Data Conversion,Migration and interface ..Why important

Do you know how many ways; we can enter the data into oracle application. Most of us can guess three different ways as:

• The Data can be entered using the application Screens.
• The data can be entered using Oracle's Open System Interface.
• The data can be stored in the database table directly

But those who works in some complex business environment may figure out some of more like:

• 3rd Party Software (for the third options)
• Taviz (formerly SmartDB) which is EAI tool.
• Crossroads
• See Beyond (formerly STC)
• Vitria
• Data Loader: They have macro enabled spreadsheet kind of tool
• More4apps

And there are many more, but most of these are used for master data, and few cases for transaction data via Open interface if available.
Importance of data conversion/migration and interfaces within any ERP implementation project can't be ignored. Since ERP mostly deals with data which finally leads into Information , thus it is equally important to understand the aspect how "data" is important in any ERP system specailly in implementation phase, no matter how simple and unified operation is. Since I been involved in some big transformation oracle apps project thus It is a absolute a good cause to share some information about integration touch point, conversion/migration and interface development to someone who is very new to ERP world as well as Oracle application.
Let's start with some common situation, we have three cases,

1. The Customer is running there some home grown IT application which cater most of the company need. Now management has decided to go for any ERP solutions, then the question what will happen for data which is already in the existing application?
2. Another situation is already using ERP
   a. They want to upgrade to higher version…presuming the structure of some table get changed? Lets say 10.7 to 11i
   b. The company is acquired or merged with some other company, and the whole data need to move into the parent or child company .
   c. They want to enable some additional modules within existing application.
3. There are few data interacting with both the cases irrespective of database technology for where data is coming and going based out of need.

The answer of the 1 is data migration and 2 is more pronounced as data conversion where as thirds is popularly known as Interface. The ways these are working haven't much difference but it is more important to understand definition and need. I never found any big difference between migration/conversion unless there is huge transformation of data, the only things figure out is conversion might required some less steps to perform, as assumption is that set up related stuff been already been taken care before execution of activity.
Let's understand like this: Data Migration as a process of moving required (and most often very large) volumes of data from our clients' existing systems to new systems. Existing systems can be anything from custom-built IT infrastructures to spreadsheets and standalone databases. Data conversion can be defined as a process of converting data from one structural form to another to suit the requirements of the system to which it is migrated.
Lets take a deep drive to understand better:
Why conversion/Migration is more important in ERP?
Before Go-Live in the production environment the required master data, the open transaction data and the historical transaction data need to be imported from the old legacy applications to Oracle Applications. Since data structure and data design in legacy systems are different from those of Oracle Applications, data need to be messaged/ converted satisfying the business rules to suite the Oracle requirement. Initial data can be migrated by any other means as discussed above depending upon cetain paramater like Volumn, usage, complexity , business rule etc..
How we Define Data Conversion

• Process where existing data from the client's old system is extracted, cleansed, formatted, and installed into a new system.
• These can be manual or automated.
• The big difference is that these are One-time only process that requires extensive testing and preparation.
• They must be executed and performed before a system goes into production.

What Is An Interface then

• These are programs for connection between Two Systems In Order To Synchronize the Data.
• They can be Manual, Batch or Real-Time.
• Used Repeatedly and Should Therefore Be Designed and Constructed In the Most Efficient Manner Possible.
• These can Be Triggered by an Event (Such As Running A Concurrent Program) Or It Can Be Scheduled To Run At A Certain Time.
• Can Be Very Costly To Construct And Maintain.

Does the conversion/migration/interface have Life Cycle
Yes, they have, because they have a significant efforts required in development and design and implementation

• Functional Designer works with business owners to determine the data mapping and complete the functional design using the Design Templates.
• If the interface/conversion is automated, the Technical Designer converts functional requirements into technical specs for the construction of the interface programs.
• The developer uses the functional and technical designs to build and test the interface/conversion programs.
• More rounds of testing are done until the interface/conversion is migrated to the production environment for deployment.

Conversion is assumed as one time activity but never looks like small activity which can be performed with couple of days.

How conversion and interface differ?
There are good numbers of parameter on which they can be categorize. Take few of them:

• Frequency
  • conversions are a one time event
  • interfaces are ongoing
• Occurrence in the project timeline
  • conversions executed before production
  • interfaces executed during production
• Manner of execution
  • conversions are batch
  • interfaces may be batch or real time
• Complexity
  • Conversion does have very complex, its totally depends upon the data mapping activity.
  • coordinating with other systems make interfaces more complex
• Maintenance
  • Maintence of interface is bit cost intensive task.

Interface Type
You have learned how interface is differ from Conversion/Migration. Now lets take few types of interfaces:
Normally in any system , there are two kinds of interface as:

Inbound Interfaces

• An inbound interface receives data from one system (legacy) and inserts into Oracle open interface tables.
• A typical inbound interface would follow these steps:
  1. Extract data from legacy system into a flat file.
  2. Use SQL*Loader or equivalent tool to upload information into a temporary table.
  3. Write a PL/SQL program to take data from the temp table and insert into the Open Interface Tables.
  4. Through the concurrent manager in Oracle Applications, run the standard Oracle Interface program to transform interface tables into Oracle data.

Outbound Interfaces
o An outbound interface takes data from Oracle tables and inserts it into an external system (via tables or flat file).
o A typical outbound interface would follow these steps:
- Write a PL/SQL program to extract data from Oracle base tables into a flat file.
- Use a custom program to read that data and post it into the legacy system

Do we have some other standard way to do interface?

• Open Interface is a table based interface registered as a concurrent program
  • process records in batches.
  • spawned(Pro-C) or PL/SQL based programs.
• API (Application Program Interface) is a parameter based stored procedure
  • directly impacts base database tables.
  • may be called from Oracle open interfaces,Forms, Reports.
• EDI (Electronic Data Interchange) uses industry standard data definitions(US/ANSI/X.12) for transmission of documents such as PO's, Invoices, Sales Order, etc.Oracle provides some EDI transactions through EDI Gateway.(
• Enterprise Application Integration (EAI) solutions are often used when complex integration requirements exist.

What Is An Open Interface Table (OIT)?

• For inbound interfaces, the interface table is the intermediary table where data from the source application temporarily resides until it is validated and processed into an Oracle base table through a standard import concurrent program.
• Open Interface Tables are standard Oracle tables.
  • Oracle uses OITs to provide a simple interface to Oracle base tables.
  • Oracle has list of all the open interface that oracle offered in there product.

Oracle Interface Program

• Most Oracle modules have standard import programs (concurrent processes) to facilitate custom inbound interfaces. The specific processing performed varies by application.
• These programs pull data from the open interface tables, validate the data, and then insert into one or more Oracle base tables.
• Upon successful completion of processing, the program deletes the processed rows from the interface table or marks them as completed.
• Depending on the import, errors can be viewed in various ways (exception reports, error tables, forms, etc…).

Examples of standard import programs:

• GL: Journal Import
• AP: Payables Open Interface
• AR: Customer Interface
• INV : Item Import
• AR - Autoinvoice

Ok, thats is all about Conversion and Interfaces briefing. I will write some more for Tools used for Conversion/Interface and will discuss some granular details about a conversion/migration project and share some information about how and where AIM's documents fits into conversion and Migration projects. So watch out this space for some more stuff for conversions.. Till than..your comment and requset you to share some information relaeted to these areas