One of the challenges when implementing a brand new software application is the migration of data from the outgoing system.

For the untrained eye this looks like a straightforward undertaking because it is not difficult to extract data from the outgoing system to a standard format using a simple query tool, and it is not difficult to take in data from this standard format into the new software application using utility software.

However, the challenge is not so much about the movement of data between the old and the new environments, but in the mismatch between the data layout of the two systems, the poor quality of data within the old system, the difference in interpretation of the data by the two systems, the missing data that is necessary for the new system to operate as well as the timing of the migration, which is different for master and transaction data. Let us consider some of these issues one by one.

It is common to find that the outgoing system includes items of data that have nowhere to be stored in the new system. This could be the case where functionality in the old system is not entirely present in the new system, or when the new system deals with specific functional issues in a different manner. This issue is typically more acute when the organisation is moving from a bespoke system to a package.

It is even more common for the new system to require data that is not found in the outgoing system. Most systems today will have specific fields that need to be populated in order for a record to be completed. These are the mandatory fields and they can be one or many depending on how the new system has been configured. Without populating these mandatory fields there will be records that cannot be migrated and could be rejected or lost.

Another common problem is the poor quality of data in the old system. Poor quality data will impact systems differently and one can expect the incoming system to fail in some aspects due to inconsistencies in the data. A related problem is the misinterpretation of data resulting in data being placed into the wrong fields in the new system. Errors of this nature could be difficult to unravel once the new system goes live and will cause havoc with reporting.

Therefore how does one manage the migration of data to the incoming system?

In the first instance, create a data dictionary. This is a catalogue of all the items of data you have in the outgoing system. This will enable you to understand the data you have and the rules to which it needs to adhere. It defines the data itself, what it represents, the validation rules, the values it can contain, etc.

With a data dictionary in hand, one can start to assess the quality of the data in the outgoing system and take decisions on the action that needs to be taken to improve it. For example, if the dictionary dictates that a particular field can contain the values ‘X’, ‘B’ or ‘T’, what action would you need to take if some of the data contains the value ‘P’?

The data elements need to be mapped to fields in the new system. In some cases, the data will need to be changed or transformed to fit the new data field; for example, an address that fits into a single field in the old system may need to be transformed to multiple fields including ‘city’, ‘postcode’ and ‘country’. It is important to understand how each field will be treated to prevent any misunderstandings and to capture any errors before the data is applied to the new system.

Errors of this nature could be difficult to unravel once the new system goes live and will cause havoc with reporting

There will be data that is not transferred to the new system as it cannot be accommodated in the current configuration. Some of this data may not be needed in the system but it may be important for reference or for reporting. Therefore, a decision needs to be made on whether to keep this data and where to keep it.

The new system may need data that is not available in the outgoing system. Identifying this data requires an understanding of the new system and access to the data dictionary. Once the gaps are identified decide how to source the missing data which may require the establishment of default values or data entry. Assess where and when the data entry is to take place keeping in mind that data would need to be kept up to date until the new system goes live.

Migration programmes can then be written and tested and the entire migration process also needs testing as multiple migration iterations will be required as migrated data is used for testing of the system by end-users before it is used for the new system to go live.

Clearly, the complexity of data migration is such that sound project management principles need to be adhered to throughout the process.

ericmuscat@kpmg.com.mt

Eric Muscat is the partner responsible for IT advisory services at KPMG Malta.