Migrating records: managing source records after migration

The process of migrating records yields two versions of the same record. The original record, known as the source record, continues to exist after a new version of it has been created by the migration process. In the majority of circumstances, there is no business need for NSW public offices to retain both the source record and the new migrated version of the record.

The General authority for source records that have been migrated (GA48) describes the circumstances and conditions under which NSW public offices may destroy source records which have been used as the input for successful migration operations. To minimise the risks associated with migration processes this authority outlines three conditions that must be met before source records can be destroyed:

1. The migration is planned, documented and managed.
2. Pre and post migration testing proves that authentic, complete, accessible and useable records can and have been migrated.
3. Source records are kept for an appropriate length of time after the migration to enable confirmation that the migration has been successful. Determination of the specific retention period must be based on an organisational risk assessment.

Complying with these conditions will help public offices to perform migration operations that produce authentic, complete, accessible and useable records.

This guidance provides advice for NSW public offices on planning and performing migration processes so that they can meet the conditions for the destruction of source records, when appropriate. Further advice about common scenarios which may necessitate the migration of records is also available:

This advice does not apply to records that have been transferred from hard copy, or paper, into a digital format. The disposal of paper records that have been scanned or digitised is covered by the General retention and disposal authority: original or source records that have been copied (GA45).

Planning for migration

Migration operations can significantly alter record content and structure. It is therefore vital that migration operations are well planned and effectively implemented so they produce authentic, complete, accessible and useable records that the organisation can rely on to support its business and as evidence of business events and transactions.

It is also a condition for the destruction of source records that have been migrated that the migration is planned (condition #1).

Determine what is to be migrated

• Understand user behaviour and requirements by establishing how users use the system and the processes the system supports.
• Identify the types of data that users input in existing fields and whether the uses of data fields are consistent.
• Identify the data which is to be migrated, based on business requirements and the functionality of the target system.
• Identify any external data sources that the system relies on. Ensure that all relevant data stores (e.g. offline records) are identified and included in the scope of the migration.
• Determine the quality of the data and consider whether data cleansing is required.
• Consider destroying records due for destruction which are no longer required for business purposes. This reduces the quantity of records to be migrated.
• Identify relationships between records which need to be preserved. Process mapping may help identify relationships between records, processes and stakeholders.
• Identify the hardware, software and format requirements of the records requiring migration. System and configuration documentation is a useful source for understanding technology dependencies.
• Identify the characteristics that are critical to the records’ meaning, use or organisational value (‘essential characteristics’) – these must be preserved by the chosen migration strategy. Essential characteristics will differ according to record type and the business purpose served by the record.
• Identify the metadata which is essential to the records’ integrity, trustworthiness and authenticity and the metadata relationships which must be preserved during migration. For example, when migrating records from an electronic document and records management system (EDRMS) the following types of metadata would be essential:
  • structural (e.g. within a document, such as a document containing a linked spreadsheet or images)
  • between records (e.g. between records documenting related aspects of business)
  • between records and/or ‘containers’ (e.g. documents aggregated to files/folders)
  • between records and other entities (e.g. between records and creating agents)
  • between records and control tools (e.g. business classification schemes, retention and disposal authorities, access and security controls and mandates).
• Identify any metadata elements that have no ongoing business or accountability relevance for the organisation. If you decide not to migrate certain metadata elements, these decisions must be fully considered and comprehensively documented.
• Complete a metadata mapping between the original system and the target system to ensure that all necessary metadata elements, their corresponding functionality and relevant business rules can be migrated between systems.

Develop an appropriate migration plan

The migration plan details the scope of the migration and how the migration will be undertaken. It defines:

• the structure, volume and quality of data being migrated
• migration strategy and methodology
• required actions including format transformations, source to target mapping and metadata mapping
• how the migration will be tested and validated (pre and post migration testing)
• risks and mitigation strategies, including recovery plans, if needed
• actions on source records and decommissioning planning
• roles and responsibilities, including who is responsible for migration validation activities.

The migration plan is a key document in any migration project. Decisions which result in adjustments to the migration project should be reflected in this document.

Maintaining authentic, complete, accessible and useable records

Successful migration operations will produce authentic, complete, accessible and useable records.

It is a condition for the destruction of source records that have been migrated that pre and post migration testing proves that authentic, complete, accessible and useable records can and have been migrated (condition #2).

Pre migration testing

Pre migration testing is an important activity that can be used to validate actions set out in the migration plan. This can be an iterative process with changes made to the migration plan based on the results of testing.

• To test the proposed migration strategy, perform a test migration on a small sample of duplicated records. Take note of exception and error reports and revise the migration methodology accordingly.
• Ask relevant technical and business personnel to verify that the resulting migrated records are complete, accessible and useable, and the migration strategy is appropriate.
• If adverse effects are noted in the migrated records, a revised migration strategy must be devised. This strategy should also be subject to pre migration testing.
• Document the results of pre migration testing. This documentation should provide the basis for the final migration plan.
• Once pre migration testing is complete, the pre migration testing and the finalised migration plan should be validated in accordance with the governance arrangements for the migration project.

Post migration testing

Post migration testing must ensure that:

• all records identified for migration have been migrated
• all necessary functionality and essential characteristics have been retained
• users are satisfied with the authenticity, completeness, accessibility and useability of the migrated records.

Post migration testing does not have to be performed individually on every document or data field – sampling may be appropriate. The sampling methodology needs to be appropriate either statistically or commensurate with the risk/business need for the records (e.g. for a document-based system, the number can be proportionate to the number of documents migrated and should, where relevant, include different document types across a range of years).

• Document the results of post migration testing.

• Once post migration testing is complete, the migration process should be validated in accordance with the governance arrangements for the migration project.

Determining an appropriate retention period for source records

Public offices must keep the source records for an appropriate length of time after a migration to enable confirmation that the migration has been successful. Retaining the source records in this way will:

• allow for any unforeseen issues associated with the migration that may emerge following post migration testing to be identified and rectified
• enable the migration to be repeated if it is discovered that some or all of the migrated records do not meet quality control standards or business requirements.

It is also a condition for the destruction of source records that have been migrated (condition #3).

The identified retention period should be calculated from the conclusion of successful post migration testing, where the migration and all outstanding issues associated with it have been validated in accordance with the governance arrangements for the migration project.

Risk assessment

Public offices must determine an appropriate retention period for source records based on an organisational risk assessment. This is a condition for the destruction of source records that have been migrated (condition #3).

Organisations should conduct this risk assessment in accordance with their internal risk management framework (as required by the Internal Audit and Risk Management Policy for the NSW Public Sector). The Risk Management Toolkit developed by NSW Treasury provides advice on risk management processes and includes risk assessment templates.

The risk assessment should consider the:

• business purpose of the records and the risks associated with this business
• value of the records
• potential business, financial and legal implications should the records be lost or corrupted
• size and complexity of the migration and the likelihood of problems associated with it
• complexity of the records being migrated
• capacities of the target system and the possibility that all aspects of this system and its impact on the records are not fully understood at the time of migration
• nature of the metadata that is not being migrated (where relevant).

For some records of long term value or relating to high risk functions and activities, retaining the source records for significant periods of time may be an appropriate risk mitigation strategy. However the best way to guarantee that records are authentic, complete, accessible and useable is to ensure that the migration planning, testing and documentation is robust. Retaining source records for long periods of time will not substitute for performing those processes adequately.

Post migration testing

As well as the outcomes of the risk assessment, public offices should also consider the quality and extent of post migration testing when determining an appropriate retention period for source records:

• If a project included rigorous, extensive post migration testing it may be appropriate to retain source records for a short period of time.
• Conversely, if a project included relatively ‘light touch’ post migration testing it may be advisable to retain source records for a longer period.

Public offices should always err on the side of caution and if any problems or concerns with the migrated records are noted, such as corruption of portions of the record, or loss of information or distortions in the records caused by the new capacities or functionality of the target system, then the initial identified retention period must be extended. Again, this additional retention period should begin to be calculated from the conclusion of successful post migration testing, where the repeated migration and all outstanding issues associated with it have been signed off by the Chief Information Officer or appropriate designated equivalent.

Incidental quality assurance

Public offices should also consider the extent of ‘incidental’ quality assurance when determining an appropriate retention period for source records.

If data is migrated and then immediately and frequently used in business as usual processes, anomalies and unforeseen issues with the migrated data are more likely to be found, and found within a shorter period of time. In these scenarios, and subject to the findings of the risk assessment, it may be appropriate to retain source records for a short period of time.

Retaining source records

The General authority for source records that have been migrated (GA48) does not require public offices to destroy source records that have been migrated.

In some situations, such as migrations between public offices, it may facilitate business processes for the transferring organisation to retain a reference copy of the source records for the length of their retention period, as specified in the relevant retention and disposal authority. Additionally, in high risk business environments, public offices may determine that retaining source records as a managed copy of the new official records is the best mitigation strategy.

However, cases where public offices need to keep source records indefinitely should be very rare. Virtually all source records should be able to be destroyed after their retention period has expired.

Where public offices decide to continue to retain the source records as well as the migrated records, they must establish and enforce cut off periods between use of the legacy system and the current system. This is necessary to ensure there is a means of establishing which version is to be relied on and used as the ‘official’ record going forward.

Documenting migration

In order to account for the outcomes of migration processes and maintain authentic records, public offices must document the entire migration process and associated project planning. It is also a condition for the destruction of source records that have been migrated that the migration is documented (condition #1).

Migration should be undertaken in accordance with best practice project management methodologies. Comprehensive records of the project should document:

• the factors triggering the migration project
• the records being migrated and the business functions and activities they relate to
• the identified essential characteristics of the records
• comparisons between original system functionality and target system functionality
• technical requirements of the original and target systems
• all system configurations, including metadata definitions and mappings
• all decisions, including decisions not to migrate certain metadata components of a record
• risk assessments
• any disposal and data clean-up performed prior to migration, including records of decisions and the processes undertaken
• management approval of migration plans
• any variations to plans
• technical reports on the migration itself, including date and time of the migration and all personnel involved
• details of all testing, pre and post migration, including comprehensive test reports which confirm the target system is functioning successfully and that all expected data and metadata has been migrated
• any necessary variation in records structure, design, metadata, format or content that will result or has resulted from the migration.

This documentation provides essential context to records that have undergone migration, and will be indispensable should the records be required for legal proceedings where their integrity and authenticity needs to be assured.

The Standard on Records Management requires NSW public offices to document the disposal of records (requirement 3.7). NSW public offices must document the destruction of source records and keep this documentation in accordance with the general retention and disposal authority for administrative records. This documentation might take the form of:

• metadata at the aggregate or system level which states that the previous version of this record group or system was destroyed on the specified date, by the identified authorised officer, in accordance with the conditions outlined in this authority
• metadata at a more granular level that documents the destruction of the previous version of specific files in accordance with necessary disposal requirements
• a final migration report that outlines the destruction of source records, providing enough detail to identify all records or groups of records that have been destroyed
• standard destruction approval forms used to authorise the destruction of records within the organisation.

NSW public offices must also document the transfer of records (e.g. to another organisation or, for State archives, to Museums of History NSW) and keep this documentation in accordance with the general retention and disposal authority for administrative records. This documentation must identify:

• the records that have been transferred
• where the records were transferred to
• the date of the transfer.

Acknowledgements and further resources

AS/NZS ISO 13008:2014 Information and documentation – Digital records conversion and migration process

Public Record Office Victoria, Decommissioning, www.prov.vic.gov.au/recordkeeping-government/a-z-topics/decommissioning (viewed 1 February 2018)

Queensland State Archives, Managing public records when decommissioning business systems methodology, April 2017, www.forgov.qld.gov.au/manage-records-when-decommissioning-business-systems (viewed 1 February 2018)

Queensland State Archives, Migrate digital records, November 2017, www.forgov.qld.gov.au/migrate-digital-records (viewed 1 February 2018)

Published April 2018