Principles for implementing metadata for records and information

The effective implementation of metadata should be based on the following principles:

Metadata requirements should be considered as part of appraisal. Your organisation should identify what metadata is necessary for the creation, capture and management of authoritative records and information, and what metadata supports organisational recordkeeping and business requirements.
Metadata is scalable. Your organisation should determine which levels of metadata can best meet its various business needs.
Metadata should be described, documented and managed. Your organisation can use metadata schemas and encoding schemes to promote the entry of meaningful, standardised and consistent metadata.
Metadata is dynamic and grows over time. Your organisation should be aware that records and information will continue to accrue metadata throughout their existence.
Metadata should be persistently linked with records and information. Your organisation should ensure that metadata is linked with the records and information to which it relates when they are transferred out of their original creating environment and through subsequent migrations.
Metadata should be managed as a record. Your organisation should document how it has configured and applied the metadata in its systems.

Metadata requirements should be considered as part of appraisal

Metadata in digital systems is abundant and permeating. The role of the records and information management professional is to identify what metadata in business applications, systems and cloud environments is necessary for the creation, capture and management of authoritative records and information, and what metadata supports organisational recordkeeping and business requirements.

Appraisal is the process of evaluating business activities and work processes to determine which records and information need to be created and captured and how long the records and information need to be kept. The results of appraisal can be used for a range of purposes, including the identification and definition of requirements for metadata for records and information. These requirements may include requirements for metadata generation or capture, and requirements for maintaining particular metadata through system changes and over time.

Metadata decision-making opportunities

Your organisation is likely to make decisions about what metadata should be created, captured and kept, and how, at certain key moments. These include:

system design
system selection
system implementation
system configuration
commencement of a service agreement
system upgrade
system migration.

Example: Appraising metadata requirements when establishing contracts for cloud-based services

The NSW Government Cloud Policy notes that when establishing contracts for cloud-based services, agencies must be able to guarantee the accuracy, integrity and reliability of data to ensure its ongoing availability. This includes including provisions for the safe return/transfer of data should the cloud service provider be the subject of a takeover, and provisions specifying what will happen to the data at the termination of the agreement (e.g. transfer to a new provider, returned to the agency, permanently deleted). These provisions should consider metadata requirements, in particular what metadata needs to be returned to the agency along with the content to show how it was managed and what processes were performed on it.

Requirements for metadata are context-dependent. Similar or identical work processes may have different metadata requirements, depending on the nature of the business being conducted.

Example: Different types of records and information need different contextual metadata

Systems designed to control and manage records and information, such as electronic document and records management systems (EDRMS), enterprise content management (ECM) systems, web content management systems and digital asset management systems, can be configured to capture and maintain a range of contextual metadata. Your organisation will need to determine the specific contextual metadata it needs to maintain for different types of records and information. For example:

If your organisation maintains documents in an EDRMS or ECM system, you may identify the need to capture information about who created or received the document, the date it was created, sent, received and captured into the system, the creating application, who edited, viewed, approved and deleted the document and when, who had access to the document at specific points in time etc.
If your organisation maintains images in a digital asset management system, you may identify the need to capture information about who took the picture, what it shows, when it was taken, what incident or event it relates to, who owns the picture, how it was used in business processes, what file format it is in, its size etc.

This metadata can be crucial for understanding and being able to use records and information over time.

Further guidance on implementing metadata in an EDRMS is available in FAQs about EDRMS.

Your organisation can implement metadata at different levels: records and information of greater value or with higher associated risks will likely warrant more detailed metadata.

Your organisation should regularly review its identified metadata requirements as part of the recurrent process of appraisal.

Metadata is scalable

Metadata can be used to describe records and information at different levels of aggregation. For example, metadata can be used to describe:

individual records at the document level in a case management system – you might apply titles and unique identifiers to each individual record in your system
groups of records in an EDRMS – you might apply retention and disposal rules at the file level to facilitate the sentencing and disposal of records and information that relate to the same matter or transaction
systems – you might apply access rules metadata to an entire system with well-defined data types, such as a personnel system or finance system, to standardise management actions and access control across the system.

You will need to determine which levels of metadata can best meet the various business needs of your organisation.

Metadata can also be used to describe things other than records and information, such as people, workgroups, organisations, business transactions, business activities, business functions, laws, regulations, policies and business rules. This type of metadata can provide the context in which records and information are created and used, enabling them to be interpreted, understood, validated, trusted and reused. You should determine whether metadata descriptions of any of these entities would be of use to your organisation.

Metadata should be described, documented and managed

To be a key organisational information resource, metadata has to be well specified and well managed, both immediately and in the long term. It cannot be created on an ad hoc basis.

Using metadata schemas and encoding schemes promotes the entry of meaningful, standardised and consistent metadata. This contributes to data quality. It can also assist in data sharing and reuse.

Metadata schemas

Metadata schemas define the overall structure of the metadata assigned to records and explain the meaning of particular metadata values. Metadata schemas:

specify the particular metadata fields that will be used in the system
provide a definition for each of the fields, indicating what can and cannot be applied within them
identify the data that will be used to populate each field if metadata will be collected automatically from other systems etc.
identify the encoding schemes or ‘picklists’ that will be used to provide data values in the system if metadata collection cannot be automated.

Encoding schemes

Encoding schemes define the content of specific metadata values. Encoding schemes:

set out what information can be entered against a value (one or more of a list of set entries or numbers, or free text, or a combination)
specify how that information is arranged
define which symbols (e.g. dashes, commas, colons) are used to separate the individual chunks of information in each value.

Use of encoding schemes:

promote standardisation, consistency and accuracy
make it easier for users to automatically apply metadata values
make searching and browsing easier and more efficient
facilitate metadata reuse in other business areas
enable automation of records and information management processes, e.g. disposal
can be critical for enabling interoperability between systems
contribute to system sustainability.

An example of an encoding scheme is the Document Form Metadata Scheme which establishes record types common to most organisations in the NSW public sector. Other examples include lists of teams within a business unit or medical procedures within a patient management system.

Good encoding schemes can add value to a system and dramatically improve its useability. You need to spend time and effort to develop good encoding schemes.

If your organisation is developing and implementing a custom-built business system, you will need to determine a comprehensive metadata schema that meets specific business needs. If your organisation is implementing an off-the-shelf software package, you will need to create a schema that identifies and clearly defines the specific elements of that system that should be used for your particular business purposes. If your organisation takes up an as service offering for a system that performs or supports business processes, you will need to work with the service provider to identify key metadata that maps to your requirements. Understanding and documenting, as appropriate, the metadata within a service offering is important to managing the records and information through transition out of the service arrangement.

In all of these situations you need to make sure you have a standardised set of metadata elements.

The following table highlights the consequences of standardising metadata creation and use in an EDRMS:

Scenario	Organisation A implemented an EDRMS with over 500 possible data fields. Organisation A did not recommend standard elements for its staff to use when registering records. Organisation A did not define elements for staff or explain what each element should and should not be used for.	Organisation B implemented an EDRMS with over 500 possible data fields. Organisation B issued procedures to staff specifying the specific elements that should be used to register records. Organisation B developed business rules for the system that identified the specific relationships, access rules and disposal requirements that could be applied in the system. Organisation B employed a range of encoding schemes that identified the specific values that could be used in a range of fields.
Outcome	There was no standardisation in the data fields used by staff when registering records – for example, some staff had used the ‘Creator’ field while others had used the ‘Author’ field to capture information about a record’s creator; some used ‘Provenance’ while others used ‘Creating organisation’ to capture information about the controlling agency. Multiple options had been used for all commonly used metadata values.	Organisation B had clearly defined documents that outlined all the fields used in the EDRMS and rules that defined how these fields were used. Regular monitoring by records staff had ensured that staff complied with these rules. Data in the EDRMS was consistent and well defined.
Consequences	Migrating to the next generation of the EDRMS was impeded as simple metadata mapping and translations were impossible. A lot of data cleansing was necessary and multiple metadata mappings were required. A lot of the metadata could not be migrated because the costs of translating multiple fields and values into specific fields in the target system were considered too excessive. This has ongoing implications for the authenticity and useability of the records which could potentially have legal and cost implications for the organisation.	Migrating to the next generation of the EDRMS was facilitated by the relatively simple one to one mappings which could be achieved for all record types. There was minimal data cleansing and all data was successfully migrated from one system to another. Because of the straightforward nature of the migration, system administrators had time to concentrate on configuring the new EDRMS, implementing business rules and making improvements to system functionality and performance.

Scenario

Organisation A implemented an EDRMS with over 500 possible data fields.

Organisation A did not recommend standard elements for its staff to use when registering records.

Organisation A did not define elements for staff or explain what each element should and should not be used for.

Organisation B implemented an EDRMS with over 500 possible data fields.

Organisation B issued procedures to staff specifying the specific elements that should be used to register records.

Organisation B developed business rules for the system that identified the specific relationships, access rules and disposal requirements that could be applied in the system.

Organisation B employed a range of encoding schemes that identified the specific values that could be used in a range of fields.

Outcome

There was no standardisation in the data fields used by staff when registering records – for example, some staff had used the ‘Creator’ field while others had used the ‘Author’ field to capture information about a record’s creator; some used ‘Provenance’ while others used ‘Creating organisation’ to capture information about the controlling agency.

Multiple options had been used for all commonly used metadata values.

Organisation B had clearly defined documents that outlined all the fields used in the EDRMS and rules that defined how these fields were used.

Regular monitoring by records staff had ensured that staff complied with these rules.

Data in the EDRMS was consistent and well defined.

Consequences

Migrating to the next generation of the EDRMS was impeded as simple metadata mapping and translations were impossible.

A lot of data cleansing was necessary and multiple metadata mappings were required.

A lot of the metadata could not be migrated because the costs of translating multiple fields and values into specific fields in the target system were considered too excessive.

This has ongoing implications for the authenticity and useability of the records which could potentially have legal and cost implications for the organisation.

Migrating to the next generation of the EDRMS was facilitated by the relatively simple one to one mappings which could be achieved for all record types.

There was minimal data cleansing and all data was successfully migrated from one system to another.
Because of the straightforward nature of the migration, system administrators had time to concentrate on configuring the new EDRMS, implementing business rules and making improvements to system functionality and performance.

Tip: Use or adapt existing metadata schemas where possible

Using or adapting an existing schema presents a range of benefits:

The schema and its usage guidelines have already been developed, saving your organisation time and effort.
An existing schema is likely to have a community of users, which means there will be access to help and advice about how best to use it.
Using an established schema facilitates interoperability between systems and in the event of administrative change.

AS /NZS 5478: 2015 Recordkeeping metadata property reference set provides a reference set of recordkeeping metadata to support systems interoperability and records sustainability.

It is important to note that existing schemas may not specifically address all of your organisation’s identified requirements for metadata for managing records and information. Your organisation will need to adapt or add to these schemas to ensure that they include all metadata necessary for effectively managing your records and information.

Tip: Automatically harvest metadata where possible

Automating as much metadata capture as possible reduces the amount of time your staff must spend on manually entering metadata and enables more consistent metadata capture.

Example: Digital images

If an image has been created by a digital camera, it is likely that the camera has also written a certain amount of information about the digital capture into the file header. This may include information about the camera make and model, its settings and the date the photograph was taken. This metadata can be automatically extracted, saving a user from having to enter it manually.

Tip: Reuse metadata where possible

You can also improve organisational efficiency if metadata created for one business purpose is shared with other applications for other business purposes. Metadata reuse in this way saves money and can dramatically improve how business is performed in your organisation. Mapping and standardising your metadata is critical to enabling this type of functionality.

Example: Reusing personnel data

Human resource management systems contain information about the people who work in your organisation, including which business unit they work in and their position title. Your organisation could establish links between its human resource management system and EDRMS or ECM system to enable the EDRMS or ECM system to reuse some of this personnel-related metadata to provide context about who created records and the business context in which they were created.

Tip: Consider business needs when developing metadata schemas

The metadata schemas you develop need to be based on specific knowledge of the business that needs to be performed within the system. Talk to staff and perform system assessments so that you know what is currently done and so that you can gain an awareness of how things can be done better.

For example:

Is information retrieval an issue? What metadata fields could be included to address this problem?
Does the business unit need to run standard reports? What data elements are necessary to generate these?
What system structure and functionality that can be enabled by metadata are required?
What information access or restrictions need to be enabled across a system?
What privacy management or data collection and management requirements can be met through metadata design and application?
What metadata is needed to account for or authenticate transactions or to enable employees to understand where a process is up to? Can the identified metadata tell the story of how the information has been used, processed and acted upon across the organisation?

Metadata schemas do not have to be uniform across your organisation, as business needs may differ between business units. Remember to support individual business requirements wherever possible through the inclusion of specific metadata fields.

Tip: Ensure users understand your metadata schema

Systems are only as good as the people who use them. You could spend significant amounts of time and money developing standardised metadata schema for business systems but if people do not implement these as intended, all the efforts involved in their development will be wasted. Spend time and money explaining the roles of the different metadata fields in your business systems and how the particular encoding schemes that provide values for these fields operate. This type of work may be difficult and repetitive but it is a critical component of the effective operation of your systems and also for the long term sustainability and integrity of the system.

Tip: Manage metadata schema and encoding schemes as records

Your organisation should regularly review metadata schema and encoding schemes to ensure they remain relevant and continue to meet needs. Versions of schema and encoding schemes that have been superseded should be retained and managed as records.

Keep your metadata up to date

Business environments are often subject to regular change. It is important to update your encoding schemes when:

workgroups are renamed
business processes change
business locations change
business rules change
employees change roles etc.

Be aware, however, that existing metadata should not be changed when circumstances change. So if an employee changes roles, any metadata for records and information they created in their previous role should reflect their role at the time they were created. If the metadata was changed, it would no longer provide an accurate representation of the business as it was when it was transacted.

Metadata is dynamic and grows over time

Metadata is dynamic and grows over time as records and information are used and managed. Initially, metadata defines a digital object at its point of capture, fixing the object into its business context and establishing management control over it. During the object’s existence it will accrue additional layers of metadata because of new roles in other business or usage contexts. This means that metadata continues to accrue information relating to the context of the records management and business processes in which an object is used, and to structural changes to the object or its appearance.

Metadata should be persistently linked with records and information

For the lifespan of the records and information to which it relates, metadata should be kept and kept in context. It should be persistently linked with the records and information to which it relates, including when they are transferred out of their original creating environment and through subsequent migrations.

Metadata should be preserved during migration

The migration of records and information between systems is a recurrent activity for organisations that conduct their business digitally. It is critical for the accuracy and reliability of records and information that your organisation identifies the metadata that should be sustained through system transitions and data migrations, and ensures that it can be successfully imported and exported in successive digital environments. Connections between records and information and this metadata should be maintained during migrations of hardware, software or technology.

It is important to think very broadly about what system metadata documents the transactional context of what was done in the system(the who, what, when and why), as well as who had the right to see and take actions in the system. This metadata can be critical for providing essential context for records and information.

For further guidance on preserving metadata during migrations, see Migrating records: managing source records after migration.

The content of records and information and their associated metadata may be managed in:

multiple coexistent locations and systems, or
a single location and system.

Keeping records and information in recordkeeping systems can provide for the creation and maintenance of logical relationships between content, point of capture metadata and process metadata. Alternatively, for records and information that are not kept in recordkeeping systems, logical relationships or linkages between content and metadata may be created and maintained using automated or manual processes.

Example: Persistently linking metadata with documents imported to a SharePoint environment

If records are imported to a SharePoint 2010 implementation from other systems, certain point of capture metadata may be overwritten. For example, if records are imported from shared or network drives, the date the record was created will be replaced with the date the record was imported to SharePoint, and the name of the record’s creator will be replaced with the name of the person who imported the record. Other information captured within the record’s properties, such as date last modified, may also be lost.

The loss of this metadata reduces the authenticity of the records: they can no longer be proven to have been created or sent by the person that created or sent them, or to have been created or sent at the time they were created or sent.

To mitigate the loss of this metadata, point of capture metadata can be bulk imported separately from records and then re-linked to each record. Alternatively, add-on software can be implemented to manage imports so that the point of capture metadata for each record is retained.

Metadata should be managed as a record

Metadata itself is a key record. Through time it will attest to the integrity of records and information by documenting their context and management history.

It is essential to document how your organisation has configured and applied the metadata in its systems. You should document:

what metadata fields you have implemented
which fields you have not implemented
what encoding schemes have been applied
what inheritance has been enabled
what automated metadata capture is in place
what default metadata values have been applied
what recordkeeping tools you have integrated
any dependencies that may exist between metadata fields
any automated capture of metadata from other business systems
any changes that you subsequently make to your configuration arrangements.

This information is very important for ongoing systems management and for facilitating system migration in the future.

Documenting configuration is really important and making point in time records about this as systems change is really important too.

Metadata should itself be managed as a record:

Metadata should be protected from loss or unauthorised deletion.
Metadata should be retained or destroyed in accordance with authorised retention and disposal authorities.
Access to metadata should be controlled.

If records and information are destroyed in accordance with authorised retention and disposal authorities, certain metadata should continue to be maintained. Retention and disposal authorities relevant to your organisation will outline the specific retention requirements that apply to the different forms of metadata maintained in your organisation. General retention and disposal authority: administrative records contains retention requirements for some metadata.

Published February 2015 / Revised November 2015

Recordkeeping Advice

Metadata for Records & Information