Case study: Housing NSW - Outsourcing the digitisation of client files

Introduction

For the past two years Housing NSW has been outsourcing a range of back-capture digitisation projects to service providers. This case study will examine the nature of these projects, achievements and lessons learned.

Background

In 2009 Housing NSW decided to close its Broadway office and the paper-based client files stored there needed to be relocated. As Housing NSW had already been establishing business process digitisation of client files (see podcast of a presentation at a Records Managers Forum on their business process digitisation program) it was not surprising that digitisation was considered as an option for these files.

The records staff at Housing NSW were approached to conduct a pilot digitisation project involving the back-capture of a small sample of client files. This then morphed into a larger pilot project.

The pilot was so successful that several other digitisation projects ensued as other offices were closed or refurbished. Time frames were dictated by the office accommodation timeframes.

The organisation has learnt a great deal from these digitisation projects and have kindly agreed to share this knowledge.

The initial pilot

Parameters

The first thing that records staff decided to do was to consider the parameters of the pilot project. They held internal discussions considering estimates, file audits, scan criteria, logistics, EDRMS registration, accessibility issues, technical specifications and metadata requirements.

They then conducted a cost analysis to determine whether in-house scanning or outsourcing to a service provider would be the best option. The cost analysis clearly showed that outsourcing was the most suitable option.

The records staff had not been involved in digitisation before. To assist them to understand what was required and what real life issues might arise, they invited a service provider to scan five sample files. The questions raised by the service provider scanning these files helped records staff to understand what additional advice they needed to give in tender documentation e.g. whether bindings could be guillotined off, how to treat post-it notes etc.

Selection of the service provider

Once approval was gained to go to select tender in December 2009, the tender documents were finalised.

One of the issues faced initially was the unknown and variable quantity of pages within the client files. This made tendering very difficult as it was hard to develop a suitable pricing structure.

Five service providers were invited to submit responses and proposals. See the Request for tender (50kb) and Tender specification (33kb) for the pilot project. These responses were evaluated against set criteria determined by the records staff.

Being new to digitisation, the records staff admit that, at the time, they did not evaluate the responses as thoroughly as they could have. They did not have a full understanding of their requirements, and had not documented everything they needed. Over time and through ‘trial and error’ and extensive experience the evaluation criteria have been tightened significantly.

The service providers were given the specification and requirements so that they could demonstrate their ability to meet these.

The Request for tender (50kb) gave respondents an opportunity to outline their experience in providing services consistent with the specifications. They were also asked to describe how the experience they had gained with other projects would influence them in undertaking the project for Housing NSW. When evaluating this, records staff found that some service providers were not as experienced as they had initially claimed to be. Among other things, respondents were also assessed for their willingness to be subject to key performance indicators (KPIs).

The contract

In January 2010, a contract was established with the successful service provider.

The successful service provider was chosen because they were competitively priced (although not the cheapest) and, more importantly, because they demonstrated a real understanding of what was needed.

A service provider was selected who was able to meet the requirements and price model. The cost of the digitisation service included transport, scanning, the return of data, storage for a six month period and then destruction of the original paper records.

Procedures

In February and March 2010, more internal discussions were held regarding resourcing, preparation tasks and file identification. The records staff determined and drafted some procedures to encourage consistency and clarity regarding the process. To view the current version of their procedure see Client File Back scanning – Technical requirements and procedure (361kb) (used as an appendix to further tender documentation).

Quality assurance

To date, benchmarks and quality assurance requirements have not been included in the specification. However, they were included in the service provider proposal and formed part of the contract. Records staff checked random samples of images for image quality and to ensure the imaging specifications were met. Initially there were some errors detected, but these were often due to misunderstandings regarding what was required or limitations of the exiting metadata. The service provider was responsive to changes required.

Metadata

Records staff recognised the need to automate as much of the metadata collection as possible. The sheer quantity of files made manual collection impossible.

Again due to the sheer quantity of records, the organisation chose not to have individual documents and forms imaged separately. The contents of each file was imaged as one whole document but, due to size, these were separated into more manageable sections.

It was recognised that it would be valuable to identify one key document within the file – the tenancy agreement. However, the process of locating this document on the file and scanning it separately to register the document with different metadata would require additional time and the benefits were not considered to be significant in relation to the time delays. The tenancy agreement can be extracted and reregistered at a later date by staff if required.

The Tender specification (33kb) shows the metadata exported from the EDRMS and supplied to the scanning vendor and how this needed to be populated by the service provider and returned. The title metadata included the following:

Title (Free Text Part) = Client Name – XXnnnnnn – Part XX – Section X

Client Name → Surname, First Name ('Title' field and on the file label)

XXnnnnnn → 8 character field ('Record Number' field and on the file label. This is the barcode of the file.)

Part XX → The ‘Part’ number of that T file (e.g. -01 or -02 from the 'Record Number' field and on the file label)

Section X → Indicates the additional 20MB ‘section’ of a file in the series when the digitised document is larger than 20MB in size.

File format and the file size of the image are recorded automatically as part of the TRIM import process.

Other technical metadata is not linked directly to the image. However, it is relatively easy to find if the authenticity of an image is questioned. A record is kept of who the service provider was at the time and/or who scanned the batch and the dates of scanning.

Some additional metadata, to meet minimum requirements is captured when the images are uploaded to the EDRMS. Some of this is defined at the file level rather than the document level, e.g. business function/process.

Changes to the contract scope

Over the coming months the contract scope was amended and increased to cover three offices: Burwood, Maroubra and Broadway. The scanning for these offices was completed between April and August 2010.

Outcome of pilot review

The pilot digitisation project (and its subsequent expansion) was generally considered to be a success and there was an increased push to digitise more client files across other offices in the central Sydney division.

The records staff reviewed their pilot carefully and recommended that no more scanning be undertaken until a business analysis was employed to document an end-to-end solution (from the original paper file to the digital image in the EDRMS). Records staff felt that the digitisation process should be improved first.

For example:
After the first few batches of files were prepared and sent to the service provider, the service provider was requesting a lot of missing metadata. It was discovered that the internal preparation process was not very rigid. A documented procedure was needed to resolve this.

Additional projects

A business analyst was hired in September 2010 to determine an end-to–end digitisation process and solution.

Records staff updated their internal procedures, processes and specifications based on their experiences in the pilot project. The procedures include the technical requirements, roles and responsibilities and reporting.

The documentation, including the Technical specification (221kb) was also revised in anticipation of the new tender process.

One additional thing built into the procedures and documentation at this stage was the option for ad hoc digitisation which involves scanning of a specific file and all parts. The whole file is scanned.

The records staff wanted to use the original service provider for the new digitisation projects as they had an established, workable relationship with them. However, the value of the contract meant that they were bound by the procurement rules in the ICT Contract 2020.

They found that many companies who were interested in the work were simply intermediaries and the digitisation work was done by major third party service providers that are not on the ICT Contract 2020. Housing NSW did not want to conduct the project through an intermediary as time is a critical factor. Therefore they communicated through the Request for quotation (54kb) that they would not accept the outsourcing of the digitisation, but that destruction and storage could be managed by a third party. As no responses were received approval was granted to directly engage the former service provider.

A new contract was then established with the service provider to cover six new Housing offices due for refurbishment or closure by September 2011. All digitisation has now been completed.

Communication with managers and staff

The new contract involved many more offices and preparation work was again devolved to office staff. In these offices it was essential to engage management upfront and help them to understand how digitisation was likely to impact on their resources.

It was common for business managers to assume that all their staff needed to do was to box up the records and send them off for digitisation. Rather, the preparation for digitisation ideally required two full time staff.

At the outset records staff would determine the file volume, cost, resource and time estimates required. Management funding for these costs and resources had to be obtained prior to any digitisation preparation taking place. A checklist was established for records staff to assist new offices with digitisation.

Once commitment was obtained the records staff trained local office staff in the documented procedure. This included important areas like file labelling, metadata preparation, boxing, listing etc. In order for the digitisation to be successful, it was made clear to staff that these processes needed to be completed correctly and the records guarded and not removed by other staff once the boxing and metadata preparation was complete.

For example:
It was common for staff within the offices to try and retrieve files from the boxes packed ready for digitisation. The checklist includes that an email needs to be sent to all staff in the office making it clear that retrieval of files needs to be managed centrally by the person responsible for local digitisation preparation.

At each of the offices there were surprises which provided management challenges.

For example:
In some offices ‘dummy files’ had been created. For example, a local office may have sent a client file to the internal Legal Services Branch. A solicitor then photocopied the file and created their own copy, a ‘dummy file’. The solicitor conducted more business with the client which was documented on the dummy file, not on the original file. The dummy file was then sent back to the local office along with the original.

In some offices there were files that existed but were not in the EDRMS. These raised the questions: Were these duplicate files? Had the tenant left? Should these be added to the EDRMS? Records staff had to research these issues, talk to staff, compare all versions of files, determine solutions to these problems and update metadata in the EDRMS after digitisation had occurred.

The need for ad hoc scanning

Once the files at the first few offices had been digitised, other business units within Housing NSW became aware of the benefits of being able to access this information electronically. It was quickly realised that not all back-capture projects would involve bulk scanning. There was an interest in ad hoc scanning of client files for legal, subpoena, fraud or Government Information Public Access Act (GIPA) requests.

For example:
When records were requested as part of GIPA requests or for subpoenas, staff wanted to scan these files. While this was out of the norm, there have been advantages for this type of digitisation. Often where litigation arises there tends to be more litigation in the future. Having the files scanned already from back-capture digitisation projects made future litigation processes more streamlined.

As a result ad hoc scanning was added to the Request for quotation (54kb) and contractual arrangements with the service provider. To date no ad hoc scanning has taken place.

Future projects

In August/September 2011 a business decision was made to link bulk digitisation to the office accommodation schedule. With over 70 Housing NSW offices across the state, this meant that where an office was identified for refurbishment, relocation or closure, the client files stored in that office would be digitised. Additionally the file preparation for digitisation was to be outsourced to the Records Management Unit at their shared service provider (Businesslink). This meant that internal resources were not being utilised and staff that would normally be taken offline for file preparation could continue with their normal work priorities.

Twenty one offices have now been identified for digitisation work with an estimated value of $1.6 million over the next two years (approximately 73,500 files). The scope of this work will involve only client files, although administrative files may also be considered in the future. Estimates have been prepared but lack of funding and procurement issues has delayed the tender.

Note: In the earlier pilot, monitoring was not particularly difficult as few projects were undertaken concurrently. Now Housing NSW is digitising in many offices concurrently, more monitoring is needed. A checklist was developed to monitor the progress and status of individual back capture projects.
The digitisation service provider reports on a weekly basis on what has been digitised. In addition, Businesslink (who is responsible for imports into the EDRMS) keeps a log of what batches of digitised images are imported. This allows the business to keep track of batches of files sent for digitisation, whether they have actually been digitised and also whether the data has been imported into the EDRMS.

There is a push within Government procurement now to use National Disability Services (NDS) where possible for goods and services, as a result of procurement policy reform by the Department of Premier and Cabinet. Under the Ready, Willing and Able Program, no tender process is required. Approval was given to pilot one small office (2,200 files) using a disability service registered with NDS as their new service provider. This pilot, which was completed in October 2011, is currently under review.

Disposal work required

The intention with the digitisation projects involving client files is to dispose of the original paper records that were digitised. A future priority for records staff will be to set some parameters around destruction practices and start implementing destruction.

Most of Housing NSW’s client files need to be retained for 10 years after they are closed, so the organisation needs to invest funds in the longer term management of the digital images to ensure they can remain accessible and useable for the period they are required.
Samples of the files need to be retained as State archives under the current retention and disposal authority. In the General retention and disposal authority: imaged records (since superseded by the General retention and disposal authority: original or source records that have been copied) records that are required as State archives and that have been created after 2000 can be destroyed after digitisation (providing all conditions of the Authority have been met) but files created before 2000 cannot. As some client files in sample categories were created before 2000 this was identified as an issue for destruction. Housing has obtained approval from State Records to destroy digitised paper records created prior to 2000 as long as a 5% sample are retained as State archives.

Another issue with the destruction of the original paper records will be that digitisation may have been applied to one part of a file, but other parts may not have been digitised as they were located in another office.

For example:
Part 3 of a 3 part file may have been digitised while parts 1 and 2 are located in a regional office and have not been digitised.
A spreadsheet tool and documentation was developed for disposing of these types of files by records staff to extract part and location metadata from the EDRMS. The metadata is then manipulated to identify destruction issues concerning individual digitised files and their parts.

Lessons learned and successes

Lessons learned

Many lessons have been learned since the initial pilot. Some early problems arose due to a limited understanding of digitisation issues and requirements. However, as records staff became more experienced in the process, they refined their processes, remedied earlier omissions and developed comprehensive procedures.

One problem they experienced was with the metadata. The initial metadata that was imported to the EDRMS during the pilot contained a number of inaccuracies. This was because the data sent to the service provider was inaccurate. Records staff subsequently introduced and took on a data verification task for each batch of files as part of the preparation work for digitisation.

Another problem was experienced with the technological infrastructure of the organisation, which could not easily work with and manage the large image sizes.

For example:
Housing NSW experienced early problems with their offices that had smaller network bandwidth. If using the EDRMS to view some digital images (of up to 10MB) it could take a considerable amount of time to open one! Upgrades and improvements to office bandwidth and the network have made this less of a problem. The maximum image file size has since been increased to 20MB as a standard.

Successes

The digitisation project has been very favourably received by the organisation to date. The initial pilot was so successful that it as been rolled out to a number of Sydney and regional offices that are being refurbished or relocated.

Feedback from tenancy staff, the fraud unit, the legal section and managers dealing with subpoenas has been particularly favourable due to the much faster turn-around time that has resulted.

For example:
Housing NSW receives about 5-6 subpoenas a week. These have a 5 day timeframe. In the past it has taken a few days to physically retrieve the relevant client files. Digitisation has made retrieval immediate, leading to significant process improvements in these areas.

Records staff list these as the most important elements which contributed to the success of their digitisation projects:

defining requirements upfront
having a rigorous selection process with clear tender documentation and evaluation criteria
finding an excellent service provider and maintaining a good working relationship with them
analysing and documenting an end-to-end solution (from the original paper file to the digital image in the EDRMS)
working closely with their shared service provider (Businesslink)
implementing rigorous quality control assessments at the start of the digitisation process which enabled any teething problems to be rectified and ensured processes and requirements could be updated quickly
paying attention to the quality of the metadata being imported back into the EDRMS
improving processes so that they could always track where specific records were at in the process, as there was always business demand for specific files and staff needed to be able to meet this business need
communicating with managers and staff in office areas to ensure that the correct resources were allocated and staff were aware of their roles and responsibilities
having a well documented preparation process when dealing with many parts of the organisation to maintain consistency and quality
allowing for the time in their own schedules to devote to the digitisation process. Management of digitisation projects became a major component on the work schedules of records staff and it was important to devote significant amounts of time to the process in order to enable quality outcomes to be achieved.

Published June 2012

Recordkeeping Advice

Case Studies