Many of the TSG customers and prospects on FileNet are struggling with old versions of FileNet that have lived for decades with millions and even billions of documents. For clients looking to reduce the cost of supporting FileNet as soon as possible, TSG can leverage some innovative solutions, including a rolling migration combined with offline access methods from TSG’s OpenMigrate to allow customers to stop running FileNet considerably earlier than a full migration effort would typically take. This post will discuss the background and unique migration approach.
Understanding FileNet Image Services
FileNet Image Services was initially created back in the 1990’s for scanning solutions when network bandwidth, database, CPU and memory were all expensive resources. Solutions, like FileNet, built at that time had some efficiencies created for the limitations of the day that really don’t make that much sense in today’s world and often can make migrations difficult and complex. Specifically for FileNet, documents are typically stored in TIFF format (think 1990’s fax machine) with a proprietary header information from FileNet. TSG typically refers to this as FileNet TIFF to not be confused with an open format TIFF. Each page of a document is stored as a separate file to reduce network traffic as well as to display the first page of a multi-page document as quickly as possible. Viewing documents required a viewer that is aware of this unique header and page storage. TSG recommend converting the FileNet TIFF single page documents to multi-page PDF to be more compatible with browser based viewing as well as other PDF capabilities. FileNet also has a proprietary compressed computer output to laser disc (COLD) format (typically FOB file extension) that also requires a FileNet viewer.
As we detail in our case study on moving one large client from FileNet to AWS, as well as our previous post on 10 Reasons why there will never be an “Easy Button” for migrations, migrations can be difficult for a number of different reasons. Specifically pressing reasons for FileNet customers typically include:
- Moving Integrations – Given the age of the system, typically many other systems are connected to FileNet for viewing and need to be moved to a new interface. The integration move needs to be coordinated with the migration to view all documents in the new system.
- Moving ingestions – Like integrations, typically there are one to many ingestion jobs that feed FileNet and will need to be moved. These moves also need to be coordinated with the move to the new system.
- User Change Management – Given the amount of content on FileNet, releasing a new interface to users timed with when all the content would be available after the migration.
For all of the above reasons and probably more, many FileNet clients tend to just keep FileNet running on old hardware and delay the decision to migrate given the complexity of all of the migration efforts. For some customers, the decision to delay has resulted in billions of documents in an old and outdated FileNet implementation.
FileNet Image Services migration with OpenMigrate
For our FileNet Migration practice, TSG determined years ago that accessing the FileNet Image Services API was not practical for large scale migrations. Major reasons included:
- API Performance – TSG found the performance of the API to not be on par with a high-volume multi-threaded approach. This could be the API but could also be the typical older hardware supporting FileNet.
- Ongoing FileNet Usage – Typically clients ask TSG to start migrating documents before the system was formally retired. Running a migration that would consume resources and might affect the performance for the FileNet users was unacceptable.
For FileNet Image Services Customers, TSG has chosen to leverage our database connector with direct access to the underlying FileNet images to avoid the two scenarios above. By taking a copy of the database that contains the meta-data and file-pointers to start the migration process, OpenMigrate doesn’t even have to access any of the FileNet components or Filenet even running to run a migration. Once the migration is complete with the copy of the database, TSG will typically do a small delta migration for any content that has been added to FileNet while the migration was running.
OpenMigrate has both source and target connectors for FileNet P8 that leverage the API. This approach works fine for low-volume migrations. TSG is working on a direct database approach for FileNet P8 for a high volume migrations.
FileNet Migration – Addressing the Ingestion Issue
To address the ties to FileNet ingestions, part of the migration process requires ingestions to begin moving their ingestion to the new system. We have seen clients be successful with a couple of different approaches.
- Feed through old system – For this approach, content continues to be fed through FileNet to either be migrated or on demand pushed through to the new system. Typically TSG can leverage OpenMigrate for this type of ingestion as it has the source adapters for FileNet. This approach does not allow FileNet to be decommissioned until a new feed is built to the new system. This approach works well when the new integration effort will take longer than the migration effort but does not let FileNet be retired early.
- Dual feed both systems – For this approach, new content is fed into both systems. For this approach, the content is available for both users on the old system as well as on the new system. With this approach the old system can continue to be run while new content and potentially migrated content is ingested into the new system. This approach works well when content (new/old) can be isolated. Users can be moved to the new system when the content they need, either through the ingestion or a migration, is moved into the new system. Once all users and documents are off of the old system, FileNet can be retired.
FileNet Migration – Retiring FileNet in Weeks
Recently, TSG was brainstorming with a client that was looking for a way to allow them to stop paying IBM for FileNet support as soon as possible since they were moving to a new alternative. For this client, the development work for ingestion into the new FileNet alternative had begun and the client was looking for an easy way to move FileNet content into an archive system. TSG identified a combination of rolling migration and the unique capability of OpenMigrate to migrate documents from FileNet without requiring FileNet to be actually running. As we have pointed out here many times before, a rolling migration involves moving some or all users to the new system immediately with the content they require being moved over when requested “on demand”. This scenario fits case management scenarios where content is broken down into cases that can be retrieved and moved in bulk but would also work for individual documents. TSG recommends a rolling migration approach for clients with large numbers of documents, users, and integrations. (see our related post on rolling migrations for Alfresco customers).
The detail of the “turn off FileNet early” approach includes:
- All the meta-data from FileNet will be migrated to a new repository from the FileNet Database. This is typically a very fast migration as it is just dumping the metadata without converting any files. Metadata will include file-pointers to actual pages/documents in the old FileNet format. TSG can support Alfresco, Documentum, Hadoop, DynamoDB out of the box for the new repository. Client is considering adding an alternative NoSQL solution.
- TSG’s OpenMigrate will begin migrating documents from the proprietary TIFF and COLD format to the new system in reverse date order (newest to oldest). OpenMigrate will be converting both TIFF and COLD into PDF format. Multiple threads of OpenMigrate will be configured to maximize throughput.
- TSG’s OpenContent interface will be installed to view documents and case folders. Users will be able to search and access meta-data content from the FileNet metadata that has been moved over. When viewing or accessing documents, if the documents have already been migrated and converted to PDF, the interface will simply show the PDF documents. If the document(s) have not been migrated, the OCMS Case will initiate an on-demand migration of the document(s) with the FileNet document links. Leveraging the OpenMigrate threads already running, the interface will place the document(s) in the OpenMigrate queue with priority to process immediately. The interface will present a timer with a message to the user that the document(s) are being migrated and, when available, will display the document(s). For previous clients, TSG has gotten 5-10 second performance for on-demand document retrieval.
By leveraging the process above, FileNet Image Services is no longer required as the rolling migration does not leverage anything on the FileNet server.
Summary
By leveraging a rolling migration approach along with TSG’s OpenMigate product, some FileNet Image Services migration scenarios can provide for the early retirement of FileNet itself before all the documents have been converted and migrated to the new platform. This approach would work for both on-premise as well as cloud migration efforts and allow clients to retire FileNet earlier.
While TSG currently has OpenMigrate connectors for FileNet P8’s APIs, TSG is working on a similar direct method to retrieve content from FileNet P8 without FileNet P8 required to be running..
If you have any thoughts or questions, please enter them below.
[…] we have been talking to multiple customers, particularly with FileNet, that have large repositories with hundreds of millions or even billions of documents. As we […]