IBM Content Manager and Content Manager onDemand (CMOD) Migrations – How to Migrate Quickly

Many of the TSG customers and prospects are struggling with old versions of Content Manager and Content Manager onDemand that have lived for decades with millions and even billions of documents.  With the 2016 announcement that, while still being branded as IBM CMOD, all future development has been moved to UNICOM, most customers are looking for a way to begin to migrate away from the product.

For clients looking to reduce the cost of supporting CM or CMOD as soon as possible, TSG can leverage some innovative migration solutions, including a rolling migration  combined with offline access methods from TSG’s OpenMigrate to allow customers to stop running CMOD considerably earlier than a full migration effort would typically take.  This post will discuss and demonstrate OpenMigrate’s abilities to migrate from CMOD.

Understanding Content Manager onDemand

Content Manager (later Content Manager onDemand) was initially created back in the 1990’s mostly for replacement of hard copies and long term archival and management of data.  Given the ability to manage lots of files, some clients used for capture/scanning solutions as well.  Solutions built at that time had some efficiencies created for the limitations of the day that really don’t make that much sense in today’s world and often can make migrations difficult and complex.  Specifically, for CMOD, the challenges of migration reside in the complex database structure for managing content as well as the proprietary compression algorithms leveraged to reduce overall storage space.

On the database side, some of the nuances that make CMOD challenging to migrate from are the database table segmentation.  Since traditional RDBMS database struggle with large tables, CMOD is architected to start new tables at a designated increment of records.  For example, once a table hits a million rows, it creates a new table for future content.

• Entries 1-1,000,000 would be stored in the table GDW1
• Entries 1,000,001-2,000,000 would be stored in the table GDW2

These database tables would have entries that look like this:

INDEX_1	REPORT_TITLE	REPORT_NAME	DOC_NAME	COMP_TPYE	PAGE_CNT
52343	CASF_0033	CASF_RUN	11AFAA	44	5

The metadata can be captured from these rows in addition to the related application database table.  The associated content is located in a folder structure built from this metadata.

When CMOD does a search, it has the built-in ability to search across all related tables.  From a migration perspective, CMOD migrations will either have separate migrations for each table or join them into one larger one.

CMOD leverages the concept of “Applications” which is like a document grouping or classification of types.  Since Applications share all the same document formats and metadata, we do see this as an obvious way of splitting migrations.

As we detail in our case study on moving one large client from FileNet to AWS, as well as our previous post on 10 Reasons why there will never be an “Easy Button” for migrations, migrations can be difficult for a number of different reasons. Specifically pressing reasons for CMOD or other legacy customers to struggle with migrations can typically include:

• Moving Integrations – Given the age of the system, how many other systems are connected to CMOD for viewing and would need to be moved to a new interface?  What is the best way to coordinate this move with the migration to view all documents in the new system?
• Moving Ingestions – How many ingestion jobs feed CMOD and will need to be moved?  How can they be coordinated to feed the new system?
• User Change Management – Given the amount of content on CMOD, how to introduce a new interface to users timed with when all the content would be available after the migration.

For all of the above reasons and probably more, many CMOD clients tend to just keep CMOD running on old hardware and delay the decision to migrate given the complexity of all of the migration efforts.  For some customers, the decision to delay has resulted in large amounts of content continuing to build up on CMOD in some cases into the billions of documents.

CMOD migration with OpenMigrate

For our CMOD Migration, TSG determined that the best approach for extracting content is to leverage our database connector with direct access to the underlying CMOD metadata and its corresponding content on the server.  Since the content is often compressed using IBM proprietary formats, we need to add a step the migration pipeline to run the content through a decompression utility on the CMOD server. 

There are several different options CMOD has for file compression (LZW12, LZW16, OD77, OD77Lite, OD77HW, OD77HWLite) and it can be determined from the database what compression is used from the RES_COMP_TYPE field of the ARSAPP table (definition of compression for all documents in an Application). The asradmin tool on the CMOD server, which we can call from OpenMigrate, allows us to pass in the source file, compression type, and output file to prepare the file for migration in an uncompressed format. See below for a quick demonstration of part of a CMOD migration.

Once the migration is complete, TSG will typically do a small delta migration for any content that has been added to CMOD while the migration was running.

CMOD Migration – Addressing the Ingestion Issue

To address the ties to CMOD ingestions, clients need to begin moving their ingestion to the new system.  We have seen clients be successful with a couple of different approaches.

• Feed through old system – For this approach, content continues to be fed through CMOD to either be migrated or on demand pushed through to the new system on a regular interval.  This approach does not allow CMOD to be decommissioned until a new feed is built to the new system.  This approach works well when the new integration effort will take longer than the migration effort but does not let CMOD be retired early.
• Dual feed both systems – For this approach, new content is fed into both systems.  For this approach, the content is available for both users on the old system as well as on the new system.  With this approach the old system can continue to be run while new content is ingested into the new system.  This approach works well when content (new/old) can be isolated.  Users can be moved to the new system when the content they need, either through the ingestion or a migration, is moved into the new system.  Once all users and documents are off of the old system, it can be retired.
• Migrate CMOD one Application at a time – Since different Applications in CMOD can feed from different sources which may have their own challenges in moving to the new system, Applications can cut over to the new system when their migration is complete and all ingestion sources have been redirected.

CMOD Migration – Retiring in Weeks

TSG has taken a different approach with some clients that were looking for a way to stop paying for legacy applications as soon as possible since they were moving to a new alternative.  For this instance, the development work for ingestion into the new alternative had begun and the client was looking for an easy way to move content into an archive system.  TSG identified a combination of rolling migration and the unique capability of OpenMigrate to migrate documents without requiring the legacy system to be actually running.  As we have pointed out here many times before, a rolling migration involves moving some or all users to the new system immediately with the content they require being moved over when requested “on demand”.  This scenario fits case management scenarios where content is broken down into cases that can be retrieved and moved in bulk but would also work for individual documents.  TSG recommends a rolling migration approach for clients with large numbers of documents, users, and integrations.  (see our related post on rolling migrations for Alfresco customers).

The detail of the “turn off CMOD early” approach includes:

• All the meta-data from CMOD will be migrated to a new repository from the CMOD Database.  This is typically a very fast migration as it is just dumping the metadata without converting or moving any files.  TSG can support Alfresco, Documentum, Hadoop, DynamoDB out of the box for the new repository. 
• TSG’s OpenMigrate will begin migrating documents from the proprietary IBM compression to the new system in reverse date order (newest to oldest).  OpenMigrate will can convert to PDF format if needed.  Multiple threads of OpenMigrate will be configured to maximize throughput. 
• TSG’s OCMS (Open Content Management Suite) interface will be installed to view content.  Users will be able to search and access meta-data content from the CMOD metadata that has been moved over.  When viewing or accessing documents, if the documents have already been migrated and converted to PDF, the interface will simply show the PDF documents.  If the document(s) have not been migrated, the OCMS application will initiate an on-demand migration of the document.  Leveraging the OpenMigrate threads, the interface will place the document(s) in the OpenMigrate queue with priority to process immediately.  The interface will present a timer with a message to the user that the document(s) are being migrated and, when available, will display the document(s).  For previous clients, TSG has gotten 5-10 second performance for on-demand document retrieval.

By leveraging the process above the content can all be available while migrating.

Summary

By leveraging a one of several migration approaches along with TSG’s OpenMigate product, content and metadata from CMOD systems can be migrated to a modern system reducing cost, complexity, and aging technology. If you have any thoughts or questions, please enter them below.