As we have discussed here before, many clients are looking to migrate away from Content Manager on Demand to new modern infrastructures like Alfresco or NoSQL (See our solutions for DynamoDB and Hbase). Whether outdated client/server interfaces, cloud initiatives or just a desire to move away from legacy solutions, TSG has seen multiple clients looking for better and more modern alternatives to CMOD. This post will highlight OpenMigrate’s CMOD migration ability along with a video demonstration moving content from CMOD to a modern environment like NoSQL or Alfresco along with a modern interface like the Alfresco Content Accelerator (formerly TSG’s OpenContent Managaement Suite)
CMOD – How is it being used?
CMOD’s roots are from the IBM Mainframe glory days when computer output and reports needed a place for high volume storage for customer communication involving documents. As many legacy mainframe applications still are very successful at producing printed statements, CMOD also continues to be used to support business critical mainframe applications. Typical examples of these types of documents include:
- Statements (Credit Card, Banking, Investment….)
- Correspondence (Explanation of Benefits, Bills, Updates…..)
- Records (Explanation of Benefits, Transactions….)
CMOD has always been more in the report archive section of content management with a focus on large volumes of documents. Typically print solutions relied on a print format like AFP (Advanced Function Presentation) that is a proprietary format created by IBM for printing out statements. Lots of paper printing applications can handle AFP but most have turned to more open standards like PDF. Another common CMOD format is Line Date, formatted for printing on a line printer. Transforming Line Data to PDF also requires some templates and transformation for the content to be formatted properly. CMOD, based on it’s ability to store large amounts of documents, has also been used to store PDF and other documents despite not having modern interfaces.
AFP and Print Formats – What are the issues?
One unique difference between AFP and other content types is that, to save storage costs, AFP contains the data needed for a print but requires a template to convert into a viewable image. For example, for a credit card statement, the AFP file might just have the detail about the customer (name, address, charges) but the template would contain the detail about the document type, formats and vendor information. In this manner, the actual data file can be more compact, providing performance and storage efficiencies. TSG found the same capability with another proprietary IBM format, FileNet COLD. (See our FileNet Cold conversion approach).
While storage and transmission costs benefit from the AFP file format, the format sacrifices portability as both the template and the data file need to be present and interpreted for viewing. As AFP was geared for print streams, often AFP files contain multiple documents, say an complete customer statement run for a certain date, making it hard to extract just a specific customers statements for the last three months. In some instances, we have even seen customer mainframes generate AFP files with thousands of different customers on a daily or hourly basis.
With modern initiatives particularly around allowing customers to access documents, records management initiatives and reduced storage costs, many clients are looking to move away from the AFP file format to more self contained and portable formats, particularly PDF, that can provide image and text in on portable document format that many browsers and printers can support natively. Another benefits of portability comes from legal holds, where large print streams might contain some documents that need legal hold.
OpenMigrate CMOD Migration – What is the smart approach?
Migrating from CMOD will typically involve large amounts of files from years of operation. Also, in order to retire CMOD, all of the ingestion processes that may have been purpose built over the years will need to be re-evaluated and pointed to the new CMOD alternative. All of this effort comes with significant costs and some risk of system downtime. Facing these risks, many customers too often decide to just postpone the decision to even begin the effort and delay their modernization, cloud or other initiatives.
TSG and Alfresco would recommend a of gradual or rolling migration approach to reduce the risk of migration. See our other posts on Reducing Risks, Stress and Costs with a Rolling Migration.
As we have posted in the past, there is never an “easy button” for complex migrations. For many of our clients, a “Rolling Migration” has allowed for a gradual rollout of both the users as well as the migration of the content to the new system. A rolling migration involves having both systems available with content from the old (CMOD) system being copied to the new system either in bulk or “on demand” when requested. The concept of a Rolling Migration is something TSG has been doing for clients since 2014.
- The cost and effort of a large migration does not delay getting the benefits of the new system in both modern interfaces and content. Pilot users are able to start using the system immediately without having to wait for all feeds to be re-pointed.
- Users can be gradually rolled onto the new system. With large numbers of users and business critical systems, the ability to roll out the new interface without any downtime is a huge benefit. Training and change management can be scheduled and conducted with small groups increasing the acceptance of the system.
- Leveraging a rolling migration significantly de-risks the project rollout as concerns about the ability to scale the system can be mitigated. By running both systems in parallel during the rollout, the users and management are able to build confidence (IT and the Business) in the new platform and interface throughout the rollout.
When the new system is in production, the integrations to the old system can gradually be moved or adjusted to the new system over time significantly reducing the risk while the new users are getting the benefits of the new capabilities.
Retiring of CMOD to NoSQL and the OpenContent Management Suite
While the rolling migration provides a way to move small amounts of content or content on demand to either Aflresco or NoSQL, if clients have billions of documents and want to retire CMOD completely, we would recommend NoSQL as a more modern way to accommodate massive file repositories. See our related post on how Big Data will disrupt Document Management. NoSQL, built for big data efforts, provides a modern infrastructure specifically designed for massive data storage like CMOD. Some unique capabilities offered for NoSQL include:
- Massive Scale – Built for Big Data, NoSQL offers the capability to support massive repositories easily extending into the billions of documents.
- Massive Ingestion Capabilities – By leveraging Schema on Read, ingestion speeds, a critical component of large print runs associated with CMOD can be easily ingested into a NoSQL repository. Because of its design, NoSQL typically allows for ingestion scaling by adding additional servers. See our DynamoDB benchmark from 2019 where we were able to scale DynamoDB to process 20,000 documents per second for ingestion.
By providing both a robust NoSQL repository as well as a modern, no-code interface, TSG/Alfresco have been addressing the massive scale needed for CMOD and other high volume repositories for years including one very health insurance client that migrated over 4 Billion documents in a 6 month period.
AFP – How to migrate and modernize
While clients that leverage CMOD for PDF files or other standalone file types (TIFF, Word….) are fairly straight forward migration efforts, migrating and modernizing AFP files is slightly more complex. Migration involves two choices.
- Convert AFP to PDF during the migration process – Converting files during the migration involves generating the PDF files from the AFP data and only migrating the PDF to the new system. This approach adds additional effort and time but results in only modern content in the new system.
- Migrate AFP files to the new system, convert on the fly – Similar to the rolling migration approach, creative clients can leverage a rolling conversion approach where files are converted when requested. In this approach AFP files as well as their templates are migrated to the new system and the new system is aware to convert these files when requested. The converted file can either be saved in the new system or discarded after viewing.
OpenMigrate and the OpenContent Management Suite can support either of these above scenarios and can even offer mix and match. Migrating the AFP files is the most practical when content is archived and will rarely be accessed. Converting to PDF makes sense when wanting to simplify the new repository as well as move away from proprietary formats. It also allows for more advanced records management as each document is directly linked to its PDF record rather than linking to a “shared” AFP file that makes legal holds and dispositions difficult.
OpenMigrate CMOD Migration to Examples
Below two videos highlighting the CMOD source component of OpenMigrate as well as the modern capabilities of the OpenContent Management Suite. In the first video:
- Client has credit card statements that need to be accessible from the Search interface with new repository. Currently these statements are being created in CMOD from some other program.
- OpenMigrate can be set up as a batch job to migrate all new credit card statements daily to the new system.
In the second scenario,
- A user wants to look at the credit cards statements for a particular client.
- When the user enters the system, the documents are automatically “rolled” from CMOD into the new system on demand.
Accessing content and potentially retiring a CMOD system can be a daunting task if not for the massive volume of content alone, but also the proprietary formats, the existing feed integrations, and finding a modern approach that can scale. TSG recommends a rolling migration approach to begin moving documents from CMOD to allow for modern formats and interfaces. If CMOD contains billions of documents and the goal is retire CMOD, NoSQL is the proven modern alternative that satisfies some of the unique requirements based on proven scale and ingestion capabilities.