Migrating to Alfresco – Reducing Risk, Stress and Cost with a Rolling Migration

Too often, migrating to Alfresco can be seen as a massive undertaking where the migration effort means moving all the content, integrations and people to the new platform in a migrate all at once, “Big Bang” approach.  Given the effort to move all the different components, along with training the users on a new system, it is easy to see why so many legacy ECM systems are still around today as their owners can, when confronted with this daunting task, choose to “wait until next year”.  TSG has been successful with multiple large clients in choosing a rolling migration to gradually move content and people on to the new system to reduce risk, stress and cost.  This post is going to review TSG’s recent experience with a large insurance client that gradually moved 16,000 users and over 350 million documents by leveraging a rolling migration.

Legacy Migration Challenges – When Big Bang doesn’t work

TSG has recently been working with many clients that are on large legacy ECM systems like FileNet, Documentum, Stellent, and ImagePlus looking to migrate off of these platforms to Alfresco. When discussing moving to Alfresco, migration efforts include:

• Moving all the content
• Moving all the users along with training to a new interface for Alfresco
• Moving all the integrations to the old legacy vendor

For many long-time large legacy ECM implementations, coordinating such a major transition is extremely risky.  Choosing to move all content at once in what we would generally call a “big bang” migration has several disadvantages including:

• Migrating effort itself – For large, multi-million document migrations, the timing of the migration might span over weeks if not months given the movement of content and meta-data from the old system to the new system and potential transformations of content and meta-data.  The timing of moving users from one system to another doesn’t easily fall into a simple weekend black out window.
• Migrating the integrations – For one legacy client, the old system had 60 different integrations to store or retrieve content from the old system.  All of these integrations would have to be moved and tested to the new environment.
• Iterative Piloting and Gradual Rollout – With a big bang migration approach, it is extremely difficult to pilot the new system with a small group as the movement is for all people. Piloting typically can reduce the risk while improving or enhancing the user experience before a system is rolled out to the masses.
• Gradual Performance Tuning – System Stabilization – Performance tuning and volume testing large, complex ECM applications is extremely difficult, as simulating user activity is often difficult to predict.  With a big bang approach, the risk of performance or stability issues are huge, as often times the users putting a huge load on portions of the system on Day 1 without the ability to gradually improve the stability and performance over time with smaller user groups and access.

Given the above risks and concerns, one reason legacy ECM systems still continue to be so prevalent is concern over the cost, risk and effort involved in the migration can lead to paralysis. Too often, while it is obvious the existing legacy ECM system does not provide all the benefits, user experience and capabilities of a new system, the concern about the risks and cost of migration can lead many customers to just stay on the legacy ECM system “for 1 more year” and “kick the migration decision can down the road” to avoid the risks and pain of a migration.

Migration Options

TSG routinely talks to clients about the different Alfresco migration options that include:

• Big Bang – Benefits include the decommissioning of the legacy system immediately but has the highest risk, downtime and complexity.  TSG rarely recommends a Big Bang approach.
• Delta Migration – Involves running the systems in parallel and moving changed content over time to the new system.  Significantly less risk and allows for better performance tuning and migration of users/integrations over time.  Most of TSG’s migrations involve some type of Delta Migration.
• Gradual Migration – Involves moving departments or subsets of users over time along with their content and integrations.  Reduces risk substantially and allows the big migration to be broken up into smaller parts.
• Hybrid Migration – Involves components of Gradual and Delta migrations.
• Rolling Migration – Involves moving some or all users to the new system immediately with the content they require being moved over when requested “on demand”.  This scenario fits case management where content is broken down into cases that can be retrieved and moved in bulk.  Typical example is Claim folders for insurance.   TSG recommends a rolling migration approach for case management applications with large numbers of documents, users, and integrations.

See a Webinar on Documentum Migrations to Alfresco TSG conducted with Alfresco to better understand the different migration approaches along with examples:

Client Experience – 350 Million Documents – 16,000 Users

TSG has recently been working with a large insurance company to roll out the OpenContent Management Suite with Alfresco for 16,000 of their claims employees. The client’s old system leveraging a custom integration from their Claims Management System with a legacy Stellent platform that provided simple document retrieval. The business was looking to transform the experience for their users to provide modern functionality such as Document Annotations, Combining Documents, Image Manipulation, and Video Annotations.

Several factors led to the client picking a rolling migration including:

• The client had over 40 integrations that create documents from various outside vendors into the existing Stellent system, the effort and timeline to implement updates to all of these feeds before being able to launch the new system was unacceptable.
• The client had 16,000 employees that would need to be trained in the new system.  Training them all in preparation for a big-bang rollout was impractical and risky.
• The client had over 350 million documents that needed to be moved and transformed.
• Alfresco and the supporting infrastructure were new to the client and there was a desire to gradually roll-out and tune the infrastructure.

Rolling Migration – How it works

A rolling migration does add some complexity to the end user interface to ”delta check” the case for any missing documents to be migrated.  For our insurance client leveraging OpenContent Case for Policy and Claim, the process is depicted below:

1. The user is working in their Insurance Operating System (Guidewire, Fineos, Custom….).  This application controls the data about a claim including claim number, policy holder and other relevant data.  From within the claim system, the user requests to look at the documents for a claim.
2. OpenContent Case is launched in a new tab from their browser.  OpenContent calls OpenMigrate to verify if the claim folder has been migrated before..  If it is the first time the case has been opened, the claim folder is created and the documents are started to migrate in reverse date order (newest documents first).
3. The user is immediately taken to the case folder and shown the documents that have already been migrated.
4. The rest of the documents in the claim are migrated in the background, and the user is able to view documents already in the claim until the rolling migration finishes.

When the system first rolled out, the delay in moving the case folder the first time was roughly 2 seconds or less with the subsequentaccesses being sub-second.  As the client has now run a delta migration to migrate the backlog, the average access time to retrieve the case document list averages under a half a second.

Rolling Migration Success

As we have posted in the past, there is never an “easy button” for complex migrations. For our client and other clients, a “Rolling Migration” allowed for a gradual rollout of both the users as well as the migration of the content to the new system. The concept of a Rolling Migration is something we have been doing for clients since 2014.

• The cost and effort of a large migration did not delay getting the benefits of the new system. Pilot users were able to start using the system immediately without having to wait for all feeds to be re-pointed.
• Users could be gradually rolled onto the new system. With over 16,000 users on this business critical system, the ability to roll out the new interface without any downtime was a strict requirement. Training and change management could be scheduled and conducted with small groups increasing the acceptance of the system.
• Leveraging a rolling migration significantly de-risked the project rollout as there were considerable concerns about the ability to scale the system. By running both systems in parallel during the rollout, the users and management were able to build confidence (IT and the Business) in the new platform and interface throughout the rollout.

Now that the system is in production, the integrations to the old system can gradually be moved or adjusted to the new system over time significantly reducing the risk while the new users are getting the benefits of the new capabilities.

Migration and Ingesting Queuing with OpenMigrate

One benefit of OpenMigrate versus other migration tools is the ability to process both migrations from legacy systems as well as on-demand migrations or ingestions.  For our insurance client, this was critical as we wanted to both process the backlog of 350 million documents when capacity was available as well as push on-demand requests to the top of the priority list when a user requested a particular claim. By leveraging OpenMigrate against an ActiveMQ queue, claims can be migrated in priority order, and “on-demand” claim views have been prioritized.

Client Implementation – By the Numbers

Now that the client is in production, some relevant numbers and performance to share include:

• 16,000 users
• 350,000,000 documents (40% PDF, 40% JPG, 10% Video/Audio, 10% MSG, Word, Excel, Powerpoint)
• 17,500,000 PDF renditions of MSG/Word/Excel/PowerPoint files leveraging an on-prem Adlib cluster
• Average retrieval of documents with OpenAnnotate of .75 seconds
• Over 140,000 document retrievals per hour during peak daily usage, or nearly 40 documents viewed every second.
• Many of the integrations have been repointed to leverage OpenMigrate to ingest approximately 400,000 new documents throughout the day.

Summary

Whether moving from FileNet, Documentum, Stellant or a host of other legacy ECM platforms, the migration effort can be daunting task that will often be delayed due to the perceived risk, cost and complexity of moving documents, users and integations.  By leveraging a rolling migration approach, certain customers can significantly reduce the risk, cost and complexity while getting the benefits of the new system earlier.

Let us know your thoughts below.