Federated Content Management – Enterprise Search with a new moniker?

One troubling point we continue to see in our industry space, whether it be content management providers like Nuxeo, ASG or even analysts like Gartner, is a continued push around the benefits of a federation search, often as an alternative to migration.  This post will explore federated content management with a comparison to a very similar approach marketed as Enterprise Search from almost 20 years ago, a technology that failed to produce value. 

Enterprise Search – What was the promise?

Back in the early 2000’s, Enterprise Search held a promise for content management very similar to what is being promised from Federated Search now:

• Knowledge workers access multiple content management systems.  A tool that could streamline knowledge worker access and activity would improve worker efficiency.  Typical quotes include “knowledge workers spend X% of their time searching for information resulting in a cost of Y.”
• Migrating from a legacy content management system is difficult.  A tool that could provide access to the old system while also providing access to the new system would reduce the need to migrate everything from the old system to the new system.

Enterprise Search promised the “single pane of glass” to view all content in an enterprise.  Stats were throw around about just how much time Knowledge workers waste searching for content.  Enterprise Search was viewed as the way to reduce the time looking for content in multiple systems. 

At the time, TSG was an Autonomy partner with multiple engagements for Autonomy Enterprise Search implementation.   

Enterprise Search – What were the problems?

For our clients, as well as other Autonomy customers, some of the struggles they encountered included:

• Adapters and Support – For Enterprise Search, Autonomy needed adapters to connect to the existing (often legacy) systems to be able to extract and index the content.  Typically, TSG would build or modify adapters as part of our implementation services.  As Autonomy (or TSG) didn’t maintain our own copies of these legacy implementations, any support would require access to the client’s system.  For issues, TSG was often “on our own” when it came to getting support from the legacy system vendor as typical legacy systems didn’t always have easy support connections.
• Security – Enterprise Search would index the content in the search engine for quick and efficient combined search.  Pushing different content repositories content required also providing a vehicle that would protect and replicate security to the search index to protect secure documents. Security added complexity and required a re-index if any security changed.
• Updating Content – While finding content makes sense, what happens when the user wants to do something with the content?  Activities could include annotation or combining with other content or linking for a specific use case.  Enterprise search suffered in that it could only view the document and would have to access the source application to work with the document.
• Licensing – Depending on the approach, users would require licenses for both the Enterprise Search tool as well as the legacy repositories.

Federated Search – What are the differences and similarities?

Federated Search is different than Enterprise Search in that the approach leverages the existing content repositories search (and security) rather than building a new index in the content repository tool.  Where the Enterprise Search approach looks like:

1. Backend – Search tool access content repository, build new search index for all repositories.
2. User accesses new search index.
3. User views document from old repository.

Federated Search looks like:

1. Search current repository and legacy content repository leveraging legacy APIs
2. Combine search results.
3. User views documents from combined interface.

The major difference and advantage for federated search is that a new search index is not created.  This allows security to be enforced as the federated search is leveraging the security of the legacy content repository.  Issues with Federated Search include:

• Combining Search is not that simple – As different repositories have different content models and values, searches done in one repository are typically different in another requiring an understanding of both (or more) repositories at a detailed level.  Federated tools rarely combine the searches, just allow a search against one repository or the other.
• Combining Search can be inefficient – Enterprise search offered one efficient index for searching.  Federated search requires access to multiple repositories with the search returning at the pace of the slowest repository.  In looking under the covers at one of the main federated search vendors, they are really migrating the meta-data to do one search and just leaving the file binaries in place in the old system. This approach requires the duplicating of the meta-data as well as constant syncing between the repositories for any changes.
• Adapters and Support – Like Enterprise Search, adapters have to be written (and supported) for the legacy platform.  Supporting the integrations is very difficult given that 1)  the product is legacy by definition, 2) the product is often competitive to the new content repository, and 3) Older systems are often less stable and harder to support than new systems.
• Updating Content – Similar to Enterprise Search, what happens when the user wants to do something with the content?  Activities could include annotation or combining with other content or linking for a specific use case.  Does this happen in the old repository?  Can the interface handle that update?
• Licensing – In this case, the license for both the old and new content repositories would need to be maintained with users having valid licenses on both systems. Older system would not be able to be decommissioned.

Of the above issues, we would say that the adapters and support from a software product perspective is the most difficult and probably looks more like a consulting engagement where the consultant has access to the legacy environments vs. software only approach.  Any support call would require the federated software firm to:

• Access the customer’s environment to debug the issue as the company probably doesn’t have their own instance of the legacy repository.
• Understand both the new repository as well as legacy repository because the legacy repository would not be providing support.

“While Enterprise Search promised value, even my clients that were able to get over the security, support and development issues of the adapters found that the final solution just wasn’t being used”, said Alan Pelz Sharpe, long time ECM/CPS analyst. “The business case for consolidating search and view of content just didn’t add enough value versus accessing the source systems themselves.”

Federated Search with a Rolling Migration

TSG approaches Federated Search as a means to a migration rather than in place of one.  From our post on Federated Search versus Rolling Migration, see the differences in the table below:

Federated Search	Federated Search with Rolling Migration
Search Retrieves the Legacy document	Search retrieves the legacy document
Transform the document for viewing	Transform the document for viewing
View the document	Store the transformed document in new repository
	View the document from new repository

In the above case, the 2^nd time the document was requested, the document would be immediately viewed from the new repository. 

In the above scenario, the benefit of Federated Search is quicker viewing of the document the first time it is accessed and those savings might just be sub-second depending on how the new system is set up. 

Benefits of the Rolling Migration approach include:

• Quicker viewing of documents the second and subsequent views.
• Ability to View/Annotate/Combine and any other document manipulation function in the new system.
• Ability to not have to transform the document the second time and beyond, a non-trival effort when working with examples like FileNet Image Services where the documents could be proprietary COLD or single page TIFF that have to be combined to PDF along with potential Daeja annotations.

When combined with a migration effort, the differences are also distinct.

• The rolling migration will always look into the legacy system to determine if something is different between the new system and legacy system and move it if required.
• The federated search will have to rely on some other method to know when to search the legacy system and the new system.  Content dropped into the old system by an old legacy integration could prove tricky.

To address the specific points brought up above:

• Combining Search is not that simple – TSG will typically handle the search as part of a migration effort where we will build and test the search and migration for the client’s specific scenario providing real-time support throughout the migration.
• Adapters and Support – As part of a consulting agreement, TSG brings previous adapters but tailor them for a specific client and their environment and versions.
• Updating Content – As content is brought over to the new system, no issue exists with updating content.
• Licensing – With a migration approach, eventually the old system can be retired.  For some approaches, like our FileNet Migration, we can access the content without requiring the system to be running or support to be paid. This approach can lead to old systems being decommissioned sooner.

Client Experiences – Federated Search and Rolling Migration

One thing we continue to struggle to understand about Federated Search is successful customers and case studies.  Most blogs talk about what it can do rather than success of it actually doing it similar to the Enterprise Search promises.  TSG did a comparison of three large customers, two that did rolling migrations and one that did more of a federated-bulk migration.  Some interesting points:

• One Rolling Migration client was able to start with only 600,000 documents in the repository and pilot/gradually roll out from 1,000 users to 19,000 users and 350,000,000 documents.
• The Bulk Migration client was not able to pilot, attempted a big bang migration approach after a 11 month migration and ended up falling back to their legacy content repository.
• One Rolling Migration client still uses old feeds into an older repository (custom – not a vendor solution) rather than moving feeds to the new repository.  Federated Search approach would not work for this client as the old custom repository has no APIs and required database look-up to support the rolling migration.

Summary

A rolling migration can be a game changer when migrating from multiple legacy systems to a new system, either on premise or in the cloud by reducing the need to move all users, content and integrations before the new system can be used and piloted.  TSG has assisted multiple large clients and customers to successfully move to new systems by leveraging a rolling migration approach.  Federated Search can play a role in accessing content from legacy systems but these applications will struggle with document actions like annotation or other document manipulation as well as licensing and other support requirements.

Let us know your thoughts below.