Documentum or Alfresco – Thoughts on Volume Testing

We have had a couple of clients, one on Documentum and one on Alfresco, work through some production volume issues over the past week.  This post will discuss the topic of volume testing for ECM solutions and present some thoughts on when and how to implement.

Why Volume Test Documentum or Alfresco?

ECM solutions have a tendency to quickly grow in size and volume compared to other IT solutions.  This is due to:

• Converting Paper to Electronic – as most implementations have moved away from paper, the amount of volume increases as more time passes and more and more paper is turned electronic.
• Lack of Archival – most implementations tend to keep the old documents and add more rather than replace.
• Cost of Storage – with the cost of storage decreasing, clients have the ability to add more storage and more documents for minimal cost.
• Content Size – with the proliferation of more digital cameras, color documents, faster networks, better viewing devices and other technologies, content size has increased.
• User Number  – As the ECM system is successful, additional users are added to the solution.  This could include other internal users as well as external users.

With both the size and number of documents increasing, clients will look to volume test their solutions for several issues:

• Performance – As volume/size/users increases, performance can suffer.  Testing would look to verify that performance is consistent.  See some related performance tuning for Documentum.
• Stability– With users/size increasing, the concern in regards to network, memory as well as the solution itself can become an issue.  Volume testing would look to confirm that, with a max load of users, the system would remain stable.

Issues with Volume Testing – Documentum or Alfresco

Volume testing comes with a number of issues for clients to navigate including:

• Cost – The cost of setting up a volume environment, adding the right amount of content as well as a tool capable of testing the user access scenarios is not trivial.  Tools such as LoadRunner add significant cost to the project.  Many clients chose to delay this testing as the ECM solution, when initially developed, will not have the number of users and content for quite some time in the future.
• Realistic Test – Does the environment, when configured, actually represent the scenario that will occur sometime in the future?  We have seen it very difficult to predict just how documents and users will be added to the system to set up a realistic test.
• Realistic Data – Given that data with different properties and sizes will be slowly migrating into the system, how can we set up a test with a proper amount of content and differences in meta-data?
• Environment Availability – Does the infrastructure support a separate volume test environment or will the volume test use production?   When is a timeframe to conduct a volume test that won’t interrupt production environments or network?
• Timing – The effort and timing associated with a volume test can delay the system going into production.

To address the issues above, we have seen clients successfully volume test when:

• New Environment is Available – Often times, clients will be adding a new server as part of an upgrade or migration.  If the infrastructure is available, clients will often migrate the data (while leaving the old system running), perform volume and performance tests before cutting over to the new environment.
• Large Data is Available – If the system currently has large amounts of data that is easy to capture, we have seen successful clients load the data to quickly performance test the results.  For one client, we successfully loaded all the meta-data (without the content) to check database performance.

If the data or environment is not available, we have seen many clients opt to conduct some basic volume testing but also rely on more monitoring and a SWAT team to quickly fix issues once they crop up.  Key to the success of the SWAT team:

• Consistency of Resources – too often, the team of developers responsible for the application has moved on to other assignments.  For the SWAT team approach to work, knowledgable and consistent resources must be able to take action quickly.  In regards to implementing the performance or volume improvements, it is best to have access to a consistent resource pool rather than have new resources having to get up to speed quickly.
• System Monitoring – Rather than waiting for issues to arise, the team should be monitoring the system in an attempt to identify and fix issues before they affect performance or system availability.
• Volume Checkpoints – Similar to monitoring, the system should schedule “health checks” to review overall capabilities thought out the lifecycle of the solution.

Summary

Volume testing Documentum or Alfresco is something that should be included with many implementations but is rarely part of the plan due to the difficulties and cost of configuring a valid volume test.  If the cost and timeframe can be contained and the appropriate data and environment can be procured, volume testing with performance considerations is a pragmatic approach to avoiding production issues.  Regardless of the volume test plan, smart implementations will have access to knowledgable resources as part of a SWAT team to address volume issues as they arise.

If you have any thoughts or comments, please add below.