• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer
TSB Alfresco Cobrand White tagline

Technology Services Group

  • Home
  • Products
    • Alfresco Enterprise Viewer
    • OpenContent Search
    • OpenContent Case
    • OpenContent Forms
    • OpenMigrate
    • OpenContent Web Services
    • OpenCapture
    • OpenOverlay
  • Solutions
    • Alfresco Content Accelerator for Claims Management
      • Claims Demo Series
    • Alfresco Content Accelerator for Policy & Procedure Management
      • Compliance Demo Series
    • OpenContent Accounts Payable
    • OpenContent Contract Management
    • OpenContent Batch Records
    • OpenContent Government
    • OpenContent Corporate Forms
    • OpenContent Construction Management
    • OpenContent Digital Archive
    • OpenContent Human Resources
    • OpenContent Patient Records
  • Platforms
    • Alfresco Consulting
      • Alfresco Case Study – Canadian Museum of Human Rights
      • Alfresco Case Study – New York Philharmonic
      • Alfresco Case Study – New York Property Insurance Underwriting Association
      • Alfresco Case Study – American Society for Clinical Pathology
      • Alfresco Case Study – American Association of Insurance Services
      • Alfresco Case Study – United Cerebral Palsy
    • HBase
    • DynamoDB
    • OpenText & Documentum Consulting
      • Upgrades – A Well Documented Approach
      • Life Science Solutions
        • Life Sciences Project Sampling
    • Veeva Consulting
    • Ephesoft
    • Workshare
  • Case Studies
    • White Papers
    • 11 Billion Document Migration
    • Learning Zone
    • Digital Asset Collection – Canadian Museum of Human Rights
    • Digital Archive and Retrieval – ASCP
    • Digital Archives – New York Philharmonic
    • Insurance Claim Processing – New York Property Insurance
    • Policy Forms Management with Machine Learning – AAIS
    • Liferay and Alfresco Portal – United Cerebral Palsy of Greater Chicago
  • About
    • Contact Us
  • Blog

Improving Large File Uploads into an ECM by 95%

You are here: Home / Alfresco / Improving Large File Uploads into an ECM by 95%

August 25, 2017

With the digital transformation of traditional ECM processes, file sizes continue to increase due to more video content and other rich media.  Regardless of the ECM solution (Alfresco, Documentum….), one of the biggest issues around digital transformation surrounds uploading large files, particularly in a synchronous mode where users want to upload the file and immediately process the file.  This post will highlight some innovative approaches TSG is currently recommending to clients surrounding the performance of ECM file uploads.

Large Files and ECM – What are the issues?

To understand why large files take so long to upload into ECM systems requires an understanding of the underpinnings of Enterprise Content Management.  ECM systems typically manage their own file stores and expose an API to allow content to be placed into that content store.  For a typical file upload process from a browser client:

  1. User identifies a file that requires upload. (Select File or Drag and Drop to Web Browser)
  2. The file is uploaded from the client to the application server via HTTP protocol over TCP/IP.
  3. The file is processed through the ECM system (API) to store into the managed content store.
  4. The user, after waiting for 2 and 3 to finish, receives back notification that the file (or files) have been uploaded.

The delays in uploading large files are typically found both in the application server as well as the ECM API.  Many times, clients will implement an asynchronous approach where files are dropped off and added to the ECM system at a later time rather than have users wait for the upload to complete.  While this approach is often acceptable for back-end migration efforts, typical user scenarios and processes often have the user wanting to upload and then tag/annotate and process the file rather than work on something different while awaiting for the long upload to complete.

Object Store and S3

TSG has had considerable success in reducing the time of large file upload both for users as well as large migrations by moving content directly into the object store and linking (rather than uploading) the files to the ECM repository.  Typically Object Stores (or Amazon S3) are built for ingestion of large files.  While a user might not notice any difference with the linking option, the process is very different.

  1. User identifies a file that requires upload. (Select File or Drag and Drop to Web Browser)
  2. The file is uploaded from the client directly to the object store/content store via HTTP protocol over TCP/IP.
  3. The file is linked with the ECM system (API).
  4. The user receives back notification that the file (or files) have been uploaded.

TSG has seen processing times improve 10 to 30 times by linking rather than the tranditional ECM method of storing documents.  The approach above also has additional benefits in regards to reducing the processing requirements of both the application server and ECM system.

ECM Storage versus ECM Linking Results

TSG has one large client leveraging the Hitachi Content Platform (HCP) as well as multiple Amazon S3 customers that are having success with the ECM linking rather than ECM storage approach within our Alfresco practice.  For results:

  • Hitachi Object Store – for a large insurance client (1 billion objects), linking was used within the migration effort to dramatically improve the initial migration (that included renditioning) to up to 200/documents per second. Linking is being used throughout the daily ingestion activities.
  • Amazon S3 – TSG has conducted multiple evaluation efforts for Amazon customers to including linking in both the initial migration as well as ongoing ingestion with large increases in performance.

Improving Network Speed – Aspera

Internally, TSG has been working with Aspera to improve upload performance for network speed.  Aspera has a rather unique approach.  Leveraging their own FASP transport technology, Aspera moves content quickly and securely over existing WAN infrastructure that is often hundreds of times faster than FTP and HTTP.  Much like bittorrent, Aspera breaks a file into small parts and quickly and redundantly sends them across network to the server leveraging the underlying components of the network.  For our internal tests, we really found that network speed was the limiting factor.  Some other observations:

  • One downside for leveraging Aspera was the need for a browser side plug-in to leverage the FASP transport technology.
  • While installing the plug-in could be an issue, the upside is that once the transfer is initiated, the browser window could close and the upload would continue.
  • Security, Retries and other nice to have’s are included with the plug-in.

Summary

Whether for initial migration or ongoing migration of more and more rich digital content, ECM users are looking for ways to improve file upload performance.  Object Linking and improving network performance with Aspera are two easy ways to improve synchronous large file uploads.

Let us know your thoughts below.

Filed Under: Alfresco, Amazon, Documentum

Reader Interactions

Leave a Reply Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Primary Sidebar

Search

Related Posts

  • ECM 2.0 – Can you build it yourself?
  • ECM Large Repositories – Volume Testing With the TSG Test Harness
  • ECM 2.0 – What does it mean?
  • Redaction for AWS, Alfresco, Documentum and Hadoop – Bulk Redaction upon Ingestion or Migration
  • Case Analytics for Insurance
  • Redaction for AWS, Alfresco, Documentum and Hadoop – Folder Case Redaction
  • Redacting Roadmap – User Scenarios
  • Alfresco on AWS – Achieving High Availability
  • Alfresco and Amazon Web Services – Disrupting Legacy Content Services – Alfresco Day London – Keynote
  • Alfresco & AWS – Modernizing Records Management – Case Study from the Department of the Navy

Recent Posts

  • Alfresco Content Accelerator and Alfresco Enterprise Viewer – Improving User Collaboration Efficiency
  • Alfresco Content Accelerator – Document Notification Distribution Lists
  • Alfresco Webinar – Productivity Anywhere: How modern claim and policy document processing can help the new work-from-home normal succeed
  • Alfresco – Viewing Annotations on Versions
  • Alfresco Content Accelerator – Collaboration Enhancements
stacks-of-paper

11 BILLION DOCUMENT
BENCHMARK
OVERVIEW

Learn how TSG was able to leverage DynamoDB, S3, ElasticSearch & AWS to successfully migrate 11 Billion documents.

Download White Paper

Footer

Search

Contact

22 West Washington St
5th Floor
Chicago, IL 60602

inquiry@tsgrp.com

312.372.7777

Copyright © 2022 · Technology Services Group, Inc. · Log in

This website uses cookies to improve your experience. Please accept this site's cookies, but you can opt-out if you wish. Privacy Policy ACCEPT | Cookie settings
Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Non-necessary
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
SAVE & ACCEPT