We recently helped a client with a difficult implementation surrounding the “Content Library” function released back in April with D2 as a primary client. The use case is focused on managing project deliverables for large-scale multi-year projects. This post will describe some of the client’s struggles and our overall thoughts.
D2 – Business Issues with the controlled nature of project documentation and deliverables
In contrast to the typical Content Library use case discussed by EMC, our client’s use case involved maintaining approximately two decades of project documentation. The organization of information was built upon a legacy structure with minimal property (metadata) values, instead using deep and complex folder structures.
Some of the D2 usability issues with our particular use case included:
- The D2 interface and had a performance impact of displaying the large number of cabinets and folders. This was similar with Webtop.
- Navigating deep folder paths was limited: Project folders can contain multiple sub-project folders
- Moving between D2 iframes can cause the adjoining frame to reset causing the user to have to browser back down the folder path again
- D2 has a default limit for displaying folders and documents, it can be increased but resulted in a performance impact
- A folder may contain a large number of documents and since they cannot all be displayed, the user must use the widget filter to find the documents they are interested in
- Searching for large number of documents
- quicksearch – given the large number of documents, the search would product too many results and was difficult to filter/navigate.
- advanced search – client wasn’t happy with a “build a search” approach due to the time to create a search. We highlight the build approach in a D2 video in the learning zone.
- D2 is unable to display a custom property on a check-in widget
- This forces the user to check in the document, navigate to it and then update the properties on it; very inefficient for the large volume of documents. A better approach is to require properties to be added as the document is being uploaded. See a demonstration of how this is accomplished HPI add documents in our learning zone.
D2 – Technical Issues
Our client struggled with a variety of different technical components. For background, this diagram from the D2 4.1 architecture white paper from EMC shows the deployment location of the D2 components.
As presented above, the multiple D2 components result in longer deployment timelines and often significant troubleshooting efforts. This complexity is also discussed in our earlier blog post on why D2 has not replaced Webtop and in a post from Scott Roth on his experiences with installing D2.
For our client, the complexity results in:
- Ongoing issues with D2 DAR installations due to a non-dmadmin install owner.
- We recommend clients utilize the default dmadmin install owner whenever possible.
- Continued struggles with the stability of the JMS – hangs and has to be restarted
- As a proactive measure, the client implemented a weekly restart process for the Documentum stack
- The D2 web application requires a restart after the JMS is restarted to prevent caching issues
- We open an SR case with EMC for each occurrence to help root cause the issue.
- Root cause determination has proven very difficult since the JMS hang has not been predictably reproducible.
- Various user and document session time outs
- session management and time outs are configured and adjusted in multiple locations
- large file transfers can result in time outs if they exceed the 5 minute transfer connection limit
The following sections summarize additional areas of complexity for the ongoing management and customization of D2 for our client.
D2 – Complexity of Deployment
Some of the items our client struggled with in regards to deployment included:
- Java Dependency on the Client
- A Java Applet is used for content transfer, including bulk export of folders and content
- HTML5 transfer can be used for content transfer, but not bulk export of folders
- Specific Java versions are required for both D2 4.2 and 4.5; auto-updates can install unsupported releases and the correct Java needs to be re-installed
- IE9 must use the Java Applet since it cannot handle the HTML5 transfer
- Lock box configurations control the encryption of the communication between D2 and other Documentum Modules
- The lockbox must be configured on the Content Server, Web Application Server, and ACS and BOCS servers
- Simple in concept, but its easy to slightly miss-configure. Administration should ensure the same password is used in the lock box steps or configuration utility
- Administration should ensure the library files to encrypt and decrypt communications are stored in the proper directories in the JMS, Web App Server, ACS, and BOCS
- D2 utilizes a custom DFS framework, D2FS, that interacts with the JMS, this is unlike the application architecture of WDK apps such as Webtop
- Unlike the DFC API, which interacts with the Content Server, the D2FS API interacts with the JMS
- JMS Components run the D2-API methods that are installed as part of the D2 DARS; runs the plug-in jar files for C2, O2, D2-Bin, D2-RPS, and D2-BOCS
- Within the D2 D2FS API Javadocs is a listing, similar to the DFC, describing all the classes available for customization of D2
- The JMS Encrypts and decrypts D2 communictions
- As part of an upgrade or patch, older files need to be deleted/removed from the JMS
- WAR file components include DFS and application components
- The D2 and D2-Config web applications are built by the D2 Install Application, the WARs are then transferred to the web app servers for deployment
- Additional D2 plugins must be included in the WAR build. These are the C2 (PDF overlay/watermarking), O2 (email), D2-Bin (recover files), D2-RPS (retention), and D2-BOCS
- D2 application configuration files can be included in a deployment on installed separately.
- In order to simplify deployments, our client does not include a configuration in the WAR.
- DFS (Documentum Foundation Services) SDK is required to build the D2 and D2-Config WARs
- D2 utilizes the DFS SDK to include lib files in the WARs it is dependent on the DFS libraries.
- DARs and hardcoded install owner
- The basic D2 deployment includes 3 DAR files: D2-DAR.dar, D2Widget-DAR.dar, Collaboration-Services.dar
- Plug-ins for D2 require deployment of up to an additional 5 DARs
- Our client used a different Install Owner than dmadmin and this has caused an issue with deploying D2 4.5.
- Deploying the DARs resulted in ACL and object ownership being changed from the Install Owner to dmadmin leading to a mix of ownership for DAR related objects.
- This causes an issue when installing a DAR using a nondmadmin file that overrides the default dmadmin owner, with mixed ownership, these DAR installs fail
- In order for the DARs to install successfully, anything where dmadmin is made an owner of the object, the owner must be manually corrected to our client’s Install Owner name
- The basic D2 deployment includes 3 DAR files: D2-DAR.dar, D2Widget-DAR.dar, Collaboration-Services.dar
D2 – Complexity of Troubleshooting
The complexity our client experienced while troubleshooting included:
- Multiple Logs to check and review for outages, performance, or user reported issues
- D2.log, D2-Config.log, D2-JMS.log, server.log and web application server logs
- Multiple repositories in one Content Server/JMS instance
- When one repository becomes unresponsive in D2 but others are available, restarting the JMS and web app server impacts all hosted repositories
- The root cause for the unresponsiveness has been difficult to identify
- The JMS and web app server should be restarted together;
- just restarting the JMS can lead to issues with D2 reporting errors, just restarting D2 often doesn’t resolve the issue
- When one repository becomes unresponsive in D2 but others are available, restarting the JMS and web app server impacts all hosted repositories
- After a restart or upgrade, users need to clear the D2 cache on the local client machine to prevent D2 error messages
- When users encounter errors, they have been instructed to do a hard reload in the browser – Ctrl+F5
- The hard reload purges the browser cache and prevents users from getting errors after a restart of the D2 web application server
D2 – Complexity of configuration
The complexity of configuration our client experienced included:
- D2 Applications provide context for the users
- D2 uses an Application construct in order to group D2 configurations together, for example the D2 example HR Sample Application available on the EMC Download site.
- Configuring a D2 application can explode into an examination of the business processes for a set of users. If this activity was not planned or the BP doesn’t fit into the configurations of a D2 application, the D2 web application may be difficult for users to adopt.
- This is particularly an area of concern our client is experiencing. They require a broad Webtop-like D2 application in order to accommodate their legacy Documentum data. Many configurations, such as Folder Import, require specific configurations which identify rules for filing, security, and functionality based on document metadata values.
- Parent / child configuration dependencies are common in configuring D2
- Many configurations are dependent on a prior configuration existing in the D2-Configuration for a D2 application
- An example is one our client encountered when trying to configure Folder Import. The folder import requires an Autolink configuration and Security configuration.
- Enabling configurations / Caching configurations
- The dashboard matrix for D2-Config allows a config to be very focused or very broadly applied.
- There is not a default configuration akin to a “Webtop” configuration. D2 requires discussions and decisions on what to configure as a default or more specifically for a D2 application
- Troubleshooting configurations
- The dependency issues can be difficult to understand and troubleshoot, for example why is a menu item is not displaying for a specific user group for a specific D2 application
- D2 4.5 now offers a way to export the specification so it can be compared to other exported specifications
D2 Development Limitations
During the rollout of D2 the client discovered specific limitations to developing add-ins for widgets and D2.
One was a custom feature built for Webtop, Folder Move Confirmation. Due to the D2FS service architecture, it was not possible to replicate the function in D2 as it was not possible to interrupt the move process and prompt the user to continue or cancel.
A second example is maintaining attribute filter values in a doclist widget. The widget is triggered to reload a list of documents after the user performed an action on one of them. Unfortunately, on the reload the filter entered by the user was lost. The user must then re-enter the filter information to reduce the number of documents displayed in the widget.
- D2 4.5 APIs
- D2FS Java API
- Unlike Webtop, D2 relies on a custom version of the DFS built specifically for D2.
- SOAP Services (legacy)
- D2FS Java API
- D2 External Widgets
- Must be a separate web app accessed via a D2 iframe
- Limited cross-communication is available between iframes
- Limited event listeners must be used
Summary
The key takeaway from our work with D2 is discovering its architectural complexity and the issues if everything isn’t addressed. Complexity included installs and configuration for the Content Server, Java Method Server, Web Application, Plug-ins and repositories. These should be carefully planned for and addressed. The complexity of D2 necessitates planning the deployment with careful thought to D2 configuration and on-going maintenance and troubleshooting. On the positive side, D2 does provide an opportunity to configure many actions based on business process and document metadata however the business must be aligned with it in order to utilize this capability.
We are interested in hearing more about your thoughts on D2, please leave your comments below.
For additional information, please find these related D2 articles in our blog.
Hi Chris:
I want to clarify that the “dmadmin” installation for DAR files can easily be configured. On page 27 of EMC Documentum D2 v4.2 Install there are instructions for creating a small XML file in order to set the install owner.
I am currently completing work on a project to migrate a large Webtop-based Documentum platform into D2. We are an small Documentum consulting firm on Chile.
While installation is not as easy as it’s on Webtop. We accomplished it on less than 2 hours on QA once you have first learn how to do it on a DEV Env.
Learning the whole process is basically carefully follow the install guide and it took us about 6 hours to complete and fully understand it. This is a one time investment so it’s basically zero significance effort compared to the complete project. And also a fully reusable knowledge.
Transferring the generated install files from QA to PRD only took about 1.5 hours to be completed including PRD specific setup.
On our project with a 4.000+ user install base and a 2+ million document, D2 has been performing better than Webtop. And the UI is a lot more user friendly also.
Setting session timeouts is also almost the same as Webtop, so no really important to mention.
As you said before, java applets are recommended but not required. There are just a pair of functions that benefit from them. Anyway, we haven’t seen any misbehavior if your applets are a more recent version than the base certified ones. On Webtop, using a newer non-certified applet would certainly fail. Anyway, EMC has been certifying and releasing patches to keep up with java versions on a monthly basis.
For solving your document access and search problem D2 has a function called “query forms” (QF) where you can create custom UIs for searching. We have successfully deployed that for some document types.
Not only QF are a great. D2 Advanced search is an order of magnitude more powerful than Webtop’s. Now you can have a fully customized attribute setup with Dictionaries, conditional V.A. And the best is you can now search for multiple doctypes at the same time. Configure the display config (also with Dictionaries and intelligent data mapping) and with real time result filtering on all attributes.
Finally, while D2 is a little harder to deploy than Webtop. Your DEV effort for your project will be about 40% of what it was before. Also, your apps will be a lot more easy to maintain and evolve making system much more successful than before. So I highly recommend customers to make the transition.
Hi Jorge –
Thank you for the thoughtful response. I’d like to clarify a few points but I’m also happy to hear you have had a good experience with D2, mine has been challenging.
You are correct that D2 4.2 can utilize a non-dmadmin Install Owner. Unfortunately, D2 4.5 cannot. It appears to be related to the installation improvement to deploy the DARs as part of the installation wizard. Per EMC support, we opened an enhancement request on behalf of our client. In the meantime, we found a workaround so the upgrade could proceed only slightly delayed.
It sounds like your Dev deployment helped you gain the knowledge to perform a speedy install of D2 in QA. It is a similar experience for us. We used the documentation to write up deployment steps and then deployed into multiple environments. I’m not sure how many D2 plug-ins or repositories you deployed to, but our client utilizes nearly all the plug-ins and each environment has several repositories. Just deploying the DARS into these large repos takes quite a bit of time. Actually, the install improvement in 4.5 was a welcome change until we found it had the install owner issue.
For the production patching we’ve done for D2 we typically allocate 4 hours for each prod environment including deployment and testing. For some smaller repos it can go quickly, our client has multiple global prod environments in clustered fail over configurations with some very large repositories. The multiple deployments just take time, there is no way around it.
As for session timeouts, if you haven’t run into issues with it that is awesome. However, our client has been struggling with session timeouts due to environment restrictions and the transfer of very large files over not very speedy remote network connections.
It sounds like the repository structure you are working with lends itself to the D2 query forms. We’ve looked at those but since our client has very little metadata on the documents they are using Quick Search extensively.
I agree that if you can leverage the D2 Application configuration in place of coding the project can go much quicker. Overall, I’d like to see a default Webtop-like D2 Application from EMC.
Again, thank you for your comment!
I wasn’t aware that D2 4.5 deploy didn’t allowed to change the install owner name. Most certainly you can workaround that just by creating the user on your repo and then modifying the deployed ACLs.
For improving search on your scenario. I would suggest to take a look at faceted search functionality. While a little harder to configure than on xCP 2.x (where it works amazingly well). You can take advantage of it for D2 and also combine them with QF. It works like the old Content intelligence services, xPlore-based.
I can see your customer has many repos and PRDs it would certainly be time consumig to deploy D2 patches on all of them. On our current case, only one PRD and 3 repos for the whole company.
Did you or your customer evaluated migrating to xCP 2.x instead of D2? We are currently about to start a new project for another customer and determined that migrating their current Webtop-based applications to xCP will be the most user-friendly approach (also a big global company BTW).
Thanks Jorje for the suggestion on faceted search. I’ll make a note to suggest it to our client, perhaps we can work it into the environments and trial it after the D2 4.5 upgrade.
Personally, I have not worked with xCP 2.x and haven’t explored it lately. I’ll put it on my R&D list. Thanks for the suggestion.
In this specific case with our client, D2 had already been chosen and 4.2 deployed in production so once we began our contract we continued with patching and stabilizing it.
Besides the fact that D2 has a bad design (dependency on JMS service looks ridiculous, moreover it has a lot of security issues – EMC is trying to fix some of them, but even last versions of D2 allow to get superuser access in 5 minutes), I believe you forgot the obvious one: EMC is 5-7 years behind of the current trends (it is bit weird because the Spring Source is part of EMC too, and Spring Source has a major influence on the current trends, looks like EMC guys do not talk to each other). Let me explain. In D2 4.x EMC had switched from ActiveX to GWT, what is the problem here? The problem is in 2006 GWT had wow effect, it had started dying somewhere in 2011, and currently it is a corpse – nobody being sane will start new project on GWT, nobody being sane will start to learn GWT, after 5 years nobody will know GWT. So, the nearest future of D2 customers is very clear – EMC will force them to switch to new wonderful stillborn platform. I have no idea how it happens, but D2/GWT was not the first EMC fault in selecting framework – the previous one was TSP/XForms.
Thank you Andrey for the comment. I had not looked at the D2 developemnt platform from the perspective of the GWT framework. It’s true technology is a moving target, there always seems to be a new and better way to build a web app and UI. Any thoughts on frameworks and UI tech that you think would be good for EMC to adopt?
> Any thoughts on frameworks and UI tech that you think would be good for EMC to adopt?
It is barely possible that EMC is able to create a good modern solution in the short term. The problem is all current modern frameworks are either based on or borrow a lot from domain driven development – if my application has a clear domain model it is not a big deal to switch between frameworks, web framework just provides a bridge between UI and domain model (i.e. today I think that GWT is suitable for my needs tomorrow I may decide that AngularJS is a cool and will switch to it – domain model remains the same). Moreover, I want to talk with business analysts using domain model terms. But Documentum is doing this all backwards: you are talking with business analysts using terms “object type” and “attribute”, you are designing object model with caution whether D2/XCP/etc will able to support your object model, etc. Actually, EMC is sick with idee fixe that business analyst is (should be) able to create a ECM application on his own, so, from EMC perspective it is absolutely normal to talk with business using wrong terms. Wrong. In order to create a robust object model you need to have a fundamental knowledge about all involved technologies, business analysts do not have such knowledge and will never have – 1000 monkeys are unable to write “War and Piece”.