Archiving e-mails and databases in Lotus Notes

Unimpeded access to all data – for all time

Whether personal or business, almost all correspondence is now handled electronically in the form of e-mails. Nevertheless, enterprises must still pay careful attention to ensure their records comply with auditing requirements. Fortunately, the days of having to print out each and every e-mail are over. Legislation now allows files to be archived electronically and the benefits of this for businesses are enormous. Electronic archiving is not only fast, but also entails fewer steps and less work than storing files in external archive facilities. What’s more, electronic archiving gives you the ability to automatically search through the files. Such archiving, however, must be made using formats that are uniform, permanent and unalterable. And, of course, the archive must contain all the functionalities and links contained in the original documents. The good news? This challenge has been resolved for the most part.

Groupware, such as the widely used Lotus Notes from IBM, simplifies communications processes within team- and project-based corporate structures. The software also systemizes the way documents are saved and stored. Everyone involved can edit data as a group, access the latest versions of needed documents at any time, and use extensive search functions. Everybody who serves and manages customers will be up to date at all times about everything that has been discussed. All communication is carried out through the system.

At least ten years – the standard for archiving

Lawmakers have put forth extensive guidance about archiving. Records of tax relevance must be kept available and deliverable in a manner that is machine readable. Archive systems in Germany are considered compliant with auditing requirements if they meet the statutory criteria (§§ 239, 257 Commercial Code (HGB) and §§ 146, 147 Tax Code (AO)) relating to the characteristics of being unalterable, complete, containing no gaps, comprehensible and machine readable. All records for each transaction of relevance to business activities must be retained for six to ten years at a recognized level of audit compliance, so that in their unalterable form they will be able to withstand the scrutiny of a tax audit or the process of being admitted as evidence in a court of law. This group includes business letters and other correspondence pertaining to commercial transactions, such as orders, invoices and letters of complaint. The retention periods for critical business documents, such as design and construction plans, are even longer.

The provisions for documenting business activities apply internationally. Examples include the Sarbanes-Oxley Act in the United States and the principles of data access and auditing of digital documents (called the GDPdU) in Germany. These regulate the way digital documents are to be treated the same as “classic” paper documents. Companies are increasingly being forced to archive everything. Management is not only responsible for the proces, but is also dependent on the ability to maintain long-term documentation. Internal auditing also requires that all transactions can be fully recorded, checked and understood. This is why it is recommended that every company have a professional computing system to ensure exhaustive and flawless record-keeping and archiving.

Document management is enough – right?

Unfortunately, Lotus Notes does not contain a complete archiving solution. What’s needed is a tool that can rigorously and methodically convert and transfer for long-term archiving files ranging from the many different types of Office formats, to scanned documents, all the way to complex CAD drafts. Saving and storing redundant copies of files is simply no longer enough. Reproduction over time and on any platform is a key prerequisite for data storage that is legally compliant and trustworthy in terms of audit quality.

There is no statutory regulation that prescribes what format must be used. In addition to inalterability, the requirement is merely that the format be readable 10 years into the future and regardless of the system used. The TIFF and PDF/A formats have become the de facto standards in practice. Because both guarantee manufacturer independence and faithful reproduction over time, they fulfill the most important prerequisites for an electronic archive. PDF/A differs from the “normal” PDF that everyone is familiar with in that all the components needed for displaying the document are directly embedded in the content and unalterably integrated within the file. Documents like these are 100 percent autarkic, since all of the content, texts, graphics, layout, and color and font specifications are contained in the PDF/A file, so that the original appearance of the source data is maintained – regardless of the application or system that was used to create it.

Easy conversion for all formats including hybrid files

One requirement for converters and archiving systems is that they must be capable of supporting all versions, namely the original PDF/A-1 format as well as the PDF/A-2 (released in June 2011) and PDF/A-3 (released in October 2012) versions. The two newer versions are parts of the standard that have been optimized to meet additional practical requirements. PDF/A-2, for example, also supports JPEG2000 images. PDF/A-3 lets users embed any file format they wish, such as original Excel files and XML datasets with static visual representation.

Lotus Notes systematizes communication, related e-mails, letters and other associated documents according to business transactions, such as by projects, customers and products. This allows employee e-mails to be collated with all the relevant corporate data in the appropriate databases. Documents with corresponding attachments, links and references to other data are characteristic for files in Lotus Notes. Because these structures are important, they must not be disregarded or fail to be included within the long-term archive, i.e. in the respective PDF/A versions into which they are converted.

Archiving in Lotus Notes: not a problem with specialized tools

Special tools are available that can be integrated in Lotus Notes to implement and fulfill these requirements using automated routines or the click of a button. One example is the combination of n2pdf and webPDF from SoftVision. n2pdf has been carefully engineered to work with content from Lotus Notes and convert such to formats that are safe for the future. Native converters also process other formats even when the actual software for these is not available in the system. WebPDF was developed to convert attachments to the PDF/A standard. These products also give you the option of validating whether documents already conform to the PDF/A standard.

These kinds of tools can be installed as gateways to the long-term archive to ensure proper document storage and retention. webPDF can also be integrated easily in communication platforms other than Lotus Notes/Domino, such as in SharePoint or SAP. An open interface ensures adequate flexibility for these systems. These tools from SoftVision can be integrated directly in Notes using LotusScript or Java, and can also be employed as a “Domino Server Task.” Openness means that the software also has to be able to be implemented with any other programming language should you want to employ external applications.

In any case, one critical requirement when integrating additional tools for long-term archiving in Lotus Notes is that the complexity of the data can be reproduced: complex PDF files, including those familiar and functional Notes navigation elements, such as cross references to documents, views and databases. The structures contained in Notes must be retained. In addition, security features and encryption must be defined before the conversion. In this way the software not only ensures that all information is retained, but simultaneously protected against unauthorized access.