Tungsten TotalAgility Data Retention: Best Practices for Enhanced Efficiency

Retained data is a powerful resource for both business and technology staff. Business staff can easily track down issues that may surface after a transaction finishes and can use historical data to help predict future workloads. Technology staff can use historical data to troubleshoot problems and ensure security.

However, saving all that data comes at a cost. Databases grow and become more difficult to manage. Performance may suffer. Data storage costs increase. A practical balance must exist between the need for data and retention costs.

While Tungsten Automation (formerly Kofax) TotalAgility® can keep all the data it gathers and generates forever, doing so is not a best practice. Different organizations will have dissimilar needs although Genus’s experience with hundreds of TotalAgility systems helped us develop retention best practices and recommended settings. This post outlines those best practices.

TotalAgility implements default retention policies during installation, and it is easy to miss adjusting those settings during solution building, testing, and deployment. For example, forever is the default retention period for most data and document images.

Audit Log Retention Best Practices

TotalAgility enables audit logging by default, with permanent retention. Audit Logs track information about, and resulting from, executing a business process or system function. Captured data helps pinpoint and analyze the cause of any change, who made the change, and when it happened.

While useful for monitoring server activity and performance, audit data will significantly increase the TotalAgility database size. Genus recommends that Audit Logging remain enabled with data retention set to one year.

Job History Retention Best Practices

Enabled by default, Job History is a setting configured for each TotalAgility process. The TotalAgility workspace Jobs, Find menu item allows viewing history for each job. Captured history details include process variable values, workflow activity history, and process timing.

The data is useful for auditing, monitoring performance, and troubleshooting process issues. Genus recommends that Job History be enabled for every process with data retention set to three to six months depending on business needs.

Document Retention Best Practices

Typically, TotalAgility stores every page of every document ingested into a process as a binary large object (BLOB) within the documents database. Document images, particularly color and grayscale ones, are large and cause database size to grow very quickly.

In certain high-volume use cases, Genus has configured the TotalAgility process to remove document images once they successfully export to the next downstream system. While such functionality is always an option, implementing it causes documents to no longer be available for reprocessing or viewing in Job history. Genus recommends this choice only when necessary.

For most processes, Genus recommends that document retention parallel the process’s setting for Job History retention. For example, if Job History retention is set to three months, make the document retention three months. Please note that the document retention period should never be longer than the Job History retention period.

Reporting Server Retention Best Practices

Most TotalAgility environments include the Insight analytics capability. Like Job History and document retention, the collection of analytics data is enabled for each process. In this case, the default is to be not enabled. Captured analytics data stores in the reporting database.

Depending on process configuration, the volume of captured data can grow extremely fast. TotalAgility’s default for reporting data retention is ten years (3650 days). The Genus team has seen business needs satisfied with retentions as low as thirty days. Genus recommends a review of business reporting needs for each process and adjusting retention to the amount of time matching those needs. Retention values that are too high can overload reporting tables and cause performance issues.

Adjusting the Retention Policy System Task Schedule

The TotalAgility Retention Policy System task marks items for deletion once they have exceeded their retention period setting and then removes them. Default settings run this task once every seven days. Genus recommends configuring the task to run every day at a time outside of normal business hours.

In Conclusion

Maintaining an effective data retention strategy in Tungsten TotalAgility is crucial for balancing the benefits of retained data with the associated costs. By implementing best practices for audit log retention, job history retention, document retention, and reporting server retention, organizations can optimize their data management processes. Adjusting the retention policy system task schedule further ensures that outdated data is systematically removed, preserving system performance and minimizing storage expenses.

For more detailed guidance or assistance with setting up your TotalAgility system or implementing updates, please contact Genus Technologies. Our team of experts is ready to help you achieve the optimal balance between data retention and operational efficiency. Contact us today!

About the Author
Don Vogel, Senior Consultant at Genus Technologies, has been a pivotal figure in the world of Kofax/Tungsten since 1996. With extensive experience in architecting, installing, upgrading, configuring, and implementing Tungsten business solutions on-premises and in the cloud, Don's expertise is a valuable resource to the Genus team and our customers. Don continues to add to his extensive list of Tungsten certifications, reinforcing his deep knowledge and authority in the field, making him a trusted advisor and TotalAgility thought leader. Connect with Don on LinkedIn!

Get more useful content, expert tech tips, and timely articles delivered to your inbox ~ Subscribe to Genus Blogs!