Lessons from the USDM Cloud Assurance Box GxP Automated Testing Tool

Long before the U.S. Food and Drug Administration (FDA) endorsed the use of tools to automate computerized system assurance activities, USDM created its Automated Testing Tool (ATT) for Cloud Assurance™ to automate the daily testing of Box GxP requirements.

We’ve now had three years to reflect on automated testing in GxP regulated environments and we’d like to share some lessons learned.

Director of Automation Jim Lyle and Compliance Manager Robert Corvino were instrumental in developing a suite of tests that would help satisfy change management requirements for USDM’s Cloud Assurance subscribers while supplementing the hundreds of automated tests that Box runs every day.

We sat down with Jim and Robert to discuss automated testing in life sciences, why USDM’s ATT is unique, and how its ATT is evolving.

Q: What is automated testing, and why should life sciences companies be paying attention?

Jim: Automated testing (as opposed to manual testing) is a scripted process to verify that software is functioning as intended; actual test results are compared with predicted or expected test results, and the pass/fail outcome is reported automatically. Testing tools can rapidly and repeatedly simulate keyboard and mouse activity or system-to-system data interactions with little or no intervention from the test engineer.

Robert: With the FDA’s upcoming computer software assurance (CSA) guidance and the agency’s endorsement of automation solutions, USDM’s ATT approach can save days or weeks of validation time while offering the inherent advantages of automated regression testing during each change cycle. A risk-based CSA mindset focuses effort where patient safety and product quality are most at stake, and automation is a natural fit for that approach.

Q: How does USDM’s ATT differ from the Box ATT?

Jim: USDM tests Box GxP cloud-based functionality at the graphical user interface (GUI) level, which means we test from a user’s perspective. Box verifies business and data-layer functionality using their application program interfaces (APIs). It’s possible that, on occasion, the internal functions of the system are working correctly, but the user interface may not let a user act or may not respond to a user’s actions correctly.

Box GxP tests are run daily by USDM since Box releases occur many times throughout the week. Ours is a much smaller set of tests, but there is a one-to-many relationship; that is, each test may cover a set of related requirements (for example, those around Box metadata). Testing includes verifying that emails are received in the user’s inbox since Box requires this email interaction in order to complete account creation and gain access.

The largest and longest-running daily script run by USDM is what I describe as the life cycle of a user: account creation, access and security requirements, admin user account controls, folder access restrictions, voluntary and forced password resets, user inactivity timeouts, and deactivation and deletion of the user account.

Testing at the user-interface level catches the failures that pass at the API level - the moments when the system is technically working but the user still can’t act.

These user-lifecycle controls are also where access management and data integrity meet day-to-day compliance, so verifying them every business day directly supports the records and electronic-signature expectations of 21 CFR Part 11.

Robert: Also, when there is a change or enhancement to Box GxP-relevant core functionality, USDM often covers it with our ATT. Sometimes we both test—USDM at the UI level and Box at the API level.

Q: How do the two of you work together, a developer and a quality reviewer?

Robert: The short answer is very well. Our development/quality partnership is what makes the ATT work for our regulated life sciences customers. Even though the FDA has stated that it does not intend to review the validation of support tools, Jim and I have a mutual respect for traditional CSV, change management, procedure, and quality assurance. Jim knows that I also have test automation experience, so I understand his team’s tasks and challenges. The quality element, while required, will not impede his progress toward getting the production set of ATT tests launched on time.

Jim: A critical factor in the development/quality partnership is understanding that UI-level testing introduces complexities that can produce false or non-functional failures, from transient Box server and/or network performance issues, to legitimate email being misidentified as spam, to an errant mouse movement invalidating the user inactivity test period. We developed our test failure severity levels to take these issues into account, and to separate them from true functionality failures or failures requiring additional change control documentation.

Robert: We operate according to our SOPs for incident and change management, and not all developers are familiar or comfortable with that. Jim and I have a synergy, and sometimes, it’s Jim who reminds me that there is a step that I may be overlooking. To return the favor, I may suggest an alternative way to solve a testing challenge, while also deferring to Jim’s expertise.

Jim: My understanding of and support for the quality role stems from my years of experience with IT systems solutions development and delivery, and regulatory compliance consulting specializing in IT infrastructure qualification and computer system validation. This included moving IT systems and infrastructure from on-premises to the cloud. I gained a clear understanding that automated regression testing is an excellent solution for maintaining and assuring a state of compliance and control for frequently changing cloud-based systems.

Q: How has USDM’s ATT process matured since its beginnings?

Robert: In March 2018, the first USDM ATT production suite consisted of 11 automated tests and two manual tests. In October 2018, we performed an internal audit of the requirements covered by the Box Validation Accelerator Pack (VAP) against the ATT and found some minor gaps. In November 2018, the USDM ATT was determined to be the automated equivalent of manually running the operational qualification (OQ) every business day, and in some ways, it exceeds the minimum testing requirements. Currently, this saves our Cloud Assurance subscribers time and money since they don’t need to execute that manual OQ as part of their validation of Box. The ATT suite is now fully automated and currently numbers 25 test cases and counting. We even offer a new Box Relay ATT subscription for Cloud Assurance customers who want to purchase the Box Relay add-on for GxP-relevant transactions.

Jim: We have added several senior automation engineers to the USDM test automation group. Their expertise allows us to develop solutions using our validated testing tools while we continue to use tool-based capabilities for rapid prototyping and proof-of-concept development.

Additionally, we continually develop new automation solutions, including those for DocuSign (in production), ServiceNow platform applications (ProcessX), and have several Okta Identity Management use case PQs in production or development. We look at other platforms to see where we can provide testing assurance for APIs, AI, and robotic process automation (RPA) solutions.

Q: What are some things to consider when taking on an automated testing effort?

Jim: First, we look at the frequency of releases or updates. Daily, weekly, or monthly updates needing OQ or PQ re-execution are excellent candidates for automated testing solutions because they save time and money over manual efforts. Secondly, we look for one testing effort can be leveraged for multiple customers. As a follow-on, customer-specific use case tests can be developed to supplement the core testing. Our tools work well with all Web-based platform GUIs that we’ve encountered so far, although some internal document model structures require more crafting effort than others.

Box is primarily a document management system and would normally be considered an indirect system, with less associated risk in relation to the development, manufacturing, and distribution of medicines and medical devices. However, a customer’s use case for this system could possibly result in a higher risk-determination, requiring a more frequent or continuous end-to-end test suite execution with real-time incident reporting. Our tools and methods can address these higher-risk use cases as well.

Robert: Other things to consider before taking on an ATT effort are that tests require a significant initial development investment and require ongoing maintenance, so you want to make sure they have a long shelf life. The ATT life cycle also requires ongoing monitoring, troubleshooting, review, and approval to remain operational and compliant. An ATT effort for regulated companies must include strategies and formal SOPs for ATT validation, administration, maintenance, use, incident and change management, reporting, and quality review. Treating automated tests as a living asset is core to validation lifecycle management across your tech stack.

Q: Is there anything else either of you would like to add?

Robert: We are encouraged by the FDA’s support of automated tools for assurance activities. As a compliance manager, I must verify that the team’s automated scripts are testing what we say they are testing for our customers. We have a documented method for that, and it works. We provide assurance to those who are paying us to act as their quality “stand-in.” Customers can’t physically be there each day to observe the test runs, and they don’t need to be. They receive test run and incident reports as evidence of our ongoing compliance assurance efforts.

Jim: I have a deep understanding of the costs and level of effort involved in qualification and validation of on-premise IT infrastructure and systems. Configuring and maintaining on-premise testing environments that are identical to the production environment is effort intensive and error prone. Currently, IT infrastructure and systems delivery are rapidly moving to a cloud-based services paradigm. The combination of the Box architecture—with any changes rolled out to all instances nearly concurrently—tested daily by the Box and USDM ATTs allows issues to be discovered, reproduced, and reported to Box in minutes. This greatly reduces the chance of undiscovered errors.

As assurance work expands to cover APIs, AI, and RPA, the same principles - risk-based scope, automated regression, and disciplined quality review - extend to AI-driven and agentic workflows in regulated environments.

Related Content:

• Services: Cloud Assurance for Box GxP
• Services: Computer Software Assurance
• Case Study: Box GxP for Controlled Clinical Management
• White Paper: Box GxP Validation
• White Paper: CSA: What You Need to Know About the FDA’s Upcoming Guidance
• White Paper: Regulated GxP Workloads in the Public Cloud
• White Paper: Automate Validation Across Your Tech Stack
• Blog: FAQ: CSV vs. CSA