The FDA has recently endorsed the use of computer system validation (CSV) tools to automate assurance activities. Long before this, USDM began creating its Automated Testing Tool for Cloud Assurance to automate the daily testing of Box GxP requirements and has three years of experience to reflect on automated testing in GxP regulated environments.
Test Automation Manager, Jim Lyle was a one-person testing “team” working with the initial USDM Cloud Assurance architects, Compliance Manager, Robert Corvino, and his predecessor on the Quality team. Together, they were tasked with developing a suite of tests that would help satisfy change management requirements for USDM’s Cloud Assurance subscribers while also supplementing the hundreds of Automated Testing Tool (ATT) tests that Box also runs every day.
We sat down with Jim and Robert to reflect on their lessons learned about automated testing in life sciences, why USDM’s ATT is unique, and how it is evolving.
Q: What is automated testing, and why should life science companies be paying attention?
Jim: Automated testing is a process that verifies that software is functioning as intended and is meeting requirements by conducting specific tests via automation (e.g., a programmed set of regression test scripts) as opposed to doing them manually. This method of software testing makes use of software tools to control the execution of tests and then compare the actual results with predicted or expected results, and report outcomes. Testing tools can rapidly and repeatedly simulate user keyboard and mouse activity or system-to-system data interactions, with little or no intervention from the test engineer.
Robert: With the FDA’s upcoming Computer Software Assurance (CSA) guidance and the agency’s recent endorsement of automation solutions, our ATT approach offers Computer Software Assurance as well as the inherent advantages of continuous automated regression testing.
Q: How does USDM’s ATT differ from the Box ATT?
Jim: USDM tests Box GxP cloud-based functionality for the web platform at the graphical user interface (GUI) level. The USDM Box GxP tests are run daily since Box releases occur many times throughout the week. This means we test functionality from a user’s perspective. On the other hand, Box verifies business and data-layer functionality using their internal and external application program interfaces (APIs). It’s possible that on occasion, the internal functions of the system are working correctly, but the user interface may not let a user act or may not respond to a user’s actions correctly.
Ours is a much smaller set of tests, but there is a one-to-many relationship from each test to multiple requirements tested. That is, each test may cover a set of related requirements, for example, those around Box metadata. The largest and longest-running daily script tests what I describe as the life cycle of a user via system admin interaction: from account creation, access and security requirements, admin user account controls, folder access restrictions, voluntary and forced password resets, user inactivity timeouts, and finally admin deactivation and deletion of the user account. This testing includes verifying that appropriate emails are received in the user’s email inbox. Box requires this email interaction to complete an account creation and gain access.
Robert: Also, when there is a change or enhancement to Box GxP-relevant core functionality, USDM often picks up this testing as part of our ATT. We assess each Box change and meet with our Box counterparts to decide who is best suited to cover the testing effort. Sometimes we both test—we at the UI level and Box at the API level.
Q: How do the two of you work together, with Jim as the developer and Robert as the quality reviewer?
Robert: The short answer is very well. And I also believe it’s our internal Automation/Quality partnership that makes our validated ATT work for our regulated life sciences customers. Even though the FDA has stated that it does not intend to review the validation of support tools, Jim and I have mutual respect for traditional CSV, change management, procedure, and quality assurance. From early on, I let Jim know that I had some test automation hands-on experience, and so I understood his tasks and their challenges. I assured him that the quality element, while required, would not impede his progress toward getting the first set of ATT tests launched on time. You cannot ignore the business needs of a company.
Jim: A critical factor in an automated testing Development/Quality partnership is the shared understanding that testing at the UI level introduces complexities and can result in many types of “false” or non-functional failures—from Box server and/or network performance issues, to various environmental factors such as a browser update in progress or legitimate email being misidentified as spam, to an errant test monitor keystroke or mouse movement invalidating the user inactivity test period. We developed our test failure incident severity categorization levels to take these issues into account, and to separate these from true functionality failures or change-based failures requiring additional change control documentation.
Robert: We mutually developed and operate according to our SOPs for incident and change management, and not all developers are familiar or comfortable with that. I have been at other organizations where there is an unfortunate disconnection or wariness between Engineering and Quality, and this can lead to deviations or lack of process control. Jim and I have a synergy, and sometimes, it’s Jim who reminds me that there is an SOP step or workflow step that I may be overlooking! To return the “favor,” when there is an ongoing testing challenge, I may suggest an alternative way to solve it, while also deferring to Jim’s expertise.
Jim: My understanding of and support for the Quality role stems from my dual career paths of many years of IT systems solutions development and delivery followed by an equal number of years of regulatory compliance consulting specializing in IT infrastructure qualification and computer system validation. This included moving IT systems and infrastructure from on-premises to the cloud. I gained a clear understanding that automated regression testing offers an excellent solution for maintaining and assuring a state of compliance and control for frequently changing cloud-based systems.
Q: How has USDM’s ATT process matured since its beginnings?
Robert: Well, in March 2018, the first USDM ATT production suite was a mixture of 11 automated tests and two manual tests. In October 2018, we performed an internal audit of the requirements covered by the Box Validation Accelerator Pack (VAP) against the ATT and found some minor gaps that required remediation. In November 2018, the USDM ATT was determined to be the automated equivalent of manually running the Operational Qualification (OQ) every business day, and in some ways, it exceeds the minimum testing requirements. Currently, this saves our Cloud Assurance subscribers time and money since they no longer need to execute that manual OQ as part of their validation of Box. The ATT test suite is now fully automated and currently numbers 25 test cases and counting. We even offer a new Box Relay ATT subscription for Cloud Assurance customers who want to purchase the Box Relay add-on for GxP-relevant transactions.
Jim: As our platform vendor opportunities have grown, we have expanded our Test Automation group to add several senior automation engineers. Their experience and expertise allow us to develop solutions using our validated testing tools and repository along with additional data-driven framework testing strategies for performance, flexibility, and maintainability gains, while we continue to use native tool-based record, learn, and playback capabilities for rapid prototyping and proof-of-concept development.
USDM’s automated testing began for the Box Cloud Assurance service, where platform updates can occur weekly. We have continued to add requirements coverage as new features, such as Box Relay, are added by Box. Additionally, we have now completed core system OQ and Performance Qualification (PQ) automation for a new Cloud Assurance offering for a cloud version of a major quality management system vendor. We also continually develop automation solutions for Oracle E-Business Suite on Oracle Cloud systems and have several customers’ Okta Identity Management platform use case PQs in production or development. We are continually looking at other platforms and are examining where we can help provide testing assurance for APIs and AI and robotic process automation (RPA) solution areas.
Q: What are some considerations around undertaking an automated testing effort?
Jim: When considering what makes a good candidate for automation, we first look at the frequency of application releases or updates. Daily, weekly or monthly updates needing OQ or PQ re-execution are excellent candidates for investing in automated testing solutions to save time and money over manual efforts. Secondly, we look for a definable core of standard functionality requirements where one testing effort can be leveraged for ongoing compliance-state assurance for multiple customers. As a follow-on, custom-configuration and customer-specific use case tests can be developed to supplement the core testing. Our tools work well with all web-based platform GUIs that we have encountered so far, though some internal document model structures require more crafting effort than others.
Box is primarily a document management system and would normally be considered an indirect system, with less associated risk in relation to the development, manufacturing, and distribution of medicines and medical devices. However, a customer’s use case for this system could possibly result in a higher risk-determination, requiring a more frequent or continuous end-to-end test suite execution with real-time incident reporting. Our tools and methods can be harnessed to address higher-risk use cases as well.
Robert: Other things to consider before taking on an ATT effort are that tests require a significant development and maintenance investment, so you want to make sure they have a long shelf life. The ATT life cycle requires ongoing monitoring, troubleshooting, maintenance, review, and approval to remain operational and compliant. Also, we feel an effort that supports regulated companies must include strategies and formal SOPs for ATT validation, administration, maintenance, use, incident and change management, reporting, and Quality review.
Q: Is there anything else either of you would like to add?
Robert: Yes. Both Jim and I are encouraged by the FDA’s recent support of automated tools for assurance activities. And I want to emphasize that we take our roles very seriously. As Compliance Manager, I must verify that the team’s automated scripts are actually testing what we say they are testing for our customers. We have a documented method for that, and it works. We provide assurance to those who are paying us to act as their Quality “stand-in.” Customers can’t physically be there each day to observe the test runs themselves, and the good news is they don’t need to be. Additionally, customers do receive test run and incident reports as evidence of our ongoing Compliance Assurance efforts.
Jim: As someone who was solely responsible for a lengthy qualification of the IT infrastructure for a major new GMP biological products facility, I gained a deep understanding of the costs and level of effort involved in qualification and validation of on-premises IT infrastructure and systems. Configuring and maintaining on-premise testing environments and data that are identical to the production environment is effort-intensive and error-prone. It became clear to me that the future of IT infrastructure and systems delivery would rapidly move to a cloud-based services paradigm. Maintaining the GxP validation state of these systems and services when the vendor controls the timing and rate of changes requires the creation of new life science solutions and services like those USDM provides. We feel our continual regression testing approach is safer for our customers than the traditional pretesting in the test environment. The combination of the Box architecture—with any changes rolled out to all instances nearly concurrently—tested by the daily Box and USDM ATTs, allows any issues to be discovered, reproduced, and reported to Box in minutes. Because we share the same Box code as our customers’ production instances, this greatly reduces the opportunity for any undiscovered errors.