Practical Concerns with Algorithms and Rapid Microbiology Systems
By Jeanne Moldenhauer Vectech Pharmaceutical Consultants, Inc.
The majority of computerized rapid microbiology systems use algorithms to make decisions about the data generated. An algorithm can be described as a formula, or series of logical statements that can be used to make decisions regarding data. For example, one algorithm, or a series of algorithms, can be used in a system that enumerates microorganisms to determine if the item detected is, or is not, a microorganism. Very little information has been published to provide guidance for industry regarding how to validate these algorithms and what things should be considered when evaluating these parameters. This discussion covers a few practical concerns the User faces.
Validation of Algorithms The companies that have developed them consider most, if not all, algorithms proprietary information. Like all proprietary information, the Vendor does not want to provide other people the details of each algorithm because they do not want other people to be able to replicate their technology. Some companies will not even provide information on how many different algorithms are used in their system. This makes it very difficult for the end-user to appropriately define validation parameters.
Typical validation studies are designed, treating the algorithms as a "black-box", i.e., one can only test the functionality of the system as a whole and cannot test each algorithm individually. One may find that a system's User's Manual or Technical Manual describes some functions that are controlled by algorithms, e.g., discriminating between particulates and microorganisms. When these functions can be identified, one may be able to design specific tests to evaluate the function of these "undefined" (to the User) algorithms. For example, if company information states that all items larger than a certain size are treated as particulate rather than microorganisms, one may want to verify that items of that size (as closely as possible to test) are discriminated as not being microorganisms.
It is useful in writing validation documents to add some verbiage that describes how your company addresses testing of the algorithms present, e.g., treated as black box testing since the algorithms are proprietary to the vendor.
Some manufacturers of rapid microbiology systems may have validated the functionality of all algorithms and included this information in their Drug Master File. This is useful since one can obtain a Cross Reference Letter of Authorization from the Vendor, which allows the FDA to review the validation of algorithms as part of the review of the regulatory submission for a company. This methodology allows for complete testing of the algorithms while protecting the Vendors proprietary information. One may choose to query the Vendor on whether this testing has been previously submitted to FDA.
Addressing Software Updates and Preventative Maintenance
It is common for the Vendor of rapid microbiology systems to provide services that include updates of software as part of a preventative maintenance system. As a User, one needs to be able to identify when these changes include changes to the algorithms. Since they are proprietary the Vendor may not routinely provide this information to the User. One way the User can address this situation is to have a contractual agreement that requires notification when these types of changes occur, along with information on what types of functions are controlled by the changed algorithms. This information is needed to limit the amount of re-validation that may be necessary.
Other Changes that May be Difficult to Address
Consider the number of microorganism identifications that have changed in Bergey's Manual over the last ten years. Depending upon how that data is used in the laboratory, one may need to address how the software will be updated, what algorithms are used to address these changes, how often changes are made throughout the year, and so forth. Say for example Bergey's Manual changes organism names every month. As a User, one must evaluate the validation work that would be required to implement these changes on a monthly basis, versus a longer periodic interval for incorporating updates to the program. Another concern is whether other algorithms or program logic has been made along with the changes that are identified.
The situations represented in this article are just a sprinkling of the types of issues faced by end-users in assessing issues with algorithms
This article is taken from the June 2004 RMUG newsletter. |
