Deployment of Next-Generation Medical Software Development System in Compliance with New FDA Requirements.
The core concept of the Pre-cert Program and AI/ML Framework is still based on the basic principles of medical device regulations, which ensure safety and confirm efficacy as the two main objectives. However, to meet the constantly evolving needs of such products, regulations not only need to ensure these objectives but also need to keep up with the challenges of the rapidly changing new generation of software. Therefore, the new review mechanism not only proposes suggested standards but also provides manufacturers with flexible opportunities.
This article will explore how to establish a system that meets these standards in practical terms, saving more development time and producing more robust software products.
Introduction:
The common points regarding the FDA’s Pre-cert Program and the requirements for the Proposed Regulatory Framework for modifications to AI/ML based SaMD are:
(1) Adoption of TPLC methodology (2) Software quality management system that adheres to robust CQOE principles (3) RWPA (Real-World Performance Analysis)
In the author’s view, the new generation of regulations provides an opportunity for developers to upgrade their structures for producing high-quality products. The purpose is not only to comply with regulatory requirements but also to ensure quality through a solid management process, saving time and maintaining high efficiency in development output.
The author’s argument focuses on practical aspects, including incorporating risk management, human factors engineering, and the Predetermine Control Plan (PCP) concept in the AI/ML framework into the design control process. Additionally, recommendations are provided for the development organization structure of the implementation process.
Differences Between the AI/ML Framework and the Pre-Cert Program
Detailed comparison can be seen as the following.
The comparison is illustrated in Table 1.
Furthermore, the proposed development process harmonized framework is shown in Fig 1 below.
Discussion on CQOE: Differences from current practices can be summarized based on the Pre-Cert Program
The following article will be served as reference.
Among the 12 evaluation domains of CQOE, as observed by the author, they are actually similar to the basic requirements of ISO-62304. However, in practical observations, the following areas are rarely implemented in many industries and therefore require strengthening.
(1) Transparency
a. Developing and maintaining systems or dashboards where all levels of the organization can rapidly see and understand how they are performing among metrics relevant to the organization.
b. Making defects, deviations, safety issues transparent to internal and external stakeholders, as appropriate.
c.Security and quality issues are communicated with internal and external stakeholders sufficiently to catalyze corrective and preventive action.
d. Buyers and users understand design assumptions about expected operational conditions/environment to use devices safely, securely, and effectively.
e. Buyers and users (patients/physicians) understand expected or minimum support lifetimes and levels.
From the author’s experience, very few companies in the world have implemented the above-mentioned items.
The transparency mentioned above can be seen as part of post-market surveillance, extending the supervision level from within the organization to external users, which increases the organization’s responsibility to the public. By making any changes after developing a product open and transparent, the organization can be held accountable by external users who can examine the changes.
(2) Configuration Management and Change Control
In some NBs in the EU, this aspect is given great importance, but it is not regulated in the current software framework of the FDA. However, from a practical operational perspective, this is a basic skill in software development processes. The author believes that it should be implemented under the requirements of regulations.
(3) Measurement, Analysis, and Continuous Improvement of Processes and Products
Typically, most businesses focus on Continuous Improvement of Processes and Products, but the measurement and analysis of the former are rarely emphasized.
If we look at the level of CMMI [2], most businesses usually fall under Level 3. If they do measure and analyze their processes, they might be at least at Level 4.
So this is also an opportunity that cannot be missed to upgrade to CMMI Level 4.
RWPA Discussions
Regarding the RWPA aspect mentioned in the Pre-Cert Program, the current industry practice involves conducting post-market surveillance (PMS) due to regulatory and quality system requirements. More proactive companies also collect customer feedback to improve their products, and some even conduct external clinical trials. These efforts not only provide additional evidence for product verification but also serve as a marketing strategy.
In my personal opinion, areas where the current industry practice needs improvement in regards to the RWPA aspect include:
Real World Health Analytics (RWHA)
Human Factors and Usability Engineering:
- Pre-Cert Organization: Support product claims by understanding user ability to comprehend and correctly navigate user interface
- All stakeholders: Demonstrate continuous improvement in usability engineering to drive health benefits and safety
Human factors engineering or usability has begun to be taken seriously in the US and EU, and more requirements have been proposed in related regulations. However, this is keeping up with the times. However, most manufacturers only focus on analysis before market launch, while the Pre-Cert Program is starting to require analysis after market launch.
Therefore, future Risk Management Reports (RM) will need to cover feedback after market launch and be combined with human factors engineering or usability reports.
Clinical Safety:
- Pre-Cert Organization: Benefit from early safety signal detection across Pre-Cert organizations
- All stakeholders: Provide assurance that safety risks are managed and mitigated in a timely way
Health Benefits
- Pre-Cert Organization: Support product claims and future claim modifications by understanding clinical benefits
- All stakeholders: Demonstrate positive impact on health outcomes
The two aforementioned items can actually be integrated into the Risk Management Report (RM) and I believe that Health Benefits must always include RM, and ultimately conduct a Risk-Benefit evaluation. This work can also be integrated with the PMCF requirements of MDR and IVDR.
User Experience Analytics (UXA)
Most companies generally meet the requirements of UXA in this area, but the requirement for clinical responsibility and excellence in product quality is lacking in the current system, which I believe is the biggest shortcoming.
User Feedback Channels:
- Pre-Cert Organization: Identify and resolve important user issues early and timely
- All stakeholders: Demonstrate clinical responsibility and excellence in product quality by ensuring that user feedback is representative of the full user population
Product Performance Analytics (PPA)
In this context, some companies continue to conduct application studies after their products are on the market, but most of them only conduct one study at most for marketing purposes, or even no studies at all after the product is on the market.
In response to this demand, a role similar to a clinical engineer will be needed to plan the experimental design and collect data.
Product Performance:
- Pre-Cert Organization: Support product claims and future claim modifications
- All stakeholders: Demonstrate sustained analytical validity and excellence in continuous improvement in product quality.
The product development approach
The design control can be applied with AI/ML framework, which can be simplified in Pre-Cert type products.
Here, it is recommended to adopt the Predetermined Change Control Plan (PCP) framework, which is basically as shown in Fig 2.
The Predetermined Change Control Plan consists of SaMD Pre-specifications (SPS) and Algorithm Change Protocol (ACP).
The basic concept of SPS is to distinguish the parts that may change in the structure and to determine the possible review format based on this.
On the other hand, ACP is a control method that controls data and procedures to meet the effectiveness and safety of the product.
For AI/ML ACP components, a comparison with non-AI/ML components is described in Table 2.
If you want to apply ACP to the Pre-Cert Program, you only need to use the last two components.
Please refer to Figure 3, which shows the audit process for software modifications in response to AI/ML architecture after the initial review, which can effectively correspond to the audit path of the Pre-Cert Program.
The use of Predetermined Change Control Plan can effectively apply to these two software management methods. More importantly, all team members only need to use one mindset to manage both products.
Therefore, the recommended harmonized PCP-based software development process framework is as follows, as shown in Fig 4.
For AI/ML frameworks, validation can be achieved through testing data. Additionally, an equivalence study must be conducted for each version, comparing it to the previous version to demonstrate equivalent or improved efficacy. The acceptance criteria for the equivalence study can be listed in the performance requirements.
At the end of each version modification, a Design Control Summary should be created to enable self-auditing using the Special 510K approval process. This summary should outline the similarities and differences between each design and be internally reviewed.
Practical recommendations
Enforcement of design inputs
Configuration management emphasized by ISO62304 can be integrated using equivalence studies, especially for baseline upgrades.
This part of the configuration needs to be defined in the Software Requirement Specifications (SRS).
More specific techniques for specification inputs will use software development techniques discussed in the GAO/2004 report such as Table 3, to establish development processes more specifically. Based on personal experience, specification inputs must be measurable (as highlighted in red in Table 3), otherwise, there can be no corresponding method for management to confirm whether they meet the requirements of the SRS.
In terms of practical operation, the implementation of design control should be carried out from the organizational structure, combined with management methods that comply with the spirit of ISO 62304 and a clear organization of output documents, such as Level 2 Procedure Documents and Level 3 Work Instructions. Detailed specifications should be developed to have measurable characteristics and can be extended to become a test plan.
Finally, combining Configuration Management with the Predetermined Change Control Plan, the design input can be illustrated as shown in Fig 5.
In addition, there are differences from traditional design inputs, as follows:
- The introduction of formative human factor testing. In terms of testing, functional testing and human factor testing are separated, and human factor testing requires two stages (formative, summative), which means that there will be at least one feedback after the design input.
- The introduction of the SPS architecture, that is, the specific introduction of the predetermined change control plan, to plan configuration management with SPS.
Organizational planning
There are three main units, establishing a development, internal testing, and review mechanism as shown in Fig 6.
The basic concept is like verifying AI with training and testing methods, where training is like the software development team, and testing is like the TE department in the organization, testing the specified specifications, similar to third-party testing. The review is like the role of an FDA reviewer.
The organization establishes three independent units to perform tasks.
The totalitarian architect sets up the coding structure, and the engineers are responsible for the coding work. These engineers can rotate tasks in the testing, review, and other departments.
In addition, the work of the review department is similar to the role of project management, even product management.
If personnel are temporarily recruited from other departments, it is project management work. If there is a product management unit to handle it, it is both product management and project management. Therefore, the design of this organization has flexibility.
Besides being the best use of limited manpower, because these engineers operate in different work functions, they can make horizontal connections in their careers.
When the company expands in the future, these people can develop towards deeper architects or horizontal project management or product management.
In fact, similar concepts can also be used in the Pre-Cert Program, testing, which can be regarded as an internal third-party testing and verification.
Post-Market Surveillance (PMS)
Post-Market Surveillance (PMS) is becoming increasingly important in current regulations such as MDR and IVDR. The requirements for PMS have shifted from passive to active, with a focus on Post-Market Clinical Follow-up (PMCF) in recent years.
In the Real-World Performance Analysis (RWPA), the PMCF plan is a key element and can be associated with the RWPA framework, as shown in Table 4. In the future, customer feedback can be collected and fed back to the design input stage.
Summary:
The above discussion presents a preliminary framework for the development and regulatory approval of SaMD and AI/ML software. This framework highlights the increasing importance of organizational considerations in addition to product requirements in the regulatory approval process. As regulations continue to evolve, companies developing new generation SaMD and AI/ML software will need to make significant adjustments to their organizational structure and thinking in order to meet regulatory requirements and market demand. The importance of robust infrastructure planning cannot be overstated, and this framework builds on the contributions of esteemed medical professionals such as Dr. Tetsu Nakamura.
[1] https://global.udn.com/global_vision/story/8663/5745058
The figure above illustrates the importance of infrastructure planning, drawing on the contributions of Dr. Tetsu Nakamura, whom the author respects.
[2] Capability Maturity Model is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.
