When it comes to assay development for pre-clinical and clinical applications, there may be confusion about the need to either qualify or validate an assay’s performance. This confusion stems from several aspects including the fact that although the terms are not interchangeable, they can mean different things within different applications. 

In order to help ensure quality of flow cytometry assay performance across pre-clinical and clinical applications, several organizations have published recommendations for the standardization of flow cytometry instrumentation and method validation for clinical applications. These included the AAPS Flow Cytometry Action Program Committee, International Council for Standardization in Haematology and the International Clinical Cytometry Society. However, no definitive guidelines have been published, leaving scientists to rely on good practices to develop, qualify and validated robust clinical tests. [1, 2] 

To this end, many laboratories, including those of FlowMetric Life Sciences have adopted principals defined within GLP and GcLP guidelines, with which a validation strategy may be applied to demonstrate that the assay is Fit for purpose (FFP) within the Context of Use (COU). A clinical validation therefore sets out to demonstrate the effectiveness of the test for the application, in other words, to confirm that the test endpoint(s) are truly relevant to the clinical condition. [3]

Within the context of this blog, we will stick to these clear definitions:

Assay Optimization: the process of determining how a range of matrix and sample elements and assay conditions effect assay parameters and performance. These data help to set the acceptance criteria for the assay qualification.

Assay Qualification: the process of determining whether an assay is suitable, (safe and effective) for its intended purpose based on limited, pre-determined performance criteria. 

Assay Validation: the process of assuring that the assay is suitable for its intended purpose on a routine basis based on pre-defined assay performance criteria.

The Life Cycle of a Flow Cytometry Assay: 

In general, according to GLP, GcLP and ISO 15189, method qualification is recommended for assays that are developed to support early drug discovery, to ensure consistent and reproducible data. Once an assay is supporting GLP-compliant (pre)clinical studies (involving biological equivalence, safety, efficacy, and PK) then the method should be validated. However, flow cytometry validation in accordance with accepted standards for other methods is challenging due to the lack of standard materials, the inherent variability of cells and their stability, and the complexity of standardizing data analysis for a platform with no absolutes.

Step 1. Assay Design- what are the endpoints for the assay? What are the key conditions, cell populations being assessed? The basics of good panel design should be incorporated: Know your cytometer; optimize marker expression patterns and clone/monoclonal antibody selection; selection of fluorophores to minimize spectral overlap (parent descent rule can be useful for high complexity panels); take into consideration antigen density- fluorophore brightness; adopt all necessary controls; review and revise the panel as needed. 

Step 2. Assay Qualification- can the assay deliver reproducible results suitable for the intended purpose?  It is important to note that this is performed to evaluate the assay itself, NOT necessarily the clinical performance. Key points include the selection of the matrix, and establishment of the experimental protocol that can be used to demonstrate the acceptance of performance criteria required for the assay’s purpose. 

Step 3. Assay Validation- involves testing the assay against specific acceptance criteria to establish and verify that the performance characteristics of the assay are suitable, reliable, and reproducible for the intended applications. There are various degrees of validation that are summarized below

Full Validation is most frequently employed when a new bioanalytical method is developed for use in the detection of a new drug entity.

Partial Validation is necessary when an already validated method is modified. This can range from intra-assay precision and accuracy determination to an almost full validation protocol. 

Cross validation is a comparison of validation parameters when two or more bioanalytical methods are used to generate data within the same study or across different studies. 

Step 4. Continued Assay Verification- involves ongoing assurance gained during routine production that the assay remains in a state of controlled performance (fit for purpose suitability, trending of results, OOS).

Flow Cytometer Instrument Qualification

Obviously, the foundation of the performance of any assay lies in maintenance and calibration of the instrumentation and flow cytometers are no exception. Most vendors provide IQ/OQ packages to demonstrate the performance of instrument hardware and software are functioning within specifications. PQ can be driven by the intended use of the instruments. Fluorescent calibration particles can be used to confirm linearity and sensitivity, and hard-dyed beads can be employed for optical alignment. Daily instrument set up and calibration is important, and optical alignment, the fluids system, laser power and PMT voltages should also be monitored regularly. The tracking of Levey-Jennings plots can support these efforts. 

Flow Cytometry Assay Qualifications and Validations

Qualification Process:

• The selection of fluorochrome based on the abundance and importance of the antigens of interest, as well as considerations of the autofluorescence and potential spectral overlap from the cellular population of interest. 
• Clone evaluation for application (cross-reactivity/stickiness, clean signal)
• Titration of conjugates is critical for optimizing the resolution and achieving robust and reproducible staining. Conjugates should be titrated on the cell type of interest, and the inclusion of a viability marker is key. There are several different approaches to assessing titration curves, the most common being the first two shown below:

• Consideration of matrix type (fresh sample, anticoagulant selection, CytoChex etc.). The stability of the sample should also be assessed. 
• Selection of lysis-, blocking, permeabilization- and fixation buffer
• Acquisition and analysis templates and gating strategy. 

The performance of all assay components should be assessed during the qualification, such as antibody cross-reactivity. It is also a critical phase for identifying the most vulnerable features of the assay. At FlowMetric statistical analysis is frequently used to identify these vulnerabilities and pre-emptively identify risk areas such as sample variability and stability, and the technical variability of different analysts.

At the end of the qualification process, there should be an established method, and a clear understanding of the performance characteristics and limitations of the assay. These are used to establish the criteria for the validation. The qualification also provides insights into the sources of variability – from sample collection, shipment, preparation, Instrument variation, patient variability etc. and can be useful in understanding the role of controls and calibrators/standards in assessing the assays’ performance. 

Fig. 1. Summary of the Assay Qualification Process for Flow Cytometry Assays. 

Validation Process 

Typically, the validation of a flow cytometry panel for GLP or GcLP applications includes test scripts to assay the following characteristics: 

• precision (includes repeatability and reproducibility)
• analytical sensitivity [limits of blank (LoB), detection (LoD), quantitation]
• analytical specificity and interference
• linearity of quantitative and semi-quantitative assays
• stability of samples, calibrators, controls; real time vs. accelerated, storage conditions versus opened prepared samples.
• Methods comparison (510K)
• Measuring range, reference range
• Software for instrument and algorithm

Analytical Sensitivity: Sensitivity of the assay depends on the type data/output generated but the goal is to minimize false positive or false negative results. For quasi-quantitative assays, this frequently refers to the ability of the test to detect and quantify low values. An example would be assays for minimal residual disease (MRD) or CD34+ progenitor cells.  For these assays analytical sensitivity involves the limit of detection (LoD), the ability to distinguish from background -limit of blank (LoB), and the lower limit of quantification (LLOQ). For assays measuring the fluorescent intensity, such as those monitoring dimly expressed antigens, it is important to assess the ability to separate negative and positive events. 

In the case of cytometric analysis, sensitivity is determined using precision alone. There are several novel methods that can be employed to determined LoB/LoD of cytometric assays, these include: the use of FMO or FMx samples or spiking healthy samples with target cells. 

Precision: Precision of the assay describes the closeness of agreement between individual measured values when the method is repeated, with run performed on different days, using different reagent lots, by different operators, using different instruments. (Different approaches are summarized in Fig. 2)

Analytical Specificity: Specificity of the assay is performed to evaluate the interference of substances and differential diagnoses and assess the cross-reactivity of the samples and reagents. For flow cytometry methods, this measures the ability of the assay to measure the intended cell populations or antigens of interest, with the effective exclusion of events resulting from cellular doublets, contamination, conjugate-degradation or inadequate compensation.

Fit-For-Purpose (FFP) Approach to Analytical Method Validation for Flow Cytometry Applications.

FFP validation may not be a one and done activity, but rather an approach of process revision over the life cycle of the assay. For most cell-based assays there are challenges in validation: i. the complexity of the analysis and the differences in control versus disease state; ii. the type of data generated; iii. the lack of specific reference ranges and standards, to list a few.

In order to determine the extend of the validation and the application of test scripts, a risk assessment is performed based on the intended use of the assay. Low Risk applications would include drug discovery and the assessment of exploratory end points in clinical trials, whereas a high-risk assay application would include the use of the assay for companion or complementary diagnostics where the readout of the assay represents a primary end point. 

Flow Cytometry data may fall into a range of different categories ranging quantitative to qualitative.  Most flow cytometry assays are not considered quantitative because of a lack of traceability to a standard however, relative quantitative endpoints can be used to describe antigen expression levels using quantitation beads and carefully controlled measurement. Examples include cytometric assays for CD64 on myeloid leukocytes and HLA-DR expression on monocytes [4]. 

The term quasi-quantitative refers to assays that lack a calibration curve or reference standard but can be used to track temporal changes rather than quantitative determination. Examples of quasi-quantitative cytometric data includes the relative percentage of a given population of cells or the cellular concentration (number of cells/unit volume) such as CD4+ T cells, CD34 progenitor cells [5] or CD19+ B-cells. However, most cytometry data is considered qualitative, reported as either a positive/negative or ordinal formats. 

According to ISO standards, the accuracy of a method incorporates both trueness and precision- trueness represents the degree of closeness between a true value and the average value of a test series. However, by definition accuracy or trueness can’t be established for the majority of cytometric methods, and without calibration or reference standards, accuracy is often not included in FFP validations. The exception is IVD/CE-marked clinical kits that usually include a QC material for confirmation. The acceptance criteria are driven by the intended use of the assay – for assays with high imprecision, an acceptance criterion based on Standard Deviation (SD) is typically used. Westgard rules for QC monitoring generally accept a 2- SD range [6].

Selectivity and Specificity should typically be accomplished during the assay development/qualification phase, however within validation, a minimum of 3 samples from the disease state should be evaluated, compared with healthy controls. As with all cytometric methods, it is important that gate positions should be reexamined to ensure that population distribution is not influenced by the disease state. Acceptance criteria is based on risk- balancing the level of observed interference with the effect on the data. This is becoming especially important for the interrogation of samples from patients undergoing immunotherapies that may interfere with the cytometric profile of sample. Spiking experiments can help assess this impact, and in general, at least 3 concentrations (based on the expected PK levels) of spiked drug compound, into 3 healthy and three disease-state samples should be employed. 

Fig. 1. Factorial Strategies for the Assessment of Precision in the Validation of Flow Cytometry Assays.

Table 1. Summary of the Flow Cytometry Assay Types and Applications that May Be Used to Drive the Selection of Validation Strategy.

The Role of QAU in Assay Validation.

GLP concepts such as the requirements for protocol, final report, and documentation practices can be applied to a validation process at various steps. 

• A study director is appointed to manage the technical conduct of the study, as well as provide the interpretation of the analysis, documentation and reporting as defined in 21CFR Part 58.33
• All study personnel are required to have appropriate training or experience which is supported by training records.
• The validation protocol, outlining the operational ranges of all key assay parameters is prepared. 
• Applicable elements as outlined in 21CFR Part 58.120 should be incorporated into the assay protocol.
• SOPs for the use of all instrumentation are required. 
• GDP are required to be followed by trained personnel and appropriate documentation/worksheets are required to be completed during the experimental procedures. 
• All claims and conclusions around the validity of the method, should be supported by experimental data in the final reviewed report. 
• QAU fulfils its responsibilities as defined under GLP to monitor the study. This includes protocol review, critical phase audits, and final report audit.
• After completion of the assay validation, all raw data, protocol documentation and the final report should be archived. 

Final Thoughts 

When developed using qualified instrumentation and appropriate validation approaches that align with the intended use, flow cytometry methodologies are highly specific, precise, and sensitive. The performance of a clinical trial across multiple instruments, personnel, and sites requires the ability to generate comparable data that can be combined into cumulative files for regulatory filing; this all relies on a foundation of validated methods. The reduction in variability at all stages of sample collection, processing, data acquisition and evaluation is essential to the integrity and analysis of clinical samples. 

FlowMetric achieves this consistency over our labs and team members by employing core lab essentials such as standardized instruments that are harmonized across sites, standardized reagent use, SOPs, QMS and training programs and dedicated data analysts to ensure consistency across the FlowMetric Life Sciences organization. 

Dr. Julie Bick is a medicinal biochemist who has spent close to 7 years with FlowMetric Life Sciences. After receiving her doctorate in Biochemistry at Southampton University in the UK, she began her career as Associate Professor at Rutgers University, NJ, before moving to the west coast to perform biomedical research with Syngenta and Novartis at the Torrey Mesa Research Institute in San Diego. Dr. Bick specializes in biomedical engineering of cells and proteins in order to provide innovative therapeutic and diagnostic solutions. She brings to FlowMetric a clinical expertise across a wide range of therapeutic areas from autoimmunity to oncology and chronic inflammatory conditions, acquired over 25 years of research experience in academic, biotechnology and pharmaceutical laboratories. In leading FlowMetric Life Sciences’ innovation initiatives, Dr. Bick has been collaborating with BurstIQ to implement Block Chain solutions into the company’s Contract Research Organization division, with a focus on enhanced big data analytics and process control solutions in the regulated clinical environment. Dr. Bick is committed to working with local Community Colleges to support STEM programs for the next generation of scientists.

References

[1] US FDA. Context of Use. https://www.fda.gov/Drugs/DevelopmentApprovalProcess/DrugDevelopmentToolsQualificationProgram/Biomarker QualificationProgram/ucm535395.htm. Accessed 9 September 2019

[2] ISO. Medical laboratories – Requirements for quality and competence. ISO 15189. Geneva, Switzerland: International Organization for Standardization; 2014.

[3] O’Hara DM, Xu Y, Liang Z, Reddy MP, Wu DY, Litwin V. Recommendations for the validation of flow cytometric testing during drug development: II assays. J Immunol Methods. 2011;363(2):120-134.

[4] Juskewitch JE, Abraham RS, League SC, et al. Monocyte HLA-DR expression and neutrophil CD64 expression as biomarkers of infection in critically ill neonates and infants. Pediatr Res. 2015;78(6):683-690.

[5] Lanza F, Healy L, Sutherland DR. Structural and functional features of the CD34 antigen: an update. J Biol Regul Homeost Agents. 2001;15(1):1-13.

[6] Westgard, J. Basic QC Practices. Training in Statistical Quality Control for Medical Laboratories. Fourth Edition. ISBN 1-886958-30-0; ISBN-13  978-1-886958-30-2