Which Tasks To Include In Human Factors Validation Testing

Failure to include critical tasks in human factors validation testing is a common cause of FDA requests for additional information following submissions, which can require re-validation and may lead to costly delays in marketing your medical device. On the other hand, including too many tasks may make the testing, analysis, and reporting unnecessarily long and expensive. Med device teams want to know, which user tasks need to be included in human factors validation and which do not. This article breaks down regulatory expectations, definitions, and practical approaches to help teams appropriately scope their human factors validation testing. The all-important flow diagram can be seen in Figure-1.

Here's the summary of the takeaways

• For HF validation of a combination product, include all tasks for which use errors could lead to any level of harm.

• For HF validation of standalone devices, at minimum include all tasks for which use errors could lead to harm that would require medical intervention to avoid permanent injury or impairment.

• As a rule of thumb, if using a 5-point severity scale, include anything of severity 3 or higher in your human factors validation for standalone medical devices

• Document rationale for your selection approach and include in your protocol and in your HFE/UE Report.

• If in doubt, include a task rather than exclude. The cost and time delay for missing a critical task is much greater than the cost and time delay for including an extra task.

• Embed your critical tasks in appropriate use scenarios during human factors validation.

• Make sure you do a thorough use-related risk analysis to uncover all potential critical tasks/hazard-related use scenarios early in product development.

• Identify which tasks are critical tasks in your use-related risk analysis.

• Include all hazard-related use scenarios during formative to uncover potential use errors that may otherwise be missed if focused on use scenarios that contain historical or assumed critical tasks.

Figure-1 Diagram showing relationship between use scenarios, hazard-related use scenarios, tasks, critical tasks, and which ones to include in human factors validation.

Those are the conclusions, for those who want to know how I came to those conclusions, you may read on for details.

Regulatory expectation of task inclusion in HF validation testing

Regulators want to see that you’ve validated the effectiveness of use-related risk controls for user tasks which have potential for harm to ensure that risks have been reduced as much as possible. That is the principle, anyways. In reality, the FDA expectations differ based on the device type. Both standalone medical devices and combination products are supposed to include all critical tasks in their human factors validation testing (Figure-2) [1]. However, the definition of critical task is different for combination products versus medical devices (Figure-3, Figure-4).

Figure-2 Excerpt from FDA Guidance on Applying Human Factors and Usability Engineering to Medical Devices Section 8 - Human Factors Validation Testing

Critical Task for standalone medical device:

Figure-3 Definition of critical task from FDA Guidance on Applying Human Factors and Usability Engineering to Medical Devices

Critical Task for combination product:

Figure-4 Definition of critical task for combination products from the FDA Guidance Application of Human Factors Engineering Principles for Combination Products: Questions and Answers

The key difference in definition of critical tasks between combination products and standalone devices is medical devices include the potential for serious harm and for combination products it is the potential for harm. This means that FDA will be expecting inclusion of any task which carries potential for harm in combination product testing, whereas for standalone medical devices FDA will be expecting inclusion of tasks which carry potential for serious harm.

Interpretation of critical tasks is relatively easy for combination products. If your use-related risk analysis uncovers any task for which there is a potential harm, it is a critical task. But the mention of serious harm in the definition of critical tasks for standalone devices begs another question.

What is serious harm?​

• FDA CDRH points to 21 CFR 803.3 (w) [3] to define serious injury.​ The definition is in Figure-5.

Figure-5 Definition of serious injury from 21 CFR 803.3 (w)

So, now it becomes clear that critical tasks for standalone devices are those for which a use error could or would lead to permanent impairment or permanent injury without medical intervention.

In summary

• For combination products FDA expects HF validation to include all tasks for which use error could lead to harm.

• For standalone devices, FDA expects HF validation to include all tasks for which use errors may result in serious harm, i.e., requiring medical intervention to avoid permanent injury or impairment.

Are expectations the same outside of the US?

Regulators and notified bodies outside of the US generally expect compliance with the international usability engineering standard 62366-1:2015 AMD:2020. In spirit, 62366-1 is aligned with the expectation of FDA CDRH. 62366-1 doesn’t use the term critical tasks but instead mentions hazard-related use scenarios, which is exactly what it sounds like (Figure-6) [2].

Figure-6  Definition of hazard-related use scenario in 62366-1:2015 AMD1:2020

You can think of a hazard-related use scenario as a sequence of tasks related to a user goal which contains a task with potential for harm. For the focus of a human factors engineer we care specifically about use errors that might lead to harm (See Figure-7). Ultimately, human factors validation testing consists of use scenarios that contain critical tasks, so when 62366-1 mentions hazard-related use scenarios it is not inconsistent with FDA expectation, although they are speaking at different levels of granularity (see Figure-7).

Figure-7  Diagram showing relationship between use scenarios, hazard-related use scenarios, tasks, and tasks with potential for use-related harm

62366-1 says that a manufacturer can choose among 3 options for inclusion of hazard related use in human factors validation (i.e. summative testing).

Figure-8 Excerpt from 62366-2: 2016 Section 12.1

Option 1 would be consistent with the combination product inclusion criteria of critical tasks. Option 2 and Option 3 would be a risk-based approach that aligns with the standalone device definition of critical task. Although 62366-1 doesn’t specify serious harm as its threshold criteria for severity-based hazard-related use scenario inclusion, it does give a parenthetical example of a severity-based selection criteria in section 5.5. of 62366-1 which says “for which medical intervention would be needed” [2]. Which aligns very closely with the definition of critical task from the FDA for standalone medical devices.

In summary

Expectations of 62366-1 are aligned with FDA’s on inclusion criteria for standalone devices. 62366-1 does not explicitly mention to include all hazard-related use scenarios for combination products. It would be good practice to include all hazard-related use scenarios in human factors validation testing for combination products even if marketing outside the united states. It is often “for free” since high-severity and low-severity use errors are embedded within the same use scenarios.

Practical Considerations

Everything mentioned so far is essentially logical conclusions made by connecting disparate guidance and standards. The remainder of the post covers practical advice coming from experience.

Do your use-related risk analysis early in product development

If you treat human factors validation and use-related risk analysis as a checkbox and don’t look at it until your QMS says it’s due, then you may identify use-related risks after it is too late. The cost of making changes to product changes by an order of magnitude with each phase of product development. A design change that costs $100 to mock up in a concept phase might cost $1,000,000 to fix during human factors validation. If teams catch critical tasks late, then they are tempted to provide additional training or labeling rather than design better risk controls or design out the potential for risk. That can get you into trouble during validation because FDA often looks for untrained groups unless you can guarantee training will be provided to all users which is often uncontrolled. Additionally, that training will add complexity and cost to marketing to product in the real world. Not only that, but additional training can be a red flag that your product is not as intuitive as it could be, and may be leading to a lesser user experience, which in turn may lead to a vulnerable market position wherein competitors may take market share by offering a better user experience.

Include all hazard-related use scenarios during formative testing

You would be wise to include all hazard-related use scenarios in formative testing, not just for combination products, but for standalone devices as well. Observations from formative often uncover new unexpected use errors and can lead to updates to use-related risk analyses and subsequently may lead to inclusion of new critical tasks in human factors validation. Testing early and often is best practice here to catch as many use errors as possible and make sure they are tested appropriately in HF validation.

Identify your critical tasks in your use-related risk analysis

Denoting which tasks are critical tasks directly in your use-related risk analysis table. This makes it easy for a reviewer to see if you considered all of the use-related risks in your protocol. If you only included critical task identification in your protocol, then the reviewer has to pull up your use-related risk analysis side-by-side with your protocol and look through line by line in your use-related risk analysis to see that you’ve considered all appropriate risks. see if you’ve omitted any risks, which can be especially difficult if your use-related risks are embedded in a dFMEA or product FMEA and scattered around.

As a rule of thumb, consider severity of 3 or higher a critical task for standalone medical devices

As a caveat, severity levels are defined by the manufacturer’s risk management process, so every situation is different.  A typical example may look like Figure-9. In the example below, a critical task would be any where use error would or could result in serious/major harm, critical harm, or catastrophic/fatal harm. Minor and Negligible harms would not trigger the ‘critical task’ label. Ultimately, it is the manufacturer’s responsibility to draw the line in determining critical tasks. This is conservative, but the time and money cost is much higher if re-validation is required (additional 8-12 weeks, $100k+ ) compared to expanding the scope of human factor validation (additional ~2 weeks, $20k).

Figure-9 Example severity table from Table 4 of 24971:2020 [4].

A note to those who are dismayed by broader definition of critical tasks for combination products

Don’t lament the broader inclusion criteria if you’re working on a combination product. It is very likely that if you were following the standalone device rules the human factors validation would look the same anyways. This is because testing is done at the scenario level. Combination devices have fewer scenarios. Using a pen injector would likely include several tasks which lead to lower and higher levels of harm in the same scenarios, meaning the same scenario would be evaluated in human factors validation under either combination product or standalone rules. In addition, the goal is to also ensure the device can be used safely for intended uses, so inclusion of the same small set of use scenarios was likely warranted for that reason as well, irrespective of the potential for harm.

Tasks inclusion decisions are sometimes simpler than task test methodology

Deciding how to test effectiveness of risk controls for a task is equally important as choosing the right tasks. Choosing the right tasks, but testing the wrong way can still lead to requests for additional information from regulators and may require re-validation and/or delay market access. This is a whole separate topic. But it is a good idea to engage a human factors professional to make sure you are using appropriate test methodology during human factors validation testing.

References

[1] Food and Drug Administration. (2016). Applying human factors and usability engineering to medical devices: Guidance for industry and Food and Drug Administration staff. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/applying-human-factors-and-usability-engineering-medical-devices

[2] International Electrotechnical Commission. (2020). IEC 62366‑1:2015+A1:2020—Medical devices—Part 1: Application of usability engineering to medical devices. IEC.

[3] U.S. Food and Drug Administration. (2024). 21 C.F.R. § 803.3 — Medical Device Reporting: Definitions.

[4] National Standards Authority of Ireland. (2020). S.R. CEN ISO/TR 24971:2020 — Medical devices: Guidance on the application of ISO 14971 (ISO/TR 24971:2020).

[5] U.S. Food and Drug Administration. (2023). Application of Human Factors Engineering Principles for Combination Products: Questions and Answers: Guidance for Industry and FDA Staff. Office of Combination Products, Center for Devices and Radiological Health, Center for Drug Evaluation and Research, & Center for Biologics Evaluation and Research. https://www.fda.gov/combination-products%20(002).pdf