Starting an AI governance program with an inventory

Why inventory first

Healthcare organizations are adopting AI through many doors at once: EHR features, vendor products, scheduling and revenue tools, and staff using general-purpose assistants. The NIST AI Risk Management Framework organizes governance into four functions, Govern, Map, Measure, and Manage, and mapping begins with knowing what systems exist and how they are used.

From procurement to ongoing review

1. 1
   Procurement or build request

   ·Intake before any AI tool is bought or built
2. 2
   Inventory entry

   ·Model type and data inputs, including PHI

   ·Human review points defined
3. 3
   Validation and audit

   ·Testing for uneven performance across groups

   ·Drift and output checks on a schedule

Inventory fields to capture

System and owner

Tool name, business sponsor, and technical lead.

Use and impact

Direct patient care, clinical documentation, billing, or administrative work.

Data scope

Which PHI, personal information, or operational data it uses for training or at run time.

Human review

Where a person must review output before it is acted on.

Validation schedule

Dates for bias, accuracy, and drift review.

What to record for each tool

Name and vendor. The intended use and the decisions it informs. The data it receives, including whether it touches PHI. Who owns it internally. Whether it influences patient care, eligibility, staffing, or other consequential decisions. How it was approved and when it will be reviewed.

Federal requirements and frameworks to track

As of this article’s review date, federal regulations and frameworks increasingly emphasize transparency and nondiscrimination in healthcare AI:

HHS Section 1557 Nondiscrimination Rule

Requires covered entities to make reasonable efforts to identify decision-support and patient care tools that use protected demographics (such as race, color, national origin, sex, age, or disability) as inputs, and to mitigate potential discrimination risks.

Health system and EHR vendor considerations

Certified EHR developers face specific transparency rules (such as ONC HTI-1) for predictive clinical decision-support interventions. For non-EHR administrative, documentation, and workflow tools, oversight focuses on vendor risk assessments, data privacy, and ensuring human-in-the-loop control.

Federal review and regulatory status

Because regulations and guidance evolve rapidly, confirm current litigation and regulatory status before relying on specific provisions. Regardless of the framework, effective compliance requires an accurate baseline inventory of all active tools.

From inventory to oversight

Tier tools by risk. Set a review and testing cadence for each tier, including checks for disparate performance where the tool affects people. Route new AI tools through the same vendor assessment as any other third party. Report the inventory and testing status to the compliance committee so oversight is documented at the governing-body level.

Closing the loop

Is it working?

Every AI tool in use is inventoried, tiered by risk, and reviewed on schedule.

Can you show it?

Show the inventory with each tool’s owner, risk tier, last review, and the committee report that covered it.