Run an AI risk assessment before the system makes business decisions

An AI risk assessment examines how a specific AI-enabled workflow could cause harm, fail its purpose, or create unacceptable exposure. It connects those scenarios with evidence, controls, and accountable decisions. The assessment should examine the complete business process, including what people and connected systems do with the output.

A model that summarizes internal notes faces different risks from one that influences hiring or sends instructions to production systems. Therefore, a generic checklist cannot establish that every use is acceptable.

This article explains how to scope an assessment, identify affected people, test relevant scenarios, and record a decision. It also covers human review, vendor evidence, monitoring, and triggers for reassessment.

Start with the use case and its limits

Describe the task in plain language. State who supplies input, what the AI produces, who receives the result, and which actions can follow.

For example, “draft a support reply for staff review” differs from “send a support reply automatically.” The second workflow gives the system more direct influence over customers.

Define prohibited uses too. A system built for document retrieval may not have suitable evidence for evaluating employee performance. Clear limits help staff recognize when a new request needs review.

The NIST AI Risk Management Framework provides a voluntary structure for considering trustworthiness throughout AI design and use. Apply a framework to the actual use case rather than treating its name as a certification.

Map people, data, and decisions

List the groups who may experience the system’s effects. Include users, customers, staff who review outputs, and people represented in the input data.

Next, map where information comes from and where it goes. A workflow can expose data through prompts, retrieved documents, logs, exports, and downstream integrations.

Identify the decision owner. The person operating a model API may not have authority to accept the business risk of a wrong recommendation.

Also involve people who understand the domain. They can explain which errors matter, which outcomes are difficult to reverse, and where existing safeguards already work.

Oxen’s AI governance framework guide covers the wider responsibility model. An assessment supplies evidence for those responsibilities at the use-case level.

Describe concrete harm scenarios

Write scenarios as a chain: a condition occurs, the system behaves in a particular way, and someone experiences a consequence.

For example, an assistant retrieves an outdated policy, presents it as current, and leads a staff member to give a customer the wrong instruction. That scenario suggests checks for source dates, retrieval scope, and review.

A vague entry such as “AI may be biased” is harder to act on. Specify which group, decision, input pattern, or outcome could create unequal treatment.

Also consider misuse and overreliance. A technically correct output can still cause problems if users apply it outside its intended purpose or assume it carries authority it does not have.

Do not limit the assessment to model errors. Access permissions, interface wording, and automation scope can change the overall risk substantially.

Use a framework without turning it into a score-only exercise

An AI risk framework can organize work across governance, context, measurement, and management. The NIST AI RMF Playbook offers suggested actions that teams can adapt to their circumstances.

Use the structure to find gaps in evidence and ownership. It should not become a reason to fill every box with a generic statement.

Risk scores can help prioritize discussion, but their meaning needs explanation. A low likelihood estimate based on little data should not appear as confident as one supported by relevant testing.

Record assumptions and uncertainty beside the score. Also identify scenarios that require a decision regardless of their ranking, such as an action the business has explicitly prohibited.

Assess the data the workflow actually uses

Review relevance, permission, quality, and coverage. Training data, live input, retrieved content, and evaluation examples can have different sources and limitations.

For a retrieval assistant, check whether users can access the underlying documents and whether the assistant preserves those boundaries. A helpful search result should not reveal another customer’s records.

Next, inspect stale or conflicting information. The workflow needs a response when several sources disagree or no suitable evidence exists.

Test missing and unusual inputs as well as normal cases. An empty form, a long document, or a language outside the expected audience can reveal a different failure pattern.

Also define retention. Sending information to an AI service can create copies in logs or operational systems that need their own approved handling rules.

Choose evaluation cases from the risks

Build a test set that reflects the task and the important failure scenarios. General benchmark performance does not prove that the system meets your business requirement.

Include common cases, difficult cases, and cases where the right response is to decline or request more information. State the expected behavior before scoring results.

For example, a policy assistant should distinguish a supported answer from a plausible answer with no source. A fluent response is not enough evidence of correctness.

Use reviewers who understand the task and a clear rubric. If reviewers disagree, inspect whether the requirement is ambiguous rather than averaging away the disagreement.

Keep the evaluation set under review. A system can improve on familiar examples while still failing on new patterns in real use.

Test connected actions separately

An AI system that can create records, send messages, or run jobs needs controls around those actions. Separate the model’s suggestion from the trusted execution rule.

For instance, a model may draft an invoice description, while deterministic application logic checks account access, amounts, and approval before issuing the invoice.

Limit available tools and data to the task. Broad access increases the consequence of a mistaken or manipulated instruction.

Also test untrusted content. Retrieved documents and user messages can contain instructions that conflict with the application’s intended behavior. The system needs boundaries that do not rely only on a reassuring prompt.

Record which actions need human confirmation and what that confirmation means. A hurried click through an unclear screen may provide little practical control.

Make human review meaningful

Human oversight needs time, information, and authority. A reviewer cannot reliably catch errors if the interface hides sources or expects instant approval of a long output.

Define the review task. Does the person verify facts, assess a recommendation, or authorize an action? Each requires different evidence.

Next, consider workload. If the AI produces more work than staff can review, the actual process may drift toward automatic acceptance.

Give reviewers a clear route to reject, correct, or escalate. Track those outcomes so the team can learn which failure patterns remain common.

Also test the experience with realistic users. A control that exists in the design document may not work under normal operating pressure.

Review suppliers and model changes

Ask vendors for evidence relevant to your use, including data handling, supported controls, known limits, and change notification. Marketing claims about responsible AI do not answer every implementation question.

Identify the components you depend on: model version, retrieval service, filters, prompts, and application code. Changes in any of them can affect behavior.

Oxen’s AI inference platform guide explains deployment responsibilities. Connect the assessment to the actual release path so a later update does not bypass the original conditions.

Keep a tested fallback for important workflows. The business should know what happens if the provider becomes unavailable or the system fails its acceptance checks.

Turn findings into owned controls

For each material scenario, record the control, owner, evidence, and remaining risk. A control might change the data scope, restrict an action, add a test, or require review.

Then verify that the control works. “Users receive training” is incomplete if nobody checks whether they understand the relevant limits.

Use a small action register that connects the assessment to delivery work. Unowned recommendations tend to remain in the report.

Also distinguish a planned control from an implemented one. The release decision should reflect the system that exists, not the safeguards the team intends to add later.

A worked example: internal policy assistance

Imagine a company wants an assistant to answer staff questions from approved policies. It will provide drafts and links, while staff remain responsible for sensitive decisions.

The assessment identifies stale sources, missing permissions, unsupported answers, and overreliance as important scenarios. The team defines source ownership, access checks, and a response for insufficient evidence.

It tests representative questions, conflicting documents, and users with different access. Reviewers assess source support and whether the answer stays within the approved task.

Before release, the owner records unresolved limits and the conditions for a pilot. This is an illustrative process, not a claim that one design eliminates all risk.

Monitor the conditions that justified approval

Choose signals tied to the assessed risks. These can include unsupported answers, reviewer corrections, failed tool actions, user reports, and changes in input mix.

Do not treat overall usage as a safety measure. A popular feature can still create frequent harmful outcomes.

Set triggers for review, such as a new audience, new data source, model change, or expansion from drafting to autonomous action. The assessment should follow the use as it evolves.

Oxen’s AI infrastructure guide covers supporting platform concerns. Monitoring and incident handling need to connect those technical signals with business impact.

Test the fallback in a real pilot

A fallback should preserve a useful business process when the AI feature cannot proceed. For a drafting assistant, that may mean returning users to the normal search or manual workflow. For an action system, it may mean holding the request for review.

Exercise that path before release. Check whether staff can find the original input, understand the reason for the handoff, and complete the task without losing work.

Also define who can pause the feature. An incident may require stopping one tool action or one audience rather than disabling the whole application. Clear controls let the team respond proportionately.

Finally, keep the pilot’s limits visible in configuration and support guidance. A temporary audience restriction should not disappear simply because the feature becomes popular.

Record a decision that others can review

The final record should state the scope, evidence, controls, limits, owner, and review date. It should also say whether the system can proceed, needs changes, or should not serve the proposed use.

Keep the decision proportionate to the consequence. A low-impact internal draft tool may need a lighter process than a system influencing access to an important service.

Preserve material uncertainty. A report that honestly identifies an evidence gap provides a better basis for action than one that turns every unknown into a low score.

Assess the workflow people will actually use

A practical AI risk assessment connects concrete scenarios with tested controls and accountable decisions. Start with the use case, preserve uncertainty, and make follow-up work visible. Then monitor the conditions that supported the decision to proceed.