AI risk assessment: how to score a use case before launch

An AI risk assessment answers one question about one use case: how much could go wrong, and so how much control does it need before it goes live? It sits between the idea and the build, and what it produces is a decision with conditions attached. A thirty-page report that nobody acts on is a sign the assessment was done for its own sake.

It is narrower than a governance framework. The framework is the organisation's rulebook: the policy, the register of AI uses, the risk tiers and the controls each tier requires. Our AI governance framework guide covers that, with a policy template you can copy. The assessment is how one use case gets placed on those tiers. If you have no framework yet, the tiers on this page will do as a starting point, and the framework can grow out of your first few assessments.

It also sits next to a data protection impact assessment (DPIA) rather than replacing it. Under UK GDPR, processing of personal data that is likely to result in a high risk to people needs a DPIA before the processing starts, and the ICO's guidance on AI and data protection says that in the vast majority of cases the use of AI will meet that bar. The ICO's list of processing likely to be high risk covers new technologies and the novel use of existing ones, naming AI, as a criterion that, combined with others, makes a DPIA necessary. The scores below give your data protection officer much of what a DPIA needs, but the DPIA stays theirs to run.

Do the assessment at the idea stage, well before the security review. A control designed in is a few lines of the build plan. The same control retrofitted after a reviewer blocks the launch means rework and a delay, and a late no from legal can cost the whole build.

Score each factor 1, 2 or 3 from the descriptions in the table, then add them up. The six are the things that change what a failure looks like: what the system can see, what it can do, who it touches, what a mistake costs, who can reach it and how often it runs. The scale is ours, so move the thresholds to fit your own risk appetite. Then keep them fixed, because the value is in comparing one use case with the next on the same terms.

Score the system as it will be built. The pitch is not the design. "A person reviews every output" only scores as a person reviewing every output if the design makes them do it, and if they will have the time.

Exposure is the factor that gets left out. A system that reads inbound email reads text written by anyone who can send you an email, and the UK government's AI Playbook names prompt injection, alongside data poisoning, perturbation attacks and hallucinations, among the threats that are specific to AI. Untrusted input plus a system that can act is the combination to look for first.

Six factors scored 1 to 3 give a total between 6 and 18. Add one rule on top, a hard trigger: a 3 on autonomy or on effect on people puts the use case in the high tier whatever the total. Adding scores up is a convenience, and it hides exactly the case you most need to see, a small internal system that runs twenty times a week and decides something about a person.

A high tier rarely means stop. It means the controls have to exist before the system does, and quite often the cheaper move is to redesign the use case until it scores lower, which is what happens in the worked example below.

For the actions that stay consequential after a redesign, the agent proposes, verifier gates pattern is how we build the check: the model proposes, deterministic rules decide whether the action goes ahead, and anything that fails goes to a person with the reason attached.

This example is invented to show the method. The retailer, the volumes and the design are illustrative, and none of them is a 1AYM client, a client result or a benchmark. A fictional online retailer receives about 3,000 customer service emails a week. The proposal is an AI assistant that reads each email, looks up the customer's orders, writes and sends the reply, and grants or refuses refunds of up to £75 on its own.

Scored as proposed, it lands in the high tier twice over: on its total, and on two hard triggers. So the team redesigns it before the build, which is cheaper than arguing about controls for a design nobody needed. In the redesign the assistant drafts every reply for an agent to send, can propose a refund but never grant or refuse one, and looks up orders only for the sender's own email address, a limit built into the lookup itself so the model cannot widen it.

Notice what did the work. The biggest drop in risk came from the redesign, before anyone wrote a line of code, and none of it was exotic. That is the case for running the assessment while the idea is still cheap to change.

A medium score says how much control to apply. Choosing the controls takes a risk register: the specific ways this design could fail, rated for likelihood and impact before any control, then a control and an owner for each, then the rating once the control is in place. These are the four risks that matter most for the fictional assistant.

The decision: proceed to build at the medium tier, with the four controls as conditions of launch. Automatic refunds are not ruled out for good. They come back as a separate assessment once three months of logs show how often agents change the refund the assistant proposes, which is evidence the original proposal never had.

A late block from legal, security or risk is usually a question nobody asked them early enough. The UK government's AI Playbook tells public servants to engage compliance, legal and data protection experts early, including during product development, and to work with commercial colleagues from the start. The advice travels well beyond government. Take each function the questions below, with the assessment, at the scoping stage.

None of this is legal advice. It is the set of questions these teams usually ask, in an order that saves a rework. Your legal adviser and data protection officer decide what the law requires of you.

For US buyers the usual reference point is the NIST AI Risk Management Framework. AI RMF 1.0 was released on 26 January 2023 and is intended for voluntary use, and NIST says it is being revised as part of the White House AI Action Plan. It has four functions: Govern, Map, Measure and Manage. The template on this page covers the Map and Manage steps, and part of Measure, for a single use case; the governance framework is where Govern lives.

For generative AI, NIST's Generative AI Profile (NIST AI 600-1, 26 July 2024) names twelve risks that are unique to generative AI or made worse by it. The four risks in the worked example fall under four of them: information security, confabulation, human-AI configuration and data privacy. The list is worth reading when you write a risk register, because it includes risks that are easy to forget, such as harmful bias or homogenization, intellectual property, and value chain and component integration.

The framework is voluntary, and sector regulators, state laws and your own contracts may ask for more. Banking is one example: SR 26-2, the revised model risk guidance the Federal Reserve issued with the FDIC and the OCC, places generative and agentic AI outside its scope and leaves their controls to each bank's own risk management, and our guide to SR 26-2 and generative AI sets out a control set a model risk team can build on. This page is not legal advice on any of these.

Central government adds two expectations. The AI Playbook for the UK Government asks that humans validate any high-risk decisions influenced by AI and, where real-time review is not possible, as with a chatbot, that human control sits at other stages. And central government departments and arm's length bodies in scope are required to use the Algorithmic Transparency Recording Standard; the Playbook encourages other public sector bodies to use it too. If you are in scope, plan that record alongside the assessment.

This section summarises published guidance and is not legal advice. Our public sector page explains how UK public bodies can work with 1AYM, and what G-Cloud covers.

Fill this in at the idea stage, share it with the functions above at scoping, and update it before launch and at each review. A line you cannot fill in is a question to answer before the build.

Scoring candidate use cases on value, feasibility and risk is part of 1AYM's AI Opportunity & Feasibility Sprint, which is fixed scope and typically takes two to four weeks. Its governance review establishes what has to be true for legal, security and risk to sign off before anything is built, so the questions in this guide get answered at scoping rather than in the last week.

The controls a medium or high score calls for are engineering: permissions the system enforces, checks on consequential actions, tests in CI and an audit trail. We build them as AI governance implementation, as part of our production AI systems work, and we take small fixed-scope statements of work as well as larger builds. On a government-accredited EdTech's speaking assessment, for example, the confidence gate is set to route about 15% of interviews to a human examiner, with a further 5% audited at random. If you have an assessment half done and want a second opinion on the scores, book the call below.