Most AI ethics commitments are written by people who cannot change what the software does.
That is the whole problem, stated plainly. A policy lives in a document maintained by legal, compliance or the executive team. The behaviour it describes lives in a prompt template maintained by a vendor or an engineer who has never read the policy. Nothing connects them. The commitment is real, and it is also inert.
There is a case running in American healthcare that shows what that gap costs, and the structure of it should bother any leader who has signed off on responsible AI language.
The specific failure a regulator named
In US Medicare Advantage, insurers are paid more for covering sicker patients. Illness is established through diagnosis codes drawn from clinical notes, so software was built to review those notes and prompt clinicians about conditions that might apply.
Federal compliance guidance now names, among conduct its investigations have identified as potentially abusive, prompts “generated by artificial intelligence algorithms” that encourage clinicians to add diagnoses patients did not have.
Read that carefully. The objection is not to AI, and not to asking a clinician a question. It is to software that leans on a professional’s judgment, every time, identically, in one direction.
A prompt that says “the record mentions X and Y, please clarify what you observed” is asking. A prompt that says “this patient appears to have condition Z, please confirm” has supplied the answer and requested a signature. Same system, same clinical situation, different sentence. No ethics policy ever written catches that difference, because the difference is in a template nobody in the policy conversation has seen.
The asymmetry worth sitting with
Here is the part that makes this a governance issue rather than a reputational one.
That same federal guidance is voluntary and says so. The obligations sitting alongside it are not. Organisations are required to report unsupported or otherwise invalid diagnosis codes, and any resulting overpayments, under US statute and regulation.
So the ethical commitment is optional. The duty that arises once a problem exists is mandatory. An organisation with sincere intentions and software that only ever prompts in one direction is still generating an obligation, and may not know it.
That asymmetry is not unique to healthcare. It is the shape of most AI governance: the aspiration is voluntary, the consequence is not.
What the numbers look like when nobody closes the gap
The US Office of Inspector General audits whether submitted diagnoses are supported by the records behind them. Across codes it considers high risk, roughly 70 percent were not supported, some categories above 90 percent.
One audit examined 97 records where an acute stroke had been submitted. None were supported as acute strokes. In 68 of those cases the patient had genuinely had a stroke, recorded in their history, and the submitted code described it as active.
No fabrication. Systems working as designed, in one direction, for years.
Three questions for your next board pack
Has anyone read what our AI actually says to our staff? Not the policy, the prompts. Pull a hundred from last month and count how many name a conclusion inside the question.
Can our systems produce a negative finding, or only a ranked positive one? If the software cannot flag something as wrong, a policy requiring correction cannot be executed.
Where do our ethics commitments create a legal duty we are not resourced to meet? That is the asymmetry, and it rarely appears on a risk register.
The uncomfortable conclusion
Ethical leadership in AI is not a values exercise. It is a procurement and architecture decision, made by people who are often not in the room when the values are agreed.
In the healthcare case, the organisations closing that gap are buying differently: a risk adjustment solution that tests records against explicit criteria rather than scoring resemblance, because an explicit criterion can fail, and a failed test is what makes a correction possible at all. That is an unglamorous technical property. It is also the only thing that turns a commitment into a control.
