Healthcare is deploying AI faster than it is defining what AI means. That distinction is not semantic – it is a patient safety question.
The pace of AI adoption in healthcare is remarkable. Across NHS trusts and clinical settings worldwide, automated systems are now handling tasks that once required human attention: summarising patient records, routing referrals, supporting diagnostic workflows, and generating documentation. Each of these deployments represents a genuine attempt to address real pressures – workforce shortages, administrative burden, and the grinding volume of modern clinical work. The intent is sound. But there is a problem embedded in the conversation that no procurement framework or governance committee has fully resolved, and it concerns the question of what we actually mean when we say “AI”.
One word, one assumption
In popular usage, and increasingly in clinical and policy discussions, the word “AI” has become synonymous with one specific kind of system: the large language model (LLM). Tools such as ChatGPT and its counterparts have shaped public understanding of what AI does and how it behaves. When clinicians, administrators, and patients imagine “AI in healthcare”, they are largely imagining a system that reads inputs and produces fluent, human-readable text in response. That association is not wrong – it simply describes a narrow slice of a much broader technical landscape, and it is creating serious blind spots in how clinical AI is evaluated and deployed.
LLMs are, at their core, statistical engines for language. They are trained to predict the most plausible sequence of words given a preceding context. They do not retrieve information the way a structured database query does. They do not reason through a problem algorithmically. They generate outputs that are probable, not verifiable – and crucially, they do not guarantee that responses are complete, consistent, or reproducible. Given the same prompt on two separate occasions, an LLM may return meaningfully different answers. It may include details absent from the source material. It may omit details that were present. For a patient with a rare presentation, a complex medication history, or abnormal values outside an LLM’s probabilistic expectations, these are not theoretical limitations. They are failure modes with direct clinical consequences.
The right tool for the task
This is not an argument against LLMs in healthcare. It is an argument for understanding where they belong. There are tasks for which their generative, probabilistic nature is well suited: drafting patient communication templates, producing first-draft summaries for human review, supporting administrative workflows where completeness is verified downstream. But there are other tasks – often the most consequential in clinical practice – where the characteristics of LLMs represent a fundamental mismatch with what is actually required.
Drug interaction checking demands exhaustive retrieval: not a plausible answer, but a guaranteed complete one. Diagnostic decision support in pathology or radiology requires that every relevant finding be identified, not a statistically likely selection of them. Regulatory audit processes require that the same query returns the same result each time. For these applications, purpose-built algorithmic systems with defined inputs, validated outputs, and traceable decision logic are the appropriate choice – not because they are fashionable, but because their architecture is aligned with the requirements of safe clinical use.
The challenge is that much of the current wave of healthcare AI deployment is not making this distinction. Governance frameworks, vendor marketing, and clinical procurement processes are frequently treating “AI” as a monolithic category, evaluating capabilities rather than architectures, and overlooking whether a given system can actually demonstrate what it did and why in an auditable fashion.
A structural audit gap
A second problem compounds this. Regardless of whether a deployed system is an LLM or a classical algorithm, AI agents now operating in clinical and administrative workflows are largely not generating the audit infrastructure that clinical governance requires. They do not consistently log which data sources were accessed, what intermediate steps were taken, or how outputs connect to the underlying inputs.
In a sector built on evidence, accountability, and the legal expectation that clinical decisions carry a defensible rationale, this represents a structural gap. When an automated system participates in a clinical pathway – reading a record, triggering a workflow, recommending an action – it should leave a trace that allows a clinician, an auditor, or a review panel to reconstruct exactly what happened. Without that trace, outputs cannot be validated. Errors cannot be identified at their origin. The clinician responsible is left standing behind a decision whose logic cannot be inspected.
The NHS already has a framework for this kind of accountability: every clinical decision requires a defensible rationale. What is needed is an extension of that principle to AI systems participating in clinical workflows, with governance specific to the type of system involved. For AI that generates language, that means enforcing human review of outputs and restricting deployment to tasks where probabilistic generation is appropriate. For AI that retrieves, classifies, or recommends, it means requiring auditability at the system level – logged inputs, traceable outputs, and the capacity to reproduce and verify results.
Where governance must go next
Neither of these requirements are technically insurmountable. Logging frameworks exist. Audit standards can be defined. Clinical AI vendors can be asked, at the procurement stage, to demonstrate traceability as a baseline requirement rather than an optional feature. What has been absent is the demand.
Healthcare’s AI governance conversation has been preoccupied with questions of bias, performance accuracy, and fairness – all legitimate concerns. But those questions assume that the outputs of AI systems can be interrogated after the fact. That assumption is not currently warranted for many deployed systems. Before asking whether an AI system is performing well, companies need to be able to ask what it actually did. The architecture of the system determines whether that question is answerable. The governance framework determines whether anyone is required to answer it.
Healthcare does not need less AI. It needs a clearer account of which AI it has, what each type can and cannot guarantee, and what evidence it leaves behind. That is not a technology problem. It is a clinical governance problem – and it deserves the same rigour that medicine applies to everything else.
By Kimber Spradlin, Chief Marketing Officer at Graylog
