In 2026, the American Medical Association reported that 81 percent of physicians surveyed used AI in practice, more than double the 2023 rate. That surge did not arrive as a single, deliberate wave. It arrived in fragments: an imaging tool in radiology, a documentation assistant in the emergency department, a staffing forecast on the nursing floor, a patient-message draft in the front office, a denial predictor in billing, a scheduling optimizer in the clinic. Each product answered a different need, ran against a different budget, and reported to a different department head.
The result is a hospital that has adopted AI without fully understanding what it adopted. Ramkumar Pichandi, CEO of Rytsense, a company that builds Agentic AI and Intelligent Automation for healthcare revenue cycle management, argues that the fragmented path has left institutions without the connective tissue needed to govern these tools as a portfolio. "Hospitals often purchased AI capability before building institutional language, governance, and operational discipline," he writes. The hardest part of adoption, he says, rarely begins with the model itself. It begins when the technology enters a workflow not designed around it.
The Real Weight of What Hospitals Took On
The early case for hospital AI was practical. Administrative work was rising. Workforce pressure was intensifying. Clinicians were spending too much time on non-clinical tasks. The pressure to improve access, reduce burnout, manage costs, and move information faster made AI look like a relief valve. And in many places, it has created real value. But AI is not just another software application. Traditional software is judged by reliable function. AI requires different questions: What data was it trained on? How does it perform in this local workflow? Who reviews its output? How often is it updated? What evidence is retained when it is used?
The stakes are higher because AI does not sit still. It can change what is captured in the medical record through documentation tools. It can change which cases are seen first through prioritization algorithms. It can change staffing decisions through forecasting models. It can change how clinical intent is translated into language through patient communication systems. Each of these touches judgment, sequencing, attention, and responsibility. Pichandi puts it plainly: hospitals adopted more than efficiency. They adopted new forms of influence over decisions they had not yet named.
AI products can influence judgment, redistribute work, alter accountability, learn from changing data, and produce outputs that are difficult to reconstruct. That is a different burden than a billing system that fails to post a payment. A failed workflow is visible. A subtly biased recommendation is not. And once AI enters a workflow, it rarely remains narrow. A documentation tool changes what is in the record. A prioritization tool changes who gets seen. A forecasting tool changes who is scheduled. Each change ripples into other systems, other decisions, other people.
Human Oversight Is Not a Slogan
The phrase "human in the loop" sounds reassuring. Pichandi argues it is not sufficient. Human oversight is only meaningful when the human has time, authority, context, and a clear reason to question the output. Clinicians receiving hundreds of AI recommendations do not review each one from first principles. They cannot. The cognitive load would be crushing.
Consider a patient-message assistant that drafts follow-up instructions after discharge. The draft must be marked as machine-generated. The reviewer must see the clinical source material behind it. Staff must be trained to recognize omissions or overstated certainty. Without those conditions, the reviewer is not overseeing. They are rubber-stamping.
Pichandi says oversight must be designed as an operational function. That means defining which outputs require review, what evidence the reviewer sees, when escalation is mandatory, how disagreement is recorded, and whether the reviewer can realistically intervene. The World Health Organization's guidance on AI for health places autonomy, accountability, transparency, safety, and equity at the center of implementation. Those are not abstract values. They are design requirements for a workflow.
The WHO also published a policy guide on understanding AI in health, making a case for informed decision-making beyond hype. It examines safety, bias, governance, regulation, and trust. The message is consistent: oversight is not a checkbox. It is a system of roles, tools, and escalation paths that must be built before the AI is switched on.
Validation Is a Lifecycle, Not a Certificate
Another common misunderstanding is that validation happens once, before deployment. Pichandi warns that AI performance can shift after the go-live date. Patient populations change. Coding practices change. Clinical documentation changes. New equipment arrives. Staff behavior evolves. Vendors push updates. Data interfaces break. Any of these can degrade a model that performed well in testing.
The National Institute of Standards and Technology developed the AI Risk Management Framework to address exactly this. It organizes risk management around governing, mapping, measuring, and managing risk as a continuous lifecycle activity. That framing matters because it treats risk as something that moves, not something that is retired after approval.
Post-deployment monitoring should include performance by patient group, override patterns, unusual output distributions, workflow delays, user complaints, downstream corrections, and signs of unintended reliance. These are not optional analytics. They are the early warning system for a tool that can silently drift.
Stay ahead of the AI curve
The most important updates, news, and content — delivered weekly.
No spam. Unsubscribe anytime.
The Office of the National Coordinator for Health Information Technology issued the HTI-1 rule, which introduced transparency requirements for predictive algorithms in certified health IT. The rule covers fairness, appropriateness, validity, effectiveness, and safety. It is a regulatory step toward forcing the questions that hospitals should already be asking. But a rule on paper does not create a monitoring dashboard. That work belongs to the institution.
Clearance Is Not Readiness
The U.S. Food and Drug Administration maintains a list of AI-enabled medical devices that have met premarket requirements. That list is useful. It is not sufficient. Pichandi is explicit: FDA clearance answers regulatory questions, not institutional readiness questions. Readiness involves data, staffing, infrastructure, training, escalation pathways, and the specific patient population in front of the hospital.
The same logic applies to other approvals. Security review does not establish clinical appropriateness. Privacy review does not establish workflow safety. Legal approval does not establish user competence. A successful pilot does not establish enterprise readiness. Each of these checks answers a different question. None of them answers the question of whether this tool, in this hospital, with these clinicians, for these patients, will do more good than harm.
Pichandi argues that every AI implementation introduces five things at once: decision influence, data dependency, workflow redesign, monitoring obligation, and trust relationship. That is why a hospital can deploy dozens of AI tools and still feel unprepared. The tools are not the problem. The absence of connective tissue is. Some hospitals lack a complete view of where AI is operating. They do not have a reliable inventory of what is running, who owns it, and what it touches.
The first practical step is exactly that inventory. It should include the purpose of each system, the decisions it influences, the data it uses, the population it affects, the vendor and model version, the human owner, the review process, the escalation path, and the evidence used for approval. An inventory alone is not governance, but it is the foundation. You cannot govern what you cannot see.
Accountability begins with every system having a named executive owner and an operational owner. The executive owner answers for the tool's existence. The operational owner answers for its daily behavior. Without those two roles, a tool can drift for months before anyone notices.
From Enthusiasm to Evidence
Hospitals do not need perfect certainty. They need to replace enthusiasm-led scaling with evidence-led scaling. That means asking hard questions before expanding a tool's role. What judgment does it influence? What happens when it is wrong? Who notices? Who can stop it? What evidence would justify expanding its role?
Pichandi says the most important question is no longer "Does this tool use AI?" The question is whether the institution can understand what the tool is doing, why it is doing it, and what happens when it fails. That understanding must reach clinical leadership, operations, compliance, technology, finance, quality, and the board. AI is too distributed to be governed by a small group of specialists. It is too consequential to be understood only as a technical subject.
The hospitals that benefit most will learn fastest from every deployment. They will treat implementation as institutional knowledge. Every pilot, every failure, every override pattern, every user complaint becomes data for the next decision. That learning loop is the real competitive advantage.
The responsible response is not retreat. AI is already creating value in healthcare, and the pressure to improve access, reduce burnout, and manage costs is not going away. The responsible response is building capacity to understand AI as it actually operates, in real workflows, with real patients, under real constraints.
Hospitals adopted AI before they fully understood what they were adopting. The path forward is not to un-adopt it. It is to build the governance, monitoring, and accountability that should have come first. The technology is already in the building. The institutional capacity to manage it is what comes next.