Many companies have spent the past few years experimenting with AI. Now, as the technology moves into business workflows in critical sectors like healthcare and finance, they are confronting a harder problem: how to stay in control once software starts taking action on its own.

By Sascha Brodsky, Staff Writer, IBM

Recent recognition of IBM as a Leader in the inaugural Gartner® Magic Quadrant™ for AI Governance Platforms comes as organizations seek ways to manage increasingly capable AI systems. Companies are moving beyond AI pilots and using the technology inside everyday business processes that require visibility into risk, controls and compliance.

“The conversation around AI has shifted dramatically over the last two years,” Dinesh Nirmal, SVP of IBM Software, told IBM Think in an interview. “Companies are no longer asking whether they should use AI; they’re asking how to deploy it responsibly across the enterprise.”

When AI joins the workflow

The first wave of enterprise AI largely centered on experimentation. Companies tested chatbots, explored large language models (LLMs) and searched for ways to improve productivity.

Many of those projects remained separate from the systems that run businesses day-to-day. IBM said in a blog post that it is beginning to change as organizations weave AI into customer service, administrative and operational tasks.

That shift brings a new set of questions. IBM said companies increasingly want to know where AI is being used, what information it can access and who is responsible for its actions. Those concerns become more pressing as organizations deploy multiple models across departments and begin experimenting with AI agents, software systems that can perform tasks rather than simply generate content.

“The organizations that succeed with AI will be the ones that can scale innovation while maintaining visibility, accountability and control,” Nirmal said.

Responsible AI in healthcare

Healthcare offers an early glimpse of what those challenges look like when AI moves from experimentation into environments where mistakes can carry real consequences.

The challenge becomes easier to see in healthcare. Hospitals want AI to reduce paperwork, streamline operations and free up clinicians’ time. They also need to know who remains responsible when technology becomes part of decisions and workflows that affect patient care.

Those competing priorities shaped how ViClinic developed its agentic healthcare operating system, or AHOS, a platform designed to deploy AI agents within clinical and administrative workflows. The company built the platform using IBM technologies, including watsonx.

“We quickly realized that AI adoption in healthcare is not primarily a technology challenge. It is a governance challenge,” Dr. Boris Jinjolava, Founder and CEO of ViClinic, a healthcare technology company and IBM partner, told IBM Think in an interview.

Healthcare organizations wanted more than sophisticated AI tools, Jinjolava said. They wanted to know how those systems would fit into existing clinical processes and who would remain responsible for the outcome.

That concern shaped ViClinic’s approach from the beginning. “Health systems need confidence that AI actions are transparent, auditable, supervised and aligned with organizational policies,” he said.

Jinjolava said healthcare organizations need to know what an AI system did, why it did it and who reviewed the result before it became part of a patient’s record.

The autonomy dilemma

Those concerns become even more important as organizations begin deploying AI agents. Unlike traditional AI tools that answer questions or generate content, AI agents can perform tasks, move information between systems and help coordinate activities across a workflow.

Organizations see agents as a way to automate tasks that once required employees to move information, complete paperwork or coordinate activities across teams. Jinjolava said the challenge is ensuring those systems operate within clearly defined boundaries.

“AI should support decisions and workflows, not act as an uncontrolled autonomous system,” he said.

The company designed the system so clinicians remained responsible for approving key outputs. “Rather than maximizing automation, we prioritized transparency, accountability and human oversight,” he said.

Building trust before scale

Building trust often comes down to deciding who reviews what and when. Jinjolava said governance frameworks help healthcare organizations establish approval thresholds, escalation paths and review processes before introducing AI into critical workflows.

Those safeguards allow organizations to begin with limited use cases, evaluate outcomes and expand adoption over time. Jinjolava said that the approach gives healthcare organizations an opportunity to test AI-assisted workflows before introducing them into broader operations.

The same questions confronting hospitals are beginning to emerge across industries as organizations experiment with more autonomous AI systems. IBM said organizations increasingly want tools that provide a clearer view of AI models, vendors and systems as AI becomes part of more business activities.

For years, companies worried about whether individual AI models would produce inaccurate answers. Jinjolava said the bigger challenge may be what happens when those systems become part of larger business processes.

“The risk shifts from a single model to an entire workflow,” he said.