While we can all agree that the AI buzz is currently loud enough to deafen a rock star, we can’t say that the buzz is necessarily ‘clear’. There are a lot of terms and ideas out there, and it can get pretty difficult to really understand what is happening with AI and what it can mean for you. This article should shed some light on that issue.

By David Shirey

Generative and Agentic

First, some fundamentals. Currently, two main AI models are in use.

The first are the Generative models.

Generative is all about finding or creating content. The content could be text, video, audio, or other formats. It requires a prompt or question to get started. That might be ‘What is the capital of Venezuela?’ or ‘What is the purpose of life?’ In both cases, the Generative model uses an LLM (Large Language Model) to search for appropriately related content, then squashes that content together to provide an ‘answer’. 

Generally, a single, standalone generative application is called a ‘bot’. It can deal with words, images, sounds, and returns either an answer that can be one word (like ‘Caracas’) or a longer prose section (like ‘This is a question that has been asked . . . ‘). There is no guarantee that the answer will be either correct or appropriate, and to a great extent, the model is just giving you its best shot. Still, the probability of a correct answer goes up if you are a) looking for a specific fact, and b) keep the question very precisely worded and the field of view narrow.

Examples of Generative AI products are all the ones we are familiar with: ChatGPT, Claude, Gemini, Jasper, and Scribe for word generation; DALL-E2 or Midjourney for visual arts and design; GitHub CoPilot, Claude Opus, and the new IBM entry, Rob, for coding; and Synthesia for audio and video generation or Elicit as a research assistant. (1)

The second type of model is Agentic.

This is all about making autonomous decisions and taking action without the need for human intervention, like actually booking a hotel room for you based on the parameters in your hotel profile or deciding that it needs to launch a preemptive nuclear strike on mankind because it thinks we don’t know what we are doing.

Agentic AI uses Agents. This term is used frequently, sometimes recklessly, but strictly it refers to small AI apps that make decisions and perform specific actions without human intervention.

Rather than being a single tool like the Generative ones, Agentic tools are really frameworks that you can use to build your application. There are quite a few of them. Still, a couple that seem to be the most often recommended are AutoGen from Microsoft, CrewAI (an open-source tool), AWS Strands Agents, Agent Bricks from DataBricks, and AskIAM from IBM, built on the Watsonx Assistant. Comparing these (and other) tools will require some effort, as their capabilities may vary significantly. Some are focused on multi-agent collaboration; some are proprietary and some open-source; some are easy to learn and use, and some have a fair learning curve. You need to decide which types of flows you will develop and who will develop them.

You may also see the term ‘foundational models’. These are sub-models that contribute to a generative or agentic model. An example would be the Large Language Model. They perform hardcore, specialized tasks and are components that you use to build the ‘bot’ or ‘agent’.

AI Project Success Surveys in the News

But how well are those initiatives translating into successful projects? What type of ROI are companies getting for their buck? How is this then driving big changes in how the company operates?

Unfortunately, the picture there is sort of vague and, for the most part, not very optimistic. There have been studies on this. Seems like everyone but me has done one. The results are the problem; there’s little agreement on the percentage of successful AI installations or on what companies are getting from them.

In fact, all of the studies agree on only one thing: that most of the AI that is happening today is being done by employees using individual AI systems, i.e., ChatGPT, etc, on their own machines, rather than a corporate AI system where the results are available to all. After that, the study results indicate a range of successful outcomes.    

Let’s start with the MIT study that has received significant publicity. This study was focused on embedded, task-specific, generative AI projects. This covered more than 300 initiatives, drawing on data from 52 executive interviews and 153 convenience surveys conducted by senior leaders. Perhaps you have already seen the results; 80% of companies reviewed are investigating AI, 60% have set up specific projects, 20% have proceeded to the pilot stage, but only 5% have been considered successful, which is measured as having been live for 6 – 12 months and showing KPI measurable increases in ROI and P&L. That success level is scary. Still, there is some controversy surrounding the study, and some articles have addressed these concerns. But other articles show business leaders buying into those figures, which of course doesn’t mean they are right (or wrong).

Fortunately, other studies have been carried out. Some mirror the MIT study, some don’t. But it is hard to compare one report with another because the ground rules may differ significantly.

The FICO report mirrors the MIT study results, but frames it as only 5% of companies that use AI have aligned their spending with business goals. I am not sure how they calculated that, and there is no explanation in the study report.

The McKinsey report reports a higher success rate (23%) but doesn’t specify at which stage of the project it is measured. Are they in in-house final testing, in a pilot, out of pilot but not installed, or installed for a fixed period? Even with a higher success rate, the report states that, for these companies, only 5% of EBIT (Earnings Before Interest & Taxes) comes from enterprise-level AI.

Wharton is the bright spot. Their survey showed that 75% of respondents (~800) reported measurable ROI. However, this study appeared to focus on Gen AI tools such as ChatGPT and CoPilot, which differ somewhat from those in other studies. The difference between this and the MIT survey may be due more to differences in measures of success than to divergent data.

Gartner did not conduct a study in the strict sense; rather, it predicted that 40% of current agentic (note: not generative) AI initiatives would be cancelled by 2027 due to rising costs, uncertain return on investment, or inadequate risk controls.

Finally, and maybe most worrisome, is that when most of these studies define the specific AI that is ‘embedded’ into the company’s workflow, it tends to be generative in nature. That is, it involves summarizing existing content, reformatting content, or creating short text items (articles, proposals, wedding vows, etc.). Most AI use is in non-operational areas, such as IT, HR, Finance, media and telecommunications, and health care. Very little of it is related to manufacturing or distribution, or on the Agentic side. 

The following table summarizes key details about the main reports.  

What Has Gone Wrong?

No matter how you slice the numbers, the odds for having a successful AI install seemed stacked against you. But what is causing this? Here, most of the above reports appear to align.

First, trying to double the company's ROI in six months, aka, trying to do too much too soon.

Second, not really understanding what the technology can do. It’s one thing to look at a flashy demo of a pre-selected case. It’s one thing to apply that to your business; it undoubtedly has different issues than the canned situation.

Third, having bad, incomplete, incomprehensible, or just plain evil data on which to base the model.

Fourth, not considering the current workflow and determining how it will need to change.

Fifth, not having strong upper management support.

Sixth, a lack of serious buy-in from the people who will use or be affected by the project. 

Seventh, departmental barriers, whether that is due to politics or the standard ‘that’s just not the way we do things here’.

Eighth, doing all of the AI development in-house. All the studies show that companies that use an external resource to assist with development or project control are more likely to succeed.

So, Can You Be Successful?

And the answer is yes. Probably. To do so, try to do as many of the following things as possible.

First, Don’t Try to do Too Much Too Soon.

You will want to think carefully about how you want to use your AI. It should be consequential, something where you will be able to measure the ROI or impact quantitatively. However, your first app will not raise your stock price by 50%.

As you consider it, decide whether this would use Generative or Agentic AI. It’s easy to think that Generative would be easier, but that is not necessarily true. Decide based on what is realistic for the current state of AI. What type of ROI can you expect (not want)? How extensive a workflow change are you looking at? Obviously, the larger the change, the harder it will be to determine the benefit and gain acceptance, but it can’t be so small that measuring ROI is difficult.

Second, Understand What the Technology can do.

Can we technologically do what you are attempting? This is important. Don’t set your sights on a ‘bridge too far’. Spec out what you want the app to do. Can each step be completed as things stand today? Really think it through.  Then research the available tools to determine which fit.

Third, Do You Have the Data?

An AI app is only as good as the data on which it is based. And, although I am sure you are different, most folk’s data is subject to error at best.

Decide what data will be needed to build this app. Do you have it somewhere? What shape is it in ( hopefully structured )? Is it accurate ( if not, how accurate )? How will you make this data ‘available’ to your AI app so that you can both train and then run the app? This is probably the most critical part of the project.

Fourth, blending AI with your existing workflows.

Once you have addressed the first three items, what impact will this app have on your current workflows? You may say ‘we don’t have any workflows,’ but that is not correct. Whether they are formally defined or not, your new AI app will fit somewhere into what someone has to do, and you have to work out (both technically and personally with the person involved) how this will affect the way things are done now. One thing – if it doesn’t seem to change things at all, then you may want to wonder if the app is really worth doing.

Fifth, have top management support.

If your AI app is worth half a farthing, you will be making some waves in the company structure. Hopefully, this project has been driven from the top down, so you do not need to build support for it. If it hasn’t, you need to secure that support, as the project budget will need to be funded. And even if that weren’t the case, you will need top-level support to smooth the way if multiple departments are involved.  And when I say ‘top’, I mean ‘TOP’. You can’t go too high in getting support for the app.

Sixth, secure buy-in from those who will use the new app.

Getting top-level involvement is key, but it is not always difficult. But getting the folks whose jobs will be affected by it, ‘that’s something else’. Especially if it involves eliminating or redefining jobs, much of this depends on the type of company you are in. Have worked with quite a few where drastic employee changes are treated as equivalent to switching one press for another. Hopefully, you are somewhere where employee effects are a big deal.

Just remember, your app has very little chance of succeeding if the people it affects are not onboard. The best way to do that is to involve users right up front.

Seventh, department barriers

Watch carefully for hidden dangers to the project. Generally, these are either personal, turf-related, or politically motivated. Try to work through that before you get to the pilot state. Involve adjacent departments as well—just a good design process.   

Eighth, going it alone versus going with some help.

As noted above, all studies agree that having a technology/PM resource to assist you increases the likelihood of success.

Yeah, I know. This makes your project more expensive. That’s life.

The End

I know. You thought it would never happen. That this article would end, that is. Hopefully, the above was worth your time.

David Shirey is president of Shirey Consulting Services, providing technical and business consulting services for the IBM i world. Among the services provided are IBM i technical support, including application design and programming services, ERP installation and support, and EDI setup and maintenance. With experience in a wide range of industries (food and beverage to electronics to hard manufacturing to drugs--the legal kind--to medical devices to fulfillment houses) and a wide range of business sizes served (from very large, like Fresh Express, to much smaller, like Labconco), SCS has the knowledge and experience to assist with your technical or business issues. You may contact Dave by email at This email address is being protected from spambots. You need JavaScript enabled to view it. or by phone at (616) 304-2466.

---

MC Press books written by David Shirey available now on the MC Press Bookstore.

		21st Century RPG: /Free, ILE, and MVC Boost your productivity, modernize your applications, and upgrade your skills with these powerful coding methods. List Price $69.95 Now On Sale