The Demo Is the Easy Part: What to Ask Before You Buy Any AI Tool
John Abbitt
·
·
8 min read
AI demos are built to win one moment, the moment you decide to buy. Whether you get your money back gets decided months later, mostly by people. Here's what the research and a famous go-live disaster say about that gap, and the questions I'd ask any vendor before signing.
A lot of the calls I get from business owners right now start the same way. They just sat through an AI demo, it went well, and they want a gut check before they sign. Sometimes it's an AI receptionist that handled a test call so smoothly nobody in the room could tell it was software. Sometimes it's a tool that read a stack of supplier invoices and put every number in the right place in the accounting system. The owner liked what they saw, the monthly price looked reasonable, and the rep would love to get started next week.
My answer is almost always the same. The demo told you the truth about one thing: the tool can do the task. I believe that, because the technology really is that good now, and it gets better every few months. What a demo can't tell you is whether your business will still be using that tool, and making money from it, six months from now. That depends on who sets it up, who owns it day to day, who trains your team, and who's around when it gets something wrong. Those are people questions, and they decide whether the purchase pays off.
So my position is simple. Treat the demo as the easy part of the decision. Spend your energy on what happens after the sale, and make every vendor answer for it before you sign, including us.
Every demo is built for the day you buy
A demo exists to win one moment, the moment you decide. So it runs on clean data, a tidy example, and an expert at the keyboard who has done it a hundred times. Your business runs on messy data, strange exceptions, and people who have twelve other things to do before lunch.
The receptionist that sounded perfect in the conference room has to handle the caller who's angry, the caller who mostly speaks Spanish, and the caller asking about a job you quoted last spring. The invoice tool has to handle the supplier who changed their PDF layout last month. The scheduling agent that booked a job in the demo has to learn that you never send that particular technician to the north side of town, a rule that lives in your dispatcher's head and nowhere else.
I don't hold this against vendors. A demo is supposed to show the product at its best, and I'd do the same thing in their seat. It just means the job of finding out what an ordinary Tuesday looks like falls to you, and the way you do that is by asking the right questions.
What Hershey learned on go-live day
There's a story from the late 1990s that every business owner buying technology should know, because the pattern hasn't changed in twenty-five years. It has only gotten cheaper to repeat.
In 1999, Hershey put in a $112 million set of new systems from SAP, Siebel and Manugistics to run its orders, warehouses, billing and production forecasting. It was some of the best enterprise software money could buy. The systems went live that year, and by October, the busiest stretch of the candy calendar, orders were stuck and product wasn't getting out of Hershey's warehouses in time for Halloween. Third-quarter profits fell 19 percent, and the company ended up sitting on piles of inventory it couldn't ship.
To be fair to everyone involved, Hershey never said publicly whether the software, the rollout or both were to blame, and SAP maintained its product wasn't the problem. What we do know is how Hershey handled its next upgrade a couple of years later. The new rollout "underwent extensive testing" and was finished months before the busy season. The CIO who led it said, "Everything's running very well."
I tell this story because the lesson is about the work around the software. Buying a world-class system was one project. Getting it to run inside a real business, at peak volume, with real people doing their real jobs, was a second project, and the second one carried nearly all the risk. AI tools are far cheaper and faster to buy than a 1999 enterprise system, and I think that makes the second project easier to skip entirely. A 25-person HVAC company has far less at stake than Hershey did, and still plenty to lose: a busy season's worth of missed calls, and a team that swears off AI for good.
The incentive to oversell is real
You should hear this from someone who sells AI services: there's an enormous amount of money chasing this technology right now, and that gives a lot of companies a reason to call their product AI whether or not much has changed under the hood. Gartner, the research firm, has a name for it, "agent washing," which it describes as rebranding existing chatbots and assistants as AI agents without giving them much new capability.
Regulators have noticed too. In September 2024, the Federal Trade Commission announced a sweep it called Operation AI Comply, a set of enforcement actions against companies making inflated AI claims. One of them was DoNotPay, which had marketed a "robot lawyer" and said it would replace the $200 billion legal industry with artificial intelligence. The FTC alleged those claims were false or misleading, and the proposed settlement included a $193,000 payment. The FTC's chair at the time put it plainly: "there is no AI exemption from the laws on the books."
Most of the vendors I deal with are honest, and plenty of the tools on the market are excellent. I'm raising this because the person across the table in a demo is paid to show you the best version of the product. Your job is to find out what the everyday version looks like.
Where AI projects actually stall
The research on what happens after the purchase lines up with the Hershey story. In 2025, S&P Global Market Intelligence surveyed more than 1,000 organizations in North America and Europe and found that the share of companies abandoning most of their AI initiatives jumped to 42 percent, up from 17 percent the year before. The average organization scrapped 46 percent of its AI proofs of concept before they reached everyday use. The obstacles named most often were cost, data privacy and security risk.
Those were mostly larger organizations, so I wouldn't read the exact percentages as a forecast for a small business. The shape of it matches what I see in companies of every size, though. The tool gets bought. A pilot starts with real energy. Then the hard questions show up. Who's responsible for this thing? What is it allowed to touch? What do we do when it gets something wrong? Why is the bill higher than the quote? Without good answers, the pilot quietly stops, and the owner decides AI just doesn't work for a business like theirs.
The good news is that almost every one of those failure points is visible before you sign. You just have to ask about it, because the demo won't show it to you.
Six questions I'd ask before signing
These are the questions I'd put to any AI vendor. They're written for an owner or an operations lead, so none of them require you to understand the technology. Pay as much attention to how the answer sounds as to what it says. A good vendor answers quickly, specifically and without getting defensive.
Can I see it work on my own data? Ask for a trial or a working session using a sample of your real calls, documents or schedule. If the tool only shines on the vendor's example, you've learned the most important thing you'll learn all week.
Who on my team runs this, and for how many hours a week? Every tool needs a person who owns it. A vendor who tells you nobody does, because it runs itself, either doesn't understand your business or isn't being straight with you. You want a real number, like two to three hours a week from your office manager for the first month, because that number tells you what the tool actually costs.
Who trains my people, and what happens when I hire someone new? Find out whether training means a video library or an actual person working with your team on your own work. Then ask what it looks like for the employee you hire next spring. In my experience, "you'll pick it up fast" is one of the most expensive sentences in software.
What can it touch, and how will I see what it did? Find out exactly which systems and data the tool can reach, and whether you can start with the smallest set of permissions that does the job. Then ask to see the record of what it did: every action it took and every answer it gave, in a form a non-technical manager can read. When something goes wrong, that record is how you find out why.
What happens when it's wrong? It will be wrong sometimes, and that's fine as long as you know how mistakes get caught. Ask who gets alerted, what the tool does when it isn't sure, and how it hands a situation to a person. You also want to know who on your team will be able to tell when the AI is wrong, and whether the vendor plans to help you build that skill.
What does month three look like, on the bill and on the phone? A lot of AI tools charge by the call, the minute, the document or the "credit." Ask for a written example of your bill at your current volume and at three times that, so a busy season doesn't surprise you. Then ask who you'll talk to after onboarding ends, how fast they respond, and whether you can speak with a current customer about it.
Two more belong in the contract. Ask who owns your data and whether it's used to train the vendor's models, and get the answer in writing. And ask how you'd leave: how you export your data, how much notice you owe, and whether you're locked into an annual term. A vendor who's confident in their product makes leaving easy.
If you only have time for three, I'd ask whether it works on your data, who has to run it, and what happens when it's wrong. Those three answers tell you most of what you need to know.
Where Avolis stands
We're tool agnostic, which means we don't make more money when you pick one product over another, and we're happy to tell you what we'd use and why. We ask these same six questions on our clients' behalf, and we expect to be asked them ourselves.
Our answers are the reason we exist. We start by finding where the work in your business actually gets stuck. We help you pick tools that fit, and we explain the reasoning. We build the workflows, integrations and agents that connect those tools to how your team really operates, with only the access each job needs. We train each person on their own part of the work, write down what your best people know, and set clear rules for what the AI handles on its own and what needs a person. And we stay on the hook for the results until the new way of working is simply how your company runs.
Some clients start with an evaluation and a roadmap, some bring us in to build, and some want a partner for the long haul. If you've got a demo on the calendar and want a second set of eyes before you sign, or you'd like to talk through where AI fits in your business at all, we're glad to talk.
Sources
Computerworld, "Halloween less spooky for Hershey's this year" (January 5, 2001): https://www.computerworld.com/article/1577198/halloween-less-spooky-for-hershey-s-this-year.html
InformationWeek, "Hershey's Biggest Treat: No Tricks" (October 29, 2002): https://www.informationweek.com/it-leadership/hershey-s-biggest-treat-no-tricks
Gartner, "Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027" (June 25, 2025): https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
Orrick, "FTC Targets Unfair or Deceptive AI Practices With Five New Enforcement Actions" (October 2, 2024): https://www.orrick.com/en/Insights/2024/10/FTC-Targets-Unfair-or-Deceptive-AI-Practices-With-Five-New-Enforcement-Actions
CIO Dive, coverage of S&P Global Market Intelligence's 2025 survey on AI project abandonment (March 14, 2025): https://www.ciodive.com/news/AI-project-fail-data-SPGlobal/742590/
Ready to make AI work for you?
Book an AI readiness evaluation. If there’s nothing worth automating, we’ll tell you.
