AI Project Data Quality Failure: How to Tell If Bad Data Really Sank It

An AI project data quality failure is one where the AI's wrong outputs trace back to the records it was given: information that was missing, wrong, or out of date when the AI used it. Plenty of projects get that label without earning it. The same wrong output can come from three other places (nobody agreed on what the right answer was, the AI misread inputs that were fine, or the process around the AI broke), and each of those needs a different fix. The way to tell them apart is to take the outputs that actually went wrong and trace each one back to where it started.

That matters because "bad data" is the most common explanation for a struggling AI project and the easiest one to accept without checking. It sounds technical and spreads the blame across everyone who ever typed into a system. It also points toward a large cleanup that may not touch the errors your customers are complaining about. If your AI is already producing wrong answers, your data is almost certainly messy somewhere, so the question worth answering is how many of those wrong answers the mess actually explains. Our guide to why AI projects fail covers the wider set of causes behind failed projects.

Key Takeaways

• An AI project data quality failure is one where the wrong outputs trace to missing, wrong, or stale input records. Many failures blamed on data come from somewhere else.
• In BARC's 2025 study of organizations delivering AI projects, 44% named data quality as an obstacle, more than named skills shortages, integration, or budget. In BARC's 2024 research, 19% had called it a major concern.
• Wrong outputs come from four places: the input record, the definition of the right answer, the AI itself, or the process around it. Sorting 30 to 60 real errors into those four groups shows which one you have.
• "The right answer" is often less settled than it looks. In a noise audit at an insurer, underwriters pricing identical policies differed by a median 55%, against the 10% median that 828 executives across industries said they'd expect.
• The Epic Sepsis Model scored 0.63 in an outside test on a standard measure of how well it ranks patients by risk, against 0.76 to 0.83 in Epic's own documentation. On the same records, the score rose to 0.80 when predictions made after sepsis began were counted.
• When the cause really is data, the errors usually start at the step where a record is created today (a photo not taken, a unit number not updated), so the fix is a change to that step.

Table of Contents

• What counts as an AI project data quality failure?
• Why "bad data" gets the blame first
• Is it really the data? Four places a wrong output can come from
• How success gets defined changes the verdict
• How do you trace failed outputs back to their cause?
• What each cause takes to fix
• When a vendor says your data is the problem
• An 80-person equipment rental company's damage charges
• Where Avolis fits
• Frequently Asked Questions
• Continue Learning

What Counts as an AI Project Data Quality Failure?

An AI project data quality failure is one where the AI did what it was built to do with the information it had, and the information was the problem. The delivery address on the order was really the billing address, a customer's tax-exempt certificate had expired without anyone updating the account, or a field meant one thing in one branch and something else in another. Give the same AI correct, complete, current inputs and it would have produced the right answer.

That last test is the one that separates a data failure from everything that looks like one. If the AI would still have been wrong with perfect inputs, the data wasn't the cause, however untidy it might be. Where two of your own experienced people would have disagreed about the right answer, the problem sits upstream of both the data and the AI. An answer that was right but reached the customer late, unchecked, or in the wrong form points to the process around the AI.

Most struggling projects have a mix of all four. What changes the decision is the proportions, because a project whose errors are 70% data needs a different plan, a different budget, and often a different owner than one whose errors are 70% disagreement about what "correct" means. Our companion page on why AI projects fail due to data quality sets out where bad data tends to enter a project and at which stage. Confirming whether data is what's going wrong in yours takes a count of real errors, not a look at the systems.

Why "Bad Data" Gets the Blame First

Bad data gets the blame first because it's the explanation that suits nearly everyone in the room. For the vendor or consultant, it locates the problem in the client's systems rather than in the build. For the business, it's a familiar complaint that nobody has to take personally, since the records were created by dozens of people over many years, and it matches the common wisdom that AI is only as good as its data.

The surveys reflect how quickly data has become the leading complaint once AI goes live. BARC, a European analyst firm, asked 421 organizations what slowed or stopped their AI projects. In 2025, 44% named data quality, more than named skills shortages, integration issues, or budget constraints. In BARC's 2024 research, only 19% had called it a major concern (BARC, "Global Study Maps the Reality of AI Delivery in 2025", December 2025; BARC, "The AI maturity divide", January 2026). BARC ties the jump to projects reaching production, where AI surfaces gaps that manual processes had kept hidden.

Those numbers describe what respondents named, not what a root-cause analysis found. "Data quality" in a survey answer can mean a missing field, a definition nobody wrote down, an integration that drops records, or a general feeling that the systems are a mess. That breadth is exactly why the label is so easy to reach for, and why it can send a business into an expensive cleanup aimed at the wrong target.

The trouble is that each misdiagnosis has a cost. Blame the data when the AI is misreading good inputs, and you pay for a cleanup while the vendor's fix waits. Blame the AI when your own team can't agree on the answer, and you replace the tool and get the same disagreement from the next one. Counting where the errors come from is what tells you which of those mistakes you're about to make.

Is It Really the Data? Four Places a Wrong Output Can Come From

Every wrong output from an AI system starts in one of four places, and you can find which by asking four questions in a fixed order about each error. Did the AI have the information a competent person would have needed, and was it correct at the time? Would two of your experienced people agree on the right answer? Given good inputs and an agreed answer, did the AI produce that answer? And if it did, what broke in the process after it?

The order matters. Data comes first because it's the cheapest to confirm (you can look at the record the AI read), and the definition question comes before the AI question because you can't say the AI was wrong until you know what right was. The table below sets out what each cause looks like in an operations business and what you'd find when you trace it.

The input was missing, wrong, or out of date

A data cause is the easiest to confirm because the evidence sits in a system. Pull up the record the AI read at the time it produced the answer, and compare it with what was true: the signed delivery ticket, the current price sheet, the supplier's actual invoice. If the record was wrong and the AI faithfully used it, you have a data error.

The detail that trips people up is timing. A record may be correct today because someone fixed it after the complaint came in, so the trace has to look at what the AI saw, not at what the system shows now. Where your software keeps a change history, use it. Where it doesn't, the person who handled the complaint usually remembers what was corrected.

Nobody agreed on the right answer

It often gets filed under "bad data," and it's a different problem. It shows up when the AI was trained or instructed to match a standard that your own people apply inconsistently, so the "wrong" output is really one of several answers your staff would give.

Experienced professionals disagree with each other far more than their managers assume. When Daniel Kahneman and colleagues ran a "noise audit" at an insurance company, they found that the median difference between two underwriters pricing the same policy was 55%, and the median difference between two claims adjusters valuing the same claim was 43%. When the authors asked 828 senior executives across industries how much variation to expect in expert judgments, the median answer was 10% (strategy+business review of Noise by Kahneman, Sibony, and Sunstein, Autumn 2021). An AI trained on those underwriters' past prices would inherit the spread, and anyone grading its output would be grading against one underwriter's view.

Generative AI adds a twist. Researchers at UC Berkeley studied people writing the criteria for judging AI outputs and found what they called criteria drift: "users need criteria to grade outputs, but grading outputs helps users define criteria" (Shankar et al., "Who Validates the Validators?", UIST 2024). In plain terms, people often don't know exactly what a good answer looks like until they've seen some bad ones. A project that never writes those criteria down will keep moving the target, and the AI's measured error rate will move with it.

The inputs were fine and the AI still got it wrong

Once the inputs check out and your people agree on the answer, a remaining error belongs to the AI or the way it was set up. That covers a document reader that misreads handwriting and a model that never saw enough examples of a rare job type. It also covers instructions that leave out a rule your staff apply without thinking, and a confidence threshold set so the AI guesses when it should ask.

These errors are the build team's to fix, and they're usually the cheapest to fix once they've been isolated, because the fix happens in one place. Isolating them is the hard part. If data and definition problems are mixed in, every proposed fix gets argued over, and the vendor can fairly say the inputs were never clean enough to judge the system on.

The output was right and the process around it failed

The last cause is the one most easily missed, because the AI's answer checks out. The customer still complained because the quote went out before anyone reviewed it, or the charge arrived two weeks after the job when nobody remembered the details, or the summary was correct and went to someone who couldn't act on it. Counting these as AI errors makes the AI look worse than it is and leaves the real fix (a review step, a deadline, a routing rule) unbuilt.

How Success Gets Defined Changes the Verdict

A well-documented public example of how much the definition of "correct" matters comes from a hospital. The Epic Sepsis Model is a tool built into Epic's electronic health record that scores hospital patients every 15 minutes for their risk of sepsis, and it's in use at hundreds of US hospitals. Researchers at the University of Michigan tested it on 38,455 hospitalizations. On the measure they used (how reliably the model ranks patients who develop sepsis above those who don't, where 0.5 is a coin flip and 1.0 is perfect), it scored 0.63, against 0.76 to 0.83 in Epic's internal documentation. It missed 67% of the patients who developed sepsis while generating alerts on 18% of all patients (Wong et al., "External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients", JAMA Internal Medicine, 2021).

Two details from the same paper show why tracing errors is harder than it looks. The researchers had to choose a definition of sepsis and used a composite of two, noting that "sepsis definitions are still debated." When they also counted predictions made up to three hours after sepsis began, the model's score rose to 0.80. They pointed to an earlier study suggesting the alerts often came after clinicians could already see the sepsis. Same model and same records, with a score of 0.63 or 0.80 depending on whether predictions made after sepsis began were counted.

In a 20-person office the same question looks like this: is a quote "wrong" when it's 5% off, or 15%? Is an invoice-coding suggestion wrong when it picks a reasonable account your controller wouldn't have picked? Until those questions have written answers, nobody can say whether the AI's error rate is data, definition, or the AI itself.

How Do You Trace Failed Outputs Back to Their Cause?

You trace failures by collecting the outputs that actually went wrong, rebuilding what the AI saw for each one, and sorting them with the four questions until the pattern is obvious. For most small and mid-sized businesses this takes two or three people a few working sessions, and it needs no data team. Google's published Rules of Machine Learning put the principle bluntly: when people are frustrated with a system, they "should do whatever it takes to turn their gripes into solid numbers" (Google, "Rules of Machine Learning," Rule #27).

1. Start from real failures. Pull 30 to 60 outputs from the last month or two that went wrong in a way someone noticed: customer complaints, credits, reversed charges, drafts your staff rewrote, escalations. This is different from a records check before a build, which samples records at random. Here you start from the damage.
2. Rebuild what the AI saw. For each case, gather the inputs as they were when the AI ran (the record, the document, the reference file version) and the output it produced. If you can't reconstruct the inputs, note that, since it's a finding in its own right: nobody can audit the system.
3. Check the inputs against the source. Compare each input with the source document or the current reference file. Missing, wrong, or stale inputs make it a data cause, and the case stops there. Note which field, and where that field gets created.
4. Get the right answer from two people separately. For the cases with clean inputs, ask two experienced staff what the correct output was, without showing each other their answers. Where they differ, mark the case as a definition cause and stop there.
5. Compare the AI's answer with the agreed one. If inputs were good and the answer was agreed, and the AI still missed, it's an AI or setup cause. Describe the miss in one line, in your words, so the vendor can reproduce it.
6. Mark the rest as workflow. Anything where the AI's answer was right but the outcome still went wrong is a workflow cause. Note what broke: no review, wrong timing, wrong recipient.
7. Count, and look for clusters. Tally the four causes, then look within each one for repeats. Ten data errors from the same field, or eight definition disputes about the same kind of case, tell you more than the totals do.

A note on step 1: the sample is biased toward the failures people noticed, which is what you want for a diagnosis, because those are the ones costing money. It won't tell you the overall error rate. If you need that, take a separate random sample of recent outputs and check them the same way.

Citation-ready summary: To tell whether an AI project is a data quality failure, collect 30 to 60 outputs that went wrong, rebuild the inputs the AI used, and ask in order: were the inputs complete, correct, and current; would two experienced staff agree on the answer; did the AI produce that answer; and did the process after it work? The tally shows how much of the failure is data, definition, the AI, or the workflow.

What Each Cause Takes to Fix

Each cause has a different fix, owner, and timeline, and only one of them is a data fix. The tally tells you which budget to spend, and the next tally tells you whether the spending worked.

A cleanup of historical records helps only when the AI learns from or looks up those specific records, and only for the fields driving the errors you counted; our data readiness guide covers how to check those fields before a build.

When a Vendor Says Your Data Is the Problem

When a vendor or consultant tells you the project is struggling because of your data, they may well be right, and the way to find out is to ask for the evidence behind it. A diagnosis made from real errors comes with specifics: which outputs went wrong, which records they came from, and which field in each record was the problem. A diagnosis made from a general impression of your systems usually doesn't.

Five questions settle it quickly:

• "Show me ten of the errors and the record each one came from." Anyone who has done the trace can do this in an afternoon.
• "Which fields, and how many errors does each account for?" A real data problem clusters in a few fields, and someone who has found it can name them.
• "How many of the errors happened on records that were correct?" If the answer is "we haven't checked," the AI's own errors are still mixed in with the data errors.
• "Did you give our people the same cases to answer?" Without that, nobody knows which errors are disagreements.
• "If we fix those fields, what error rate do you expect, and by when?" A target gives both sides a way to check the fix in the next tally.

This also belongs in the contract before the build starts. Agreeing on how errors will be counted and categorized, who grades the right answer, and which inputs the vendor may assume are correct turns a future argument about blame into a shared count. Our guide to which AI consulting company to choose covers what else to ask before you sign. If the project is already deep in trouble, our page on how to recover a failing AI project covers the decision to recover, narrow, restart, or stop.

An 80-Person Equipment Rental Company's Damage Charges

Consider an 80-person equipment rental company with three branches renting aerial lifts, skid steers, and compact excavators to contractors (an illustrative composite). About 900 units come back each month. A yard technician inspects each return, fills in a checklist, and takes photos, and a counter manager decides whether to bill the customer for damage, using a rate card of standard repair charges.

The company brought in an AI tool to draft those damage charges from the return checklist, the photos, the checkout record, and the rate card, with the counter manager approving each one. Four months in, disputed charges had climbed, and two large contractor accounts had complained to the owner. The vendor's view was that the inspection data was too inconsistent for the tool to work well, and it proposed a three-month project to re-label two years of past inspection records.

Before approving that, the operations manager pulled the 60 charges from the previous two months that had been disputed or reversed. With the yard lead and one counter manager, she rebuilt what the tool had seen for each and sorted them with the four questions.

Data accounted for 26 of the 60 disputes, and none of it sat in the past inspection records the vendor wanted to re-label. In 19 cases the unit had gone out without a checkout photo, so the company couldn't show the damage was new, and the charge collapsed as soon as the customer disputed it. In another 7, the tool had compared the return against the wrong machine. When the yard swapped a unit at the last minute, the checkout record stayed attached to the original unit number, so the returned lift was judged against another lift's condition. Both problems started at checkout, in records created that month.

The AI's own errors came next, at 17. Ten were misreads on clean inputs (mud read as a dent, and the same dent charged twice because it appeared in two photos). Another 16 disputes were about scuffs, worn tires, and seat tears that customers called normal wear, and the operations manager asked two counter managers to rule on each of them separately. On 7, both managers called it wear, so the tool had charged for something the company itself wouldn't bill, which made those AI errors too. On the other 9 the managers disagreed with each other, which is the definition problem: the tool had learned from two years of decisions that followed no consistent rule. The last 8 charges were correct and were disputed anyway, because they reached the customer more than a week after the return, without photos, after the contractor's site supervisor had already signed off on the invoice.

The fixes took about six weeks. Checkout photos became a required step in the yard app, and a unit swap now reassigns the checkout record before the unit can leave the yard. The counter managers wrote a one-page wear-and-damage rule with example photos and re-graded the roughly 300 wear-or-damage decisions from the previous six months under it, about two days' work in place of the vendor's two-year re-labeling. The vendor retrained the tool on those re-graded cases, removed duplicate photos, and routed low-confidence charges to a counter manager instead of drafting them. Charges now go out within two business days with the photos attached, and every disputed charge gets one of the four cause codes at the counter, so the tally keeps running on its own. Two months later, disputes were running at about a third of their earlier level, and the cause codes showed which bucket the remaining ones came from.

Where Avolis Fits

Every engagement starts with our diagnostic, and for a workflow where AI is already producing wrong answers, the errors themselves are part of what gets measured. We go through the workflow with the people who run it, count what moves through it and how long each step takes, find the records it depends on, and look at the outputs that went wrong and where each one started. That way the decision about what to fix next rests on counts rather than on whoever explains the problem most confidently, and you keep the ranked findings whether or not we build anything next.

When we build, the error categories, the written definition of a correct answer, and the review step go in from the first day, each with an owner in your business. Because we keep working alongside your team after launch, a rise in errors arrives with a count of where it's coming from and someone who knows the system well enough to act on it. Our guide to AI readiness assessment services explains what a good diagnostic should produce.

Frequently Asked Questions

What is an AI project data quality failure?

An AI project data quality failure is one where the AI's wrong outputs trace back to its inputs: records that were missing, wrong, or out of date when the AI used them. The test is whether the AI would have been right with correct, complete, current inputs. If it would still have been wrong, the cause lies elsewhere.

How do I know if bad data is causing my AI project to fail?

Collect 30 to 60 outputs that went wrong, rebuild the inputs the AI used for each, and ask in order whether the inputs were complete and correct, whether two experienced staff agree on the right answer, whether the AI produced it, and whether the process after it worked. The share that fails the first question is your data problem.

What else causes AI errors besides bad data?

Three other causes are common: your own staff disagreeing about the right answer, the AI misreading or mishandling inputs that were fine, and the process around the AI failing even when its answer was right. In a noise audit at an insurer, underwriters pricing identical policies differed by a median 55%, so disagreement alone can make a sound AI look wrong.

How many failed outputs do I need to review?

Thirty to 60 is usually enough to show the pattern, as long as they're outputs that went wrong in a way someone noticed: complaints, credits, reversed charges, drafts your staff rewrote. That sample shows where the costly errors come from, not the overall error rate. For an error rate, check a separate random sample of recent outputs the same way.

Who is responsible when an AI project fails because of data?

It depends on the cause. Data errors belong to whoever owns the step that creates the record, definition disputes to the manager who owns the outcome, AI errors to the vendor or builder, and workflow failures to the team lead. Agreeing on how errors will be counted and assigned before the build starts avoids most arguments about blame later.

Continue Learning

An AI project data quality failure is real and common, and bad data still gets blamed for more than its share. A count of where your errors come from turns that argument into a plan, and it usually shows fixes that are smaller and faster than a cleanup.

If your AI project is producing wrong answers, pull the last 30 complaints or corrections this week and sort them with the four questions before anyone approves a cleanup.

Why projects fail, in detail:

• Why do AI projects fail? Common reasons and statistics
• AI project failure statistics
• Why AI projects fail due to data quality
• How to recover a failing AI project

Checking your data:

• Data readiness assessment for AI

Starting the right way:

• AI readiness assessment services
• Which AI consulting company should I choose?

---

Sources

• BARC, "Global Study Maps the Reality of AI Delivery in 2025," 3 December 2025, retrieved 2026-10-26: https://barc.com/news/ai-leading-edge-study/
• Shawn Rogers, BARC, "The AI maturity divide: leaders vs. laggards," 5 January 2026, retrieved 2026-10-26: https://barc.com/ai-maturity-divide-leaders-vs-laggards/
• Theodore Kinni, "How noisy is your company?," strategy+business, Issue 104, Autumn 2021, reviewing Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein, Noise: A Flaw in Human Judgment (Little, Brown, 2021), retrieved 2026-10-26: https://strategy-business.com/article/How-noisy-is-your-company
• Shreya Shankar, J.D. Zamfirescu-Pereira, Björn Hartmann, Aditya G. Parameswaran, and Ian Arawjo, "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences," UIST 2024, arXiv:2404.12272 (submitted 18 April 2024), retrieved 2026-10-26: https://arxiv.org/abs/2404.12272
• Andrew Wong et al., "External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients," JAMA Internal Medicine 181(8), 2021, doi:10.1001/jamainternmed.2021.2626, retrieved 2026-10-26: https://pmc.ncbi.nlm.nih.gov/articles/PMC8218233/
• Martin Zinkevich, Google, "Rules of Machine Learning: Best Practices for ML Engineering," Rule #27, retrieved 2026-10-26: https://developers.google.com/machine-learning/guides/rules-of-ml

On the BARC figures. The 19% and 44% shares are two different measures: 19% called data quality a major concern in BARC's 2024 research, and 44% named it as an obstacle in the 2025 Lessons from the Leading Edge study. The 2025 chart answers "What obstacles slowed/stopped your organization from delivering on your AI projects?" with n=408, and the study overall drew on 421 organizations worldwide. Respondents named the obstacle; the survey didn't trace errors to causes.

On the noise audit. The 55% and 43% figures are median differences between two professionals judging the same case at one insurance company. The 10% figure is the median answer from 828 senior executives across industries asked how much variation to expect in expert judgments, as reported in the strategy+business review of the authors' 2021 book.

On the Epic Sepsis Model. The 0.63 figure is the hospitalization-level area under the receiver operating characteristic curve at Michigan Medicine, 2018–2019. The 0.76 to 0.83 range is what the authors report from Epic's internal documentation. The 0.80 figure comes from the authors' sensitivity analysis that included predictions made up to three hours after sepsis onset.

On the equipment rental company. It's an illustrative composite, not a client. Its figures (80 people, three branches, about 900 returns a month, four months, 60 disputed charges and their split by cause, 9 of 16 disagreements, about 300 re-graded decisions, six weeks, two business days, and a third of the earlier dispute level) are representative examples, not measurements or benchmarks.

On the four causes. The four-cause sort, the question order, and the fix table are Avolis's own synthesis, not a published standard.

---

About Avolis Research Group

Avolis Research Group is Avolis's in-house research practice, focused on how operations-heavy small and mid-sized businesses actually adopt AI. It synthesizes primary economic research, government survey data, and results from real implementations into practical, vendor-neutral guidance.

More about Avolis and how we work · Get in touch