Technical Due Diligence: 3 Checks for AI Automation Claims

A deck reports 55.8% faster work from a controlled experiment with 95 professional developers. The task was one JavaScript HTTP server built from a template against 12 visible tests. Technical due diligence accepts the measurement and still asks whether it supports the conclusion attached to it.

Technical Due Diligence in the AI Era: The Three Checks

Technical due diligence in the AI era means testing three things before accepting an automation number: whether human labor tracks workload volume, where the automation boundary actually sits, and who validated the quality judgment. Soukaïna Abelhad, an AI tech lead who has built AI systems used by investment professionals and now leads AI evaluation in production, proposed this framework in a 2026 AI Engineer talk on diligence against automation claims.

Her starting observation is that the dangerous number is rarely invented. It is often a real measurement carried into a conclusion it cannot support. A vendor can quote a controlled study accurately and still mislead a buyer about what that study covers.

The three checks run in order. Arithmetic first because it needs no model expertise. Definition second, because a precise boundary is required before any rate means anything. Evaluation third, because a measurement is only as good as the judgment behind it.

Only claims that survive all three get translated into unit economics, where the cost per accepted autonomous task becomes the number a buyer can defend.

Check One: Does Human Labor Track Workload Volume?

The arithmetic check compares workload growth with the fully loaded human labor required to deliver it. Put output volume next to everyone whose work exists because of the AI: operators, reviewers, trainers, escalation handlers and rework. If human labor rises in proportion to workload, the deck has described assistance, not labor removal.

Abelhad cites observational telemetry from Faros AI, an engineering intelligence platform, covering more than 10,000 developers across 1,255 teams. In that data, teams with high AI adoption merged 98% more pull requests, review time rose 91%, and the average pull request became 154% larger.

Those figures do not prove that AI increased headcount. The data is observational telemetry, not a controlled labor study, and Abelhad is explicit about that limit. The warning is narrower and still useful: more work entered the system, and the review burden grew along with it.

The diligence request that follows is simple. Ask for the company's own 18-month data and put workload volume beside fully loaded labor, counting every role that exists downstream of the model.

A vendor dashboard is not the only source of this arithmetic. The buyer can reconstruct it from internal ticket volume, review queues and reviewer headcount, which is exactly why the check works without model expertise.

Check Two: AI Attributed Is Not Autonomous

The definition check asks what level of automation the claim actually covers inside a named workflow. Uber, the ride-hailing and delivery company, has said that 70% of its pull requests are AI attributed, which is an impressive figure that does not tell a buyer the full autonomous rate. Abelhad notes the period and total count were not disclosed either.

Every automation claim sits somewhere on a ladder. The model drafted the work. Humans edited it. Humans approved it. Or no human intervened at any point. Finance already has a term for the last rung: straight-through processing, defined as completion end to end without human intervention inside a boundary the claimant names.

The boundary is the hard part. Does it end when the code was drafted, when the pull request was approved, or when the change reached production? Each answer produces a different numerator and a different denominator.

A percentage without a stated denominator cannot be audited, because the buyer has no way to tell whether the work counted as automated still depended on people to finish or correct it.

Treat AI attributed work and autonomously completed work as separate claims. They require separate evidence, and only the second one supports a labor-removal argument.

Check Three: Who Validated the Automated Judge?

The evaluation check asks what the automated grader was validated against and whether domain experts agree with it. A model judging another model's output is useful evidence only inside limits that someone has measured.

Abelhad cites a Meta study reporting a 24 percentage point gap between an automated grader and the merge decision after adjustment to the human baseline. Four maintainers from three repositories reviewed 296 AI generated pull requests tied to different issues, without knowing whether a patch came from AI or a human. The automated grader was substantially more optimistic than the people responsible for accepting the work.

The same pattern appears from the build side. In one production project Abelhad worked on, 10 out of 10 outputs passed automated validation, while very few survived once domain experts reviewed them.

Evaluation maturity is therefore the first credibility signal to inspect. Ask who labeled the ground truth, whether an expert agrees with the automated verdict, whether anyone measures that agreement, what the judge was validated against, and which bad outputs escaped the check.

Rubric agreement and expert agreement are different measurements. A pipeline can report high automated grader agreement while diverging from the assessment of the people accountable for the result.

From Surviving Claims to Cost per Accepted Autonomous Task

Once a claim survives the arithmetic, definition and evaluation checks, it still has to survive the unit economics. Start with the eligible workflow value, apply actual adoption, straight-through success and accepted quality, then subtract the cost of AI, human review, escalation and rework.

The resulting metric is cost per accepted autonomous task. Failures belong in the denominator and the cost account, because a task that fails, escalates or needs rework consumed resources without producing an accepted result.

This reframing changes the diligence conversation. Instead of arguing about whether a percentage is impressive, the buyer and vendor can argue about a cost figure both sides can reconstruct from the same inputs.

The economic question also exposes optimistic modeling. A workflow that looks cheap per task at a high straight-through rate becomes expensive once escalation and rework are priced in at their observed rates.

Abelhad's card summarises the sequence: arithmetic, definition, evaluation, then unit economics, with only the surviving claims translated into money.

What the Builder.ai Case Shows About Warnings Without Gates

Warnings that are published but never converted into a gate are the failure mode this framework targets. Builder.ai, the London-headquartered company that marketed AI-assisted app development, is Abelhad's illustration, and she states plainly that this is not a claim that weak AI positioning caused its collapse.

Her account gives dates. In 2019 the Wall Street Journal publicly questioned the company's AI capability claims. In May 2025 the Financial Times reported that its 2024 revenue estimate had been revised from $220 million to roughly $55 million. On 2 June 2025 the company filed Chapter 7, the liquidation proceeding under United States bankruptcy law.

The insolvency involved revenue financing issues beyond AI claims, and Abelhad separates that from her diligence point. The lesson she draws is about process, not causation: the warning existed for years, and the process did not turn it into a gate.

The framework is the gate. Turn each published doubt into a test with a named owner and a threshold, or it stays a note in a file while the claim keeps circulating.

Running These Checks in Your Next Vendor Conversation

These three checks compress into questions you can ask without model expertise. Each one forces the claimant to name a denominator, a boundary or a validation method.

The Crazystack TypeScript stack used by Gustavo Dev Doido at the Bootcamp do Dev Doido teaches engineering teams to build observable pipelines, and that habit transfers directly to diligence. Instrument the workflow, log every human intervention, and the arithmetic check becomes a query rather than a negotiation.

Abelhad's closing line reframes the whole exercise. The job of diligence is not to ask whether the number is impressive, but to ask what must be true for the number to matter.

FAQ

• What is technical due diligence for an AI product? It is the process of testing whether a vendor's technical and automation claims hold up under inspection before an investment or purchase. For AI products, the useful tests are arithmetic on human labor, a precise automation boundary, and validation of the quality judge.

• Why is a 70% automated claim hard to audit? A percentage without a stated denominator or boundary does not say what was counted or where human work stopped. AI attributed work, human-approved work and straight-through completion are different measurements that produce different numbers from the same workflow.

• What is straight-through processing? Straight-through processing describes a task completed end to end without human intervention inside a boundary the claimant names. The boundary matters, because a rate measured up to code drafting is not comparable to a rate measured up to production deployment.

• Why can an automated grader disagree with human reviewers? An automated grader scores against a rubric that someone designed, and rubric agreement is not the same as expert agreement. In the Meta study Abelhad cites, the automated grader was more optimistic than the maintainers who had to accept the work.

• What metric should replace an automation percentage? Cost per accepted autonomous task, with failures, escalations and rework included in the cost account. It combines adoption, straight-through success, accepted quality and the human labor that remains.

• Does higher AI adoption reduce engineering headcount? The Faros data Abelhad cites shows merged pull requests rising 98% while review time rose 91%, which is observational telemetry rather than a controlled labor study. The data warns that downstream review burden can grow with output but does not establish a headcount effect.

• What went wrong at Builder.ai according to this framework? Public doubts about the AI capability were reported in 2019, and its 2024 revenue estimate was revised from $220 million to roughly $55 million in May 2025 before a Chapter 7 filing on 2 June 2025. Abelhad frames the lesson as process failure, not proof that weak AI claims caused the collapse.

• Who labeled the ground truth in an AI evaluation? That is a question the vendor should answer with names, roles and a measured agreement rate against domain experts. If nobody measures expert agreement, the evaluation reports rubric agreement and nothing more.

• Can these checks be run without machine learning expertise? Yes. The arithmetic check is pure accounting, the definition check is pure specification, and the evaluation check asks about process and validation rather than model internals.

• What must be true for an automation number to matter? The measurement has to cover the scope the claim asserts, human labor has to fall rather than track volume, the quality judge has to agree with accountable experts, and the economics have to survive after escalation and rework are priced in.

Turn a Diligence Talk Into an Article with Skala Blog

The framework above works because each check forces a claimant to state a denominator, a boundary or a validation method. That habit of asking what must be true for a number to matter applies to your own material as much as to a vendor deck.

If you have recorded talks, interviews or internal walkthroughs that carry this kind of reasoning, Skala Blog turns a YouTube video into a written article. Paste the video URL, let Skalablog transcribe it, and generate a draft you can review and publish.

Source video