What a technical AI audit should deliver

Decide what the review must answer

Agree on the system, users, data and scenarios in scope. A document assistant and an agent that takes actions need different tests.

Keep results separate from assumptions

Record tested cases, sources, access rules and observed errors. Explain how the sample was chosen and what remains outside it.

If behaviour under realistic load or data was not tested, make that limit visible.

Prioritise the next move

An actionable conclusion separates blockers, worthwhile improvements and decisions for the accountable team. It may recommend continuing, narrowing or stopping; a rebuild is not the automatic answer.

A report outline another person can inspect

A technical report should let someone follow the conclusion: objective and scope; data and versions used; agreed criteria; executed tests and outcomes; observed failures with a reasoned severity; limitations; remediation options; and a conditional recommendation. If production or real-user access was unavailable, state that as a limit rather than burying it in a footnote.

For an internal assistant, a sample might include correct cited answers, questions with no source and cross-role permission tests. Decorative scores are less useful than reproducible examples and the consequence of each failure for the task. The accountable organisation retains the decision to accept or reject a risk.

Start with decisions, not a catalogue of tools

List the candidate workflows, the person who owns each one and the consequence of an incorrect answer or action. Rank them against expected value, data readiness and operational risk. Record why a tempting case is deferred. A workshop that produces twenty ideas without a priority and a reason to stop has not yet reduced uncertainty.

Map the constraints before choosing an architecture

The review should identify data categories, who may access them, permitted processing locations, integrations, procurement constraints and the team that would operate a pilot. Compare a small number of feasible options, such as an external API, a hybrid flow and an owned environment, against the same task and data boundary. The report should say which constraints rule an option out and which assumptions still need a specialist to confirm. An architecture diagram alone cannot settle those questions.

Define a pilot another team could assess

An actionable pilot brief names its users, source collections, permitted actions and a small set of representative tests. It defines what success and failure mean for this task, what a person must review, and what would stop or narrow the pilot. For a document assistant, include a question with a cited answer, one without a supported answer, and one where a different role must not see the source. These are example test categories, not results from a Hesperia engagement.

Check the evidence behind the recommendation

Ask whether the report separates observed results from assumptions, versions and test data; whether failures can be reproduced; and whether the owner could repeat the decisive checks. A recommendation may be to proceed, repair a data source, run a narrower experiment or stop. Each option should name what information would change the decision. If the review did not include production-like load, legal assessment or real-user trial, the conclusion must not imply that it did.

Compare the technical audit service with the private AI readiness checklist: the first concerns an existing system; the second helps decide whether a use case is ready to test.