AI Analyst

Commercial teams get asked about campaign performance by clients, by managers, just before a call: why did cost per click jump last week? Is conversion rate falling for this advertiser? Getting a reliable answer usually means waiting for an analyst or building a query, and a quick answer is easy to get subtly wrong.

AI Analyst lets you ask that question in plain language, or ask for a briefing on one advertiser. The answer comes back as:

• a short summary;
• findings, each with the exact figures behind it and where they come from;
• possible explanations, labelled as hypotheses, not causes;
• checks to run next;
• what the data cannot tell you.

If a question is ambiguous — “last week” with no dates — it asks first instead of guessing. It cannot change a campaign, and it is built not to claim why something happened; section 04 shows where it still overreaches.

The goal is a tool a commercial team can use without an analyst on hand, and trust because every number can be checked. That is the target, not the current state: it runs on synthetic data only, and in the 27 September retest it failed all five cases about the planted issues — the part that matters most. Re-run later that day, the two questions about those issues passed; the advertiser briefings did not, and one of ten briefing runs passed source review. In the full run on 28 September all five passed, including the three briefings — one run per case, so an observed result, not a reliability rate. No time saved or business result has been measured.

The hard part is not the arithmetic but stating it correctly. Answers go wrong in ways that read well: a rate compared as if it were a volume, two periods of different length compared as totals, a cause asserted from a correlation.

The simpler alternative is a fixed dashboard or a hand-written query. The bet here is that a model can turn a plain question into the right scoped calculation and explain the result, as long as it never does the arithmetic itself and every number it states points back to computed evidence. That is a hypothesis: no time saved, cost reduced or business outcome has been measured.

The data is synthetic: 7,290 rows for three fictional advertisers, 23 April to 21 July 2026, with a few issues planted deliberately by the generator. Currency is unspecified. Dates inside the data are not the dates the model ran.

The decisions that matter:

• Python does the numbers. Two read-only tools filter the data and compute metrics. Rates are computed from raw totals before rounding; periods of different length are compared as daily averages for volumes and as ratios of sums for rates.
• The model chooses, cites and explains. It picks a tool and its arguments, then returns a fixed structure: summary, findings with evidence paths, hypotheses, suggested checks and limitations.
• Every model request is approved. No request is sent without owner approval, there is no automatic retry, and a transient provider error allows one freshly approved retry. Turns, tool calls and spend per request are capped.
• Replies are validated, not trusted. Python checks the structure and resolves each cited value. That catches format and citation errors; it does not prove the analysis is right, so each evaluated answer also gets a source review.
• It cannot act. No campaign changes, no arbitrary file reads, no real client data, and no claims about actual causes.

The comparison rule, as written in the tool code:

def period_comparison(current, reference, current_days, reference_days):
    """Compare volume per day and weighted rates, never unequal-duration totals."""
    a = {k: v / current_days if k in COUNTS else v for k, v in current.items()}
    b = {k: v / reference_days if k in COUNTS else v for k, v in reference.items()}
    result = comparison(a, b)
    for k, value in result.items():
        value['basis'] = 'daily_average' if k in COUNTS else 'ratio_of_sums'
        value['current_days'] = int(current_days)
        value['reference_days'] = int(reference_days)
    return result

The code and tests were written under my direction by OpenAI Codex until 28 September 2026, then by Claude Code (Anthropic). This page describes the system and its evidence; it does not claim I wrote it by hand.

Each line below is a different kind of check. They are not added together.

By category in the 27 September retest: plain questions 5 / 5, ambiguous questions 3 / 3, invalid requests 3 / 3, multi-step questions 2 passed and 2 not run — and anomaly cases (briefings and questions about the planted issues) 0 / 5. In the re-runs that followed, both anomaly questions and both multi-step questions passed. Across those four re-runs, one of ten briefing runs passed.

The four 27 September re-run rows each ran a different version and cover only the cases listed, so they are not a new score for the whole suite. The same goes for the 3 October targeted re-run.

The full run on 28 September put all 20 cases through one frozen version. By category: anomaly cases 5 / 5, plain questions 5 / 5, ambiguous questions 2 / 3, invalid requests 3 / 3, multi-step questions 3 / 4. All three briefings passed and named the three issues the generator planted — three issues in one synthetic dataset, not a detection rate. I accepted that result and the version went live the same day.

Then I tried a newer model, claude-sonnet-5-5, with the same settings plus fixes for the two 28 September failures. Its first full run passed both, and failed three multi-step questions the live version had passed. Two controller fixes followed and the three passed when re-run. Before the next full run I set the rule for deploying it: at least 18 / 20 and no new runtime failure. It scored 18 / 20, the same as the live version, but both failures were new runtime failures, so it was not deployed.

Same score, different failure. The live version's two failures still published an answer, and one of those answers made a false claim. The candidate's two failures stopped and published nothing. Which is worse depends on what a wrong answer costs compared with no answer. Each version ran each case once, so 18 against 18 is within the variation between runs.

“Source review” means every figure and claim in the reply was checked against the CSV. I ran that review with an AI coding assistant — OpenAI Codex for the runs up to 27 September, Claude Code from the 28 September full run; it is not an independent human rating.

Cost is estimated from the provider's token counts at recorded standard rates, not taken from an invoice: USD 0.667876 for 36 requests on 26 September, USD 0.6703 for 31 requests on 27 September. Each batch ran under a separate cap of 60 requests and USD 0.75. The four re-runs that day cost USD 0.373406 for 13 requests, 0.221412 for 5, 0.136216 for 3 and 0.136968 for 3, each under its own cap. The 28 September full run cost USD 0.785616 for 34 requests (cap 40 requests, USD 8.00); the hosted question USD 0.031488 for 2. The claude-sonnet-5-5 runs cost USD 0.740886 for 37 requests, 0.174616 for 9 and 0.751742 for 37 (caps 40 / USD 3.00, 12 / USD 0.50 and 40 / USD 3.00).

This covers every failed or unrun case in the dated live runs: 12 failures and 2 cases not run in the 26 September run and the 27 September retest, then 8 failures and 1 case cut off in the four re-runs later on 27 September; 2 failures in the 28 September full run; and 3, then 2, failures in the two full runs of the claude-sonnet-5-5 candidate on 3 and 4 October, which was not deployed. A case can fail for more than one reason, so counts overlap. The 54 earlier development attempts are kept privately and not itemised here.

Question, exactly as asked: “Compare cost per click during June 15-18 with the other days of June 12-21, 2026, for Northwind Retail - Retargeting Core.”

Model claude-sonnet-5, run started 27 Sep 2026, 07:57 UTC; tools query_metrics and get_briefing_evidence, both read-only; 90 synthetic rows in scope.

The generator multiplies this campaign's spend by 3.0 on 15–18 June. Measured on the data, cost per click is 4.04 in those four days against 1.49 on the other six: a ratio of 2.713, not 3.0, because clicks vary too.

The model's third reply, uncut and unedited:

{"status":"answer","summary":"For Northwind Retail - Retargeting Core, cost per click during the focus window of June 15-18, 2026 was substantially higher than during the remaining days of the June 12-21, 2026 window (June 12-14 and June 19-21 combined). Click and impression daily averages were also higher in the focus window, while spend rose by a larger margin than clicks, which is consistent with the elevated CPC.","findings":[{"metric":"cpc","what":"CPC for the June 15-18 focus days was much higher than CPC on the other days of the June 12-21 window","severity":"high","confidence":"medium","evidence":[{"evidence_id":"e1","path":"comparison.cpc.change_pct","label":"CPC change_pct, focus vs reference"},{"evidence_id":"e1","path":"focus.totals.cpc","label":"CPC total, focus window (Jun 15-18)"},{"evidence_id":"e1","path":"reference.totals.cpc","label":"CPC total, reference days (Jun 12-14 & 19-21)"}]}],"hypotheses":["A bid or budget change during June 15-18 may have raised the average cost per click","A shift in auction competition or targeting mix during June 15-18 could have increased click costs"],"actions":[{"action":"Review campaign bid strategy and budget pacing settings for Retargeting Core specifically for the June 15-18 dates","why":"To check whether a bid or budget adjustment coincides with the CPC increase"},{"action":"Break down CPC by device and country for the same date ranges","why":"To see whether the increase is concentrated in a specific segment rather than spread evenly"}],"limitations":["Currency for spend and cpc figures is unknown, so only relative comparisons are meaningful","This is a single specific-window comparison; it does not establish a longer-term trend or root cause","Reference days (June 12-14 and June 19-21) form a non-contiguous six-day set as defined by the query, not a separate matched period"]}

What happened. Earlier in the same run the model asked for the wrong advertiser; the scope check refused it and the model corrected itself. This reply and the next were then rejected because my validator did not accept a valid label for the six reference days — a false alarm in the checker, not a model error. After the checker was repaired, an offline recheck still rejects this reply, now for a real reason: the summary mentions click, impression and spend movements that no finding cites. The next saved reply passes the repaired checker offline. The live run stays recorded as failed; the repaired version has not been re-run against the model. the repaired version was re-run against the model later the same day, and that answer passed source review: cost per click 171.3% higher on 15–18 June than on the other six days, a ratio of 2.713, with the right advertiser and no invented cause. See Corrections.

Question, exactly as asked: “What was mobile CTR for BlueOak Travel during May 1-14, 2026?”

Model claude-sonnet-5-5, adaptive thinking, medium effort; run started 4 Oct 2026, 06:02 UTC, in the candidate's final full run. One read-only query_metrics call for mobile CTR over those 14 days; 126 synthetic rows in scope. Computed from the CSV, the answer is 0.190825%.

The model's first draft cited an evidence path that does not exist, a real error, and was rejected. Its repair, uncut and unedited:

{"status":"answer","summary":"Mobile CTR for BlueOak Travel over the 14 days from May 1 to May 14, 2026 is shown in the cited evidence. It is a weighted ratio of clicks to impressions across all campaigns and countries.","findings":[{"metric":"ctr","what":"Mobile CTR for BlueOak Travel across all its campaigns and countries during the requested 14-day window.","severity":"low","confidence":"high","evidence":[{"evidence_id":"e1","path":"totals.ctr","label":"Mobile CTR, May 1-14, 2026 (percent)"}]}],"hypotheses":[],"actions":[],"limitations":["No comparison period was requested, so this does not say whether the value is high, low or changing.","The data is synthetic and the result covers only the MOBILE device."]}

What happened. The repair cites the right value. The checker rejected it anyway: the summary names clicks and impressions, which no finding cites. It names them only to define CTR. One repair is allowed, so nothing was published and the case counts as failed. The same question passed with the deployed model on 28 September and with this candidate on 3 October. This is the cost of strict output rules: they stop unsupported claims, and sometimes correct answers too.

• Synthetic data only. Nothing here shows how it would behave on real campaigns or what it would be worth to anyone.
• It observes; it cannot establish why something changed or exactly when it started.
• The hosted version has live model requests switched off. There is no production-readiness claim.
• Each full run is one run per case: 18 / 20 is an observed result for that version, not a reliability rate.
• Successful answers selected across earlier iterations are not a current pass rate. Re-running only the failed cases would not make a fresh 20 / 20 either.

Still open: the deployed version has no fix for its two 28 September failures; the fixes have run only with the claude-sonnet-5-5 candidate.

• 2026-09-28 — This page said the repaired version had not been re-run against the model. That was already wrong when the revised page went live on 27 September: a targeted live re-run had started ten minutes earlier. The same claim appeared in the current state, in the failure table (“No live re-run yet”; “the contribution case was not re-run”) and in the next step, which asked for the re-run that had already begun. Those places now reflect the re-runs; the original sentence stays struck in section 05.

• 2026-10-04 — The hosted version moved to its own address, analyst.pietrolauria.com; same tested version, live model requests still off; hosted checks 28 / 28.
• 2026-10-04 — 28 Sep: one full 20-case run on a frozen version, 18 / 20; that version is hosted with live model requests off (hosted checks 29 / 29). 3–4 Oct: a claude-sonnet-5-5 candidate scored 17 / 20, then 18 / 20 after fixes; not deployed under the rule set before its final run. Code written by Claude Code from 28 Sep.
• 2026-09-28 — Four targeted live re-runs on 27 Sep: 4 / 7, then briefings 1 / 3, 0 / 2 and 0 / 2. The four re-run questions passed; briefings still state directions the numbers contradict. Corrected the claim that no live re-run had happened.
• 2026-09-27 — Added the commercial explanation; the failure table now covers every failed or unrun case in both live runs.
• 2026-09-27 — First published, in progress: 27 Sep live retest 13 / 20 passed source review; repaired version not yet re-run live.