Three experiments to cut our AI video pipeline's cost: one passed, one passed with conditions, one failed
Table of Contents
Our AI Shorts pipeline uses Claude Opus for topic selection, scripting and review; voice, rendering, image generation and uploading all run locally. On October 10 I asked: how much of this pipeline can move to local tools or cheaper models without losing quality?
Of three experiments, one passed, one passed with conditions, and one failed. The one that failed is the one everybody thinks of first: switch to a cheaper model.
First, measure where the money goes
Cut without measuring and you cut the wrong thing. We classified every turn in the worker transcripts by the tool command it ran and priced it at API list price (Opus 5.5: input $4, cache write $5, cache read $0.20, output $20 per million tokens). What we actually pay is a monthly subscription; dollars here are only for comparison.
| Used for | Share of Claude usage |
|---|---|
| Writing, viewing and fixing the visual spec | 36% to 38% |
| Handling local text-to-speech mispronunciations | 19% to 24% |
| Writing YouTube descriptions | 5% to 14% |
| Actually writing the narration | 3% to 6% |
Narration was 3% to 6%. A cheaper writer can at most save that slice. Local voice is cheap, but the rework triggered by its mispronunciations eats about a fifth of the Claude spend.
Along the way we also tried a local model: qwen3.5:9b on CPU only, writing descriptions for 7 published videos, compared against the Opus final versions. Wording was clean, 0 errors in 57 sentences (no Simplified characters, no mainland or Cantonese terms, no banned words). But 4 of 7 would have been rejected: two labeled real data as "illustrative," one overclaimed "confirmed by official data," and one put an internal source code into a public description. Each also took 2 minutes. Good at wording, bad at facts. Not switching.
E1: record the voice first, fix every line at once (passed)
The old order: finish the script, design the visuals, render, listen, then fix mispronunciations one at a time, each fix meaning edit, re-record, re-render, look again. In week two, 22 of the 66 review fixes across 7 videos were about the voice.
The new order: as soon as the narration is written, record it and transcribe it with faster-whisper. Every mispronounced line goes to the model in one batch, which returns three phrasings per line; we record all three locally, pick the one with the fewest errors whose duration is no more than 1.3 times the original, and only then design the visuals.
Results for 7 videos, 49 lines:
- 5 lines needed replacing after the first recording, 2 of them with syllable errors. Trying three phrasings took 5.6 GPU minutes; afterwards all 49 lines had 0 syllable errors.
- Voice-related Claude cost US$0.12 per video, or US$0.17 counting the pre-check and waiting on recordings. Week two was US$0.76.
- Voice GPU time for 7 videos was 21.0 minutes, versus 34.0 in week two.
We added tone to the replacement threshold. Checking only syllables lets through a hook where "five hundred million people" (五億人) is heard as "not a single person" (無一人), which flips the meaning.
This run had two limits. This run stopped at the voice: no rendering, no review, so part of the saving comes from having no back half, not purely from the new order. And we had written a pre-check word list for characters likely to be misread; it caught 0 of the 5 lines that actually needed replacing, and all 5 of its "hook too long" warnings were false alarms. What worked was recording first and listening. The list is only a reminder.
E1 is now in the Shorts autopilot. Later we also made headless workers start in a clean folder without loading the whole workspace's settings, and the same rephrasing task dropped from US$0.31 to US$0.10 per call.
E2: lower the reviewer's reasoning effort (medium passed with conditions, low failed)
claude -p accepts --effort. Second-round review is mostly "confirm last round's problems were fixed," which doesn't look like it needs full reasoning.
How we tested: copy 6 already-approved videos and seed 16 known errors into their descriptions: 4 removed qualifiers, 6 numbers changed by one digit, 6 overclaims the source doesn't support. The answer key sat where the reviewer couldn't read it. The same review prompt ran at default, medium and low: 18 reviews, 20 minutes, CPU only.
| effort | Caught | Flawed videos passed | Five-score average | API equivalent per review |
|---|---|---|---|---|
| default | 15/16 | 0/6 | 6.20 | US$0.47 |
| medium | 15/16 | 0/6 | 6.47 | US$0.34 (28% cheaper) |
| low | 15/16 | 0/6 | 6.80 | US$0.27 (43% cheaper) |
All three missed the same one: we removed "(illustrative)" but the sentence still said "assume," so the qualifier hadn't really disappeared. A weak seed.
On seeded errors alone, low looks like the best deal. But seeding only measures "a description was altered"; it can't measure a reviewer's ability to go looking for problems. Reading the reviews line by line: default and medium also caught mispronounced characters, a spec that dropped the qualifier "more than," a time-sensitive claim not rechecked within 24 hours, and low contrast in the layout. Low mostly didn't (one video went from 5 issues to 2, another from 5 to 3), and it scored 0.6 points more leniently.
Medium had its own wrinkle: total tokens went up 23%, all of it cheap cache reads, so in dollars it still saved. Our threshold had said "25% fewer tokens," and by that letter medium failed. We settled on dollars as the measure. Medium is now used from the second review round on and for redos; first-round review and scripting stay at default; low is used for no review at all.
E3: Sonnet drafts, Opus reviews (failed)
This is the obvious one: for the same token mix, Sonnet costs about 0.6 times what Opus does. We took 7 videos on insurance and tax, used the same topic data and the same scripting prompt, changed only --model sonnet, renamed both versions to neutral names with a random x/y assignment, and had the same Opus reviewer judge them blind.
| Opus | Sonnet | |
|---|---|---|
| Drafting, API equivalent per video | US$1.90 | US$1.24 (0.66x) |
| Blind five-score average | 6.80 | 6.66 |
| Fixes per video | 5.7 | 7.4 |
| Accuracy below 8 | 4/7 videos | 4/7 videos |
| Factual errors | 1 | 4 |
An average gap of 0.14 looks usable, and the count of videos below 8 on accuracy was the same on both sides (a bar even Opus didn't clear on this run). The difference was the kind of error. Sonnet's clustered in facts: six conditions written as four, an approval described as sufficient on its own with an added sentence the source doesn't contain, and two scenes in the same video contradicting each other about "the same day." In insurance and tax content, a factual error means it can't publish.
To be fair we also pulled the scores Opus's own first drafts got at the time: average 6.86, 8.3 fixes. So a Sonnet first draft is roughly an Opus first draft, with more factual errors. Also, two Opus final versions that had received SHIP in production both got FIX on this re-review, one dropping from 7.6 to 6.6. The reviewer has noise of its own; we don't read anything into per-video differences under 0.5.
We also tried a variant: Sonnet as the main model with Opus as advisor (--advisor opus). Both videos scored higher than pure Opus, with accuracy 8 and 9. But the advisor sent the entire conversation uncached each time, 160k to 320k advisor input tokens for a single video, which put the cost at 1.22 to 1.60 times Opus writing alone. Best quality, no savings.
The mistake we made
The entire first round of E3 was thrown out. All 9 Sonnet drafts, while being written, went and read the Opus version's folder for the same topic, review fixes included. That is copying the answer key. One of them also wrote 8 files into a production folder. We reran in a sandbox containing only the rules, the engine and templates, and used a script to scan the transcripts and confirm 0 reads of Opus content. The voided round cost about US$16 equivalent; all of E3 came to about US$47.
In a blind test, the agent under test will go looking for the answer. Isolation has to be done with the file system, not with a line in the prompt saying "don't look."
Putting the three together
After paying for the extra fixes, Sonnet would still net about US$0.50 per video, but what it adds is factual errors, and insurance and tax content can't trade facts for money. Lowering effort is workable, as long as you test it against real, unseeded problems and not only the seeded score.
The biggest saving came from changing the order: move the step that causes the most rework earlier, check it with cheap local tools, and catch the error when it is cheapest to fix.
Same lesson as the channel autopilot ledger: redos cost more than unit price.
UltraProbe follows the same idea: check whether a system prompt has its defenses written, with a static check of 25 defense vectors that calls no model and returns in milliseconds; then use red-team tools to see whether the model actually holds. We open-sourced the prompt defense checks as the npm package prompt-defense-audit, which you can drop into your own CI.
FAQ
Where does the Claude spend in an AI video pipeline go?
We classified the worker's usage turn by turn: writing, viewing and fixing the visual spec took 36% to 38%; handling mispronunciations from our local text-to-speech took 19% to 24%; writing YouTube descriptions took 5% to 14%; writing the narration itself took only 3% to 6%. The first thing to cut was the rework caused by mispronunciations, not the model.
Does setting Claude's effort to medium or low hurt review quality?
We seeded 16 known errors across 6 videos and ran the same review prompt at default, medium and low effort. All three caught 15 and none let a flawed video through. Medium was 28% cheaper per review and still listed most of the real, unseeded problems; low was 43% cheaper but often missed those real problems (mispronounced characters, a dropped qualifier, a time-sensitive claim not rechecked) and scored more leniently. We use medium for second-round review and redos, and low for no review at all.
Can you save money by having Sonnet draft and Opus review?
Drafting was 34% cheaper and the blind-review average was only 0.14 lower, but Sonnet needed 1.7 more fixes per video, and factual errors were 4 for Sonnet versus 1 for Opus. The topic was insurance and tax, where a factual error means it can't publish, so we did not switch. A variant with Sonnet as the main model and Opus as advisor scored highest, but the advisor resent the whole conversation uncached each time, so it cost 1.22 to 1.60 times what Opus alone did.