Having established in the previous piece that the input side of the bill is where the money goes, this is about the three levers you can pull on it.
They are not equally valuable and they are not equally safe, and the order people reach for them is close to backwards. Model choice has the largest headline number and the most ways to go wrong. Caching has the smallest headline and is almost free. Batching sits in between and is simply forgotten.
Table of contents
Open Table of contents
Caching, the one that is already switched off
Prompt caching stores a prefix of your request so that subsequent requests reusing that prefix read it at a fraction of the price. On current models a cache read costs roughly a tenth of the normal input rate, and on the larger models closer to a twentieth. Writing to the cache costs about one and a quarter times the normal rate.
That write cost sets the break even. One write and one read is a loss. Two reads roughly breaks even. Everything after that is close to free input, which is why anything with a stable preamble and repeated calls should be caching and most of them are not.
The mechanism is a prefix match, and that is where it goes wrong. Requests are assembled in a fixed order, tool definitions first, then the system prompt, then the messages. Any byte that changes anywhere in the prefix invalidates everything after it.
Which means the following all silently destroy your cache, and I have seen every one of them in real code:
A timestamp in the system prompt. Something as innocent as telling the model today’s date, rendered to the second, guarantees a miss on every single request.
A request identifier or trace ID placed near the front for logging convenience.
Tool definitions assembled from a dictionary or map with no deterministic ordering, so the list comes out in a different sequence between processes.
A user name, account identifier or other per request detail in the system prompt rather than in the message.
None of these produce an error. You simply pay full price forever and nothing tells you. The check is one field in the response: if the cached read count is zero across requests that should share a prefix, something in front is moving.
The design rule that follows is simple and worth building into the shape of the code. Stable content first, volatile content last. The frozen system prompt, the deterministic tool list, the long reference document, all before the last cache marker. The timestamp, the request ID and the user’s actual question after it.
Batching, the one that is forgotten
Asynchronous batch processing runs at half price. Submit a set of requests, poll until they finish, read the results.
Half the bill, for work that does not need an answer in the next few seconds. Nightly classification, backfills, bulk extraction, evaluation runs, anything populating a cache or an index, report generation. In most products a meaningful share of inference is not interactive, and almost none of it is batched, because the synchronous path was written first and nobody revisited it.
Two caveats. Results come back in any order, so key them by the identifier you supplied rather than by position. And you are trading latency for money, so it only applies where nobody is waiting.
I would look at this before touching model choice. It is a fifty percent reduction on a slice of traffic with no quality tradeoff whatsoever, which is a rare thing.
Model choice, the one with the trap
The spread runs from GPT-6 Luna at ten cents per million input tokens to Claude Fable 5.1 and GPT-6 Astra at ten dollars. A hundred fold. Routing cheap work to cheap models is obviously correct in principle, and in practice it is the lever most likely to cost you money.
Three reasons.
Caches are scoped to a model. Split your traffic across three models and you have three cache namespaces, each with a lower hit rate than the one you had before. A cascade can lose more on cache misses than it saves on rate, and the only way to know is to measure both together rather than comparing list prices.
A cheaper model that needs more turns is not cheaper. The unit that matters is cost per completed task. A model that takes four attempts at a tenth of the price has saved you sixty percent and spent four times the latency and four times the review attention, which may not be a trade you wanted.
And the comparison people skip is the capable model at lower effort. Thinking depth is adjustable within a single model, and on current models the lower effort settings on a newer model often match or beat the higher settings on an older one. One model, one cache namespace, a dial you can turn per route. That is usually a better first experiment than building a routing layer, and it takes an afternoon rather than a sprint.
Where routing genuinely earns its place is when the work is obviously different in kind. Bulk classification, extraction from a known format, summarising something short, routing a ticket. These do not need a frontier model and never did. The mistake is building a general cascade before establishing that the simple version does not work.
Context hygiene
Underneath all three levers is what actually sets the bill, which is how much you resend.
Every turn carries the whole conversation. A loop that accumulates tool output, pastes whole files, and keeps everything forever has a cost curve that grows with the square of its length, and the only question is how long it runs before somebody notices.
The mitigations are all about throwing things away deliberately. Clearing old tool results once they have served their purpose. Summarising earlier turns rather than carrying them verbatim. Passing a path and a line range instead of a whole file. Not reading a hundred thousand token document into context to answer a question about one paragraph of it.
This is ordinary engineering discipline applied to a resource people have not yet learned to see. It is the same instinct as not loading an entire table into memory to count its rows, and it tends to arrive in a team at about the same point as the first surprising invoice.
The order I would work in
Measure the ratio of input to output on real traffic. If input dominates by ten to thirty times, which it usually does, everything below follows.
Check the cache hit rate. Fix the silent invalidators. This is free and it is typically the largest single improvement available.
Move everything that nobody is waiting for onto the batch path. Another free one.
Trim what you resend. Slightly less free, because it takes thought, but no quality tradeoff if done carefully.
Then turn the effort dial down on routes that do not need the depth, and measure quality as you go.
Only then consider routing to different models, and measure the cache loss alongside the rate saving rather than assuming the list price difference is what you will get.
The first three have no downside, and most teams I have seen skip straight past them to the fourth and fifth, which is where all the ways to make things worse are kept.