Tag: cost
All the articles with the tag "cost".
Token cost as a design constraint
Caching, batching and routing are the three levers on an inference bill. They are worth roughly ten times, two times and up to a hundred times respectively, and almost everybody reaches for them in the wrong order.
When inference costs more than the engineer
An hour of engineer time buys roughly four million output tokens at top tier rates. Working out where that stops being true, and why the answer has almost nothing to do with which model you picked.