"So we've been talking about AI..." - the series, chapter 5 - Costs and Efficiency (the last one)

"The only values that exist are subjective values" - probably Friedrich Nietzsche (until he got hit with an unexpected cloud bill)

AI is expensive. Not only for training but inference, data traffic, running models and all the scarcity around that due to GPU dependencies bump its unit cost to the moon. This tend to change and commoditize but until then there are budgets being blown right and left.

Depending on the task, local hardware and local models with consumer grade GPUs and CPUs can be set as an advantage, a stage before using a more expensive vendor. Two of the forefronts of this are Ollama and BitNet.

One of the cases I like to discuss is video generation from PDF for training purposes, which can easily exhaust hundreds of thousands of dollars and can easily start on a local beefy machine and finished on Gemini, OpenAI or Claude. Same for audio transcription. The price of a good gaming machine and the evolution of models as LLAMA, DeepSeek and derived models makes for a great setup for batch pre-processing.

I wrote about my planning toolbox before. AI has the potential to be one of the biggest operational expenses before showing results on operational and comercial efficiency. I had two partners companies that together spent more than 1MM dollars by enabling LLM APIs without much form and not integrating it on their cloud economics discipline. Worst, it is hard to measure progress when you are looking for a problem which the answer should be “use AI”.

I think that efficiency gains by automation beyond processes review, waste reduction, shorter customer waiting time, better management of company risk and collaterals are pretty easy to spot.

Workforce reduction is not something that I think is productive to communicate and focus without clear goals. Since way before AI executives and middle managers have to perform their duty of team and budget management. Thinking that AI will change that overnight is not aligned with my beliefs and frankly I think it is a poor management decision for team morale to shout that out loud without actionable items.

Important metrics: Price per token, long term contracts, usage limits and quotas, impact of automation in face of user and service groweth versus vendor cost ratio.

I have released a new book: “Go for Gophers”. It is a progressive, hands on book for teams and engineers that are adopting Go and look for an idiomatic path forward. Quick, practical and loaded with useful examples. Check it out.
If you are a CTO, Tech Lead or Product Leader check The CTO Field Guide. It is a book to help product engineering leaders of all levels. If you need personalized help, check the mentoring program based on the book at https://ctofieldguide.com/mentoring.html.