"So we've been talking about AI..." - the series, chapter 1

Chapter 1 - intro and old intro and old old intro and glossary

TL;DR - Engineering and Product leaders are challenged with questions, ideas, committees, investments and new roles due to AI FOMO. Companies want in on AI, teams were already full and the actionable items ranged from using coding assistants to hiring AI executives to… optimize operations. I've organized a playbook to help you navigate it.

Introduction

I've changed this intro a lot of times and decided to have two - an intro for now and the old intro as of the first week of June 2025. I've also changed this post to a series as it became a huge stream of text. The first reason was early feedback, then there were some triggers:

The first trigger: Last month I got an email of a cloud company that I respect a lot, love their products and consider myself a heavy user asking me to answer a survey about a new offer based on monitoring and AI. Monitoring is an expected capability of any cloud provider, something as old we get for granted, but what caught me was “monitoring with AI capabilities”.

I've answered the survey out of empathy as probably a fellow product person is executing the company strategy and targeting whatever is needed to pass their performance evaluation.

By the end of the survey form there were something like a “What else do you think this product should provide” box, a classic on product discovery so I felt that it was safe to give a feedback that this product smelled like some sort of AI washing of something basic, resulting in something no one needed.

The second trigger was that I did around 10 mentoring sessions that the central subject were AI and what to do with it, the frustration that teams were not “programming more” or giving lower estimates in days. Almost all of these conversations concluded that the leadership should talk more amongst them instead of expecting that the team plus an AI would come with a strategic or productivity change alone.

The third trigger was noting that thousands of text generators and bots released each and every week that got obsoleted by an AI Provider capability - ChatGPT meeting transcription comes to mind after seeing at least 4 startups based on that down the sink.

Besides being thrown out of the market by your upstream AI provider, the credibility loss risk in front of the target audience is too high. The revenue return will be very low. In meantime of that a lot of money will be spent.

Now you can read the old intro and follow up with a Glossary and one post per category with a micro playbook help you navigate through this uncertainty.

The old intro

So as I said, I felt I survived the first wave of AI for coding promises in a good shape. In my case it was overlapped with adopting teams designs based on bigger companies and books so I had enough to dodge.

But that won't go away like this. So after a year of “What if we used AI” meetings, AI committees and VCs asking us what was the plan to be an “AI Company”, even the best finance statement ever won't keep you, a engineering or product lead, free from some sort of time sink.

So to help you and me, I ran some experiments to find out how to better connect to AI. As it happened with 90s application builders, 2000s “build any application collection” CDs, 2010s “cookie cutter” code templates and PaaS offerings, I walked out with something positive along with a lot of spaghetti code.

AI products, specially LLMs are sought after to increase delivery velocity and flexibility, so they require good quality inputs, cost management and an assessment of the impact on data privacy and security - these are not clear beyond advisory statements within usage and legalese terms.

I've divided our AI strategy in five categories of interest to help you build your plan. If you want me to expand in one or more of them just drop me a line !

Glossary

Models - a broad term but for our context are AI models, programs trained on large amount of data to generate information and execute tasks. The most know type of model we discuss today are of the LLM (Large Language Model ) category.

You will read about Generative AI too which for our purposes is just what we describe. There is plenty of material to dig deeper into types of model, specialized training, tuning and so on.

I recommend creating and account and perusing Hugging Face a community driven hub for datasets, models, integration and documentation. Each model will have a “card” which describes how it was trained, if it is derived from another model and the purpose. Look into BERT and Transformers too if you want to learn what is behind running a model

Prompt - command - or conversation - you establish with a model in natural language. If the model has a chat interface, prompt will be the text. If a model has an API, part of its call will be a prompt. It is interesting to note that most of the AI Products we consume are built around prompts, and you can find some of the most famous prompts in this github repository.

Tokens - each prompt and model answer are composed of tokens. You can think of them as each word in a prompt but that wouldn't be too accurate. Each model and vendor define a their tokens as a unit or chunk of information.

AI models use them to establish relation as AI answers can be seen from a statistical standpoint as “the best next word - or token - for that position”.

Token windows - Tokens are part of an important trait of an AI vendor: the token window. Token Window is the size, or max size of tokens that can be processed by each request and response. This is important because we will see RAG and other augmentation methods as memories that end up adding tokens to a prompt or request, which can consume precious space.

The bigger the token window the bigger the context the model can take, meaning that between prompts and responses the model can “remember” better and “more” about the session.

Agents - autonomous components that can do one or a series of actions with or without human intervention. There are frameworks and methodologies to build and manage agents. They come from the fine lineage of cronjobs and job schedulers but adapted to the model scale.

RAG (Retrieval-Augmented Generation) - used to improve the accuracy and relevance of responses from large language models (LLMs) by creating indexes and external information sources. Think of creating an index to help enrich a question based on local legislation, translating words to tokens using the model API and incorporate this data into the response. Remember the token window right ? It is directly impacted by it.

MCP (Model Context Protocol) - one of the intercommunication standards available to extend and create “tools” for models. MCP is an open standard, open-source framework introduced by Anthropic in November of 2024 with a surprising adoption rate. Other big players released their standards but for reasons MCP got a wide adoption and quickly became available from software as Github, JIRA, Figma, Blender and a host of platforms. It can even help connect two distinct models and create an ensemble.

Internally it is based on the JSON-RPC protocol and can be deployed either as a local command, standalone service or API. Imagine you want your model to run a security check with external tools, execute a web browser automation to test a site or scrap some information, examine your issues on Github and JIRA, consolidate all the outputs and produce a report or issue commands for your cloud provider or feature switch platform.

This is possible without breaking the model sandbox by enabling a MCP server with tools - new functions that the model learn to use. I recommend testing the Github MCP and my project MCP OSV which makes the OpenSource Vulnerability Database project scanning available for platforms as Cursor, enabling proper package scanning for vulnerabilities.

Before that you would have to code partial prompts and automation between them with code to glue the logic and outputs. Too complex.

I have released a new book: “Go for Gophers”. It is a progressive, hands on book for teams and engineers that are adopting Go and look for an idiomatic path forward. Quick, practical and loaded with useful examples. Check it out.
If you are a CTO, Tech Lead or Product Leader check The CTO Field Guide. It is a book to help product engineering leaders of all levels. If you need personalized help, check the mentoring program based on the book at https://ctofieldguide.com/mentoring.html.