Working with AI Models and APIs

When you chat with an AI assistant, a lot is hidden. Behind the friendly window, your software is quietly having a very structured exchange with a model: it packages up your words, adds some settings, sends the bundle across the internet, and waits for a reply. Understanding that exchange is what lets you build tools instead of just typing into a box.

It also lets you reason about the things that matter in practice — how much a feature will cost, how fast it will feel, and whether you are locked to one supplier. None of that is visible from the chat window, but all of it becomes real the moment you build something that runs many times a day.

The request and the response

Talking to a model through an API follows one simple loop. Your program sends a request — the prompt (your instructions and any context) plus a few parameters (settings that shape the output). The provider's model reads it and returns a response: the generated text. Providers such as OpenAI and Anthropic both use this same general request–response shape. One common parameter is temperature, which nudges the output toward being more predictable or more varied; another caps how long the reply may be.

Tokens, cost, and latency

Models do not read words; they read tokens — small chunks of text, often a few characters or part of a word. Both what you send and what you get back are measured in tokens, and providers typically charge by the token. So longer prompts and longer answers cost more. Tokens also cap length: every model has a limit on how many it can handle at once. Latency is the delay between sending a request and getting a reply. A bigger, more capable model may give better answers but respond more slowly — a trade-off you feel directly in any tool that people wait on.

Choosing a model: open vs closed

Models come in two broad flavours. Closed models are accessed only through a provider's API; you send requests and pay per use, but you cannot see or hold the model itself. Open-weight models can be downloaded and run on your own hardware. Meta's Llama family, first released in 2023, is a prominent example of open-weight models that can be self-hosted. Closed models are convenient and need no infrastructure; open models offer control, privacy, and no per-token fee — but you must supply and pay for the computing power to run them.

Suppose you build a tool that turns meeting notes into a short summary. Your program assembles a request: the prompt ("Summarise these notes in three bullet points") plus the notes as context, and a parameter capping the reply length. It sends this to a model's API. Seconds later a response comes back with three bullets. If the notes are long, the request uses more tokens, so it costs a little more and may take slightly longer. Run it once a week and the cost is trivial; run it for ten thousand meetings a day and tokens, cost, and latency suddenly govern every design choice you make.

It is tempting to always reach for the biggest, most capable model. But consider a tool that classifies incoming messages as "urgent" or "not urgent." That is a narrow, simple task. A large model would answer correctly — while costing more per call and responding more slowly than a smaller model that handles the job just as well. Bigger is not automatically better. Matching the model to the difficulty of the task is often the difference between a tool that is affordable at scale and one that is not.

A concrete, verifiable reference is Meta's Llama family of large language models, first released in 2023 and made available with open weights that developers can download and run themselves. This matters because it gives builders a genuine alternative to calling a closed, hosted API: an organisation with privacy requirements or steady, heavy usage can self-host an open-weight model on its own machines, keeping data in-house and avoiding per-token fees — at the cost of managing the hardware and setup. Meanwhile, closed providers such as OpenAI and Anthropic offer their models only through APIs, trading that control for convenience. Both paths use the same underlying idea of sending a prompt and receiving a response; they differ in who runs the model. Details and licence terms evolve, so check current sources before relying on specifics.

• "The model reads whole words." It reads tokens — sub-word chunks — which is why cost and limits are measured that way.
• "A bigger model is always the right choice." Larger often means slower and pricier; match the model to the task.
• "Open-weight models are free to use." The weights may be free, but running them requires hardware and electricity you pay for.
• "Every provider works completely differently." The core request–response pattern is broadly shared across providers like OpenAI and Anthropic.

Pick a tool you would like to build — say, one that drafts polite replies to reviews. Write out its single request in plain words: what goes in the prompt, what context you would attach, and one parameter you would set. Then note one cost factor (tokens) and one experience factor (latency) you would watch as it scales.

Think Like a Maester: Every AI feature is one request and one response — master that loop, and cost, speed, and model choice stop being mysteries.

1. What two things make up a request sent to a model API?
2. What is a token, and why does it matter for cost and length?
3. What is latency, and how does model size tend to affect it?
4. What is the key difference between an open-weight model and a closed one?
5. Give one reason an organisation might self-host an open model like Llama instead of calling a hosted API.

Software talks to an AI model through a simple loop: it sends a request — a prompt plus parameters — and receives a generated response. This request–response pattern is shared broadly across providers such as OpenAI and Anthropic. The practical constraints live in the details: models read tokens, which drive both cost and length limits; latency trades against capability, since bigger models can be slower; and model choice should match the difficulty of the task rather than always defaulting to the largest option. Finally, models split into closed ones reached only through an API and open-weight ones, like Meta's Llama family released from 2023, that can be self-hosted for more control and privacy at the price of running the hardware yourself. Understanding this loop is what turns AI from something you chat with into something you can build with.

• How Large Language Models Work
• No-Code and Low-Code AI