Messages, parameters and tokens

Most model APIs share the same core idea: you send a list of messages and receive the model's next message.

Roles

• System (or developer) instructions set the overall behaviour: who the assistant is, rules, tone and output requirements. Depending on the provider, this is a separate parameter or the first message.
• User messages contain the request.
• Assistant messages contain previous model replies, included so the model sees the conversation so far.

The API is stateless in its basic form: each request must include all the context the model should see. Conversation memory is your application's job, which also means you control exactly what the model knows.

A thin wrapper

Provider SDKs differ in naming, so we hide them behind one function:

from dataclasses import dataclass

@dataclass
class Reply:
    text: str
    input_tokens: int
    output_tokens: int

def complete(system: str, messages: list[dict], *, model: str,
             max_tokens: int = 800, temperature: float = 0.2) -> Reply:
    # calls the configured provider's SDK and maps its response to Reply
    ...

The course repository includes implementations for several providers. Everything else in our code calls complete and never touches a vendor SDK directly.

Tokens

Models read and write tokens, chunks of text that are often parts of words. Pricing, speed and limits are measured in tokens:

• The context window is the maximum number of tokens the model can consider at once, including your instructions, conversation history, documents and its answer.
• max_tokens (or a similarly named parameter) caps the length of the reply. Set it deliberately: too low cuts answers off mid sentence, too high risks long, expensive replies.
• Input and output tokens are usually priced differently, with output often costing more.

Log token counts for every call from the start.

Temperature and sampling

Temperature controls randomness. Lower values make output more focused and repeatable; higher values add variety. For extraction, classification and tool use, keep it low. For brainstorming, higher can help. Some newer reasoning models fix or ignore these parameters, so check your provider's documentation.

Streaming

With streaming, the API sends the reply in small pieces as they are generated. Users see text appear immediately, which makes a large difference to perceived speed in chat interfaces. Streaming complicates parsing structured output, so we use it for user facing text and not for machine readable responses.

Errors and retries

Expect rate limit errors, timeouts and occasional server errors. Wrap calls with retries using exponential backoff and jitter, set timeouts and make failures visible in logs. A model call is a network call to a busy service, and it should be treated like one.