From chat window to product feature

When you chat with an assistant, you are the quality control. You read every answer, notice when it is wrong and ask again. In a product feature, nobody reviews each response before it reaches a user or a downstream system. That changes what matters.

What a product needs from an LLM

• Predictable shape: code needs JSON with known fields, not a friendly paragraph.
• Grounding: answers about orders or policies must come from real data, not the model's guesses.
• Bounded actions: if the model can trigger actions, such as issuing a refund, those actions need limits and confirmations.
• Measured quality: you need to know whether today's prompt is better than yesterday's, across hundreds of cases, not three you tried by hand.
• Cost and latency control: every call costs money and time, at scale.
• Safety: users and documents can contain instructions that try to manipulate the model.

This course is organised around these needs.

The project: Desk

Desk is an internal helpdesk assistant for a mid sized online shop. Employees ask questions such as "Why was order 4821 refunded twice?" or "What is our policy on late returns from Norway?". Desk can:

1. Look up orders, customers and refunds through tools backed by a mock API.
2. Search a small policy handbook.
3. Draft a reply to a customer for an agent to review.
4. Escalate to a human with a summary when it is unsure or when policy requires it.

We build it step by step, with an evaluation suite growing alongside the code.

Calling models directly first

Frameworks for LLM apps can save time, but they also hide what is sent to the model, which makes debugging hard. We start with provider SDKs and a thin wrapper of our own, about 50 lines. Once you understand the moving parts, you can judge whether a framework adds enough value for your situation.

Models and providers

Several providers offer capable models through similar APIs, including OpenAI, Anthropic and Google, and open weight models can be self hosted or used through other providers. Models differ in quality, speed, price, context length and features. We design Desk so the model can be swapped by configuration, and we use evaluations to choose, rather than reputation or leaderboards.

Budget

Running the course exercises costs a small amount in API usage. Set a monthly spending limit in your provider's console before you start, and use small, fast models for development where possible.