Intermediate · About 56 hours · 3 projects · Free

AI Roadmap for Product leaders

In about 56 hours you carry one AI feature from idea to launch review: a PRD with its evaluation written first, a prototype that has run your golden set, and a launch review with cost per task and guardrails.

0of 56 hours · 0%

Saved in this browser. No account, no email.

What you'll build

Go at your own pace. Stop at any milestone: each one leaves you with something to show.

  1. 1
    A PRD for one AI feature, with its evaluation written first

    Carry one feature from your own product through a PRD where the evaluation exists before any code.

    hour 23
  2. 2
    A working prototype on an API key, a free course Space, or a tool you already have

    Build the smallest working version of your feature and make it face your golden set.

    hour 39
  3. 3
    A launch review with cost per task and the guardrails

    Put numbers and limits around the feature, then make a go or no-go call.

    hour 56

Skills you'll learn

  • Model limits
  • Prompt design
  • Output schemas
  • Eval rubrics
  • Golden sets
  • Error UX
  • Retrieval decisions
  • Workflow design
  • Autonomy levels
  • Prototyping
  • Eval runs
  • Token economics
  • Model pricing
  • Guardrails
  • Privacy review
  • A/B testing
  • Buy vs build

Prerequisites

  • You own or influence a product roadmap.
  • You can write a one-page spec.
  • You can get an API key with a small credit, or use a chat tool your company allows.
  • No coding beyond pasting and editing a short script.

Program outline

3 milestones · 10 sections · 50 lessons · 3 projects · Every course and source is free

Learn as a feed

Milestone 1 · Hours 1 to 23 · 4 sections · 20 lessons

Decide what to build, and write the test first

Learn what models can do, then write the prompt, the output shape and the test before building.

Hours 1 to 50/17

By the end you can explain to your team what a language model does and does not do, why it makes things up, and what fits in its context, and you have chosen the one feature you will carry through the plan.

In the Encyclopedia: Large Language Models (LLMs), AI Hallucinations, Context Window & Long Context

  1. What the model does, in one paragraph55 min

  2. Karpathy's hour on LLMs, and your one feature75 min

    • Watch
      Andrej Karpathy · Intro to Large Language Models (1hr) · Andrej Karpathy
    • Do
      Pick the one AI feature you will carry through the whole plan (default if you have none: an assistant that answers questions from your help center). Write its first user story: as a [user], I want [outcome], so that [value]. Add the three limits Karpathy names that apply to it.

  3. Why it makes things up65 min

  4. The context window as a product limit60 min

  5. Where AI is paying off now, and where it bit someone60 min

Hours 6 to 110/14

By the end you have a system prompt for your feature with its failure cases written beside it, and an output shape your engineers can build against.

In the Encyclopedia: Prompt Engineering & Using LLMs Effectively, Prompt Engineering, Structured Outputs

  1. Prompting 101, the vendor way65 min

    • Watch
      Prompting 101 | Code w/ Claude · Anthropic
    • Read
      Prompt engineering overview · Anthropic
    • Do
      Write version 1 of your feature's system prompt: the role, the task in one sentence, the rules, and three worked examples. Keep it under one page and paste it into a chat model with one real input.

  2. The DeepLearning.AI short course, first half70 min

    • Course
      DeepLearning.AI · ChatGPT Prompt Engineering (Free) · DeepLearning.AI · free, certificate extra
    • Do
      Work through Guidelines, Iterative and Summarizing with your own feature's input instead of the course's: write three prompt variants for the same input and note which output you would ship and why.

  3. Second half: inferring, transforming, and the chatbot lesson75 min

    • Course
      DeepLearning.AI · ChatGPT Prompt Engineering (Free) · DeepLearning.AI · free, certificate extra
    • Do
      Write your feature's failure-case list: ten inputs where you expect the prompt to break (empty input, two questions at once, a request outside scope, an instruction hidden in the input, a very long document, the wrong language, a rude user, a question the source cannot answer, a date-sensitive question, a request for personal data). Keep it; it seeds your golden set.

  4. Structured outputs: the shape engineers build on65 min

  5. Prompt tactics, and prompts that fail quietly75 min

    • Read
      OpenAI · Prompt Engineering Guide · OpenAI
    • Code
      anthropics/prompt-eng-interactive-tutorial · anthropics/prompt-eng-interactive-tutorial
    • Do
      Revise your system prompt to version 2 with the tactics you found most useful. For each of your ten failure cases, record whether v2 handles it, with the exact output you saw if you ran it (a chat window is enough; the tutorial notebooks need an API key).

Hours 12 to 170/15

By the end you have an eval rubric and a 30-case golden set for your feature, with ten cases scored by hand as the baseline, before anyone builds anything.

In the Encyclopedia: LLM Evaluation & Benchmarks, Offline and Online Evaluation, Data Quality for AI

  1. Why your AI product needs evals55 min

  2. Success criteria and test cases75 min

  3. Build the golden set75 min

  4. Label quality: your golden set is data too65 min

  5. Rules of ML, read as a product manager70 min

    • Read
      Google's Rules of ML · practical evaluation wisdom · Google
    • Do
      Score ten of your golden cases by hand against the rubric using a current chat model and your v2 prompt. Record the pass rate per criterion as the baseline, then write the two numbers that would make you ship (offline pass rate) and stop (the live metric).

Hours 18 to 230/15

By the end your PRD is complete: the problem, the user stories, the prompt and output shape, the eval, the UX for confidence and errors, and the points where a person reviews.

In the Encyclopedia: AI Product Management, Designing AI Products (Human-AI Interaction), Human in the Loop

  1. What changes when the product is probabilistic65 min

  2. Becoming an AI PM (listen at 1.5x)75 min

  3. The PAIR codelab, part 1: user need to AI problem, and the dataset75 min

  4. PAIR part 2: confidence, limits, and a way forward from errors75 min

  5. Human in the loop: where a person must sign60 min

Milestone 1 project · about 6 hours

A PRD for one AI feature, with its evaluation written first

Carry one feature from your own product through a PRD where the evaluation exists before any code.

  • Model limits
  • Prompt design
  • Output schemas
  • Eval rubrics
  • Golden sets
  • Error UX

Courses that teach this

Build it

  1. Pick one AI feature from your own product and write the user problem it solves.

  2. Write what the model can and cannot do for it, and list how it can fail.

  3. Write the system prompt and the output schema your engineers will build against.

  4. Build the rubric and a 30-case golden set, and score ten cases by hand as the baseline.

  5. Design the error and low-confidence experience and the human review points.

  6. Set the ship and stop numbers, then get one colleague to review the PRD.

You finish with

A PRD (problem, user stories, model limits, system prompt v2 with failure cases, output schema, rubric, golden set spreadsheet with a hand-scored baseline, error UX, review points, ship and stop numbers) reviewed by one colleague.

Check your work

  • The eval exists before any code.

  • Ten cases are hand-scored as the baseline.

  • Ship and stop numbers are written down.

  • The error and low-confidence experience is designed.

See how others did it

Milestone 2 · Hours 24 to 39 · 3 sections · 15 lessons

Build the prototype and make it face the golden set

Learn retrieval and workflow choices, then build a prototype that faces your test.

Hours 24 to 280/18

By the end you can say whether your feature needs retrieval, which sources it should read, and what goes into the context on every call, written as a spec.

In the Encyclopedia: Retrieval-Augmented Generation (RAG), Context Engineering, AI in Sales and Customer Support

  1. RAG in plain words55 min

  2. RAG versus fine-tuning, and why enterprise RAG fails65 min

  3. Context engineering: what goes in the window on every call65 min

  4. Contextual retrieval and the citation rule55 min

  5. Support assistants in the wild, and golden sets that go stale75 min

Hours 29 to 340/16

By the end you have chosen the workflow shape for your feature (a fixed chain or a true agent), its autonomy level, and the approval points, with reasons an engineer will accept.

In the Encyclopedia: AI Agents, Agentic Workflows & Patterns, Levels of Agent Autonomy

  1. Agentic workflows: the four patterns75 min

  2. Anthropic's Building Effective Agents65 min

  3. OpenAI's practical guide to building agents75 min

    • Paper
      A practical guide to building agents · OpenAI
    • Do
      Fill in the guide's checklist for your feature on one page: the tools it needs, the instructions, the guardrails, and the triggers for handing off to a person.

  4. Levels of autonomy: how much the system decides alone65 min

  5. Agents in the wild: commerce and voice60 min

Hours 35 to 390/17

By the end you have a working prototype on an API key, a free course Space, or a tool you already have, and it has run your golden set at least once with a recorded score.

In the Encyclopedia: Tool Use & Function Calling, Model Context Protocol (MCP), Vibe Coding

  1. Tool use: how the model asks for an action65 min

  2. MCP: one plug for many tools55 min

  3. Vibe coding: what it is good for and where it breaks75 min

  4. Build it: three routes to a running prototype75 min

    • Course
      AI Agents Course · Hugging Face
    • Read
      OpenRouter quickstart · OpenRouter
    • Do
      Choose one route and build: (A) in the course's Space, adapt the chapter 1 agent to run your system prompt on one golden case; (B) with an API key, a short script that reads a golden case, calls the model with your prompt, and prints the structured output; (C) the same flow in a no-code tool you already have. Save five outputs.

  5. Run the golden set and record the score70 min

    • Code
      Anthropic's educational courses · Anthropic
    • Do
      Open the Prompt evaluations course for the pattern, then run all 30 golden cases through your prototype, score them against the rubric (by hand, or with a model grader for the format checks), and record the pass rate per criterion beside your hand-scored baseline. Write the three failures you will fix first.

Milestone 2 project · about 8 hours

A working prototype on an API key, a free course Space, or a tool you already have

Build the smallest working version of your feature and make it face your golden set.

  • Retrieval decisions
  • Workflow design
  • Autonomy levels
  • Prototyping
  • Eval runs

Courses that teach this

  • Embeddings

    Google · 45 minutes

    What embeddings are and how similarity search uses them.

  • Agentic AI

    DeepLearning.AI, Andrew Ng · about 8 hours · Free videos; labs and certificate extra

    Build agents with reflection, tools, planning and evals, with no framework.

  • AI Agents Course

    Hugging Face · 3 to 4 hours per chapter

    Learn what agents are, how they use tools, and build one, with a free certificate.

Build it

  1. Decide whether the feature needs retrieval, and write what goes into the context on every call.

  2. Choose the workflow shape, a fixed chain or an agent, and the autonomy level.

  3. Build the smallest version: a script, a notebook, a Space or a no-code flow.

  4. Run all 30 golden cases and record a pass rate per criterion beside your baseline.

  5. List the three failures to fix first, fix one, and run the set again.

You finish with

A script, notebook, Space, or no-code flow that runs all 30 golden cases, plus a scorecard with pass rate per criterion beside your hand-scored baseline and the three failures to fix first.

Check your work

  • It runs all 30 golden cases, not just a demo case.

  • Pass rates sit beside your baseline.

  • You can name the top three failures.

  • One fix moved the score.

See how others did it

Milestone 3 · Hours 40 to 56 · 3 sections · 15 lessons

Price it, guard it, measure it, decide

Learn to price, guard and measure the feature, then decide.

Hours 40 to 440/19

By the end you have a pricing table for your feature: tokens per task, cost per task at three model choices with and without caching, and the value per task it must beat.

In the Encyclopedia: Inference Cost & Token Economics, Token Density and Information Density, Measuring AI Value and ROI

  1. Tokens are the unit of cost60 min

  2. Input and output are priced differently60 min

  3. Caching and compression: cutting the bill without cutting quality55 min

  4. Value per task: the evidence65 min

  5. Productivity claims, read carefully70 min

Hours 45 to 500/20

By the end you have the guardrail list for your feature, injection test cases in your golden set, and a one-page privacy review.

In the Encyclopedia: AI Guardrails, Prompt Injection & Jailbreaking, Data Protection Law and AI (GDPR and beyond)

  1. Guardrails: what sits around the model70 min

  2. Prompt injection: the attack that hides in your data70 min

  3. The lethal trifecta and agent security60 min

  4. Personal data: GDPR Article 22 and your feature75 min

  5. When the chatbot said it: two cases and the promise you make60 min

Hours 51 to 560/17

By the end you have the launch review: the A/B plan with its stop rule, the ongoing eval, the buy-versus-build call, and a go or no-go recommendation.

In the Encyclopedia: A/B Testing for ML, Agent Evaluation, Build, Buy or Partner

  1. Seven rules of thumb for experimenters75 min

  2. How not to run an A/B test, and watching it live70 min

  3. Evaluating agents after launch65 min

  4. Buy, boost or build, and the vendor questions75 min

  5. The launch review65 min

Milestone 3 project · about 6 hours

A launch review with cost per task and the guardrails

Put numbers and limits around the feature, then make a go or no-go call.

  • Token economics
  • Model pricing
  • Guardrails
  • Privacy review
  • A/B testing
  • Buy vs build

Courses that teach this

Build it

  1. Count tokens per task and price the feature on three models, with and without caching.

  2. Write the value per task the feature has to beat.

  3. List the guardrails and add injection cases to the golden set.

  4. Do a one-page privacy review.

  5. Write the A/B plan with its stop rule, the ongoing eval and the buy-versus-build call.

  6. Present the launch review, with a go or no-go, to one stakeholder.

You finish with

A launch review memo with a pricing table, guardrail table, one-page privacy review, A/B and online eval plan, buy-versus-build call, and a go or no-go recommendation presented to one stakeholder.

Check your work

  • Cost per task is priced on three models.

  • The value per task beats the cost.

  • Injection cases are in the golden set.

  • The A/B plan has a stop rule.

  • It ends in a clear go or no-go.

See how others did it

Check yourself at the end

  • You can explain to your team what a language model can and cannot do for your product, and choose which tasks to give it and which to keep with people.

  • You can write the evaluation before the feature: a rubric, a golden set with refusal cases, a hand-scored baseline, and the two numbers that mean ship or stop.

  • You can treat the prompt, the context, and the output schema as product surface, with failure cases and grounding rules written beside them.

  • You can choose a workflow shape and an autonomy level for an agent feature, and defend the approval points to an engineer.

  • You can price a feature per task across models, with caching, and set the break-even pass rate it must beat.

  • You can run a launch review that covers guardrails, prompt injection, personal data, the A/B plan, the ongoing eval, and the buy-versus-build call.