Experienced leaders · About 55 hours · 3 projects · Free

AI Roadmap for Tech leaders

In about 55 hours you hold an architecture decision record, a model and vendor scorecard with real prices, and an evaluation and rollout plan with guardrails, all for one real AI feature in your company.

0of 55 hours · 0%

Saved in this browser. No account, no email.

What you'll build

Go at your own pace. Stop at any milestone: each one leaves you with something to show.

  1. 1
    Architecture decision record for one AI feature

    Decide how one AI feature should be built, and write down why and what would change the decision.

    hour 17
  2. 2
    Model and vendor selection scorecard with real prices

    Compare models and vendors for one feature on real prices and your own test cases, then recommend one with an exit plan.

    hour 33
  3. 3
    Evaluation and rollout plan with its guardrails

    Write the plan that takes the feature from a working prototype to production and keeps it there.

    hour 55

Skills you'll learn

  • Architecture decisions
  • Retrieval vs fine-tuning
  • Context budgets
  • Output contracts
  • Permissions
  • Autonomy levels
  • Cost modeling
  • Caching
  • Open vs closed models
  • Gateways
  • Vendor evaluation
  • Exit planning
  • Golden sets
  • Rubrics
  • Threat modeling
  • Guardrails
  • Red teaming
  • Rollout
  • Drift monitoring
  • Governance

Prerequisites

  • You have led an engineering team or owned a system's architecture.
  • You can read a pricing page and a short technical paper.
  • You have one real AI feature, planned or live, to use as your running example.
  • No code or math is needed.

Program outline

3 milestones · 10 sections · 50 lessons · 3 projects · Every course and source is free

Learn as a feed

Milestone 1 · Hours 1 to 17 · 3 sections · 15 lessons

Architecture: what you are buying and how it fits

Learn what a language model is and how the four ways of giving it your data compare.

Hours 1 to 60/15

By the end you can explain to your board in plain words what a language model does and does not do, show where each dollar in the AI stack goes, and name the one feature your architecture decision record will cover.

In the Encyclopedia: Large Language Models (LLMs), The AI Stack, AI Strategy

  1. What a language model actually does, and does not70 min

  2. The Karpathy hour: how these models are built, tuned and served75 min

    • Watch
      Andrej Karpathy · Intro to Large Language Models (1hr) · Andrej Karpathy
    • Do
      Write a ten-line glossary in your own words (pretraining, fine-tuning, RLHF, inference, tokens, context window, hallucination, tool use, open weights, scaling) and send it to your leadership team as the shared vocabulary.

  3. The AI stack: layers, lock-in and where the money goes65 min

  4. Where the opportunities are, and why most pilots stall65 min

  5. Pick the feature and write its decision frame75 min

    • Course
      Transform your business with AI · Microsoft Learn
    • Read
      Making AI Work: Leadership, Lab, and Crowd · Ethan Mollick
    • Do
      Do the 'Create business value with AI' module of the Microsoft path. Then choose the one AI feature your architecture decision record will cover and write its decision frame in five lines: the user, the job to be done, what good looks like as a number, what it must never do, and the date a decision is due. Have one peer check it.

Hours 7 to 110/17

By the end you can name the four ways to put your data in front of a model (prompt, retrieval, fine-tuning, tools), pick one for your feature and defend the choice with its cost driver and its risk in a paragraph.

In the Encyclopedia: Context Engineering, Retrieval-Augmented Generation (RAG), Fine-Tuning LLMs

  1. Context engineering: what goes in the window and what it costs65 min

  2. RAG, explained for the person who signs off on it70 min

  3. Retrieval that works: contextual retrieval, hybrid search and agentic loops65 min

    • Read
      Contextual retrieval in AI systems · Anthropic
    • Watch
      What is Agentic RAG? · IBM Technology
    • Do
      Decide whether your feature needs plain retrieval, hybrid search with reranking, or an agentic retrieval loop. Write the decision in three sentences with the one measurement (retrieval hit rate on 20 real questions) that would prove you wrong.

  4. Fine-tuning: when it earns its keep70 min

  5. Structured outputs: the contract between the model and your software60 min

    • Read
      Structured outputs · Anthropic
    • Read
      Structured model outputs · OpenAI
    • Do
      Write the output contract for your feature: the fields the model must return, which are required, what a malformed answer triggers, and the fallback the user sees. Add it as the interfaces section of your architecture decision record.

Hours 12 to 170/15

By the end you can tell a workflow from an agent, set the autonomy level and the permissions for your feature, and finish the architecture decision record.

In the Encyclopedia: AI Agents, Model Context Protocol (MCP), Agentic Coding

  1. Agentic design patterns: Andrew Ng's course, first lessons65 min

    • Course
      Agentic AI, Andrew Ng (free, certificate extra) · DeepLearning.AI · free, certificate extra
    • Do
      Watch the first module. Then write one paragraph on which of the four patterns (reflection, tool use, planning, multi-agent collaboration) your feature needs, and which it does not.

  2. Workflows versus agents: the patterns that ship65 min

  3. Tool use and MCP: the contract with your systems65 min

  4. Harnesses, autonomy and multi-agent systems: how much rope, at what cost70 min

  5. Coding agents in your engineering org, then finish the record65 min

Milestone 1 project · about 6 hours

Architecture decision record for one AI feature

Decide how one AI feature should be built, and write down why and what would change the decision.

  • Architecture decisions
  • Retrieval vs fine-tuning
  • Context budgets
  • Output contracts
  • Permissions
  • Autonomy levels

Courses that teach this

  • Agentic AI

    DeepLearning.AI, Andrew Ng · about 8 hours · Free videos; labs and certificate extra

    Build agents with reflection, tools, planning and evals, with no framework.

  • Building with the Claude API

    Anthropic · 9 hr

    Call the Claude API, then build tool use, RAG, evals, MCP and agents.

Build it

  1. Pick one AI feature your company wants or already runs, and write the decision to be made.

  2. Compare the four ways to give the model your data: prompt, retrieval, fine-tuning, tools.

  3. Set the context budget and the output contract your engineers will build against.

  4. Write the permissions table and the autonomy level, with the human approval points.

  5. Write the decision, its consequences, and what would reverse it.

You finish with

A two to three page architecture decision record in your team's format: context, options considered (prompt only, retrieval, fine-tuning, tools and agents), decision, interfaces, permissions, consequences, and what would change the decision.

Check your work

  • At least three options are compared on cost, risk and effort.

  • The permissions table names what the model may do alone.

  • It says what would make you reverse the decision.

  • An engineer could start building from it.

See how others did it

Milestone 2 · Hours 18 to 33 · 3 sections · 15 lessons

Models, vendors and cost

Learn what AI really costs, then compare models and vendors on your own terms.

Hours 18 to 220/15

By the end you have a cost-per-task model for your feature at three volumes, built from published prices you recorded with their dates, and can name the two levers that cut it most.

In the Encyclopedia: Inference Cost & Token Economics, Prompt Caching, Semantic Caching

  1. Token economics: why input and output are priced differently60 min

  2. Latency: prefill, decode, and what the user feels60 min

  3. Prompt caching: the cheapest saving you will ever make65 min

    • Read
      Prompt caching · Anthropic
    • Read
      Prompt caching · OpenAI
    • Do
      Reorder your feature's prompt so the stable parts come first, using your context budget. Recompute the cost model with the cached input price from both vendor pages, and record the cache discount and its conditions (minimum length, time to live).

  4. Semantic caching: when a near-duplicate question should not reach the model (short course, first half)65 min

    • Course
      Semantic Caching for AI Agents · DeepLearning.AI
    • Do
      Do the first half of the short course, about 45 minutes. Then decide whether your feature's traffic repeats: estimate the share of near-duplicate questions, the hit rate at which the cache pays for itself, and the risk of a stale or wrong cached answer. Write the decision as a short paragraph with the similarity threshold you would set.

  5. Cost per task at three volumes, and the agent multiplier70 min

Hours 23 to 280/18

By the end you have a shortlist of five to eight candidate models and gateways with published prices and context limits, and a written decision on API versus self-hosting with the numbers behind it.

In the Encyclopedia: Open-Weights & Small Language Models, Model Routing & AI Gateways, GPU & TPU Computing

  1. Open weights versus closed APIs: what you actually get65 min

  2. Small models and where they belong70 min

  3. Leaderboards and arenas: the gap between a benchmark and your task60 min

  4. Gateways and routing: one interface, many models, a spend cap70 min

  5. GPUs, AI factories and the self-hosting decision65 min

    • Read
      Which GPU for deep learning? · Tim Dettmers
    • Read
      What Is an AI Data Center? | IBM · IBM
    • Do
      Write the self-host versus API decision for your feature: the GPU memory the model needs, a monthly cloud GPU price from any provider page you can read (with the date), the engineer-months to run it, and the API cost at your expected volume. Decide, and put the paragraph in the scorecard.

Hours 29 to 330/15

By the end you have a finished model and vendor scorecard with weighted criteria, real prices and a one-paragraph recommendation you could read to the board.

In the Encyclopedia: Build, Buy or Partner, Evaluating AI Vendors and Procurement, AI Platforms

  1. Buy, boost or build65 min

  2. What enterprises actually buy, and why projects fail65 min

  3. Vendor evaluation: the questions to ask and the evidence to demand65 min

  4. Platforms: hosted models, evals, serving and observability on one bill65 min

  5. Finish the scorecard and write the recommendation65 min

Milestone 2 project · about 8 hours

Model and vendor selection scorecard with real prices

Compare models and vendors for one feature on real prices and your own test cases, then recommend one with an exit plan.

  • Cost modeling
  • Caching
  • Open vs closed models
  • Gateways
  • Vendor evaluation
  • Exit planning

Courses that teach this

Build it

  1. Build a cost-per-task model from published prices, with the date you read each price.

  2. Run it at three volumes, with and without caching.

  3. Shortlist five to eight open and closed models and gateways.

  4. Weight your criteria and score each candidate, including API versus self-hosting.

  5. Run your own test cases on the top candidates and write the recommendation and exit plan.

You finish with

A spreadsheet with the cost model at three volumes, five to eight candidates scored against weighted criteria with prices and the date read, a vendor question list, and a one-paragraph recommendation with the exit plan.

Check your work

  • Every price has the date you read it.

  • Cost per task is shown at three volumes.

  • Your own test cases, not public benchmarks, decide the ranking.

  • The recommendation includes how you would leave.

See how others did it

Milestone 3 · Hours 34 to 55 · 4 sections · 20 lessons

Evals, security, operations and governance

Learn to evaluate, secure, roll out and govern an AI feature.

Hours 34 to 390/16

By the end you have written success criteria, a 30-case golden set and a judging rubric for your feature, and can explain to the board why a vendor demo is not an evaluation.

In the Encyclopedia: LLM Evaluation & Benchmarks, Agent Evaluation, AI Hallucinations

  1. Why your AI product needs evals, and what an eval is65 min

  2. Build the golden set75 min

  3. Judges, and hallucination as a measured thing60 min

    • Read
      Why language models hallucinate · OpenAI
    • Read
      Reduce hallucinations · Anthropic
    • Do
      Write the judging rubric: a 1 to 5 scale with an example at each level, the rule for an unsupported claim, and when a person must grade instead of a model. Grade five golden cases by hand to calibrate it.

  4. Reading a vendor's numbers: benchmarks, contamination and your own run70 min

  5. Evaluating agents: trajectories, cost and safety, not just answers65 min

    • Read
      Demystifying evals for AI agents · Anthropic
    • Read
      What is AI Agent Evaluation? | IBM · IBM
    • Do
      If your feature calls tools, add 10 agent cases to the golden set that check the path taken: the right tool called, no forbidden action, steps under a cap, cost under a cap. Then write the eval section of the rollout plan: what runs on every prompt change and what runs weekly.

Hours 40 to 440/16

By the end you have a threat model for your feature, a guardrail design with its human approval points, and a red-team checklist your team can run before launch.

In the Encyclopedia: Prompt Injection & Jailbreaking, AI Guardrails, Red Teaming AI Systems

  1. Prompt injection: what it is and why a filter does not fix it75 min

  2. The lethal trifecta: private data, untrusted content and a way out60 min

    • Read
      The lethal trifecta for AI agents · Simon Willison
    • Read
      What is AI Agent Security? | IBM · IBM
    • Do
      Check your feature against the trifecta: does it read private data, process untrusted content, and have any channel out (send, post, call, write)? If all three, decide which one you remove or put behind approval. Write the decision and update your permissions table.

  3. Guardrails: input checks, output checks, permissions and approval65 min

    • Read
      How to implement LLM guardrails · OpenAI
    • Read
      What Are AI Guardrails? | IBM · IBM
    • Do
      Design the guardrail stack for your feature as a table: layer (input filter, schema validation, tool allow-list, output check, human approval), what it catches, what it costs in latency, and what the user sees when it fires. Mark which layers block and which only log.

  4. Red teaming: break it before someone else does65 min

  5. NIST's attack taxonomy and the security section of the plan65 min

Hours 45 to 500/14

By the end you can describe the production loop for a model feature (version, evaluate, trace, watch cost, catch regressions) and have written the rollout stages, dashboard and alerts for your feature.

In the Encyclopedia: LLMOps, ML Monitoring & Observability, Model & Data Drift

  1. What changes when the model is an API: LLMOps in one page65 min

  2. Production ML systems: Google's checklist against your feature75 min

  3. Monitoring and observability: traces, cost and quality in production60 min

    • Read
      What is AI Observability? | IBM · IBM
    • Read
      Evidently AI · ML Monitoring · evidentlyai.com
    • Do
      Specify the dashboard for your feature: ten panels (requests, p50 and p95 latency, cost per task, cache hit rate, error rate, guardrail fires, judge score on sampled traffic, thumbs-down rate, tool failures, tokens per task), each with an alert threshold and the person paged.

  4. Drift: the world changes and no error is raised65 min

    • Read
      What Is Model Drift? | IBM · IBM
    • Read
      Data distribution shifts and monitoring · Chip Huyen
    • Do
      List the drift sources for your feature (user mix, document corpus, vendor model updates, prompt edits, seasonal topics), and for each the signal that would reveal it and how often you check. Add 'the vendor silently updated the model' as a row and decide how you pin versions.

  5. Rollout stages: offline gate, shadow, canary, A/B and the kill switch65 min

    • Read
      What is Model Evaluation? | IBM · IBM
    • Read
      LLM evaluation: a beginner's guide (Evidently AI) · evidentlyai.com
    • Do
      Write the rollout stages for your feature: offline gate (golden set pass rate), shadow (model runs, nobody sees), canary (1 to 5 percent of traffic, which users), A/B (metric, duration, decision rule), full release. For each: entry criteria, exit criteria, who decides, and the kill switch.

Hours 51 to 550/17

By the end you can state which rules apply to your feature (NIST AI RMF, the EU AI Act tier, GDPR Article 22, your own policy), have a governance and incident process in the rollout plan, and have a team proposal ready for the board.

In the Encyclopedia: AI Governance, AI Regulation & Governance, Data Protection Law and AI (GDPR and beyond)

  1. Governance and the NIST AI RMF: govern, map, measure, manage60 min

  2. The EU AI Act and US rules: which tier is your feature60 min

    • Read
      What is the EU AI Act? | IBM · IBM
    • Read
      AI Act | Shaping Europe’s digital future · European Commission
    • Do
      Classify your feature under the EU AI Act (prohibited, high risk, transparency duty, minimal) with the article or annex you relied on, and note the US rules that touch it (state law, your sector's regulator). Write three sentences on what the classification obliges you to do before launch.

  3. Data protection: GDPR, Article 22 and the CNIL recipe60 min

  4. Incidents: the database, the definition and your response process70 min

  5. Team shape, hiring, and the board conversation75 min

Milestone 3 project · about 10 hours

Evaluation and rollout plan with its guardrails

Write the plan that takes the feature from a working prototype to production and keeps it there.

  • Golden sets
  • Rubrics
  • Threat modeling
  • Guardrails
  • Red teaming
  • Rollout
  • Drift monitoring
  • Governance

Courses that teach this

  • Production ML systems

    Google · about 1 hour

    How machine learning systems fail in production and how to test and monitor them.

  • AI security fundamentals

    Microsoft Learn

    Learn AI attack techniques and controls, then plan red-teaming exercises for LLM apps.

  • Evaluating AI Agents

    DeepLearning.AI · 2h41m · Free for now, certificate extra

    Trace an agent, evaluate router and skills, run experiments to improve it.

Build it

  1. Write success criteria, a 30 to 40 case golden set and a judging rubric.

  2. Run the golden set against a working prototype and record pass rates.

  3. Write the threat model, the guardrails and a red-team checklist your team can run.

  4. Define rollout stages, the dashboard, the alerts and the drift checks.

  5. Write the governance page, the data protection check and the incident process, with named owners.

You finish with

A plan of about ten pages plus a 30 to 40 case golden set, ready to hand to the engineering team and to show the security lead, the data protection officer and the board.

Check your work

  • Success criteria are numbers, set before launch.

  • Each threat maps to a guardrail and a test.

  • Each rollout stage has a rule to stop.

  • Every task has a named owner.

See how others did it

Check yourself at the end

  • You can explain to the board what a language model does and does not do, and defend one architecture choice (prompt, retrieval, fine-tuning, agent) in writing.

  • You can put a cost per task on an AI feature from published prices, with and without caching, and say which lever cuts it most.

  • You can run a model and vendor selection with weighted criteria, real prices and your own test cases instead of a vendor demo.

  • You can specify an evaluation (success criteria, golden set, judging rubric) and a staged rollout with a dashboard, alerts and a kill switch.

  • You can threat-model an AI feature against prompt injection and the lethal trifecta, design its guardrails, and hand your team a red-team checklist.

  • You can place the feature under the rules that apply (NIST AI RMF, the EU AI Act tier, GDPR Article 22), run an incident process, and propose the team that owns it.