Noble Vision Academy

Course 1 · in production

Seeing What Your AI Is Doing

Observability for AI agents — what the agent did, whether it was right, what it cost, and who is at risk.

Your dashboards tell you the system is up. They do not tell you what your agent decided to do. This course closes that gap — in short lessons, with hands-on checks along the way.

Get notified when early access opensSee what is inside

8 modules · 25 lessons of 5–7 minutes · self-paced · certificate of completion

Same question · two runsSynthetic data
Run 13 steps
  1. Looks up the order
  2. Checks the refund policy
  3. Answers
HTTP 200 · inside the latency target9,216 tokens

Right answer.

Run 25 steps
  1. Searches the knowledge base
  2. Searches the knowledge base
  3. Searches the knowledge base
  4. Never looks up the order
  5. Answers anyway
HTTP 200 · inside the latency target37,730 tokens

Wrong answer, just as confident. Nothing paged.

Hollenby Support Agent, “Where is my refund?” — an illustrative example, not a client result.

The four questions

For any single request, you should be able to answer four questions.

Classic monitoring answers “is it up, is it slow, is it erroring?” Those still matter. For agents they are not enough.

01

What did the agent do?

Every model call, tool call and retry, in order — a path you can replay.

02

Was it the right work?

A clean finish is not a right answer. Evals tell you whether it did the job.

03

What did it cost, and how long did it take?

Tokens, money and latency per request, so a change in behaviour shows up.

04

Is anyone at risk?

What was sent, what came back, and what should never have been logged.

How it works

Not a video you sit through. A course you work through.

Short lessons

Five to seven minutes each, with chapters, captions and a searchable transcript. Pick up exactly where you stopped, on any device.

Checkpoints

The video pauses at the moments that matter and asks you one question. The answer explains itself and links back.

Try it

Small hands-on steps in a trace viewer, on synthetic data: find the repeated tool call, spot the step that never ran.

Labs and incidents

Each module closes with a lab or a branching incident — choose what to check next and see what happens.

Scenario exam

Thirty questions, mostly real-looking traces: what went wrong, and what would you do? Eighty percent to pass.

Certificate

A certificate of completion with a public verification page, ready for LinkedIn. No accreditation is claimed.

The modules

Eight modules, from the first trace to the 2 a.m. incident.

  1. Module 1Why agent observability is different3 lessons
    1. 1.1From SNMP and APM to agents that decide
    2. 1.2The four questions you have to answer in production
    3. 1.3A working map of the agent observability stack
  2. Module 2Instrumentation and tracing4 lessons
    1. 2.1OpenTelemetry in plain language
    2. 2.2Spans for prompts, completions, tool calls, and loops
    3. 2.3Context propagation across services and clouds
    4. 2.4Instrumentation mistakes that hide the real path
  3. Module 3Logging prompts and outputs safely3 lessons
    1. 3.1What to log — and what never to log by default
    2. 3.2Redacting PII and PHI without blinding yourself
    3. 3.3Retention, access, and audit in regulated environments
  4. Module 4Evals: knowing if the agent is still doing the job3 lessons
    1. 4.1Offline evals and golden regression sets
    2. 4.2Online evals and LLM-as-judge — uses and limits
    3. 4.3Wiring evals into release and production gates
  5. Module 5Cost, tokens, latency, and SLOs3 lessons
    1. 5.1Token accounting and cost visibility
    2. 5.2Latency, tail behavior, and agent SLOs
    3. 5.3When cost and latency tell you the agent changed
  6. Module 6Alerting and incident response for agents3 lessons
    1. 6.1Failure modes unique to agents
    2. 6.2Designing alerts from traces, evals, cost, and safety signals
    3. 6.3Incident response: a worked illustrative example
  7. Module 7Multi-cloud, private cloud, and dashboards3 lessons
    1. 7.1Collecting telemetry on AWS, Azure, and GCP
    2. 7.2Private cloud, on-prem, and data boundaries
    3. 7.3Dashboards that answer the four questions
  8. Module 8Capstone: instrument an agent end to end3 lessons
    1. 8.1The capstone brief and observability plan
    2. 8.2Walkthrough: traces, logs, evals, cost, and alerts
    3. 8.3The shipping checklist and what to do next

Who teaches it

Ian Noble

Ian has worked in network and systems monitoring since 1998 — network management, NetOps platforms, telemetry and enterprise dashboards — and now on what it takes to see what AI agents are doing in production.

About the presenters

This course is taught by Ian Noble and uses AI presenters. Ian's on-screen likeness and voice are synthetic, made with his consent, and he reviewed every lesson. Q and Janet are AI presenters, not real people. Dashboards, traces and logs show synthetic data.

Early access

Early access opens when Modules 1 and 2 are ready.

The remaining modules follow on a published schedule. Leave your address and we will tell you the day it opens.

Get notified

Teams of 50 or more: write to the same address about an enterprise licence.