I've spent a lot of time over the past few years tracking various parts of the technology ecosystem, most recently focusing on the AI-native engineering landscape. Typically, each time I've built a Google Sheet: company names, funding, headcount, partnerships, pasted in by hand as I do desk research, flipping back and forth between various websites, and later adding in further details whenever I read something new worth keeping. The power of a sheet is that it's easy to add in new columns, as you learn of a new feature that is relevant, but the difficulty is in going back and re-capturing that data for earlier rows. It works, to a point, but over time the data loses shape, and keeping everything up-to-date gets unwieldy.
It feels like the sort of problem that agents should be able to help solve. So when I took the Kaggle × Google 5-Day AI Agents Intensive, I decided not to stop at the course exercises. The course is organised around five agent components — models, tools, orchestration, memory, evaluation — plus the operational concerns of running agents for real. I rebuilt my spreadsheet as Nebula: add a company as just {name, website}, research agents enrich it, the data lands in a Neo4j graph, and a React app presents it. This allowed me to test each course concept against a real problem. Sometimes this confirmed what I expected, and in one case it taught me a lesson I wouldn't have got from the class alone.
Day 1: when do you actually need an agent?
An early and useful contrast came from building two LLM-powered components side by side — only one of which was an agent.
The importer that migrated my old spreadsheet is a single structured-output call: freeform notes in, typed fields out (year founded, funding, people and their titles). No loop, no tools, no complex decisions. This is perfect for something that needs to be cheap, fast, and deterministic enough to run over hundreds of rows.
The enrichment agent is different — given a company name and website, I need something that has to make some decisions: fetch the site, notice the gaps, fall back to web search, fetch a promising result, then write once. It's not perfect at doing this, but it is a reasoning loop with tools, and I can watch it decide — and decide myself where I want to steer it the next time.
Agents provide autonomy, but that has a development cost, and it requires adjustment and fine-tuning for that to work. If the task has a fixed shape, a plain model call is cheaper, faster, and easier to trust. What I've learned to ask is whether the task involves decisions that I couldn't script in advance. If I can, that's the approach I should probably take, and not leap straight to agents every time.
Day 2: tools, and letting other agents in via MCP
Tool design turned out to be mostly about defining clear actions. The agent's tools are deliberately simple: web_search, fetch_page, and a single write path, save_company. When client discovery kept missing logo walls, the fix turned out to be a better tool, not a cleverer or more emphatic prompt: fetch_page learned to surface links and images, but I needed a vision tool to actually identify the company logos (metadata was too inconsistent).
Day 2 also covered MCP, so I experimented by having Nebula expose its graph through an MCP server. This means in my day-to-day conversations with Claude I can ask things like "which employee-owned companies partner with Anthropic?" directly. Recalling "put the guardrails on tools, not hope in prompts", the read tools are open, but the raw Cypher tool rejects write clauses.
Day 3: two kinds of memory
Adding an in-app chat assistant over the graph made the session/long-term distinction concrete. Session memory is just multi-turn context within a conversation. Long-term memory is durable facts stored as nodes in the same graph the assistant researches — tell it "I focus on employee-owned companies" today, and a completely fresh process recalls this next week. Keeping memory in the graph rather than a separate store means it's inspectable: the memories are just nodes, so I can query exactly what the assistant believes with the same read-only tools I use for everything else. (Deleting one still means going to the database directly — a forget tool is an obvious gap - there's a lot more to explore here)
Day 4: the eval that had to be evaluated
The lesson that justified the whole exercise came from the eval harness. I built one for the enrichment agent: performing deterministic field checks against known values, trajectory checks (did it search, fetch, and save exactly once?), and an LLM-as-Judge for faithfulness. The judge promptly flagged the agent for hallucinating — real 2026 funding figures were marked "overstated", and actual past dates marked "future projections".
That seemed strange, until I realised that it was the judge that was the hallucinator! It was scoring the agent's output against its own stale training knowledge, not against what the agent had actually found.
To fix it required a change to the architecture, not just the eval. The agent now cites a source URL and date for every financial figure and headcount number, stored as citation edges in the graph, and a guardrail drops any figure that arrives without one. The judge's question changed from "is this true?" (unanswerable by a model with a cutoff) to "is this supported by the evidence the agent retrieved?" (checkable). Faithfulness scores went from an average of 3.3 to 4.7, and — more importantly — the remaining flags became trustworthy: the harness still correctly catches an unsupported number when the agent produces one.
You have to eval your eval. And data provenance turned out to be the thing that makes faithfulness checkable at all — an LLM judge can't verify a claim against the world, only against cited evidence, and without citations there's no evidence to check against. This helps in the eval harness and in the product, where every agent-sourced fact now shows its citations in the UI.
The same distrust-by-default shaped the write path. The chat assistant can research a company but can't write to the graph: it proposes an enrichment, the UI shows a review card with fields and citations, and only my explicit commit writes it. The agent proposes the updates; but I'm the one who commits.
Day 5: the shift to production is where real lessons get learned
The final day's material — deployment, auth, operations — became a day of actual shipping: Firebase Auth with an allowlist in front of everything, slow research moved to background jobs so nothing times out, the API on Cloud Run scaling to zero, CI/CD on every merge. Nebula now runs at a real URL, replacing the spreadsheet it grew out of.
Production also forced fixes for issues that you wouldn't cover in a class that tends to show the happy path: rate limits that turn a five-call agent loop into a burst of 429s, an SVG logo crashing the vision tool, and background tasks failing silently until the failure was logged and surfaced.
A brief aside on tooling: the entire build took a weekend, built with Claude Code — and the repo's PR reviews are now handled by an agent too, working from a written review guide. Agents building agent tooling, reviewed by agents. It's turtles most of the way down.
The difference between a course, and building something
I finished all five days before writing a line of Nebula, so it wasn't just that I stepped through with my own examples. What mattered was having a real problem to hang the principles on. The eval material stayed abstract to me until I watched my own judge hallucinate hallucinations, and then it clicked: this is what that day was about. The course gave me the vocabulary, but a few of the ideas only made sense once a concrete failure sent me back to the material to work out what had gone wrong. Everything is on GitHub, including a LEARNING.md mapping each course day to the code that implements it. Next up from the course: structured tracing, and splitting the enrichment agent into specialist sub-agents. Next up for the tool: driving new feature requests directly from my research usage.