8 Hard-Won Lessons from a Year Building PostHog AI: Agents Are Here, But Not Without Pitfalls

PostHog, the open-source product analytics platform, has launched PostHog AI, an AI agent integrated directly into its interface. After a year of iteration—from a hackathon prototype with a single "create trends chart" tool to a versatile agent handling multi-step analyses, SQL queries, feature flags, and error investigations—it's now used by thousands weekly. This evolution, detailed in a PostHog blog post by Michael Matloka, reveals eight critical learnings that reshape how developers should approach agentic AI systems.

The journey highlights the paradoxical nature of agents: simultaneously hard to design and surprisingly easy to implement with modern models. As PostHog AI demonstrates real-world utility in untangling 'data noodles'—interconnected events, sessions, and errors—these lessons carry implications for AI/ML engineers and product teams aiming to embed agents into their stacks.

1. Model Improvements Are a Relentless Bulldozer

Model advancements outpace expectations. A year ago, reasoning models were experimental; today, they're table stakes. PostHog's pivotal shifts include OpenAI's o4-mini for cost-effective reasoning in complex queries and Anthropic's Claude 4 family for reliable tool use. Currently powered by Claude Sonnet 4.5 for its quality-speed-cost balance, PostHog AI's core loop exemplifies how these upgrades simplify architectures previously plagued by confusion.

"It's shocking how reasoning models were still experimental 12 months ago, because today reasoning is essential." — Michael Matloka

Predicting upgrade impacts remains tricky, as noted in Sean Goedecke's analysis.

2. Agents Beat Rigid Workflows

Early attempts with graph-style workflows failed for free-form tasks. In the GPT-4o era, looped tool calls with fixed prompts led to confusion, prompting experiments like the 'hedgehog' assistant Max v0 , which could only create queries.

Modern single-loop architectures prevail: the LLM self-corrects across dozens of steps while verifying outputs. A "switch mode tool" enables scaling to PostHog's vast toolset—a technique akin to tool search for handling complexity without graphs.

3. Single Loops Outshine Subagents

Subagent hierarchies sound clever but erode context, crippling self-correction. PostHog found subagents useful only for parallel, self-contained tasks. Emergent behavior from a single LLM loop with simple tools—mirroring Claude Code's success—delivers superior results.

4. To-Dos: The Simple Superpower

The todo_write tool is deceptively powerful. Used at step ends, it reinforces next actions without executing anything substantive. This intuitive mechanism, akin to chain-of-thought prompting, keeps agents on track across extended interactions.

5. Context Is the Unsung Hero

Ambiguous user queries demand broad context. PostHog AI's /init command, inspired by Claude Code, auto-generates project memory via GPT-5-mini-powered web searches or minimal questions. This 'CLAUDE.md' equivalent ensures agents grasp domain specifics, like acronyms tied to user products.

6. Transparency Builds Trust

Hiding reasoning chains or failed calls backfired—users demanded visibility. Streaming every token, tool call, and error fosters confidence, mirroring how LLMs thrive on full context.

7. Frameworks: Proceed with Caution

PostHog regrets early adoption of LangChain and LangGraph. These lock developers into fragile ecosystems amid rapid LLM evolution. Low-level calls via native SDKs offer flexibility as providers diverge (e.g., varying web search formats).

8. Evals Fall Short—Real Usage Reigns

While useful, evals struggle with production complexity. PostHog prioritizes "Traces Hour"—weekly reviews of live traces using their LLM analytics—over synthetic benchmarks. This grounds improvements in actual user paths.

Where PostHog AI Stands Today—and What's Next

Internally, it handles SQL debugging, behavior analysis, experiments, and errors across tangled data. Users access it via the UI post-/init. Future plans: research tools, session replays, proactive alerts, code ties.

PostHog AI proves agents can tame production analytics' chaos. For builders, the mandate is straightforward: embrace model momentum, prioritize simple loops with rich context and todos, shun frameworks, and iterate from real traces. As agents mature, those who adapt nimbly will lead the field.