Interrogatory LLMs: Structuring Knowledge Extraction in Complex Systems

In the evolving landscape of AI-assisted software development, Martin Fowler introduces a compelling pattern: using Large Language Models not just as content generators, but as interrogators that can extract and structure knowledge from human experts. This approach addresses a fundamental challenge in complex system design: how to capture the nuanced context required for building robust distributed architectures.

The Problem: Context Gathering in Complex Systems

Building distributed systems requires understanding multiple dimensions of knowledge: business requirements, technical constraints, data models, API contracts, and operational considerations. Traditionally, gathering this context has been a manual, often ad-hoc process that frequently leads to information gaps and inconsistencies.

The challenge intensifies as systems scale. A microservices architecture might require dozens of teams to coordinate their understanding of shared data contracts and service interfaces. In such environments, documentation becomes critical but burdensome. Technical documentation often suffers from three key problems:

1. Temporal decay - Documentation quickly becomes outdated as systems evolve
2. Incomplete coverage - Humans naturally focus on happy paths and edge cases are overlooked
3. Knowledge silos - Critical context exists only in the minds of subject matter experts

Fowler identifies that when we need an LLM to perform complex tasks like designing a new feature, we often need to feed it substantial context - potentially several pages of markdown covering user requirements, implementation guidelines, and external system dependencies. The traditional approach requires humans to author this documentation, a process that's time-consuming and error-prone.

The Solution: LLM as Interrogator

Fowler proposes an alternative approach: using an LLM to interview human experts and generate the required context. This "interrogatory LLM" would ask targeted questions to extract the necessary information, then synthesize it into a structured document for subsequent use.

The process follows a clear pattern:

1. Initialization - The LLM is given the task description and initial constraints
2. Interview phase - The LLM asks questions one at a time to clarify requirements
3. Information synthesis - The LLM compiles the gathered information into a structured format
4. Context generation - The resulting document becomes input for another LLM session to perform the actual task

This approach transforms how we think about LLM interactions. Instead of treating them as black-box generators, we position them as collaborative partners in the knowledge extraction process.

Implementation Patterns

Fowler outlines several concrete implementations of this pattern:

1. Requirements Gathering

For feature design, an interrogatory LLM can interview stakeholders to capture:

• User experience requirements
• Business rules and constraints
• Integration points with existing systems
• Performance and scalability requirements
• Security and compliance considerations

The LLM would ask clarifying questions about ambiguous requirements, probe for edge cases, and ensure all necessary constraints are documented before proceeding to design.

2. Documentation Validation

A particularly valuable application is verifying existing documentation. Given a software specification or architecture document, an interrogatory LLM can interview domain experts to assess accuracy. This approach leverages the LLM's ability to:

• Identify inconsistencies in technical descriptions
• Ask follow-up questions about ambiguous statements
• Verify that documented behavior matches actual implementation
• Flag outdated information that needs updating

This pattern is especially valuable when the original documentation is poorly written or incomplete. The conversational format makes the review process more engaging and thorough than traditional document reading.

3. Knowledge Transfer

For organizations dealing with knowledge silos, interrogatory LLMs can help extract expertise from experienced personnel and transfer it to newer team members. The LLM can guide experts through structured interviews to capture:

• Debugging approaches for common failure modes
• Operational procedures for maintenance tasks
• Historical context for architectural decisions
• Lessons learned from past incidents

Technical Considerations

Implementing an effective interrogatory LLM requires careful attention to several technical aspects:

Prompt Engineering

The quality of the interrogation depends heavily on the initial prompt. Effective prompts should:

• Clearly define the scope and objectives of the interview
• Specify the desired output format and structure
• Establish constraints on the types of questions to ask
• Define the level of technical depth required

As Fowler notes, a striking element of Harper Reed's approach is insisting that the LLM ask only one question at a time. This constraint prevents information overload and ensures each response is focused and complete. However, implementing this requires careful prompt design and may need frequent reminders during the conversation.

Context Management

The interrogatory LLM must maintain appropriate context throughout the conversation while avoiding token limitations. This requires:

• Strategic summarization of previous exchanges
• Identification of key constraints and requirements
• Recognition when information gathering is complete

For complex interviews, the system might benefit from a hybrid approach where human reviewers periodically validate the captured information before continuing.

Multi-LLM Workflows

The most sophisticated implementations might use multiple specialized LLMs in a pipeline:

1. An interrogatory LLM to gather initial requirements
2. A validation LLM to check for completeness and consistency
3. A synthesis LLM to structure the information appropriately
4. A generation LLM to produce the final output

This approach allows each model to focus on its specific strength while maintaining overall coherence in the knowledge extraction process.

Trade-offs and Limitations

While powerful, the interrogatory LLM approach comes with significant trade-offs:

Advantages

• Reduced cognitive load on human experts who don't need to structure their thoughts
• Improved consistency in documentation through standardized questioning
• Enhanced coverage of edge cases through systematic probing
• Knowledge preservation that captures tacit expertise
• Scalability for large organizations with multiple subject matter experts

Limitations

• AI-generated artifacts may lack the nuanced understanding of human-authored content
• Question quality depends entirely on the LLM's training and prompt design
• Over-reliance risk might lead to undervaluing human expertise
• Context window constraints may limit the scope of complex interviews
• Verification overhead requires human review to ensure accuracy

Ethical Considerations

The approach raises important questions about:

• Attribution - How to credit human experts versus AI contributions
• Accountability - Who bears responsibility for errors in AI-generated documentation
• Bias amplification - How the LLM's questioning might inadvertently reinforce existing biases
• Knowledge ownership - Who controls the extracted knowledge in organizational settings

Practical Implementation

For organizations looking to implement interrogatory LLMs, Fowler suggests a phased approach:

1. Start with high-value, well-defined domains where knowledge capture is critical
2. Establish human review checkpoints to validate AI-generated documentation
3. Iterate on prompt engineering based on feedback from domain experts
4. Develop specialized prompts for different types of knowledge extraction
5. Create feedback loops to improve the interrogation process over time

The most successful implementations will likely combine LLM interrogation with human review, leveraging the strengths of both approaches while mitigating their respective weaknesses.

Broader Implications

The interrogatory LLM pattern represents a fundamental shift in how we think about AI's role in knowledge work. Rather than replacing human expertise, it positions AI as a collaborator that can help structure and amplify that expertise.

This approach has particular relevance for:

• API design - Capturing complex contract requirements and versioning strategies
• Database schema evolution - Documenting data models and migration paths
• Distributed system architecture - Capturing consistency requirements and failure handling patterns
• DevOps practices - Preserving operational knowledge and incident response procedures

As Fowler notes, this technique extends beyond software development. For individuals who find writing difficult, an interrogatory LLM offers a pathway to share their knowledge without struggling with the mechanics of composition. While the resulting text may carry "that tang of AI-writing" that experienced writers find off-putting, it's preferable to the alternative of critical knowledge remaining unshared.

In the end, the interrogatory LLM pattern isn't about replacing human expertise but about creating new pathways for that expertise to flow into systems where it can create value. As our systems grow increasingly complex, these knowledge extraction techniques may become essential infrastructure for maintaining coherence and continuity in our technical organizations.

For more insights from Martin Fowler, you can explore his website and blog. Harper Reed's original approach that inspired this pattern can be found on his blog.