Taming the AI Wild West: Governance, Pipelines, and the Single Foundation Model Approach

The rush to implement AI across organizations has created a new set of distributed systems challenges that many companies are unprepared for. As CXOs face mandates to go 'AI-first' from boards and investors, the resulting landscape is a messy patchwork of uncontrolled AI usage, complex data pipelines, and significant security risks that threaten the very value these technologies promise to deliver.

The Shadow AI Problem

What's emerging is what Hema Raghavan, co-founder and head of engineering at Kumo.ai, calls 'shadow AI' - the unauthorized use of AI tools by employees across departments. This creates a governance nightmare where sensitive company data is egressing to unauthorized services through seemingly innocent actions like cleaning up a sales deck or connecting CRM tools to unvetted AI platforms.

'The company does not have the governance to understand what's going in and outside that perimeter,' Raghavan explains. 'I think that's starting to worry the CIOs quite a bit.' This concern extends to the broader supply chain, as company data flows through multiple third-party services, each adding potential points of failure and security exposure.

Architectural Approaches to AI Governance

To address these challenges, organizations are implementing several governance models:

1. Deployment within approved platforms: Solutions like Snowflake's Snowpark Container Services allow AI models to be deployed inside the database perimeter, keeping data within approved security boundaries. This approach leverages existing infrastructure while maintaining governance.

2. Gateway monitoring systems: Companies are implementing centralized gateways that route all AI calls through monitored endpoints, providing visibility into data flows and enabling detection of PII or sensitive information. This requires significant resources but offers comprehensive oversight.

3. VPC deployments: Deploying AI models within Virtual Private Clouds provides an additional layer of control and isolation, though it requires substantial DevOps expertise to implement and maintain.

The most sophisticated approaches combine these methods with AI-powered monitoring systems that can detect sensitive data patterns in real-time, creating a self-governing security perimeter around AI usage.

Pipeline Sprawl: The Hidden Cost of Complex AI Systems

Before founding Kumo.ai, Raghavan led AI teams at LinkedIn, where she witnessed firsthand the challenges of maintaining dozens of models with hundreds of supporting pipelines. This 'pipeline sprawl' creates a complex web of dependencies that becomes increasingly difficult to manage as organizations scale.

'Think recommender systems, think lead scoring, think fraud risk prediction,' Raghavan illustrates. 'They're often coming from feature engineering. Because you have a data scientist who's saying, you know what? I need to aggregate the last 30 days of clicks. And you have one pipeline that collects that aggregation and ETLs it, and then three people rely on that ETL'ed pipeline.'

The problems multiply when upstream pipelines fail. At LinkedIn, a broken front-end tracking pipeline caused model behavior to deteriorate, requiring extensive debugging efforts across multiple interconnected systems. This complexity only increases as teams grow, people leave, and pipelines suffer from bit rot.

The Single Foundation Model Approach

Kumo.ai's solution to pipeline sprawl is architectural simplification: a single foundation model that supports multiple use cases through on-the-fly database queries rather than pre-computed features.

'Can we have one foundation model?' Raghavan asks. 'Can you imagine that a company, just for all those use cases that I described, you just have one foundation model that you need to maintain? Yes, but very elegant, right? You just have one foundation model, and then of course, each use case is different.'

This approach leverages patterns from generative AI, using in-context learning where the model queries the database in real-time for relevant examples. For instance, when making job recommendations, the system would query for the user's current profile and relevant job postings on the fly, rather than relying on pre-computed features stored in separate pipelines.

'Instead of moving the pipelines between data sources, you're moving the pipelines to the AI model itself,' Raghavan explains. 'So, you don't have tons of pipelines that reach to training data that then, so when you're debugging the model, you're not debugging 20 other pipelines.'

Trade-offs in AI Architecture

The single foundation model approach offers significant advantages in maintainability and reduced complexity, but it's not without trade-offs:

1. Performance characteristics: Data warehouses often don't meet the low-latency requirements of online services like recommendation engines. This necessitates additional layers for real-time applications.

2. Specialization vs. simplicity: While reducing the number of databases is generally beneficial, some use cases may require specialized systems like vector databases or time-series databases for optimal performance.

3. Governance boundaries: Keeping AI within the data warehouse simplifies governance for analytics but may not address all security concerns for online services that require different deployment patterns.

Raghavan emphasizes that the fewer data sources, the better: 'When my team comes up with a new design, the first thing I ask is, 'can we not have one more database?' One more source of truth that we have to maintain. Because again, you have to keep all of these different sources in sync.'

The Evolution of Engineering Roles

The rise of AI and coding agents is fundamentally changing what it means to be an engineer. Junior engineers can no longer simply accept the outputs of AI tools without understanding the underlying design choices.

'Suddenly junior engineers have to grow up much faster because they have to be asking the agent design choice questions,' Raghavan observes. 'You can't just accept the answer. You have to do the work of understanding it, right?'

This shift is reflected in hiring practices, with companies like Kumo.ai focusing on candidates who can explain and evaluate AI-generated code rather than simply produce it quickly. The interview process now includes take-home problems where candidates must articulate why an agent made certain design choices.

'The interview process is elevated to a new class of engineers,' Raghavan notes. 'In the past, when you came out of college, the first thing all of the fan companies looked for was, can this person code fast? Do they know their algorithms class? I think that's gone. It's now– the code fast is the agent. Now it's, can you read this code? Do you understand the design choices? Have you given it enough test cases that you actually trust the output of this?'

Future Directions in AI Governance

Looking ahead, Raghavan anticipates several developments in AI governance and architecture:

1. Standardized visibility tools: Organizations will demand tools that provide CIOs and CISOs with visibility into API and egress flows, with these capabilities becoming standard requirements for AI vendors.

2. Cost-aware architecture: As experimentation costs become more apparent, organizations will increasingly consider the trade-offs between using external API-based services and deploying open models internally within their perimeter.

3. Design reviews for AI: AI-related design choices will become formal parts of engineering design reviews, with teams evaluating not just technical merits but also security implications and governance considerations.

The most successful organizations, Raghavan suggests, will be those that balance innovation with the lessons learned from past distributed systems challenges. 'It's an exciting time to be in engineering and I think we have to embrace the change, but I think it's also a time to take lessons from the past that we've known of places we've been burned on, choices that we've made, and not repeat past mistakes.'

As the industry matures, we're likely to see more sophisticated approaches to AI architecture that address these fundamental challenges while maintaining the velocity and innovation that makes AI so valuable. The key, as with all distributed systems, is finding the right balance between simplicity and flexibility, control and experimentation.

For organizations looking to implement these approaches, Kumo.ai provides tools for training and running state-of-the-art AI models on relational data, allowing predictions about users and transactions in seconds. More information is available on their official website or by contacting Hema Raghavan directly at [email protected].