Busting AI Myths and Embracing Realities in Privacy & Security

Katharine Jarmul delivered a compelling keynote at InfoQ Dev Summit Munich, addressing the growing tension between AI automation and privacy/security concerns. As AI systems increasingly shift from augmentation to automation, organizations face unprecedented challenges in maintaining data protection while leveraging these powerful tools.

The Guardrail Myth: Why Technical Safeguards Fall Short

The first myth Jarmul tackles is the belief that guardrails will save us. She explains that guardrails—technical mechanisms designed to filter harmful or private content—come in multiple forms, each with distinct vulnerabilities.

Software-based guardrails use input-output filters and memory architectures like Bloom filters to block problematic content. However, these can be easily bypassed through simple techniques like variable renaming or ASCII art obfuscation, as demonstrated by researchers who circumvented copyright protections by translating code into French.

External algorithmic guardrails employ machine learning models or LLM-as-a-judge systems to evaluate prompts and responses. While more sophisticated, these too have weaknesses—ArtPrompt attacks show how ASCII art can mask harmful keywords from detection systems.

Model alignment through RLHF/DPO represents the most comprehensive approach, retraining models to produce desired outputs. Yet even these systems can be manipulated through carefully crafted prompts that activate memorized training data.

Jarmul's key insight: guardrails are valuable tools but insufficient as standalone solutions. Organizations must understand their limitations and implement multiple layers of protection.

Performance Isn't Privacy: The Overparameterization Problem

A second dangerous myth suggests that better-performing models will inherently solve privacy issues. Jarmul traces the evolution of large language models, highlighting how overparameterization—having more model parameters than training data points—has fundamentally changed how these systems operate.

Historically, machine learning models suffered from overfitting, requiring early stopping to prevent poor generalization. Modern overparameterized models, however, can memorize training data while still generalizing well—a phenomenon researchers like Chiyuan Zhang have extensively studied.

This memorization poses serious privacy risks. Training datasets often contain sensitive information: medical records, mugshots, watermarked images, and personal data scraped without proper consent. The larger and more capable models become, the greater their potential to memorize and inadvertently expose private information.

Differential privacy offers one mitigation, with recent advances like VaultGemma demonstrating that privacy-preserving training is possible without catastrophic performance loss. However, organizations must carefully consider when memorization is acceptable (lyrics, trivia) versus when generalization is preferable (confidential business data).

Risk Frameworks: Necessary but Insufficient

Jarmul critiques the proliferation of AI risk taxonomies, from MIT repositories to NIST guidelines and the EU AI Act. While comprehensive, these frameworks often present impractical recommendations for most organizations.

For instance, OWASP's suggestions to implement "automated scanning for anomalies and cryptographic validation of stored data" may exceed the capabilities of teams without dedicated security infrastructure. Similarly, directives to "limit knowledge propagation" and avoid "low-trust inputs" fail to address the fundamental challenge of controlling training data provenance.

Her solution: establish interdisciplinary risk radar sessions that bring together developers, data scientists, privacy experts, and security professionals. Regular collaborative discussions help teams identify relevant threats, debunk myths, and develop practical mitigation strategies tailored to their specific AI implementations.

Red Teaming: Beyond One-Time Assessments

The fourth myth—that a single red teaming exercise provides lasting security—ignores the dynamic nature of AI systems and threat landscapes. Jarmul advocates for iterative, ongoing red teaming that evolves with your architecture and threat models.

Effective AI red teaming requires understanding attacker motivations: data theft, service disruption, brand damage, or cost inflation. Teams should model specific attack scenarios, test systematically, and iterate based on findings. This process builds security expertise across the organization while creating reusable testing infrastructure.

Practical implementation involves integrating threat modeling (using tools like PLOT4AI), incorporating testing into MLOps pipelines, conducting cost and stress testing, developing evaluation frameworks, and maintaining robust monitoring systems.

The Next Model Won't Save You

Perhaps the most sobering myth is the belief that future model versions will resolve current privacy and security challenges. Jarmul presents data showing that most AI usage focuses on practical advice, writing assistance, and information retrieval—not security-critical applications.

Product priorities drive model development. Companies optimize for engagement, usability, and market share rather than privacy by default. Features like OpenAI's memory function enable sophisticated user profiling for advertising purposes, while model designers explicitly craft personalities to increase user engagement.

The solution isn't waiting for vendor improvements but taking ownership of AI security. Jarmul recommends diversifying model providers, experimenting with local models (Ollama, GPT4All), and exploring open-weight alternatives like Apertus, which provides transparency about training data and security testing.

Building Responsibility and Ownership

Throughout her presentation, Jarmul emphasizes that effective AI privacy and security requires cultural change. Organizations must move beyond blame cultures where security incidents are hidden due to fear of repercussions. Instead, they should foster environments where responsibility, agency, and ownership are distributed across teams.

Practical steps include:

• Implementing interdisciplinary risk radar sessions
• Developing robust security and privacy testing frameworks
• Evaluating and using open-weight and local models
• Creating reusable testing infrastructure
• Building organizational muscle memory for security practices

Jarmul's "feminist AI LAN party" demonstration—serving multiple LLMs from a single gaming laptop to 30 participants—illustrates the democratization of AI capabilities. When organizations can run models locally, they gain control over data flows and reduce dependency on potentially problematic cloud providers.

The Path Forward

The keynote concludes with a call to action: only through our own care and intervention can we address AI privacy and security challenges. Drawing on Smokey the Bear's message about forest fire prevention, Jarmul reminds us that responsibility lies with each practitioner.

Organizations should ask themselves which mitigations they're willing to implement:

• Testing and implementing guardrails
• Using or training differentially private models
• Running interdisciplinary risk radar sessions
• Developing robust security and privacy testing
• Evaluating and using open-weight and local models

By taking ownership of these challenges rather than waiting for vendors to solve them, organizations can build AI systems that balance innovation with the privacy and security expectations of users and regulators alike.

The presentation serves as both a wake-up call and a practical guide, helping organizations navigate the complex intersection of AI capabilities and security requirements in an era where automation increasingly replaces human judgment.