Source: Hacker News discussion

When Surveillance Starts to Feel Like a Feature

“I think a year ago I would have expected this, but I’ve grown pretty complacent using AI apps and lately have started becoming more shocked realizing how much they are collecting constantly.”

That comment, buried in a Hacker News thread, captures a shift that should make every engineer, product owner, and CTO pause. Not because it’s novel—but because it isn’t.

We walked into the AI era assuming heavy telemetry and broad data capture were part of the deal: log prompts, store embeddings, analyze usage patterns, improve the model. Sensible. Expected. Necessary.

Then something changed. The tooling got better, the UX friction disappeared, models moved from novelty to infrastructure—and with that normalization came a subtle failure mode:

  • Users stopped asking what was being collected.
  • Builders stopped being explicit about what was being collected.
  • Everyone quietly agreed this was "just how AI works."

Now the shock is back. And it’s warranted.

What “Constantly Collecting” Actually Means in AI Products

For a technical audience, "data collection" isn’t an abstract privacy talking point. It’s a concrete pipeline design decision.

Modern AI apps—SaaS copilots, browser extensions, integrated IDE assistants, AI CRMs, chatbots in productivity suites—frequently ingest:

  • Raw user prompts and documents (often including secrets, source code, contracts, and PII)
  • Clickstreams and interaction logs (what you ask, when you ask, how you correct)
  • Workspace or repo context (repository scans, file trees, code metadata, comments)
  • Application and device metadata (IP, user agent, rough location, org identifiers)
  • Sometimes voice, screen context, or screenshots for multimodal assistance

Many of these products:

  • Retain logs beyond what’s necessary for immediate inference.
  • Use data to train or fine-tune models, either by default or via dark-pattern consent.
  • Pipe data through third-party infrastructure (observability, analytics, error reporting).
  • Are vague, fragmented, or quietly mutable in their privacy disclosures.

None of this is hypothetical; it’s the natural outcome of combining:

  1. A performance arms race (better models need more feedback).
  2. A growth arms race (PMs want more data for engagement and personalization).
  3. A compliance posture that lags behind the product roadmap.

The result: pervasive, normalized surveillance woven into tools that developers now use to write production code, handle production data, and reason about sensitive systems.

Why Complacency Is Now a Technical Risk

For seasoned builders, the core issue isn’t moral panic; it’s risk management.

If you lead engineering at a SaaS company, a bank, a health-tech startup, a devtools vendor, or a critical infrastructure provider, quiet data collection by AI tools threatens you on multiple fronts:

  • Supply chain risk:

    • AI plugins, browser extensions, and IDE assistants can exfiltrate codebases and configs.
    • Third-party AI integrations become unvetted data processors with privileged context.

  • Regulatory and contractual exposure:

    • GDPR, CCPA, HIPAA, PCI, and sector-specific regimes increasingly treat “improve our AI” as an insufficient justification for broad data reuse.
    • Using customer data to train general models without explicit, informed consent is a lawsuit waiting to happen.

  • IP and confidentiality leakage:

    • Proprietary algorithms, novel architectures, and sensitive configs leaking via training pipelines or logs.
    • Even "anonymized" code or documents can be re-identifiable given enough correlated context.

  • Model governance complexity:

    • Harder audits: Who had access to what? Which datasets trained which internal models?
    • Incident response gets messier when prompts and context are scattered across vendors.

The complacency is costly because it’s architectural. Many organizations are building AI capabilities on top of assumptions that would not survive a serious data governance review.

Design AI Like an Adversary Is Watching (Because One Might Be)

If you’re shipping AI products—or integrating them into your stack—this is the moment to reset your defaults.

Here are concrete technical and product principles that distinguish responsible AI apps from surveillanceware with a chat UI:

  1. Minimize by design

    • Collect only what is needed for the immediate operation of the feature.
    • Make long-term retention opt-in, not default.
    • Resist the reflexive "we might use this later for model improvement" mindset.

  2. Isolate sensitive contexts

    • For enterprise: strict tenant isolation in both inference and logging.
    • For IDE and productivity tools: explicit scoping of what can be read (per-project, per-folder, per-app), not carte blanche.

  3. Offer real training controls

    • Clear, front-and-center toggles: "Use my data to improve models: Yes/No" with honest consequences.
    • Separate system metrics (latency, error rates) from semantic content used for training.

  4. Cryptography and boundaries over vibes and trust

    • End-to-end encryption where feasible; at minimum, encrypt at rest and in transit with strong key management.
    • Treat third-party observability and analytics as part of your threat surface, not invisible plumbing.

  5. Log with intent

    • Redact secrets in prompts and responses before they hit logs.
    • Short retention windows by default, with configuration for stricter environments.
    • Keep an auditable map: which services see what classes of data, and why.

  6. Human-readable transparency

    • For a developer audience, publish a data flow diagram, not a marketing paragraph.
    • If your AI tool can see source code, say exactly how, where it’s processed, for how long, and under what legal terms.

These are not just ethical stance points; they are differentiators. In a market where AI capabilities commoditize fast, privacy posture becomes a feature serious customers will pay for.

The Strategic Opportunity in Saying the Quiet Part Out Loud

The HN commenter’s discomfort is a leading indicator of a reputational turn. Developers who once shrugged at over-collection are now:

  • Running internal security reviews of AI tools.
  • Asking vendors pointed questions about training data, tenancy, and retention.
  • Building internal, self-hosted models—even at higher cost—purely to regain control.

There is a wide-open lane for:

  • AI copilots that run on-device or at least on VPC-isolated infrastructure.
  • Vendor contracts that cryptographically and contractually guarantee no cross-tenant training.
  • Open source frameworks that make "private by default" the easiest implementation path.

If you’re building in this space, your competitive story in 2025 won’t just be "we’re smarter." It will be:

"We are predictably boring about your data. Here is the diagram. Here is the config. Here is the proof."

And if you’re integrating AI tools into your org, now is the right time to become, once again, a little less complacent and a lot more specific:

  • Which prompts are logged?
  • Where are they stored, and for how long?
  • Are they used for training? Of which models?
  • Who, exactly, can see them—now and in a breach scenario?

The rediscovery of "how much they are collecting" doesn’t have to end in cynicism. It can trigger a higher standard—set by the very engineers who once accepted surveillance as collateral damage of progress.

That standard is overdue.