Meredith Whittaker, President of Signal and long-time critic of surveillance-driven technology, has raised urgent concerns about the rise of so-called 'agentic AI.' Her writings and talks argue that AI agents—systems that promise convenience by automating complex tasks—pose existential risks to privacy, security, and democratic freedoms. She’s been widely featured at SXSW and most recently the Economist. We want to summarize her views on this cutting edge concern and explore their implications when applied to observability and incident response in modern software systems.
The dominant tech narrative imagines AI agents as 'robot butlers'—systems capable of anticipating our needs, booking restaurants, managing schedules, or even initiating personal interactions. To deliver such convenience, these agents require near-total access to user data: browser history, credit-card information, private messages, location data, and more. In practice, this means granting 'root-level' permissions across devices and platforms to AI systems operated by a handful of powerful companies (see Whittaker's analysis in The Economist).
Whittaker warns that these agents blur the boundaries between applications and operating systems, effectively breaking what she calls the 'blood-brain barrier' of digital life. Sensitive data from encrypted apps like Signal could be swept into undifferentiated data contexts, undermining their guarantees. Already, research has shown that agents can be tricked into revealing sensitive information or manipulated into executing harmful actions.
One striking example was Apple’s rollout of 'Apple Intelligence,' which transmitted transcripts of WhatsApp messages to Apple servers—compromising WhatsApp’s end-to-end encryption by introducing Apple as another data endpoint.
Beyond privacy, Whittaker highlights competition risks. AI agents, integrated directly into operating systems by Apple, Google, and Microsoft, bypass traditional APIs to extract data directly from screens or interfaces. This allows platform owners to aggregate market insights from billions of users—advantages competitors cannot match. For independent apps and services, this represents both a loss of data sovereignty and a structural disadvantage.
We have reviewed Whittaker’s concerns as they apply directly to observability and site reliability engineering (SRE). We are seeing AI agents being widely deployed to investigate logs, traces, dashboards, and code, they are starting to dramatically accelerate root cause analysis and reduce mean time to resolution. The efficiency gains could drive near-universal adoption.
However, granting such agents unrestricted access to production telemetry is equivalent to granting access to the nervous system of modern infrastructure. If compromised, these systems could expose sensitive intellectual property, leak customer data, or trigger destructive actions if they misapply automated fixes.
Imagine an observability agent that misinterprets a log pattern and automatically issues a configuration change that cascades into an outage. Worse, adversaries could exploit these agents, tricking them into extracting secrets or executing malicious commands. In effect, the very tools meant to increase resilience could introduce systemic fragility.
Whittaker urges fundamental safeguards for agentic systems:
- Privacy as default: Developers must be able to designate 'sensitive zones'—applications or datasets off-limits to agents.
- Radical transparency: OS vendors should clearly disclose what data agents access, how it is used, and how it is secured.
- Security-first design: Investments must prioritize protecting private communications and critical systems, rather than discarding safeguards in pursuit of profitability.
Applied to observability, this means limiting AI agents to narrowly scoped datasets, requiring human-in-the-loop approval before fixes are applied, and ensuring independent auditability of agent actions. Without such measures, the drive for faster incident resolution could create systemic vulnerabilities in the very systems agents are tasked to protect.
Whittaker reminds us that privacy is not an abstract or sterile concern—it underpins intimacy, freedom of expression, and democratic accountability. Just as surveillance threatens civil society, surveillance-style AI agents threaten engineering ecosystems. By embedding themselves in the fabric of critical infrastructure, agentic systems may normalize unsafe practices and erode trust. Observability tools must resist this trajectory by centering transparency, user control, and verifiable protections.
Observability AI agents need a great deal of context.. To be useful, they require access to:
There are an array of processes to maintain PII anonymization, such as for Open Telemetry - an open-source observability framework for collecting, generating, and exporting telemetry data. Every major observability vendor has a process for this.
For example, explaining how to scrub sensitive data (like PII, authentication tokens, etc.) from OpenTelemetry logs, traces, and metrics using three processors in the OpenTelemetry Collector: Attributes Processor (for knowing which fields to delete/update/hash), Redaction Processor (for pattern-based masking or filtering, or enforcing allowlists), and Transform Processor (using OTTL for more complex or conditional redaction logic). It also emphasizes best practices: redact at the source when possible, use multiple layers of protection, test your rules, and balance data privacy with observability.
For code, to mitigate PII exposure, never hardcode secrets or sensitive values — store them in secure secrets managers (e.g., Vault, AWS Secrets Manager) — and enforce static analysis or pre-commit hooks to block accidental inclusion. Use anonymized or synthetic data for test fixtures instead of real user data.
These approaches are well established but to be sure this is not narrow, well-scoped access. It is “root” access to the telemetry and control plane of modern systems. Once granted, only the above PII practices separate operational data and sensitive data, the only boundary between performance debugging and potential security exposures.
If this data flows without these checks or if they are compromised to cloud servers for analysis, the attack surface grows. A misconfiguration or exploit could expose an entire company’s operational nervous system.
One of Whittaker’s most urgent points is that AI agents can be manipulated. They may be tricked into revealing sensitive data or taking harmful actions. In observability, the equivalent risk is serious:
In other words, the very features that make observability agents powerful — autonomy, breadth of access, speed — also make them uniquely dangerous when compromised.
Transparency is a core principle of observability: engineers must see what’s happening inside complex systems. Yet if AI agents become the primary lens on that data, transparency risks giving way to automation.
A generation of engineers may grow accustomed to accepting the agent’s diagnosis without questioning it. Root cause analysis may be tacitly agreed with and fixes may be deployed because “the AI said so.” Over time, investigative skills could erode. And when the AI is wrong — because it missed a subtle signal or weighted evidence poorly — the result could be a longer, more damaging outage.
According to Charity Majors, CTO of honeycomb.io - an observability platform for software engineering teams, AI generated code without strong production feedback loops and observability, may actually sabotage ROI by increasing maintenance overhead. While the black box problem is only exacerbated because this makes debugging harder - engineers can’t always reproduce a failure or trace exactly what the AI decided.
This is the paradox: the very tools designed to strengthen reliability, including those that use AI to augment software development may create new forms of fragility. This is why we see doubling down on transparency of agentic AI observability as crucial.
If current trends continue, the observability world might soon face:
These are not distant hypotheticals. They are foreseeable outcomes if we rush toward agentic observability without safeguards.
The good news: Whittaker’s critique also points to solutions. We think every AI SRE or AI powered debugging platform should embody these values as we have at Relvy. We can and design observability agents for this:
These are not optional enhancements. They are minimum requirements if we want observability AI to strengthen, rather than undermine, reliability.
Observability AI SRE and AI powered debugging agents can make a real difference. They can cut through data noise, speed investigations, and help engineers recover from incidents faster. But the stakes are high. Reliability isn’t just about convenience — it’s about trust, security, and resilience.
If we embed AI agents into our observability stacks without transparency and control, we risk creating brittle, centralized systems that are vulnerable to error and abuse. If we build them carefully, with privacy, competition, and human oversight in mind, they could become true partners in reliability.
The path we choose now will shape whether AI in observability fulfills its promise — or repeats the mistakes Meredith Whittaker warns against.
At Relvy, we believe engineers—not black-box AI—should stay in control. Incidents are high-stakes, so every step must be transparent, explainable, and guided by the team on call.
We’re building AI-powered debugging that eliminates manual, time consuming investigations - cutting resolution time while keeping humans in the loop. With Relvy, teams reduce failures, improve uptime, and trust their tools as much as their teammates.
Our mission: make reliability faster, smarter, and human-centered.
