An OpenTelemetry gotcha for anyone instrumenting AI agents

Fri, 17 Jul 2026 14:40:00 -0300

If you added OpenTelemetry to an AI agent this year, you probably followed the same path everyone follows. Add the ADOT distro. Run your entrypoint through opentelemetry-instrument. Point the exporter at CloudWatch. Ship it. And it works. Traces show up, token counts show up, latency shows up. Nothing broken.

But there’s a subtle question hiding in that setup that most people never ask, and getting it wrong is a real security bug. One that just cost the AWS Bedrock AgentCore Python SDK a CVE this week.

The design decision most people miss

OpenTelemetry has semantic conventions for generative AI (the gen_ai.* namespace). Version 1.41 of the spec makes a deliberate design decision that’s easy to miss: it splits the attributes an agent span can carry into two different categories.

Structural attributes are metadata about the call. Which model. Which provider. How many input tokens, how many output tokens, how long it took, which guardrail applied. These are safe to emit on every span by default. They tell you about performance and cost without exposing what the agent was working on.

Content attributes are the message bodies themselves. gen_ai.input.messages (the prompt). gen_ai.output.messages (the response). gen_ai.system_instructions (the system prompt). The spec says these must not be emitted by default. They have to be an explicit opt-in.

The reason for the split isn’t paranoia. If your agent is a customer support bot, the prompt is what your customer wrote in the chat window. If it’s a financial-analysis agent, the prompt might include the numbers being analyzed. If it’s a medical-summary agent, the prompt might have identifiable patient data. Whatever the domain, the message bodies are customer data. The token counts and latency numbers aren’t.

What happened this week

The AWS Bedrock AgentCore Python SDK, which is the library you use to build agents on Amazon Bedrock’s agent runtime, shipped two versions (1.4.8 and 1.5.0) whose OpenTelemetry instrumentation wrote the full user prompt and the full agent response into span attributes on every invocation, without checking the opt-in flag. Those spans then exported to the customer’s aws/spans CloudWatch log group.

The exposure was local: you needed IAM access to the account and logs:GetLogEvents on the span log group to read anything. Nothing on the open internet. But once inside, you saw plaintext prompts and plaintext responses. That’s CVE-2026-15737, disclosed on July 16, with a fixed release (bedrock-agentcore 1.5.1) available the same day.

I want to be clear about the reason for writing this post. It isn’t to pick on the AgentCore team. AWS published a clear advisory on the same day the fix shipped, told users exactly what to purge from CloudWatch, made the whole thing consumable through a public GitHub Security Advisory, and got the fix into pip within hours. That’s what a mature responsible-disclosure pipeline looks like. Being able to write a post like this one, from public sources, on the same day the CVE lands, is only possible because the transparency is there.

The reason I’m writing is that the same class of mistake is easy to make on every project that instruments an agent with OpenTelemetry, and the fix is a config choice, not a code choice.

The check I want to leave you with

Two questions.

Question 1: Do your CloudWatch log groups contain content attributes?

Run this in CloudWatch Logs Insights against your /aws/spans/* groups, or wherever your agent’s OTel exporter lands:

1
2
3
4


fields @timestamp, @message
| filter @message like /gen_ai\.input\.messages|gen_ai\.output\.messages|gen_ai\.prompt|gen_ai\.completion/
| sort @timestamp desc
| limit 100

If it returns rows, message bodies are landing in your logs. That’s either intentional (you opted in) or a design escape (something in your stack has content capture on by default). Either way, worth knowing.

Question 2: If content bodies are being captured, who can read them?

The default CloudWatchLogsReadOnlyAccess managed policy grants read on every log group in the account. If you’ve got engineers or contractors with broad read access on the observability side, they can also read every conversation your agent has had. That’s a call to make deliberately, not by accident.

For most agents in production, the right pattern is:

Keep content capture off by default.
Route structural spans (token counts, latency, model id) to your general observability pipeline.
If you need content for debugging, route it to a separate log group with a stricter KMS key, tighter IAM, and shorter retention.

The instrumentation-side switch is one environment variable. AGENT_OBSERVABILITY_ENABLED=true gets you structural coverage. Content capture only happens if a library explicitly opts in via its own flag. Traceloop calls it TRACELOOP_TRACE_CONTENT. Others vary. If you never set the content flag, and your SDK follows the v1.41 opt-in default, you’re safe by design.

Fix and audit checklist

If you’re building on Bedrock AgentCore:

Upgrade to bedrock-agentcore>=1.5.1. The current stable is 1.15.1 (June 25, 2026), so you’re 10 minor bumps behind if you’re on the affected releases.
Run the CWL Insights query above on your aws/spans log groups.
Purge whatever leaked (delete the affected log streams, or delete the log groups and let AgentCore recreate them).
Audit who has logs:GetLogEvents on those groups.
If you have forked or vendored versions of the SDK, patch them too.

If you’re building on any other agent framework (LangChain with OTel, Strands Agents, LlamaIndex with observability, your own custom thing), check what your instrumentation does with content by default. Not every library respects the v1.41 opt-in signal. The check takes ten minutes.

The design decision takes ten seconds. But it’s a lot easier to make the right choice on day one than to purge months of CloudWatch spans later.

See you around, Leo

Opentelemetry on Café Com Cloud

An OpenTelemetry gotcha for anyone instrumenting AI agents

The design decision most people miss

What happened this week

The check I want to leave you with

Fix and audit checklist