AI Tracing Without Exporting Private Content

An AI trace can explain which model call failed, which tool was slow and where a retry occurred. It can also contain the user's question, retrieved documents, tool arguments and generated answer. Turning on tracing therefore creates a data flow that deserves its own configuration review.

Start with the failures you need to diagnose. Most routine performance questions need timings, operation identifiers, model versions, error categories and attempt counts. They do not automatically require complete message bodies.

Check the SDK actually in use

Different tracing systems have different defaults. The OpenTelemetry GenAI span conventions are marked Development and recommend that instruction, input and output capture be opt-in. This is guidance for instrumentation, not proof of your installed library's behaviour.

The OpenAI Agents Python tracing documentation says generation and function spans can include sensitive data, with RunConfig.trace_include_sensitive_data defaulting to true. It also documents a separate control for voice audio. Record the installed SDK version and explicitly set the intended behaviour rather than relying on a remembered default.

Inspect ordinary application logging as well. A trace configuration cannot prevent a wrapper from printing a full request when an exception occurs. Error handlers, debug middleware and tool implementations are common places for an otherwise restrained design to gain another payload copy.

Follow every exporter

Draw up a small destination inventory: the application process, local buffers, collector, tracing provider, log service and any external content store. For each destination, record who can read it, how long it retains data and how deletion is handled.

Pay particular attention to fan-out. The Agents SDK documentation distinguishes add_trace_processor(), which leaves the default exporter registered, from set_trace_processors(), which replaces the processor list. Adding your own processor does not establish that it is the only destination.

A redaction processor also needs a defined failure path. If another exporter can send the original event independently, a failed redaction step may leave that route untouched. Put mandatory filtering and delivery into a path whose failure prevents that payload from being exported. Keep a minimal error counter so the fault remains visible without including the rejected content.

Treat those SDK details as version-specific checks. Confirm them during dependency updates instead of carrying an old assumption into the next release.

Choose fields deliberately

Use an allowlist for routine operational telemetry. A useful starting record can contain a task identifier, operation type, start time, duration, model identifier, tool name, result category and retry count. Add business identifiers only when there is a clear diagnostic need and appropriate access control.

Avoid inserting private text into span names. A name such as retrieve_policy is easier to aggregate than one containing a user's question. Store prompt template versions rather than duplicating templates and filled-in values on every event.

OpenTelemetry's sensitive-data guidance discusses filtering and redaction processors and cautions that hashing predictable identifiers does not make them anonymous. A hash of a small set of customer numbers can remain linkable. Choose an internal surrogate where that meets the operational need, and still control access to the mapping.

If a difficult incident requires payload capture, make it a scoped, time-limited diagnostic mode. Define eligible users or synthetic test traffic, the authorised storage destination and expiry. Keep it separate from the default production configuration. The MCP audit logging note covers the broader action record; trace payloads need this additional handling.

Prove the route with synthetic markers

Use a test conversation containing unique fake markers in the user message, tool arguments, tool response and final answer. Exercise success, timeout and error paths. Search every configured telemetry destination for each marker.

The expected result depends on the policy: markers should be absent from routine traces, or present only in the explicitly approved diagnostic store. Check buffered exports and delayed delivery too. An empty live dashboard immediately after a request does not establish that nothing was queued.

Also force the filtering stage to fail in a test environment. Verify that the system records a safe operational error and does not send the unfiltered event through an alternative exporter. Keep the observed result with the pipeline configuration and SDK version.

Deploy the restrictive configuration to a small service first and confirm that ordinary incidents remain diagnosable. Keep the previous configuration for comparison, but do not use an accidental payload-export setting as the automatic rollback. The practical fallback is minimal telemetry while the new pipeline is repaired.

References

• OpenTelemetry, GenAI client span conventions. Accessed 17 October 2026.
• OpenAI, Agents Python SDK tracing. Accessed 17 October 2026.
• OpenTelemetry, Handling sensitive data. Accessed 17 October 2026.