OpenTelemetry Support

This page is under active development. Specifications are not final and subject to change.

"Proof is in the progress" - Drake

This document details Sentry's work in integrating and supporting OpenTelemetry, the open standard for metrics, traces and logs. In particular, it focuses on the integration between Sentry's performance monitoring product and OpenTelemetry's tracing spec.

Background

When Sentry performance monitoring was initially introduced, OpenTelemetry was in early stages. This lead to us adopt a slightly different model from OpenTelemetry, notably we have this concept of transactions that OpenTelemetry does not have. We've described this, and some more historical background, in our performance monitoring research document.

Approach

TODO: Talk about the approach we are using, based on Matt's hackweek project - https://github.com/getsentry/sentry-ruby/pull/1876

Span Protocol

Below describe the transformations between an OpenTelemetry span and a Sentry Span. Related: the interface for a Sentry Span, the Relay spec for a Sentry Span and the spec for an OpenTelemetry span.

This is based on a mapping done as part of work on the OpenTelemetry Sentry Exporter.

OpenTelemetry Span	Sentry Span	Notes
trace_id	trace_id
span_id	span_id
parent_span_id	parent_span_id	If a span does not have a parent span ID, it is a root span. For a root span: If there is an active Sentry transaction, add it to the transaction If there is no active Sentry transaction, construct a new transaction from that span
name	description
name,attributes,kind	op
attributes,kind,status	tags	The OpenTelemetry Span Status message and span kind are set as tags on the Sentry span. Note, if you are constructing a transaction, DO NOT ADD TO TAGS.
attributes,status	status	See Span Status for more details
start_time_unix_nano	start_timestamp
end_time_unix_nano	timestamp
event		See Span Events for more details

Currently there is no spec for how Span.link in OpenTelemetry should appear in Sentry.

Span Status

In OpenTelemetry, Span Status is an enum of 3 values, while Sentry's Span Status is an enum of 17 values that map to the GRPC status codes. Each of the Sentry Span Status codes also map to HTTP codes. Sentry adopted it's Span Status spec from OpenTelemetry, who used the GRPC status code spec, but later on changed to the current spec it uses today.

To map from OpenTelemetry Span Status to, you need to rely on both OpenTelemetry Span Status and Span attributes. This approach was adapted from a PR by GH user @anguisa to the OpenTelemetry Sentry Exporter.

Copied
// OpenTelemetry span status can be Unset, Ok, Error. HTTP and Grpc codes contained in tags can make it more detailed.

// canonicalCodesHTTPMap maps some HTTP codes to Sentry's span statuses. See possible mapping in https://develop.sentry.dev/sdk/event-payloads/span/
var canonicalCodesHTTPMap = map[string]sentry.SpanStatus{
	"400": sentry.SpanStatusFailedPrecondition, // SpanStatusInvalidArgument, SpanStatusOutOfRange
	"401": sentry.SpanStatusUnauthenticated,
	"403": sentry.SpanStatusPermissionDenied,
	"404": sentry.SpanStatusNotFound,
	"409": sentry.SpanStatusAborted, // SpanStatusAlreadyExists
	"429": sentry.SpanStatusResourceExhausted,
	"499": sentry.SpanStatusCanceled,
	"500": sentry.SpanStatusInternalError, // SpanStatusDataLoss, SpanStatusUnknown
	"501": sentry.SpanStatusUnimplemented,
	"503": sentry.SpanStatusUnavailable,
	"504": sentry.SpanStatusDeadlineExceeded,
}

// canonicalCodesGrpcMap maps some GRPC codes to Sentry's span statuses. See description in grpc documentation.
var canonicalCodesGrpcMap = map[string]sentry.SpanStatus{
	"1":  sentry.SpanStatusCanceled,
	"2":  sentry.SpanStatusUnknown,
	"3":  sentry.SpanStatusInvalidArgument,
	"4":  sentry.SpanStatusDeadlineExceeded,
	"5":  sentry.SpanStatusNotFound,
	"6":  sentry.SpanStatusAlreadyExists,
	"7":  sentry.SpanStatusPermissionDenied,
	"8":  sentry.SpanStatusResourceExhausted,
	"9":  sentry.SpanStatusFailedPrecondition,
	"10": sentry.SpanStatusAborted,
	"11": sentry.SpanStatusOutOfRange,
	"12": sentry.SpanStatusUnimplemented,
	"13": sentry.SpanStatusInternalError,
	"14": sentry.SpanStatusUnavailable,
	"15": sentry.SpanStatusDataLoss,
	"16": sentry.SpanStatusUnauthenticated,
}

code := spanStatus.Code()
if code < 0 || int(code) > 2 {
    return sentry.SpanStatusUnknown, fmt.Sprintf("error code %d", code)
}
httpCode, foundHTTPCode := tags["http.status_code"]
grpcCode, foundGrpcCode := tags["rpc.grpc.status_code"]
var sentryStatus sentry.SpanStatus
switch {
case code == 1 || code == 0:
    sentryStatus = sentry.SpanStatusOK
case foundHTTPCode:
    httpStatus, foundHTTPStatus := canonicalCodesHTTPMap[httpCode]
    switch {
    case foundHTTPStatus:
        sentryStatus = httpStatus
    default:
        sentryStatus = sentry.SpanStatusUnknown
    }
case foundGrpcCode:
    grpcStatus, foundGrpcStatus := canonicalCodesGrpcMap[grpcCode]
    switch {
    case foundGrpcStatus:
        sentryStatus = grpcStatus
    default:
        sentryStatus = sentry.SpanStatusUnknown
    }
default:
    sentryStatus = sentry.SpanStatusUnknown
}
return sentryStatus

Span Events

OpenTelemetry, has the concept of Span Events. As per the spec:

An event is a human-readable message on a span that represents “something happening” during it’s lifetime

In Sentry, we have two options for how to treat span events. First, we can add them as breadcrumbs to the transaction the span belongs to. Second, we can create an artificial "point-in-time" span (a span with 0 duration), and add it to the span tree. TODO on what approach we take here.

In the special case that the span event is an exception span, where the name of the span event is exception, we also have the possibility of generating a Sentry error from an exception. In this case, we can create this exception based on the attributes of an event, which include the error message and stacktrace. This exception can also inherit all other attributes of the span event + span as tags on the event.

In the OpenTelemetry Sentry exporter, we've used this strategy to generate Sentry errors.

Transaction Protocol

There is no concept of a transaction within OpenTelemetry, so we rely on promoting spans to become transactions. The span description becomes the transaction name, and the span op becomes the transaction op. Therefore, OpenTelemetry spans must be mapped to Sentry spans before they can be promoted to become a transaction.

We should not set tags on transactions. Instead, we should populate the attributes field on the otel context, see below

OpenTelemetry Context

Aside from information from Spans and Transactions, OpenTelemetry has meta-level information about the SDK, resource, and service that generated spans. To track this information, we generate a new OpenTelemetry Event Context.

The existence of this context on an event (transaction or error) is how the Sentry backend will know that the incoming event is an OpenTelemetry event.

It uses the following schema:

Copied
type Primitive = number | string | boolean | bigint;

interface Attributes<T extends Primitive> {
    [key: string]: T | Array<T> | Record<string, T>;
}

interface OpenTelemetryContext {
    type?: "otel",

    // https://github.com/open-telemetry/opentelemetry-proto/blob/724e427879e3d2bae2edc0218fff06e37b9eb46e/opentelemetry/proto/trace/v1/trace.proto#L174-L186
    attributes?: Attributes;

    // https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/resource/semantic_conventions/README.md#service
    service?: {
        // from: service.name
        name: string;
        // from: service.namespace
        namespace?: string;
        // from: service.instance.id
        instance_id?: string;
        // from: service.version
        version?: string;
    };

    // https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/resource/semantic_conventions/README.md#telemetry-sdk
    otel_sdk?: {
        // from: telemetry.sdk.name
        name?: string;
        // from: telemetry.sdk.language
        language?: SDKLanguage;
        // from: telemetry.sdk.version
        version?: string;
        // from: telemetry.auto.version
        auto_version?: string;
    };

    // Rest of attribute keys excluding service and sdk keys
    // https://github.com/open-telemetry/opentelemetry-proto/blob/724e427879e3d2bae2edc0218fff06e37b9eb46e/opentelemetry/proto/resource/v1/resource.proto#L32
    [key: string]?: string,
}

The reason sdk and service are split are so they can be indexed as top level fields in the future for easier usage within Sentry.

SDK Spec

You can edit this page on GitHub.

Docs

General

Development

Application

Self-Hosted

Frontend

Backend

Services

SDK Development

Integrations

Resources

Meta Documentation