Set up tracing
A trace follows one operation, such as an HTTP request or background job. A span records a timed step within it. Parent and child spans show where the operation spent time and which step failed.
Leilani collects spans that your application exports with OpenTelemetry. Printing logs or serving HTTP requests does not create application traces automatically. There is no tracing switch in leilani.json.
Export traces from Rust
Add an OpenTelemetry SDK, OTLP HTTP exporter, and instrumentation to each package you want to trace. This example uses the dependency versions below. Add them under [dependencies] in the package’s Cargo.toml:
opentelemetry = { version = "0.32", default-features = false, features = ["trace"] }
opentelemetry_sdk = { version = "0.32", default-features = false, features = ["trace"] }
opentelemetry-otlp = { version = "0.32", default-features = false, features = ["trace", "http-proto", "reqwest-blocking-client"] }
tracing = "0.1"
tracing-opentelemetry = { version = "0.33", default-features = false }
tracing-subscriber = { version = "0.3", features = ["env-filter"] }This small program initializes export, creates a parent operation and child span, and flushes before exiting. In your application, initialize once at startup, keep the provider alive while serving requests, and call shutdown() during graceful shutdown after active work finishes. Merge these layers into an existing subscriber if you already have one.
use opentelemetry::{trace::TracerProvider, KeyValue};
use opentelemetry_sdk::{trace::SdkTracerProvider, Resource};
use tracing_subscriber::{
filter::Targets, layer::SubscriberExt, util::SubscriberInitExt, Layer,
};
fn init_telemetry() -> Result<SdkTracerProvider, Box<dyn std::error::Error + Send + Sync>> {
let exporter = opentelemetry_otlp::SpanExporter::builder()
.with_http()
.build()?;
let provider = SdkTracerProvider::builder()
.with_resource(
Resource::builder()
.with_service_name(std::env::var("OTEL_SERVICE_NAME")
.unwrap_or_else(|_| env!("CARGO_PKG_NAME").into()))
.with_attribute(KeyValue::new("leilani.package.name", env!("CARGO_PKG_NAME")))
.build(),
)
.with_batch_exporter(exporter)
.build();
let tracer = provider.tracer(env!("CARGO_PKG_NAME"));
tracing_subscriber::registry()
.with(tracing_subscriber::fmt::layer().with_ansi(false).with_writer(std::io::stdout))
.with(tracing_opentelemetry::layer().with_tracer(tracer)
.with_filter(Targets::new().with_target(env!("CARGO_CRATE_NAME"), tracing::Level::TRACE)))
.try_init()?;
Ok(provider)
}
#[tracing::instrument(skip_all)]
fn process_order() {
tracing::info!(order_id = 42, "Processing order");
tracing::info_span!("check_inventory").in_scope(|| {
// Perform the operation you want to measure here.
});
}
fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let provider = init_telemetry()?;
process_order();
provider.shutdown()?;
Ok(())
}The export layer includes this crate’s spans and excludes exporter-internal HTTP calls. Add your own library crate targets to the filter when you instrument them. The example’s text formatter writes all log levels to stdout, so those log rows have Informational severity; trace errors use span status independently.
Instrument request handlers and useful child operations, such as database calls or outbound requests. #[tracing::instrument(skip_all)] also supports async functions; use tracing::Instrument to attach spans to futures, rather than holding a span-enter guard across an await. Record an OpenTelemetry error status when an operation fails, for example:
let span = tracing::info_span!(
"payment",
otel.status_code = "ERROR",
otel.status_description = "Payment provider timed out"
);
span.in_scope(|| tracing::error!("Payment failed"));Leilani supplies these defaults to deployed packages unless you override them:
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_SERVICE_NAME=<package name>The trace endpoint is http://127.0.0.1:4318/v1/traces inside the workspace VM. It accepts OTLP/HTTP protobuf and OTLP JSON, including gzip. It accepts traces only; OTLP/gRPC on port 4317 is not available. Keep the base endpoint above for an exporter that appends /v1/traces; a signal-specific OTEL_EXPORTER_OTLP_TRACES_ENDPOINT must include the full path. Explicit exporter options can override these defaults.
Leilani also adds leilani.package.name to OTEL_RESOURCE_ATTRIBUTES. The example sets that attribute directly too. Configure OTEL_SERVICE_NAME in Environment if you want a service name different from the package name. Leilani attaches workspace and deployment identity; your exporter does not need a tenant header.
Trace delivery is best effort. The collector accepts at most 1 MiB per request before and after decompression; in-memory queue overflow, export failures, or shutdown can lose spans. A successful collector response acknowledges queuing, not durable storage.
For local development, run a local OTLP/HTTP collector or set the exporter endpoint to one you control. The deployed collector is not exposed at your public application hostname. After deploying instrumentation, send requests through your application and allow the exporter’s batch interval and indexing time to pass.