# Set up tracing

A trace follows one operation, such as an HTTP request or background job. A span records a timed step within it. Parent and child spans show where the operation spent time and which step failed.

Leilani collects spans that your application exports with OpenTelemetry. Printing logs or serving HTTP requests does not create application traces automatically. There is no tracing switch in `leilani.json`.

## Export traces from Rust

Add an OpenTelemetry SDK, OTLP HTTP exporter, and instrumentation to each package you want to trace. This example uses the dependency versions below. Add them under `[dependencies]` in the package’s Cargo.toml:

```toml
opentelemetry = { version = "0.32", default-features = false, features = ["trace"] }
opentelemetry_sdk = { version = "0.32", default-features = false, features = ["trace"] }
opentelemetry-otlp = { version = "0.32", default-features = false, features = ["trace", "http-proto", "reqwest-blocking-client"] }
tracing = "0.1"
tracing-opentelemetry = { version = "0.33", default-features = false }
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
```

This small program initializes export, creates a parent operation and child span, and flushes before exiting. In your application, initialize once at startup, keep the provider alive while serving requests, and call `shutdown()` during graceful shutdown after active work finishes. Merge these layers into an existing subscriber if you already have one.

```rust
use opentelemetry::{trace::TracerProvider, KeyValue};
use opentelemetry_sdk::{trace::SdkTracerProvider, Resource};
use tracing_subscriber::{
    filter::Targets, layer::SubscriberExt, util::SubscriberInitExt, Layer,
};

fn init_telemetry() -> Result<SdkTracerProvider, Box<dyn std::error::Error + Send + Sync>> {
    let exporter = opentelemetry_otlp::SpanExporter::builder()
        .with_http()
        .build()?;
    let provider = SdkTracerProvider::builder()
        .with_resource(
            Resource::builder()
                .with_service_name(std::env::var("OTEL_SERVICE_NAME")
                    .unwrap_or_else(|_| env!("CARGO_PKG_NAME").into()))
                .with_attribute(KeyValue::new("leilani.package.name", env!("CARGO_PKG_NAME")))
                .build(),
        )
        .with_batch_exporter(exporter)
        .build();
    let tracer = provider.tracer(env!("CARGO_PKG_NAME"));
    tracing_subscriber::registry()
        .with(tracing_subscriber::fmt::layer().with_ansi(false).with_writer(std::io::stdout))
        .with(tracing_opentelemetry::layer().with_tracer(tracer)
            .with_filter(Targets::new().with_target(env!("CARGO_CRATE_NAME"), tracing::Level::TRACE)))
        .try_init()?;
    Ok(provider)
}

#[tracing::instrument(skip_all)]
fn process_order() {
    tracing::info!(order_id = 42, "Processing order");
    tracing::info_span!("check_inventory").in_scope(|| {
        // Perform the operation you want to measure here.
    });
}

fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let provider = init_telemetry()?;
    process_order();
    provider.shutdown()?;
    Ok(())
}
```

The export layer includes this crate’s spans and excludes exporter-internal HTTP calls. Add your own library crate targets to the filter when you instrument them. The example’s text formatter writes all log levels to stdout, so those log rows have Informational severity; trace errors use span status independently.

Instrument request handlers and useful child operations, such as database calls or outbound requests. `#[tracing::instrument(skip_all)]` also supports async functions; use `tracing::Instrument` to attach spans to futures, rather than holding a span-enter guard across an await. Record an OpenTelemetry error status when an operation fails, for example:

```rust
let span = tracing::info_span!(
    "payment",
    otel.status_code = "ERROR",
    otel.status_description = "Payment provider timed out"
);
span.in_scope(|| tracing::error!("Payment failed"));
```

Leilani supplies these defaults to deployed packages unless you override them:

```dotenv
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_SERVICE_NAME=<package name>
```

The trace endpoint is `http://127.0.0.1:4318/v1/traces` inside the workspace VM. It accepts OTLP/HTTP protobuf and OTLP JSON, including gzip. It accepts traces only; OTLP/gRPC on port 4317 is not available. Keep the base endpoint above for an exporter that appends `/v1/traces`; a signal-specific `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` must include the full path. Explicit exporter options can override these defaults.

Leilani also adds `leilani.package.name` to `OTEL_RESOURCE_ATTRIBUTES`. The example sets that attribute directly too. Configure `OTEL_SERVICE_NAME` in Environment if you want a service name different from the package name. Leilani attaches workspace and deployment identity; your exporter does not need a tenant header.

Trace delivery is best effort. The collector accepts at most 1 MiB per request before and after decompression; in-memory queue overflow, export failures, or shutdown can lose spans. A successful collector response acknowledges queuing, not durable storage.

For local development, run a local OTLP/HTTP collector or set the exporter endpoint to one you control. The deployed collector is not exposed at your public application hostname. After deploying instrumentation, send requests through your application and allow the exporter’s batch interval and indexing time to pass.
