Documentation

How Leilani works

Leilani builds and hosts Rust applications. Push your source code with Git, and use the browser console to inspect builds, read logs, and manage configuration.

A Leilani workspace is named after a Cargo workspace and mirrors its organization: one source tree can contain multiple Rust packages that are built and deployed together. It holds their source, deployment, and shared environment variables, with package-specific overrides where needed.

The leilani.json file in your project tells Leilani which packages to start and where to send HTTP requests. Leilani builds the code and starts the configured packages in a virtual machine.

Before you start, have Git, Rust, and Cargo installed, and a Rust HTTP application that runs locally. Keep the Cargo workspace’s root Cargo.toml at the root of your Git project. A single-package project works too.

1. Create a workspace

1. Open Leilani, enter your email, and follow the sign-in link. The link expires after five minutes.
2. On your first sign-in, create a password of at least 12 characters. Keep it for Git authentication and unlocking encrypted workspace files such as .env.
3. Open Deploy in your workspace. Copy the workspace ID and the Git remote shown there.

Deploying and running an application requires an active trial or paid subscription. Check your access and manage your subscription in Profile.

2. Configure your app

Create leilani.json beside Cargo.toml. For an application named my-app listening on port 3000, start with:

{
  "packages": {
    "my-app": {
      "args": []
    }
  },
  "http": {
    "binds": {
      "WORKSPACE-ID.aloha.leilani.dev": 3000
    }
  }
}

• Replace my-app with your Cargo package name. Use the default binary name matching that package; see Packages for binary naming requirements.
• Replace WORKSPACE-ID with your workspace ID. This hostname is your application's public address.
• Make your application listen on 0.0.0.0:3000, or change both the application's listening port and the port in http.binds to match. The configuration maps requests to a port; it does not set your application's listener.

args supplies command-line arguments to the package. This minimal configuration adds no access-control rules, so the application is public. Keep secrets out of source control.

3. Deploy with Git

Commit your application source, Cargo.toml, Cargo.lock, and leilani.json to your local main branch. Run these commands from the project directory, replacing WORKSPACE-ID:

git remote add leilani https://leilani.dev/git/WORKSPACE-ID
git push leilani main

When Git requests credentials, use the Git username shown on Deploy and the password you created in Leilani. Add the remote once; use the push command for later deployments.

A push that updates main triggers a build. Watch the output in your terminal and check the status on Deploy. Once the application is running, open https://WORKSPACE-ID.aloha.leilani.dev.

Leilani builds all Cargo workspace members in release mode with a locked dependency graph. Commit an up-to-date Cargo.lock; a build that needs to change it will fail.

To deploy changes, commit them and push again. Pushing another branch does not deploy it.

Environment variables

After deploying, open Environment to add or remove variables. Choose a shared workspace value or a package-specific value. A package value overrides a shared value with the same key.

Use the plain value option when your application needs the original value, such as a database URL. Select Rebuild & restart on Deploy to apply changes to running packages.

Manage your application

Deploy

Find Git connection details, inspect the latest build, and use Rebuild & restart to compile the latest pushed source without a new push.

Logs

Read application output, build events, and HTTP events. See Logs for collection behavior and search examples.

Traces

Find slow or failing operations and inspect their spans. Start with tracing setup, then explore traces.

Config

Inspect the parsed configuration. This view is read-only. Edit leilani.json locally, commit it, and push to change package arguments, HTTP routing, or access-control rules.

The top-bar status shows whether the application is running. Having deployed configuration does not by itself mean the application is healthy.

Logs

Logs contains package output, build messages, package start and stop events, and HTTP request and response events recorded by Leilani’s proxy. Everything is scoped to your signed-in workspace. No logging SDK or change to leilani.json is required.

Write application logs

Write newline-terminated messages to stdout or stderr. Each nonblank line becomes a PackageLog entry labeled with its package name. Stdout receives Informational severity; stderr receives Error. The collector does not infer severity from words or JSON fields inside your message.

println!("order_id=42 action=created");
eprintln!("order_id=42 action=payment_failed reason=timeout");

A logging library works too, provided its output goes to stdout or stderr. If your formatter sends every level to stderr, every line will be labeled Error in Leilani. Configure its writer deliberately. Files written by your application are not automatically collected. Blank lines are skipped; multi-line messages appear as separate entries. Avoid putting passwords, tokens, or request bodies in logs.

Read and refresh logs

Open Logs. The table shows UTC time, severity, package, event kind, and message, newest first. A query returns up to 200 entries from the last 24 hours, across deployments in this workspace. This is the console’s search window, not a promise of retention. Narrow your query if recent traffic hides the event you need.

Auto-refresh defaults to every two seconds. Choose another interval or Refresh off while investigating. Refresh pauses when the tab is hidden, you scroll down to older rows, or you select text. Return to the top and clear the selection to resume. Drag column dividers to resize them, or focus a divider and use the arrow keys.

Search with LogQL

Enter a log query in Search (LogQL) and select Run query. Start with a stream selector in braces, then add filters. = selects an exact label value, =~ matches a label with a regular expression, |= keeps lines containing text, and != excludes lines containing text. Names and exact matches are case-sensitive. Use log queries here; metric aggregations are not displayed as log rows.

All events (the default)

{kind=~".+"}

Output from one package

{kind="PackageLog", package="api"}

Errors written by a package

{kind="PackageLog", package="api", severity="Error"}

Find a message and exclude noise

{kind="PackageLog"} |= "order_id=42" != "healthcheck"

Match text without case sensitivity

{kind="PackageLog"} |~ "(?i)(timeout|connection refused)"

Requests to a path

{kind="HttpRequest"} | json | path="/orders"

Server-error responses

{kind="HttpResponse"} | json | status_code=~"5[0-9][0-9].*"

Package starts and stops

{kind=~"PackageStarted|PackageStopped"}

Build output

{kind=~"CompileMessage|CargoOutputChunk"}

Replace api, paths, and search text with your own values. The package stream label belongs to PackageLog entries; lifecycle events instead carry package data in their JSON message. HTTP events carry fields such as request_id, path, method, and status_code. Use | json to extract fields from those events. Application output is stored as a JSON string, so its embedded fields are not automatically indexed as labels.

For a failing request, find its HttpResponse entry, copy its request_id, then search both HTTP event kinds:

{kind=~"HttpRequest|HttpResponse"} | json | request_id="REQUEST-ID"

To restrict events to a known build or deployment, filter the attached metadata after the selector, for example {kind=~".+"} | build_id="BUILD-ID" or {kind=~".+"} | deployment_id="DEPLOYMENT-ID". Use an actual identifier from your deployment; these are not stream-selector labels.

If a search returns nothing, select Reset, confirm the package ran and emitted a complete line within the last 24 hours, then add filters one at a time. A query error is different from an empty result: check syntax and label capitalization, or retry if log storage is temporarily unavailable.

Set up tracing

A trace follows one operation, such as an HTTP request or background job. A span records a timed step within it. Parent and child spans show where the operation spent time and which step failed.

Leilani collects spans that your application exports with OpenTelemetry. Printing logs or serving HTTP requests does not create application traces automatically. There is no tracing switch in leilani.json.

Export traces from Rust

Add an OpenTelemetry SDK, OTLP HTTP exporter, and instrumentation to each package you want to trace. The following versions match Leilani’s example application. Add them under [dependencies] in the package’s Cargo.toml:

opentelemetry = { version = "0.32", default-features = false, features = ["trace"] }
opentelemetry_sdk = { version = "0.32", default-features = false, features = ["trace"] }
opentelemetry-otlp = { version = "0.32", default-features = false, features = ["trace", "http-proto", "reqwest-blocking-client"] }
tracing = "0.1"
tracing-opentelemetry = { version = "0.33", default-features = false }
tracing-subscriber = { version = "0.3", features = ["env-filter"] }

This small program initializes export, creates a parent operation and child span, and flushes before exiting. In your application, initialize once at startup, keep the provider alive while serving requests, and call shutdown() during graceful shutdown after active work finishes. Merge these layers into an existing subscriber if you already have one.

use opentelemetry::{trace::TracerProvider, KeyValue};
use opentelemetry_sdk::{trace::SdkTracerProvider, Resource};
use tracing_subscriber::{
    filter::Targets, layer::SubscriberExt, util::SubscriberInitExt, Layer,
};

fn init_telemetry() -> Result<SdkTracerProvider, Box<dyn std::error::Error + Send + Sync>> {
    let exporter = opentelemetry_otlp::SpanExporter::builder()
        .with_http()
        .build()?;
    let provider = SdkTracerProvider::builder()
        .with_resource(
            Resource::builder()
                .with_service_name(std::env::var("OTEL_SERVICE_NAME")
                    .unwrap_or_else(|_| env!("CARGO_PKG_NAME").into()))
                .with_attribute(KeyValue::new("leilani.package.name", env!("CARGO_PKG_NAME")))
                .build(),
        )
        .with_batch_exporter(exporter)
        .build();
    let tracer = provider.tracer(env!("CARGO_PKG_NAME"));
    tracing_subscriber::registry()
        .with(tracing_subscriber::fmt::layer().with_ansi(false).with_writer(std::io::stdout))
        .with(tracing_opentelemetry::layer().with_tracer(tracer)
            .with_filter(Targets::new().with_target(env!("CARGO_CRATE_NAME"), tracing::Level::TRACE)))
        .try_init()?;
    Ok(provider)
}

#[tracing::instrument(skip_all)]
fn process_order() {
    tracing::info!(order_id = 42, "Processing order");
    tracing::info_span!("check_inventory").in_scope(|| {
        // Perform the operation you want to measure here.
    });
}

fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    let provider = init_telemetry()?;
    process_order();
    provider.shutdown()?;
    Ok(())
}

The export layer includes this crate’s spans and excludes exporter-internal HTTP calls. Add your own library crate targets to the filter when you instrument them. The example’s text formatter writes all log levels to stdout, so those log rows have Informational severity; trace errors use span status independently.

Instrument request handlers and useful child operations, such as database calls or outbound requests. #[tracing::instrument(skip_all)] also supports async functions; use tracing::Instrument to attach spans to futures, rather than holding a span-enter guard across an await. Record an OpenTelemetry error status when an operation fails, for example:

let span = tracing::info_span!(
    "payment",
    otel.status_code = "ERROR",
    otel.status_description = "Payment provider timed out"
);
span.in_scope(|| tracing::error!("Payment failed"));

Leilani supplies these defaults to deployed packages unless you override them:

OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_SERVICE_NAME=<package name>

The trace endpoint is http://127.0.0.1:4318/v1/traces inside the workspace VM. It accepts OTLP/HTTP protobuf and OTLP JSON, including gzip. It accepts traces only; OTLP/gRPC on port 4317 is not available. Keep the base endpoint above for an exporter that appends /v1/traces; a signal-specific OTEL_EXPORTER_OTLP_TRACES_ENDPOINT must include the full path. Explicit exporter options can override these defaults.

Leilani also adds leilani.package.name to OTEL_RESOURCE_ATTRIBUTES. The example sets that attribute directly too. Configure OTEL_SERVICE_NAME in Environment if you want a service name different from the package name. Leilani attaches workspace and deployment identity; your exporter does not need a tenant header.

For local development, run a local OTLP/HTTP collector or set the exporter endpoint to one you control. The deployed collector is not exposed at your public application hostname. After deploying instrumentation, send requests through your application and allow the exporter’s batch interval and indexing time to pass.

Explore traces

Open Traces after your application has exported spans. JavaScript must be enabled. Use these controls to find an operation:

• Time range: last 15 minutes, last hour (the default), or last 24 hours.
• Search traces: case-insensitive text in operation, service, or package names, or a complete 32-character hexadecimal trace ID. This is a text search, not a LogQL or TraceQL editor; arbitrary attributes are not searched here.
• Package / Service: narrow the result to an instrumented package or service name. Choices come from received trace data in the selected time range.
• Status: Error finds traces containing an error span. OK means no error span was reported; it does not prove the HTTP response succeeded. Set span status in your instrumentation.
• Min. duration: enter milliseconds, such as 500, to find traces lasting at least half a second.

Filters update the results automatically. Select Refresh to fetch again, or Clear filters to start over. Auto-refresh runs every two seconds while you are viewing the latest results; it pauses while the tab is hidden, you select text, read older rows, or inspect a trace. Uncheck Auto-refresh to keep results steady.

For a slow-request investigation, choose a package, set Min. duration to 500, and choose Longest duration under Sort by. To investigate failures, clear the duration threshold and choose Error. Searches return at most 100 traces. Sorting, the duration chart, error rate, median, and p95 describe the returned results, not all workspace traffic. Narrow the time range or filters when the limit notice appears.

Inspect a trace

Select an operation in the table or a point on the duration chart. The detail dialog shows the full trace ID, total duration, services, and a span waterfall. Bars show span start offsets and durations relative to the trace’s start. Select a span to inspect its attributes and timestamped events. Look for long spans and error spans; concurrent spans overlap, so their durations do not add up to the total.

Copy the trace ID from the dialog to share it with someone who can access the workspace. A direct link has the form /aloha/traces?trace=TRACE-ID. It opens that trace directly rather than searching the selected time window. The trace must still exist in this workspace. Press Escape or use Close trace details to return to the search.

Connect traces and logs

Include the current trace ID in your application’s logs so you can find the same operation in both views. With the subscriber above, log it while inside an instrumented span:

use opentelemetry::trace::TraceContextExt;
use tracing_opentelemetry::OpenTelemetrySpanExt;

let context = tracing::Span::current().context();
let trace_id = context.span().span_context().trace_id().to_string();
tracing::info!(%trace_id, "Order processed");

Copy the full ID from Traces and search Logs with {kind="PackageLog"} |= "TRACE-ID", replacing the placeholder. You can also copy an ID from a log into Search traces or the direct-link URL.

The proxy’s X-Leilani-Request-ID correlates HttpRequest and HttpResponse log entries. It is separate from an OpenTelemetry trace ID. Log both in your application if you need to connect proxy events to spans. Returning an x-trace-id response header is something your application can implement; Leilani does not add one automatically.

For traces spanning multiple applications, configure W3C Trace Context propagation: extract incoming traceparent and tracestate into the request span’s parent context, and inject the current context into outbound calls. Without propagation, each application can create a separate trace for the same request. Each Leilani workspace can only view spans exported into its own workspace.

When traces are missing

• Confirm the instrumented code ran, its spans finished, and your subscriber filter and sampler allow those spans.
• Check the exporter protocol and endpoint, including signal-specific overrides. Sending a request to a static file served by Leilani never reaches your application and creates no application span.
• Check Logs for exporter errors. Keep the provider alive, allow time for batches to arrive, and flush on shutdown.
• Clear package, service, status, and duration filters; choose a wider time range. If you know the full trace ID, try the direct link while search indexing catches up.
• Recent traces can be incomplete while more spans arrive. Close and reopen the detail view after another batch is exported. An unavailable-backend error is not an empty search result; retry once storage is available.

leilani.json reference

Place one leilani.json at the Cargo workspace root, alongside its Cargo.toml. It configures the whole Leilani workspace. Use valid JSON, without comments or trailing commas. Commit changes and push to deploy them; Config shows the parsed configuration and is read-only.

All four top-level fields are optional. An empty configuration starts no packages and defines no HTTP listeners. There is no separate domains field: hostnames belong in http.binds.

Packages & restarts

Cargo determines workspace membership through Cargo.toml. Leilani builds all members, then starts only the entries in packages. For example, a workspace can contain an API, a worker, and a shared library; list the API and worker here, and leave the library out.

Use a package name whose binary target has the same name. Startup resolves each key against built executable target names, while the console lists Cargo package names. If you customize [[bin]].name, keep these names aligned. Executable names must be unique across the workspace.

Each package is supervised independently. It runs from the source root, receives shared environment variables plus its own overrides, and writes stdout and stderr to Logs. There is no configured startup dependency order; packages that depend on one another should tolerate the other process not being ready yet.

{
  "packages": {
    "api": {
      "args": ["--port", "3000"],
      "restart": { "policy": "on-failure", "max_restarts": 5, "delay_secs": 1 }
    },
    "worker": {
      "args": [],
      "restart": { "policy": "always", "max_restarts": 10, "delay_secs": 3 }
    }
  }
}

The API must implement the example’s --port argument. Leilani does not add command-line options to your application. Omit restart, or any of its fields, to use the defaults; unknown fields inside restart are rejected.

HTTP & static files

Bind your application to the configured port, for example 0.0.0.0:3000. Leilani terminates public HTTPS and forwards HTTP to that listener. A binding does not set the application’s listening address, select a package, or set a PORT environment variable.

Hostnames contain no scheme, port, or path. Use app.example.com, not a URL. Names are case-insensitive and trailing dots are normalized. IP addresses and wildcard hostnames are not supported.

With static_dir: "./public", a request for /styles.css serves public/styles.css if that file exists. Missing files and directory paths fall through to the application: there is no automatic index.html or single-page application fallback. Access rules run before static-file serving. Include the intended public files in your pushed source, and keep secrets outside this directory.

Custom domains

The built-in hostname is WORKSPACE-ID.aloha.leilani.dev. To add a custom hostname:

1. Add the hostname and its destination port to http.binds.
2. At your DNS provider, create a CNAME record pointing that hostname to your workspace’s built-in hostname.
3. Commit and push the configuration. Allow time for DNS propagation and certificate issuance, then visit the custom hostname over HTTPS.

app.example.com.  CNAME  WORKSPACE-ID.aloha.leilani.dev.

Both DNS and configuration are required: the current CNAME chain selects the workspace, and that workspace must explicitly bind the custom hostname. Pointing only an A or AAAA record at Leilani is insufficient. Use a subdomain if your provider flattens apex aliases into address records; Leilani needs a visible CNAME chain.

Leilani obtains and renews certificates automatically using Let’s Encrypt HTTP-01 validation. DNS must resolve to Leilani, and public HTTP on port 80 must reach its challenge handler. A DNS proxy or redirect that intercepts the challenge can prevent issuance. HTTPS becomes available once the certificate is issued.

Add each hostname separately, including www if needed. To protect the same paths on multiple hostnames, include all of them in the relevant rule’s hosts array.

Application users

users defines credentials for visitors to your deployed application. These accounts are separate from workspace sign-in and Git credentials. Defining users alone does not protect any route; add matching access rules.

To configure a password without committing it, open Environment, add ADMIN_PASSWORD_HASH, choose All packages, enter the intended password, and select Argon2id as the transform. Reference the stored hash with "password": "${ADMIN_PASSWORD_HASH}". Set it before enabling the rule, then deploy or rebuild. The user logs in with the original password.

Use unique usernames. Password hashes are omitted from the Config view. For a request authenticated by a rule, Leilani removes the Basic Authorization header and forwards X-Leilani-User-ID containing the normalized username and X-Leilani-User-Roles containing a JSON array of roles. Incoming values for these identity headers are removed before authentication.

Access rules

Each rule applies to an exact hostname, a listed HTTP method, and a path matching its regular expression. A matching request must supply valid HTTP Basic credentials for a user with at least one of the rule’s roles. Missing credentials, an incorrect password, or no allowed role returns HTTP 401 with a Basic authentication challenge.

Unmatched requests are public. A rule for GET does not protect POST. A rule for a custom hostname does not protect the built-in hostname unless both are listed. To require authentication across a hostname, use ^/.*$ and list every HTTP method your application accepts.

Use Rust regex syntax for paths, with JSON escaping: a literal dot in a pattern is written \\. in JSON. Look-around and backreferences are not supported.

Keep rules from overlapping. Evaluation continues after a successful match; priority does not make a rule override later rules. Authentication consumes the Authorization header, so a second matching rule cannot reuse those credentials and returns 401. Combine alternative roles and hostnames into one rule, or use disjoint path and method matches. Rules with the same priority and host set are treated as duplicates, even if their paths differ.

Variable substitution

String values in leilani.json can reference environment variables using ${NAME}. For example, "args": ["--database", "${DATABASE_URL}"] passes a stored connection string as an argument.

• Values inside a package’s configuration use shared variables plus that package’s overrides. Other fields, including users, rules, and HTTP settings, use shared workspace variables only.
• Substitution applies to string values, not object keys. You can substitute a hostname in a rule’s hosts array, but not a hostname key in http.binds.
• The first placeholder in each string is replaced once. Expansion is not recursive; additional placeholders remain unchanged. Use one placeholder per string.
• A missing variable leaves the placeholder unchanged. Define every referenced variable before relying on the configuration.
• Substitution keeps the value a string. Numeric fields such as ports, body limits, and restart counts must be JSON numbers, not placeholders.

Variables configured through Environment are supplied to launched packages; they are not passed to the Cargo build process. There is no environment-variable map in leilani.json. Rebuild & restart after changing variables to apply them to running packages. Config may show expanded values, so avoid placing secrets in ordinary configuration strings when your application can read them directly from its environment.

Complete example

This configuration starts an API and a worker from one Cargo workspace. The API must listen on port 3000. Replace WORKSPACE-ID and app.example.com, configure the CNAME, and create the shared ADMIN_PASSWORD_HASH variable as described above. The two hostnames serve the same API; the worker needs no HTTP binding.

{
  "packages": {
    "api": {
      "args": [],
      "restart": {
        "policy": "on-failure",
        "max_restarts": 5,
        "delay_secs": 1
      }
    },
    "worker": {
      "args": [],
      "restart": {
        "policy": "always",
        "max_restarts": 10,
        "delay_secs": 3
      }
    }
  },
  "http": {
    "binds": {
      "WORKSPACE-ID.aloha.leilani.dev": 3000,
      "app.example.com": 3000
    },
    "max_body_size": 1048576,
    "static_dir": "./public"
  },
  "users": [
    {
      "username": "admin",
      "password": "${ADMIN_PASSWORD_HASH}",
      "roles": ["admin"],
      "xref_id": "customer-123"
    }
  ],
  "rules": [
    {
      "priority": 0,
      "hosts": [
        "WORKSPACE-ID.aloha.leilani.dev",
        "app.example.com"
      ],
      "methods": ["GET", "HEAD", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"],
      "path": "^/admin(?:/.*)?$",
      "roles": ["admin"]
    }
  ]
}

The listed methods on /admin and its child paths require the admin role on both hostnames, including matching static files. Other paths and methods remain public at the Leilani proxy. Extend the method list if your application accepts additional methods.