How to Stream OTLP to Local DuckLake

Use the duckdb-otlp Docker image in local-ducklake mode to stream OTLP/HTTP exports into a local DuckLake lakehouse.

The container initializes DuckDB, loads the required extensions, attaches DuckLake, starts the ingest server, and commits accepted rows in batches.

Live ingestion uses OTLP/HTTP on port 4318. WASM builds do not include the ingest server.

Configure

Create .env:

DUCKDB_MODE=local-ducklake
DUCKDB_OTLP_TOKEN=dev-token-123456

DUCKLAKE_NAME=lake
DUCKLAKE_CATALOG_PATH=/data/ducklake/catalog.duckdb
DUCKLAKE_DATA_PATH=/data/ducklake/storage

DUCKDB_QUACK_ENABLED=1
DUCKDB_QUACK_ADDR=0.0.0.0:9494
DUCKDB_QUACK_TOKEN=dev-quack-token-123456

DUCKLAKE_CATALOG_PATH stores DuckLake metadata. DUCKLAKE_DATA_PATH stores Parquet data files.

Start the server

mkdir -p data

docker run --rm --name duckdb-otlp \
  --env-file .env \
  -p 4318:4318 \
  -p 9494:9494 \
  -v "$(pwd)/data:/data" \
  ghcr.io/smithclay/duckdb-otlp:latest

The container creates the target tables in lake.main if they do not exist:

otlp_logs
otlp_traces
otlp_metrics_gauge
otlp_metrics_sum
otlp_metrics_histogram
otlp_metrics_exp_histogram

Leave the container running while clients send OTLP/HTTP requests.

POST a log record

In another terminal:

curl -sS http://localhost:4318/v1/logs \
  -H 'Authorization: Bearer dev-token-123456' \
  -H 'Content-Type: application/json' \
  -d '{"resourceLogs":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"local-ducklake-demo"}},{"key":"deployment.environment","value":{"stringValue":"docs"}}]},"scopeLogs":[{"scope":{"name":"duckdb-otlp-guide"},"logRecords":[{"timeUnixNano":"1704067200000000000","observedTimeUnixNano":"1704067200123456789","severityNumber":9,"severityText":"INFO","body":{"stringValue":"hello from local DuckLake"},"attributes":[{"key":"guide","value":{"stringValue":"stream-to-local-ducklake"}}]}]}]}]}'

Response:

{"status":"buffered","rows":1,"batches":1}

Rows are accepted before they are durable. They commit automatically in the background, on graceful shutdown, or immediately after an explicit flush.

Query committed rows

Flush and query through Quack from a host DuckDB process. The server process owns the DuckLake catalog while it runs, and the distroless image has no shell or bundled DuckDB CLI.

The server image is distroless and has no shell or DuckDB CLI, so do not use docker exec ... sh -c for inspection SQL. The examples in this guide enable Quack and publish port 9494 for this purpose.

duckdb <<'SQL'
INSTALL quack;
LOAD quack;

FROM quack_query(
  'quack:localhost:9494',
  'SELECT * FROM otlp_flush(''otlp:0.0.0.0:4318'')',
  token = 'dev-quack-token-123456'
);

FROM quack_query(
  'quack:localhost:9494',
  $$
  SELECT time_unix_nano, service_name, severity_text, body
  FROM lake.main.otlp_logs
  WHERE service_name = 'local-ducklake-demo'
  ORDER BY time_unix_nano DESC
  LIMIT 5
  $$,
  token = 'dev-quack-token-123456'
);
SQL

Stop cleanly

docker stop duckdb-otlp

The image sends otlp_stop('otlp:0.0.0.0:4318') during shutdown, so remaining buffered rows are committed before the process exits.

otlp_flush seals buffered ingest rows. Run DuckDB or DuckLake maintenance commands when you need compaction.