docker-otel-lgtm

docker-otel-lgtm

An OpenTelemetry backend in a Docker image. It bundles the OpenTelemetry Collector, Prometheus (metrics), Tempo (traces), Loki (logs), Pyroscope (profiles), and Grafana into a single container — with optional OBI (eBPF auto-instrumentation).

The grafana/otel-lgtm Docker image is an open source backend for OpenTelemetry that’s intended for development, demo, and testing environments.

[!IMPORTANT] If you are looking for a production-ready, out-of-the box solution to monitor applications and minimize MTTR (mean time to resolution) with OpenTelemetry and Prometheus, you should try Grafana Cloud Application Observability.

Documentation

• Blog posts:
  • An OpenTelemetry backend in a Docker image: Introducing grafana/otel-lgtm
  • Observability in under 5 seconds: Reflecting on a year of grafana/otel-lgtm

Get the Docker image

The Docker image is available on Docker Hub.

docker pull grafana/otel-lgtm:latest

Run the Docker image

Linux/Unix

./run-lgtm.sh

Windows (PowerShell)

./run-lgtm

macOS

With docker/podman:

./run-lgtm.sh

With Apple Container:

Before building or running, ensure Apple Container’s services is started with container system start:

container system start
CONTAINER_RUNTIME_OVERRIDE=container ./run-lgtm.sh

The build and run scripts automatically select Podman, then Docker, then Apple Container. Set CONTAINER_RUNTIME_OVERRIDE to docker, podman, or container to select a runtime when multiple are installed. This also works with mise, for example:

CONTAINER_RUNTIME_OVERRIDE=container mise run lgtm

Caveats with Apple Container:

• OBI eBPF auto-instrumentation is not supported (--privileged / --pid=host don’t exist).
• The first run of container system start prompts to download and install a VM kernel, which can take several minutes.
• Apple Container defaults to 1G of memory, which is insufficient for the stack; the run script sets 2G by default. To override it, use the LGTM_CONTAINER_MEMORY environment variable (for example, 4G or 4096M):

CONTAINER_RUNTIME_OVERRIDE=container LGTM_CONTAINER_MEMORY=4G ./run-lgtm.sh

Linux/Unix/macOS Using mise

You can also use mise to run the Docker image:

mise run lgtm

Query and troubleshoot with gcx

Use gcx with your local LGTM instance to query fresh application telemetry and investigate an empty dashboard with existing gcx agent skills. gcx runs on your host; no Grafana Cloud account is required.

Upgrading from the generated MCP setup? See the migration note for removed files and credential handling.

Configuration

Enable logging

You can enable logging in the .env file for troubleshooting:

This has nothing to do with any application logs, which are collected by OpenTelemetry.

Configure the shutdown timeout

The container forwards SIGTERM and SIGINT to every backend and waits up to five seconds for a graceful shutdown. Set LGTM_SHUTDOWN_TIMEOUT_SECONDS to change this grace period. Any process still running after the timeout is forcefully stopped.

Enable OBI (eBPF auto-instrumentation)

OpenTelemetry eBPF Instrumentation (OBI) uses eBPF to automatically generate traces and RED metrics for HTTP/gRPC services — with zero code changes.

To enable OBI, add ENABLE_OBI=true to your .env file or pass it as an environment variable:

ENABLE_OBI=true ./run-lgtm.sh

# Using mise
mise run lgtm-obi

Requirements: Linux kernel 5.8+ with BTF support. The run-lgtm.sh and run-lgtm.ps1 scripts automatically add the required --pid=host and --privileged Docker flags when OBI is enabled. If you run docker run directly, you must add these flags manually. OBI is not supported on Apple Container (container CLI has no --privileged / --pid=host); use Docker or Podman.

[!NOTE] The --pid=host flag shares the host’s PID namespace with the container, so OBI can discover and instrument processes running on the host — not just inside the container. For example, OBI_TARGET=java will instrument Java processes running on the host as well.

Target specific applications

By default, OBI discovers services on common ports (80, 443, 8080-8099, 3000-3999, 5000-5999). You can target specific applications:

# Monitor all Java processes
ENABLE_OBI=true OBI_TARGET=java ./run-lgtm.sh

# Monitor all Python processes
ENABLE_OBI=true OBI_TARGET=python ./run-lgtm.sh

# Monitor a specific executable by name
ENABLE_OBI=true OBI_TARGET=myapp ./run-lgtm.sh

# Monitor specific ports
ENABLE_OBI=true OTEL_EBPF_OPEN_PORT=8080,9090 ./run-lgtm.sh

Send data to vendors

In addition to the built-in observability tools, you can also send data to vendors. That way, you can easily try and switch between different backends.

If the OTEL_EXPORTER_OTLP_ENDPOINT variable is set, the OpenTelemetry Collector will send data (logs, metrics, and traces) to the specified endpoint using “OTLP/HTTP”.

You can also configure per-signal endpoints:

• OTEL_EXPORTER_OTLP_LOGS_ENDPOINT
• OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
• OTEL_EXPORTER_OTLP_TRACES_ENDPOINT

If both global and per-signal endpoints are set, per-signal values take precedence. Endpoints must include the scheme (for example, http://jaeger:4318).

In addition, you can provide OTEL_EXPORTER_OTLP_HEADERS, for example, to authenticate with the backend.

Send data to Grafana Cloud

You can find the values for the environment variables in your Grafana Cloud account.

Persist data across container instantiation

The various components in the repository are configured to write their data to the /data directory. If you need to persist data across containers being created and destroyed, you can mount a volume to the /data directory. Note that this image is intended for development, demo, and testing environments and persisting data to an external volume doesn’t change that. However, this feature could be useful in certain cases for some users even in testing situations.

Customize backend configuration

Each backend supports a *_EXTRA_ARGS environment variable for passing additional CLI flags without modifying any files:

For example, to set a 90-day retention period for Prometheus:

docker run -e PROMETHEUS_EXTRA_ARGS="--storage.tsdb.retention.time=90d" grafana/otel-lgtm

[!NOTE] The value is split on whitespace into separate arguments. For options that require values with spaces, mount a custom configuration file instead (see below).

You can also pass OpenTelemetry Collector CLI options as container arguments by prefixing each option with --otelcol-. Use --name=value for options with values; each container argument becomes one collector argument, so spaces and newlines in a value are preserved. When the first container argument uses this prefix, the default stack launcher runs and ignores any other unprefixed arguments. Ordinary command overrides (such as bash) continue to run as commands. For example, this passes TLS certificate and key overrides to the collector (with the files mounted at /certs):

docker run -v ./certs:/certs:ro grafana/otel-lgtm \
  '--otelcol-config=yaml:receivers::otlp::protocols::grpc::tls::cert_file: "/certs/cert.pem"' \
  '--otelcol-config=yaml:receivers::otlp::protocols::grpc::tls::key_file: "/certs/key.pem"'

This is an alternative to OTELCOL_EXTRA_ARGS, which splits its value on whitespace and therefore cannot preserve spaces within an argument. Use a single --otelcol-name=value argument rather than separate option and value arguments.

For deeper customization, you can mount custom configuration files into the container:

docker run -v ./my-loki-config.yaml:/otel-lgtm/loki-config.yaml:ro grafana/otel-lgtm

Grafana is configured via GF_* environment variables — see the Grafana documentation for details.

Pre-install Grafana plugins

You can pre-install Grafana plugins by adding them to the GF_PLUGINS_PREINSTALL environment variable. See the Grafana documentation for more information.

Add custom dashboards

You can add custom Grafana dashboards by mounting them into the container with a provisioning configuration.

Create a dashboard JSON file and a provisioning YAML file:

dashboards-provisioning.yaml:

apiVersion: 1

providers:
  - name: "Custom Dashboards"
    type: file
    options:
      path: /otel-lgtm/grafana/conf/provisioning/dashboards/custom
      foldersFromFilesStructure: false

Mount both files in your docker-compose.yml (Docker/Podman only; Apple Container has no official compose equivalent):

services:
  lgtm:
    image: grafana/otel-lgtm
    volumes:
      - ./custom-dashboard.json:/otel-lgtm/grafana/conf/provisioning/dashboards/custom/custom-dashboard.json:ro
      - ./dashboards-provisioning.yaml:/otel-lgtm/grafana/conf/provisioning/dashboards/custom.yaml:ro

See the Java example for a complete working example.

To set a custom dashboard as the home dashboard, add the GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH environment variable:

services:
  lgtm:
    image: grafana/otel-lgtm
    environment:
      GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH: /otel-lgtm/grafana/conf/provisioning/dashboards/custom/custom-dashboard.json

Run as a non-root user

The image runs as root by default, but it also supports running as an arbitrary UID.

docker run --user 1000820000:0 --tmpfs /data:rw,mode=1777,size=3g -p 3000:3000 grafana/otel-lgtm

Run lgtm in Kubernetes

# Create k8s resources
kubectl apply -f k8s/lgtm.yaml

# Configure port forwarding
kubectl port-forward service/lgtm 3000:3000 3200:3200 4040:4040 4317:4317 4318:4318 9090:9090

# Using mise
mise k8s-apply
mise k8s-port-forward

Run lgtm with Nomad

See nomad/ for a Nomad job spec (optional Consul service registration) that mirrors the Kubernetes example and run-lgtm.sh ports.

nomad job run nomad/lgtm.nomad.hcl

Send OpenTelemetry Data

There’s no need to configure anything: the Docker image works with OpenTelemetry’s defaults.

# Not needed, but these are the defaults in OpenTelemetry
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318

View Grafana

Navigate to http://127.0.0.1:3000 and log in with the default built-in user admin and password admin.

Build the Docker image from scratch

cd docker/
docker build . -t grafana/otel-lgtm

# Using mise
mise build-lgtm

With Apple Container, run from the repository root after starting its services as described above:

CONTAINER_RUNTIME_OVERRIDE=container ./build-lgtm.sh

# Using mise
CONTAINER_RUNTIME_OVERRIDE=container mise run build-lgtm

The first container build downloads a builder image before building the LGTM image, so the first build can take longer.

[!TIP] If you built your image locally, you can use the run-lgtm scripts with the parameters latest true to run your local image (or mise run local-lgtm).

Build and run the example app

Run

Run the example REST service:

Run on Unix/Linux

./run-example.sh

Run on Windows (PowerShell)

./run-example

Run on Unix/Linux using mise

mise run example

Generate traffic

Generate on Unix/Linux

./generate-traffic.sh

Generate on Windows (PowerShell)

./generate-traffic

Generate on Unix/Linux using mise

mise run generate-traffic

Run example apps in different languages

The example apps are in the examples/ directory. Each example has a run.sh or run.cmd script to start the app.

Every example implements a rolldice service, which returns a random number between 1 and 6.

Each example uses a different application port (to be able to run all applications at the same time).

Verifying Container Image Signatures

The container images that are published are signed using cosign v3+. You can verify the signatures using a command similar to the following example:

VERSION="0.29.0"
IMAGE="docker.io/grafana/otel-lgtm:${VERSION}"
CERTIFICATE_IDENTITY_REGEX="^https://github\.com/grafana/shared-workflows/\.github/workflows/sign-and-attest\.yml@"
CERTIFICATE_OIDC_ISSUER="https://token.actions.githubusercontent.com"
CERTIFICATE_REPO="grafana/docker-otel-lgtm"

cosign verify "${IMAGE}" --certificate-identity-regexp "${CERTIFICATE_IDENTITY_REGEX}" --certificate-oidc-issuer "${CERTIFICATE_OIDC_ISSUER}" --certificate-github-workflow-repository "${CERTIFICATE_REPO}"

It is also possible to verify the signatures of images from our continuous integration that are published to the GitHub Container Registry. For example for the main branch:

VERSION="main"
IMAGE="ghcr.io/grafana/docker-otel-lgtm:${VERSION}"
WORKFLOW="ghcr-image-build-and-publish.yml"
CERTIFICATE_IDENTITY="https://github.com/grafana/docker-otel-lgtm/.github/workflows/${WORKFLOW}@refs/heads/${VERSION}"
CERTIFICATE_OIDC_ISSUER="https://token.actions.githubusercontent.com"

cosign verify "${IMAGE}" --certificate-identity "${CERTIFICATE_IDENTITY}" --certificate-oidc-issuer "${CERTIFICATE_OIDC_ISSUER}"

Verifying Container Image Attestations

The container images that are published are also attested. You can verify the attestations using the GitHub CLI as shown in the following example:

VERSION="0.29.0"
IMAGE="oci://docker.io/grafana/otel-lgtm:${VERSION}"
REPOSITORY="grafana/docker-otel-lgtm"
SIGNER_WORKFLOW="grafana/shared-workflows/.github/workflows/sign-and-attest.yml"

gh attestation verify --repo "${REPOSITORY}" "${IMAGE}" --signer-workflow "${SIGNER_WORKFLOW}"

It is also possible to verify the attestations of images from our continuous integration that are published to the GitHub Container Registry. For example for the main branch:

VERSION="main"
REPOSITORY="grafana/docker-otel-lgtm"
IMAGE="oci://ghcr.io/${REPOSITORY}:${VERSION}"

gh attestation verify --repo "${REPOSITORY}" "${IMAGE}"

Related Work

• Metrics, Logs, Traces and Profiles in Grafana
• OpenTelemetry Acceptance Tests (OATs)