Object Storage (MinIO / COS)

Logs (Loki) and traces (Tempo) are stored on local volumes by default. Switch them to object storage when the data volume grows or when storage has to be shared across nodes.

Which components actually need it

• Single-node deployment: only logs (Loki) are worth moving to object storage. A single-node deployment has no distributed tracing capability and produces no trace data, so ops-tempo neither needs nor benefits from it.
• Cluster deployment: both logs and traces can use object storage, and they must use two different buckets.

1. Prepare MinIO

Skip to step 2 if you already have MinIO or a cloud object store.

docker run -d --name minio \
  -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin \
  -e MINIO_ROOT_PASSWORD='<strong-random-password>' \
  -v /data/minio:/data \
  --restart always \
  minio/minio server /data --console-address ":9001"

• 9000 is the S3 API port (this is what the platform connects to); 9001 is the web console.
• In production use a strong random MINIO_ROOT_PASSWORD and keep port 9000 on the internal network only.

Open http://minio-host:9001 and sign in with those credentials.

2. Create buckets

Loki and Tempo must use separate buckets — sharing one corrupts each other's metadata.

Console: Buckets → Create Bucket. Or from the command line:

docker run --rm --network host --entrypoint sh minio/mc -c "
  mc alias set local http://<minio-host>:9000 minioadmin '<password>' &&
  mc mb local/mdis-loki &&
  mc mb local/mdis-tempo &&
  mc ls local"

The names are arbitrary; this page uses mdis-loki / mdis-tempo.

3. Create an access key (AK/SK)

Do not reuse the root account. Console: Access Keys → Create access key, then save the Access Key and Secret Key (the secret is shown only once).

Command line — create a dedicated user with read/write access:

docker run --rm --network host --entrypoint sh minio/mc -c "
  mc alias set local http://<minio-host>:9000 minioadmin '<password>' &&
  mc admin user add local mdis '<secret-key, 8+ chars>' &&
  mc admin policy attach local readwrite --user mdis"

4. Apply the configuration

Single node (ops.yaml)

Add the variables to the ops-loki service only:

  ops-loki:
    environment:
      ENV_S3_ENDPOINT: 192.168.1.10:9000     # MinIO S3 API address; http:// prefix optional
      ENV_S3_BUCKET_LOKI: mdis-loki          # create this bucket in MinIO first
      ENV_S3_ACCESS_KEY: mdis
      ENV_S3_SECRET_KEY: '<secret-key>'
      ENV_S3_FORCE_PATH_STYLE: 'true'

Then recreate that container with docker compose -f ops.yaml up -d ops-loki.

Cluster (10-secret.yaml)

stringData:
  ENV_S3_ENDPOINT: "minio.default.svc.cluster.local:9000"
  ENV_S3_ACCESS_KEY: "mdis"
  ENV_S3_SECRET_KEY: "<secret-key>"
  ENV_S3_FORCE_PATH_STYLE: "true"

The bucket name does not belong in the Secret

ENV_S3_BUCKET is set individually for ops-loki and ops-tempo in 30-stateful.yaml. Putting it in the shared Secret would make both components write to the same bucket and overwrite each other's metadata.

5. Getting the endpoint right (the step that goes wrong most often)

ENV_S3_ENDPOINT is the S3 API address, not the console address — either 192.168.1.10:9000 or http://192.168.1.10:9000 works: the http:// prefix is optional. Tempo's minio-go client only accepts host:port, so the platform strips the scheme when starting Tempo; Loki always connects over HTTP (insecure: true is hard-coded). ⚠️ For the same reason, HTTPS-only object storage is not currently supported.

Deployment relationship	What to put in the endpoint
MinIO in the same compose network	the container name, e.g. minio:9000
MinIO in another compose stack or on the host	host IP + published host port, e.g. 192.168.1.10:9011
MinIO in the same Kubernetes cluster	the Service DNS name, e.g. minio.default.svc.cluster.local:9000
Cloud object storage	the endpoint domain given by the provider

A container name from another stack will not resolve

Outside a shared compose network the container name cannot be resolved and ops-loki fails to start with:

dial tcp: lookup minio: no such host

Use the host IP instead. The published host port is not necessarily 9000 — use whatever docker ps shows (if it is mapped as 9011:9000, use 9011).

by default.

Path style

Object store	ENV_S3_FORCE_PATH_STYLE
MinIO	true
Tencent Cloud COS	false (virtual-hosted style only; true always fails)
UCloud US3	true

6. Verify

After recreating the container, check the logs and the bucket:

# 1. The container is up with no S3 errors
docker logs <loki-container> 2>&1 | grep -iE "error|fail" | head

# 2. After some logs have been produced, objects should appear in the bucket
docker run --rm --network host --entrypoint sh minio/mc -c "
  mc alias set local http://<minio-host>:9000 mdis '<secret-key>' &&
  mc ls -r local/mdis-loki | head"

Objects under fake/ or index/ mean it works. Allow a few minutes right after the switch — Loki uploads in batches.

7. Common errors

Log message	Cause
dial tcp: lookup xxx: no such host	the endpoint uses a name the container network cannot resolve, see step 5
connection refused	wrong port — use the mapping shown by docker ps
SignatureDoesNotMatch	wrong access key or secret key
AccessDenied	credentials are valid but lack permission on the bucket; check the user policy
NoSuchBucket	the bucket was never created, or the name is misspelt

Existing logs are not migrated

Loki does not move existing data when the backend changes; old logs stay in the original local volume. You need to generate new logs to see anything in the new backend — this is expected, not a misconfiguration.