Object Storage Integration (MinIO/COS)

Logs (Loki) and traces (Tempo) are stored in local volumes by default. When data volume is large or data needs to be shared across nodes, switch to object storage.

Scope

• Standalone deployment: Only logs (Loki) need object storage consideration. Standalone deployment does not provide tracing and does not generate trace data, so ops-tempo does not need and is not recommended to configure object storage.
• Cluster deployment: Both logs and traces can use object storage, and they must use two different buckets.

1. Prepare MinIO

If MinIO or cloud object storage already exists, skip to Step 2.

docker run -d --name minio \
  -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin \
  -e MINIO_ROOT_PASSWORD='<strong-random-password>' \
  -v /data/minio:/data \
  --restart always \
  minio/minio server /data --console-address ":9001"

• 9000 is the S3 API port used by the Ops Platform. 9001 is the Web console.
• In production, replace MINIO_ROOT_PASSWORD with a strong random string and expose only port 9000 inside the internal network.

Open http://<MinIO-host-IP>:9001 in a browser and log in with the account and password above.

2. Create Buckets

Loki and Tempo must use different buckets. Sharing a bucket causes metadata to overwrite each other.

Use the console Buckets → Create Bucket, or run:

docker run --rm --network host --entrypoint sh minio/mc -c "
  mc alias set local http://<MinIO-host-IP>:9000 minioadmin '<password>' &&
  mc mb local/mdis-loki &&
  mc mb local/mdis-tempo &&
  mc ls local"

Names are not mandatory. This document uses mdis-loki/mdis-tempo.

3. Create Access Keys (AK/SK)

Do not use the root account directly. In the console, go to Access Keys → Create access key, and save the generated Access Key and Secret Key (Secret Key is shown only once).

Command-line method (create a dedicated user and authorize the two buckets):

docker run --rm --network host --entrypoint sh minio/mc -c "
  mc alias set local http://<MinIO-host-IP>:9000 minioadmin '<password>' &&
  mc admin user add local mdis '<secret-key-8-chars-or-more>' &&
  mc admin policy attach local readwrite --user mdis"

4. Fill in Configuration

Standalone (ops.yaml)

Only add environment variables to the ops-loki service:

  ops-loki:
    environment:
      ENV_S3_ENDPOINT: 192.168.1.10:9000     # MinIO S3 API address, use host:port, without http://
      ENV_S3_BUCKET_LOKI: mdis-loki          # Create this bucket in MinIO in advance
      ENV_S3_ACCESS_KEY: mdis
      ENV_S3_SECRET_KEY: '<SK>'
      ENV_S3_FORCE_PATH_STYLE: 'true'

After modification, run docker compose -f ops.yaml up -d ops-loki to recreate the container.

Cluster (10-secret.yaml)

stringData:
  ENV_S3_ENDPOINT: "minio.default.svc.cluster.local:9000"
  ENV_S3_ACCESS_KEY: "mdis"
  ENV_S3_SECRET_KEY: "<SK>"
  ENV_S3_FORCE_PATH_STYLE: "true"

Bucket names are not in Secret, and each component uses its own bucket

In 30-stateful.yaml, configure ENV_S3_BUCKET_LOKI for ops-loki and ENV_S3_BUCKET_TEMPO for ops-tempo. The two buckets must be different.

Do not write bucket names into the shared Secret, and do not use the old generic variable ENV_S3_BUCKET. It is passed to both components. If Loki writes the directory structure first, Tempo initialization fails and enters CrashLoopBackOff. (ENV_S3_BUCKET is retained only for compatibility with existing deployments.)

⚠️ Both buckets must be created before deployment. If a bucket does not exist, Loki reports NoSuchBucket and restarts repeatedly.

5. Endpoint Format (Key Check)

ENV_S3_ENDPOINT is the MinIO S3 API address, not the console address. It should look like 192.168.1.10:9000 or http://192.168.1.10:9000. Both with and without http:// are accepted: Tempo uses minio-go, which only accepts host:port, and the Ops Platform automatically strips the scheme when starting Tempo. Loki uses HTTP according to the fixed configuration (insecure: true). ⚠️ For this reason, object storage that only provides HTTPS is not currently supported.

Deployment Relationship	endpoint Value
MinIO and Ops Platform in the same compose network	Container name, such as minio:9000
MinIO in another compose stack or deployed on the host	Host IP + host mapped port, such as 192.168.1.10:9011
MinIO in the same Kubernetes cluster	Service DNS, such as minio.default.svc.cluster.local:9000
Cloud object storage	Endpoint domain provided by the vendor

Do not use a container name from another stack

If the containers are not in the same compose network, container names cannot be resolved. ops-loki fails to start and reports:

dial tcp: lookup minio: no such host

Use the host IP in this case. The host mapped port may not be 9000. Use the actual value shown by docker ps. For example, if it is mapped as 9011:9000, use 9011.

Path Style

Object Storage	ENV_S3_FORCE_PATH_STYLE
MinIO	true
Tencent Cloud COS	false (only virtual-hosted style is supported; true always fails)
UCloud US3	true

6. Verification

After recreating the container, check logs and the bucket:

# 1. Container has started and has no S3 errors
docker logs <loki-container> 2>&1 | grep -iE "error|fail" | head

# 2. After triggering log writes, objects should appear in the bucket
docker run --rm --network host --entrypoint sh minio/mc -c "
  mc alias set local http://<MinIO-host-IP>:9000 mdis '<SK>' &&
  mc ls -r local/mdis-loki | head"

Objects starting with fake/ or index/ in the bucket indicate success. It may take several minutes after configuration because Loki uploads only after batch writes.

7. Common Errors

Error in Logs	Cause
dial tcp: lookup xxx: no such host	endpoint uses a name that cannot be resolved in the container network. See Step 5
connection refused	Wrong port. Use the actual mapped port shown by docker ps
SignatureDoesNotMatch	Incorrect AK/SK
AccessDenied	AK/SK is correct, but lacks permission to the bucket. Check the user policy
NoSuchBucket	Bucket not created or bucket name misspelled

Existing logs are not migrated after switching storage

Loki does not migrate existing data when switching storage backends. Old logs remain in the original local volume. After the switch, new logs must be generated before content appears in the new backend. This does not mean the configuration failed.