Tracing Ingestion

Tracing shows the call path and latency of a request across multiple microservices. Data is reported by the business side through OTLP to the Ops Platform Alloy and then forwarded to Tempo for storage.

Only available for HAP/HDP cluster deployment

Tracing is intended for Kubernetes cluster deployment. Standalone (Docker Compose) deployment is not supported.

Recommended Method: Istio Sidecar Automatic Reporting

When services run inside an Istio mesh, sidecars can automatically generate and report spans with zero business code changes. HAP/HDP cluster mode usually runs inside the mesh.

Injecting sidecar does not mean traces are reported

After Istio is installed and istio-injection=enabled is set on a namespace, spans are still not generated by default. You must configure the reporting address and sampling rate for Istio. Without this configuration, errors are usually not reported, but the tracing page has no data.

Self-check:

kubectl -n istio-system get cm istio \
  -o jsonpath='{.data.mesh}' | grep -A3 extensionProviders

No output means it has not been configured. Follow the steps below.

Step 1: Add a Reporting Endpoint from Istio to Ops Platform Alloy

kubectl -n istio-system edit configmap istio

Add extensionProviders under mesh:

data:
  mesh: |-                                  # This is a string block
    extensionProviders:
      - name: mdis-otel                    # Must match the name used by Telemetry below
        opentelemetry:
          service: ops-alloy.hap-ops.svc.cluster.local
          port: 4317

Complete example:

data:
  mesh: |-
    accessLogFile: /dev/stdout
    defaultConfig:
      holdApplicationUntilProxyStarts: true
    extensionProviders:
      - name: mdis-otel
        opentelemetry:
          service: ops-alloy.hap-ops.svc.cluster.local
          port: 4317

Common Configuration Errors

1. Do not indent extensionProviders under defaultConfig. extensionProviders is a top-level field under mesh. If placed at the wrong level, istiod silently ignores it. The configuration exists, but spans are not generated.

2. service must use a full FQDN, and the namespace must be the actual namespace where the Ops Platform is deployed. The default manifest uses hap-ops, but real deployments may use another namespace. If the namespace is wrong, the sidecar cannot resolve the service and usually does not report an error. Confirm it with:

kubectl get svc -A | grep ops-alloy

After saving, restart istiod:

kubectl -n istio-system rollout restart deploy/istiod
kubectl -n istio-system rollout status deploy/istiod

Verify that the configuration was written:

kubectl -n istio-system get cm istio -o jsonpath='{.data.mesh}' | grep -A4 extensionProviders

Step 2: Enable Sampling

The previous step only configures where Istio reports traces. Sampling rate must also be configured. When no Telemetry resource is created, the sampling rate is 0, and no span is generated.

kubectl apply -f - <<'EOF'
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mdis-tracing
  namespace: istio-system      # Place in the Istio root namespace to apply to the whole mesh
spec:
  tracing:
    - providers:
        - name: mdis-otel      # Must exactly match the name in Step 1
      randomSamplingPercentage: 100
EOF

Confirm that the Telemetry resource exists:

kubectl get telemetry -A

Use 100 for sampling during verification, then reduce it

Set randomSamplingPercentage to 100 during verification. Otherwise, under 10% sampling, a small number of requests may not be sampled and can be mistaken for configuration failure. After confirming the trace path is working, change it back to 1–10. Higher sampling increases Tempo storage and write pressure. Changing this value does not require restarting istiod, but business Pods must be restarted to take effect.

Telemetry API Version

The example uses telemetry.istio.io/v1 (Istio 1.22+, verified on 1.29). Earlier Istio versions use v1alpha1; other fields are the same. Confirm the current version with:

kubectl api-resources | grep telemetry

Step 3: Restart Business Pods

Business Pods must be restarted after configuring tracing. Existing sidecars do not automatically reload the new mesh config.

kubectl rollout restart deploy -n <business-namespace>

Confirm that sidecars were injected:

kubectl get pod -n <business-namespace> -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].name}{"\n"}{end}' | grep istio-proxy

The corresponding row must show enabled to have a sidecar.

Step 4: Confirm Spans Have Reached the Ops Platform

This step confirms whether the Ops Platform Alloy has received spans:

kubectl -n <Ops-Platform-namespace> exec deploy/ops-alloy -- \
  curl -s localhost:12345/metrics | grep otelcol_receiver_accepted_spans_total

Trigger several cross-service calls first, and then run the check again. If the metric increases, the reporting path is connected:

otelcol_receiver_accepted_spans_total{...,transport="grpc"} 83

If the metric remains 0 or cannot be found, troubleshoot with the table below:

Symptom	Likely Cause
Metric is 0	Business Pods were not restarted (Step 3), or the sampling rate is too low and requests were not sampled
Metric does not exist	Alloy has not enabled OTLP receiving; check whether the ops-alloy Service exposes 4317
Metric increases but page is empty	Alloy→Tempo path issue; check ENV_TEMPO_GRPC_URL and ops-tempo container logs

Reporting Endpoints

Alloy OTLP receiving endpoints:

Protocol	Endpoint
OTLP gRPC	4317
OTLP HTTP	4318

If an application has its own OpenTelemetry SDK, point the exporter to the above endpoint. Inside the cluster, use Service DNS, for example: http://ops-alloy.<Ops-Platform-namespace>.svc.cluster.local:4317 (replace the namespace with the actual Ops Platform namespace; kubectl get svc -A | grep ops-alloy can be used to check it).

Reporters in another cluster cannot access it directly

ops-alloy is ClusterIP and can be resolved only inside the cluster where the Ops Platform is deployed. Cross-cluster reporting requires exposing 4317 first (NodePort or Ingress). Because trace volume can be large, plan exposure and rate limiting according to the actual network environment.

Data Retention and Storage

Trace retention is controlled by ENV_TEMPO_RETENTION, default 30 days. See Environment Variables.

To retain trace data long term and avoid node disk limits, prepare an object storage bucket for Tempo during deployment (recommended name: mdis-tempo). See Object Storage.

Verification

After configuration, open "Tracing → Trace Analysis" and filter by service or Trace ID. If no data appears, first check whether otelcol_receiver_accepted_spans_total has increased, and then check the Tempo logs and ENV_TEMPO_GRPC_URL.