Kubernetes Cluster Monitoring
The Ops Platform collects Kubernetes node resources, container resources (cAdvisor), and cluster object status (kube-state-metrics), and displays node, Pod, Deployment, container, and other views in "Resource Monitoring → Kubernetes".
Confirm the Ops Platform Deployment Location
The Ops Platform deployment location determines how Kubernetes metrics are collected:
| Scenario | Configuration |
|---|---|
Ops Platform deployed inside the HAP/HDP Kubernetes cluster (reusing the existing cluster, either ops.yaml or k8s-cluster/) | No extra configuration required. Data from this cluster is collected automatically after deployment. Go directly to Verification |
| Ops Platform outside the target HAP/HDP Kubernetes cluster (for example, standalone Ops Platform deployment while another Kubernetes cluster needs monitoring) | Deploy collection components inside the monitored cluster |
| Monitoring multiple Kubernetes clusters | The HAP/HDP Kubernetes cluster where the Ops Platform runs is collected automatically; deploy one set of collection components in each other monitored cluster. See Monitoring Multiple Clusters |
Regardless of the method, the processes that scrape kubelet/cAdvisor/kube-state-metrics run inside the monitored cluster and access apiserver through their mounted ServiceAccount. Therefore, no NodePort exposure is required, and the Bearer Token does not leave the cluster.
Ops Platform Inside the HAP/HDP Kubernetes Cluster: Automatic Collection
Kubernetes deployment manifests already include the following configuration, so no extra action is required:
| Component | Location |
|---|---|
ENV_K8S_MONITOR_MODE: "incluster" | ConfigMap (ops.yaml top section/11-configmap.yaml) |
| Prometheus ServiceAccount + ClusterRole | End of the same ops.yaml/20-rbac.yaml |
| kube-state-metrics | ops-ksm at the end of the same ops.yaml/40-stateless.yaml |
The built-in Prometheus directly collects the HAP/HDP Kubernetes cluster where the Ops Platform runs, and stores data in the local TSDB. Do not configure remote_write.
Configuring it would mean pushing data back to itself and adds an unnecessary write path.
The single-file ops.yaml manifest before 1.5.7 did not include RBAC or kube-state-metrics, so the Kubernetes panel had no data.
Download ops.yaml again and apply it over the existing deployment to add them without affecting existing data.
The added items are ServiceAccount/ClusterRole/kube-state-metrics and three ConfigMap keys.
To disable the Kubernetes panel, set ENV_K8S_MONITOR_MODE in the ConfigMap to off and remove kube-state-metrics related objects.
Collection Component Mode: Ops Platform Outside the Target Cluster or Monitoring Multiple Clusters
Deploy a lightweight set of collection components inside the monitored cluster: a Prometheus dedicated to Kubernetes metrics + RBAC + kube-state-metrics. It pushes metrics back to the Ops Platform through remote_write; the Ops Platform does not need reverse access to this cluster.
This mode only requires outbound access from the monitored cluster to the Ops Platform, and is suitable for customer environments with restricted inbound access.
After extraction, modify the following three variables in 30-prometheus-incluster.yaml first, and then apply:
- name: ENV_PROMETHEUS_REMOTE_WRITE_URL
value: "http://<Ops-Platform-address>:30881/mdis/prometheus/server/api/v1/write"
- name: ENV_PROMETHEUS_REMOTE_WRITE_TOKEN
value: "<Ops-Platform-ENV_OPS_TOKEN>"
- name: ENV_K8S_CLUSTER_NAME
value: "prod-k8s" # Cluster identifier; dashboards use it to distinguish clusters. Avoid duplicate names
The write address is assembled from three parts:
http://<Ops Platform gateway address>:<port> + <sub-path> + /prometheus/server/api/v1/write
↑ ↑ ↑
Kubernetes deployment uses ops-gateway Ops Platform Ops Platform Prometheus runs
NodePort (default 30881) ENV_OPS_ under --web.external-url=
Standalone deployment uses 48881 SUB_PATH /prometheus/server/
default /mdis
1. Do not use the Ops Platform :9090. The Ops Platform ops-prometheus is ClusterIP and can be resolved only inside the Ops Platform cluster. Other clusters cannot access it. Use the external port of the gateway.
2. Do not omit /prometheus/server/. Missing this segment returns 404.
3. Token is required. This gateway write endpoint uses Bearer authentication. Missing token returns 401. Use the Ops Platform ENV_OPS_TOKEN (the same one used to log in to the Ops Platform).
The errors above only retry continuously in the collection-side container logs. In the UI they usually appear as "No Data". After configuration, run the self-check first inside the collection component Pod:
# Without token: should return 401, proving the entry exists and authentication is enabled
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
http://<Ops-Platform-address>:30881/mdis/prometheus/server/api/v1/write
# With token: should return 400 (endpoint exists, but rejects the empty request body)
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
-H 'Authorization: Bearer <Ops-Platform-ENV_OPS_TOKEN>' \
http://<Ops-Platform-address>:30881/mdis/prometheus/server/api/v1/write
404 means the address is assembled incorrectly. If both requests return 401, the token is wrong.
Apply after modification:
kubectl apply \
-f k8s-addons/00-namespace.yaml \
-f k8s-addons/10-kube-state-metrics.yaml \
-f k8s-addons/30-prometheus-incluster.yaml
Data flow:
┌─ Monitored cluster ─────────────────────────────┐
│ kubelet / cAdvisor ──┐ │
│ kube-state-metrics ──┼─→ Collection Prometheus │
│ apiserver ───────────┘ (uses local SA) │
└────────────────────────── ──────┬────────────────┘
│ remote_write (outbound only, with Bearer token)
↓
Ops Platform gateway :30881/mdis/prometheus/server/api/v1/write
↓
Ops Platform Prometheus (writes to local TSDB)
↓
Grafana "Resource Monitoring → Kubernetes" panel
The collector pushes only kube_*/container_*/kubelet_*/machine_*/node_* metrics back to the Ops Platform. Its own Prometheus runtime metrics are not pushed, avoiding extra storage usage.
Collection Component Manifests
| File | Purpose | Required |
|---|---|---|
00-namespace.yaml | Namespace mdis-monitoring | ✅ |
10-kube-state-metrics.yaml | kube-state-metrics + RBAC + Service | ✅ |
20-node-exporter.yaml | node_exporter DaemonSet + Service | Optional (node/container metrics are already covered by kubelet+cAdvisor) |
30-prometheus-incluster.yaml | In-cluster ops-prometheus + RBAC + Service | ✅ |
40-alloy-logs.yaml | Container log collection DaemonSet | Deploy only when container logs need to be collected. See Service Log Ingestion |
40-alloy-logs.yaml is responsible for log collection, not metrics collection, and is not included in the apply command above. Before using it, modify ENV_LOKI_PUSH_URL.
If this file is not deployed, the Kubernetes panel still has metrics data, but the "Container Console" has no data.
The Ops Platform already automatically collects the HAP/HDP Kubernetes cluster where it runs. Deploying another set of collection components causes duplicate collection: two copies of data are written to the same TSDB with different cluster names, which creates confusing entries in the dashboard dropdown and adds storage overhead.
Monitoring Multiple Clusters
Metrics from each cluster carry the origin_prometheus=<cluster name> label when they enter the Ops Platform. The "Resource Monitoring → Kubernetes" panel displays a Cluster dropdown at the top, allowing cluster switching. The default All view shows merged data.
The following example uses "the Ops Platform deployed in cluster A and managing cluster B" to describe the configuration process.
Step 1: Name Cluster A Where the Ops Platform Runs
Cluster A is already collected automatically by the Ops Platform. Configure a readable cluster name by modifying the Ops Platform ConfigMap:
kubectl -n hap-ops edit configmap ops-config
Change ENV_K8S_CLUSTER_NAME from the default default to a meaningful name:
ENV_K8S_CLUSTER_NAME: "main-k8s"
Restart Prometheus for the configuration to take effect (this value is written to Prometheus external_labels at startup):
kubectl -n hap-ops rollout restart deployment/ops-prometheus
origin_prometheus is a label on metrics. After renaming, new data uses the new name while old data remains under default.
The dashboard dropdown will show both entries. Confirm the name early during rollout to avoid later changes affecting historical data identification.
Step 2: Deploy Collection Components in Cluster B
First confirm that cluster B can access cluster A's gateway port outbound (default NodePort 30881). Cluster A does not need reverse access to cluster B:
# On any node in cluster B
curl -s -o /dev/null -w '%{http_code}\n' http://<cluster-A-address>:30881/mdis/web/
Then download k8s-addons.tar.gz and modify three variables in 30-prometheus-incluster.yaml:
- name: ENV_PROMETHEUS_REMOTE_WRITE_URL
value: "http://<cluster-A-address>:30881/mdis/prometheus/server/api/v1/write"
- name: ENV_PROMETHEUS_REMOTE_WRITE_TOKEN
value: "<cluster-A-Ops-Platform-ENV_OPS_TOKEN>"
- name: ENV_K8S_CLUSTER_NAME
value: "prod-k8s"
Apply it (using cluster B's kubeconfig):
kubectl apply \
-f k8s-addons/00-namespace.yaml \
-f k8s-addons/10-kube-state-metrics.yaml \
-f k8s-addons/30-prometheus-incluster.yaml
Step 3: Confirm Successful Push
Check the collection component logs in cluster B. There should be no remote_write retry errors:
kubectl -n mdis-monitoring logs deploy/ops-prometheus | grep -iE '\[k8s\]|remote|error' | tail
Back in the Ops Platform, the "Cluster" dropdown at the top of "Resource Monitoring → Kubernetes" should show both main-k8s and prod-k8s.
You can also query the Ops Platform Prometheus directly to see how many clusters are reporting data:
count by (origin_prometheus) (kube_node_info)
Add a Third or Fourth Cluster
Repeat Step 2. For each cluster, only ENV_K8S_CLUSTER_NAME needs to be changed. The write address and token all point to the same Ops Platform:
| Cluster | Configuration | ENV_K8S_CLUSTER_NAME |
|---|---|---|
| A (Ops Platform location) | Already self-collected; modify ConfigMap | main-k8s |
| B | One set of collection components | prod-k8s |
| C | One set of collection components | test-k8s |
| … | … | … |
The dashboard dropdown displays this name. If two clusters use the same name, their metrics are mixed and cannot be distinguished.
No extra configuration is required on the Ops Platform side. Pay attention to storage capacity: metrics from N clusters are all written to the Ops Platform TSDB.
For many clusters or large clusters, increase ENV_PROMETHEUS_RETENTION_SIZE or expand the disk as needed.
Verification
ServiceAccount + apiserver proxy automatically discovers kubelet, cAdvisor, kube-state-metrics, and pods. No kubelet port exposure or kubeconfig is required.
Ops Platform automatically collecting the HAP/HDP Kubernetes cluster where it runs (ops.yaml uses hap-ops, and k8s-cluster/ uses the same namespace):
kubectl -n hap-ops get deploy kube-state-metrics
kubectl -n hap-ops exec deploy/ops-prometheus -- \
curl -s localhost:9090/prometheus/server/api/v1/targets | grep -o '"job":"k8s-[^"]*"'
Collection component mode (deployed in mdis-monitoring of the monitored cluster):
kubectl -n mdis-monitoring get pods
kubectl -n mdis-monitoring logs deploy/ops-prometheus | grep '\[k8s\]'
Startup logs should include [k8s] incluster collection enabled, cluster identifier origin_prometheus="...".
After remote_write is configured, the logs also show the push target. If the address prefix is wrong, the log gives a direct warning.
In both cases, the three jobs k8s-kubelet/k8s-cadvisor/k8s-kube-state-metrics should be UP.
Then open "Resource Monitoring → Kubernetes" to view node and Pod data.