Kubernetes Cluster Monitoring

The Ops Platform collects node resources, container resources (cAdvisor) and cluster object state (kube-state-metrics) from Kubernetes, and presents nodes, Pods, Deployments and containers under Resource Monitoring → K8s.

First answer one question: is the platform installed inside the monitored cluster?

That determines what you have to do — the two paths differ substantially:

Your situation	What to do
Platform runs inside the monitored cluster (K8s deployment, either `ops.yaml` or `k8s-cluster/`)	Nothing. Data appears right after installation — skip to Verification
Platform runs outside the cluster (standalone install, separate K8s to monitor)	Install the collection components in the monitored cluster
You need to monitor multiple K8s clusters	The local cluster self-collects as usual; install one collector set in each remaining cluster — see Monitoring multiple clusters

Collection always happens inside the cluster

On either path, kubelet / cAdvisor / kube-state-metrics are scraped by a Pod running in that cluster, using its own mounted ServiceAccount to reach the apiserver. So no NodePort needs to be exposed, and no bearer token ever leaves the cluster.

The out-of-cluster pull mode (static) removed in 1.5.3 was the opposite: it required a bearer token plus apiserver, kubelet and kube-state-metrics exposed through three NodePorts — the more clusters, the larger the exposed surface. The "Kubernetes cluster" data source type was removed at the same time: K8s is no longer configured through data sources, so its absence in the UI is expected.

Platform inside the monitored cluster: works out of the box

The K8s deployment manifests already ship all three pieces — nothing to configure:

Component	Where
`ENV_K8S_MONITOR_MODE: "incluster"`	ConfigMap (top of `ops.yaml` / `11-configmap.yaml`)
Prometheus ServiceAccount + ClusterRole	End of the same `ops.yaml` / `20-rbac.yaml`
kube-state-metrics	End of the same `ops.yaml` / `ops-ksm` in `40-stateless.yaml`

The platform's own Prometheus scrapes the local cluster directly and stores the data in its local TSDB. Do not configure remote_write here — it would just push data to itself, adding a pointless hop.

Upgrading from a release older than 1.5.7

ops.yaml before 1.5.7 shipped without the RBAC objects and kube-state-metrics, so the K8s dashboards stayed empty. Download ops.yaml again and apply it over the existing install — existing data is unaffected (the additions are a ServiceAccount, a ClusterRole, kube-state-metrics and three ConfigMap keys).

To turn the K8s dashboards off: set ENV_K8S_MONITOR_MODE to off in the ConfigMap and skip the kube-state-metrics objects.

Collector mode: platform outside the cluster, or multiple clusters

Install a lightweight collector set inside the monitored cluster: a dedicated K8s-scraping Prometheus plus RBAC and kube-state-metrics. It pushes metrics back to the platform via remote_write, so the platform never needs inbound access to the cluster.

Customer environments typically allow outbound but not inbound traffic, which makes this direction decisive.

⬇ Download k8s-addons.tar.gz

After extracting, edit the three variables in 30-prometheus-incluster.yaml first, then apply:

- name: ENV_PROMETHEUS_REMOTE_WRITE_URL
  value: "http://<ops-platform>:30881/mdis/prometheus/server/api/v1/write"
- name: ENV_PROMETHEUS_REMOTE_WRITE_TOKEN
  value: "<the platform's ENV_OPS_TOKEN>"
- name: ENV_K8S_CLUSTER_NAME
  value: "prod-k8s"          # cluster identifier used by the dashboards; must be unique

Build the write URL from three parts — none of them optional:

http://<platform gateway>:<port> + <sub path> + /prometheus/server/api/v1/write
        ↑                           ↑            ↑
   ops-gateway NodePort             the          the platform's Prometheus runs
   (30881 by default) for K8s;      platform's   under --web.external-url=
   48881 for standalone             ENV_OPS_     /prometheus/server/
                                    SUB_PATH
                                    (/mdis)

The three things that go wrong

① Do not use the platform's :9090. ops-prometheus is a ClusterIP Service, resolvable only inside the platform's own cluster. Other clusters cannot reach it — go through the gateway port.

② Do not drop /prometheus/server/. Without it the endpoint returns 404.

③ The token is mandatory. This gateway write endpoint uses Bearer authentication; without it you get 401. Use the platform's ENV_OPS_TOKEN (the same one you log in with).

All three failures only ever appear as retries in the collector's container log — in the UI they all look like plain "no data". Verify from inside the collector Pod before moving on:

# without a token — expect 401, proving the endpoint exists and auth is active
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
  http://<ops-platform>:30881/mdis/prometheus/server/api/v1/write

# with the token — expect 400 (endpoint exists, empty body rejected)
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
  -H 'Authorization: Bearer <ENV_OPS_TOKEN>' \
  http://<ops-platform>:30881/mdis/prometheus/server/api/v1/write

A 404 means the URL is wrong; 401 on both means the token is wrong.

Then apply:

kubectl apply \
  -f k8s-addons/00-namespace.yaml \
  -f k8s-addons/10-kube-state-metrics.yaml \
  -f k8s-addons/30-prometheus-incluster.yaml

Data flow:

┌─ Monitored cluster ───────────────────────┐
│  kubelet / cAdvisor ──┐                   │
│  kube-state-metrics ──┼─→ Collector Prom  │
│  apiserver ───────────┘   (its own SA)    │
└────────────────────────────────┬──────────┘
                                 │ remote_write (outbound only, Bearer token)
                                 ↓
   Ops gateway :30881/mdis/prometheus/server/api/v1/write
                                 ↓
              Platform Prometheus (writes to local TSDB)
                                 ↓
              Grafana "Resource Monitoring → K8s"

Only K8s metrics are pushed

The collector forwards only kube_* / container_* / kubelet_* / machine_* / node_*. Its own Prometheus runtime metrics are not pushed and do not consume platform storage.

What is in the bundle

File	Purpose	Required
`00-namespace.yaml`	Namespace `mdis-monitoring`	✅
`10-kube-state-metrics.yaml`	kube-state-metrics + RBAC + Service	✅
`20-node-exporter.yaml`	node_exporter DaemonSet + Service	Optional (node/container metrics already come from kubelet + cAdvisor)
`30-prometheus-incluster.yaml`	In-cluster ops-prometheus + RBAC + Service	✅
`40-alloy-logs.yaml`	Container log collection DaemonSet	Only if you want container logs — see Service Log Ingestion

40-alloy-logs.yaml handles logs, not metrics, which is why it is absent from the apply command above — it needs ENV_LOKI_PUSH_URL set first. Without it the K8s dashboards still work; only the Container Console stays empty.

Do not install this set into the cluster hosting the platform

The platform already scrapes its own cluster. A second set means duplicate collection: two copies of the same data arrive under different cluster names, the dashboard dropdown shows two entries that never agree, and you pay for the storage twice.

Monitoring multiple clusters

Metrics from each cluster arrive labelled origin_prometheus=<cluster name>, and the Resource Monitoring → K8s dashboards gain a cluster dropdown at the top (default: All, showing everything merged).

The walkthrough below covers the common case: the platform lives in cluster A, and you want to onboard cluster B.

Step 1: Give cluster A (the platform's own) a name

A already self-collects; it just needs a recognisable name. Edit the platform ConfigMap:

kubectl -n hap-ops edit configmap ops-config

Change ENV_K8S_CLUSTER_NAME from the default default to something meaningful:

ENV_K8S_CLUSTER_NAME: "main-k8s"

Restart Prometheus to apply it (the value is written into Prometheus's external_labels at startup):

kubectl -n hap-ops rollout restart deployment/ops-prometheus

Renaming splits historical data

origin_prometheus is a label on the metrics themselves. After a rename, new data carries the new name while old data keeps default, and both appear in the dropdown. Either set the name early, while there is little data, or keep default — do not rename after months of history.

Step 2: Install the collector components in cluster B

First confirm B can reach A's gateway port outbound (NodePort 30881 by default); A never needs inbound access to B:

# from any node in cluster B
curl -s -o /dev/null -w '%{http_code}\n' http://<cluster-A>:30881/mdis/web/

Then download k8s-addons.tar.gz and set the three variables in 30-prometheus-incluster.yaml:

- name: ENV_PROMETHEUS_REMOTE_WRITE_URL
  value: "http://<cluster-A>:30881/mdis/prometheus/server/api/v1/write"
- name: ENV_PROMETHEUS_REMOTE_WRITE_TOKEN
  value: "<ENV_OPS_TOKEN of the platform in cluster A>"
- name: ENV_K8S_CLUSTER_NAME
  value: "prod-k8s"

Apply (against cluster B's kubeconfig):

kubectl apply \
  -f k8s-addons/00-namespace.yaml \
  -f k8s-addons/10-kube-state-metrics.yaml \
  -f k8s-addons/30-prometheus-incluster.yaml

Step 3: Confirm the data arrives

In cluster B, the collector log should show no remote_write retries:

kubectl -n mdis-monitoring logs deploy/ops-prometheus | grep -iE '\[k8s\]|remote|error' | tail

Back on the platform, the Cluster dropdown on the Resource Monitoring → K8s dashboards should now offer both main-k8s and prod-k8s. You can also ask the platform's Prometheus directly how many clusters are reporting:

count by (origin_prometheus) (kube_node_info)

Adding a third, fourth cluster

Repeat step 2. Each cluster changes only ENV_K8S_CLUSTER_NAME; the write URL and token all point at the same platform:

Cluster	How	`ENV_K8S_CLUSTER_NAME`
A (hosts the platform)	Already self-collecting — edit the ConfigMap	`main-k8s`
B	One collector set	`prod-k8s`
C	One collector set	`test-k8s`
…	…	…

Names must be self-explanatory, never IPs, never duplicated

The dropdown shows exactly this string. Two clusters sharing a name produce metrics that merge together with no way to tell them apart.

No extra configuration is needed on the platform side. Do watch storage: metrics from N clusters all land in the platform's TSDB, so raise ENV_PROMETHEUS_RETENTION_SIZE or grow the disk as the number or size of clusters increases.

Verification

The ServiceAccount plus apiserver proxy discovers kubelet, cAdvisor, kube-state-metrics and pods automatically — no exposed kubelet port, no kubeconfig.

Platform self-collecting its own cluster (hap-ops for both ops.yaml and k8s-cluster/):

kubectl -n hap-ops get deploy kube-state-metrics
kubectl -n hap-ops exec deploy/ops-prometheus -- \
  curl -s localhost:9090/prometheus/server/api/v1/targets | grep -o '"job":"k8s-[^"]*"'

Collector mode (installed in the monitored cluster's mdis-monitoring):

kubectl -n mdis-monitoring get pods
kubectl -n mdis-monitoring logs deploy/ops-prometheus | grep '\[k8s\]'

The startup log should contain a [k8s] line reporting the cluster identifier (origin_prometheus="..."), followed by the push target if remote_write is configured. A malformed URL prefix is reported right there as a warning.

In both cases the k8s-kubelet, k8s-cadvisor and k8s-kube-state-metrics jobs should all be UP. The Resource Monitoring → K8s dashboards then show node and Pod views.

First answer one question: is the platform installed inside the monitored cluster?​

Platform inside the monitored cluster: works out of the box​

Collector mode: platform outside the cluster, or multiple clusters​

What is in the bundle​

Monitoring multiple clusters​

Step 1: Give cluster A (the platform's own) a name​

Step 2: Install the collector components in cluster B​

Step 3: Confirm the data arrives​

Adding a third, fourth cluster​

Verification​