Kubernetes
Deploy the full Ops Platform inside the monitored HAP/HDP Kubernetes cluster. Prometheus discovers the current cluster through ServiceAccount permissions, with no token or NodePort required.
Follow the steps below to complete a Kubernetes deployment. The process covers the full deployment path for the Kubernetes form.
If the Ops Platform is deployed outside the cluster, it can monitor only hosts and middleware. It cannot directly collect Kubernetes node, Pod, or Deployment metrics from the cluster itself. The out-of-cluster collection mode has been removed since 1.5.3.
Before You Start
Prepare the following
- A running HAP/HDP cluster
- Monitored middleware such as MongoDB/MySQL/Redis/Kafka/Elasticsearch, based on the actual deployment. These services must be reachable from inside the cluster, and read-only accounts must be prepared according to Deployment Overview
- Flink is optional. Leave it empty if it is not deployed
kubectlcan operate the target cluster. If the cluster is not ready, see Appendix A: Build a Test Cluster
Download k8s-cluster.tar.gz. After extraction, it contains:
00-namespace.yaml 11-configmap.yaml 30-stateful.yaml 50-daemonsets.yaml
10-secret.yaml.example 20-rbac.yaml 40-stateless.yaml
Choose the persistence mode
Stateful components, including Prometheus, Grafana, and ops-mongo, must persist data. Choose one of the following two modes. This decision determines how Step 2 is performed:
| PVC (default, recommended) | hostPath (alternative) | |
|---|---|---|
| Data location | Storage volume. Data is retained when nodes are rebuilt | Local directory on a specific node |
| Manifest set | Split manifests under k8s-cluster/ | Single ops.yaml |
| Prerequisite | The cluster has a default StorageClass | None, but the node must be fixed |
| Suitable for | Managed cloud clusters (TKE/ACK/EKS) and production | Self-managed clusters with fixed nodes, or clusters without an available StorageClass |
Cloud nodes are replaceable. Self-healing, scaling, and version upgrades may rebuild nodes, while hostPath data does not move with Pods.
Losing the Prometheus TSDB affects historical metric queries. Losing ops-mongo means losing alert rules, notification channels, data source configuration, and slow-query history.
The following sections use PVC as the main path. Differences for hostPath are called out separately.
Step 1: Pull the Image
crictl pull nocoly/ops-allinone:1.5.7
ops-nodeagent runs as a DaemonSet, so every node in the cluster must have this image. For offline environments, see Offline Package.
The image is hosted in Alibaba Cloud Hangzhou (registry.cn-hangzhou.aliyuncs.com). Pulling 2.16GB from overseas clusters may take more than ten minutes, and different nodes may finish at different times.
If a node is stuck, you can kubectl cordon that node and delete non-DaemonSet Pods on it so they are rescheduled to nodes that already have the image cached. DaemonSet Pods still need to wait for image pull completion on every node. For overseas deployments, sync the image to a nearby registry first.
Step 2: Prepare Persistence
PVC (Recommended)
The cluster must have a default StorageClass. Check it with:
kubectl get sc
The output should include a StorageClass marked with (default). If it does not exist, create one based on the environment.
Managed Cloud Clusters (TKE/ACK/EKS)
Cloud vendors usually provide a default StorageClass, but creating a StorageClass that supports online expansion is recommended. Changing the StorageClass after PVC creation usually requires recreating the cloud disk. Tencent Cloud TKE example:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: cbs-expandable
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: com.tencent.cloud.csi.cbs
parameters:
diskType: CLOUD_PREMIUM # Explicit declaration. TKE default SC often uses `type: cbs`, which is not a valid parameter and falls back to the driver default.
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true # TKE default SC usually does not enable this switch, although the driver supports online expansion.
After creating it, remove the default annotation from the previous default StorageClass to avoid having two defaults:
kubectl patch sc <previous-default-sc-name> -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
PVC.status.capacity is updated only after the second phase, where resize2fs runs on the node, and that phase requires a Pod to mount the PVC.
If no Pod mounts the PVC, it may remain at the old value and show FileSystemResizePending. This does not mean expansion failed; the cloud disk has usually already been expanded.
Self-Managed or Single-Node Test Clusters
If no cloud disk is available, use local-path-provisioner. Data is stored in a local node directory:
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
kubectl patch sc local-path -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
30-stateful.yaml requests prometheus 100Gi + ops-mongo 20Gi + grafana 10Gi = 130Gi by default. This is sized for production clusters.
local-path-provisioner does not check remaining disk capacity. The PVC can become Bound, and Prometheus reports no space left on device only after the disk is full. This is easy to misdiagnose as a service failure. Test machines often have around 100G of disk space, so adjust the request before applying:
# 30-stateful.yaml, three volumeClaimTemplates
resources:
requests:
storage: 20Gi # ops-prometheus, test environment
# storage: 5Gi # ops-mongo
# storage: 5Gi # ops-grafana
In addition, local-path does not support online expansion. Increasing the size requires recreating the PVC. Use a cloud disk StorageClass with allowVolumeExpansion in production.
hostPath (Alternative)
If no StorageClass is used, pin stateful components to a specific node and store data on that node's local disk. Label the node:
kubectl label node <node-name> hap-ops=true
Use
kubectl get node -o wideto view node names. Data directories are under/data/mdis/on that node, with one subdirectory for each component: prometheus, grafana, loki, tempo, and mongo.
Step 3: Prepare Object Storage (Optional)
Logs (Loki) and traces (Tempo) write to local volumes by default. To retain them for a long time without being limited by a single node's disk, prepare object storage with two buckets:
| Bucket | Purpose | Suggested name |
|---|---|---|
| Logs | Loki block storage | mdis-loki |
| Traces | Tempo block storage | mdis-tempo |
There are two configuration locations. Keep them separate:
| What to configure | Where | Description |
|---|---|---|
| endpoint, AK/SK, path style | ENV_S3_* in 10-secret.yaml | Fill in Step 4 |
| Bucket names | ENV_S3_BUCKET_LOKI and ENV_S3_BUCKET_TEMPO under ops-loki and ops-tempo in 30-stateful.yaml | The manifest already presets the two names above. Create buckets with these names and no manifest change is required |
The two components must use different buckets. If only the generic ENV_S3_BUCKET is set, Loki and Tempo write to the same bucket and their directory structures overwrite each other. Startup logs will show a warning. For endpoint formats and compatibility notes for each cloud provider, see Object Storage.
If long-term retention is not required, skip this step and add object storage later when needed.
Step 4: Fill Configuration
# 1. Namespace
kubectl apply -f 00-namespace.yaml
# 2. Secret: copy the example and fill it in
cp 10-secret.yaml.example 10-secret.yaml
The following two items are required in 10-secret.yaml:
| Variable | Description |
|---|---|
ENV_OPS_TOKEN | Login token. Use a strong random string |
ENV_ALERT_CRYPTO_KEY | 32 hexadecimal characters. Generate it with openssl rand -hex 16 |
If object storage was prepared in Step 3, also fill ENV_S3_ENDPOINT/ENV_S3_ACCESS_KEY/ENV_S3_SECRET_KEY/ENV_S3_FORCE_PATH_STYLE. Bucket names are configured in the previous step.
kubectl apply -f 10-secret.yaml
Other adjustable items, such as data retention and component addresses, are in 11-configmap.yaml. Defaults can be used directly. See Environment Variables when changes are needed.
Use kubectl create secret or write YAML directly. Heredoc escaping may write quotation marks into variable values, causing login authentication failures that are hard to diagnose.
hostPath mode: Do not use the split manifests above. Download the single manifest and modify it as needed:
The manifest includes all components, including gateway, Prometheus, Grafana, Loki, Tempo, Alloy, MongoDB, middleware exporters, node-exporter, and their Service definitions. The namespace is hap-ops. After downloading, modify ENV_OPS_TOKEN, ENV_ALERT_CRYPTO_KEY, the image tag, and the node label.
The manifest includes Prometheus ServiceAccount/ClusterRole and kube-state-metrics. When the Ops Platform runs inside the monitored cluster, it can collect the current cluster after deployment, the same as PVC mode.
Do not deploy k8s-addons additionally. That component is for the collection mode where the Ops Platform is outside the monitored cluster. It creates another namespace and coexists with this manifest. Use it only when the current Ops Platform needs to manage other clusters. See Kubernetes Cluster Monitoring.
Manifests earlier than v1.5.7 did not include RBAC and kube-state-metrics, so Kubernetes panels had no data after deployment. Download the latest manifests again and apply them over the existing resources.
Step 5: Start
kubectl apply -f 11-configmap.yaml -f 20-rbac.yaml \
-f 30-stateful.yaml -f 40-stateless.yaml -f 50-daemonsets.yaml
# Wait until ready. The first run needs to pull a 2.16GB image, which may take more than ten minutes across regions.
kubectl get pods -n hap-ops -w
hostPath mode: run
kubectl apply -f ops.yaml(the manifest includes thehap-opsnamespace). To stop it, runkubectl delete -f ops.yaml. Both modes use thehap-opsnamespace, so the verification commands below are the same.
After all Pods are Running, confirm that the alert subsystem is mounted:
kubectl logs deploy/ops-gateway -n hap-ops | grep alert
# Expected: [alert] alert subsystem mounted at /api/alert/
Kubernetes does not have Compose depends_on. When image pull times differ, gateway may enter Running before ops-mongo. Since 1.4.5, if ops-server cannot connect to ops-mongo, it retries in the background with 3s/6s/12s backoff. After mongo is ready, the alert subsystem is mounted automatically, with no manual restart required.
The log above means the alert subsystem has been mounted successfully.
Step 6: Expose and Access
ops-gateway-ui is exposed through NodePort 30881. Use nginx as a reverse proxy for access outside the cluster:
upstream hap-ops {
server <node1>:30881 max_fails=0;
server <node2>:30881 max_fails=0;
server <node3>:30881 max_fails=0;
}
location /mdis/ {
allow <ops-egress-ip>;
deny all;
proxy_pass http://hap-ops;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
ENV_OPS_SUB_PATH=/mdis is already set in ConfigMap, so /mdis/ can be passed through as-is without rewrite. Visit https://domain/mdis/web/. The login token is ENV_OPS_TOKEN.
Security boundary = IP allowlist + the Ops Platform 32-character token + session cookie. No additional Basic Auth is required.
type: NodePort assigns a NodePort to every port on that Service. This would also expose 8081, the ops-server internal API that should only be used by Prometheus inside the cluster for service discovery. Therefore, UI traffic uses NodePort, while internal APIs use ClusterIP only.
Step 7: Verification
pip install playwright && playwright install chromium
python3 mdis_regression.py --base http://<node-ip>:30881 --token <ENV_OPS_TOKEN>
Exit code 0 means all checks passed. If a middleware panel has no data, first check the Collection Status column on the UI "Data Sources" page. Hover over the red status to see the failure reason. Most collection issues can be located directly from that reason. For additional troubleshooting, see FAQ.
Follow-Up Integration
After the Ops Platform is deployed, continue integrating monitored objects:
- Data Sources: register hosts and middleware. Collection configuration is configured only in the UI, not in ConfigMap
- Kubernetes Cluster: collect node, container, and cluster object status
- Node Exporter: monitor servers outside the cluster. Nodes inside the cluster are already covered by
ops-nodeagent - Service Logs: let HAP/HDP microservices push logs to the Ops Platform
- Tracing: let business services report traces
Appendix A: Build a Test Cluster
Skip this appendix if a cluster already exists. The following is a minimal process for creating a test cluster with kubeadm on Debian 12. It is only for test environments. For production, follow the official Kubernetes documentation.
# 1. Disable swap. kubelet requires swap to be disabled by default.
swapoff -a && sed -i '/ swap / s/^/#/' /etc/fstab
# 2. Kernel modules and network parameters
cat > /etc/modules-load.d/k8s.conf <<'EOF'
overlay
br_netfilter
EOF
modprobe overlay && modprobe br_netfilter
cat > /etc/sysctl.d/k8s.conf <<'EOF'
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system
# 3. containerd. Enable SystemdCgroup, otherwise kubelet restarts repeatedly.
apt-get update && apt-get install -y containerd
mkdir -p /etc/containerd && containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
systemctl restart containerd && systemctl enable containerd
# 4. kubeadm / kubelet / kubectl
apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.30/deb/Release.key \
| gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.30/deb/ /' \
> /etc/apt/sources.list.d/kubernetes.list
apt-get update && apt-get install -y kubelet kubeadm kubectl
apt-mark hold kubelet kubeadm kubectl
Initialize the control plane:
kubeadm init \
--pod-network-cidr=10.244.0.0/16 \
--image-repository registry.aliyuncs.com/google_containers
export KUBECONFIG=/etc/kubernetes/admin.conf
echo 'export KUBECONFIG=/etc/kubernetes/admin.conf' >> ~/.bashrc
kubeadm init pulls control-plane images from registry.k8s.io by default. Access from Mainland China is restricted, so kubeadm init may stay at [preflight] Pulling images and then time out. Use --image-repository registry.aliyuncs.com/google_containers.
kubeadm config images pull must use the same parameter, or image pre-pull will fail in the same way.
Install CNI. Without CNI, nodes remain NotReady:
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
For a single-node cluster, also remove the control-plane taint, otherwise business Pods cannot be scheduled:
kubectl taint nodes --all node-role.kubernetes.io/control-plane-
Verify:
kubectl get nodes # Should be Ready
kubectl get pods -A # Should all be Running
Appendix B: Multi-Node and Dedicated Monitoring Nodes
The slim and standard editions use the same manifests. For multi-node clusters, pay attention to the following items:
- Dedicated monitoring node (optional): taint the node and add tolerations to each workload:
kubectl taint nodes <node-name> hap-ops=true:NoSchedule# Add under spec.template.spectolerations:- { key: "hap-ops", operator: "Equal", value: "true", effect: "NoSchedule" }
- Multiple middleware sets: standard clusters often have multiple mongo/kafka/es sets. Do not deploy multiple agents. Since 1.3.0, a single
ops-agentsupports multiple targets with automatic port allocation. Register multiple data sources on the "Data Sources" page. - Node metrics:
ops-nodeagentis a DaemonSet and automatically covers all nodes. To monitor servers outside the cluster, deploy Node Exporter separately on the target servers.
Appendix C: Three Structural Differences from Standalone Deployment
The following items are specific to cluster mode and should not be copied directly from the standalone Compose configuration.
1. The ops-agent Service must be headless (clusterIP: None)
ops-agent dynamically starts one exporter for each data source instance, with dynamically allocated ports:
mongodb -> 19400, 19401, 19402 ... one additional port for each MongoDB instance
kafka -> 19500, 19501 ...
mysql -> 19100 ...
A regular ClusterIP Service forwards only the ports listed in ports. All dynamic ports return connection refused, and Prometheus marks all middleware targets as DOWN. A headless Service resolves DNS directly to Pod IPs, so any port can be reached without enumerating it.
2. Container logs rely on the alloy-logs DaemonSet
ops-alloy (ROLE=alloy) collects logs through docker.sock, while Kubernetes uses containerd. It cannot collect container logs in Kubernetes. Periodic docker.sock connection errors in logs are expected and can be ignored; in Kubernetes, this component receives OTLP.
The alloy-logs DaemonSet in 50-daemonsets.yaml reads Pod logs through the Kubernetes API and is used for container log collection in Kubernetes scenarios.
3. Stateful components use StatefulSet instead of Deployment
Deployment rolling updates may cause old and new Pods to mount the same disk at the same time. Therefore, hostPath mode needs strategy: Recreate. A single-replica StatefulSet naturally deletes the old Pod before creating the new one, avoiding this issue.