[alibaba/higress]想把prometheus采集指标转成prometheus-adapter规则用于hpa

4

一般来说根据 CPU 或者内存来做 HPA 就足够了，你们是否有 CPU 和内存不是瓶颈，但是 Higress Gateway 又需要扩容的场景呢？

cr7258

8

我们现在想加几个关于http的指标来进行hpa，比如请求量、500错误数量等。有些业务cpu和内存不足以覆盖使用场景。

w7team

0

请求量可以根据 envoy_http_downstream_rq_total 和 envoy_cluster_upstream_rq_total 指标。

500 错误可以根据 envoy_http_downstream_rq, envoy_http_downstream_rq_total 和 envoy_cluster_upstream_rq, envoy_cluster_upstream_rq 指标。

Envoy 指标说明的文档可以参考：https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/stats

cr7258

6

这几个指标我们也看到了，但是筛选条件有限，无法定位到关联的pod，只能定位到gateway的pod，基本无法使用。


envoy_http_downstream_rq_total {app="higress-gateway",container="higress-gateway",higress="higress-system-higress-gateway",http_conn_manager_prefix="agent",instance="10.42.0.214:15020",job="w7-system/vm-operator-k8s-offline-metrics-auto",namespace="higress-system",node="server1",pod="higress-gateway-6f4b5d6f79-zlbr8",pod_template_hash="6f4b5d6f79",prometheus="w7-system/vm-operator-k8s-offline-metrics-agent",sidecar_istio_io_inject="false"}
min:465,564
median:466,014
max:466,464

envoy_http_downstream_rq_total {app="higress-gateway",container="higress-gateway",higress="higress-system-higress-gateway",http_conn_manager_prefix="outbound_0.0.0.0_80",instance="10.42.0.214:15020",job="w7-system/vm-operator-k8s-offline-metrics-auto",namespace="higress-system",node="server1",pod="higress-gateway-6f4b5d6f79-zlbr8",pod_template_hash="6f4b5d6f79",prometheus="w7-system/vm-operator-k8s-offline-metrics-agent",sidecar_istio_io_inject="false"}
min:12,622
median:12,782
max:12,888

envoy_http_downstream_rq_total {app="higress-gateway",container="higress-gateway",higress="higress-system-higress-gateway",http_conn_manager_prefix="admin",instance="10.42.0.214:15020",job="w7-system/vm-operator-k8s-offline-metrics-auto",namespace="higress-system",node="server1",pod="higress-gateway-6f4b5d6f79-zlbr8",pod_template_hash="6f4b5d6f79",prometheus="w7-system/vm-operator-k8s-offline-metrics-agent",sidecar_istio_io_inject="false"}
min:5,927
median:5,957
max:5,987

envoy_http_downstream_rq_total {app="higress-gateway",container="higress-gateway",higress="higress-system-higress-gateway",http_conn_manager_prefix="stats",instance="10.42.0.214:15020",job="w7-system/vm-operator-k8s-offline-metrics-auto",namespace="higress-system",node="server1",pod="higress-gateway-6f4b5d6f79-zlbr8",pod_template_hash="6f4b5d6f79",prometheus="w7-system/vm-operator-k8s-offline-metrics-agent",sidecar_istio_io_inject="false"}
min:5,925
median:5,955
max:5,985

envoy_http_downstream_rq_total {app="higress-gateway",container="higress-gateway",higress="higress-system-higress-gateway",http_conn_manager_prefix="outbound_0.0.0.0_443",instance="10.42.0.214:15020",job="w7-system/vm-operator-k8s-offline-metrics-auto",namespace="higress-system",node="server1",pod="higress-gateway-6f4b5d6f79-zlbr8",pod_template_hash="6f4b5d6f79",prometheus="w7-system/vm-operator-k8s-offline-metrics-agent",sidecar_istio_io_inject="false"}
min:4,502
median:4,504
max:4,504

w7team

1

你是想根据 Higress Gateway 上的 HTTP 请求指标来 HPA 你的业务 Pod?

cr7258

2

是的

w7team

7

那这样的话可能依靠 Higress Gateway 的指标就不太行了，可以考虑在业务的 Pod 暴露一下自身的 HTTP 指标来实现了。

cr7258

4

应该是有方案的，这个项目都在阿里云上用的，应该是有实践案例

w7team

[alibaba/higress]想把prometheus采集指标转成prometheus-adapter规则用于hpa

回答