“Monitoring with prometheus”的版本间的差异
(未显示同一用户的117个中间版本) | |||
第5行: | 第5行: | ||
[https://www.bookstack.cn/read/prometheus-book/README.md Prometheus操作指南] | [https://www.bookstack.cn/read/prometheus-book/README.md Prometheus操作指南] | ||
+ | =进阶= | ||
+ | 以后全改成自己 docker-compose build | ||
+ | |||
+ | [https://blog.51cto.com/wzlinux/2487300 prometheus 监控概述(一)有常用监控警告] | ||
+ | |||
+ | [https://blog.51cto.com/wzlinux/2490670 PromQL 内置函数(三)] | ||
+ | |||
+ | |||
+ | |||
+ | [https://www.jb51.net/article/189679.htm prometheus python 库编写自定义指标的方法(完整代码)] | ||
+ | |||
+ | |||
+ | [https://www.jianshu.com/p/19442595e886 promethues + python + flask监控后端服务状态] | ||
+ | |||
+ | [https://blog.csdn.net/weixin_33754913/article/details/91803530 prometheus python client] | ||
+ | |||
+ | [https://blog.csdn.net/JackLiu16/article/details/80381201?utm_medium=distribute.pc_relevant.none-task-blog-title-7&spm=1001.2101.3001.4242 使用Python和Flask编写Prometheus监控] | ||
+ | |||
+ | [https://blog.csdn.net/specter11235/article/details/87927202 python开发prometheus exporter] | ||
+ | |||
+ | ==服务发现== | ||
+ | === 基于文件的服务发现 Use file-based service discovery to discover scrape targets=== | ||
+ | ==== 安装 node-exporter by ansible==== | ||
+ | https://github.com/evan886/my-ansible/tree/main/prome-node-add | ||
+ | |||
+ | ==== 配置 ==== | ||
+ | <pre> | ||
+ | #这里以json格式为例子其实 另外一种 也可以用YAML格式的 | ||
+ | cd apps/prom/prometheus | ||
+ | sudo mkdir targets | ||
+ | |||
+ | cat targets/nodes.json | ||
+ | [{ | ||
+ | "targets": [ | ||
+ | "172.16.0.20:9100", | ||
+ | "172.16.0.29:9100" | ||
+ | ], | ||
+ | "labels": { | ||
+ | "server": "node_export_live" | ||
+ | } | ||
+ | }] | ||
+ | |||
+ | vi prometheus.yml | ||
+ | |||
+ | # - cadvisor:8080 | ||
+ | |||
+ | - job_name: 'cadvisor-nodejs' | ||
+ | static_configs: | ||
+ | - targets: ['121.41.101.200:8080'] | ||
+ | |||
+ | - job_name: 'nodes_server_discoverry' | ||
+ | file_sd_configs: | ||
+ | - files: | ||
+ | - targets/*.json | ||
+ | refresh_interval: 1m | ||
+ | |||
+ | |||
+ | sudo curl -X POST http://172.16.0.42:9090/-/reload | ||
+ | |||
+ | 解析如下 | ||
+ | |||
+ | scrape_configs: | ||
+ | - job_name: 'prometheus' | ||
+ | static_configs: # 这个配置表示 静态发现 | ||
+ | - targets: ['localhost:9090'] | ||
+ | |||
+ | - job_name: 'node' # 增加一个节点,取名为 node | ||
+ | metrics_path: "/metrics" # 获取数据的路径 http://10.0.20.12:9100/metrics | ||
+ | file_sd_configs: # 这个配置表示通过文件发现 | ||
+ | - files: | ||
+ | - targets/nodes/*.json # 读取目录下的所有json结尾的文件 | ||
+ | refresh_interval: 1m # 刷新频率,1 分钟 | ||
+ | |||
+ | |||
+ | |||
+ | |||
+ | 打开 | ||
+ | https://prom.com/targets | ||
+ | |||
+ | 有 job_name: 'nodes_server_discoverry' 就是成功 | ||
+ | </pre> | ||
+ | [https://prometheus.io/docs/guides/file-sd/ Use file-based service discovery to discover scrape targets] | ||
=ins on docker-composer= | =ins on docker-composer= | ||
+ | git clone https://github.com/evan886/docker-compose-monitor.git | ||
+ | |||
+ | |||
https://gitee.com/atompi/PrometheusStackDockerCompose | https://gitee.com/atompi/PrometheusStackDockerCompose | ||
+ | 注意 grafana update 升级为gra 8.2.7 就行了 不行有 漏洞 | ||
+ | |||
+ | docker-compose up -d grafana | ||
=安全= | =安全= | ||
第18行: | 第106行: | ||
=prometheus= | =prometheus= | ||
+ | |||
+ | Prometheus/prəˈmiː.θi.əs/ | ||
+ | |||
+ | |||
[[Nginx用户密码认证配置 Basic HTTP authentication]] | [[Nginx用户密码认证配置 Basic HTTP authentication]] | ||
第33行: | 第125行: | ||
[https://zhuanlan.zhihu.com/p/24811652 基于Prometheus的分布式在线服务监控实践] | [https://zhuanlan.zhihu.com/p/24811652 基于Prometheus的分布式在线服务监控实践] | ||
+ | |||
+ | [https://juejin.im/post/6844903866064977934 Prometheus学习系列(十三)之配置解析 ] | ||
==usage== | ==usage== | ||
第40行: | 第134行: | ||
https://github.com/prometheus/blackbox_exporter/blob/master/blackbox.yml | https://github.com/prometheus/blackbox_exporter/blob/master/blackbox.yml | ||
+ | |||
+ | === reload === | ||
+ | <pre> | ||
+ | 检查配置文件 | ||
+ | bin/promtool check config /opt/prometheus-2.14/config/prometheus.yml | ||
+ | |||
+ | 重新加载配置文件 | ||
+ | curl -X POST httP://172.31.248.19:9090/-/reload | ||
+ | </pre> | ||
=grafana= | =grafana= | ||
+ | [https://www.oschina.net/news/213575/grafana-9-2-released Grafana 9.2 发布] | ||
<pre> | <pre> | ||
+ | 首次用户和密码为 admin admin | ||
+ | |||
+ | |||
#防火墙 grafana ng 在 10.3.10.10 | #防火墙 grafana ng 在 10.3.10.10 | ||
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.3.10.10" port protocol="tcp" port="3000" accept " | firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.3.10.10" port protocol="tcp" port="3000" accept " | ||
第64行: | 第171行: | ||
docker-grafana/config.env https://github.com/stefanwalther/docker-grafana/blob/master/config.env | docker-grafana/config.env https://github.com/stefanwalther/docker-grafana/blob/master/config.env | ||
+ | |||
+ | ==grafana 添加源和数据 == | ||
+ | |||
+ | [https://www.bookstack.cn/read/prometheus-book/quickstart-use-grafana-create-dashboard.md 使用Grafana创建可视化Dashboard] | ||
+ | |||
+ | ===Panel plugin not found: grafana-piechart-panel=== | ||
+ | <pre> | ||
+ | |||
+ | 升级为7.0.0就好了 不过要看其它模板还能用么 | ||
+ | |||
+ | #这个放弃了 是在 6.x系列的 还不好用 | ||
+ | grafana-cli plugins install grafana-piechart-panel | ||
+ | |||
+ | docker-compose restart grafana | ||
+ | |||
+ | |||
+ | 执行过程 | ||
+ | grafana-cli plugins install grafana-piechart-panel | ||
+ | installing grafana-piechart-panel @ 1.6.1 | ||
+ | from: https://grafana.com/api/plugins/grafana-piechart-panel/versions/1.6.1/download | ||
+ | into: /var/lib/grafana/plugins | ||
+ | |||
+ | ✔ Installed grafana-piechart-panel successfully | ||
+ | |||
+ | Restart grafana after installing plugins . <service grafana-server restart> | ||
+ | |||
+ | [https://github.com/unifi-poller/unifi-poller/issues/248 RESOLVED: Panel plugin not found: table-old ] | ||
+ | |||
+ | |||
+ | </pre> | ||
+ | [https://www.jianshu.com/p/153d944ab836 Panel plugin not found: grafana-piechart-panel] | ||
+ | |||
==grafana 添加用户 == | ==grafana 添加用户 == | ||
<pre> | <pre> | ||
+ | 0. 自己的话 在左下角就可以改了 | ||
+ | |||
1.Users--> 2.Invite --> | 1.Users--> 2.Invite --> | ||
第85行: | 第226行: | ||
http://localhost:3000/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa | http://localhost:3000/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa | ||
+ | |||
+ | http://a.linuxsa.org/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa | ||
--> | --> | ||
第95行: | 第238行: | ||
[https://blog.csdn.net/kozazyh/article/details/93759717?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.nonecase grafana-利用Google OAuth2 身份验证] | [https://blog.csdn.net/kozazyh/article/details/93759717?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.nonecase grafana-利用Google OAuth2 身份验证] | ||
+ | |||
+ | == ng config== | ||
+ | <pre> | ||
+ | cat gf.conf | ||
+ | server | ||
+ | { | ||
+ | listen 80; | ||
+ | server_name gf.linuxsa.org; | ||
+ | # access_log /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log; | ||
+ | #error_log /var/log/nginx/test-nux-bcb.evan.error.log; | ||
+ | |||
+ | |||
+ | client_max_body_size 10m; | ||
+ | |||
+ | location / { | ||
+ | proxy_pass http://10.3.10.124:3000; | ||
+ | proxy_http_version 1.1; | ||
+ | |||
+ | #set_real_ip_from 10.0.0.0/8; | ||
+ | #set_real_ip_from 172.16.0.0/12; | ||
+ | #set_real_ip_from 192.168.0.0/16; | ||
+ | #real_ip_header X-Forwarded-For; | ||
+ | |||
+ | proxy_set_header Host $host; | ||
+ | proxy_set_header X-Real-IP $remote_addr; | ||
+ | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; | ||
+ | proxy_set_header X-Remote-Port $remote_port; | ||
+ | proxy_set_header X-Forwarded-Proto $scheme; | ||
+ | proxy_redirect off; | ||
+ | |||
+ | proxy_connect_timeout 1800s; | ||
+ | proxy_send_timeout 1800s; | ||
+ | proxy_read_timeout 1800s; | ||
+ | |||
+ | |||
+ | |||
+ | real_ip_recursive on; | ||
+ | # real_ip_header X-Forwarded-For; | ||
+ | # set_real_ip_from 0.0.0.0/0; | ||
+ | # proxy_set_header Host $host; | ||
+ | # proxy_set_header X-Real-IP $remote_addr; | ||
+ | # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; | ||
+ | |||
+ | |||
+ | #proxy_set_header Host $host:$server_port; | ||
+ | #proxy_set_header HTTP_X_FORWARDED_FOR $remote_addr; | ||
+ | #proxy_set_header X-NginX-Proxy true; | ||
+ | |||
+ | # proxy_set_header X-Real-IP $remote_addr; | ||
+ | # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; | ||
+ | # proxy_set_header Upgrade $http_upgrade; | ||
+ | # proxy_set_header Connection "upgrade"; | ||
+ | } | ||
+ | } | ||
+ | |||
+ | </pre> | ||
==使用教程== | ==使用教程== | ||
+ | |||
+ | [https://mojotv.cn/2019/08/20/dash-graph-of-prometheus 使用Prometheus和Grafana做可视化大屏展示] | ||
+ | |||
+ | [https://www.cnblogs.com/cherylgi/p/13927231.html grafana +Prometheus API 使用 ] | ||
+ | |||
[https://yuerblog.cc/2019/01/04/grafana-usage/ grafana配置与简单使用] | [https://yuerblog.cc/2019/01/04/grafana-usage/ grafana配置与简单使用] | ||
第115行: | 第319行: | ||
− | + | [https://www.cnblogs.com/fatyao/p/11192234.html Grafana报警] | |
第145行: | 第349行: | ||
可以如上的 node | 可以如上的 node | ||
[[Node-exporter]] | [[Node-exporter]] | ||
+ | |||
+ | =Blackbox= | ||
+ | |||
+ | [[Blackbox exporter]] | ||
+ | |||
+ | =cadvisor= | ||
+ | <pre> | ||
+ | cadvisor docker container fails to start "mountpoint for cpu not found" | ||
+ | |||
+ | 用 Try replacing image by gcr.io/cadvisor/cadvisor | ||
+ | registry.cn-hangzhou.aliyuncs.com/evan886/my_k8s_containers:cadvisor2024 | ||
+ | |||
+ | |||
+ | |||
+ | https://github.com/google/cadvisor/issues/1943 | ||
+ | </pre> | ||
+ | |||
+ | =pushgateway= | ||
+ | |||
+ | [[pushgateway]] | ||
+ | |||
+ | [https://www.cnblogs.com/xiao987334176/p/9933963.html 基于Prometheus的Pushgateway实战 ] | ||
+ | |||
+ | https://songjiayang.gitbooks.io/prometheus/content/pushgateway/how.html | ||
+ | |||
+ | =[[Process-exporter]]进程监控= | ||
=应用监控= | =应用监控= | ||
+ | ==redis== | ||
[[prometheus监控Redis集群]] | [[prometheus监控Redis集群]] | ||
+ | |||
+ | [https://blog.csdn.net/weixin_43832846/article/details/90024274 pika主备部署(redis-sentinel)+监控(prometheus+grafana)] | ||
+ | |||
+ | [https://blog.csdn.net/weixin_34289744/article/details/92725774?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-16.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-16.nonecase Prometheus监控 Redis , redis-cli_exporter] | ||
+ | |||
+ | ==监控MQ== | ||
+ | [https://www.rabbitmq.com/prometheus.html Monitoring with Prometheus & Grafana ] | ||
+ | [https://blog.csdn.net/weixin_38937840/article/details/103144734 prometheus监控RabbitMQ] | ||
+ | |||
+ | [https://chenzhonzhou.github.io/2020/12/03/prometheus-jian-kong-rabbitmq/ Prometheus 监控 Rabbitmq] | ||
+ | |||
+ | |||
+ | |||
+ | [https://grafana.com/blog/2021/06/01/monitor-and-alert-on-essential-rabbitmq-cluster-metrics-with-the-new-grafana-cloud-integration/?src=ggl-s&mdm=cpc&camp=nb-rabbitmq-bmm&cnt=135709386153&trm=rabbitmq%20monitor&device=c&gclid=CjwKCAjwoduRBhA4EiwACL5RPyO-9bp3COrXnxBNs5VYyBYZAxt7p9jUPqueQ8m0Lz_cO1rkVYJ8cRoCs6AQAvD_BwE Monitor and alert on essential RabbitMQ cluster metrics with the new Grafana Cloud integration] | ||
+ | |||
+ | ==监控ES== | ||
+ | https://github.com/slok/ecs-exporter | ||
+ | ==监控docker== | ||
+ | 像普通export 一样 docker运行后 , | ||
+ | |||
+ | 与 prometheus 集成 | ||
+ | |||
+ | - job_name: docker | ||
+ | static_configs: | ||
+ | - targets: ['192.168.111.65:8088'] | ||
+ | |||
+ | |||
+ | [https://www.cnblogs.com/user-sunli/p/14738679.html Prometheus监控docker容器 还带报警呢] | ||
+ | |||
+ | [https://www.cnblogs.com/Dev0ps/p/10546276.html cAdvisor+Prometheus+Grafana监控docker ] | ||
+ | |||
+ | [https://blog.csdn.net/Sebastien23/article/details/113645177#t7 Prometheus+Grafana监控Docker容器和Linux主机] | ||
+ | |||
+ | [https://cloud.tencent.com/developer/article/1584309 docker监控: cAdvisor] | ||
+ | |||
+ | [https://blog.csdn.net/palet/article/details/82889493 Prometheus+grafana监控:cAdvisor输出的容器内存相关的指标的解读] | ||
+ | |||
+ | [https://www.cnblogs.com/edisonchou/p/docker_monitor_introduction_part2.html 你必须知道的容器监控 (2) cAdvisor ] | ||
+ | |||
+ | [https://www.jianshu.com/p/efc7c66638cc prometheus 监控Docker] | ||
+ | |||
+ | [https://www.cnblogs.com/xiangsikai/p/11289518.html Prometheus 监控Docker服务器及Granfanna可视化 ] | ||
+ | |||
+ | [https://www.jianshu.com/p/41831207bbf6 prometheus监控docker] | ||
+ | |||
+ | == mysql== | ||
+ | [https://blog.csdn.net/qq_25934401/article/details/82594478 prometheus 监控之 mysql 篇(含mysql报警规则)] | ||
+ | |||
+ | [https://jiajunhuang.com/articles/2018_12_16-prometheus_mysqld_exporter.md.html Prometheus MySQL Exporter源码阅读与分析] | ||
+ | |||
+ | [https://www.jianshu.com/p/27b979554ef8 python + promethues 监控mysql] | ||
+ | |||
+ | ==Spring boot== | ||
+ | https://www.bookstack.cn/read/prometheus_practice/application-spring.md | ||
+ | |||
+ | |||
+ | [https://blog.csdn.net/liangcha007/article/details/86699013#7.%E5%85%B3%E4%BA%8Epushgateway Prometheus基于java开发exporter样例以及pushgateway的应用] | ||
+ | |||
+ | =ng conf= | ||
+ | <pre> | ||
+ | cat alert.conf | ||
+ | server | ||
+ | { | ||
+ | listen 80; | ||
+ | server_name alert.com; | ||
+ | # access_log /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log; | ||
+ | #error_log /var/log/nginx/test-nux-bcb.evan.error.log; | ||
+ | |||
+ | |||
+ | client_max_body_size 10m; | ||
+ | |||
+ | location / { | ||
+ | # auth_basic "Prometheus"; | ||
+ | # auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd"; | ||
+ | proxy_pass http://10.3.10.124:9093; | ||
+ | proxy_http_version 1.1; | ||
+ | |||
+ | #s#et_real_ip_from 10.0.0.0/8; | ||
+ | #set_real_ip_from 172.16.0.0/12; | ||
+ | #set_real_ip_from 192.168.0.0/16; | ||
+ | #real_ip_header X-Forwarded-For; | ||
+ | proxy_set_header Host $host; | ||
+ | proxy_set_header X-Real-IP $remote_addr; | ||
+ | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; | ||
+ | proxy_set_header X-Remote-Port $remote_port; | ||
+ | proxy_set_header X-Forwarded-Proto $scheme; | ||
+ | proxy_redirect off; | ||
+ | proxy_connect_timeout 1800s; | ||
+ | proxy_send_timeout 1800s; | ||
+ | proxy_read_timeout 1800s; | ||
+ | |||
+ | real_ip_recursive on; | ||
+ | # real_ip_header X-Forwarded-For; | ||
+ | # set_real_ip_from 0.0.0.0/0; | ||
+ | # proxy_set_header Host $host; | ||
+ | # proxy_set_header X-Real-IP $remote_addr; | ||
+ | # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; | ||
+ | |||
+ | |||
+ | #proxy_set_header Host $host:$server_port; | ||
+ | #proxy_set_header HTTP_X_FORWARDED_FOR $remote_addr; | ||
+ | #proxy_set_header X-NginX-Proxy true; | ||
+ | |||
+ | # proxy_set_header X-Real-IP $remote_addr; | ||
+ | # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; | ||
+ | # proxy_set_header Upgrade $http_upgrade; | ||
+ | # proxy_set_header Connection "upgrade"; | ||
+ | } | ||
+ | } | ||
+ | |||
+ | |||
+ | |||
+ | cat pro.conf | ||
+ | server | ||
+ | { | ||
+ | listen 80; | ||
+ | server_name pro.linuxsa.org; | ||
+ | # access_log /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log; | ||
+ | #error_log /var/log/nginx/test-nux-bcb.evan.error.log; | ||
+ | |||
+ | |||
+ | client_max_body_size 10m; | ||
+ | |||
+ | location / { | ||
+ | auth_basic "Prometheus"; | ||
+ | auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd"; | ||
+ | proxy_pass http://10.3.10.124:9090; | ||
+ | proxy_http_version 1.1; | ||
+ | |||
+ | #set_real_ip_from 10.0.0.0/8; | ||
+ | #set_real_ip_from 172.16.0.0/12; | ||
+ | #set_real_ip_from 192.168.0.0/16; | ||
+ | #real_ip_header X-Forwarded-For; | ||
+ | |||
+ | proxy_set_header Host $host; | ||
+ | proxy_set_header X-Real-IP $remote_addr; | ||
+ | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; | ||
+ | proxy_set_header X-Remote-Port $remote_port; | ||
+ | proxy_set_header X-Forwarded-Proto $scheme; | ||
+ | proxy_redirect off; | ||
+ | |||
+ | proxy_connect_timeout 1800s; | ||
+ | proxy_send_timeout 1800s; | ||
+ | proxy_read_timeout 1800s; | ||
+ | |||
+ | |||
+ | |||
+ | real_ip_recursive on; | ||
+ | # real_ip_header X-Forwarded-For; | ||
+ | # set_real_ip_from 0.0.0.0/0; | ||
+ | # proxy_set_header Host $host; | ||
+ | # proxy_set_header X-Real-IP $remote_addr; | ||
+ | # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; | ||
+ | |||
+ | |||
+ | #proxy_set_header Host $host:$server_port; | ||
+ | #proxy_set_header HTTP_X_FORWARDED_FOR $remote_addr; | ||
+ | #proxy_set_header X-NginX-Proxy true; | ||
+ | |||
+ | # proxy_set_header X-Real-IP $remote_addr; | ||
+ | # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; | ||
+ | # proxy_set_header Upgrade $http_upgrade; | ||
+ | # proxy_set_header Connection "upgrade"; | ||
+ | } | ||
+ | } | ||
+ | |||
+ | |||
+ | cat blackbox.conf | ||
+ | server | ||
+ | { | ||
+ | listen 80; | ||
+ | server_name blackbox.com; | ||
+ | # access_log /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log; | ||
+ | #error_log /var/log/nginx/test-nux-bcb.evan.error.log; | ||
+ | client_max_body_size 10m; | ||
+ | location / { | ||
+ | # auth_basic "Prometheus"; | ||
+ | # auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd"; | ||
+ | proxy_pass http://10.3.10.124:9115; | ||
+ | proxy_http_version 1.1; | ||
+ | |||
+ | proxy_set_header Host $host; | ||
+ | proxy_set_header X-Real-IP $remote_addr; | ||
+ | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; | ||
+ | proxy_set_header X-Remote-Port $remote_port; | ||
+ | proxy_set_header X-Forwarded-Proto $scheme; | ||
+ | proxy_redirect off; | ||
+ | |||
+ | proxy_connect_timeout 1800s; | ||
+ | proxy_send_timeout 1800s; | ||
+ | proxy_read_timeout 1800s; | ||
+ | real_ip_recursive on; | ||
+ | } | ||
+ | } | ||
+ | |||
+ | |||
+ | |||
+ | </pre> | ||
+ | |||
+ | =[[Alertmanager]]= | ||
+ | |||
+ | =教程= | ||
+ | [https://www.cnblogs.com/caoweixiong/p/12736520.html Prometheus + Grafana(八)系统监控之Kafka ] | ||
+ | |||
+ | [https://www.cnblogs.com/caoweixiong/p/12736573.html Prometheus + Grafana(九)系统监控之RabbitMQ ] | ||
+ | |||
+ | |||
+ | |||
+ | [https://www.cnblogs.com/zhaojiedi1992/p/zhaojiedi_liunx_60_prometheus_config.html prometheus学习系列五: Prometheus配置文件 ] | ||
=see also= | =see also= | ||
+ | |||
+ | [https://www.cnblogs.com/hong-fithing/p/14695803.html Docker系列——Grafana+Prometheus+Node-exporter服务器监控平台(一) ] | ||
+ | |||
+ | [https://www.jianshu.com/p/93412a925da2 规划 Prometheus 的存储用量] | ||
+ | |||
+ | [https://blog.csdn.net/lemon89/article/details/104088184 深入理解监控系统——Prometheus核心特点] | ||
+ | |||
+ | [https://blog.csdn.net/qq_21816375/article/details/80462445?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-12.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-12.nonecase prometheus部署node,server以及域名加告警] | ||
+ | |||
+ | |||
+ | [https://zhuanlan.zhihu.com/p/56081874 通过gitlab-ci自动添加prometheus业务监控] | ||
+ | |||
+ | [http://www.linuxea.com/1881.html linuxea:gitlab-ci/cd Gitlab和Gitlab Prometheus监控(十二) ] | ||
+ | |||
+ | [https://www.cnblogs.com/chenqionghe/p/10494868.html 从零搭建Prometheus监控报警系统] | ||
+ | |||
+ | [https://www.jianshu.com/p/1f05476ebcee 使用prometheus自定义监控] | ||
+ | |||
[https://zhuanlan.zhihu.com/p/91215371 为什么说 Prometheus 是足以取代 Zabbix 的监控神器?] | [https://zhuanlan.zhihu.com/p/91215371 为什么说 Prometheus 是足以取代 Zabbix 的监控神器?] | ||
− | [[category:ops]] [[category:container]] | + | [https://blog.51cto.com/wzlinux/2487300 prometheus 监控概述(一)] |
+ | |||
+ | [https://my.oschina.net/54188zz/blog/3030618 基于prometheus + grafana + mysql + Telegram 监控告警] | ||
+ | |||
+ | [https://netsecurity.51cto.com/art/201912/608600.htm 为什么我们放弃Zabbix采用Prometheus] | ||
+ | |||
+ | [https://www.ctolib.com/topics-135208.html 容器监控利器 - prometheus在生产落地过程中的思考] | ||
+ | |||
+ | [https://blog.csdn.net/heian_99/article/details/103952955 Centos7安装Promethus(普罗米修斯)监控系统完整版] | ||
+ | |||
+ | [http://soa.ctocio.com.cn/soa/2020/0821/33627.html Prometheus 的局限 这 30 个坑点不要踩] | ||
+ | [[category:ops]] [[category:container]] [[category:prom]] |
2024年10月6日 (日) 02:29的最新版本
==
目录
book
进阶
以后全改成自己 docker-compose build
prometheus python 库编写自定义指标的方法(完整代码)
promethues + python + flask监控后端服务状态
服务发现
基于文件的服务发现 Use file-based service discovery to discover scrape targets
安装 node-exporter by ansible
https://github.com/evan886/my-ansible/tree/main/prome-node-add
配置
#这里以json格式为例子其实 另外一种 也可以用YAML格式的 cd apps/prom/prometheus sudo mkdir targets cat targets/nodes.json [{ "targets": [ "172.16.0.20:9100", "172.16.0.29:9100" ], "labels": { "server": "node_export_live" } }] vi prometheus.yml # - cadvisor:8080 - job_name: 'cadvisor-nodejs' static_configs: - targets: ['121.41.101.200:8080'] - job_name: 'nodes_server_discoverry' file_sd_configs: - files: - targets/*.json refresh_interval: 1m sudo curl -X POST http://172.16.0.42:9090/-/reload 解析如下 scrape_configs: - job_name: 'prometheus' static_configs: # 这个配置表示 静态发现 - targets: ['localhost:9090'] - job_name: 'node' # 增加一个节点,取名为 node metrics_path: "/metrics" # 获取数据的路径 http://10.0.20.12:9100/metrics file_sd_configs: # 这个配置表示通过文件发现 - files: - targets/nodes/*.json # 读取目录下的所有json结尾的文件 refresh_interval: 1m # 刷新频率,1 分钟 打开 https://prom.com/targets 有 job_name: 'nodes_server_discoverry' 就是成功
Use file-based service discovery to discover scrape targets
ins on docker-composer
git clone https://github.com/evan886/docker-compose-monitor.git
https://gitee.com/atompi/PrometheusStackDockerCompose
注意 grafana update 升级为gra 8.2.7 就行了 不行有 漏洞
docker-compose up -d grafana
安全
node exporter 添加防火墙 curl -Lo /etc/yum.repos.d/_copr_ibotty-prometheus-exporters.repo https://copr.fedorainfracloud.org/coprs/ibotty/prometheus-exporters/repo/epel-7/ibotty-prometheus-exporters-epel-7.repo # yum install node_exporter
prometheus
Prometheus/prəˈmiː.θi.əs/
Nginx用户密码认证配置 Basic HTTP authentication
不错的书 https://yunlzheng.gitbook.io/prometheus-book/
新型监控告警工具prometheus(普罗米修斯)的入门使用(附视频讲解)
【视频】新型监控告警工具prometheus(普罗米修斯)入门使用(附视频讲解)
为什么说 Prometheus 是足以取代 Zabbix 的监控神器?
usage
Grafana单图表告警邮件&面板图(Dashboard)日报邮件
https://github.com/prometheus/blackbox_exporter/blob/master/blackbox.yml
reload
检查配置文件 bin/promtool check config /opt/prometheus-2.14/config/prometheus.yml 重新加载配置文件 curl -X POST httP://172.31.248.19:9090/-/reload
grafana
首次用户和密码为 admin admin #防火墙 grafana ng 在 10.3.10.10 firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="10.3.10.10" port protocol="tcp" port="3000" accept " firewall-cmd --reload
grafana.com Run Grafana Docker image
中文 Node Exporte for Prometheus Dashboard
usage creat-->import --> id 8919
Container
https://grafana.com/grafana/dashboards/11174
docker-grafana/config.env https://github.com/stefanwalther/docker-grafana/blob/master/config.env
grafana 添加源和数据
Panel plugin not found: grafana-piechart-panel
升级为7.0.0就好了 不过要看其它模板还能用么 #这个放弃了 是在 6.x系列的 还不好用 grafana-cli plugins install grafana-piechart-panel docker-compose restart grafana 执行过程 grafana-cli plugins install grafana-piechart-panel installing grafana-piechart-panel @ 1.6.1 from: https://grafana.com/api/plugins/grafana-piechart-panel/versions/1.6.1/download into: /var/lib/grafana/plugins ✔ Installed grafana-piechart-panel successfully Restart grafana after installing plugins . <service grafana-server restart> [https://github.com/unifi-poller/unifi-poller/issues/248 RESOLVED: Panel plugin not found: table-old ]
Panel plugin not found: grafana-piechart-panel
grafana 添加用户
0. 自己的话 在左下角就可以改了 1.Users--> 2.Invite --> 3.进行邀请 Email or Username:输入用户邮箱,建议先跟用户名保持相同,然后在后面在写邮箱,后面会说到,这样不会报错 Name:输入用户名 Role:配置用户权限(只读Viewer 编辑Editor 管理员Admin) Send invite email:勾选时,会将新用户的设置密码的链接发到邮箱 点击创建之后回调到新用户设置密码 --> 4.创建好之后,点击Pending Invites 复制链接到浏览器的地址栏中访问(也可从邮件中收到链接) 进去之后是下面的界面,这里有个坑,复制的IP地址是localhost,我们这里改为自己搭建的IP地址,然后在Email填写邮箱地址,设置新用户的密码 设置完成后,会直接以新用户身份登陆 http://localhost:3000/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa http://a.linuxsa.org/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa -->
ng config
cat gf.conf server { listen 80; server_name gf.linuxsa.org; # access_log /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log; #error_log /var/log/nginx/test-nux-bcb.evan.error.log; client_max_body_size 10m; location / { proxy_pass http://10.3.10.124:3000; proxy_http_version 1.1; #set_real_ip_from 10.0.0.0/8; #set_real_ip_from 172.16.0.0/12; #set_real_ip_from 192.168.0.0/16; #real_ip_header X-Forwarded-For; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Remote-Port $remote_port; proxy_set_header X-Forwarded-Proto $scheme; proxy_redirect off; proxy_connect_timeout 1800s; proxy_send_timeout 1800s; proxy_read_timeout 1800s; real_ip_recursive on; # real_ip_header X-Forwarded-For; # set_real_ip_from 0.0.0.0/0; # proxy_set_header Host $host; # proxy_set_header X-Real-IP $remote_addr; # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; #proxy_set_header Host $host:$server_port; #proxy_set_header HTTP_X_FORWARDED_FOR $remote_addr; #proxy_set_header X-NginX-Proxy true; # proxy_set_header X-Real-IP $remote_addr; # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # proxy_set_header Upgrade $http_upgrade; # proxy_set_header Connection "upgrade"; } }
使用教程
grafana使用教程之基本概念(basic concepts)
Grafana快速入门:InfluxDB数据源以及曲线图表仪表盘配置
apached conf
##/etc/httpd/conf.d/gf.conf <VirtualHost *:80> ServerName gf.linuxsa.org ServerAlias gf.linuxsa.org ProxyRequests off <Proxy *> Order deny,allow Allow from all </Proxy> <Location /> ProxyPass http://localhost:3000/ ProxyPassReverse http://localhost:3000/ </Location> </VirtualHost>
vm监控Node-exporter
可以如上的 node Node-exporter
Blackbox
cadvisor
cadvisor docker container fails to start "mountpoint for cpu not found" 用 Try replacing image by gcr.io/cadvisor/cadvisor registry.cn-hangzhou.aliyuncs.com/evan886/my_k8s_containers:cadvisor2024 https://github.com/google/cadvisor/issues/1943
pushgateway
https://songjiayang.gitbooks.io/prometheus/content/pushgateway/how.html
Process-exporter进程监控
应用监控
redis
pika主备部署(redis-sentinel)+监控(prometheus+grafana)
Prometheus监控 Redis , redis-cli_exporter
监控MQ
Monitoring with Prometheus & Grafana prometheus监控RabbitMQ
Monitor and alert on essential RabbitMQ cluster metrics with the new Grafana Cloud integration
监控ES
https://github.com/slok/ecs-exporter
监控docker
像普通export 一样 docker运行后 ,
与 prometheus 集成
- job_name: docker static_configs: - targets: ['192.168.111.65:8088']
cAdvisor+Prometheus+Grafana监控docker
Prometheus+Grafana监控Docker容器和Linux主机
Prometheus+grafana监控:cAdvisor输出的容器内存相关的指标的解读
Prometheus 监控Docker服务器及Granfanna可视化
mysql
prometheus 监控之 mysql 篇(含mysql报警规则)
Prometheus MySQL Exporter源码阅读与分析
Spring boot
https://www.bookstack.cn/read/prometheus_practice/application-spring.md
Prometheus基于java开发exporter样例以及pushgateway的应用
ng conf
cat alert.conf server { listen 80; server_name alert.com; # access_log /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log; #error_log /var/log/nginx/test-nux-bcb.evan.error.log; client_max_body_size 10m; location / { # auth_basic "Prometheus"; # auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd"; proxy_pass http://10.3.10.124:9093; proxy_http_version 1.1; #s#et_real_ip_from 10.0.0.0/8; #set_real_ip_from 172.16.0.0/12; #set_real_ip_from 192.168.0.0/16; #real_ip_header X-Forwarded-For; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Remote-Port $remote_port; proxy_set_header X-Forwarded-Proto $scheme; proxy_redirect off; proxy_connect_timeout 1800s; proxy_send_timeout 1800s; proxy_read_timeout 1800s; real_ip_recursive on; # real_ip_header X-Forwarded-For; # set_real_ip_from 0.0.0.0/0; # proxy_set_header Host $host; # proxy_set_header X-Real-IP $remote_addr; # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; #proxy_set_header Host $host:$server_port; #proxy_set_header HTTP_X_FORWARDED_FOR $remote_addr; #proxy_set_header X-NginX-Proxy true; # proxy_set_header X-Real-IP $remote_addr; # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # proxy_set_header Upgrade $http_upgrade; # proxy_set_header Connection "upgrade"; } } cat pro.conf server { listen 80; server_name pro.linuxsa.org; # access_log /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log; #error_log /var/log/nginx/test-nux-bcb.evan.error.log; client_max_body_size 10m; location / { auth_basic "Prometheus"; auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd"; proxy_pass http://10.3.10.124:9090; proxy_http_version 1.1; #set_real_ip_from 10.0.0.0/8; #set_real_ip_from 172.16.0.0/12; #set_real_ip_from 192.168.0.0/16; #real_ip_header X-Forwarded-For; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Remote-Port $remote_port; proxy_set_header X-Forwarded-Proto $scheme; proxy_redirect off; proxy_connect_timeout 1800s; proxy_send_timeout 1800s; proxy_read_timeout 1800s; real_ip_recursive on; # real_ip_header X-Forwarded-For; # set_real_ip_from 0.0.0.0/0; # proxy_set_header Host $host; # proxy_set_header X-Real-IP $remote_addr; # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; #proxy_set_header Host $host:$server_port; #proxy_set_header HTTP_X_FORWARDED_FOR $remote_addr; #proxy_set_header X-NginX-Proxy true; # proxy_set_header X-Real-IP $remote_addr; # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # proxy_set_header Upgrade $http_upgrade; # proxy_set_header Connection "upgrade"; } } cat blackbox.conf server { listen 80; server_name blackbox.com; # access_log /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log; #error_log /var/log/nginx/test-nux-bcb.evan.error.log; client_max_body_size 10m; location / { # auth_basic "Prometheus"; # auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd"; proxy_pass http://10.3.10.124:9115; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Remote-Port $remote_port; proxy_set_header X-Forwarded-Proto $scheme; proxy_redirect off; proxy_connect_timeout 1800s; proxy_send_timeout 1800s; proxy_read_timeout 1800s; real_ip_recursive on; } }
Alertmanager
教程
Prometheus + Grafana(八)系统监控之Kafka
Prometheus + Grafana(九)系统监控之RabbitMQ
prometheus学习系列五: Prometheus配置文件
see also
Docker系列——Grafana+Prometheus+Node-exporter服务器监控平台(一)
prometheus部署node,server以及域名加告警
linuxea:gitlab-ci/cd Gitlab和Gitlab Prometheus监控(十二)
为什么说 Prometheus 是足以取代 Zabbix 的监控神器?
基于prometheus + grafana + mysql + Telegram 监控告警
容器监控利器 - prometheus在生产落地过程中的思考