“Monitoring with prometheus”的版本间的差异

来自linux中国网wiki
跳到导航 跳到搜索
 
（未显示同一用户的84个中间版本）
第15行： 第15行：
  
 
[https://www.jb51.net/article/189679.htm  prometheus python 库编写自定义指标的方法(完整代码)]
 
[https://www.jb51.net/article/189679.htm  prometheus python 库编写自定义指标的方法(完整代码)]
 +
 +
 +
[https://www.jianshu.com/p/19442595e886 promethues + python + flask监控后端服务状态]
 +
 +
[https://blog.csdn.net/weixin_33754913/article/details/91803530  prometheus python client]
 +
 +
[https://blog.csdn.net/JackLiu16/article/details/80381201?utm_medium=distribute.pc_relevant.none-task-blog-title-7&spm=1001.2101.3001.4242  使用Python和Flask编写Prometheus监控]
 +
 +
[https://blog.csdn.net/specter11235/article/details/87927202 python开发prometheus exporter]
 +
 +
==服务发现==
 +
=== 基于文件的服务发现 Use file-based service discovery to discover scrape targets===
 +
==== 安装 node-exporter by ansible====
 +
https://github.com/evan886/my-ansible/tree/main/prome-node-add
 +
 +
==== 配置  ====
 +
<pre>
 +
#这里以json格式为例子其实 另外一种 也可以用YAML格式的
 +
cd  apps/prom/prometheus
 +
sudo mkdir targets
 +
 +
cat targets/nodes.json
 +
[{
 +
  "targets": [
 +
    "172.16.0.20:9100",
 +
    "172.16.0.29:9100"
 +
  ],
 +
  "labels": {
 +
    "server": "node_export_live"
 +
  }
 +
}]
 +
 +
vi prometheus.yml
 +
 +
#          - cadvisor:8080
 +
 +
  - job_name: 'cadvisor-nodejs'
 +
    static_configs:
 +
    - targets: ['121.41.101.200:8080']
 +
 +
  - job_name: 'nodes_server_discoverry'
 +
    file_sd_configs:         
 +
    - files:
 +
      - targets/*.json   
 +
      refresh_interval: 1m   
 +
 +
 +
sudo  curl -X POST http://172.16.0.42:9090/-/reload   
 +
     
 +
解析如下
 +
 +
scrape_configs:
 +
  - job_name: 'prometheus'
 +
    static_configs:            # 这个配置表示 静态发现
 +
    - targets: ['localhost:9090']
 +
 +
  - job_name: 'node'            # 增加一个节点，取名为 node
 +
    metrics_path: "/metrics"    # 获取数据的路径 http://10.0.20.12:9100/metrics
 +
    file_sd_configs:            # 这个配置表示通过文件发现
 +
    - files:
 +
      - targets/nodes/*.json    # 读取目录下的所有json结尾的文件
 +
      refresh_interval: 1m      # 刷新频率，1 分钟
 +
   
 +
 +
 +
 +
打开
 +
https://prom.com/targets
 +
 +
有  job_name: 'nodes_server_discoverry' 就是成功
 +
</pre>
 +
[https://prometheus.io/docs/guides/file-sd/ Use file-based service discovery to discover scrape targets]
  
 
=ins on docker-composer=
 
=ins on docker-composer=
 +
git clone https://github.com/evan886/docker-compose-monitor.git
 +
 +
 
https://gitee.com/atompi/PrometheusStackDockerCompose
 
https://gitee.com/atompi/PrometheusStackDockerCompose
 +
注意 grafana update 升级为gra 8.2.7 就行了 不行有 漏洞
 +
 +
docker-compose  up -d  grafana
  
 
=安全=
 
=安全=
第28行： 第106行：
  
 
=prometheus=
 
=prometheus=
 +
 +
Prometheus/prəˈmiː.θi.əs/
 +
 +
 
[[Nginx用户密码认证配置 Basic HTTP authentication]]
 
[[Nginx用户密码认证配置 Basic HTTP authentication]]
  
第43行： 第125行：
  
 
[https://zhuanlan.zhihu.com/p/24811652 基于Prometheus的分布式在线服务监控实践]
 
[https://zhuanlan.zhihu.com/p/24811652 基于Prometheus的分布式在线服务监控实践]
 +
 +
[https://juejin.im/post/6844903866064977934  Prometheus学习系列（十三）之配置解析 ]
  
 
==usage==
 
==usage==
第50行： 第134行：
  
 
https://github.com/prometheus/blackbox_exporter/blob/master/blackbox.yml
 
https://github.com/prometheus/blackbox_exporter/blob/master/blackbox.yml
 +
 +
=== reload ===
 +
<pre>
 +
检查配置文件
 +
bin/promtool check config /opt/prometheus-2.14/config/prometheus.yml
 +
 +
重新加载配置文件
 +
curl -X POST httP://172.31.248.19:9090/-/reload
 +
</pre>
  
 
=grafana=
 
=grafana=
 +
[https://www.oschina.net/news/213575/grafana-9-2-released Grafana 9.2 发布]
 
<pre>
 
<pre>
 +
首次用户和密码为 admin admin
 +
 +
 
#防火墙  grafana ng 在 10.3.10.10
 
#防火墙  grafana ng 在 10.3.10.10
 
  firewall-cmd  --permanent --add-rich-rule="rule family="ipv4" source address="10.3.10.10" port protocol="tcp" port="3000" accept "
 
  firewall-cmd  --permanent --add-rich-rule="rule family="ipv4" source address="10.3.10.10" port protocol="tcp" port="3000" accept "
第78行： 第175行：
  
 
[https://www.bookstack.cn/read/prometheus-book/quickstart-use-grafana-create-dashboard.md 使用Grafana创建可视化Dashboard]
 
[https://www.bookstack.cn/read/prometheus-book/quickstart-use-grafana-create-dashboard.md 使用Grafana创建可视化Dashboard]
 +
 +
===Panel plugin not found: grafana-piechart-panel===
 +
<pre>
 +
 +
升级为7.0.0就好了 不过要看其它模板还能用么
 +
 +
#这个放弃了 是在 6.x系列的  还不好用
 +
grafana-cli plugins install grafana-piechart-panel
 +
 +
docker-compose  restart  grafana
 +
 +
 +
执行过程
 +
grafana-cli plugins install grafana-piechart-panel
 +
installing grafana-piechart-panel @ 1.6.1
 +
from: https://grafana.com/api/plugins/grafana-piechart-panel/versions/1.6.1/download
 +
into: /var/lib/grafana/plugins
 +
 +
✔ Installed grafana-piechart-panel successfully
 +
 +
Restart grafana after installing plugins . <service grafana-server restart>
 +
 +
[https://github.com/unifi-poller/unifi-poller/issues/248 RESOLVED: Panel plugin not found: table-old ]
 +
 +
 +
</pre>
 +
[https://www.jianshu.com/p/153d944ab836 Panel plugin not found: grafana-piechart-panel]
 +
 
==grafana 添加用户 ==
 
==grafana 添加用户 ==
 
<pre>
 
<pre>
 +
0. 自己的话 在左下角就可以改了
 +
 
1.Users-->  2.Invite -->  
 
1.Users-->  2.Invite -->  
  
第99行： 第226行：
  
 
http://localhost:3000/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa
 
http://localhost:3000/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa
 +
 +
http://a.linuxsa.org/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa
  
 
   -->
 
   -->
第109行： 第238行：
  
 
[https://blog.csdn.net/kozazyh/article/details/93759717?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.nonecase  grafana-利用Google OAuth2 身份验证]
 
[https://blog.csdn.net/kozazyh/article/details/93759717?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-2.nonecase  grafana-利用Google OAuth2 身份验证]
 +
 +
== ng config==
 +
<pre>
 +
cat gf.conf
 +
server
 +
{
 +
    listen 80;
 +
    server_name gf.linuxsa.org;
 +
#    access_log  /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
 +
    #error_log /var/log/nginx/test-nux-bcb.evan.error.log;
 +
 +
 +
    client_max_body_size 10m;
 +
 
 +
    location / {
 +
      proxy_pass http://10.3.10.124:3000;
 +
      proxy_http_version 1.1;
 +
 +
#set_real_ip_from 10.0.0.0/8;
 +
#set_real_ip_from 172.16.0.0/12;
 +
#set_real_ip_from 192.168.0.0/16;
 +
#real_ip_header X-Forwarded-For;
 +
 +
        proxy_set_header        Host                $host;
 +
        proxy_set_header        X-Real-IP            $remote_addr;
 +
        proxy_set_header        X-Forwarded-For      $proxy_add_x_forwarded_for;
 +
        proxy_set_header        X-Remote-Port        $remote_port;
 +
        proxy_set_header        X-Forwarded-Proto    $scheme;
 +
        proxy_redirect          off;
 +
 +
          proxy_connect_timeout 1800s;
 +
          proxy_send_timeout 1800s;
 +
          proxy_read_timeout 1800s;
 +
 +
 +
 +
      real_ip_recursive on;
 +
#      real_ip_header X-Forwarded-For;
 +
#      set_real_ip_from 0.0.0.0/0;
 +
#      proxy_set_header Host            $host;
 +
#      proxy_set_header X-Real-IP $remote_addr;
 +
#      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 +
 +
 +
#proxy_set_header Host            $host:$server_port;
 +
#proxy_set_header    HTTP_X_FORWARDED_FOR $remote_addr;
 +
#proxy_set_header X-NginX-Proxy true;
 +
 +
#      proxy_set_header X-Real-IP        $remote_addr;
 +
#    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 +
#      proxy_set_header Upgrade $http_upgrade;
 +
#      proxy_set_header Connection "upgrade";
 +
    }
 +
}
 +
 +
</pre>
  
 
==使用教程==
 
==使用教程==
 +
 +
[https://mojotv.cn/2019/08/20/dash-graph-of-prometheus 使用Prometheus和Grafana做可视化大屏展示]
 +
 +
[https://www.cnblogs.com/cherylgi/p/13927231.html  grafana +Prometheus API 使用 ]
 +
 
[https://yuerblog.cc/2019/01/04/grafana-usage/ grafana配置与简单使用]
 
[https://yuerblog.cc/2019/01/04/grafana-usage/ grafana配置与简单使用]
  
第129行： 第319行：
  
  
−
 
+
[https://www.cnblogs.com/fatyao/p/11192234.html Grafana报警]
  
  
第163行： 第353行：
  
 
[[Blackbox exporter]]
 
[[Blackbox exporter]]
 +
 +
=cadvisor=
 +
<pre>
 +
cadvisor docker container fails to start "mountpoint for cpu not found"
 +
 +
用 Try replacing image by gcr.io/cadvisor/cadvisor
 +
registry.cn-hangzhou.aliyuncs.com/evan886/my_k8s_containers:cadvisor2024
 +
 +
 +
 +
https://github.com/google/cadvisor/issues/1943
 +
</pre>
  
 
=pushgateway=
 
=pushgateway=
第172行： 第374行：
 
https://songjiayang.gitbooks.io/prometheus/content/pushgateway/how.html
 
https://songjiayang.gitbooks.io/prometheus/content/pushgateway/how.html
  
−
=[[Process-exporter]]=
+
=[[Process-exporter]]进程监控=
  
 
=应用监控=
 
=应用监控=
第184行： 第386行：
 
==监控MQ==
 
==监控MQ==
 
[https://www.rabbitmq.com/prometheus.html  Monitoring with Prometheus & Grafana ]
 
[https://www.rabbitmq.com/prometheus.html  Monitoring with Prometheus & Grafana ]
 +
[https://blog.csdn.net/weixin_38937840/article/details/103144734  prometheus监控RabbitMQ]
 +
 +
[https://chenzhonzhou.github.io/2020/12/03/prometheus-jian-kong-rabbitmq/ Prometheus 监控 Rabbitmq]
 +
 +
 +
 +
[https://grafana.com/blog/2021/06/01/monitor-and-alert-on-essential-rabbitmq-cluster-metrics-with-the-new-grafana-cloud-integration/?src=ggl-s&mdm=cpc&camp=nb-rabbitmq-bmm&cnt=135709386153&trm=rabbitmq%20monitor&device=c&gclid=CjwKCAjwoduRBhA4EiwACL5RPyO-9bp3COrXnxBNs5VYyBYZAxt7p9jUPqueQ8m0Lz_cO1rkVYJ8cRoCs6AQAvD_BwE Monitor and alert on essential RabbitMQ cluster metrics with the new Grafana Cloud integration]
  
 
==监控ES==
 
==监控ES==
 
https://github.com/slok/ecs-exporter
 
https://github.com/slok/ecs-exporter
−
==docker==
+
==监控docker==
−
[https://blog.csdn.net/palet/article/details/82889493?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-7.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-7.nonecase Prometheus+grafana监控：cAdvisor输出的容器内存相关的指标的解读]
+
像普通export 一样 docker运行后  ,
 +
 
 +
与 prometheus 集成
 +
 
 +
  - job_name: docker
 +
    static_configs:
 +
    - targets: ['192.168.111.65:8088']
 +
 
 +
 
 +
[https://www.cnblogs.com/user-sunli/p/14738679.html  Prometheus监控docker容器 还带报警呢]
 +
 
 +
[https://www.cnblogs.com/Dev0ps/p/10546276.html  cAdvisor+Prometheus+Grafana监控docker ]
 +
 
 +
[https://blog.csdn.net/Sebastien23/article/details/113645177#t7  Prometheus+Grafana监控Docker容器和Linux主机]
 +
 
 +
[https://cloud.tencent.com/developer/article/1584309 docker监控: cAdvisor]
 +
 
 +
[https://blog.csdn.net/palet/article/details/82889493 Prometheus+grafana监控：cAdvisor输出的容器内存相关的指标的解读]
 +
 
 +
[https://www.cnblogs.com/edisonchou/p/docker_monitor_introduction_part2.html  你必须知道的容器监控 (2) cAdvisor ]
 +
 
 +
[https://www.jianshu.com/p/efc7c66638cc prometheus 监控Docker]
 +
 
 +
[https://www.cnblogs.com/xiangsikai/p/11289518.html Prometheus 监控Docker服务器及Granfanna可视化 ]
 +
 
 +
[https://www.jianshu.com/p/41831207bbf6 prometheus监控docker]
  
 
== mysql==
 
== mysql==
第194行： 第428行：
  
 
[https://jiajunhuang.com/articles/2018_12_16-prometheus_mysqld_exporter.md.html Prometheus MySQL Exporter源码阅读与分析]
 
[https://jiajunhuang.com/articles/2018_12_16-prometheus_mysqld_exporter.md.html Prometheus MySQL Exporter源码阅读与分析]
 +
 +
[https://www.jianshu.com/p/27b979554ef8 python + promethues 监控mysql]
  
 
==Spring boot==
 
==Spring boot==
第201行： 第437行：
 
[https://blog.csdn.net/liangcha007/article/details/86699013#7.%E5%85%B3%E4%BA%8Epushgateway  Prometheus基于java开发exporter样例以及pushgateway的应用]
 
[https://blog.csdn.net/liangcha007/article/details/86699013#7.%E5%85%B3%E4%BA%8Epushgateway  Prometheus基于java开发exporter样例以及pushgateway的应用]
  
−
=告警=
+
=ng conf=
 +
<pre>
 +
cat  alert.conf
 +
server
 +
{
 +
    listen 80;
 +
    server_name alert.com;
 +
#    access_log  /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
 +
    #error_log /var/log/nginx/test-nux-bcb.evan.error.log;
 +
 
 +
 
 +
    client_max_body_size 10m;
 +
 
 +
    location / {
 +
  #      auth_basic "Prometheus";
 +
  #      auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
 +
      proxy_pass http://10.3.10.124:9093;
 +
      proxy_http_version 1.1;
 +
 
 +
#s#et_real_ip_from 10.0.0.0/8;
 +
#set_real_ip_from 172.16.0.0/12;
 +
#set_real_ip_from 192.168.0.0/16;
 +
#real_ip_header X-Forwarded-For;
 +
        proxy_set_header        Host                $host;
 +
        proxy_set_header        X-Real-IP            $remote_addr;
 +
        proxy_set_header        X-Forwarded-For      $proxy_add_x_forwarded_for;
 +
        proxy_set_header        X-Remote-Port        $remote_port;
 +
        proxy_set_header        X-Forwarded-Proto    $scheme;
 +
        proxy_redirect          off;
 +
          proxy_connect_timeout 1800s;
 +
          proxy_send_timeout 1800s;
 +
          proxy_read_timeout 1800s;
 +
 
 +
      real_ip_recursive on;
 +
#      real_ip_header X-Forwarded-For;
 +
#      set_real_ip_from 0.0.0.0/0;
 +
#      proxy_set_header Host            $host;
 +
#      proxy_set_header X-Real-IP $remote_addr;
 +
#      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  
−
[[Alertmanager]]
 
  
−
==telegram==
+
#proxy_set_header Host            $host:$server_port;
−
通过 prome or grafana ?
+
#proxy_set_header    HTTP_X_FORWARDED_FOR $remote_addr;
 +
#proxy_set_header X-NginX-Proxy true;
  
 +
#      proxy_set_header X-Real-IP        $remote_addr;
 +
#    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 +
#      proxy_set_header Upgrade $http_upgrade;
 +
#      proxy_set_header Connection "upgrade";
 +
    }
 +
}
  
−
[https://blog.csdn.net/weixin_34242331/article/details/91875514 基于prometheus + grafana + mysql + Telegram 监控告警]
+
 
−
https://my.oschina.net/54188zz/blog/3030618
+
 
 +
cat  pro.conf
 +
server
 +
{
 +
    listen 80;
 +
    server_name pro.linuxsa.org;
 +
#    access_log  /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
 +
    #error_log /var/log/nginx/test-nux-bcb.evan.error.log;
 +
 
 +
 
 +
    client_max_body_size 10m;
 +
 
 +
    location / {
 +
        auth_basic "Prometheus";
 +
        auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
 +
      proxy_pass http://10.3.10.124:9090;
 +
      proxy_http_version 1.1;
 +
 
 +
#set_real_ip_from 10.0.0.0/8;
 +
#set_real_ip_from 172.16.0.0/12;
 +
#set_real_ip_from 192.168.0.0/16;
 +
#real_ip_header X-Forwarded-For;
 +
 
 +
        proxy_set_header        Host                $host;
 +
        proxy_set_header        X-Real-IP            $remote_addr;
 +
        proxy_set_header        X-Forwarded-For      $proxy_add_x_forwarded_for;
 +
        proxy_set_header        X-Remote-Port        $remote_port;
 +
        proxy_set_header        X-Forwarded-Proto    $scheme;
 +
        proxy_redirect          off;
 +
 
 +
          proxy_connect_timeout 1800s;
 +
          proxy_send_timeout 1800s;
 +
          proxy_read_timeout 1800s;
 +
 
 +
 
 +
 
 +
      real_ip_recursive on;
 +
#      real_ip_header X-Forwarded-For;
 +
#      set_real_ip_from 0.0.0.0/0;
 +
#      proxy_set_header Host            $host;
 +
#      proxy_set_header X-Real-IP $remote_addr;
 +
#      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 +
 
 +
 
 +
#proxy_set_header Host            $host:$server_port;
 +
#proxy_set_header    HTTP_X_FORWARDED_FOR $remote_addr;
 +
#proxy_set_header X-NginX-Proxy true;
 +
 
 +
#      proxy_set_header X-Real-IP        $remote_addr;
 +
  #    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 +
#      proxy_set_header Upgrade $http_upgrade;
 +
#      proxy_set_header Connection "upgrade";
 +
    }
 +
}
 +
 
 +
 
 +
cat blackbox.conf
 +
server
 +
{
 +
    listen 80;
 +
    server_name blackbox.com;
 +
#    access_log  /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
 +
    #error_log /var/log/nginx/test-nux-bcb.evan.error.log;
 +
    client_max_body_size 10m;
 +
    location / {
 +
  #      auth_basic "Prometheus";
 +
  #      auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
 +
      proxy_pass http://10.3.10.124:9115;
 +
      proxy_http_version 1.1;
 +
 
 +
        proxy_set_header        Host                $host;
 +
        proxy_set_header        X-Real-IP            $remote_addr;
 +
        proxy_set_header        X-Forwarded-For      $proxy_add_x_forwarded_for;
 +
        proxy_set_header        X-Remote-Port        $remote_port;
 +
        proxy_set_header        X-Forwarded-Proto    $scheme;
 +
        proxy_redirect          off;
 +
 
 +
          proxy_connect_timeout 1800s;
 +
          proxy_send_timeout 1800s;
 +
          proxy_read_timeout 1800s;
 +
      real_ip_recursive on;
 +
    }
 +
}
 +
 
 +
 
 +
 
 +
</pre>
 +
 
 +
=[[Alertmanager]]=
  
 
=教程=
 
=教程=
 +
[https://www.cnblogs.com/caoweixiong/p/12736520.html  Prometheus + Grafana（八）系统监控之Kafka ]
 +
 +
[https://www.cnblogs.com/caoweixiong/p/12736573.html  Prometheus + Grafana（九）系统监控之RabbitMQ ]
 +
 +
 +
 
[https://www.cnblogs.com/zhaojiedi1992/p/zhaojiedi_liunx_60_prometheus_config.html  prometheus学习系列五： Prometheus配置文件 ]
 
[https://www.cnblogs.com/zhaojiedi1992/p/zhaojiedi_liunx_60_prometheus_config.html  prometheus学习系列五： Prometheus配置文件 ]
  
 
=see also=
 
=see also=
 +
 +
[https://www.cnblogs.com/hong-fithing/p/14695803.html  Docker系列——Grafana+Prometheus+Node-exporter服务器监控平台（一） ]
  
 
[https://www.jianshu.com/p/93412a925da2 规划 Prometheus 的存储用量]
 
[https://www.jianshu.com/p/93412a925da2 规划 Prometheus 的存储用量]
第222行： 第598行：
  
 
[https://blog.csdn.net/qq_21816375/article/details/80462445?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-12.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-12.nonecase  prometheus部署node,server以及域名加告警]
 
[https://blog.csdn.net/qq_21816375/article/details/80462445?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-12.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-12.nonecase  prometheus部署node,server以及域名加告警]
 +
 +
 +
[https://zhuanlan.zhihu.com/p/56081874 通过gitlab-ci自动添加prometheus业务监控]
 +
 +
[http://www.linuxea.com/1881.html linuxea:gitlab-ci/cd Gitlab和Gitlab Prometheus监控(十二) ]
  
 
[https://www.cnblogs.com/chenqionghe/p/10494868.html 从零搭建Prometheus监控报警系统]
 
[https://www.cnblogs.com/chenqionghe/p/10494868.html 从零搭建Prometheus监控报警系统]
第231行： 第612行：
 
[https://blog.51cto.com/wzlinux/2487300 prometheus 监控概述(一)]
 
[https://blog.51cto.com/wzlinux/2487300 prometheus 监控概述(一)]
  
 +
[https://my.oschina.net/54188zz/blog/3030618 基于prometheus + grafana + mysql + Telegram 监控告警]
 +
 +
[https://netsecurity.51cto.com/art/201912/608600.htm 为什么我们放弃Zabbix采用Prometheus]
 +
 +
[https://www.ctolib.com/topics-135208.html  容器监控利器 - prometheus在生产落地过程中的思考]
  
 +
[https://blog.csdn.net/heian_99/article/details/103952955  Centos7安装Promethus(普罗米修斯）监控系统完整版]
  
 +
[http://soa.ctocio.com.cn/soa/2020/0821/33627.html Prometheus 的局限 这 30 个坑点不要踩]
 
[[category:ops]] [[category:container]] [[category:prom]]
 
[[category:ops]] [[category:container]] [[category:prom]]

2024年10月6日 (日) 02:29的最新版本

==


book

Prometheus操作指南

进阶

以后全改成自己 docker-compose build

prometheus 监控概述(一)有常用监控警告

PromQL 内置函数(三)


prometheus python 库编写自定义指标的方法(完整代码)


promethues + python + flask监控后端服务状态

prometheus python client

使用Python和Flask编写Prometheus监控

python开发prometheus exporter

服务发现

基于文件的服务发现 Use file-based service discovery to discover scrape targets

安装 node-exporter by ansible

https://github.com/evan886/my-ansible/tree/main/prome-node-add

配置

#这里以json格式为例子其实 另外一种 也可以用YAML格式的
cd  apps/prom/prometheus
sudo mkdir targets

 cat targets/nodes.json 
[{
  "targets": [
    "172.16.0.20:9100",
    "172.16.0.29:9100"
  ],
  "labels": {
    "server": "node_export_live"
  }
}]

vi prometheus.yml

#           - cadvisor:8080

  - job_name: 'cadvisor-nodejs'
    static_configs:
    - targets: ['121.41.101.200:8080']

  - job_name: 'nodes_server_discoverry'
    file_sd_configs:           
    - files:
      - targets/*.json    
      refresh_interval: 1m    
 
 
 sudo   curl -X POST http://172.16.0.42:9090/-/reload     
      
 解析如下 
 
 scrape_configs:
  - job_name: 'prometheus'
    static_configs:             # 这个配置表示 静态发现
    - targets: ['localhost:9090']

  - job_name: 'node'            # 增加一个节点，取名为 node
    metrics_path: "/metrics"    # 获取数据的路径 http://10.0.20.12:9100/metrics
    file_sd_configs:            # 这个配置表示通过文件发现
    - files:
      - targets/nodes/*.json    # 读取目录下的所有json结尾的文件
      refresh_interval: 1m      # 刷新频率，1 分钟
     



打开 
https://prom.com/targets

有  job_name: 'nodes_server_discoverry' 就是成功

Use file-based service discovery to discover scrape targets

ins on docker-composer

git clone https://github.com/evan886/docker-compose-monitor.git


https://gitee.com/atompi/PrometheusStackDockerCompose 

注意 grafana update 升级为gra 8.2.7 就行了 不行有 漏洞 

docker-compose  up -d  grafana

安全

 node exporter 添加防火墙

curl -Lo /etc/yum.repos.d/_copr_ibotty-prometheus-exporters.repo https://copr.fedorainfracloud.org/coprs/ibotty/prometheus-exporters/repo/epel-7/ibotty-prometheus-exporters-epel-7.repo
# yum install node_exporter

prometheus

Prometheus/prəˈmiː.θi.əs/


Nginx用户密码认证配置 Basic HTTP authentication

不错的书 https://yunlzheng.gitbook.io/prometheus-book/



新型监控告警工具prometheus（普罗米修斯）的入门使用（附视频讲解）

【视频】新型监控告警工具prometheus（普罗米修斯）入门使用（附视频讲解）

为什么说 Prometheus 是足以取代 Zabbix 的监控神器？

基于Prometheus的分布式在线服务监控实践

Prometheus学习系列（十三）之配置解析

usage

Grafana单图表告警邮件&面板图（Dashboard）日报邮件

Recording rules

https://github.com/prometheus/blackbox_exporter/blob/master/blackbox.yml

reload

检查配置文件
bin/promtool check config /opt/prometheus-2.14/config/prometheus.yml

 重新加载配置文件
 curl -X POST httP://172.31.248.19:9090/-/reload

grafana

Grafana 9.2 发布 

首次用户和密码为 admin admin


#防火墙   grafana ng 在 10.3.10.10
 firewall-cmd   --permanent --add-rich-rule="rule family="ipv4" source address="10.3.10.10" port protocol="tcp" port="3000" accept "
firewall-cmd --reload


grafana.com Run Grafana Docker image

中文 Node Exporte for Prometheus Dashboard

usage creat-->import --> id 8919

Container


https://grafana.com/grafana/dashboards/11174 


docker-grafana/config.env  https://github.com/stefanwalther/docker-grafana/blob/master/config.env

grafana 添加源和数据

使用Grafana创建可视化Dashboard

Panel plugin not found: grafana-piechart-panel


升级为7.0.0就好了 不过要看其它模板还能用么 

#这个放弃了 是在 6.x系列的  还不好用 
grafana-cli plugins install grafana-piechart-panel

docker-compose  restart  grafana 


执行过程
 grafana-cli plugins install grafana-piechart-panel
installing grafana-piechart-panel @ 1.6.1
from: https://grafana.com/api/plugins/grafana-piechart-panel/versions/1.6.1/download
into: /var/lib/grafana/plugins

✔ Installed grafana-piechart-panel successfully 

Restart grafana after installing plugins . <service grafana-server restart>

[https://github.com/unifi-poller/unifi-poller/issues/248 RESOLVED: Panel plugin not found: table-old ]

Panel plugin not found: grafana-piechart-panel

grafana 添加用户

0. 自己的话 在左下角就可以改了 

1.Users-->  2.Invite --> 

3.进行邀请
Email or Username:输入用户邮箱，建议先跟用户名保持相同，然后在后面在写邮箱，后面会说到，这样不会报错
Name：输入用户名
Role:配置用户权限(只读Viewer 编辑Editor 管理员Admin)
Send invite email：勾选时，会将新用户的设置密码的链接发到邮箱

点击创建之后回调到新用户设置密码

 -->

4.创建好之后，点击Pending Invites
复制链接到浏览器的地址栏中访问（也可从邮件中收到链接）
进去之后是下面的界面，这里有个坑，复制的IP地址是localhost，我们这里改为自己搭建的IP地址，然后在Email填写邮箱地址，设置新用户的密码
设置完成后，会直接以新用户身份登陆


http://localhost:3000/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa

http://a.linuxsa.org/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa

  -->

grafana的用户及权限的配置

grafana的用户及权限的配置

grafana-利用Google OAuth2 身份验证

ng config

cat gf.conf 
server
{
    listen 80;
    server_name gf.linuxsa.org;
#    access_log  /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
    #error_log /var/log/nginx/test-nux-bcb.evan.error.log;


    client_max_body_size 10m;
  
    location / {
      proxy_pass http://10.3.10.124:3000;
      proxy_http_version 1.1;

#set_real_ip_from 10.0.0.0/8;
#set_real_ip_from 172.16.0.0/12;
#set_real_ip_from 192.168.0.0/16;
#real_ip_header X-Forwarded-For;

        proxy_set_header        Host                 $host;
        proxy_set_header        X-Real-IP            $remote_addr;
        proxy_set_header        X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header        X-Remote-Port        $remote_port;
        proxy_set_header        X-Forwarded-Proto    $scheme;
        proxy_redirect          off;

          proxy_connect_timeout 1800s;
          proxy_send_timeout 1800s;
          proxy_read_timeout 1800s;



      real_ip_recursive on;
#      real_ip_header X-Forwarded-For;
#      set_real_ip_from 0.0.0.0/0;
#      proxy_set_header Host             $host;
#      proxy_set_header X-Real-IP $remote_addr;
#      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


#proxy_set_header Host             $host:$server_port;
#proxy_set_header    HTTP_X_FORWARDED_FOR $remote_addr;
#proxy_set_header X-NginX-Proxy true;

#      proxy_set_header X-Real-IP        $remote_addr;
 #     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#      proxy_set_header Upgrade $http_upgrade;
#      proxy_set_header Connection "upgrade";
    }
}

使用教程

使用Prometheus和Grafana做可视化大屏展示

grafana +Prometheus API 使用

grafana配置与简单使用

Grafana使用教程

grafana数据可视化-使用教程

grafana快速入门

grafana使用教程之基本概念（basic concepts）

Grafana从无到有的一个相当粗略的资料总结


grafana数据可视化-使用教程

Grafana快速入门：InfluxDB数据源以及曲线图表仪表盘配置


Grafana报警 


apached conf

##/etc/httpd/conf.d/gf.conf


<VirtualHost *:80>
      ServerName gf.linuxsa.org
      ServerAlias gf.linuxsa.org
     
      ProxyRequests off
     
      <Proxy *>
        Order deny,allow
        Allow from all
      </Proxy>
     
      <Location />
        ProxyPass http://localhost:3000/
        ProxyPassReverse http://localhost:3000/
      </Location>
    </VirtualHost>

vm监控Node-exporter

可以如上的 node Node-exporter

Blackbox

Blackbox exporter

cadvisor

cadvisor docker container fails to start "mountpoint for cpu not found"

用 Try replacing image by gcr.io/cadvisor/cadvisor
registry.cn-hangzhou.aliyuncs.com/evan886/my_k8s_containers:cadvisor2024



https://github.com/google/cadvisor/issues/1943

pushgateway

pushgateway

基于Prometheus的Pushgateway实战

https://songjiayang.gitbooks.io/prometheus/content/pushgateway/how.html

Process-exporter进程监控

应用监控

redis

prometheus监控Redis集群

pika主备部署（redis-sentinel）+监控（prometheus+grafana）

Prometheus监控 Redis , redis-cli_exporter

监控MQ

Monitoring with Prometheus & Grafana prometheus监控RabbitMQ

Prometheus 监控 Rabbitmq


Monitor and alert on essential RabbitMQ cluster metrics with the new Grafana Cloud integration

监控ES

https://github.com/slok/ecs-exporter

监控docker

像普通export 一样 docker运行后 ,

与 prometheus 集成 

 - job_name: docker
   static_configs:
   - targets: ['192.168.111.65:8088']


Prometheus监控docker容器 还带报警呢

cAdvisor+Prometheus+Grafana监控docker

Prometheus+Grafana监控Docker容器和Linux主机

docker监控: cAdvisor

Prometheus+grafana监控：cAdvisor输出的容器内存相关的指标的解读

你必须知道的容器监控 (2) cAdvisor

prometheus 监控Docker

Prometheus 监控Docker服务器及Granfanna可视化

prometheus监控docker

mysql

prometheus 监控之 mysql 篇（含mysql报警规则）

Prometheus MySQL Exporter源码阅读与分析

python + promethues 监控mysql

Spring boot

https://www.bookstack.cn/read/prometheus_practice/application-spring.md


Prometheus基于java开发exporter样例以及pushgateway的应用

ng conf

cat  alert.conf 
server
{
    listen 80;
    server_name alert.com;
#    access_log  /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
    #error_log /var/log/nginx/test-nux-bcb.evan.error.log;


    client_max_body_size 10m;
  
    location / {
  #      auth_basic "Prometheus";
  #      auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
      proxy_pass http://10.3.10.124:9093;
      proxy_http_version 1.1;

#s#et_real_ip_from 10.0.0.0/8;
#set_real_ip_from 172.16.0.0/12;
#set_real_ip_from 192.168.0.0/16;
#real_ip_header X-Forwarded-For;
        proxy_set_header        Host                 $host;
        proxy_set_header        X-Real-IP            $remote_addr;
        proxy_set_header        X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header        X-Remote-Port        $remote_port;
        proxy_set_header        X-Forwarded-Proto    $scheme;
        proxy_redirect          off;
          proxy_connect_timeout 1800s;
          proxy_send_timeout 1800s;
          proxy_read_timeout 1800s;

      real_ip_recursive on;
#      real_ip_header X-Forwarded-For;
#      set_real_ip_from 0.0.0.0/0;
#      proxy_set_header Host             $host;
#      proxy_set_header X-Real-IP $remote_addr;
#      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


#proxy_set_header Host             $host:$server_port;
#proxy_set_header    HTTP_X_FORWARDED_FOR $remote_addr;
#proxy_set_header X-NginX-Proxy true;

#      proxy_set_header X-Real-IP        $remote_addr;
 #     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#      proxy_set_header Upgrade $http_upgrade;
#      proxy_set_header Connection "upgrade";
    }
}



cat  pro.conf 
server
{
    listen 80;
    server_name pro.linuxsa.org;
#    access_log  /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
    #error_log /var/log/nginx/test-nux-bcb.evan.error.log;


    client_max_body_size 10m;
  
    location / {
        auth_basic "Prometheus";
        auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
      proxy_pass http://10.3.10.124:9090;
      proxy_http_version 1.1;

#set_real_ip_from 10.0.0.0/8;
#set_real_ip_from 172.16.0.0/12;
#set_real_ip_from 192.168.0.0/16;
#real_ip_header X-Forwarded-For;

        proxy_set_header        Host                 $host;
        proxy_set_header        X-Real-IP            $remote_addr;
        proxy_set_header        X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header        X-Remote-Port        $remote_port;
        proxy_set_header        X-Forwarded-Proto    $scheme;
        proxy_redirect          off;

          proxy_connect_timeout 1800s;
          proxy_send_timeout 1800s;
          proxy_read_timeout 1800s;



      real_ip_recursive on;
#      real_ip_header X-Forwarded-For;
#      set_real_ip_from 0.0.0.0/0;
#      proxy_set_header Host             $host;
#      proxy_set_header X-Real-IP $remote_addr;
#      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


#proxy_set_header Host             $host:$server_port;
#proxy_set_header    HTTP_X_FORWARDED_FOR $remote_addr;
#proxy_set_header X-NginX-Proxy true;

#      proxy_set_header X-Real-IP        $remote_addr;
 #     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#      proxy_set_header Upgrade $http_upgrade;
#      proxy_set_header Connection "upgrade";
    }
}


cat blackbox.conf 
server
{
    listen 80;
    server_name blackbox.com;
#    access_log  /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
    #error_log /var/log/nginx/test-nux-bcb.evan.error.log;
    client_max_body_size 10m;
    location / {
  #      auth_basic "Prometheus";
  #      auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
      proxy_pass http://10.3.10.124:9115;
      proxy_http_version 1.1;

        proxy_set_header        Host                 $host;
        proxy_set_header        X-Real-IP            $remote_addr;
        proxy_set_header        X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header        X-Remote-Port        $remote_port;
        proxy_set_header        X-Forwarded-Proto    $scheme;
        proxy_redirect          off;

          proxy_connect_timeout 1800s;
          proxy_send_timeout 1800s;
          proxy_read_timeout 1800s;
      real_ip_recursive on;
    }
}

Alertmanager

教程

Prometheus + Grafana（八）系统监控之Kafka

Prometheus + Grafana（九）系统监控之RabbitMQ


prometheus学习系列五： Prometheus配置文件

see also

Docker系列——Grafana+Prometheus+Node-exporter服务器监控平台（一）

规划 Prometheus 的存储用量

深入理解监控系统——Prometheus核心特点

prometheus部署node,server以及域名加告警


通过gitlab-ci自动添加prometheus业务监控

linuxea:gitlab-ci/cd Gitlab和Gitlab Prometheus监控(十二)

从零搭建Prometheus监控报警系统

使用prometheus自定义监控

为什么说 Prometheus 是足以取代 Zabbix 的监控神器？

prometheus 监控概述(一)

基于prometheus + grafana + mysql + Telegram 监控告警

为什么我们放弃Zabbix采用Prometheus

容器监控利器 - prometheus在生产落地过程中的思考

Centos7安装Promethus(普罗米修斯）监控系统完整版

Prometheus 的局限 这 30 个坑点不要踩

取自“http://wiki.linuxchina.net/index.php?title=Monitoring_with_prometheus&oldid=12032”