“Monitoring with prometheus”的版本间的差异

来自linux中国网wiki
跳到导航 跳到搜索
第447行: 第447行:
  
 
==telegram==
 
==telegram==
  通过 prome  
+
  通过 prome
 
 
 
 
===*  创建 tg  机器和 报警组===
 
 
 
====** 创建机器 ====
 
 
 
=====*** 202011 创建bot的例子=====
 
<pre>
 
#2020
 
evan lai, [29.10.20 16:50]
 
/start
 
 
 
BotFather, [29.10.20 16:50]
 
I can help you create and manage Telegram bots. If you're new to the Bot API, please see the manual (https://core.telegram.org/bots).
 
 
 
You can control me by sending these commands:
 
 
 
/newbot - create a new bot
 
/mybots - edit your bots [beta]
 
 
 
Edit Bots
 
/setname - change a bot's name
 
/setdescription - change bot description
 
/setabouttext - change bot about info
 
/setuserpic - change bot profile photo
 
/setcommands - change the list of commands
 
/deletebot - delete a bot
 
 
 
Bot Settings
 
/token - generate authorization token
 
/revoke - revoke bot access token
 
/setinline - toggle inline mode (https://core.telegram.org/bots/inline)
 
/setinlinegeo - toggle inline location requests (https://core.telegram.org/bots/inline#location-based-results)
 
/setinlinefeedback - change inline feedback (https://core.telegram.org/bots/inline#collecting-feedback) settings
 
/setjoingroups - can your bot be added to groups?
 
/setprivacy - toggle privacy mode (https://core.telegram.org/bots#privacy-mode) in groups
 
 
 
Games
 
/mygames - edit your games (https://core.telegram.org/bots/games) [beta]
 
/newgame - create a new game (https://core.telegram.org/bots/games)
 
/listgames - get a list of your games
 
/editgame - edit a game
 
/deletegame - delete an existing game
 
 
 
BotFather, [29.10.20 16:50]
 
Alright, a new bot. How are we going to call it? Please choose a name for your bot.
 
 
 
evan lai, [29.10.20 16:50]
 
/newbot
 
 
 
evan lai, [29.10.20 16:51]
 
evan_alert_bot
 
 
 
BotFather, [29.10.20 16:51]
 
Good. Now let's choose a username for your bot. It must end in `bot`. Like this, for example: TetrisBot or tetris_bot.
 
 
 
evan lai, [29.10.20 16:51]
 
evan_alert_bot
 
 
 
BotFather, [29.10.20 16:51]
 
Done! Congratulations on your new bot. You will find it at t.me/evan_alert_bot. You can now add a description, about section and profile picture for your bot, see /help for a list of commands. By the way, when you've finished creating your cool bot, ping our Bot Support if you want a better username for it. Just make sure the bot is fully operational before you do this.
 
 
 
Use this token to access the HTTP API:
 
1363904888:AAGeUIoxxRMlxk9zHUa2MTRi1My9HDBP69w
 
Keep your token secure and store it safely, it can be used by anyone to control your bot.
 
 
 
For a description of the Bot API, see this page: https://core.telegram.org/bots/api
 
</pre>
 
 
 
=====有用的信息 =====
 
<pre>
 
 
 
 
 
evan lai, [10.05.20 21:55]
 
lxtx_prom_alert_bot
 
 
 
BotFather, [10.05.20 21:55]
 
Done! Congratulations on your new bot. You will find it at t.me/lxtx_prom_alert_bot. You can now add a description, about section and profile picture for your bot, see /help for a list of commands. By the way, when you've finished creating your cool bot, ping our Bot Support if you want a better username for it. Just make sure the bot is fully operational before you do this.
 
 
 
Use this token to access the HTTP API:
 
1157710367:AAFD9YLsjdQ_t7botbVLa4xxWrOc9LVHNYc
 
Keep your token secure and store it safely, it can be used by anyone to control your bot.
 
 
 
For a description of the Bot API, see this page: https://core.telegram.org/bots/api
 
 
 
 
 
使用API/bottoken/API方法getMe获取自己的id
 
 
 
 
 
curl https://api.telegram.org/bot1157710367:AAFD9YLsjdQ_t7xxxxxLa4imWrOV9LVHNYc/getMe
 
 
 
 
 
#前面有bot字母
 
sns:~# curl https://api.telegram.org/bot1157710367:AAFD9YLsjdQ_xxxxotbVLa4imWrOV9LVHNYc/getMe
 
{"ok":true,"result":{"id":1157710367,"is_bot":true,"first_name":"prom_alert_bot","username":"lxtx_prom_alert_bot","can_join_groups":true,"can_read_all_group_messages":false,"supports_inline_queries":false}}
 
 
 
</pre>
 
 
 
==== 创建组====
 
<pre>
 
获取群ID
 
 
 
在Telegram新建group,然后添加成员刚创建的机器人 (prom_alert_bot) ,调用API方法getUPdates获取群ID
 
 
 
curl https://api.telegram.org/bot1157710367:AAFD9YLsjdQ_t7botbVLa4imWrOV9LVHNYc/getUpdates
 
{"ok":true,"result":[{"update_id":367831744,
 
"message":{"message_id":1,"from":{"id":796717144,"is_bot":false,"first_name":"evan","last_name":"lai","username":"linuxsa"},"chat":{"id":-470646458,"title":"alerm","type":"group","all_members_are_administrators":true},"date":1597202656,"new_chat_participant":{"id":1157710367,"is_bot":true,"first_name":"prom_alert_bot","username":"lxtx_prom_alert_bot"},"new_chat_member":{"id":1157710367,"is_bot":true,"first_name":"prom_alert_bot","username":"lxtx_prom_alert_bot"},"new_chat_members":[{"id":1157710367,"is_bot":true,"first_name":"prom_alert_bot","username":"lxtx_prom_alert_bot"}]}}]}
 
</pre>
 
 
 
===webhook ===
 
==== 1.先把 webhook 跑起来 ====
 
<pre>
 
 
 
 
 
git clone https://github.com/evan886/alertmanager-webhook-telegram-python.git
 
cd  alertmanager-webhook-telegram-python/docker
 
docker build -t alertmanager-webhook-telegram:1.0 .
 
docker run -d --name telegram-bot \
 
-e "bottoken=1157710367:AxxxxxxQ_t7botbVLa4imWrOV9LVHNYc" \
 
-e "chatid=4706458" \
 
-e "username=evan" \
 
-e "password=evanLxx123" \
 
-p 9119:9119 alertmanager-webhook-telegram:1.0
 
</pre>
 
 
 
==== 配置 ====
 
<pre>
 
cat alertmanager/config.yml
 
 
 
# 定义路由树信息,这个路由可以接收到所有的告警,还可以继续配置路由,比如project: zhidaoAPP(prometheus 告警规则中自定义的lable)发给谁,project: baoxian的发给谁
 
route:
 
  group_by: ['alertname'] # 报警分组依据
 
  group_wait: 10s        # 最初即第一次等待多久时间发送一组警报的通知
 
  group_interval: 60s    # 在发送新警报前的等待时间
 
  repeat_interval: 1h    # 发送重复警报的周期 对于email配置中,此项不可以设置过低,否则将会由于邮件发送太多频繁,被smtp服务器拒绝
 
  receiver: 'telegram-webhook'      # 发送警报的接收者的名称,以下receivers name的名称
 
 
 
# 定义警报接收者信息
 
receivers:
 
  - name: 'telegram-webhook'
 
    webhook_configs:
 
    - url: http://evan:[email protected]:9119/alert
 
 
 
</pre>
 
 
 
==== 结果查看====
 
正常来说 这时候 你的TG 组就有信息了 ,没的话 就停止一个node export  收不到就有问题喽
 
 
 
=== trouble===
 
起不来 老报错 level=error ts=2019-08-26T05:52:52.19072198Z caller=main.go:337 msg="Loading configuration file failed" file=/usr/local/prometheus/alertmanager/alertmanager.yml err="yaml: unmarshal errors:\n  line 12: field receivers not found in type config.plain"  解决办法 用了聪的办法  - url: 'http://用户:密码@172.24.103.122:9119/alert'
 
 
 
=== see also===
 
https://prometheus.io/docs/alerting/latest/configuration/
 
 
 
https://core.telegram.org/bots
 
 
 
[https://techsoftcenter.com/how-to-create-a-telegram-bot-id-chat-id/ How to Create a Telegram Bot ID/Chat ID]
 
 
 
[https://www.cnblogs.com/KillBugMe/p/13140226.html 创建telegram 机器人 并发送消息]
 
 
 
[https://www.teleme.io/articles/create_your_own_telegram_bot?hl=zh-hans 如何创建我自己的电报机器人(Telegram Bot)]
 
 
 
[https://nova.moe/manage-host-alert-on-telegram-with-grafana/ 在 Telegram 中管理主机监控和警报信息]
 
 
 
https://github.com/inCaller/prometheus_bot
 
 
 
https://github.com/metalmatze/alertmanager-bot
 
 
 
[https://blog.csdn.net/weixin_34242331/article/details/91875514  基于prometheus + grafana + mysql + Telegram 监控告警]
 
 
 
https://my.oschina.net/54188zz/blog/3030618
 
 
 
[https://yunlzheng.gitbook.io/prometheus-book/parti-prometheus-ji-chu/alert/prometheus-alert-rule 自定义Prometheus告警规则]
 
 
 
[https://www.linux.org.ru/forum/general/14894302  prometheus alertmanager telegram ]
 
 
 
[https://www.cnblogs.com/wangxu01/articles/11654836.html 部署Alertmanager实现邮件/钉钉/微信报警]
 
 
 
[https://www.cnblogs.com/xiaobaozi-95/p/10740511.html prometheus告警插件-alertmanager]
 
  
 
=教程=
 
=教程=

2020年10月30日 (五) 08:42的版本

==


book

Prometheus操作指南

进阶

以后全改成自己 docker-compose build

prometheus 监控概述(一)有常用监控警告

PromQL 内置函数(三)


prometheus python 库编写自定义指标的方法(完整代码)


promethues + python + flask监控后端服务状态

prometheus python client

使用Python和Flask编写Prometheus监控

python开发prometheus exporter

ins on docker-composer

git clone https://github.com/evan886/docker-compose-monitor.git


https://gitee.com/atompi/PrometheusStackDockerCompose

安全

 node exporter 添加防火墙

curl -Lo /etc/yum.repos.d/_copr_ibotty-prometheus-exporters.repo https://copr.fedorainfracloud.org/coprs/ibotty/prometheus-exporters/repo/epel-7/ibotty-prometheus-exporters-epel-7.repo
# yum install node_exporter

prometheus

Nginx用户密码认证配置 Basic HTTP authentication

不错的书 https://yunlzheng.gitbook.io/prometheus-book/



新型监控告警工具prometheus(普罗米修斯)的入门使用(附视频讲解)

【视频】新型监控告警工具prometheus(普罗米修斯)入门使用(附视频讲解)

为什么说 Prometheus 是足以取代 Zabbix 的监控神器?

基于Prometheus的分布式在线服务监控实践

usage

Grafana单图表告警邮件&面板图(Dashboard)日报邮件

Recording rules

https://github.com/prometheus/blackbox_exporter/blob/master/blackbox.yml

grafana

#防火墙   grafana ng 在 10.3.10.10
 firewall-cmd   --permanent --add-rich-rule="rule family="ipv4" source address="10.3.10.10" port protocol="tcp" port="3000" accept "
firewall-cmd --reload


grafana.com Run Grafana Docker image

中文 Node Exporte for Prometheus Dashboard

usage creat-->import --> id 8919

Container


https://grafana.com/grafana/dashboards/11174


docker-grafana/config.env  https://github.com/stefanwalther/docker-grafana/blob/master/config.env  

grafana 添加源和数据

使用Grafana创建可视化Dashboard

Panel plugin not found: grafana-piechart-panel


升级为7.0.0就好了 不过要看其它模板还能用么 

#这个放弃了 是在 6.x系列的  还不好用 
grafana-cli plugins install grafana-piechart-panel

docker-compose  restart  grafana 


执行过程
 grafana-cli plugins install grafana-piechart-panel
installing grafana-piechart-panel @ 1.6.1
from: https://grafana.com/api/plugins/grafana-piechart-panel/versions/1.6.1/download
into: /var/lib/grafana/plugins

✔ Installed grafana-piechart-panel successfully 

Restart grafana after installing plugins . <service grafana-server restart>

[https://github.com/unifi-poller/unifi-poller/issues/248 RESOLVED: Panel plugin not found: table-old ]


Panel plugin not found: grafana-piechart-panel

grafana 添加用户

1.Users-->  2.Invite --> 

3.进行邀请
Email or Username:输入用户邮箱,建议先跟用户名保持相同,然后在后面在写邮箱,后面会说到,这样不会报错
Name:输入用户名
Role:配置用户权限(只读Viewer 编辑Editor 管理员Admin)
Send invite email:勾选时,会将新用户的设置密码的链接发到邮箱

点击创建之后回调到新用户设置密码

 -->

4.创建好之后,点击Pending Invites
复制链接到浏览器的地址栏中访问(也可从邮件中收到链接)
进去之后是下面的界面,这里有个坑,复制的IP地址是localhost,我们这里改为自己搭建的IP地址,然后在Email填写邮箱地址,设置新用户的密码
设置完成后,会直接以新用户身份登陆


http://localhost:3000/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa

http://a.linuxsa.org/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa

  -->

grafana的用户及权限的配置

grafana的用户及权限的配置

grafana-利用Google OAuth2 身份验证

ng config

cat gf.conf 
server
{
    listen 80;
    server_name gf.linuxsa.org;
#    access_log  /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
    #error_log /var/log/nginx/test-nux-bcb.evan.error.log;


    client_max_body_size 10m;
  
    location / {
      proxy_pass http://10.3.10.124:3000;
      proxy_http_version 1.1;

#set_real_ip_from 10.0.0.0/8;
#set_real_ip_from 172.16.0.0/12;
#set_real_ip_from 192.168.0.0/16;
#real_ip_header X-Forwarded-For;

        proxy_set_header        Host                 $host;
        proxy_set_header        X-Real-IP            $remote_addr;
        proxy_set_header        X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header        X-Remote-Port        $remote_port;
        proxy_set_header        X-Forwarded-Proto    $scheme;
        proxy_redirect          off;

          proxy_connect_timeout 1800s;
          proxy_send_timeout 1800s;
          proxy_read_timeout 1800s;



      real_ip_recursive on;
#      real_ip_header X-Forwarded-For;
#      set_real_ip_from 0.0.0.0/0;
#      proxy_set_header Host             $host;
#      proxy_set_header X-Real-IP $remote_addr;
#      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


#proxy_set_header Host             $host:$server_port;
#proxy_set_header    HTTP_X_FORWARDED_FOR $remote_addr;
#proxy_set_header X-NginX-Proxy true;

#      proxy_set_header X-Real-IP        $remote_addr;
 #     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#      proxy_set_header Upgrade $http_upgrade;
#      proxy_set_header Connection "upgrade";
    }
}

使用教程

grafana配置与简单使用

Grafana使用教程

grafana数据可视化-使用教程

grafana快速入门

grafana使用教程之基本概念(basic concepts)

Grafana从无到有的一个相当粗略的资料总结


grafana数据可视化-使用教程

Grafana快速入门:InfluxDB数据源以及曲线图表仪表盘配置


Grafana报警


apached conf
##/etc/httpd/conf.d/gf.conf


<VirtualHost *:80>
      ServerName gf.linuxsa.org
      ServerAlias gf.linuxsa.org
     
      ProxyRequests off
     
      <Proxy *>
        Order deny,allow
        Allow from all
      </Proxy>
     
      <Location />
        ProxyPass http://localhost:3000/
        ProxyPassReverse http://localhost:3000/
      </Location>
    </VirtualHost>

vm监控Node-exporter

可以如上的 node Node-exporter

Blackbox

Blackbox exporter

pushgateway

pushgateway

基于Prometheus的Pushgateway实战

https://songjiayang.gitbooks.io/prometheus/content/pushgateway/how.html

Process-exporter

应用监控

redis

prometheus监控Redis集群

pika主备部署(redis-sentinel)+监控(prometheus+grafana)

Prometheus监控 Redis , redis-cli_exporter

监控MQ

Monitoring with Prometheus & Grafana

监控ES

https://github.com/slok/ecs-exporter

docker

Prometheus+grafana监控:cAdvisor输出的容器内存相关的指标的解读

mysql

prometheus 监控之 mysql 篇(含mysql报警规则)

Prometheus MySQL Exporter源码阅读与分析

python + promethues 监控mysql

Spring boot

https://www.bookstack.cn/read/prometheus_practice/application-spring.md


Prometheus基于java开发exporter样例以及pushgateway的应用

ng conf

cat  alert.conf 
server
{
    listen 80;
    server_name alert.com;
#    access_log  /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
    #error_log /var/log/nginx/test-nux-bcb.evan.error.log;


    client_max_body_size 10m;
  
    location / {
  #      auth_basic "Prometheus";
  #      auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
      proxy_pass http://10.3.10.124:9093;
      proxy_http_version 1.1;

#s#et_real_ip_from 10.0.0.0/8;
#set_real_ip_from 172.16.0.0/12;
#set_real_ip_from 192.168.0.0/16;
#real_ip_header X-Forwarded-For;
        proxy_set_header        Host                 $host;
        proxy_set_header        X-Real-IP            $remote_addr;
        proxy_set_header        X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header        X-Remote-Port        $remote_port;
        proxy_set_header        X-Forwarded-Proto    $scheme;
        proxy_redirect          off;
          proxy_connect_timeout 1800s;
          proxy_send_timeout 1800s;
          proxy_read_timeout 1800s;

      real_ip_recursive on;
#      real_ip_header X-Forwarded-For;
#      set_real_ip_from 0.0.0.0/0;
#      proxy_set_header Host             $host;
#      proxy_set_header X-Real-IP $remote_addr;
#      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


#proxy_set_header Host             $host:$server_port;
#proxy_set_header    HTTP_X_FORWARDED_FOR $remote_addr;
#proxy_set_header X-NginX-Proxy true;

#      proxy_set_header X-Real-IP        $remote_addr;
 #     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#      proxy_set_header Upgrade $http_upgrade;
#      proxy_set_header Connection "upgrade";
    }
}



cat  pro.conf 
server
{
    listen 80;
    server_name pro.linuxsa.org;
#    access_log  /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
    #error_log /var/log/nginx/test-nux-bcb.evan.error.log;


    client_max_body_size 10m;
  
    location / {
        auth_basic "Prometheus";
        auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
      proxy_pass http://10.3.10.124:9090;
      proxy_http_version 1.1;

#set_real_ip_from 10.0.0.0/8;
#set_real_ip_from 172.16.0.0/12;
#set_real_ip_from 192.168.0.0/16;
#real_ip_header X-Forwarded-For;

        proxy_set_header        Host                 $host;
        proxy_set_header        X-Real-IP            $remote_addr;
        proxy_set_header        X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header        X-Remote-Port        $remote_port;
        proxy_set_header        X-Forwarded-Proto    $scheme;
        proxy_redirect          off;

          proxy_connect_timeout 1800s;
          proxy_send_timeout 1800s;
          proxy_read_timeout 1800s;



      real_ip_recursive on;
#      real_ip_header X-Forwarded-For;
#      set_real_ip_from 0.0.0.0/0;
#      proxy_set_header Host             $host;
#      proxy_set_header X-Real-IP $remote_addr;
#      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


#proxy_set_header Host             $host:$server_port;
#proxy_set_header    HTTP_X_FORWARDED_FOR $remote_addr;
#proxy_set_header X-NginX-Proxy true;

#      proxy_set_header X-Real-IP        $remote_addr;
 #     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#      proxy_set_header Upgrade $http_upgrade;
#      proxy_set_header Connection "upgrade";
    }
}


cat blackbox.conf 
server
{
    listen 80;
    server_name blackbox.com;
#    access_log  /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
    #error_log /var/log/nginx/test-nux-bcb.evan.error.log;
    client_max_body_size 10m;
    location / {
  #      auth_basic "Prometheus";
  #      auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
      proxy_pass http://10.3.10.124:9115;
      proxy_http_version 1.1;

        proxy_set_header        Host                 $host;
        proxy_set_header        X-Real-IP            $remote_addr;
        proxy_set_header        X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header        X-Remote-Port        $remote_port;
        proxy_set_header        X-Forwarded-Proto    $scheme;
        proxy_redirect          off;

          proxy_connect_timeout 1800s;
          proxy_send_timeout 1800s;
          proxy_read_timeout 1800s;
      real_ip_recursive on;
    }
}



告警

Alertmanager

telegram

通过 prome

教程

prometheus学习系列五: Prometheus配置文件

see also

规划 Prometheus 的存储用量

深入理解监控系统——Prometheus核心特点

prometheus部署node,server以及域名加告警

从零搭建Prometheus监控报警系统

使用prometheus自定义监控

为什么说 Prometheus 是足以取代 Zabbix 的监控神器?

prometheus 监控概述(一)

基于prometheus + grafana + mysql + Telegram 监控告警