“Monitoring with prometheus”的版本间的差异

来自linux中国网wiki
跳到导航 跳到搜索
第511行： 第511行：
  
 
Use this token to access the HTTP API:
 
Use this token to access the HTTP API:
−
1363904888:AAGeUIoTyRMlxk9zHUa2MTRi1My9HDBP69w
+
1363904888:AAGeUIoxxRMlxk9zHUa2MTRi1My9HDBP69w
 
Keep your token secure and store it safely, it can be used by anyone to control your bot.
 
Keep your token secure and store it safely, it can be used by anyone to control your bot.
  
 
For a description of the Bot API, see this page: https://core.telegram.org/bots/api
 
For a description of the Bot API, see this page: https://core.telegram.org/bots/api
 
</pre>
 
</pre>
−
 
  
 
=====有用的信息 =====
 
=====有用的信息 =====

2020年10月30日 (五) 07:04的版本

==


book

Prometheus操作指南

进阶

以后全改成自己 docker-compose build

prometheus 监控概述(一)有常用监控警告

PromQL 内置函数(三)


prometheus python 库编写自定义指标的方法(完整代码)


promethues + python + flask监控后端服务状态

prometheus python client

使用Python和Flask编写Prometheus监控

python开发prometheus exporter

ins on docker-composer

git clone https://github.com/evan886/docker-compose-monitor.git


https://gitee.com/atompi/PrometheusStackDockerCompose

安全

 node exporter 添加防火墙

curl -Lo /etc/yum.repos.d/_copr_ibotty-prometheus-exporters.repo https://copr.fedorainfracloud.org/coprs/ibotty/prometheus-exporters/repo/epel-7/ibotty-prometheus-exporters-epel-7.repo
# yum install node_exporter

prometheus

Nginx用户密码认证配置 Basic HTTP authentication

不错的书 https://yunlzheng.gitbook.io/prometheus-book/



新型监控告警工具prometheus（普罗米修斯）的入门使用（附视频讲解）

【视频】新型监控告警工具prometheus（普罗米修斯）入门使用（附视频讲解）

为什么说 Prometheus 是足以取代 Zabbix 的监控神器？

基于Prometheus的分布式在线服务监控实践

usage

Grafana单图表告警邮件&面板图（Dashboard）日报邮件

Recording rules

https://github.com/prometheus/blackbox_exporter/blob/master/blackbox.yml

grafana

#防火墙   grafana ng 在 10.3.10.10
 firewall-cmd   --permanent --add-rich-rule="rule family="ipv4" source address="10.3.10.10" port protocol="tcp" port="3000" accept "
firewall-cmd --reload


grafana.com Run Grafana Docker image

中文 Node Exporte for Prometheus Dashboard

usage creat-->import --> id 8919

Container


https://grafana.com/grafana/dashboards/11174 


docker-grafana/config.env  https://github.com/stefanwalther/docker-grafana/blob/master/config.env

grafana 添加源和数据

使用Grafana创建可视化Dashboard

Panel plugin not found: grafana-piechart-panel


升级为7.0.0就好了 不过要看其它模板还能用么 

#这个放弃了 是在 6.x系列的  还不好用 
grafana-cli plugins install grafana-piechart-panel

docker-compose  restart  grafana 


执行过程
 grafana-cli plugins install grafana-piechart-panel
installing grafana-piechart-panel @ 1.6.1
from: https://grafana.com/api/plugins/grafana-piechart-panel/versions/1.6.1/download
into: /var/lib/grafana/plugins

✔ Installed grafana-piechart-panel successfully 

Restart grafana after installing plugins . <service grafana-server restart>

[https://github.com/unifi-poller/unifi-poller/issues/248 RESOLVED: Panel plugin not found: table-old ]

Panel plugin not found: grafana-piechart-panel

grafana 添加用户

1.Users-->  2.Invite --> 

3.进行邀请
Email or Username:输入用户邮箱，建议先跟用户名保持相同，然后在后面在写邮箱，后面会说到，这样不会报错
Name：输入用户名
Role:配置用户权限(只读Viewer 编辑Editor 管理员Admin)
Send invite email：勾选时，会将新用户的设置密码的链接发到邮箱

点击创建之后回调到新用户设置密码

 -->

4.创建好之后，点击Pending Invites
复制链接到浏览器的地址栏中访问（也可从邮件中收到链接）
进去之后是下面的界面，这里有个坑，复制的IP地址是localhost，我们这里改为自己搭建的IP地址，然后在Email填写邮箱地址，设置新用户的密码
设置完成后，会直接以新用户身份登陆


http://localhost:3000/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa

http://a.linuxsa.org/invite/PAhyq4gzfA3ugRiw0PH2RC2vsIuSTa

  -->

grafana的用户及权限的配置

grafana的用户及权限的配置

grafana-利用Google OAuth2 身份验证

ng config

cat gf.conf 
server
{
    listen 80;
    server_name gf.linuxsa.org;
#    access_log  /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
    #error_log /var/log/nginx/test-nux-bcb.evan.error.log;


    client_max_body_size 10m;
  
    location / {
      proxy_pass http://10.3.10.124:3000;
      proxy_http_version 1.1;

#set_real_ip_from 10.0.0.0/8;
#set_real_ip_from 172.16.0.0/12;
#set_real_ip_from 192.168.0.0/16;
#real_ip_header X-Forwarded-For;

        proxy_set_header        Host                 $host;
        proxy_set_header        X-Real-IP            $remote_addr;
        proxy_set_header        X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header        X-Remote-Port        $remote_port;
        proxy_set_header        X-Forwarded-Proto    $scheme;
        proxy_redirect          off;

          proxy_connect_timeout 1800s;
          proxy_send_timeout 1800s;
          proxy_read_timeout 1800s;



      real_ip_recursive on;
#      real_ip_header X-Forwarded-For;
#      set_real_ip_from 0.0.0.0/0;
#      proxy_set_header Host             $host;
#      proxy_set_header X-Real-IP $remote_addr;
#      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


#proxy_set_header Host             $host:$server_port;
#proxy_set_header    HTTP_X_FORWARDED_FOR $remote_addr;
#proxy_set_header X-NginX-Proxy true;

#      proxy_set_header X-Real-IP        $remote_addr;
 #     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#      proxy_set_header Upgrade $http_upgrade;
#      proxy_set_header Connection "upgrade";
    }
}

使用教程

grafana配置与简单使用

Grafana使用教程

grafana数据可视化-使用教程

grafana快速入门

grafana使用教程之基本概念（basic concepts）

Grafana从无到有的一个相当粗略的资料总结


grafana数据可视化-使用教程

Grafana快速入门：InfluxDB数据源以及曲线图表仪表盘配置


Grafana报警 


apached conf

##/etc/httpd/conf.d/gf.conf


<VirtualHost *:80>
      ServerName gf.linuxsa.org
      ServerAlias gf.linuxsa.org
     
      ProxyRequests off
     
      <Proxy *>
        Order deny,allow
        Allow from all
      </Proxy>
     
      <Location />
        ProxyPass http://localhost:3000/
        ProxyPassReverse http://localhost:3000/
      </Location>
    </VirtualHost>

vm监控Node-exporter

可以如上的 node Node-exporter

Blackbox

Blackbox exporter

pushgateway

pushgateway

基于Prometheus的Pushgateway实战

https://songjiayang.gitbooks.io/prometheus/content/pushgateway/how.html

Process-exporter

应用监控

redis

prometheus监控Redis集群

pika主备部署（redis-sentinel）+监控（prometheus+grafana）

Prometheus监控 Redis , redis-cli_exporter

监控MQ

Monitoring with Prometheus & Grafana

监控ES

https://github.com/slok/ecs-exporter

docker

Prometheus+grafana监控：cAdvisor输出的容器内存相关的指标的解读

mysql

prometheus 监控之 mysql 篇（含mysql报警规则）

Prometheus MySQL Exporter源码阅读与分析

python + promethues 监控mysql

Spring boot

https://www.bookstack.cn/read/prometheus_practice/application-spring.md


Prometheus基于java开发exporter样例以及pushgateway的应用

ng conf

cat  alert.conf 
server
{
    listen 80;
    server_name alert.com;
#    access_log  /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
    #error_log /var/log/nginx/test-nux-bcb.evan.error.log;


    client_max_body_size 10m;
  
    location / {
  #      auth_basic "Prometheus";
  #      auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
      proxy_pass http://10.3.10.124:9093;
      proxy_http_version 1.1;

#s#et_real_ip_from 10.0.0.0/8;
#set_real_ip_from 172.16.0.0/12;
#set_real_ip_from 192.168.0.0/16;
#real_ip_header X-Forwarded-For;
        proxy_set_header        Host                 $host;
        proxy_set_header        X-Real-IP            $remote_addr;
        proxy_set_header        X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header        X-Remote-Port        $remote_port;
        proxy_set_header        X-Forwarded-Proto    $scheme;
        proxy_redirect          off;
          proxy_connect_timeout 1800s;
          proxy_send_timeout 1800s;
          proxy_read_timeout 1800s;

      real_ip_recursive on;
#      real_ip_header X-Forwarded-For;
#      set_real_ip_from 0.0.0.0/0;
#      proxy_set_header Host             $host;
#      proxy_set_header X-Real-IP $remote_addr;
#      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


#proxy_set_header Host             $host:$server_port;
#proxy_set_header    HTTP_X_FORWARDED_FOR $remote_addr;
#proxy_set_header X-NginX-Proxy true;

#      proxy_set_header X-Real-IP        $remote_addr;
 #     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#      proxy_set_header Upgrade $http_upgrade;
#      proxy_set_header Connection "upgrade";
    }
}



cat  pro.conf 
server
{
    listen 80;
    server_name pro.linuxsa.org;
#    access_log  /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
    #error_log /var/log/nginx/test-nux-bcb.evan.error.log;


    client_max_body_size 10m;
  
    location / {
        auth_basic "Prometheus";
        auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
      proxy_pass http://10.3.10.124:9090;
      proxy_http_version 1.1;

#set_real_ip_from 10.0.0.0/8;
#set_real_ip_from 172.16.0.0/12;
#set_real_ip_from 192.168.0.0/16;
#real_ip_header X-Forwarded-For;

        proxy_set_header        Host                 $host;
        proxy_set_header        X-Real-IP            $remote_addr;
        proxy_set_header        X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header        X-Remote-Port        $remote_port;
        proxy_set_header        X-Forwarded-Proto    $scheme;
        proxy_redirect          off;

          proxy_connect_timeout 1800s;
          proxy_send_timeout 1800s;
          proxy_read_timeout 1800s;



      real_ip_recursive on;
#      real_ip_header X-Forwarded-For;
#      set_real_ip_from 0.0.0.0/0;
#      proxy_set_header Host             $host;
#      proxy_set_header X-Real-IP $remote_addr;
#      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


#proxy_set_header Host             $host:$server_port;
#proxy_set_header    HTTP_X_FORWARDED_FOR $remote_addr;
#proxy_set_header X-NginX-Proxy true;

#      proxy_set_header X-Real-IP        $remote_addr;
 #     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#      proxy_set_header Upgrade $http_upgrade;
#      proxy_set_header Connection "upgrade";
    }
}


cat blackbox.conf 
server
{
    listen 80;
    server_name blackbox.com;
#    access_log  /var/log/nginx/test-nuxt-bcb-blockchain-website.lliao.net.log;
    #error_log /var/log/nginx/test-nux-bcb.evan.error.log;
    client_max_body_size 10m;
    location / {
  #      auth_basic "Prometheus";
  #      auth_basic_user_file "/usr/local/openresty/nginx/conf/hosts/.htpasswd";
      proxy_pass http://10.3.10.124:9115;
      proxy_http_version 1.1;

        proxy_set_header        Host                 $host;
        proxy_set_header        X-Real-IP            $remote_addr;
        proxy_set_header        X-Forwarded-For      $proxy_add_x_forwarded_for;
        proxy_set_header        X-Remote-Port        $remote_port;
        proxy_set_header        X-Forwarded-Proto    $scheme;
        proxy_redirect          off;

          proxy_connect_timeout 1800s;
          proxy_send_timeout 1800s;
          proxy_read_timeout 1800s;
      real_ip_recursive on;
    }
}

告警

Alertmanager

telegram

通过 prome


* 创建 tg 机器和 报警组

** 创建机器

*** 202011 创建bot的例子
#2020
evan lai, [29.10.20 16:50]
/start

BotFather, [29.10.20 16:50]
I can help you create and manage Telegram bots. If you're new to the Bot API, please see the manual (https://core.telegram.org/bots).

You can control me by sending these commands:

/newbot - create a new bot
/mybots - edit your bots [beta]

Edit Bots
/setname - change a bot's name
/setdescription - change bot description
/setabouttext - change bot about info
/setuserpic - change bot profile photo
/setcommands - change the list of commands
/deletebot - delete a bot

Bot Settings
/token - generate authorization token
/revoke - revoke bot access token
/setinline - toggle inline mode (https://core.telegram.org/bots/inline)
/setinlinegeo - toggle inline location requests (https://core.telegram.org/bots/inline#location-based-results)
/setinlinefeedback - change inline feedback (https://core.telegram.org/bots/inline#collecting-feedback) settings
/setjoingroups - can your bot be added to groups?
/setprivacy - toggle privacy mode (https://core.telegram.org/bots#privacy-mode) in groups

Games
/mygames - edit your games (https://core.telegram.org/bots/games) [beta]
/newgame - create a new game (https://core.telegram.org/bots/games)
/listgames - get a list of your games
/editgame - edit a game
/deletegame - delete an existing game

BotFather, [29.10.20 16:50]
Alright, a new bot. How are we going to call it? Please choose a name for your bot.

evan lai, [29.10.20 16:50]
/newbot

evan lai, [29.10.20 16:51]
evan_alert_bot

BotFather, [29.10.20 16:51]
Good. Now let's choose a username for your bot. It must end in `bot`. Like this, for example: TetrisBot or tetris_bot.

evan lai, [29.10.20 16:51]
evan_alert_bot

BotFather, [29.10.20 16:51]
Done! Congratulations on your new bot. You will find it at t.me/evan_alert_bot. You can now add a description, about section and profile picture for your bot, see /help for a list of commands. By the way, when you've finished creating your cool bot, ping our Bot Support if you want a better username for it. Just make sure the bot is fully operational before you do this.

Use this token to access the HTTP API:
1363904888:AAGeUIoxxRMlxk9zHUa2MTRi1My9HDBP69w
Keep your token secure and store it safely, it can be used by anyone to control your bot.

For a description of the Bot API, see this page: https://core.telegram.org/bots/api
有用的信息


evan lai, [10.05.20 21:55]
lxtx_prom_alert_bot

BotFather, [10.05.20 21:55]
Done! Congratulations on your new bot. You will find it at t.me/lxtx_prom_alert_bot. You can now add a description, about section and profile picture for your bot, see /help for a list of commands. By the way, when you've finished creating your cool bot, ping our Bot Support if you want a better username for it. Just make sure the bot is fully operational before you do this.

Use this token to access the HTTP API:
1157710367:AAFD9YLsjdQ_t7botbVLa4imWrOV9LVHNYc
Keep your token secure and store it safely, it can be used by anyone to control your bot.

For a description of the Bot API, see this page: https://core.telegram.org/bots/api


使用API/bottoken/API方法getMe获取自己的id


curl https://api.telegram.org/bot1157710367:AAFD9YLsjdQ_t7botbVLa4imWrOV9LVHNYc/getMe


#前面有bot字母 
sns:~# curl https://api.telegram.org/bot1157710367:AAFD9YLsjdQ_t7botbVLa4imWrOV9LVHNYc/getMe
{"ok":true,"result":{"id":1157710367,"is_bot":true,"first_name":"prom_alert_bot","username":"lxtx_prom_alert_bot","can_join_groups":true,"can_read_all_group_messages":false,"supports_inline_queries":false}}

创建组

获取群ID

在Telegram新建group,然后添加成员刚创建的机器人 (prom_alert_bot) ，调用API方法getUPdates获取群ID

 curl https://api.telegram.org/bot1157710367:AAFD9YLsjdQ_t7botbVLa4imWrOV9LVHNYc/getUpdates
{"ok":true,"result":[{"update_id":367831744,
"message":{"message_id":1,"from":{"id":796717144,"is_bot":false,"first_name":"evan","last_name":"lai","username":"linuxsa"},"chat":{"id":-470646458,"title":"alerm","type":"group","all_members_are_administrators":true},"date":1597202656,"new_chat_participant":{"id":1157710367,"is_bot":true,"first_name":"prom_alert_bot","username":"lxtx_prom_alert_bot"},"new_chat_member":{"id":1157710367,"is_bot":true,"first_name":"prom_alert_bot","username":"lxtx_prom_alert_bot"},"new_chat_members":[{"id":1157710367,"is_bot":true,"first_name":"prom_alert_bot","username":"lxtx_prom_alert_bot"}]}}]}

webhook

1.先把 webhook 跑起来



git clone https://github.com/evan886/alertmanager-webhook-telegram-python.git
cd   alertmanager-webhook-telegram-python/docker 
docker build -t alertmanager-webhook-telegram:1.0 .
docker run -d --name telegram-bot \
	-e "bottoken=1157710367:AAFD9YLsjdQ_t7botbVLa4imWrOV9LVHNYc" \
	-e "chatid=470646458" \
	-e "username=evan" \
	-e "password=evanLXTX123" \
	-p 9119:9119 alertmanager-webhook-telegram:1.0

trouble

起不来 老报错 level=error ts=2019-08-26T05:52:52.19072198Z caller=main.go:337 msg="Loading configuration file failed" file=/usr/local/prometheus/alertmanager/alertmanager.yml err="yaml: unmarshal errors:\n  line 12: field receivers not found in type config.plain"   解决办法 用了聪的办法  - url: 'http://用户:密码@172.24.103.122:9119/alert'

see also

https://core.telegram.org/bots

How to Create a Telegram Bot ID/Chat ID

创建telegram 机器人 并发送消息

如何创建我自己的电报机器人(Telegram Bot)

在 Telegram 中管理主机监控和警报信息

https://github.com/inCaller/prometheus_bot

https://github.com/metalmatze/alertmanager-bot

基于prometheus + grafana + mysql + Telegram 监控告警

https://my.oschina.net/54188zz/blog/3030618

自定义Prometheus告警规则

prometheus alertmanager telegram

部署Alertmanager实现邮件/钉钉/微信报警

prometheus告警插件-alertmanager

教程

prometheus学习系列五： Prometheus配置文件

see also

规划 Prometheus 的存储用量

深入理解监控系统——Prometheus核心特点

prometheus部署node,server以及域名加告警

从零搭建Prometheus监控报警系统

使用prometheus自定义监控

为什么说 Prometheus 是足以取代 Zabbix 的监控神器？

prometheus 监控概述(一)

基于prometheus + grafana + mysql + Telegram 监控告警

取自“http://wiki.linuxchina.net/index.php?title=Monitoring_with_prometheus&oldid=4982”