页面“运维规划2018-2024”与“Filebeat nginx log”之间的差异

来自linux中国网wiki
(页面间的差异)
跳到导航 跳到搜索
 
 
第1行: 第1行:
[[category:devops]]  
+
[[category:ops]]
=大龄运维=
 
go ,cmdb, js  devops
 
  
https://www.v2ex.com/t/598440?__cf_chl_jschl_tk__=4fd0e10bc1ee83eb05176806d9433076e1fa08e8-1597039021-0-Af0TkNavo215SSTNLsdKOV9tqmTbckqd9aV_DrcqROga2l0U_TbpmLo5RYrY1VFEtsDNtYp82uGAiF9rOj-5LQw_8R9ic8SMnnk-lvBz1VxXZnlc4af3Qr6KC5Rqe9v1nX3W3hmXhO6NtxE8dgXbs13Wo5-xE5wgt5At15GuUC8UDd6v0W2w6eGdw9VBxks5jXlk-nZ2sc9xycmrjEwl6l4HZYVEvLnNdBRJpV9Aa6LG8T7qnoAvGeT-eQYqbUN0b7T6fRXFkjxqNbTOgOLm_xuL7sou1BKNe9zd1kNNfFtgcoKekkuwYNFcMP9ve9PTKRddmq1E5ELO8PzFSOZWMB8
+
=ins and config=
 +
==Download and install Filebeat==
 +
<pre>
  
  
[https://www.zhihu.com/question/27214752 什么是运维?运维工程师这个职位有多重要?]
+
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.12.0-amd64.deb
 +
sudo dpkg -i filebeat-7.12.0-amd64.deb
 +
</pre>
 +
==Edit the configuration ==
 +
<pre>
 +
Modify /etc/filebeat/filebeat.yml to set the connection information:
  
=2020=
+
output.elasticsearch:
==english==
+
  hosts: ["<es_url>"]
 +
  username: "elastic"
 +
  password: "<password>"
 +
setup.kibana:
 +
  host: "<kibana_url>"
  
==redis==
+
Where <password> is the password of the elastic user, <es_url> is the URL of Elasticsearch, and <kibana_url> is the URL of Kibana.
wiki上相关的可以看看先
 
  
[[Redis运维]]
+
</pre>
  
==lua==
+
==Enable and configure the nginx module ==
[[Lua基础]]
 
  
==golang==
+
<pre>
 
+
sudo filebeat modules enable nginx
== Zookeeper==
 
https://www.jianshu.com/u/3829416c7065
 
==see also==
 
 
 
[https://zhuanlan.zhihu.com/p/66483660 运维超级好用工具大PK]
 
 
 
=k8s=
 
搞个国内的吧  debian  主要就是这下面三个站内资源
 
 
 
[https://developer.aliyun.com/mirror/kubernetes?spm=a2c6h.13651102.0.0.3e221b11Au1vN6 Kubernetes 镜像 aliyun]
 
  
 +
Modify the settings in the /etc/filebeat/modules.d/nginx.yml file.
  
 +
</pre>
  
[[Install and Configure Kubernetes (k8s) on debian10]]
+
== Start Filebeat==
  
 
[[使用kubeadm离线部署kubernetesv1.9.0 on centos7]]
 
 
 
 
[[2018k8s]]
 
 
 
[https://zhuanlan.zhihu.com/p/46341911 kubernetes安装(国内环境)]
 
 
 
[https://www.cnblogs.com/chenzhenqi/p/10695959.html 国内环境安装k8s]
 
 
 
 
国内k8s集群部署的几种方式
 
部署方式
 
 
目前我所了解有以下几种方式:
 
 
<pre>
 
<pre>
    使用kubeadmin通过离线镜像的方式,网上教程和镜像包挺多的
+
The setup command loads the Kibana dashboards. If the dashboards are already set up, omit this command.
 
 
    通过厂商集成的方式如:rancher
 
    这里推荐一篇blog:CentOS7环境安装Kubernetes四部曲https://blog.csdn.net/boling_cavalry/article/details/79613037
 
 
 
    直接使用类似阿里云这样的公有云平台k8s 。只要有钱,一键搞定 : )
 
 
 
    通过yum官方仓库安装,需要启用extras仓库。
 
    不过里面版本都比较老,目前最新的也只是1.5.2版本,对于想学新的东西,比如k8s角色管理、helm等来说不太方便,因此也不建议使用。
 
 
 
    通过二进制包的形式进行安装,这里推荐一个repo:kubeasz https://github.com/gjmzj/kubeasz
 
    里面文档也很丰富清晰,对于初学者来说很有帮助。 </pre>
 
 
 
原文链接:https://blog.csdn.net/lusyoe/article/details/80217291
 
 
 
=服务器费用=
 
自动续费 ?
 
 
 
=cmdb=
 
搞在xk intra 然后 frp到外面
 
 
 
[https://www.linuxprobe.com/cmdb-things.html 前聚美优品运维负责人谈CMDB]
 
 
 
[https://www.jianshu.com/p/f3a70cddccc5 CMDB和运维自动化]
 
 
 
[http://www.yunweipai.com/archives/6856.html (深度好文)重构CMDB,避免运维之耻]
 
 
 
[https://segmentfault.com/q/1010000003838760 开源CMDB]
 
 
 
=运维自动化=
 
 
 
[http://os.51cto.com/art/201806/576802.htm 小团队如何从零搭建一个自动化运维体系]
 
 
 
[http://os.51cto.com/art/201806/576888.htm 如何做好自动化运维?自动化运维必备技能有哪些]
 
  
 +
sudo filebeat setup
 +
sudo service filebeat start
 +
</pre>
  
[[category:ops]] [[category:生活与情感]]
+
== Module status==
==python 自动化 ==
 
 
 
[https://blog.csdn.net/aeolus_pu/article/details/52605109 利用python实现批量自动化运维脚本案例]
 
 
 
早下班时回家记得看书和动手例子
 
 
 
=监控=
 
==zabbix==
 
 
 
[[Zabbix报警]]
 
 
 
[https://blog.csdn.net/wyl9527/article/details/73928853 action_email 还有报警前执行脚本]
 
 
 
email 报警
 
短信报警
 
wechat 报警
 
 
 
zabbix new server docker 化
 
[https://blog.csdn.net/heyongluoyao8/article/details/51413668 服务器监控(包括性能指标与web应用程序)]
 
 
 
==pinpoint==
 
 
 
== Nightingale ==
 
[https://www.oschina.net/p/nightingale-didi nightingale-didi]
 
 
 
Open-Falcon 是小米运维团队开源的企业级监控解决方案,在国内广泛使用。
 
 
 
Nightingale 采用 Apache-2.0 开源协议,Copyright © 滴滴 2020
 
 
 
=python=
 
[https://blog.csdn.net/chengxuyuanyonghu/article/details/59486631 python常用运维脚本实例]
 
 
 
 
 
[https://blog.csdn.net/hardywang/article/details/51536510  python用于系统运维管理和灰度发布部署]
 
 
 
[http://www.voidcn.com/article/p-poyppygk-ng.html 基于python的一个运维自动化的项目(进度更新)【已开源】]
 
 
 
=django=
 
 
 
Gunicorn
 
 
 
=ruby=
 
 
 
=tomcat=
 
[http://blog.51cto.com/jowin/1739400 Linux运维 第三阶段 (十九) tomcat]
 
 
 
[https://blog.csdn.net/lifetragedy/article/details/7708724 通向架构师的道路(第四天)之Tomcat性能调优]
 
 
 
[https://www.jianshu.com/p/d45e12241af4 Tomcat性能调优-JVM监控与调优]
 
 
 
[https://blog.csdn.net/xyang81/article/details/51530979 Tomcat配置与优化(内存、并发、管理)]
 
 
 
 
 
[https://www.centos.bz/2017/08/tomcat-jvm-performance-tuning/ Tomcat和JVM的性能调优总结]
 
 
 
[https://www.oschina.net/translate/tomcat-performance-tuning Tomcat 生产服务器性能优化 【已翻译100%】]
 
 
 
[http://yunzhu.iteye.com/blog/953387 监控Tomcat解决方案(监控应用服务器系列文章)]
 
 
 
==tomcat优化==
 
 
 
[https://blog.csdn.net/coolwzjcool/article/details/2544448 TOMCAT内存大小调整]
 
 
 
[http://elf8848.iteye.com/blog/378805 Tomcat内存设置详解]
 
 
 
[https://blog.csdn.net/xyang81/article/details/51530979 Tomcat配置与优化(内存、并发、管理)]
 
 
 
[https://blog.csdn.net/lifetragedy/article/details/7708724 通向架构师的道路(第四天)之Tomcat性能调优-让小猫飞奔]
 
 
 
==tomcat 启动脚本==
 
 
<pre>
 
<pre>
#!/bin/bash
+
Module status 右边的 check data 按键 -- > Nginx logs dashboard
#cat /etc/init.d/pp-col 在centos7上试过 可运行
 
# chkconfig: 345 99 28
 
# description: Starts/Stops Apache Tomcat
 
 
 
#by evan
 
#修改几个目录
 
##export JAVA_HOME=/usr/java/jdk1.8.0_121/
 
#sed -i "s/JAVA_HOME=\/usr\/java\/default\//JAVA_HOME=\/usr\/java\/jdk1.8.0_121\//g" pp-collector.init
 
#sed -i "s/CATALINA_HOME=\/data\/service\/pinpoint-collector\//CATALINA_HOME=\/data\/pp-col\//g" pp-collector.init
 
#sed -i "s/CATALINA_BASE=\/data\/service\/pinpoint-collector\//CATALINA_BASE=\/data\/pp-col\//g" pp-collector.init
 
#by evan
 
 
 
# Tomcat  start/stop/status script
 
# Forked from: https://gist.github.com/valotas/1000094
 
# @author: Miglen Evlogiev <[email protected]>
 
#
 
# Release updates:
 
# Updated method for gathering pid of the current proccess
 
# Added usage of CATALINA_BASE
 
# Added coloring and additional status
 
# Added check for existence of the tomcat user
 
#
 
 
#Location of JAVA_HOME (bin files)
 
export JAVA_HOME=/usr/java/jdk1.8.0_121/
 
 
#Add Java binary files to PATH
 
export PATH=$JAVA_HOME/bin:$PATH
 
 
#CATALINA_HOME is the location of the bin files of Tomcat 
 
export CATALINA_HOME=/data/pp-col/
 
   
 
#CATALINA_BASE is the location of the configuration files of this instance of Tomcat
 
export CATALINA_BASE=/data/pp-col/
 
 
#TOMCAT_USER is the default user of tomcat
 
export TOMCAT_USER=root
 
 
#TOMCAT_USAGE is the message if this script is called without any options
 
TOMCAT_USAGE="Usage: $0 {\e[00;32mstart\e[00m|\e[00;31mstop\e[00m|\e[00;32mstatus\e[00m|\e[00;31mrestart\e[00m}"
 
 
#SHUTDOWN_WAIT is wait time in seconds for java proccess to stop
 
SHUTDOWN_WAIT=3
 
 
tomcat_pid() {
 
        echo `ps -fe | grep $CATALINA_BASE | grep -v grep | tr -s " "|cut -d" " -f2`
 
}
 
 
start() {
 
  pid=$(tomcat_pid)
 
  if [ -n "$pid" ]
 
  then
 
    echo -e "\e[00;31mTomcat is already running (pid: $pid)\e[00m"
 
  else
 
    # Start tomcat
 
    echo -e "\e[00;32mStarting tomcat\e[00m"
 
    #ulimit -n 100000
 
    #umask 007
 
    #/bin/su -p -s /bin/sh tomcat
 
        if [ `user_exists $TOMCAT_USER` = "1" ]
 
        then
 
                su $TOMCAT_USER -c $CATALINA_HOME/bin/startup.sh
 
        else
 
                sh $CATALINA_HOME/bin/startup.sh
 
        fi
 
        status
 
  fi
 
  return 0
 
}
 
 
status(){
 
          pid=$(tomcat_pid)
 
          if [ -n "$pid" ]; then echo -e "\e[00;32mTomcat is running with pid: $pid\e[00m"
 
          else echo -e "\e[00;31mTomcat is not running\e[00m"
 
          fi
 
}
 
 
stop() {
 
  pid=$(tomcat_pid)
 
  if [ -n "$pid" ]
 
  then
 
    echo -e "\e[00;31mStoping Tomcat\e[00m"
 
    #/bin/su -p -s /bin/sh tomcat
 
        sh $CATALINA_HOME/bin/shutdown.sh
 
 
    let kwait=$SHUTDOWN_WAIT
 
    count=0;
 
    until [ `ps -p $pid | grep -c $pid` = '0' ] || [ $count -gt $kwait ]
 
    do
 
      echo -n -e "\n\e[00;31mwaiting for processes to exit\e[00m";
 
      sleep 1
 
      let count=$count+1;
 
    done
 
 
    if [ $count -gt $kwait ]; then
 
      echo -n -e "\n\e[00;31mkilling processes which didn't stop after $SHUTDOWN_WAIT seconds\e[00m"
 
      kill -9 $pid
 
    fi
 
  else
 
    echo -e "\e[00;31mTomcat is not running\e[00m"
 
  fi
 
 
  return 0
 
}
 
 
user_exists(){
 
        if id -u $1 >/dev/null 2>&1; then
 
        echo "1"
 
        else
 
                echo "0"
 
        fi
 
}
 
 
case $1 in
 
 
        start)
 
          start
 
        ;;
 
     
 
        stop) 
 
          stop
 
        ;;
 
     
 
        restart)
 
          stop
 
          start
 
        ;;
 
     
 
        status)
 
                status
 
             
 
        ;;
 
     
 
        *)
 
                echo -e $TOMCAT_USAGE
 
        ;;
 
esac   
 
exit 0
 
  
 
</pre>
 
</pre>
 +
systemctl daemon-reload
  
<pre>[root@node2 tomcat]# vim /etc/init.d/tomcat(tomcat启动脚本)
+
=see also=
 
 
------------script start--------------
 
 
 
#!/bin/sh
 
 
 
# Tomcat init script for Linux.
 
 
 
#
 
 
 
# chkconfig: 2345 96 14
 
 
 
# description: The Apache Tomcatservlet/JSP container.
 
 
 
export JAVA_HOME=/usr/java/latest
 
 
 
export CATALINA_HOME=/usr/local/tomcat
 
 
 
exec $CATALINA_HOME/bin/catalina.sh $*
 
 
 
--------------script end---------------
 
 
 
[root@node2 tomcat]# chmod +x !$
 
 
 
chmod +x /etc/init.d/tomcat
 
 
 
[root@node2 tomcat]# chkconfig --add tomcat
 
 
 
[root@node2 tomcat]# chkconfig tomcat on
 
 
 
[root@node2 tomcat]# chkconfig --list tomcat
 
 
 
tomcat            0:关闭      1:关闭      2:启用      3:启用      4:启用      5:启用      6:关闭
 
 
 
[root@node2 tomcat]# service tomcat stop
 
 
 
……
 
 
 
[root@node2 tomcat]# service tomcat start</pre>
 
 
 
=safe=
 
[https://blog.linuxeye.cn/454.html fail2ban阻止SSH暴力破解]
 
 
 
== safe dns==
 
http://www.freenom.world/zh/index.html?lang=zh
 
 
 
Freenom是世界上第一个也是唯一的免费域名提供商。 我们的使命是把互联网带给世界人民,帮助各国发展自己的数字经济。
 
 
 
通过打破壁垒、整合免费域名与最新网站建设和托管技术,Freenom让企业或个人建立网站和管理内容很容易,没有任何成本。
 
 
 
=docker k8s=
 
 
 
=intra=
 
==内网穿透打洞==
 
[[内网穿透打洞]]
 
 
 
==内网环境==
 
oracle 17
 
 
 
nginx  other 11
 
 
 
frp
 
 
 
但是没有开机自启动呢
 
 
 
=apps=
 
==7.1redis==
 
公司的redis要 HA
 
 
 
[https://my.oschina.net/u/2600078/blog/1923696 redis架构演变与redis-cluster群集读写方案 ]
 
 
 
==7.2 nginx==
 
 
 
redis  lua+ cc
 
[[Nginx Lua Redis防止CC攻击]]
 
 
 
==7.3 mysql==
 
mysql gpt
 
 
 
==7.4 dns==
 
[http://blog.51cto.com/sweetpotato/1596973 Linux运维实战之DNS基础]
 
 
 
==7.5 CI jenkins==
 
  
==备份==
+
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-nginx.html
==关键文件备份==
 
passwd  etc
 
  
== 更新==
 
 
mysql 文件  这个尽量做成自动化
 
接下何焱的活
 
==db备份==
 
  
=on  =
+
[https://www.cnblogs.com/kuku0223/p/8317965.html ELK--filebeat nginx模块]
# 内网开发环境
 
# jenkins 自动化更新 (dev test)
 
# zabbix 报警
 
# pinpoint
 
#
 
#
 
#
 
# 有序列表项
 
#
 
# 有序列表项
 
#
 
# 有序列表项
 
#
 

2021年4月27日 (二) 09:25的版本


ins and config

Download and install Filebeat



curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.12.0-amd64.deb
sudo dpkg -i filebeat-7.12.0-amd64.deb

Edit the configuration

Modify /etc/filebeat/filebeat.yml to set the connection information:

output.elasticsearch:
  hosts: ["<es_url>"]
  username: "elastic"
  password: "<password>"
setup.kibana:
  host: "<kibana_url>"

Where <password> is the password of the elastic user, <es_url> is the URL of Elasticsearch, and <kibana_url> is the URL of Kibana.

Enable and configure the nginx module

sudo filebeat modules enable nginx

Modify the settings in the /etc/filebeat/modules.d/nginx.yml file.

Start Filebeat

The setup command loads the Kibana dashboards. If the dashboards are already set up, omit this command.

sudo filebeat setup
sudo service filebeat start

Module status

Module status  右边的 check data 按键  -- > Nginx logs dashboard

systemctl daemon-reload

see also

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-nginx.html


ELK--filebeat nginx模块