haproxy+kibana日志分析可视化展示平台

kibnan

haproxy配置

json格式输出日志到rsyslog日志格式不一定要按照我这个配置。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
haproxy.cfg

global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local3
log-format {"haproxy_dateTime":"%t","haproxy_clientIP":"%ci","haproxy_backendSourceIP":"%bi","http_request":"%r","reponse_headers":"%hrl","status_code":%ST,"haproxy_bytesRead":"%B","haproxy_serverName":"%s","haproxy_Tw":"%Tw","haproxy_Tc":"%Tc","haproxy_Tt":"%Tt"}
---
cat /etc/rsyslog.conf
local3.* @@192.168.200.75:516

lostash配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
[root@loanaudit02 zhphuser]# cat /etc/logstash/conf.d/haproxy.conf 
#input {
input {
truesyslog{
truetruetype => "haproxy"
truetrueport => "516"
true}
}
filter {
json {
source => "message"
#target => "doc"
#remove_field => ["message"]
}
}
filter {
truegeoip {
source => "haproxy_clientIP"
target => "geoip"
database => "/usr/local/logstash/config/GeoLite2-City.mmdb" //ip数据库地址
add_field => ["[geoip][coordinates]","%{[geoip][longitude]}"] //添加字段coordinates,值为经
add_field => ["[geoip][coordinates]","%{[geoip][latitude]}"] //添加字段coordinates,值为纬度
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}
}
#filter {
# grok {
# match => {
# "message" => "%{IPV4:cip}"
# }
# }
#}


output{
elasticsearch{
hosts=>["192.168.1.100:9200"]
index=> "logstash-haproxy-%{+YYYY-MM-DD}"
}
}

index必须以logstash开头

geoip配置

1
2
3
在logstash服务器下载IP地址归类查询库
wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
gunzip GeoLite2-City.mmdb.gz

配置kibana

1
2
3
4
5
6
kibana使用地图统计ip访问地域

安装logstash插件
/usr/share/logstash/bin/logstash-plugin install logstash-filter-geoip
kibana.yml最后一行添加使用高德地图
tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'

下面是kibana自带的几种visualize类型

类型               用途
Area chart       用区块图来可视化多个不同序列的总体贡献。
Data table       用数据表来显示聚合的原始数据。其他可视化可以通过点击底部的方式显示数据表。
Line chart        用折线图来比较不同序列。
Markdown widget    用 Markdown 显示自定义格式的信息或和你仪表盘有关的用法说明。
Metric         用指标可视化在你仪表盘上显示单个数字。
Pie chart        用饼图来显示每个来源对总体的贡献。
Tile map       用瓦片地图将聚合结果和经纬度联系起来。
Vertical bar chart    用垂直条形图作为一个通用图形。

top5ip统计

kibnan

当前请求pv

kibnan

当前请求UV

kibnan

公网ip地址分布

kibnan

请求响应代码统计

kibnan

请求URL统计

kibnan