nginx日誌統計相關命令和python腳本

src=(s1,s2,s3,s3)

for i in ${src[@]

 

 

nginx訪問量統計

1.根據訪問IP統計UV
awk '{print $1}'  access.log|sort | uniq -c |wc -l

嘉圖的統計 awk '{print $1}'  /var/log/nginx/access.log|sort | uniq -c |wc -l
2.統計訪問URL統計PV
awk '{print $7}' access.log|wc -l
3.查詢訪問最頻繁的URL
awk '{print $7}' access.log|sort | uniq -c |sort -n -k 1 -r|more
4.查詢訪問最頻繁的IP
awk '{print $1}' access.log|sort | uniq -c |sort -n -k 1 -r|more
5.根據時間段統計查看日誌
 cat  access.log| sed -n '/14\/Mar\/2015:21/,/14\/Mar\/2015:22/p'|more

6．根據時間點統計查看日誌.

[root@hemlock nginx]#  sed -n '/22\/Apr\/2016:10:10:00/,/22\/Apr\/2016:10:11:00/p' access.log|awk '{print $1,$6}'|sort|uniq -c |sort -nr|more

查看某個時間點的

 

7.查看服務器併發的

[root@hemlock php-fpm]# netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'

 

[root@hemlock ~]#  netstat -ant | grep ESTABLISHED|more

查看鏈接從哪裏來

 

8．Mongo查看單個IP信息的語法

. db.access20160417.find({"remote_addr" : "222.64.69.202"})

 

10.在10.1.0.9查看這個IP有沒登錄網站

mysql> SELECT * from garmentoffice.login_log where ip_address='5.9.98.178';

11顯示主要信息

 

11[root@hemlock nginx]# grep "120.236" access.log |awk '{print $1,$5,$6,$8,$9,$10,$11,$12,$13,$14,$15}' |sort |uniq -c |sort -nr|more

   1287 120.236.141.135 "GET /websocket/GetMemberInfo 403 0.002 "http://www.haodingdan.com/company-335380.html?spm=10001" "Mozilla/5.0 (Windows NT 6.1; WOW64)

顯示了ip數和ip  顯示了useragent  顯示了utl  顯示了主頁  顯示了狀態 和顯示request_time=0.002  GET是訪問的uri  後面url是referer

 

12.統計目前訪問最多的uri 和ip和其餘參數

awk '{print $1,$5,$6,$8,$9,$10,$11,$12,$13,$14,$15}' access.log |sort |uniq -c |sort -nr|more

 

13,知道ip查找來自哪一個機房域名的網站

http://ip.chinaz.com/?qq-pf-to=pcqq.group

 

14數據庫監控負載的地址

http://10.1.0.12:8080/render/?width=1133&height=650&target=collectd.quivertree.load.load.shortterm&target=collectd.lime.load.load.shortterm&target=collectd.apple.load.load.shortterm&from=-8hours&fontSize=14

 

15mongo查看最慢的全部信息中最慢的request_time

db.access.find({},{"http_x_forwarded_for":1,"request_time":1,"request":1,"time":1}).sort({"request_time":-1}).limit(10);

注意這裏的-1  這裏是以request_time排序而且限制出來10條 ，其實mongo最多也就一次出來20條信息

 

16mongo查看單個IP的信息這裏是request.time排序

db.access.find({http_x_forwarded_for:'220.181.108.160'},{"time":1,_id:0,status:1,request_time:1,http_user_agent:1,request:1,http_x_forwarded_for:1}).sort({request_time:1})

 

 

17 阿里雲工單地址

https://workorder.console.aliyun.com/console.htm#/ticket/detail/?ticketId=DG6M0A9

 

18.雲監控安裝步驟

https://help.aliyun.com/document_detail/cms/User_Manual/CloudServiceMonitor/CloudMonitorGuide.htm

選擇安裝新版雲監控插件

 

19. mongodb數據庫刪除集合

db.collection.drop()刪除一個集合的命令

好比 db.access20160511.drop()

 

20.centos7.1安裝計劃任務

yum install crontabs  -y

 

21 show variables   like  "%gen%";

搜索mysql的某個配置

 

 

22 php的平滑重啓

kill  -USR2  `cat /run/php-fpm/php-fpm.pid` 

 

23  mongo的正則匹配查詢只要加一個$regex 就能夠查詢/sync/check/後面可能還不少。均可以查詢獲得

db.access20160622.find({"request":{$regex:"/sync/check"}}).limit(5)

 

24 cedar每日統計錯誤代碼好比500 404

 nohup php script.php backend/mongo_log/countStatus/access20160603 >> access.log &

 

25  查詢某個字段相關一共有多少條數據

 db.user_inbox_session.find({unread_count:-1}).count()

 

26 查看訪問的request、

 sh check_options.sh  danxin2.haodingdan.com|awk '{print $6,$7,$15}'|sort|uniq -c |sort -nr|more

 

27阿里雲app地址

https://push.console.aliyun.com/?spm=5176.doc34370.2.2.kJCWpr#/app/deviceStat/?appId=2308580&appKey=23287577&appName=%E5%8D%83%E8%AF%A2&source=

 

28 mongo的模糊查詢

db.access20160701.find({"request":{$regex:"session_id=9/-1000038/0,0&to=-1000038&user_id=342322"}},{request:1,time:1,_id:0,request_time:1}).sort({request_time:-1})

 

 

29 查看mongo的索引

HaodingdanRepSet:PRIMARY> db.user_inbox_session.getIndexes();

 

db.purchase_intention.getIndexes();

 

30 查看mongo表中的一條信息i

db.user_inbox_session.find().limit(1);

 

31 若是須要屏蔽改爲1 下面是屏蔽的寫法，必定指定id ，要否則所有都會被屏蔽   不屏蔽是0 ·執行的時候給老大看下····

 update haodingdan.com_base_info set is_shield = 1 where member_id in( 1212 ,12121,);

 

32 統計danxin2站有referer_http

cat  /var/log/nginx/access.log|grep danxin2.haodingdan.com|awk '{print $10}'|sort |uniq -c |sort -nr|more

說明 $10正好是_這個referer位置

 

33 統計日誌請求大於3就

sh query_time.sh 08:00 09:00 |grep  /websocket/CheckOnline |awk '{if ($5 > 3) print $1,$2,$3,$4,$5,$6,$7,$8}'

 

34 查看是否有索引

show index from tblname;

 

35  svn拉取以前的版本代碼下來，注意我這裏只能拉取目錄 不能拉取文件  -r後面的1862是指定show log最後一次的版本號

 

svn  co -r 1862  svn://packages.garmentoffice.org/hdd_modules/business_objects/branches/1.6.0/src/Haodingdan/BusinessModule/

 

Python

#統計PV，UV

#-*-  coding: UTF-8  -*-

ips = []

with open(r'C:\Users\qinghesh\Desktop\view\nxd-frontal-gateway.2017-12-17.log') as f:
for line in f:
ip = line.split(',')[1].split(':')[1].strip('"')
ips.append(ip)

print('PV is {0}'.format(len(ips)))
print('UV is {0}'.format(len(set(ips))))
print('The ip is {}'.format(set(ips)))

 

#統計httpd code

d = {}

with open('access_log',encoding='utf-8') as f:
for line in f:
key = line.split()[8]
d.setdefault(key,0)
d[key] += 1


sum_request = 0
err_request = 0


for key,val in d.items():
print(key,val)
if int(key) >= 400:
err_request += val
sum_request+= val
print(sum_request)


print('error rate: {0:.2f}%'.format(err_request * 100.0 / sum_request))

 

 

#統計url

from collections import Counter
from collections import Iterator

c = Counter()
with open('access_log') as f:
for line in f:
c[line.split()[6]] += 1

print("Poular resources : {0}".format(c.most_common(3)))