工做中用到的linux、SQL、Python語句

時間 2019-12-08

標籤用到 linux sql python 語句欄目 Linux 简体版

原文原文鏈接

SQL:html

分組取最新值後求和python

select sum(fff.asset_balance) as "總資產",sum(fff.enable_balance) as "可用資金",sum(fff.fund_balance) as "現金資產" from (
select tt.fund_account,tt.fund_balance,tt.asset_balance,tt.enable_balance,date_time from fund_real_log as tt group by fund_account,fund_balance,asset_balance,date_time having
date_time=(select max(date_time) from fund_real_log where fund_account=tt.fund_account and fund_account in ('30001691','30001708'))
) fff;
複製代碼

Python解決unicode編碼
linux

s = '\xe9\x9d\x92\xe8\x9b\x99\xe7\x8e\x8b\xe5\xad\x90'
ss = s.encode('raw_unicode_escape')
print(ss)  # 結果：b'\xe9\x9d\x92\xe8\x9b\x99\xe7\x8e\x8b\xe5\xad\x90'
sss = ss.decode()
print(sss)複製代碼

轉碼：shell

>>> t1 = '\xe6\x9f\xa5\xe8\xaf\xa2\xe6\x95\xb0\xe6\x8d\xae\xe5\xba\x93\xe5\x87\xba\xe7\x8e\xb0\xe5\xbc\x82\xe5\xb8\xb8' >>> print t1.decode('utf-8') 查詢數據庫出現異常複製代碼

python比對Excel（新版本openpyxl）數據庫

#-*- coding:utf-8 -*-
from openpyxl import *


def compareFile(ename1, ename2):
    # A bool to verify if 2 xlsx is the same.
    fileSame = True
    # Load workbook, and get sheetname
    src_book = load_workbook(filename=ename1)
    dlo_book = load_workbook(filename=ename2)
    src_sheets = src_book.sheetnames
    dlo_sheets = dlo_book.sheetnames

    if (src_sheets != dlo_sheets):
        print "Two file has different sheets."
        print ename1, "has sheet names:", src_sheets
        print ename2, "has sheet names:", dlo_sheets
    else:
        print "Two file has the same sheets."
        sn = src_sheets
        for wsn in sn:
            # Get worksheet
            src_sheet_object = src_book[wsn]
            dlo_sheet_object = dlo_book[wsn]
            c = src_sheet_object.max_column
            r = src_sheet_object.max_row
            # This can be replace by src_sheet_object.get_dimension
            if ((dlo_sheet_object.max_column != c) or (dlo_sheet_object.max_row != r)):
                print "DIFFERDENT at SHEET-", wsn, ": Rows or columns not the same!"
                fileSame = False
            else:
                print "SHEET-", wsn, ":Rows or columns are same!"
                # Compare every cell.
                flag = True
                for i in range(1, r + 1):
                    for j in range(1, c + 1):
                        c1 = src_sheet_object.cell(i, j)
                        c2 = dlo_sheet_object.cell( i, j)
                        if (c1):
                            if (c2):
                                if (c1.value != c2.value):
                                    if ((wsn == "Internal Info") and ((i == 4) and (j == 2)) or (
                                        (i == 5) and (j == 3))):
                                        continue
                                    print "DIFFERDENT_VALUE at SHEET-", wsn, ": At (", i, ",", j, ")",
                                    print "diff FROM", c1.value, "TO", c2.value
                                    flag = False
                            else:
                                print "DIFFERDENT_TO_NONE at SHEET-", wsn, ": At (", i, ",", j, ")"
                                print "diff FROM", c1.value
                                flag = False
                        else:
                            if (c2):
                                print "DIFFERDENT_TO_NONE at SHEET-", wsn, ": At (", i, ",", j, ")"
                                print "diff FROM", c2.value
                                flag = False
                fileSame = fileSame and flag
        if fileSame:
            print "SAME_FILE:", ename1, ename2


print "------------------文件比對開始------------------"
srcfile = u"文件絕對路徑"
dlofile = u"文件絕對路徑"compareFile(srcfile,dlofile)
print "------------------文件比對結束------------------"


複製代碼

出現echo展現命令執行結果未換行時加雙引號便可:bash

echo "`cmd`"複製代碼

開啓FTP服務傳文件:服務器

service vsftpd status
sudo service vsftpd start
複製代碼

抓包:tcp

sudo tcpdump -i any host IP -s0 -w tmp.cap
strings tmp.cap > tmp.txt
複製代碼

awk提取數據:性能

ps -fux | grep python | awk '{for(i=1;i<=10;i++)printf $i ";;";for(a=11;a<=NF;a++)printf $a " ";printf "\n"}'
複製代碼

memUse=`awk -v x="$memUseRate" -v y=100 'BEGIN{printf "%.2f\n",x*y}'`
awk -F '|' '{print $3}' 1120capstats.csv| awk '{sum+=$1} END {print "Average = ", sum/(NR-1)}'
複製代碼

一行命令（循環）:測試

for i in `cat 1214.txt` ; do if [[ i-tmp=1 ]]; then   tmp=i;   echo "right"; else   echo "wrong";   exit 2; fi; done
複製代碼

查看進程的具體狀態, 包括進程ID，進程啓動的路徑

jps -lv複製代碼

殺進程

for pid in `ps fux | grep excu.py | grep -v grep | awk '{print $2}'`;do `kill -9 ${pid}`; done複製代碼

檢查返回結果是否連續:

#!/bin/bash
#getResult.sh
resultFile=$1
readonly resultFile
jobid=$2
grepCondition="\"40014\":\"${jobid}\""
result=`grep -ai ${grepCondition} ${resultFile} | awk -F ',' '{print $10}'`
#echo "$result"
if [ ! "$result" ]
then
  echo "No result!!!!!!!!!!Please check your jobid!!!!!!!!!!!"
  exit 2
fi
initNum=$3
tmp=($(echo "$result"))
for item in ${tmp[@]};do
  #echo $item
  len=${#item}
  part2=${item:9:-1}
  echo $part2
  differ=`expr ${part2}-${initNum}`
  if [[ $differ -eq 1 ]] 
  then
    echo "The result is right!!!"
    ((initNum++));
  else
    echo "Wrong result!!!!!!!!!!"
  exit 2
  fi
done
複製代碼

執行語句:

./getResult.sh msg.log 745 0
複製代碼

linux浮點數據計算問題

性能測試時統計服務器指標一（比較粗）

#!/bin/bash
#script to capture system statistics
OUTFILE=/home/test/tmp.csv
`:>$OUTFILE`
echo " 時間 用戶鏈接數 系統平均負載 空閒內存 空閒CPU百分比" >> $OUTFILE

int=1

while(($int<=10000));do
    TIME=`date "+%Y-%m-%d %H:%M:%S"`
    TIMEOUT=`uptime`
    VMOUT=`vmstat 1 1`
    USERS=`echo $TIMEOUT | gawk -F ',' '{print $3}' | sed 's/^[ \t]*//g'`
    #LOAD=`echo $TIMEOUT | gawk '{for(i=9;i<=NF;i++)printf $i}'`
    LOAD=`echo $TIMEOUT | gawk '{print $9,$10}' | sed "s/,//"`
    FREE=`echo $VMOUT | gawk '{for(i=24;i<=NF;i++)printf $i " "}' | gawk '{print $4}'`
    #FREE=`echo $VMOUT | sed -n '/[0-9]/p' | sed -n '1p' | gawk '{print $4} '`
    IDLE=`echo $VMOUT | gawk '{for(i=24;i<=NF;i++)printf $i " "}' | gawk '{print $15}'`
    #IDLE=`echo $VMOUT | sed -n '/[0-9]/p' | sed -n '2p' |gawk '{print $15}'`
    echo " $TIME | $USERS | $LOAD | $FREE | $IDLE" >> $OUTFILE
    ((int++))
    sleep 1
done
複製代碼

性能測試時統計服務器指標二（比較粗）

#!/bin/bash 
OUTFILE=/home/test/newTmp.csv

`:>newTmp.csv`

echo " 時間 系統平均負載 CPU使用率 內存使用率 " > $OUTFILE

int=1

while(($int<=100000));do
    TIME=`date "+%Y-%m-%d %H:%M:%S"`
    cmd="`top -b -n 1 | head -n 5|sed s/[[:space:]]//g`"
    loadaverage=`echo "$cmd" | sed -n "1p" | gawk -F ',' '{print $4,$5,$6}'`

    us=`echo "$cmd" | sed -n "3p" | gawk -F ',' '{print $1,$2,$4}' | gawk '{print $1}' | gawk -F ':' '{print $2}' | sed "s/us//"`
    #echo $us
    sy=`echo "$cmd" | sed -n "3p" | gawk -F ',' '{print $1,$2,$4}' | gawk '{print $2}' | sed "s/sy//"`
    #echo $sy
    cpuUs=`echo "$us+$sy"|bc`
    #echo $cpuUs

    memused=`echo "$cmd" | sed -n "4p"| gawk -F ':' '{print $2}' | gawk -F ',' '{print $3}'| sed "s/used//"`
    memtotal=`echo "$cmd" | sed -n "4p"| gawk -F ':' '{print $2}' | gawk -F ',' '{print $1}' | sed "s/total//"`
    memUseRate=`echo | awk -v a=$memused -v b=$memtotal '{printf("%.4f",a/b)}'`
    memUse=`echo "$memUseRate*100"|bc`   #linux shell浮點計算

    echo "$TIME | $loadaverage | $cpuUs | $memUse " >> $OUTFILE
    ((int++))
    sleep 1
done
複製代碼

性能測試時統計服務器指標計算：

#!/bin/bash 
#index=$1

file=$1

#if [ ! $3 ];then
# $3=' '
#
#rmcode=$3

echo "12123213123213"

initlist=`cat $file | gawk -F '|' '{print $3}' | gawk -F ':' '{print $2}' | sed "s/ //"`

echo "34534534534543"

tmp=($(echo "$initlist"))
sum=0.0
nums=0
for item in ${tmp[@]};do
  sum=`echo "$sum+$item"|bc`
  ((nums+=1))
done

echo $sum

averageLoad=`echo | awk -v a=$sum -v b=$nums '{printf("%.2f",a/b)}'`

echo $averageLoad
複製代碼