linux shell & bash

shell & bash

shell指容許用戶經過文本操做計算機的程序。

interactive shell：從是否經過標準輸入輸出與用戶進行交互的角度分爲交互式shell（interactive）和非交互式shell（non-interactive）。

login shell：從是否以一個指定用戶及其環境進入shell角度分爲登陸式shell和非登陸式shell，登陸式shell會額外source /etc/profile,/etc/profile.d,~/.profile,~/.bash_login等profile相關腳本。

變量$-包含了當前shell的選項參數，一個字母表明一種選項。部分以下：

• H histexpand
• m monitor
• h hashall
• B braceexpand
• i interactive

判斷是不是交互式shell：根據$-輸出是否包含字母i來判斷。

case "$-" in
*i*)    echo This shell is interactive ;;
*)      echo This shell is not interactive ;;
esac

if [[ "$-" == *i* ]]; then
    echo This shell is interactive
fi

當前使用的shell類型（bash, cshell...）：echo \(0；注意\)SHELL是默認shell，並不是當前shell，也就說若是在echo $SHELL不是當前shell的類型。

uniq只會刪除連續重複的行，也就說uniq不在輸入的全部行上作distinct，只是在剪掉連續出現的行。想要對在輸入的全部行上作distinct，能夠考慮先sort，而後uniq，但須要注意這樣會改變輸入中行字符串出現的順序。

iostat查看系統io，iftop查看網絡流的狀況。

ssh鏈接在很短的不活躍時間後自動斷開，如需延長這個閒置時間，經過在ssh配置文件中修改ServerAliveInterval ServerAliveCountMax兩個量實現，前者指給遠程發送心跳包的時間間隔，後者指連續發送心跳包的最大次數。
~/.ssh/config或/etc/ssh/ssh_config中添加兩行：

# 每60秒發送一個心跳包
ServerAliveInterval 60
# 最多連續發送3次
ServerAliveCountMax 3

ssh命令行參數提供密碼：

sshpass -p PASSWORD ssh -p PORT USER@HOST

sudo -E 保留當前環境變量執行命令

sudo -i 登入root用戶shell

查看當前使用的shell：
echo $SHELL

查看系統可用的shel：
cat /etc/shells

更改用戶默認的shell：
修改/etc/passwd文件對應用戶最後的shell配置。

bash相關配置文件

登陸bash shell相關（ubuntu）：/etc/profile -> /etc/bash.bashrc（在/etc/profile中被source） -> /etc/profile.d/ -> ~/.bash_profile -> ~/.bash_login -> .bashrc

交互式bash shell： /etc/bash.bashrc -> .bashrc

若是shell不是bash，/etc/profile中不會source /etc/bash.bashrc，如啓動intellij idea IDE時。

shell script腳本

shell腳本（shell script）是指由一系列shell命令組成的腳本程序文件。shell script不是interative shell，也不是login shell。

當心ls、grep等帶有顏色的輸出，輸出字符串看起來同樣的，在作uni時被視爲不同，則當心其中有個輸出可能帶有顏色字符，用cat -A確認。

for, while中計數

counter=0
for/while...
  counter=$((counter+1))

定義環境變量：
export name=value
取消定義：

交換文件中以逗號分割的前兩列的純bash腳本實現：

while IFS=, read x1 x2 remainning
do
   echo ${x2},${x1},$remainning
done < FILE

$PS1中的特殊符號：
\u：用戶名
\h：短主機名
\W：working directory(basename)
\s: shell名字（bash、sh）
\v: shell version
\d: 英文日期
\D{fmt}：日期格式（man strftime）
\n: 換行（多行提示符相關）
\w: full path of working directory
\H: full hostname
以及顏色標籤[\exxx]

若是不存在則建立目錄
[ -d /path/to/dir ] || mkdir -p /path/to/dir

io重定向 io redirect:
stdout stderr輸出到同一個文件：
ls -l /usr > out 2>&1 或者 ls -l /usr &> out。
stdout stderr輸出到單獨文件 ls -l /usr > stdout 2> stderr。
忽略stderr輸出 ls -l /usr > stdout 2>/dev/null

date

將epoc seconds格式化輸出 date -d @1305741420。注意，需帶@前綴。 若想將millis（毫秒）格式化輸出則先去掉數值後3位：echo 1305741420 | cut -b1-10 | xargs -I {} date -d @{}

xargs

將xargs的輸入（一般是前面管道的輸出）做爲待執行命令最後一個參數，若是須要拼接固定（或其餘形式格式化），能夠指定替換串形式，如：echo 1305741420 | xargs -I {} | date -d @{}，則指定{}爲替換串，會將命令形式date -d @{}中的{}替換爲xargs接收到的輸入。

awk

行處理，根據列是否知足條件輸出、或作進一步處理，這樣的需求徹底能夠考慮使用awk。
忽略#開頭的行 awk '$1 ~ /^[^#]/'
忽略#開頭的行，輸出第三列（默認以空格分隔）以<開頭的行awk '$1 ~ /^[^#]/ && $3 ~ /^</'
設置分隔符爲逗號,：awk -F, awk 'BEGIN{FS=,}'

sed

comm

找出已排序的兩個文件中的相同行、不一樣行，差集、交集、並集皆可實現。
輸出3列，分別是：FILE1中獨有行、FILE2中獨有行、共有行。
comm接受- 參數，表示不輸出第 列，能夠是多個列
comm -12僅輸出兩文件共有行
comm -23僅輸出FILE1獨有行
comm -13僅輸出FILE2獨有行

join

合併已排序的兩個文件中的行。

top

• -b 輸出到文件
• -p 指定pid, -pN1,N2,N3
• -O 打印能夠輸出的字段，至關於對-o的幫助信息
• -n 迭代次數 -n num
• -O 排序字段 [+|-]fieldName

lsblk

分辨ssd/hdd：lsblk -d -o name,rota 輸出0表明ssd，1表明hdd。
lsblk -o name可展現出物理盤的從屬虛擬硬盤（slave disk），如物理盤虛擬出的lvm盤。

迭代文件中的行

cat 默認會以空格分割字符串，所以若是文件中行包含空格，則cat不能迭代成功，解決方案是在cat文件以前把IFS變量設爲換行符，而後用cat迭代，完成後應該把IFS重置爲以前的值
對稍大一點（上百M）的文件，速度至關慢。

IFS=$'\n'
for line in `cat file`
do
    ...
done

控制流程：

if ... ; then
...
elif ... ; then
...
else
...
fi

function

#定義函數，關鍵字function是可不寫
function fun(){
    #獲取參數
    echo $1 , $2;
    #返回值，可選
    return $(($1 + $2))
}
#調用函數，須要先定義
fun 5 7;
#獲取函數返回值
s = $?

輸出錯誤信息到stderr： echo .... >&2

字符串

判斷字符串是否包含給定字符串：

if [[ <string> == *"sub-string"* ]]; then
    ...
fi

下載

wget
根據url下載小文件 wget -nd URL。選項-nd指明不建立目錄，不然程序將根據url建立層級目錄；選項-b指明後臺下載；選項-d打印處理過程；-v指明更詳細輸出。

定時任務

編寫crontab文件

crontab <crontab-file>
crontabl -l # 列出任務

linux文件權限rwx

文件：

目錄：

鏈接其餘電腦

ssh ...
#在ssh中
export DISPLAY=:0
[sudo] notify-send <msg>  #右上角顯示提示信息
[sudo] xmessage <msg> [-timeout <n>(單位：sec)] [-buttons OK,Cancle] #以xmessage醜陋的窗口彈窗顯示信息

#某些能夠從命令行打開GUI的程序，如redis-desktop-manager
...

命令及功能：

[ -d "~" ] is false
[ -d ~ ] is true
[ -d "$HOME" ] is true
apt-cache show app # check whether app is installed
apt-get update # update apt repository

telnet <CTRL+]> back to telnet then quit or <CTRL+D> to quit telnet

tail -n +NUM FILE # show line NUM to end

export NAME=xxx
unset NAME

# detect encoding of tile content
enca FILE
# convert encoding of file content
enconv FILE
# convert encoding of text
iconv -f from-encoding -t to-encoding
# convert encoding of file name
convmv -f from-encoding -t to-encoding --notest
# convert file content between character sets
recode FROM-ENC..TO-ENC FILE    (e.g. recode gbk..utf8 /a.txt)
# list the supported character sets of recode
recode -l

lsusb
lscpu

# specify an encoding before extracting
unzip -O encoding
unzip -O cp936 -l chap04.zip | head -n -2 | sed -n '4,$p' | awk '{print $4}' | grep 
# read the contents of files in a zip file; extract file into stdout
unzip -c ZIPFILE ENTRY

locate:
 locate -b(basename) xx ; locate -r regex

# manual for command
man xx; man [1-7] xx; man 7 man
# help for shell-built command 
help <CMD> # help cd; help pwd
# document
info xx
# search man page names and descriptions
apropos xx

# block device, uuid, &...
(sudo) blkid /device...  ;  blkid -s UUID /device ;  blkid -s LABEL /device


# To cut out the third field of text from file or stdin, which is delimited by a #:
cut -d # -f3
# first char of column instead of whole column, cut twice
cut -d # -f3 | cut -c 1  # cannot cut -f3 -c 1

# sort file name numerically
ls /dev/sda* | sort -k 1.9n
# list files sorted by size(desending order)
ls -S
# list files sorted by time(desending order)
ls -t

# list files, dereference symbolic link
ls -L ...

# short desciption of ascii characters
man 7 ascii

# directory or file usage
du -hs FILE
du -a <directory> # including files not only directory
du -d <depth> FILE 

# disk usage
df -lh /mount/point /# or mounted /dev/xxx, if /dev/xx not mounted, output is not what we expected

# disk ssd or hdd
lsblk -d -o name,rota  # 0 for ssd, 1 for hdd
cat /sys/block/sdx/quene/rotational

# create an iso image file from a cd/dvd/bd
dd if=/dev/cdrom of=/path/to/image.iso

# eject a disc
eject   eject cdrom    eject /mount/point    eject /dev/cdrom

# mount Windows shared folder, mount samba share (cifs <- sambfs)
mount -t cifs -o username="WinUser",password="secret",uid="localuser",gid="localgroup",dir_mode=0777 //192.168.x.x /mnt/win
# maybe need run the following command before mount
apt install cifs-utils # if no cifs filesystem supporting lib in system (by checking if /sbin/mount.cifs exists)

# mount an iso image file
mount -o loop /path/to/image.iso /path/to/mount/point
# mount partition by non-superuser
gvfs-mount -d /dev/sda1 /mnt/point/ # gvfs-* for gnome user only
# unmount
gvfs-mount -u <location>
# list
gvfs-mount -l

無密碼訪問遠程主機R
# generate key pair on local host
ssh-keygen -t rsa 
#copy public key id_rsa.pub into remote host R
scp x.pub user@host:/home/u/.ssh/
# on remote host
cat x.pub >> .ssh/authrized_keys

# cgroup: control group
# sudo apt-get install cgroup-bin

# share a read-only/readable-writable directory over lan
?????

# find devices on lan
????

# find a device on lan then mount
?????

# list open files for a process
ls /proc/PID/fd
lsof -p PID

# determine / identify processes using files or sockets
fuser name...

# merge two folders
rsync -abviuzP src/ dest/
    -i turns on the itemized format, which shows more information than the default format
    -b makes rsync backup files that exist in both folders, appending ~ to the old file. You can control this suffix with --suffix .suf
    -u makes rsync transfer skip files which are newer in dest than in src
    -z turns on compression, which is useful when transferring easily-compressible files over slow links
    -P turns on --partial and --progress
    --partial makes rsync keep partially transferred files if the transfer is interrupted
    --progress shows a progress bar for each transfer, useful if you transfer big files

# merge two folders with cp, tip: hard link: cp -r --link dir1/* dir2/* merged/ && rm -r dir1/ dir2/
# IMPOSSIBLE: create necessary destination directories when copying file, e.g. cp foo.txt /path/not/existing/foo.txt

# telnet to a ssl port(e.g. 443)
openssl s_client -connect host:port

# shared library dependencies
ldd BINARY

# list symbols from object files
nm OBJ(?.o) LIB(?.so ?.a)

# set shared library search paths(directories), add paths to /etc/ld.so.conf, then run ldconfig
vim /etc/ld.so.conf
ldconfig
# report missing objects
ldd -r /path/to/ELF_binary
ldd /path/to/executable | grep 'not found'

# 
readelf -d <binary> | grep RPATH

# restore trash that deleted as root
su
cd /path/to/parent/of/file/trashed
restore-trash

# change label of partitions
# mlabel or fatlabel for fat32
# ntfslabel for ntfs
# exfatlabel for exFAT
# e2label for ext2/ext3/ext4
# btrfs for BTRFS

# change label of a fat32 filesystem
sudo mlabel -i /dev/sdb1 ::"new_label"
#if fails & print '... add mtools_skip_check=1 to .mtoolsrc', do
#echo "mtools_skip_check=1" >> ~/.mtoolsrc # as the program says
# get or set label of ms-dos filesystem
sudo fatlabel /dev/sdc1 [new_label]
# change label of a ntfs filesystem
sudo ntfslabel /dev/sda7 "new_label"
# change label of a exfat filesystem
sudo exfatlabel /dev/sdc1 "new_label"
# change label of a ext2/3/4 filesystem
sudo e2label /dev/sdc1 "new_label"

# list background jobs
jobs
bg
# foreground jobs
fg

# get gpu information
lspci -v | grep VGA -A 12

# grep
grep PATTERN {FILE | stdin}
 -r # deep in FILE recursively
 -i # case-insensitive
 -v # not matching
 -A NUM, --after-context=NUM # print NUM lines after matching lines
 -B NUM, --before-context=NUM # print NUM lines before matching lines
 -C NUM, --context=NUM
 -H, --with-filename
 -h, --no-filename


sudo ssldump -i lo port 443 

# delete entries from a zip file
zip -d foo.zip foo/entry/to/delete
# delete a directory from a zip file
zip -d foo.zip "foo/dir/*"

# command information
type <command>
# locate a command
which <command> 

# make naming fifo
mkfifo
# make special files
mknod <name> <type>

# change password of current user
passwd
# change another user
sudo passwd <username>

# add group
sudo addgroup <group-name>
# add user
sudo adduser -ingroup <group-name> <user-name>

# login as / become another user, default as root
su - <user-name>

# tell login shell or not
shopt login_shell

# add user in sudoers
# just put the user into 'sudo' group
usermod -a -G sudo <USER-NAME>
su -        # login as root
visudo
# then move to the last line, write <user-name> ALL=(ALL:ALL) ALL  then ctrl-X to exit, Alt-B to backup previous version file

# screen resolution
xdpyinfo | grep dimensions

# groups of user
groups <user-name>...
# list groups of current user
groups

# add user into group
usermod -a -G <group-name> <user-name>  #e.g usermod -a -G sudo jack

# top usage
memory usage summary value scale: E (circle in KB<-->PB)
memory usage of process: swap between size/percentage: ?
sort by memory usage:M
sort by cpu usage:P
toggle command/command-line: c

# memory usage
free -m (by M unit)
free -g (by G unit)

ps aux -p <PID>

screen [-S session-name] # create a screen sesion
# run some long-time-consuming script
screen -d [session-name] # <Ctrl-A d> detach from session
screen -ls # list sessions
screen -r <session-name> # attach to session
screen -d -r <session-name> # attach , detach first if necessary
# which sesseion i am in
echo $STY
# exit: close session then exit

# to leave the process running after closing the terminal
<ctrl+z> <- suspend the process and back to terminal
bg  <- show background jobs, find out the job id of corresponding process
disown -h %<jobib>  <- let the job run in background

pgrep -f <pattern> # search pattern in command line (not only in command)
pgrep -a <pattern> # print full command

# find files whose name ends with .c in (and desending into) current directory
find . -name "*.c" # quotes are must. or: find . -name \*.c  or: find .-name '*.c'
# 在大範圍內查找（如根目錄/）是很魯莽的，find在大範圍內查找的開銷較大

#!!! * in a shell command line will be expanded as file names(hidden files excluded), if it is a wildcard in an argument, should be enclosed by qutoes or escaped
# find . -name *.c => means find . -name x.c y.c z.c  where x.c y.c z.c are only files ending with .c in current directory
# find . -name \*.c or "*.c" '*.c' => means find files in .(and desending), which name ending with .c
# find files on which no read-permission
find <FILE> ! -perm -u+r # user-non-readable
find <FILE> ! -perm -g+r # no permision on group
# find files by size
find . -size +100M # larger than 100M
find . -size -100M # smaller than 100M
# find files by owner
find . -user <USER> # username or user id all allowed
# find files by time in day unit, -ctime, -atime, -mtime;  c: change(attributes been changed), a: access, m: modify
# by time in minute unit, -cmin, -amin, -mmin
# time argument, +<NUM> over NUM day/min, -<NUM> within NUM day/min
find . -ctime +2 -and -mmin 60 # 
# striping leading './' of output of find .   ; find * will not search hidden file/dir(file of name starts with '.')
find -printf '%P\n'


# md5 checksum
md5sum FILE...

# gui sudo
gksudo <cmd line>

# print http headers
curl -v <URL>

# startup services management
sysv-rc-conf

# change owner of symbolic file, put -h (no dereference) option
chown -h user:group FILE

# mount remote directory conneting by ssh
sshfs user@host:/path /local/path
# umount remote directory
fusermount -u /local/path

# determine linux distribution info
ls /etc/*-release
# or
lsb_release -a

# print kernal version
uname -a

# print listening, established connections with specific port
(sudo) lsof -i:<PORT> # lsof -i:8080
(sudo) netstat -tunlp|grep <PORT> # -tunlp: tcp, udp

# list symbols in a shared library
readelf -s /path/to/libx.s

# encode a image or a file in base64
base64 <FILE>

# system time(date)
date
# hardware clock time
hwclock   # sudo hwclock

# dmesg, messages from kernel
dmesg

# awk
# swap columns
awk '{print $2"\t"$1}' <FILE>
# accumulate numbers, one per line (sum)
awk '{s+=$1} END {print $s}' <FILE>
# using shell variable in awk, you cannot use one directly
awk -v home=$HOME 'print(home)'  # careful: it's home, not $home
# output columns satisfy criteria, e.g not begin with #, and 3rd column starts with <
awk '$1 ~ /^[^#]/ && $3 ~ /^</'


# create directory if not exist
[ -d /dir ] || mkdir -p /dir

ss -tlnp

# rename network interface temporarily, 'eth0' -> 'eth1'
ifconfig eth0 down
ip link set eth0 name eth1
ifconfig eth1 up
# rename network interface permanently, edit /etc/udev/rules.d/some-file


# io statistics
iostat
iostat -x 2 5 # extented info, update every 2sec, total 5 updates

# top io wait process
iotop

# list open files
lsof -p <pid>

# copy into gui clipboard
xsel -b

# compare unique file line by line, make lines *sorted* & *unique* in each file
comm FILE1 FILE2   # a hyphen - means stdin, only one hyphen allowed
 -1 supress column 1 - unique lines in FILE1
 -2 supress column 2 - unique lines in FILE2
 -3 supress column 3 - unique lines in FILE1 & FILE2
 -12 tells intersect, -23 tells {FILE1} - {FILE2}, -13 tells {FILE2} - {FILE1}

# file status
stat FILE

# system uptime
uptime

# network monitor
iftop

# change user name
usermod -l new_name user_name
# move home dir, -m -d
usermod -m(move) -d <NEW_DIR> <USER>
# change uid
usermod -u <NEW_UID> <USER>
# append user to group, add group to user, -a -G
usermod -a -G <GROUP,GROUP2...> <USER>
# list user and group id
id
# list id and belonging groups
id <USER>
#change group name
groupmod -n <NEW_GP> <GROUP>