Linux升級wget/curl用於下載https文件的過程

時間 2019-11-11

標籤 linux 升級 wget curl 用於下載 https 文件過程欄目 Linux 简体版

原文原文鏈接

由於須要提高服務器的性能以及支持mysql更新版本的某些特性，所以決定升級mysql版本從5.1.30到8.0，目標肯定下來就開始幹。html

`Mysql`安裝方式選擇

在Linux上安裝應用，通常有三種方式，優劣對比分別爲：
java

所以咱們選擇二進制安裝，安裝簡單、方便，支持多個Mysql版本同時存在。mysql

在Linux上安裝二進制版本的應用，統一爲三步：linux

// 經過配置自動生成文件
./configure
// 編譯文件
make
// 檢查自測單元，看編譯是否經過，能夠省略該步，不影響安裝
make check
// 安裝
make install

卸載經過二進制安裝的程序：git

// 方式一: 在編譯目錄裏執行卸載
make uninstall
// 方式二：找到安裝目錄，而後刪除，如nettle程序
$find / -name nettle
/usr/include/nettle
rm -rf /usr/include/nettle

`wget&curl`不支持`https`下載

咱們能夠在mysql官網下載最新版本的mysql8.0.13二進制文件，注意官網提供的下載連接是https協議的，當咱們在服務器執行下載命令：github

// 使用wget或者curl來下載文件
wget https://dev.mysql.com/downloads/file/?id=480751
curl -O https://dev.mysql.com/downloads/file/?id=480751

會報錯：redis

// wget 加上--no-check-certificate 依然不能夠
$wget https://dev.mysql.com/downloads/file/?id=480751
--2018-12-12 16:57:54--  https://dev.mysql.com/downloads/file/?id=480751
Resolving dev.mysql.com (dev.mysql.com)... 137.254.60.11
Connecting to dev.mysql.com (dev.mysql.com)|137.254.60.11|:443... connected.
GnuTLS: A TLS fatal alert has been received.
GnuTLS: received alert [40]: Handshake failed
Unable to establish SSL connection.

// curl 加上--insecure依然不能夠
$curl https://dev.mysql.com/downloads/file/?id=480751
curl: (35) SSL connect error

根據網上查詢到的答案，緣由均爲版本太低，當前的版本不支持https協議的下載:算法

So the error actually happens with www.coursera.org and the reason is missing support for SNI. You need to upgrade your version of wget.

當前的版本：sql

$wget --version
GNU Wget 1.16.3 built on linux-gnu.

+digest +https +ipv6 +iri +large-file +nls +ntlm +opie -psl +ssl/gnutls 

Wgetrc: 
    /usr/local/etc/wgetrc (system)
Locale: 
    /usr/local/share/locale 
Compile: 
    gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc" 
    -DLOCALEDIR="/usr/local/share/locale" -I. -I../lib -I../lib 
    -DHAVE_LIBGNUTLS -DNDEBUG 
Link: 
    gcc -DHAVE_LIBGNUTLS -DNDEBUG -lpcre -lnettle -lgnutls -lz -lidn 
    -lrt ftp-opie.o gnutls.o http-ntlm.o ../lib/libgnu.a 

Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Please send bug reports and questions to <bug-wget@gnu.org>.

------------------

$curl.7.19.7 --version
curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp 
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

`wget`升級

既然是版本太低，升級便可，直接安裝新版本的wget，而後卸載掉原有的wget文件便可。vim

下載完成1.20版本以後

// 得到文件wget-1.20.tar.gz
wget http://mirror.sergal.org/gnu/wget/wget-1.20.tar.gz
// 解壓縮
tar -xzvf wget-1.20.tar.gz
// 進入解壓後的文件夾
cd wget-1.20
// 開始配置
./configure
// 然而報錯了：
...
checking for GNUTLS... no
configure: error: Package requirements (gnutls) were not met:

No package 'gnutls' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables GNUTLS_CFLAGS
and GNUTLS_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.

報錯信息顯示咱們沒有安裝gnutls依賴，須要繼續進行安裝，更爲詳細的報錯信息，能夠查看config.log:

// 查看詳細報錯信息
vim config.log

...
PKG_CONFIG='/usr/local/bin/pkg-config'
...
configure:44443: checking for GNUTLS
configure:44450: $PKG_CONFIG --exists --print-errors "gnutls"
Package gnutls was not found in the pkg-config search path.
Perhaps you should add the directory containing `gnutls.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gnutls' found
configure:44453: $? = 1
configure:44467: $PKG_CONFIG --exists --print-errors "gnutls"
Package gnutls was not found in the pkg-config search path.
Perhaps you should add the directory containing `gnutls.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gnutls' found
configure:44470: $? = 1
configure:44484: result: no
No package 'gnutls' found
configure:44500: error: Package requirements (gnutls) were not met:

No package 'gnutls' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you                                                                            
installed software in a non-standard prefix.                                                                                                  
                                                                                                                                              
Alternatively, you may set the environment variables GNUTLS_CFLAGS                                                                            
and GNUTLS_LIBS to avoid the need to call pkg-config.                                                                                         
See the pkg-config man page for more details.

這裏能夠看出配置腳本實際是執行了：

$ /usr/local/bin/pkg-config --exists --print-errors "gnutls"
Package gnutls was not found in the pkg-config search path.
Perhaps you should add the directory containing `gnutls.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gnutls' found

安裝`gnutls`庫

gnutls全稱 GNU Transport Layer Security Library，即基於GNU版權協議的傳輸層安全協議，是wget支持https中的ssl協議的基礎庫。

咱們能夠在官方提供的鏡像庫裏快速下載並安裝：

// 下載gnutls二進制文件
wget http://www.ring.gr.jp/pub/net/gnupg/gnutls/v3.6/gnutls-3.6.4.tar.xz
// 解壓xz文件
xz -d gnutls-3.6.4.tar.xz
tar -xvf gnutls-3.6.4.tar
cd gnutls-3.6.4
./configure

// 報錯：
...
checking for NETTLE... no
configure: error: 
  ***
  *** Libnettle 3.4 was not found.

// 若是以爲可能不安全，能夠下載md5簽名文件驗證文件，可是這個文件驗證後發現簽名過時了，因此沒辦法驗證了
wget http://www.ring.gr.jp/pub/net...
gpg --verify gnutls-3.6.4.tar.xz.sig gnutls-3.6.4.tar
gpg --recv-key F1679A65
gpg --verify --verbose gnutls-3.6.4.tar.xz.sig gnutls-3.6.4.tar

查看詳細報錯信息：

$ vim config.log

...
configure:10032: checking for NETTLE
configure:10039: $PKG_CONFIG --exists --print-errors "nettle >= 3.4"
Package nettle was not found in the pkg-config search path.
Perhaps you should add the directory containing `nettle.pc'
to the PKG_CONFIG_PATH environment variable
No package 'nettle' found
configure:10042: $? = 1
configure:10056: $PKG_CONFIG --exists --print-errors "nettle >= 3.4"
Package nettle was not found in the pkg-config search path.
Perhaps you should add the directory containing `nettle.pc'
to the PKG_CONFIG_PATH environment variable
No package 'nettle' found
configure:10059: $? = 1
configure:10073: result: no
No package 'nettle' found
configure:10090: error:
  ***
  *** Libnettle 3.4 was not found.

結果顯示咱們須要3.4版本以上的Libnettle庫，繼續安裝。

安裝`Libnettle`庫

Nettle庫是用於跨平臺的底層密碼庫，包含加密和解密的不一樣算法。咱們下載並安裝nettle庫：

wget ftp://ftp.gnu.org/gnu/nettle/nettle-3.4.1.tar.gz
tar -xzvf nettle-3.4.1.tar.gz
cd nettle-3.4.1
./configure
// 安裝成功
...
configure: summary of build options:

  Version:           nettle 3.4.1
  Host type:         x86_64-unknown-linux-gnu
  ABI:               64
  Assembly files:    x86_64
  Install prefix:    /usr/local
  Library directory: ${exec_prefix}/lib64
  Compiler:          gcc
  Static libraries:  yes
  Shared libraries:  yes
  Public key crypto: no
  Using mini-gmp:    no
  Documentation:     yes

make
make install

根據官方文檔，咱們安裝完成後應該會有兩個文件lib{hogweed,nettle}.so，然而咱們只能發現其中一個：

make install &&
chmod -v 755 /usr/lib/lib{hogweed,nettle}.so &&
install -v -m755 -d /usr/share/doc/nettle-3.4.1 &&
install -v -m644 nettle.html /usr/share/doc/nettle-3.4.1

$ ll | grep '\.so'
-rwxr-xr-x 1 root root 3675341 Dec 12 19:15 libnettle.so
$ ll | grep weed
-rw-rw-r-- 1 work work     529 Dec 10 15:30 hogweed.pc
-rw-r--r-- 1 work work     590 Nov 19  2017 hogweed.pc.in
-rw-rw-r-- 1 work work     298 Dec 10 15:30 libhogweed.map
-rw-r--r-- 1 work work     338 Nov 19  2017 libhogweed.map.in

少了一個libhogweed.so文件，稍後咱們編譯gnutls時會發現這個致使的問題。

繼續編譯`gnutls`

既然nettle安裝完成了，咱們能夠繼續安裝gnutls：

./configure

...
configure: error: 
  ***
  *** Libnettle 3.4 was not found.

依然報錯缺失庫，但咱們明明已經安裝了，爲何找不到呢？咱們用包管理工具查找一下：

$ pkg-config --modversion nettle
Package nettle was not found in the pkg-config search path.
Perhaps you should add the directory containing `nettle.pc'
to the PKG_CONFIG_PATH environment variable
No package 'nettle' found

咱們找下這個nettle.pc剛纔安裝到哪裏去了：

$ locate nettle.pc
/home/work/lib/nettle-3.4.1/nettle.pc
/home/work/lib/nettle-3.4.1/nettle.pc.in
/usr/lib64/pkgconfig/nettle.pc
/usr/local/lib64/pkgconfig/nettle.pc

而咱們pkg-config默認的管理包檢索路徑爲/usr/lib/pkgconfig，因此沒法正常找到，參考pkgconfig文檔，有兩種方案：

// 方案一：連接該文件到默認目錄中
ln -s /usr/local/lib64/pkgconfig/nettle.pc /usr/lib/pkgconfig/nettle.pc  
// 方案二：全局變量中更改包的檢索路徑（只在本次終端窗口生效，退出後恢復，因此只能臨時使用一下）
$ echo $PKG_CONFIG_PATH

$ export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib64/pkgconfig/
$ echo $PKG_CONFIG_PATH
:/usr/local/lib64/pkgconfig/

// 任一方案執行後結果
$ pkg-config --modversion nettle
3.4.1

此時，咱們繼續安裝：

$ ./configure
...
checking for NETTLE... yes
checking for HOGWEED... no
configure: error: 
  ***
  *** Libhogweed (nettle's companion library) was not found. Note that you must compile nettle with gmp support.

能夠看到，咱們的nettle庫已經安裝成功，可是hogweed卻檢查沒有找到，提示中也寫明瞭libhogweed須要字gmp庫已經安裝完成的狀況下從新編譯nettle才能夠被安裝。

有人提出過相關的問題，咱們也能夠從官網文檔上更詳細的知道這個Nettle對於libhogweed的依賴：

5 Linking
Nettle actually consists of two libraries, libnettle and libhogweed. The libhogweed library contains those functions of Nettle that uses bignum operations, and depends on the GMP library. With this division, linking works the same for both static and dynamic libraries.
If an application uses only the symmetric crypto algorithms of Nettle (i.e., block ciphers, hash functions, and the like), it’s sufficient to link with -lnettle. If an application also uses public-key algorithms, the recommended linker flags are -lhogweed -lnettle -lgmp. If the involved libraries are installed as dynamic libraries, it may be sufficient to link with just -lhogweed, and the loader will resolve the dependencies automatically.

總而言之，就是沒有libhogweed.so這個文件不行，而它只能由nettle進行安裝。根據nettle庫官方資料顯示，libhogweed.so應該在nettle安裝時被自動生成，然而咱們在上面的安裝過程當中並無生成。那是否是由於我沒有安裝gmp致使的呢？

安裝`gmp`庫

咱們下載gmp庫並安裝，能夠在編譯Nettle的config.log中查看有一條warning，指明瞭版本需求：

$ vim config.log

...
configure:6583: result: no
configure:6594: WARNING: GNU MP not found, or too old. GMP-6.0 or later is needed, see https://gmplib.org/.
    Support for public key algorithms will be unavailable.

因此咱們須要下載6.0版本後的：

// 這裏我只找到了官網的https版本，沒辦法，只好本地下載，而後rz到服務器，由於是二進制文件，要帶上-be參數
rz -be
// 而後正常編譯
$ ./configure & make & make install
...
Libraries have been installed in:
   /usr/local/lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the 'LD_RUN_PATH' environment variable
     during linking
   - use the '-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to '/etc/ld.so.conf'

這裏提醒咱們須要將動態庫連接到緩存中，咱們採用第四種方案，能夠參考ldconfig命令：

$ vim /etc/ld.so.conf

// 添上安裝的.so文件路徑
/usr/local/lib
:wq

$ ldconfig
$ ldconfig -v | grep gmp
        libgmp.so.10 -> libgmp.so.10.3.2
        libgmpxx.so.4 -> libgmpxx.so.4.1.0
        libgmp.so.3 -> libgmp.so.3.5.0

看到libgmp.so.10就是咱們安裝的最新版本，如今OK了。

而後從新編譯安裝nettle，會生成libhogweed.so文件：

$ ll | grep weed
-rw-r--r-- 1 root root     541 Dec 12 22:12 hogweed.pc
-rw-r--r-- 1 work work     590 Nov 19  2017 hogweed.pc.in
-rw-r--r-- 1 root root 6154192 Dec 12 22:13 libhogweed.a
-rw-r--r-- 1 root root     298 Dec 12 22:12 libhogweed.map
-rw-r--r-- 1 work work     338 Nov 19  2017 libhogweed.map.in
-rwxr-xr-x 1 root root 5519996 Dec 12 22:13 libhogweed.so
-rw-r--r-- 1 root root       8 Dec 12 22:13 libhogweed.stamp

請注意若是安裝完成後，若是出現多個版本的gmp庫，請刪除老版本的。具體刪除哪一項請自行斟酌，我刪除了全部的，而後在編譯的過程當中，會報錯：can't find libgmp.so.3，說明libgmp.so.3這個是基礎庫，請不要動！
等我刪除了老版本的，從新編譯nettle就OK。若是你安裝成功了新版本，依然編譯不成功，請參考這個。

依賴地獄

用二進制來安裝的時候，老是會出現各類各樣的問題，缺乏各類依賴的包，解決方法就是缺什麼就去安什麼，可是會很是恐怖。爲了解決nettle安裝的問題，除了上面的gmp，我還安裝了最新版本的各類庫：

libunistring： https://www.gnu.org/software/...
p11-kit：https://github.com/p11-glue/p...
libffi：https://sourceware.org/libffi/
pkg-config: https://pkg-config.freedeskto...
libtasn1: https://ftp.gnu.org/gnu/libta...

同時，因爲gnutls編譯不經過的問題，又升級了pkg-config，它依賴於Libtasn1。

繼續安裝`gnutls`庫（失敗、暫時放棄更新）

./configure
// 此時沒有錯誤信息了，可是還有不少WARNING信息
*** autogen not found. Will not link against libopts.
*** You will not be able to create source packages with 'make dist'
  because gtk-doc >= 1.14 is not found.
*** LIBIDN2 was not found. You will not be able to use IDN2008 support
*** libunbound was not found. Libdane will not be built.
*** trousers was not found. TPM support will be disabled.
*** `guile-snarf' from Guile not found.  Guile bindings not built.

*** The DNSSEC root key file in /etc/unbound/root.key was not found.
*** This file is needed for the verification of DNSSEC responses.
*** Use the command: unbound-anchor -a "/etc/unbound/root.key"
*** to generate or update it.

// 繼續編譯，又報錯了
make
...
tlsproxy/buffer.c:40: error: redefinition of typedef 'buffer_t'
tlsproxy/buffer.h:31: note: previous declaration of 'buffer_t' was here

暫時放棄更新wget，過幾天繼續嘗試，解決各類問題太費時間了

若是想要減小warning信息，能夠更新autogen等庫：

autogen: https://ftp.gnu.org/gnu/autog...
guile: http://alpha.gnu.org/gnu/guile/
gcc: http://ftp.tsukuba.wide.ad.jp...
mpc: https://ftp.gnu.org/gnu/mpc/

安裝autogen過程當中須要依賴guile，然而安裝guile時又報錯：guile configure: error: Cannot find a type to use in place of socklen_t
放棄更新autogen。

嘗試`curl`更新，層層依賴，放棄

$ curl https://dev.mysql.com/downloads/file/?id=480751
curl: (35) SSL connect error

根據報錯緣由和網上資料是因爲版本過老，須要更新curl版本。

從官方地址下載curl後安裝，再次用新版本的curl請求：

$ curl https://dev.mysql.com/downloads/file/?id=480751
curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

仍是報錯，根據上面的資料，若是依然不能解決問題，須要更新NSS，NSS和OpenSSl相似，都屬於底層密碼學，由Mozilla維護，MDN文檔提供安裝說明，它跟前面的二進制文件略有不一樣，不提供configure自動配置，詳細的查看它的文檔。

安裝NSS庫比較麻煩，還須要再安裝GYP庫，想起來又是層層嵌套的依賴關係，放棄更新。

解決方案：本地下載，`rz`上傳

在耗費兩天的時間後，我及時的終止了無畏的嘗試，轉而使用本地下載mysql8.0文件，而後rz -be上傳到服務器，搞定。

總結

Linux上層層依賴的二進制文件安裝簡直是地獄版的體驗，在給咱們帶來高自由度的同時也有無盡的煩惱，然而yum安裝版本又太低，不能知足需求。雖然最終仍是沒有成功更新wget或者curl，可是在整個過程當中，也學習到了不少的新東西，在這篇文章總結一下過程，但願也能幫助一些人在某一步驟遇到的問題。