java Jsoup爬取靜態網站

 1    public static void configUrl() throws IOException {
 2         setAgent("10.1.111.14", "1080");
 3         System.setProperty("https.protocols", "TLSv1.2,TLSv1.1,SSLv3");
 4         Document document = Jsoup.connect(url)
 5                 .header("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8")
 6                 .header("Accept-Encoding", "gzip, deflate, sdch")
 7                 .header("Accept-Language", "zh-CN,zh;q=0.8")
 8                 .header("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36")
 9                 .get();
10         urlElementList = new ArrayList<Element>();
11         Elements aSrcs = document.select("#News");
12         for (Element aSrc : aSrcs) {
13             Elements aes = aSrc.getElementsByTag("a");
14             for (Element ac : aes) {
15                 urlElementList.add(ac);
16             }
17 
18         }
19     }

header()內是爲了模擬請求頭，setAgent(ip,port)是設置代理服務器（第一篇有介紹），用來爬取國外網站，國內的不用加。

加入jsoup的maven依賴

<!-- https://mvnrepository.com/artifact/org.jsoup/jsoup -->
    <dependency>
      <groupId>org.jsoup</groupId>
      <artifactId>jsoup</artifactId>
      <version>1.8.3</version>
    </dependency>