最近在公司作個系統,因爲要獲取網頁的一些數據,以及一些網頁的數據,因此就寫的一個公用的HttpUtils.下面是針對烏雲網我寫的一個例子。java
1、首先是獲取指定路徑下的網頁內容。node
public static String httpGet(String urlStr, Map<String, String> params) throws Exception { StringBuilder sb = new StringBuilder(); if (null != params && params.size() > 0) { sb.append("?"); Entry<String, String> en; for (Iterator<Entry<String, String>> ir = params.entrySet().iterator(); ir.hasNext();) { en = ir.next(); sb.append(en.getKey() + "=" + URLEncoder.encode(en.getValue(),"utf-8") + (ir.hasNext() ? "&" : "")); } } URL url = new URL(urlStr + sb); HttpURLConnection conn = (HttpURLConnection) url.openConnection(); conn.setConnectTimeout(5000); conn.setReadTimeout(5000); conn.setRequestMethod("GET"); if (conn.getResponseCode() != 200) throw new Exception("請求異常狀態值:" + conn.getResponseCode()); BufferedInputStream bis = new BufferedInputStream(conn.getInputStream()); Reader reader = new InputStreamReader(bis,"gbk"); char[] buffer = new char[2048]; int len = 0; CharArrayWriter caw = new CharArrayWriter(); while ((len = reader.read(buffer)) > -1) caw.write(buffer, 0, len); reader.close(); bis.close(); conn.disconnect(); //System.out.println(caw); return caw.toString(); }
瀏覽器詢問結果:正則表達式
代碼詢問結果與上面一致:瀏覽器
2、經過指定url獲取,網頁部分想要的數據。app
對於這個方法,要導入Jsoup包,這個能夠本身在網上下載。ide
Document doc = null; try { doc = Jsoup.connect("http://www.wooyun.org//bugs//wooyun-2016-0225856").userAgent("Mozilla/5.0 (Windows NT 10.0; Trident/7.0; rv:11.0) like Gecko").timeout(30000).get(); } catch (IOException e) { e.printStackTrace(); } for(Iterator<Element> ir = doc.select("h3").iterator();ir.hasNext();){ System.out.println(ir.next().text()); }
對於那個select選擇器,根據條件來選擇,doc.select("h3").iterator(),對於Jsoup有如下規則:工具
jsoup 是一款基於Java 的HTML解析器,可直接解析某個URL地址或HTML文本內容。它提供了一套很是省力的API,可經過DOM,CSS以及相似於jQuery的操做方法來取出和操做數據。
jsoup的強大在於它對文檔元素的檢索,Select方法將返回一個Elements集合,並提供一組方法來抽取和處理結果,要掌握Jsoup首先要熟悉它的選擇器語法。
一、Selector選擇器基本語法ui
二、Selector選擇器組合使用語法編碼
三、Selector僞選擇器語法url
注意:上述僞選擇器索引是從0開始的,也就是說第一個元素索引值爲0,第二個元素index爲1等。
瀏覽器訪問:
代碼訪問:
源代碼HttpUtils.java:
package com.ffcs.lsoc.bug.utils; import java.io.BufferedInputStream; import java.io.CharArrayWriter; import java.io.InputStreamReader; import java.io.Reader; import java.net.HttpURLConnection; import java.net.URL; import java.net.URLEncoder; import java.util.Iterator; import java.util.Map; import java.util.Map.Entry; import org.jsoup.nodes.Document; import org.jsoup.nodes.Element; /** * 抓取網頁工具類 * @author g-gaojp * @date 2016-7-10 */ public class HttpUtils { /** * 獲取網頁數據 * @param urlStr 訪問地址 * @param params 參數 * @param charset 字符編碼 * @return * @throws Exception */ public static String httpGet(String urlStr, Map<String, String> params,String charset) throws Exception { StringBuilder sb = new StringBuilder(); if (null != params && params.size() > 0) { sb.append("?"); Entry<String, String> en; for (Iterator<Entry<String, String>> ir = params.entrySet().iterator(); ir.hasNext();) { en = ir.next(); sb.append(en.getKey() + "=" + URLEncoder.encode(en.getValue(),"utf-8") + (ir.hasNext() ? "&" : "")); } } URL url = new URL(urlStr + sb); HttpURLConnection conn = (HttpURLConnection) url.openConnection(); conn.setConnectTimeout(5000); conn.setReadTimeout(5000); conn.setRequestMethod("GET"); if (conn.getResponseCode() != 200){ throw new Exception("請求異常狀態值:" + conn.getResponseCode()); } BufferedInputStream bis = new BufferedInputStream(conn.getInputStream()); Reader reader = new InputStreamReader(bis,charset); char[] buffer = new char[2048]; int len = 0; CharArrayWriter caw = new CharArrayWriter(); while ((len = reader.read(buffer)) > -1) caw.write(buffer, 0, len); reader.close(); bis.close(); conn.disconnect(); return caw.toString(); } /** * 獲取網頁數據 * @param urlStr 訪問地址 * @param params 參數 * @return * @throws Exception */ public static String httpGet(String urlStr, Map<String, String> params) throws Exception { StringBuilder sb = new StringBuilder(); if (null != params && params.size() > 0) { sb.append("?"); Entry<String, String> en; for (Iterator<Entry<String, String>> ir = params.entrySet().iterator(); ir.hasNext();) { en = ir.next(); sb.append(en.getKey() + "=" + URLEncoder.encode(en.getValue(),"utf-8") + (ir.hasNext() ? "&" : "")); } } URL url = new URL(urlStr + sb); HttpURLConnection conn = (HttpURLConnection) url.openConnection(); conn.setConnectTimeout(5000); conn.setReadTimeout(5000); conn.setRequestMethod("GET"); if (conn.getResponseCode() != 200) throw new Exception("請求異常狀態值:" + conn.getResponseCode()); BufferedInputStream bis = new BufferedInputStream(conn.getInputStream()); Reader reader = new InputStreamReader(bis,"gbk"); char[] buffer = new char[2048]; int len = 0; CharArrayWriter caw = new CharArrayWriter(); while ((len = reader.read(buffer)) > -1) caw.write(buffer, 0, len); reader.close(); bis.close(); conn.disconnect(); //System.out.println(caw); return caw.toString(); } /** * 從得到的網頁的document中獲取指定條件的內容 * @param document * @param condition 條件 * @return */ public static String catchInfomationFromDocument(Document document , String condition , int position){ if(document != null){ Iterator<Element> iterator = document.select(condition).iterator(); if(iterator.hasNext()){ String str = iterator.next().text(); return str.substring(position).trim(); } } return null; } /** * 判斷從得到的網頁的document中<br/> * 獲取指定條件的內容是否存在 * @param document * @param condition 條件 * @return */ public static boolean isExistInfomation(Document document , String condition){ if(document != null){ Iterator<Element> iterator = document.select(condition).iterator(); if(iterator.hasNext()){ return true; } } return false; } }