網頁快照實現-Java探險之旅

引言

在監控系統、網頁爬蟲、審覈流程等領域，常常須要保存系統在某一時刻的狀態-網頁快照。實現網頁快照主要有phantomjs、selenium,可是網上資料都是零碎的，本文主要採用Selenium,詳細闡述從編碼到部署的整個流程.

1、添加Selenium依賴

<dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-java</artifactId>
    <version>3.11.0</version>
</dependency>

2、實現網頁快照

(1) 建立WebDriver

a.手機設備

public WebDriver createWebDriver(String device) {
    Map<String, String> mobileEmulation = new HashMap<String, String>();
    //設置設備,例如:Google Nexus 7/Apple iPhone 6
    //mobileEmulation.put("deviceName", "Google Nexus 7");
    if(StringUtils.isBlank(device))
        device = "iPhone 6";
    mobileEmulation.put("deviceName", device);   //這裏是要使用的模擬器名稱，就是瀏覽器中模擬器中的頂部型號
    Map<String, Object> chromeOptions = new HashMap<String, Object>();
    chromeOptions.put("mobileEmulation", mobileEmulation);
    DesiredCapabilities capabilities = DesiredCapabilities.chrome();
    capabilities.setCapability(ChromeOptions.CAPABILITY, chromeOptions);
    WebDriver driver = new ChromeDriver(capabilities);
    Dimension dimension = new Dimension(420, 900);
    driver.manage().window().setSize(dimension);
    return driver;
}

b.PC設備

public WebDriver createWebDriver(String device) {
    //暫時採用特定分辨率
    ChromeOptions options = new ChromeOptions();
    options.addArguments("window-size=1200,800");
    WebDriver driver = new ChromeDriver(options);
    driver.manage().window();
    return driver;
}

(2) 繪製圖像

public String execute(String uri, String device) {
        PerfCounter.count("miui_ad_schedule_capture.execute.total", 1L);
        long start = System.currentTimeMillis();
        System.setProperty("webdriver.chrome.driver", "/home/rd/chromedriver");
        //windows: System.setProperty("webdriver.chrome.driver", "src/main/resources/driver/chromedriver.exe");
        WebDriver driver = createWebDriver(device);
        String cdnUrl = "";
        try {
            driver.get(uri);
            cdnUrl = savePage(driver, url, "capture_" + System.currentTimeMillis() + ".png");
        } catch (Exception e) {
            logger.error("capture fail: [{},{}],{} !", uri ,device,e);
        } finally {
            driver.close();
            driver.quit();
        }
        PerfCounter.count("miui_ad_schedule_capture.execute.success", 1L, System.currentTimeMillis() - start);
        return cdnUrl;
    }

    private String savePage(WebDriver driver, String filePath, String fileName) throws IOException, InterruptedException, NoSuchAlgorithmException {
        JavascriptExecutor jsExecutor = (JavascriptExecutor) driver;
        TakesScreenshot takesScreenshot = (TakesScreenshot) driver;
        jsExecutor.executeScript("window.scrollTo(0,0)");
        BufferedImage imageOriginal = new BufferedImage(1, 1, BufferedImage.TYPE_INT_RGB);// 建立全屏截圖
        int lastScroll = -1;
        int currentScroll = 0;
        Thread.sleep(1000);
        while (lastScroll != currentScroll) {
            byte[] bytesScroll = takesScreenshot.getScreenshotAs(OutputType.BYTES);
            BufferedImage imageScroll = ImageIO.read(new ByteArrayInputStream(bytesScroll));
            int screenHeight = imageScroll.getHeight();
            int screenWidth = imageScroll.getWidth();
            System.out.println("scree:" + screenHeight + ":" + screenWidth);

            BufferedImage combined = new BufferedImage(screenWidth, currentScroll + screenHeight, BufferedImage.TYPE_INT_RGB);
            Graphics g = combined.getGraphics();
            g.drawImage(imageOriginal, 0, 0, null);
            g.drawImage(imageScroll, 0, currentScroll, null);
            imageOriginal = combined;

            logger.info("lastScroll:" + lastScroll + "    currentScroll:" + currentScroll + "    screenHeight:" + screenHeight);
            int scrollTo = currentScroll + screenHeight;
            lastScroll = currentScroll;
            jsExecutor.executeScript("window.scrollTo(0," + scrollTo + ")");
            currentScroll = Double.valueOf(jsExecutor.executeScript("return document.documentElement.scrollTop").toString()).intValue();
            System.out.println(screenHeight + ":" + scrollTo);
            if (lastScroll > 5000) {
                break;
            }
            Thread.sleep(1000);
        }

        File path = new File(filePath);
        if (!path.exists() || !path.isDirectory()) {
            path.mkdirs();
        }
        File file = new File(path.getAbsolutePath() + File.separatorChar + fileName);
        ImageIO.write(imageOriginal, "png", file);
        String uploadPath = FileStorageUtils.upload(file.getAbsolutePath(), CHANNEL_ID);
        System.out.println("uploadPath:" + uploadPath);
        if(file.exists()){
            file.delete();
        }
        return uploadPath;
    }

3、部署(ubantu16.04)

(1) 安裝chrome

sudo apt-get install libxss1 libappindicator1 libindicator7 #install dependency
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo dpkg -i google-chrome*.deb  
sudo apt-get install -f

安裝完成後測試：

google-chrome --headless --remote-debugging-port=9222 https://chromium.org --disable-gpu

這裏是使用headless模式進行遠程調試，ubuntu上大多沒有gpu，因此--disable-gpu以避免報錯。以後使用另外一個命令行訪問本地的9222端口：

curl http://localhost:9222

可以看到調試信息應該就是安裝成功了

(2) 下載chromedriver

chromedriver提供了操做chrome的api，是selenium控制chrome的橋樑。
能夠到：https://sites.google.com/a/ch...。
下載並解壓：

wget https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip
unzip chromedriver_linux64.zip

Chrome與WebDriver的版本對應關係：

chromedriver版本	支持的Chrome版本
v2.37	v64-66
v2.36	v63-65
v2.35	v62-64
v2.34	v61-63
v2.33	v60-62
v2.32	v59-61
v2.31	v58-60
v2.30	v58-60
v2.29	v56-58
v2.28	v55-57
v2.27	v54-56
v2.26	v53-55
v2.25	v53-55
v2.24	v52-54
v2.23	v51-53
v2.22	v49-52
v2.21	v46-50
v2.20	v43-48
v2.19	v43-47
v2.18	v43-46
v2.17	v42-43
v2.13	v42-45
v2.15	v40-43
v2.14	v39-42
v2.13	v38-41
v2.12	v36-40
v2.11	v36-40
v2.10	v33-36
v2.9	v31-34
v2.8	v30-33
v2.7	v30-33
v2.6	v29-32
v2.5	v29-32
v2.4	v29-32

(3) 安裝Xfvb

若是不安裝Xfvb,java -jar xxx.jar 將報一下錯誤：

Starting ChromeDriver 2.30.477691 (6ee44a7247c639c0703f291d320bdf05c1531b57) on port 14103
Only local connections are allowed.
Exception in thread "main" org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally
  (Driver info: chromedriver=2.30.477691 (6ee44a7247c639c0703f291d320bdf05c1531b57),platform=Linux 4.9.15-x86_64-linode81 x86_64) (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 60.13 seconds
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: 'localhost', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.9.15-x86_64-linode81', java.version: '1.8.0_131'
Driver info: driver.version: ChromeDriver
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    ...

安裝方式：
a.安裝Xvfb

sudo apt install Xvfb

b.啓動Xvfb服務

Xvfb -ac :7 -screen 0 1280x1024x8 &

c.鏈接服務7

export DISPLAY=:7

關於Xvfb的具體應用請參考:https://www.x.org/archive/X11R7.6/doc/man/man1/Xvfb.1.xhtml

參考連接:
https://blog.csdn.net/codebat...
https://jiayi.space/post/zai-...

博客編寫不易，若是以爲寫得能夠，請幫忙點個贊