玩玩puppeteer，實現一個小‘爬蟲’

Puppeteer

puppeteer官網

1、puppeteer是幹什麼的？

引用puppeteer官網解釋： Most things that you can do manually in the browser can be done using Puppeteer!

• 生成頁面的屏幕截圖和PDF。
• 抓取SPA並生成預渲染內容（即「SSR」）。
• 自動錶單提交，UI測試，鍵盤輸入等。
• 建立最新的自動化測試環境。 使用最新的JavaScript和瀏覽器功能直接在最新版本的Chrome中運行測試。
• 捕獲您網站的[時間線跟蹤]

2、經常使用API

• page.setViewport() 設置獲取屏幕大小，默認獲取屏幕大小爲800px * 600px

• page.pdf(路徑，大小) 保存爲pdf格式圖片

  • 舉例：page.pdf({path: 'hn.pdf', format: 'A4'});

• page.evaluate(fn) 執行chrome的api

  • 舉例：

    await page.evaluate(() => {
         return {
             width: document.documentElement.clientWidth,
             height: document.documentElement.clientHeight,
        deviceScaleFactor: window.deivcePixelRatio
        };        
    })

• puppeteer.launch({headless: false}); 打開瀏覽器，默認值是true

更多API

3、舉個栗子：截取屏幕

3.1 代碼

const puppeteer = require('puppeteer');
// 引用default.js的sceenshot路徑，將截取的屏幕pdf保存到該路徑下。
const { screenshot } = require('./config/default.js');

(async () => {
  // 獲取browser實例
  const browser = await puppeteer.launch();
  // 獲取瀏覽器tab頁面實例
  const page = await browser.newPage();
  // 連接到百度首頁
  await page.goto('https://www.baidu.com');
  // 截屏
  await page.screenshot({
    // 將截屏按時間戳保存到指定路徑下。
    path: `${screenshot}/${Date.now()}.png`
  });
  // 關閉
  await browser.close();
})();

3.2 而後執行命令

node src/screenshot.js

3.3 最後在screenshot文件指定路徑下生成百度首頁的截屏。

4、爬取百度圖片列表

4.1 實現思路

1. 模擬用戶打開瀏覽器
2. 模擬打開tab頁
3. 模擬前往百度圖片頁面
4. 模擬focus到輸入框，輸入查詢值, 點擊查詢按鈕
5. 抓取圖片
6. 經過writeFile，將圖片下載到指定路徑下。

4.2 目錄結構

.
|-mn
|-src
|   |-config
|   |    |-default.js     
|   |-helper
|   |    |-srcToImg.js
|   |-mn.js
|-package.json

4.3 mn.js 主文件

const puppeteer = require('puppeteer');
const { mn } = require('./config/default');
const srcToImg = require('./helper/srcToImg');
(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://image.baidu.com');
  console.log('go to https://image.baidu.com');
  await page.setViewport({
    width: 1920,
    height: 1080
  });
  console.log('reset viewport');
  await page.focus('#kw');
  await page.keyboard.sendCharacter('狗');
  await page.click('.s_search');
  console.log('go to search list');
  page.on('load', async () => {
    console.log('page loading done, start fetch ...');
    const srcs = await page.evaluate(() => {
      const images = document.querySelectorAll('img.main_img');
      return Array.prototype.map.call(images, img => img.src);
    });
    console.log(`get ${srcs.length} image, start download`);
    
    srcs.forEach(async (src) => {
      await srcToImg(src, mn);
    });
    await browser.close();
  })
})();

4.4 default.js 路徑

const path = require('path');
module.exports = {
  screenshot: path.resolve(__dirname, '../../screenshot'),
  mn: path.resolve(__dirname, '../../mn')
}

4.5 srcToImg.js 解析圖片地址

const http = require('http');
const https = require('https');
const fs = require('fs');
const path = require('path');
const { promisify } = require('util');
const writeFile = promisify(fs.writeFile);

module.exports = async(src, dir) => {
  if(/\.(jpg|png|gif)$/.test(src)) {
    await urlToImg(src, dir);
  }else {
    await base64ToImg(src, dir); 
  }
}

// 識別src爲http或者https的圖片
const urlToImg = promisify((url, dir, callback) => {
  const mod = /^https:/.test(url) ? https : http;
  const ext = path.extname(url);
  const file = path.join(dir, `${Date.now()}${ext}`);
  mod.get(url, res => {
    res.pipe(fs.createWriteStream(file))
      .on('finish', () => {
        callback();
        console.log(file);
      })
  })
})

// 識別src爲base64地址的圖片
const base64ToImg = async (base64Str, dir) => {
  // data: image/jpeg;base64,/raegreagearg
  const matchs = base64Str.match(/^data:(.+?);base64,(.+)$/);
  try {
    const ext = matches[1].split('/')[1]
      .replace('jpeg', 'jpg');
    const file = path.join(dir, `${Date.now()}.${ext}`);
    await writeFile(file, match[2], 'base64');
    console.log(file);
  } catch (ex) {
    console.log('非法 base64 字符串');
  }
}

4.6 最終在mn文件夾中存入爬取到的圖片。

go to https://image.baidu.com
reset viewport
go to search list
page loading done, start fetch ...
get 46 image, start download
非法 base64 字符串
非法 base64 字符串
非法 base64 字符串
非法 base64 字符串
非法 base64 字符串
非法 base64 字符串
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351397.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351396.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351398.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351400.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351405.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351386.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351399.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351405.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351405.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351402.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351412.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351413.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351403.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351398.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351399.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351403.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351406.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351401.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351408.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351404.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351414.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351400.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351402.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351413.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351408.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351414.jpg
/Users/lius/Desktop/web spider/headless-crawler/headless_crawler/mn/1530800351413.jpg
......

4.7 mn文件夾下