puppeteer爬蟲-爬取豆瓣電影

時間 2020-03-04

標籤 puppeteer 爬蟲豆瓣欄目網絡爬蟲简体版

原文原文鏈接

具體使用請參看 Puppeteer官網

puppeteer爬蟲和通常爬蟲的區別

爬蟲是一個模擬人類請求網站行爲的程序。不少爬蟲程序是經過請求http獲取數據，用puppeteer作爬蟲是經過DevTool協議控制Chrome或Chromium來抓取頁面信息node

puppeteer下載

兩種模式git

1.完整模式（自動下載最新版的Chrome）github

npm i puppeteer
...
const puppeteer = require('puppeteer');

2.簡潔模式（手動指定Chrome路徑）chrome

npm i puppeteer-core
...
const puppeteer = require('puppeteer-core');
puppeteer.launch({
    executablePath:Chrome.exe // 指定Chrome或Chromium路徑
})

爬取豆瓣電影

直接上代碼npm

const puppeteer = require('puppeteer-core');

const launchConfig = {
    headless: false,
    executablePath: 'C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe',
    slowMo: 1000,
    defaultViewport: {
        width: 1600,
        height: 900,
    }
}

puppeteer.launch(launchConfig).then(async browser => {
    const page = await browser.newPage();
    await page.goto('https://movie.douban.com/cinema/nowplaying/beijing/');
    const result = await page.evaluate(() => {
        const items = document.querySelectorAll('#nowplaying > div.mod-bd > ul >li');
        const links = [];
        if (items.length >= 1) {
            items.forEach((item) => {
                const data = Array.from(item.attributes);
                const link = {};
                data.forEach((v) => {
                    link[v.nodeName] = v.value;
                });
                const a = item.querySelector('.poster > a');
                const img = a.querySelector('img');
                link.href = a.getAttribute('href');
                link.src = img.getAttribute('src');
                links.push({
                    ...link,
                });
            });
        }
        return links;
    });
    console.log('result', result)
    await browser.close();
});