Web Spider -- 作一個簡單的爬蟲 （願給您啓示）

Web Spider

一個簡單的爬蟲

代碼已託管

這裏有一個簡單的例子

public class SpiderZhiHuDemo {


    private Logger logger = LoggerFactory.getLogger(SpiderZhiHuDemo.class);


    /**
     * 根據提供的種子爬取數據
     */
    public void initSeedDemo() {
        String seed = "https://www.zhihu.com/question/28629300";
        new Spider() {
            {
                downloader = new ProxyDownloader(new JSoupDownloader());
                extractPattern = "https://www.zhihu.com/question/.*"; // 指定對應的抓取規則
            }
        }.init(seed).start();
    }

    /**
     * 本身定義 抓取的url連接 (簡單的)
     */
    public void easyStreamDemo(){
        String url = "https://www.zhihu.com/question/";
        new Spider() {
            {
                downloader = new ProxyDownloader(new JSoupDownloader());
                pipeline = (title, page) -> {
                    logger.info("url: {} , title : {} ", page.getRequest(), title);
                };
            }
        }.init(Stream.iterate(28629300, i -> i + 1).limit(60).map(i -> url + i).parallel()).start();
    }
}

控制檯打印結果

18:35:54.278 [ForkJoinPool.commonPool-worker-3] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629326 , title : 建築工地上的青年如何自我成長？ - 知乎 
18:35:54.525 [ForkJoinPool.commonPool-worker-1] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629319 , title : 國內專作進口行業的公司多不？（不包括貨代公司） - 知乎 
18:35:54.553 [main] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629338 , title : 如何有效地進行後天性豐胸？ - 知乎 
18:35:54.881 [ForkJoinPool.commonPool-worker-3] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629328 , title : Android開發中list和適配器是怎樣關聯的？適配器具體做用是什麼？list能夠作什麼？ - 知乎 
18:35:55.168 [ForkJoinPool.commonPool-worker-1] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629321 , title : 互聯網公司，粉絲運營如何作？ - 知乎 
18:35:55.317 [ForkJoinPool.commonPool-worker-2] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629311 , title : 爲何一些大公司的特喜歡裝X？ - 知乎 
18:35:55.506 [ForkJoinPool.commonPool-worker-1] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629315 , title : 大二中 剛開學 一閒下來卻總想給家裏打電話 沒法控制 這是爲何？ - 知乎 
18:35:55.664 [main] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629342 , title : 中國雷達哪家強？ - 知乎 
18:35:56.113 [ForkJoinPool.commonPool-worker-3] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629325 , title : 美麗加芬蝸牛系列怎麼樣? - 知乎 
18:35:56.472 [ForkJoinPool.commonPool-worker-3] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629300 , title : "截石位"的說法是從哪裏來的或者怎麼翻譯來的？ - 知乎 
18:35:56.713 [ForkJoinPool.commonPool-worker-1] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629304 , title : 我想考南京大學的歷史系中國史的其中一組 就是01～09方向 請問 是按組命題的麼? - 知乎 
18:35:56.803 [ForkJoinPool.commonPool-worker-2] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629353 , title : K線的精髓在哪裏？ - 知乎 
18:35:57.022 [ForkJoinPool.commonPool-worker-3] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629301 , title : 華爲加班到底有多恐怖？ - 知乎 
18:35:57.143 [main] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629335 , title : 人體的哪些疾病（症狀）能夠自行診斷？ - 知乎 
18:35:57.167 [ForkJoinPool.commonPool-worker-2] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629354 , title : 既然ISIS這麼罪大惡極，那麼聯合國爲何還不派兵剿滅? - 知乎 
18:35:57.333 [ForkJoinPool.commonPool-worker-3] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629302 , title : 誰能幫我起個專門針對聲樂培訓，演出，小明星，小歌手教學工做室名字？能讓人很快記住大氣的名字？謝謝！？ - 知乎 
18:35:58.014 [ForkJoinPool.commonPool-worker-2] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629357 , title : 大學畢業後留有一大堆書不捨得丟掉，每到搬家時就會發愁了，請問如何處理不會以爲惋惜並且便捷？ - 知乎 
18:35:58.022 [ForkJoinPool.commonPool-worker-3] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629349 , title : 華爲研發員工的發展，待遇，福利？和互聯網公司比呢？ - 知乎 
18:35:58.209 [ForkJoinPool.commonPool-worker-1] INFO  c.s.s.sample.zhihu.SpiderZhiHuDemo - url: https://www.zhihu.com/question/28629332 , title : 朋友圈瘋傳: 求擴散！一個小孩的生命結束了就是由於她把手機充電接口放到嘴裏，而另外一頭還在插線板上。 求解? - 知乎