Scrapy中添加隨機User-Agent

方法：使該模塊scrapy-fake-useragent

1，pip install scrapy-fake-useragent

2，在setting.py 寫：

DOWNLOADER_MIDDLEWARES = {
    'lagoujob.middlewares.RandomUesrAgent': 1,
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
}

3，在 middlewares.py 寫入：

class RandomUesrAgent(object):
    def process_request(self, request, spider):
        ua = UserAgent()
        request.headers.setdefault("User-Agent", ua.random)

　或者：

而後就能夠用啦