1、http協議原理html
http是無狀態的應用層協議python
2、urllib、urllib二、requestsgit
在python2中,有urllib和urlib2兩個包,python3全變成urllib了github
urllib和urllib2是相互獨立的模塊,好比,urllib不能僞造user agent,可是請求參數的構建,須要使用到urllib,全部常常是二者配合使用的;json
requests使用了urlib3(屢次請求重複使用一個socket)api
3、環境準備瀏覽器
http://httpbin.org/ 安全
若是將該站點搭建在本地,使用pip安裝就能夠了服務器
pip install gunicorn httpbincookie
啓動:
gunicorn httpbin:app --bind 0.0.0.0:8000
發送一個http的get請求,獲取本機的IP地址
import urllib2 import urllib URL_IP='http://192.168.74.20:8000/ip' def use_simple_urllib2(): response=urllib2.urlopen(URL_IP) print ">>>>Response Headers:" print response.info() print ''.join([line for line in response.readlines()]) if __name__ == '__main__': print ">>>Use Simple urllib2" use_simple_urllib2() 結果爲: >>>Use Simple urllib2 >>>>Response Headers: Server: gunicorn/19.7.1 Date: Sat, 08 Apr 2017 13:06:18 GMT Connection: close Content-Type: application/json Access-Control-Allow-Origin: * Access-Control-Allow-Credentials: true Content-Length: 31 { "origin": "192.168.74.1" }
使用urllib構建請求參數,使用urllib2發送請求
def use_params_urllib2(): """ 構建請求參數 :return: """ params = urllib.urlencode({'param1':'hello','param2':'world'}) print 'Request Params:' print params #發送請求 response=urllib2.urlopen('?'.join([URL_GET,'%s'])%params) #處理響應 print ">>>>Response Headers:" print response.info() print ">>>Status code" print response.getcode() print ''.join([line for line in response.readlines()]) if __name__ == '__main__': print ">>>Use Simple urllib2" use_simple_urllib2() print ">>>Use params urllib2" use_params_urllib2() 結果: >>>Use params urllib2 Request Params: param2=world¶m1=hello >>>>Response Headers: Server: gunicorn/19.7.1 Date: Sat, 08 Apr 2017 13:22:05 GMT Connection: close Content-Type: application/json Access-Control-Allow-Origin: * Access-Control-Allow-Credentials: true Content-Length: 322 >>>Status code 200 { "args": { "param1": "hello", "param2": "world" }, "headers": { "Accept-Encoding": "identity", "Connection": "close", "Host": "192.168.74.20:8000", "User-Agent": "Python-urllib/2.7" }, "origin": "192.168.74.1", "url": "http://192.168.74.20:8000/get?param2=world¶m1=hello" }
使用requests模塊實現上述功能
#!/usr/bin/env python # _*_ coding:utf-8 _*_ __author__ = 'Charles Chang' import requests import urllib2 import urllib URL_IP='http://192.168.74.20:8000/ip' URL_GET='http://192.168.74.20:8000/get' def use_simple_requests(): response=requests.get(URL_IP) print ">>>>Response Headers:" print response.headers print '>>>>Response Body:' print response.text def use_params_resuqets(): """ 構建請求參數 :return: """ params = {'param1':'hello','param2':'world'} print 'Request Params:' print params #發送請求 response=requests.get(URL_IP,params=params) #處理響應 print ">>>>Response Headers:" print response.headers print ">>>Status code" print response.status_code print response.reason print ">>>>Response Body:" print response.json() if __name__ == '__main__': print ">>>Use Simple urllib2" use_simple_requests() print print ">>>Use params requests" use_params_resuqets() 結果: >>>Use Simple urllib2 >>>>Response Headers: {'Content-Length': '31', 'Server': 'gunicorn/19.7.1', 'Connection': 'close', 'Access-Control-Allow-Credentials': 'true', 'Date': 'Sat, 08 Apr 2017 14:29:38 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json'} >>>>Response Body: { "origin": "192.168.74.1" } >>>Use params requests Request Params: {'param2': 'world', 'param1': 'hello'} >>>>Response Headers: {'Content-Length': '31', 'Server': 'gunicorn/19.7.1', 'Connection': 'close', 'Access-Control-Allow-Credentials': 'true', 'Date': 'Sat, 08 Apr 2017 14:29:38 GMT', 'Access-Control-Allow-Origin': '*', 'Content-Type': 'application/json'} >>>Status code 200 OK >>>>Response Body: {u'origin': u'192.168.74.1'}
4、github api,requests 請求方法
https://developer.github.com/v3/
GET:查看資源
POST:增長資源
PUT:修改資源
DELETE:刪除資源
HEAD:查看響應頭
OPTIONS:查看可用請求方法
PATCH:使用少許的json格式的數據去更新數據的方式;
Verb | Description |
---|---|
HEAD |
Can be issued against any resource to get just the HTTP header info. |
GET |
Used for retrieving resources. |
POST |
Used for creating resources. |
PATCH |
Used for updating resources with partial JSON data. For instance, an Issue resource hastitle and body attributes. A PATCH request may accept one or more of the attributes to update the resource. PATCH is a relatively new and uncommon HTTP verb, so resource endpoints also accept POST requests. |
PUT |
Used for replacing resources or collections. For PUT requests with no body attribute, be sure to set the Content-Length header to zero. |
DELETE |
Used for deleting resources. |
requests請求方法調用github api
import json import requests URL='https://api.github.com' def build_uri(endpoint): return '/'.join([URL,endpoint]) def better_print(json_str): return json.dumps(json.loads(json_str),indent=4) def request_method(): response=requests.get(build_uri('users/myuser'),auth=('myuser','mypassword')) print response.text print better_print(response.text) if __name__ == '__main__': request_method() 結果: {"login":"CharlesQQ","id":18431718,"avatar_url":"https://avatars2.githubusercontent.com/u/18431718?v=3","gravatar_id":"","url":"https://api.github.com/users/CharlesQQ","html_url":"https://github.com/CharlesQQ","followers_url":"https://api.github.com/users/CharlesQQ/followers","following_url":"https://api.github.com/users/CharlesQQ/following{/other_user}","gists_url":"https://api.github.com/users/CharlesQQ/gists{/gist_id}","starred_url":"https://api.github.com/users/CharlesQQ/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CharlesQQ/subscriptions","organizations_url":"https://api.github.com/users/CharlesQQ/orgs","repos_url":"https://api.github.com/users/CharlesQQ/repos","events_url":"https://api.github.com/users/CharlesQQ/events{/privacy}","received_events_url":"https://api.github.com/users/CharlesQQ/received_events","type":"User","site_admin":false,"name":null,"company":null,"blog":null,"location":null,"email":null,"hireable":null,"bio":null,"public_repos":7,"public_gists":0,"followers":0,"following":8,"created_at":"2016-04-13T00:58:55Z","updated_at":"2017-04-08T14:36:42Z","private_gists":0,"total_private_repos":0,"owned_private_repos":0,"disk_usage":14729,"collaborators":0,"two_factor_authentication":false,"plan":{"name":"free","space":976562499,"collaborators":0,"private_repos":0}} { "disk_usage": 14729, "private_gists": 0, "public_repos": 7, "site_admin": false, "subscriptions_url": "https://api.github.com/users/CharlesQQ/subscriptions", "gravatar_id": "", "hireable": null, "id": 18431718, "followers_url": "https://api.github.com/users/CharlesQQ/followers", "following_url": "https://api.github.com/users/CharlesQQ/following{/other_user}", "collaborators": 0, "total_private_repos": 0, "blog": null, "followers": 0, "location": null, "type": "User", "email": null, "bio": null, "gists_url": "https://api.github.com/users/CharlesQQ/gists{/gist_id}", "owned_private_repos": 0, "company": null, "events_url": "https://api.github.com/users/CharlesQQ/events{/privacy}", "html_url": "https://github.com/CharlesQQ", "updated_at": "2017-04-08T14:36:42Z", "plan": { "collaborators": 0, "name": "free", "private_repos": 0, "space": 976562499 }, "received_events_url": "https://api.github.com/users/CharlesQQ/received_events", "starred_url": "https://api.github.com/users/CharlesQQ/starred{/owner}{/repo}", "public_gists": 0, "name": null, "organizations_url": "https://api.github.com/users/CharlesQQ/orgs", "url": "https://api.github.com/users/CharlesQQ", "created_at": "2016-04-13T00:58:55Z", "avatar_url": "https://avatars2.githubusercontent.com/u/18431718?v=3", "repos_url": "https://api.github.com/users/CharlesQQ/repos", "following": 8, "login": "CharlesQQ", "two_factor_authentication": false }
5、帶參數的請求
方式1:
search_product.html?cat=19283377&...
params: requests.get(url,params={'key1':''value1})
方式2:表單數據提交
Content-Type: application/x-www-form-urlencoded
requests.post(url,data={'key1':'value1','keys':'value2'})
方式3:json參數提交
Content-type:application/json
requests.post(url,json={'key1':'value1','key2':'value2'})
使用params獲取user信息
def params_requests(): response=requests.get(build_uri('users'),params={'since':11}) print better_print(response.text) print response.request.headers print response.url if __name__ == '__main__': params_requests() 結果: { "following_url": "https://api.github.com/users/tomtt/following{/other_user}", "events_url": "https://api.github.com/users/tomtt/events{/privacy}", "organizations_url": "https://api.github.com/users/tomtt/orgs", "url": "https://api.github.com/users/tomtt", "gists_url": "https://api.github.com/users/tomtt/gists{/gist_id}", "html_url": "https://github.com/tomtt", "subscriptions_url": "https://api.github.com/users/tomtt/subscriptions", "avatar_url": "https://avatars1.githubusercontent.com/u/31?v=3", "repos_url": "https://api.github.com/users/tomtt/repos", "received_events_url": "https://api.github.com/users/tomtt/received_events", "gravatar_id": "", "starred_url": "https://api.github.com/users/tomtt/starred{/owner}{/repo}", "site_admin": false, "login": "tomtt", "type": "User", "id": 31, "followers_url": "https://api.github.com/users/tomtt/followers" }, { "following_url": "https://api.github.com/users/railsjitsu/following{/other_user}", "events_url": "https://api.github.com/users/railsjitsu/events{/privacy}", "organizations_url": "https://api.github.com/users/railsjitsu/orgs", "url": "https://api.github.com/users/railsjitsu", "gists_url": "https://api.github.com/users/railsjitsu/gists{/gist_id}", "html_url": "https://github.com/railsjitsu", "subscriptions_url": "https://api.github.com/users/railsjitsu/subscriptions", "avatar_url": "https://avatars1.githubusercontent.com/u/32?v=3", "repos_url": "https://api.github.com/users/railsjitsu/repos", "received_events_url": "https://api.github.com/users/railsjitsu/received_events", "gravatar_id": "", "starred_url": "https://api.github.com/users/railsjitsu/starred{/owner}{/repo}", "site_admin": false, "login": "railsjitsu", "type": "User", "id": 32, "followers_url": "https://api.github.com/users/railsjitsu/followers" }, { "following_url": "https://api.github.com/users/nitay/following{/other_user}", "events_url": "https://api.github.com/users/nitay/events{/privacy}", "organizations_url": "https://api.github.com/users/nitay/orgs", "url": "https://api.github.com/users/nitay", "gists_url": "https://api.github.com/users/nitay/gists{/gist_id}", "html_url": "https://github.com/nitay", "subscriptions_url": "https://api.github.com/users/nitay/subscriptions", "avatar_url": "https://avatars1.githubusercontent.com/u/34?v=3", "repos_url": "https://api.github.com/users/nitay/repos", "received_events_url": "https://api.github.com/users/nitay/received_events", "gravatar_id": "", "starred_url": "https://api.github.com/users/nitay/starred{/owner}{/repo}", "site_admin": false, "login": "nitay", "type": "User", "id": 34, "followers_url": "https://api.github.com/users/nitay/followers" }, { "following_url": "https://api.github.com/users/kevwil/following{/other_user}", "events_url": "https://api.github.com/users/kevwil/events{/privacy}", "organizations_url": "https://api.github.com/users/kevwil/orgs", "url": "https://api.github.com/users/kevwil", "gists_url": "https://api.github.com/users/kevwil/gists{/gist_id}", "html_url": "https://github.com/kevwil", "subscriptions_url": "https://api.github.com/users/kevwil/subscriptions", "avatar_url": "https://avatars1.githubusercontent.com/u/35?v=3", "repos_url": "https://api.github.com/users/kevwil/repos", "received_events_url": "https://api.github.com/users/kevwil/received_events", "gravatar_id": "", "starred_url": "https://api.github.com/users/kevwil/starred{/owner}{/repo}", "site_admin": false, "login": "kevwil", "type": "User", "id": 35, "followers_url": "https://api.github.com/users/kevwil/followers" }, { "following_url": "https://api.github.com/users/KirinDave/following{/other_user}", "events_url": "https://api.github.com/users/KirinDave/events{/privacy}", "organizations_url": "https://api.github.com/users/KirinDave/orgs", "url": "https://api.github.com/users/KirinDave", "gists_url": "https://api.github.com/users/KirinDave/gists{/gist_id}", "html_url": "https://github.com/KirinDave", "subscriptions_url": "https://api.github.com/users/KirinDave/subscriptions", "avatar_url": "https://avatars1.githubusercontent.com/u/36?v=3", "repos_url": "https://api.github.com/users/KirinDave/repos", "received_events_url": "https://api.github.com/users/KirinDave/received_events", "gravatar_id": "", "starred_url": "https://api.github.com/users/KirinDave/starred{/owner}{/repo}", "site_admin": false, "login": "KirinDave", "type": "User", "id": 36, "followers_url": "https://api.github.com/users/KirinDave/followers" }, { "following_url": "https://api.github.com/users/jamesgolick/following{/other_user}", "events_url": "https://api.github.com/users/jamesgolick/events{/privacy}", "organizations_url": "https://api.github.com/users/jamesgolick/orgs", "url": "https://api.github.com/users/jamesgolick", "gists_url": "https://api.github.com/users/jamesgolick/gists{/gist_id}", "html_url": "https://github.com/jamesgolick", "subscriptions_url": "https://api.github.com/users/jamesgolick/subscriptions", "avatar_url": "https://avatars1.githubusercontent.com/u/37?v=3", "repos_url": "https://api.github.com/users/jamesgolick/repos", "received_events_url": "https://api.github.com/users/jamesgolick/received_events", "gravatar_id": "", "starred_url": "https://api.github.com/users/jamesgolick/starred{/owner}{/repo}", "site_admin": false, "login": "jamesgolick", "type": "User", "id": 37, "followers_url": "https://api.github.com/users/jamesgolick/followers" }, { "following_url": "https://api.github.com/users/atmos/following{/other_user}", "events_url": "https://api.github.com/users/atmos/events{/privacy}", "organizations_url": "https://api.github.com/users/atmos/orgs", "url": "https://api.github.com/users/atmos", "gists_url": "https://api.github.com/users/atmos/gists{/gist_id}", "html_url": "https://github.com/atmos", "subscriptions_url": "https://api.github.com/users/atmos/subscriptions", "avatar_url": "https://avatars0.githubusercontent.com/u/38?v=3", "repos_url": "https://api.github.com/users/atmos/repos", "received_events_url": "https://api.github.com/users/atmos/received_events", "gravatar_id": "", "starred_url": "https://api.github.com/users/atmos/starred{/owner}{/repo}", "site_admin": false, "login": "atmos", "type": "User", "id": 38, "followers_url": "https://api.github.com/users/atmos/followers" }, { "following_url": "https://api.github.com/users/errfree/following{/other_user}", "events_url": "https://api.github.com/users/errfree/events{/privacy}", "organizations_url": "https://api.github.com/users/errfree/orgs", "url": "https://api.github.com/users/errfree", "gists_url": "https://api.github.com/users/errfree/gists{/gist_id}", "html_url": "https://github.com/errfree", "subscriptions_url": "https://api.github.com/users/errfree/subscriptions", "avatar_url": "https://avatars1.githubusercontent.com/u/44?v=3", "repos_url": "https://api.github.com/users/errfree/repos", "received_events_url": "https://api.github.com/users/errfree/received_events", "gravatar_id": "", "starred_url": "https://api.github.com/users/errfree/starred{/owner}{/repo}", "site_admin": false, "login": "errfree", "type": "Organization", "id": 44, "followers_url": "https://api.github.com/users/errfree/followers" }, { "following_url": "https://api.github.com/users/mojodna/following{/other_user}", "events_url": "https://api.github.com/users/mojodna/events{/privacy}", "organizations_url": "https://api.github.com/users/mojodna/orgs", "url": "https://api.github.com/users/mojodna", "gists_url": "https://api.github.com/users/mojodna/gists{/gist_id}", "html_url": "https://github.com/mojodna", "subscriptions_url": "https://api.github.com/users/mojodna/subscriptions", "avatar_url": "https://avatars1.githubusercontent.com/u/45?v=3", "repos_url": "https://api.github.com/users/mojodna/repos", "received_events_url": "https://api.github.com/users/mojodna/received_events", "gravatar_id": "", "starred_url": "https://api.github.com/users/mojodna/starred{/owner}{/repo}", "site_admin": false, "login": "mojodna", "type": "User", "id": 45, "followers_url": "https://api.github.com/users/mojodna/followers" }, { "following_url": "https://api.github.com/users/bmizerany/following{/other_user}", "events_url": "https://api.github.com/users/bmizerany/events{/privacy}", "organizations_url": "https://api.github.com/users/bmizerany/orgs", "url": "https://api.github.com/users/bmizerany", "gists_url": "https://api.github.com/users/bmizerany/gists{/gist_id}", "html_url": "https://github.com/bmizerany", "subscriptions_url": "https://api.github.com/users/bmizerany/subscriptions", "avatar_url": "https://avatars1.githubusercontent.com/u/46?v=3", "repos_url": "https://api.github.com/users/bmizerany/repos", "received_events_url": "https://api.github.com/users/bmizerany/received_events", "gravatar_id": "", "starred_url": "https://api.github.com/users/bmizerany/starred{/owner}{/repo}", "site_admin": false, "login": "bmizerany", "type": "User", "id": 46, "followers_url": "https://api.github.com/users/bmizerany/followers" }, { "following_url": "https://api.github.com/users/jnewland/following{/other_user}", "events_url": "https://api.github.com/users/jnewland/events{/privacy}", "organizations_url": "https://api.github.com/users/jnewland/orgs", "url": "https://api.github.com/users/jnewland", "gists_url": "https://api.github.com/users/jnewland/gists{/gist_id}", "html_url": "https://github.com/jnewland", "subscriptions_url": "https://api.github.com/users/jnewland/subscriptions", "avatar_url": "https://avatars1.githubusercontent.com/u/47?v=3", "repos_url": "https://api.github.com/users/jnewland/repos", "received_events_url": "https://api.github.com/users/jnewland/received_events", "gravatar_id": "", "starred_url": "https://api.github.com/users/jnewland/starred{/owner}{/repo}", "site_admin": true, "login": "jnewland", "type": "User", "id": 47, "followers_url": "https://api.github.com/users/jnewland/followers" }, { "following_url": "https://api.github.com/users/joshknowles/following{/other_user}", "events_url": "https://api.github.com/users/joshknowles/events{/privacy}", "organizations_url": "https://api.github.com/users/joshknowles/orgs", "url": "https://api.github.com/users/joshknowles", "gists_url": "https://api.github.com/users/joshknowles/gists{/gist_id}", "html_url": "https://github.com/joshknowles", "subscriptions_url": "https://api.github.com/users/joshknowles/subscriptions", "avatar_url": "https://avatars0.githubusercontent.com/u/48?v=3", "repos_url": "https://api.github.com/users/joshknowles/repos", "received_events_url": "https://api.github.com/users/joshknowles/received_events", "gravatar_id": "", "starred_url": "https://api.github.com/users/joshknowles/starred{/owner}{/repo}", "site_admin": false, "login": "joshknowles", "type": "User", "id": 48, "followers_url": "https://api.github.com/users/joshknowles/followers" }, { "following_url": "https://api.github.com/users/hornbeck/following{/other_user}", "events_url": "https://api.github.com/users/hornbeck/events{/privacy}", "organizations_url": "https://api.github.com/users/hornbeck/orgs", "url": "https://api.github.com/users/hornbeck", "gists_url": "https://api.github.com/users/hornbeck/gists{/gist_id}", "html_url": "https://github.com/hornbeck", "subscriptions_url": "https://api.github.com/users/hornbeck/subscriptions", "avatar_url": "https://avatars0.githubusercontent.com/u/49?v=3", "repos_url": "https://api.github.com/users/hornbeck/repos", "received_events_url": "https://api.github.com/users/hornbeck/received_events", "gravatar_id": "", "starred_url": "https://api.github.com/users/hornbeck/starred{/owner}{/repo}", "site_admin": false, "login": "hornbeck", "type": "User", "id": 49, "followers_url": "https://api.github.com/users/hornbeck/followers" }, { "following_url": "https://api.github.com/users/jwhitmire/following{/other_user}", "events_url": "https://api.github.com/users/jwhitmire/events{/privacy}", "organizations_url": "https://api.github.com/users/jwhitmire/orgs", "url": "https://api.github.com/users/jwhitmire", "gists_url": "https://api.github.com/users/jwhitmire/gists{/gist_id}", "html_url": "https://github.com/jwhitmire", "subscriptions_url": "https://api.github.com/users/jwhitmire/subscriptions", "avatar_url": "https://avatars0.githubusercontent.com/u/50?v=3", "repos_url": "https://api.github.com/users/jwhitmire/repos", "received_events_url": "https://api.github.com/users/jwhitmire/received_events", "gravatar_id": "", "starred_url": "https://api.github.com/users/jwhitmire/starred{/owner}{/repo}", "site_admin": false, "login": "jwhitmire", "type": "User", "id": 50, "followers_url": "https://api.github.com/users/jwhitmire/followers" }, { "following_url": "https://api.github.com/users/elbowdonkey/following{/other_user}", "events_url": "https://api.github.com/users/elbowdonkey/events{/privacy}", "organizations_url": "https://api.github.com/users/elbowdonkey/orgs", "url": "https://api.github.com/users/elbowdonkey", "gists_url": "https://api.github.com/users/elbowdonkey/gists{/gist_id}", "html_url": "https://github.com/elbowdonkey", "subscriptions_url": "https://api.github.com/users/elbowdonkey/subscriptions", "avatar_url": "https://avatars0.githubusercontent.com/u/51?v=3", "repos_url": "https://api.github.com/users/elbowdonkey/repos", "received_events_url": "https://api.github.com/users/elbowdonkey/received_events", "gravatar_id": "", "starred_url": "https://api.github.com/users/elbowdonkey/starred{/owner}{/repo}", "site_admin": false, "login": "elbowdonkey", "type": "User", "id": 51, "followers_url": "https://api.github.com/users/elbowdonkey/followers" }, { "following_url": "https://api.github.com/users/reinh/following{/other_user}", "events_url": "https://api.github.com/users/reinh/events{/privacy}", "organizations_url": "https://api.github.com/users/reinh/orgs", "url": "https://api.github.com/users/reinh", "gists_url": "https://api.github.com/users/reinh/gists{/gist_id}", "html_url": "https://github.com/reinh", "subscriptions_url": "https://api.github.com/users/reinh/subscriptions", "avatar_url": "https://avatars0.githubusercontent.com/u/52?v=3", "repos_url": "https://api.github.com/users/reinh/repos", "received_events_url": "https://api.github.com/users/reinh/received_events", "gravatar_id": "", "starred_url": "https://api.github.com/users/reinh/starred{/owner}{/repo}", "site_admin": false, "login": "reinh", "type": "User", "id": 52, "followers_url": "https://api.github.com/users/reinh/followers" }, { "following_url": "https://api.github.com/users/knzconnor/following{/other_user}", "events_url": "https://api.github.com/users/knzconnor/events{/privacy}", "organizations_url": "https://api.github.com/users/knzconnor/orgs", "url": "https://api.github.com/users/knzconnor", "gists_url": "https://api.github.com/users/knzconnor/gists{/gist_id}", "html_url": "https://github.com/knzconnor", "subscriptions_url": "https://api.github.com/users/knzconnor/subscriptions", "avatar_url": "https://avatars0.githubusercontent.com/u/53?v=3", "repos_url": "https://api.github.com/users/knzconnor/repos", "received_events_url": "https://api.github.com/users/knzconnor/received_events", "gravatar_id": "", "starred_url": "https://api.github.com/users/knzconnor/starred{/owner}{/repo}", "site_admin": false, "login": "knzconnor", "type": "User", "id": 53, "followers_url": "https://api.github.com/users/knzconnor/followers" } ] {'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.10.0'} https://api.github.com/users?since=11
使用patch修改數據
def json_request(): response=requests.patch(build_uri('user'),auth=('charlesQQ','mypass'), json={'name':'charlesQQ11','email':'charles_ali@qq.com'}) print better_print(response.text) print response.request.headers print response.request.body print response.status_code { "disk_usage": 14729, "private_gists": 0, "public_repos": 7, "site_admin": false, "subscriptions_url": "https://api.github.com/users/CharlesQQ/subscriptions", "gravatar_id": "", "hireable": null, "id": 18431718, "followers_url": "https://api.github.com/users/CharlesQQ/followers", "following_url": "https://api.github.com/users/CharlesQQ/following{/other_user}", "collaborators": 0, "total_private_repos": 0, "blog": null, "followers": 0, "location": null, "type": "User", "email": "charles_ali@qq.com", "bio": null, "gists_url": "https://api.github.com/users/CharlesQQ/gists{/gist_id}", "owned_private_repos": 0, "company": null, "events_url": "https://api.github.com/users/CharlesQQ/events{/privacy}", "html_url": "https://github.com/CharlesQQ", "updated_at": "2017-04-08T14:36:42Z", "plan": { "collaborators": 0, "name": "free", "private_repos": 0, "space": 976562499 }, "received_events_url": "https://api.github.com/users/CharlesQQ/received_events", "starred_url": "https://api.github.com/users/CharlesQQ/starred{/owner}{/repo}", "public_gists": 0, "name": "charlesQQ11", "organizations_url": "https://api.github.com/users/CharlesQQ/orgs", "url": "https://api.github.com/users/CharlesQQ", "created_at": "2016-04-13T00:58:55Z", "avatar_url": "https://avatars2.githubusercontent.com/u/18431718?v=3", "repos_url": "https://api.github.com/users/CharlesQQ/repos", "following": 8, "login": "CharlesQQ", "two_factor_authentication": false } {'Content-Length': '54', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.10.0', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Authorization': 'Basic Y2hhcmxlc1FROnN0YXJ0MzMzMzM='} {"name": "charlesQQ11", "email": "charles_ali@qq.com"} 200
使用post增長數據
def json_request(): #response=requests.patch(build_uri('user'),auth=('charlesQQ','mypass'), # json={'name':'charlesQQ11','email':'charles_ali@qq.com'}) response=requests.post(build_uri('user/emails'),auth=('charlesQQ','start33333'),json=['qq_c1231@163.com']) print better_print(response.text) print response.request.headers print response.request.body print response.status_code
6、請求異常處理
通常在咱們調用第三方服務的時候,難以免的是第三方服務調用出現異常,這種時候,就須要使用異常處理,來保證程序能夠正常運行;
requests的異常處理,須要使用其exceptions類來實現;
將超時時間設置爲0.1秒,來觸發超時異常
import requests from requests import exceptions def timeout_request(): try: response=requests.get(build_uri('user/emails'),timeout=0.1) response.raise_for_status() except exceptions.Timeout as e: print e.message except exceptions.HTTPError as e: print e.message else: print response.text print response.status_code 調用結果: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /user/emails (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x0000000003148CF8>, 'Connection to api.github.com timed out. (connect timeout=0.1)'))
將超時時間設置爲10s,觸發401錯誤
def timeout_request(): try: response=requests.get(build_uri('user/emails'),timeout=10) response.raise_for_status() except exceptions.Timeout as e: print e.message except exceptions.HTTPError as e: print e.message else: print response.text print response.status_code 調用結果: 401 Client Error: Unauthorized for url: https://api.github.com/user/emails
7、自定義request
requests經過urlib3實現能夠hold住整個socket,經過session實現;
發送的request包,包含發送的body,headers,auth,proxy,timeout,verify等,回覆消息response爲text,json等;
有時間去研究源碼吧
def hard_requests(): from requests import Request,Session s=Session() headers={'User-Agent':'fake1.3.4'} req=Request('GET',build_uri('user/emails'),auth=('myuser','mypass'),headers=headers) prepared=req.prepare() print prepared.body print prepared.headers resp=s.send(prepared,timeout=5) print resp.status_code print resp.request.headers print resp.text 結果: None {'Authorization': 'Basic Y2hhcmxlc1FROnN0YXJ0MzMzMzM=', 'User-Agent': 'fake1.3.4'} 200 {'Authorization': 'Basic Y2hhcmxlc1FROnN0YXJ0MzMzMzM=', 'User-Agent': 'fake1.3.4'} [{"email":"qq_c123@163.com","primary":false,"verified":false,"visibility":null},{"email":"charles_ali@qq.com","primary":true,"verified":true,"visibility":"public"},{"email":"qq_c1231@163.com","primary":false,"verified":false,"visibility":null}]
8、響應基本API
Response對象的API:
status_code:以下
400:Bad request,請求服務器端沒法解析; 401:unauthorized,請求沒有認證; 403:forbidden; 404:Not found;
500:內部錯誤;
reason:
headers:
url:
history:
elapsed:
request:
encoding:
raw:讀取原始的對象
content:string類型:
text:unicode;
json:
下面分別舉例介紹
status_code和reason
import requests response = requests.get('https://api.github.com') >>> response.status_code 200 >>> response.reason 'OK'
響應headers
>>> response.headers {'X-XSS-Protection': '1; mode=block', 'Content-Security-Policy': "default-src 'none'", 'Access-Control-Expose-Headers': 'ETag, Link, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval', 'Transfer-Encoding': 'chunked', 'Access-Control-Allow-Origin': '*', 'X-Frame-Options': 'deny', 'Status': '200 OK', 'X-Served-By': 'eef8b8685a106934dcbb4b7c59fba0bf', 'X-GitHub-Request-Id': '059D:17F65:2FAC88D:3DE5D57:58EA0875', 'ETag': 'W/"7dc470913f1fe9bb6c7355b50a0737bc"', 'Date': 'Sun, 09 Apr 2017 10:09:57 GMT', 'X-RateLimit-Remaining': '38', 'Strict-Transport-Security': 'max-age=31536000; includeSubdomains; preload', 'Server': 'GitHub.com', 'X-GitHub-Media-Type': 'github.v3; format=json', 'X-Content-Type-Options': 'nosniff', 'Content-Encoding': 'gzip', 'Vary': 'Accept, Accept-Encoding', 'X-RateLimit-Limit': '60', 'Cache-Control': 'public, max-age=60, s-maxage=60', 'Content-Type': 'application/json; charset=utf-8', 'X-RateLimit-Reset': '1491735173'}
響應url
>>> response.url u'https://api.github.com/'
響應history
>>> response.history #若是沒有重定向,默認爲空 [] >>> response = requests.get('http://api.github.com') #通常訪問http,會跳轉到https >>> response.history [<Response [301]>]
響應時間
>>> response.elapsed datetime.timedelta(0, 3, 158723)
能夠調用的方法
>>> dir(response) ['__attrs__', '__bool__', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__getstate__', '__hash__', '__init__', '__iter__', '__module__', '__new__', '__nonzero__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_content', '_content_consumed', 'apparent_encoding', 'close', 'connection', 'content', 'cookies', 'elapsed', 'encoding', 'headers', 'history', 'is_permanent_redirect', 'is_redirect', 'iter_content', 'iter_lines', 'json', 'links', 'ok', 'raise_for_status', 'raw', 'reason', 'request', 'status_code', 'text', 'url']
請求對象和請求頭
>>> response.request <PreparedRequest [GET]> >>> response.request.headers {'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-327.el7.x86_64'}
請求的編碼
>>> response.encoding 'utf-8'
響應的內容,content爲str類型
>>> response.raw.read(10) '' >>> >>> >>> response.content '{"current_user_url":"https://api.github.com/user","current_user_authorizations_html_url":"https://github.com/settings/connections/applications{/client_id}","authorizations_url":"https://api.github.com/authorizations","code_search_url":"https://api.github.com/search/code?q={query}{&page,per_page,sort,order}","commit_search_url":"https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}","emails_url":"https://api.github.com/user/emails","emojis_url":"https://api.github.com/emojis","events_url":"https://api.github.com/events","feeds_url":"https://api.github.com/feeds","followers_url":"https://api.github.com/user/followers","following_url":"https://api.github.com/user/following{/target}","gists_url":"https://api.github.com/gists{/gist_id}","hub_url":"https://api.github.com/hub","issue_search_url":"https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}","issues_url":"https://api.github.com/issues","keys_url":"https://api.github.com/user/keys","notifications_url":"https://api.github.com/notifications","organization_repositories_url":"https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}","organization_url":"https://api.github.com/orgs/{org}","public_gists_url":"https://api.github.com/gists/public","rate_limit_url":"https://api.github.com/rate_limit","repository_url":"https://api.github.com/repos/{owner}/{repo}","repository_search_url":"https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}","current_user_repositories_url":"https://api.github.com/user/repos{?type,page,per_page,sort}","starred_url":"https://api.github.com/user/starred{/owner}{/repo}","starred_gists_url":"https://api.github.com/gists/starred","team_url":"https://api.github.com/teams","user_url":"https://api.github.com/users/{user}","user_organizations_url":"https://api.github.com/user/orgs","user_repositories_url":"https://api.github.com/users/{user}/repos{?type,page,per_page,sort}","user_search_url":"https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"}'
響應內容,text爲unicode類型
>>> response.text u'{"current_user_url":"https://api.github.com/user","current_user_authorizations_html_url":"https://github.com/settings/connections/applications{/client_id}","authorizations_url":"https://api.github.com/authorizations","code_search_url":"https://api.github.com/search/code?q={query}{&page,per_page,sort,order}","commit_search_url":"https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}","emails_url":"https://api.github.com/user/emails","emojis_url":"https://api.github.com/emojis","events_url":"https://api.github.com/events","feeds_url":"https://api.github.com/feeds","followers_url":"https://api.github.com/user/followers","following_url":"https://api.github.com/user/following{/target}","gists_url":"https://api.github.com/gists{/gist_id}","hub_url":"https://api.github.com/hub","issue_search_url":"https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}","issues_url":"https://api.github.com/issues","keys_url":"https://api.github.com/user/keys","notifications_url":"https://api.github.com/notifications","organization_repositories_url":"https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}","organization_url":"https://api.github.com/orgs/{org}","public_gists_url":"https://api.github.com/gists/public","rate_limit_url":"https://api.github.com/rate_limit","repository_url":"https://api.github.com/repos/{owner}/{repo}","repository_search_url":"https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}","current_user_repositories_url":"https://api.github.com/user/repos{?type,page,per_page,sort}","starred_url":"https://api.github.com/user/starred{/owner}{/repo}","starred_gists_url":"https://api.github.com/gists/starred","team_url":"https://api.github.com/teams","user_url":"https://api.github.com/users/{user}","user_organizations_url":"https://api.github.com/user/orgs","user_repositories_url":"https://api.github.com/users/{user}/repos{?type,page,per_page,sort}","user_search_url":"https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"}'
響應內容,爲json對象
>>> response.json( ... ) {u'issues_url': u'https://api.github.com/issues', u'current_user_repositories_url': u'https://api.github.com/user/repos{?type,page,per_page,sort}', u'rate_limit_url': u'https://api.github.com/rate_limit', u'repository_search_url': u'https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}', u'user_organizations_url': u'https://api.github.com/user/orgs', u'commit_search_url': u'https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}', u'repository_url': u'https://api.github.com/repos/{owner}/{repo}', u'emojis_url': u'https://api.github.com/emojis', u'hub_url': u'https://api.github.com/hub', u'keys_url': u'https://api.github.com/user/keys', u'following_url': u'https://api.github.com/user/following{/target}', u'emails_url': u'https://api.github.com/user/emails', u'authorizations_url': u'https://api.github.com/authorizations', u'code_search_url': u'https://api.github.com/search/code?q={query}{&page,per_page,sort,order}', u'followers_url': u'https://api.github.com/user/followers', u'public_gists_url': u'https://api.github.com/gists/public', u'organization_url': u'https://api.github.com/orgs/{org}', u'gists_url': u'https://api.github.com/gists{/gist_id}', u'feeds_url': u'https://api.github.com/feeds', u'user_search_url': u'https://api.github.com/search/users?q={query}{&page,per_page,sort,order}', u'user_url': u'https://api.github.com/users/{user}', u'events_url': u'https://api.github.com/events', u'organization_repositories_url': u'https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}', u'current_user_url': u'https://api.github.com/user', u'issue_search_url': u'https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}', u'notifications_url': u'https://api.github.com/notifications', u'starred_url': u'https://api.github.com/user/starred{/owner}{/repo}', u'starred_gists_url': u'https://api.github.com/gists/starred', u'current_user_authorizations_html_url': u'https://github.com/settings/connections/applications{/client_id}', u'user_repositories_url': u'https://api.github.com/users/{user}/repos{?type,page,per_page,sort}', u'team_url': u'https://api.github.com/teams'} >>> response.json()['team_url'] #訪問json的對象 u'https://api.github.com/teams'
9、下載圖片/文件
圖片/文件下載過程
瀏覽器模擬-->構建request(調試策略)-->讀取流data-->打開文件,存入數據
有一些返回的context的內容是沒有辦法讀取的,只有經過流的方式保存到文件中
import requests def download_image(): """ demo:下載圖片 :return: """ url= "http://img0.tech2ipo.com/upload/img/article/2015/06/1434420238601.png" headers={'User-Agent':'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'} response=requests.get(url,headers=headers,stream=True) with open('demo.png','wb') as fd: for chunk in response.iter_content(128): fd.write(chunk) print response.status_code # print response.reason # print response.content
使用上下午文管理的方式,讀物數據流,寫入文件,最後關閉流
def download_image_improved(): """ demo:下載圖片 :return: """ #僞造headers信息 headers={'User-Agent':'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'} #限定url url= "http://img0.tech2ipo.com/upload/img/article/2015/06/1434420238601.png" response=requests.get(url,headers=headers,stream=True) from contextlib import closing with closing(requests.get(url,headers=headers,stream=True)) as response: #打開文件 with open('demo.png','wb') as fd: #每128寫入一次 for chunk in response.iter_content(128): fd.write(chunk)
10、事件鉤子(Event hooks)
相似於js中的回調函數;
IO請求-->response-->回調函數
說白了,就是異步請求,request發出以後,直接返回,剩餘的IO請求,由回調函數處理;
異步處理demo
import requests def get_key_info(response,*args,**kwargs): """ 回調函數 :return: """ print response.headers['Content-Type'] def main(): """ 主程序 :return: """ requests.get('https://api.github.com',hooks=dict(response=get_key_info)) main() 執行結果: application/json; charset=utf-8
11、HTTP認證
一、http基本認證
requests a protected resource-->requests a username:password-->sends username:password-->returns requests resource
import requests BASE_URL='https://api.github.com' def construct_url(end_point): return '/'.join([BASE_URL,end_point]) def basic_auth(): """ 基本認證 :return: """ response=requests.get(construct_url('user'),auth=('imoocdeo','imoocdeo123')) print response.text print response.request.headers basic_auth() 打印結果: {"message":"Bad credentials","documentation_url":"https://developer.github.com/v3"} {'Authorization': 'Basic aW1vb2NkZW86aW1vb2NkZW8xMjM=', 'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.10.0'}
這裏字符串 aW1vb2NkZW86aW1vb2NkZW8xMjM=是使用base64編碼的結果
在終端打開bpython(pip install bpython安裝)
>>> import base64 >>> >>> base64.b64decode('aW1vb2NkZW86aW1vb2NkZW8xMjM=') 'imoocdeo:imoocdeo123' #是用戶名和密碼拼湊起來的,可是這樣可能不安全
二、OAUTH認證
https://www.codewars.com/ 使用的登陸認證方式
APP---github
github生成access_token: https://github.com/settings/tokens-->Generate new token-->選擇該token擁有的權限
def basic_oauth(): headers={'Authorization':'token 51684c8452ff83556f1a6705d2d85d7f68cd5ca7'} #user/emails,經過請求頭把token帶過去 response=requests.get(construct_url('user/emails'),headers=headers) print response.request.headers print response.text print response.status_code basic_oauth() 打印結果: {'Authorization': 'token 51684c8452ff83556f1a6705d2d85d7f68cd5ca7', 'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.10.0'} [{"email":"qq_c123@163.com","primary":false,"verified":false,"visibility":null},{"email":"charles_ali@qq.com","primary":true,"verified":true,"visibility":"public"},{"email":"qq_c1231@163.com","primary":false,"verified":false,"visibility":null}] 200
使用python自帶的語法糖類實現上述內容,__call__能夠將實例化的對象當作類進行實例化調用,即將實例化的對象在加(),就能夠調用__call__函數,f()();
from requests.auth import AuthBase class GithubAuth(AuthBase): def __init__(self,token): self.token=token def __call__(self, r): #request 加headers r.headers['Authorization']=' '.join(['token',self.token]) print r return r def oauth_advaced(): auth=GithubAuth('51684c8452ff83556f1a6705d2d85d7f68cd5ca7') response=requests.get(construct_url('user/emails'),auth=auth) print response.text oauth_advaced() 調用結果: <PreparedRequest [GET]> [{"email":"qq_c123@163.com","primary":false,"verified":false,"visibility":null},{"email":"charles_ali@qq.com","primary":true,"verified":true,"visibility":"public"},{"email":"qq_c1231@163.com","primary":false,"verified":false,"visibility":null}]
12、requests庫的proxy代理
import requests proxies={'http':'sock5://127.0.0.1:1080','https':'sock5://127.0.0.1:1080'} #設置代理地址 url='https://www.facebook.com' response=requests.get(url,proxies=proxies,timeout=10)
十3、Session和cookie
cookie:
瀏覽器請求(無cookie)-->服務器-->HTTP響應(響應頭有set cookie,設置有效期,權限範圍等等)-->解析cookie保存本地-->HTTP請求(攜帶cookie)--->解析cookie識別信息--HTTP響應
缺點:cookie是在瀏覽器端的,每次請求,都要攜帶cookie,消耗帶寬;cookie能夠僞造,不安全;
session:
瀏覽器HTTP請求-->服務器存儲session-->HTTP響應(set cookie-session-id 叫作session id的cookie)-->解析cookie保存本地(這個cookie很是小,僅含有session id)-->解析session id(到存儲中查,該session id對於的用戶信息,權限信息是什麼)-->HTTP響應