【Python之路】特別篇--服務商API認證、Restful、一致性哈希

時間 2019-11-21

標籤 Python之路特別務商 api 認證 restful 一致性哈希欄目 Zookeeper 简体版

原文原文鏈接

API加密方式

1/ 加密方式:html

Md5 (隨機字符串 + 時間戳)node

2/ 發送方式:python

http://127.0.0.1:8888/index?pid= MD5加密值 | 時間戳 | 序號

服務端接收:web

(1) 判斷時間戳是否在有效期內json

(2) 判斷url地址是否已經訪問過api

(3) 判斷md5加密值與服務端生成的md5 是否匹配緩存

import hashlib
access_record = [

]

PID_LIST = [
    'abcd',
    'ddd',
    'ks',
]
class MainHandler(tornado.web.RequestHandler):
    def get(self):
        import time
        # 獲取url中所有數據
        pid = self.get_argument('pid', None)
        # 獲取變量
        m5, client_time, i = pid.split('|')

        server_time = time.time()
        # 時間超過10s禁止
        if server_time > float(client_time) + 10:
            self.write(' Error !! ')
            return
        # 處理10s內容重複的請求
        if pid in access_record:
            self.write(' Error !! ')
            return
        access_record.append(pid)

        pid = PID_LIST[int(i)]
        ramdom_str = "%s|%s" %(pid, client_time)
        h = hashlib.md5()
        h.update(bytes(ramdom_str, encoding='utf-8'))
        server_m5 = h.hexdigest()
        # print(m5,server_m5)
        if m5 == server_m5:
            self.write("Hello, world")
        else:
            self.write(' Error !! ')

客戶端: 生成帶規則的url鏈接 -> SDK服務器

import requests
import time
import hashlib

url = 'http://127.0.0.1:8888/index?pid=%s'

pid = 'aabc'
client_time = time.time()

h = hashlib.md5()
ramdom_str = "%s|%s" %(pid, client_time)
h.update(bytes(ramdom_str,encoding='utf-8'))

m5 = h.hexdigest()
url_payloads = '%s|%s|%d'%(m5,client_time,0)
url = url%(url_payloads)

print(url)
ret = requests.get(url)
print(ret.text)

客戶端

Access_LIST 能夠用Redis 設置過時時間,到期限自動清空內容。restful

Restful

1、名稱網絡

REST，即Representational State Transfer的縮寫。我對這個詞組的翻譯是 "表現層狀態轉化" 。

若是一個架構符合REST原則，就稱它爲RESTful架構。

2、資源（Resources）

REST的名稱"表現層狀態轉化"中，省略了主語。"表現層"其實指的是"資源"（Resources）的"表現層"。

所謂"資源"，就是網絡上的一個實體，或者說是網絡上的一個具體信息。它能夠是一段文本、一張圖片、一首歌曲、一種服務，總之就是一個具體的實在。你能夠用一個URL（統一資源定位符）指向它，每種資源對應一個特定的URL。要獲取這個資源，訪問它的URL就能夠，所以URL就成了每個資源的地址或獨一無二的識別符。

3、表現層（Representation）

"資源"是一種信息實體，它能夠有多種外在表現形式。咱們把"資源"具體呈現出來的形式，叫作它的"表現層"（Representation）。

好比，文本能夠用txt格式表現，也能夠用HTML格式、XML格式、JSON格式表現，甚至能夠採用二進制格式；圖片能夠用JPG格式表現，也能夠用PNG格式表現。

URL只表明資源的實體，不表明它的形式。嚴格地說，有些網址最後的".html"後綴名是沒必要要的，由於這個後綴名錶示格式，屬於"表現層"範疇，而URL應該只表明"資源"的位置。它的具體表現形式，應該在HTTP請求的頭信息中用Accept和Content-Type字段指定，這兩個字段纔是對"表現層"的描述。

4、狀態轉化（State Transfer）

訪問一個網站，就表明了客戶端和服務器的一個互動過程。在這個過程當中，勢必涉及到數據和狀態的變化。

互聯網通訊協議HTTP協議，是一個無狀態協議。這意味着，全部的狀態都保存在服務器端。所以，若是客戶端想要操做服務器，必須經過某種手段，讓服務器端發生"狀態轉化"（State Transfer）。而這種轉化是創建在表現層之上的，因此就是"表現層狀態轉化"。

客戶端用到的手段，只能是HTTP協議。具體來講，就是HTTP協議裏面，四個表示操做方式的動詞：GET、POST、PUT、DELETE。它們分別對應四種基本操做：GET用來獲取資源，POST用來新建資源（也能夠用於更新資源），PUT用來更新資源，DELETE用來刪除資源。

5、綜述

綜合上面的解釋，咱們總結一下什麼是RESTful架構：

　　（1）每個URL表明一種資源；

　　（2）客戶端和服務器之間，傳遞這種資源的某種表現層；

　　（3）客戶端經過四個HTTP動詞，對服務器端資源進行操做，實現"表現層狀態轉化"。

6、誤區

RESTful架構有一些典型的設計誤區。

最多見的一種設計錯誤，就是URI包含動詞。由於"資源"表示一種實體，因此應該是名詞，URI不該該有動詞，動詞應該放在HTTP協議中。

舉例來講，某個URI是/posts/show/1，其中show是動詞，這個URI就設計錯了，正確的寫法應該是/posts/1，而後用GET方法表示show。

好比網上匯款，從帳戶1向帳戶2匯款500元，錯誤的URI是：

POST /accounts/1/transfer/500/to/2

正確的寫法是把動詞transfer改爲名詞transaction，資源不能是動詞，可是能夠是一種服務：

POST /transaction HTTP/1.1
Host: 127.0.0.1
　　
from=1&to=2&amount=500.00

另外一個設計誤區，就是在URI中加入版本號：

http://www.example.com/app/1.0/foo
http://www.example.com/app/1.1/foo
http://www.example.com/app/2.0/foo

由於不一樣的版本，能夠理解成同一種資源的不一樣表現形式，因此應該採用同一個URI。版本號能夠在HTTP請求頭信息的Accept字段中進行區分

Accept: vnd.example-com.foo+json; version=1.0
Accept: vnd.example-com.foo+json; version=1.1
Accept: vnd.example-com.foo+json; version=2.0

👉 詳情點擊

一致性哈希

py2.7版本哈希

# -*- coding: utf-8 -*-
"""
    hash_ring
    ~~~~~~~~~~~~~~
    Implements consistent hashing that can be used when
    the number of server nodes can increase or decrease (like in memcached).

    Consistent hashing is a scheme that provides a hash table functionality
    in a way that the adding or removing of one slot
    does not significantly change the mapping of keys to slots.

    More information about consistent hashing can be read in these articles:

        "Web Caching with Consistent Hashing":
            http://www8.org/w8-papers/2a-webserver/caching/paper2.html

        "Consistent hashing and random trees:
        Distributed caching protocols for relieving hot spots on the World Wide Web (1997)":
            http://citeseerx.ist.psu.edu/legacymapper?did=38148


    Example of usage::

        memcache_servers = ['192.168.0.246:11212',
                            '192.168.0.247:11212',
                            '192.168.0.249:11212']

        ring = HashRing(memcache_servers)
        server = ring.get_node('my_key')

    :copyright: 2008 by Amir Salihefendic.
    :license: BSD
"""

import math
import sys
from bisect import bisect

if sys.version_info >= (2, 5):
    import hashlib
    md5_constructor = hashlib.md5
else:
    import md5
    md5_constructor = md5.new

class HashRing(object):

    def __init__(self, nodes=None, weights=None):
        """`nodes` is a list of objects that have a proper __str__ representation.
        `weights` is dictionary that sets weights to the nodes.  The default
        weight is that all nodes are equal.
        """
        self.ring = dict()
        self._sorted_keys = []

        self.nodes = nodes

        if not weights:
            weights = {}
        self.weights = weights

        self._generate_circle()

    def _generate_circle(self):
        """Generates the circle.
        """
        total_weight = 0
        for node in self.nodes:
            total_weight += self.weights.get(node, 1)

        for node in self.nodes:
            weight = 1

            if node in self.weights:
                weight = self.weights.get(node)

            factor = math.floor((40*len(self.nodes)*weight) / total_weight);

            for j in xrange(0, int(factor)):
                b_key = self._hash_digest( '%s-%s' % (node, j) )

                for i in xrange(0, 3):
                    key = self._hash_val(b_key, lambda x: x+i*4)
                    self.ring[key] = node
                    self._sorted_keys.append(key)

        self._sorted_keys.sort()

    def get_node(self, string_key):
        """Given a string key a corresponding node in the hash ring is returned.

        If the hash ring is empty, `None` is returned.
        """
        pos = self.get_node_pos(string_key)
        if pos is None:
            return None
        return self.ring[ self._sorted_keys[pos] ]

    def get_node_pos(self, string_key):
        """Given a string key a corresponding node in the hash ring is returned
        along with it's position in the ring.

        If the hash ring is empty, (`None`, `None`) is returned.
        """
        if not self.ring:
            return None

        key = self.gen_key(string_key)

        nodes = self._sorted_keys
        pos = bisect(nodes, key)

        if pos == len(nodes):
            return 0
        else:
            return pos

    def iterate_nodes(self, string_key, distinct=True):
        """Given a string key it returns the nodes as a generator that can hold the key.

        The generator iterates one time through the ring
        starting at the correct position.

        if `distinct` is set, then the nodes returned will be unique,
        i.e. no virtual copies will be returned.
        """
        if not self.ring:
            yield None, None

        returned_values = set()
        def distinct_filter(value):
            if str(value) not in returned_values:
                returned_values.add(str(value))
                return value

        pos = self.get_node_pos(string_key)
        for key in self._sorted_keys[pos:]:
            val = distinct_filter(self.ring[key])
            if val:
                yield val

        for i, key in enumerate(self._sorted_keys):
            if i < pos:
                val = distinct_filter(self.ring[key])
                if val:
                    yield val

    def gen_key(self, key):
        """Given a string key it returns a long value,
        this long value represents a place on the hash ring.

        md5 is currently used because it mixes well.
        """
        b_key = self._hash_digest(key)
        return self._hash_val(b_key, lambda x: x)

    def _hash_val(self, b_key, entry_fn):
        return (( b_key[entry_fn(3)] << 24)
                |(b_key[entry_fn(2)] << 16)
                |(b_key[entry_fn(1)] << 8)
                | b_key[entry_fn(0)] )

    def _hash_digest(self, key):
        m = md5_constructor()
        m.update(key)
        return map(ord, m.digest())

View Code

py3.5版本哈希

# -*- coding: utf-8 -*-
"""
    hash_ring
    ~~~~~~~~~~~~~~
    Implements consistent hashing that can be used when
    the number of server nodes can increase or decrease (like in memcached).

    Consistent hashing is a scheme that provides a hash table functionality
    in a way that the adding or removing of one slot
    does not significantly change the mapping of keys to slots.

    More information about consistent hashing can be read in these articles:

        "Web Caching with Consistent Hashing":
            http://www8.org/w8-papers/2a-webserver/caching/paper2.html

        "Consistent hashing and random trees:
        Distributed caching protocols for relieving hot spots on the World Wide Web (1997)":
            http://citeseerx.ist.psu.edu/legacymapper?did=38148


    Example of usage::

        memcache_servers = ['192.168.0.246:11212',
                            '192.168.0.247:11212',
                            '192.168.0.249:11212']

        ring = HashRing(memcache_servers)
        server = ring.get_node('my_key')

    :copyright: 2008 by Amir Salihefendic.
    :license: BSD
"""

import math
import sys
from bisect import bisect

if sys.version_info >= (2, 5):
    import hashlib
    md5_constructor = hashlib.md5
else:
    import md5
    md5_constructor = md5.new

class HashRing(object):

    def __init__(self, nodes=None, weights=None):
        """`nodes` is a list of objects that have a proper __str__ representation.
        `weights` is dictionary that sets weights to the nodes.  The default
        weight is that all nodes are equal.
        """
        self.ring = dict()
        self._sorted_keys = []

        self.nodes = nodes

        if not weights:
            weights = {}
        self.weights = weights

        self._generate_circle()

    def _generate_circle(self):
        """Generates the circle.
        """
        total_weight = 0
        for node in self.nodes:
            total_weight += self.weights.get(node, 1)

        for node in self.nodes:
            weight = 1

            if node in self.weights:
                weight = self.weights.get(node)

            factor = math.floor((40*len(self.nodes)*weight) / total_weight);

            for j in range(0, int(factor)):
                b_key = self._hash_digest( '%s-%s' % (node, j) )

                for i in range(0, 3):
                    key = self._hash_val(b_key, lambda x: x+i*4)
                    self.ring[key] = node
                    self._sorted_keys.append(key)

        self._sorted_keys.sort()

    def get_node(self, string_key):
        """Given a string key a corresponding node in the hash ring is returned.

        If the hash ring is empty, `None` is returned.
        """
        pos = self.get_node_pos(string_key)
        if pos is None:
            return None
        return self.ring[ self._sorted_keys[pos] ]

    def get_node_pos(self, string_key):
        """Given a string key a corresponding node in the hash ring is returned
        along with it's position in the ring.

        If the hash ring is empty, (`None`, `None`) is returned.
        """
        if not self.ring:
            return None

        key = self.gen_key(string_key)

        nodes = self._sorted_keys
        pos = bisect(nodes, key)

        if pos == len(nodes):
            return 0
        else:
            return pos

    def iterate_nodes(self, string_key, distinct=True):
        """Given a string key it returns the nodes as a generator that can hold the key.

        The generator iterates one time through the ring
        starting at the correct position.

        if `distinct` is set, then the nodes returned will be unique,
        i.e. no virtual copies will be returned.
        """
        if not self.ring:
            yield None, None

        returned_values = set()
        def distinct_filter(value):
            if str(value) not in returned_values:
                returned_values.add(str(value))
                return value

        pos = self.get_node_pos(string_key)
        for key in self._sorted_keys[pos:]:
            val = distinct_filter(self.ring[key])
            if val:
                yield val

        for i, key in enumerate(self._sorted_keys):
            if i < pos:
                val = distinct_filter(self.ring[key])
                if val:
                    yield val

    def gen_key(self, key):
        """Given a string key it returns a long value,
        this long value represents a place on the hash ring.

        md5 is currently used because it mixes well.
        """
        b_key = self._hash_digest(key)
        return self._hash_val(b_key, lambda x: x)

    def _hash_val(self, b_key, entry_fn):
        return (( b_key[entry_fn(3)] << 24)
                |(b_key[entry_fn(2)] << 16)
                |(b_key[entry_fn(1)] << 8)
                | b_key[entry_fn(0)] )

    def _hash_digest(self, key):
        m = md5_constructor()
        m.update(bytes(key, encoding='utf-8'))
        return list(m.digest())

View Code

區別在於py2.7下

m = md5_constructor()
m.update(key)
return map(ord, m.digest())

digest() 返回的是字符串, 最後map函數調用ord,返回數字列表

py3.5中: digest() 返回的是字節類型，for循環後就能獲得數字，省去了ord這一步，即便用 list進行for循環便可生成數字列表！

def _hash_digest(self, key):
    m = md5_constructor()
    m.update(bytes(key, encoding='utf-8'))
    return list(m.digest())

例子:

假設有5臺memcache服務器 ( 利用ip進行哈希 )

memcache_servers = ['192.168.0.245:11212',
                    '192.168.0.246:11212',
                    '192.168.0.247:11212',
                    '192.168.0.248:11212',
                    '192.168.0.249:11212']

主機ip 哈希完成後造成的哈希值以下

    cacheA       key1
    cacheB       key2
    cacheC       key3
    cacheD       key4
    cacheE       key5

而後5臺節點圍繞稱一個環形，如圖

主機造成哈希值後，若是須要對某個值進行hash,(這裏和普通hash不同的就是這裏只hash不求餘)，當hash求出一個數值後，

查看這個值介於哪兩個值之間。好比介於key4和key5之間，而後從key4這個落點開始順時針查找，最早遇到的第一個cache服務器後，就把這個服務器看成這次緩存的落點，

從而把這個值放到這個落點上。如圖:

在使用普通的哈希時，當其中任意一臺機器down掉後，咱們須要重計算落點，由於除數已經發生了改變，因此都要從新設置緩存。

而使用一致性哈希，當其中一臺機器down掉後，只有它上面到它這一段的環路緩存失效了，如：當cacheC down掉後，掉落在cacheB到cacheC之間的值都將失效，那麼失效的落點將繼續順時針查找cache服務器，因此新的落點就會降落到cacheD上。

memcache服務器中，爲某些ip添加權重，就是對環上設置多個相同的key增長几率

memcache_servers = ['192.168.0.246:11212',
                    '192.168.0.247:11212',
                    '192.168.0.249:11212']
weights = {
    '192.168.0.246:11212': 1,
    '192.168.0.247:11212': 2,
    '192.168.0.249:11212': 10000
}

ring = HashRing(memcache_servers, weights)
server = ring.get_node('my_key1')
print(server)

# 鏈接mem(server)
# mem.set()