Django rest framework源碼分析（3）----節流

時間 2019-12-11

標籤 django rest framework 源碼分析節流欄目 Python 简体版

原文原文鏈接

自定義節流

有些時候爲了對用戶的訪問頻率進行限制和防止爬蟲，須要在規定的時間中對用戶訪問的次數進行限制python

下面自定義一個用戶每分鐘只能訪問3次，代碼以下：api

from rest_framework.throttling import BaseThrottle
import time
VISIT_RECORD = {}   #保存訪問記錄

class VisitThrottle(BaseThrottle):
    '''60s內只能訪問3次'''
    def __init__(self):
        self.history = None   #初始化訪問記錄

    def allow_request(self,request,view):
        #獲取用戶ip (get_ident)
        remote_addr = self.get_ident(request)
        ctime = time.time()
        #若是當前IP不在訪問記錄裏面，就添加到記錄
        if remote_addr not in VISIT_RECORD:
            VISIT_RECORD[remote_addr] = [ctime,]     #鍵值對的形式保存
            return True    #True表示能夠訪問
        #獲取當前ip的歷史訪問記錄
        history = VISIT_RECORD.get(remote_addr)
        #初始化訪問記錄
        self.history = history

        #若是有歷史訪問記錄，而且最先一次的訪問記錄離當前時間超過60s，就刪除最先的那個訪問記錄，
        #只要爲True，就一直循環刪除最先的一次訪問記錄
        while history and history[-1] < ctime - 60:
            history.pop()
        #若是訪問記錄不超過三次，就把當前的訪問記錄插到第一個位置（pop刪除最後一個）
        if len(history) < 3:
            history.insert(0,ctime)
            return True

    def wait(self):
        '''還須要等多久才能訪問'''
        ctime = time.time()
        return 60 - (ctime - self.history[-1])

上面的大碼就是當用戶第一次訪問的時候把它的IP地址和當前訪問的時間添加到字典 VISIT_RECORD 中，循環取出最早添加的時間判斷其時間有沒有過了60s，若是過了則將其刪掉，而後在看列表的長度是否小於3次，若是小於3次證實，是能夠訪問的的。wait表示的是還須要等待多久能夠繼續訪問緩存

咱們在使用的時候，只需在類視圖中簡單在類屬性 throttle_classes = [ 節流控制的類] 便可app

下面咱們在用戶登錄的時候來簡單的測試下，代碼代碼以下：ide

class AuthView(APIView):
    """
    用於用戶登陸認證
    """
    authentication_classes = []
    permission_classes = []
    throttle_classes = [VisitThrottle,] # 進行訪問次數的限制

    def post(self,request,*args,**kwargs):

        ret = {'code':1000,'msg':None}
        try:
            user = request._request.POST.get('username')
            pwd = request._request.POST.get('password')
            obj = models.UserInfo.objects.filter(username=user,password=pwd).first()
            if not obj:
                ret['code'] = 1001
                ret['msg'] = "用戶名或密碼錯誤"
            # 爲登陸用戶建立token
            token = md5(user)
            # 存在就更新，不存在就建立
            models.UserToken.objects.update_or_create(user=obj,defaults={'token':token})
            ret['token'] = token
        except Exception as e:
            print(e)
            ret['code'] = 1002
            ret['msg'] = '請求異常'

        return JsonResponse(ret)

咱們連續訪問3次登錄測試的結果以下函數

節流源碼分析

源碼入口 dispatch 代碼以下源碼分析

    def dispatch(self, request, *args, **kwargs):
        """
        `.dispatch()` is pretty much the same as Django's regular dispatch,
        but with extra hooks for startup, finalize, and exception handling.
        """
        self.args = args
        self.kwargs = kwargs
        # 對原生的request進行封裝
        request = self.initialize_request(request, *args, **kwargs)
        self.request = request
        self.headers = self.default_response_headers  # deprecate?

        try:
            self.initial(request, *args, **kwargs)

            # Get the appropriate handler method
            if request.method.lower() in self.http_method_names:
                handler = getattr(self, request.method.lower(),
                                  self.http_method_not_allowed)
            else:
                handler = self.http_method_not_allowed

            response = handler(request, *args, **kwargs)

        except Exception as exc:
            response = self.handle_exception(exc)

        self.response = self.finalize_response(request, response, *args, **kwargs)
        return self.response

執行認證 self.initial 代碼以下post

    def initial(self, request, *args, **kwargs):
        """
        Runs anything that needs to occur prior to calling the method handler.
        """
        self.format_kwarg = self.get_format_suffix(**kwargs)

        # Perform content negotiation and store the accepted info on the request
        neg = self.perform_content_negotiation(request)
        request.accepted_renderer, request.accepted_media_type = neg

        # Determine the API version, if versioning is in use.
        version, scheme = self.determine_version(request, *args, **kwargs)
        request.version, request.versioning_scheme = version, scheme

        # Ensure that the incoming request is permitted
        # 實現認證
        self.perform_authentication(request)
        # 權限判斷
        self.check_permissions(request)
        # 訪問頻率控制
        self.check_throttles(request)

訪問頻率控制self.check_throttles(request)代碼以下：測試

    def check_throttles(self, request):
        """
        Check if request should be throttled.
        Raises an appropriate exception if the request is throttled.
        """
        for throttle in self.get_throttles():
            if not throttle.allow_request(request, self):
                self.throttled(request, throttle.wait())

在上面的源碼中咱們能夠知道，若是沒有經過會返回 false 執行 self.throttled(request, throttle.wait()),拋出異常，代碼以下ui

 def throttled(self, request, wait):
        """
        If request is throttled, determine what kind of exception to raise.
        """
        raise exceptions.Throttled(wait)

若是返回True表示能夠繼續訪問，讓咱們來繼續追蹤 self.get_throttles() 代碼以下：

    def get_throttles(self):
        """
        Instantiates and returns the list of throttles that this view uses.
        """
        # 返回訪問頻率控制的列表
        return [throttle() for throttle in self.throttle_classes]

經過上面的源碼咱們知道爲何咱們在須要進行節流限制的接口中設置類屬性 throttle_class = [ 節流限制的類 ] ，還有就是在咱們節流限制的類中爲何要重寫

allow_request(request, self) 和 wait 方法

進行全局節流的配置

上面是咱們本身定義的節流控制，走咱們本身定義的類屬性 throttle_class = [ 節流限制的類 ] 若是咱們想使用父類的從而實現全局的配置又該如何去實現呢

父類中的 thtottle_classes 設置以下

因此咱們只需在配置文件中的 REST_FRAMEWORK 添加 DEFAULT_THROTTLE_CLASSES 的路徑便可，配置以下

REST_FRAMEWORK = {
    # 全局使用的認證類
    "DEFAULT_AUTHENTICATION_CLASSES":['api.utils.auth.FirstAuthtication','api.utils.auth.Authtication', ],
    "UNAUTHENTICATED_USER":lambda :"匿名用戶",
    "UNAUTHENTICATED_TOKEN":None,
    "DEFAULT_PERMISSION_CLASSES":['api.utils.permission.SVIPPermission'],# 默認的權限認證
    "DEFAULT_THROTTLE_CLASSES":["api.utils.throttle.VisitThrottle"], # 進行節流的限制
}

根據上面節流限制類路徑的定義，咱們在應用 api 下的utils目錄下建立 throttle.py 把代碼以下

from rest_framework.throttling import BaseThrottle,SimpleRateThrottle



import time
VISIT_RECORD = {}
class VisitThrottle(BaseThrottle):

    def __init__(self):
        self.history = None

    def allow_request(self,request,view):
        # 1. 獲取用戶IP
        remote_addr = self.get_ident(request)

        ctime = time.time()
        if remote_addr not in VISIT_RECORD:
            VISIT_RECORD[remote_addr] = [ctime,]
            return True
        history = VISIT_RECORD.get(remote_addr)
        self.history = history

        while history and history[-1] < ctime - 60:
            history.pop()

        if len(history) < 3:
            history.insert(0,ctime)
            return True

        # return True    # 表示能夠繼續訪問
        # return False # 表示訪問頻率過高，被限制

    def wait(self):
        # 還須要等多少秒才能訪問
        ctime = time.time()
        return 60 - (ctime - self.history[-1])

咱們的視圖類中就不須要再添加節流的類屬性的配置了，就能夠實現節流的控制。簡單的代碼以下

class AuthView(APIView):
    """
    用於用戶登陸認證
    """
    authentication_classes = []
    permission_classes = []

    def post(self,request,*args,**kwargs):
        pass

內置的控制頻率類

上面是寫的自定義節流，drf內置了不少節流的類，用起來比較方便。

（1）BaseThrottle

本身要寫allow_request和wait方法
get_ident就是獲取ip

class BaseThrottle(object):
    """
    Rate throttling of requests.
    """

    def allow_request(self, request, view):
        """
        Return `True` if the request should be allowed, `False` otherwise.
        """
        raise NotImplementedError('.allow_request() must be overridden')

    def get_ident(self, request):
        """
        Identify the machine making the request by parsing HTTP_X_FORWARDED_FOR
        if present and number of proxies is > 0. If not use all of
        HTTP_X_FORWARDED_FOR if it is available, if not use REMOTE_ADDR.
        """
        xff = request.META.get('HTTP_X_FORWARDED_FOR')
        remote_addr = request.META.get('REMOTE_ADDR')
        num_proxies = api_settings.NUM_PROXIES

        if num_proxies is not None:
            if num_proxies == 0 or xff is None:
                return remote_addr
            addrs = xff.split(',')
            client_addr = addrs[-min(num_proxies, len(addrs))]
            return client_addr.strip()

        return ''.join(xff.split()) if xff else remote_addr

    def wait(self):
        """
        Optionally, return a recommended number of seconds to wait before
        the next request.
        """
        return None

咱們上面的例子就是經過繼承 BaseThrottle 來實現的，相對較麻煩一點

（2）SimpleRateThrottle

class SimpleRateThrottle(BaseThrottle):
    """
    A simple cache implementation, that only requires `.get_cache_key()`
    to be overridden.

    The rate (requests / seconds) is set by a `rate` attribute on the View
    class.  The attribute is a string of the form 'number_of_requests/period'.

    Period should be one of: ('s', 'sec', 'm', 'min', 'h', 'hour', 'd', 'day')

    Previous request information used for throttling is stored in the cache.
    """
    cache = default_cache
    timer = time.time
    cache_format = 'throttle_%(scope)s_%(ident)s'
    scope = None   #這個值自定義，寫什麼均可以
    THROTTLE_RATES = api_settings.DEFAULT_THROTTLE_RATES

    def __init__(self):
        if not getattr(self, 'rate', None):
            self.rate = self.get_rate()
        self.num_requests, self.duration = self.parse_rate(self.rate)

    def get_cache_key(self, request, view):
        """
        Should return a unique cache-key which can be used for throttling.
        Must be overridden.

        May return `None` if the request should not be throttled.
        """
        raise NotImplementedError('.get_cache_key() must be overridden')

    def get_rate(self):
        """
        Determine the string representation of the allowed request rate.
        """
        if not getattr(self, 'scope', None):
            msg = ("You must set either `.scope` or `.rate` for '%s' throttle" %
                   self.__class__.__name__)
            raise ImproperlyConfigured(msg)

        try:
            return self.THROTTLE_RATES[self.scope]
        except KeyError:
            msg = "No default throttle rate set for '%s' scope" % self.scope
            raise ImproperlyConfigured(msg)

    def parse_rate(self, rate):
        """
        Given the request rate string, return a two tuple of:
        <allowed number of requests>, <period of time in seconds>
        """
        if rate is None:
            return (None, None)
        num, period = rate.split('/')
        num_requests = int(num)
        duration = {'s': 1, 'm': 60, 'h': 3600, 'd': 86400}[period[0]]
        return (num_requests, duration)

    def allow_request(self, request, view):
        """
        Implement the check to see if the request should be throttled.

        On success calls `throttle_success`.
        On failure calls `throttle_failure`.
        """
        if self.rate is None:
            return True

        self.key = self.get_cache_key(request, view)
        if self.key is None:
            return True

        self.history = self.cache.get(self.key, [])
        self.now = self.timer()

        # Drop any requests from the history which have now passed the
        # throttle duration
        while self.history and self.history[-1] <= self.now - self.duration:
            self.history.pop()
        if len(self.history) >= self.num_requests:
            return self.throttle_failure()
        return self.throttle_success()

    def throttle_success(self):
        """
        Inserts the current request's timestamp along with the key
        into the cache.
        """
        self.history.insert(0, self.now)
        self.cache.set(self.key, self.history, self.duration)
        return True

    def throttle_failure(self):
        """
        Called when a request to the API has failed due to throttling.
        """
        return False

    def wait(self):
        """
        Returns the recommended next request time in seconds.
        """
        if self.history:
            remaining_duration = self.duration - (self.now - self.history[-1])
        else:
            remaining_duration = self.duration

        available_requests = self.num_requests - len(self.history) + 1
        if available_requests <= 0:
            return None

        return remaining_duration / float(available_requests)

讓咱們來繼續追蹤如下其內部的源碼，咱們來看一下函數

    def allow_request(self, request, view):
        """
        Implement the check to see if the request should be throttled.

        On success calls `throttle_success`.
        On failure calls `throttle_failure`.
        """
        if self.rate is None:
            return True
        # 獲取一個用於緩存的鍵來記錄限流的
        self.key = self.get_cache_key(request, view)
        if self.key is None:
            return True

        self.history = self.cache.get(self.key, [])
        self.now = self.timer()

        # Drop any requests from the history which have now passed the
        # throttle duration
        while self.history and self.history[-1] <= self.now - self.duration:
            self.history.pop()
        if len(self.history) >= self.num_requests:
            return self.throttle_failure()
        return self.throttle_success()

咱們能夠看到它的內部首先判斷 self.rate 是否爲None ，咱們不知道其是什麼不放追蹤如下看看

 if not getattr(self, 'rate', None):
            self.rate = self.get_rate()
        self.num_requests, self.duration = self.parse_rate(self.rate)

get_rate 的代碼以下

    def get_rate(self):
        """
        Determine the string representation of the allowed request rate.
        """
        if not getattr(self, 'scope', None):
            msg = ("You must set either `.scope` or `.rate` for '%s' throttle" %
                   self.__class__.__name__)
            raise ImproperlyConfigured(msg)

        try:
            return self.THROTTLE_RATES[self.scope]
        except KeyError:
            msg = "No default throttle rate set for '%s' scope" % self.scope
            raise ImproperlyConfigured(msg)

咱們看到若是沒有定義self.scope這個屬性的話，直接會拋出異常，因此在這裏若是咱們要是繼承 SimpleRateThrottle 類來實現限流的話，必須定義類屬性 scope ，這個是須要咱們本身寫的

讓咱們繼續追蹤代碼下面的 self.THROTTLE_RATES[self.scope]

    THROTTLE_RATES = api_settings.DEFAULT_THROTTLE_RATES

咱們能夠看到它是從配置文件中讀取 DEFAULT_THROTTLE_CLASSES 能夠看到它是從配置文件中根據鍵 self.scrope去進行讀取的

讓咱們來繼續追蹤 allow_request 中的 self.key = self.get_cache_key(request, view) 其代碼以下：

    def get_cache_key(self, request, view):
        """
        Should return a unique cache-key which can be used for throttling.
        Must be overridden.

        May return `None` if the request should not be throttled.
        """
        raise NotImplementedError('.get_cache_key() must be overridden')

在上面的源碼中，咱們發現其拋出了一個異常，這個是用來獲取一個用於緩存的鍵來記錄限流的，因此這個方法咱們要重寫，重寫的代碼以下

from rest_framework.throttling import BaseThrottle,SimpleRateThrottle
class VisitThrottle(SimpleRateThrottle):
    scope = "NBA"
　　 def get_cache_key(self, request, view): 
　　　　return self.get_ident(request) # 默認根據IP做爲鍵 

class UserThrottle(SimpleRateThrottle): 
　　scope = "NBAUser" 
　　def get_cache_key(self, request, view): 

　　　　return request.user.username # 登錄後用戶名做爲鍵

下緬的流程就和咱們自定義的限流的邏輯同樣了，早這裏就不作陳述了

繼承 SimpleRateThrottle的寫法以下

咱們能夠經過繼承SimpleRateThrottle類，來實現節流，會更加的簡單，由於SimpleRateThrottle裏面都幫咱們寫好了，

from rest_framework.throttling import SimpleRateThrottle

class VisitThrottle(SimpleRateThrottle):
    '''匿名用戶60s只能訪問三次（根據ip）'''
    scope = 'NBA'   #這裏面的值，本身隨便定義，settings裏面根據這個值配置Rate

    def get_cache_key(self, request, view):
        #經過ip限制節流
        return self.get_ident(request)

class UserThrottle(SimpleRateThrottle):
    '''登陸用戶60s能夠訪問10次'''
    scope = 'NBAUser'    #這裏面的值，本身隨便定義，settings裏面根據這個值配置Rate

    def get_cache_key(self, request, view):
        return request.user.username

（2）settings.py

#全局
REST_FRAMEWORK = {
    #節流
    "DEFAULT_THROTTLE_CLASSES":['API.utils.throttle.UserThrottle'],   #全局配置，登陸用戶節流限制（10/m）
    "DEFAULT_THROTTLE_RATES":{
        'NBA':'3/m',         #沒登陸用戶3/m，NBA就是scope定義的值
        'NBAUser':'10/m',    #登陸用戶10/m，NBAUser就是scope定義的值
    }
}

（3）views.py

局部配置方法

class AuthView(APIView):
    .
    .    
    .
    # 默認的節流是登陸用戶（10/m）,AuthView不須要登陸，這裏用匿名用戶的節流（3/m）
    throttle_classes = [VisitThrottle,]
　 　.

說明：

API.utils.throttle.UserThrottle 這個是全局配置（根據ip限制，10/m）
DEFAULT_THROTTLE_RATES --->>>設置訪問頻率的
throttle_classes = [VisitThrottle,] --->>>局部配置（不適用settings裏面默認的全局配置）

總結

基本使用

建立類，繼承BaseThrottle, 實現：allow_request ，wait
建立類，繼承SimpleRateThrottle, 實現： get_cache_key， scope='NBA' (配置文件中的key)

全局

   #節流
    "DEFAULT_THROTTLE_CLASSES":['API.utils.throttle.UserThrottle'],   #全局配置，登陸用戶節流限制（10/m）
    "DEFAULT_THROTTLE_RATES":{
        'NBA':'3/m',         #沒登陸用戶3/m，NBA就是scope定義的值
        'NBAUser':'10/m',    #登陸用戶10/m，NBAUser就是scope定義的值
    }
}