內存淘汰,和平時咱們設置redis key的過時時間,不是一回事;內存淘汰是說,假設咱們限定redis只能使用8g內存,如今已經使用了這麼多了(包括設置了過時時間的key和沒設過時時間的key),那,後續的set操做,還怎麼辦呢?java
是否是隻能報錯了?面試
那不行啊,不科學吧,由於有的key,可能已經好久沒人用了,可能之後也不會再用到了,那咱們是否是能夠把這類key給幹掉呢?redis
幹掉key的過程,就是內存淘汰。算法
當咱們在配置文件裏設置了以下屬性時:數據庫
# maxmemory <bytes>
默認,該屬性是被註釋掉的。緩存
其實,這個配置項的註釋,至關有價值,咱們來看看:服務器
# Don't use more memory than the specified amount of bytes. # When the memory limit is reached Redis will try to remove keys # according to the eviction policy selected (see maxmemory-policy). # # If Redis can't remove keys according to the policy, or if the policy is # set to 'noeviction', Redis will start to reply with errors to commands # that would use more memory, like SET, LPUSH, and so on, and will continue # to reply to read-only commands like GET. # # This option is usually useful when using Redis as an LRU cache, or to set # a hard memory limit for an instance (using the 'noeviction' policy). # # WARNING: If you have slaves attached to an instance with maxmemory on, # the size of the output buffers needed to feed the slaves are subtracted # from the used memory count, so that network problems / resyncs will # not trigger a loop where keys are evicted, and in turn the output # buffer of slaves is full with DELs of keys evicted triggering the deletion # of more keys, and so forth until the database is completely emptied. # # In short... if you have slaves attached it is suggested that you set a lower # limit for maxmemory so that there is some free RAM on the system for slave # output buffers (but this is not needed if the policy is 'noeviction'). # # maxmemory <bytes>
渣翻譯以下:app
不能使用超過指定數量bytes的內存。當該內存限制被達到時,redis會根據過時策略(eviction policy,經過參數 maxmemory-policy來指定)來驅逐key。dom
若是redis根據指定的策略,或者策略被設置爲「noeviction」,redis會開始針對以下這種命令,回覆錯誤。什麼命令呢?會使用更多內存的那類命令,好比set、lpush;只讀命令仍是不受影響,能夠正常響應。ide
該選項一般在redis使用LRU緩存時有用,或者在使用noeviction策略時,設置一個進程級別的內存limit。
所謂策略,意思是,當咱們要刪除部分key的時候,刪哪些,不刪哪些?是否是須要一個策略?好比是隨機刪,就像滅霸同樣?仍是按照lru時間來刪,lru的策略意思就是,最近最少使用的key,將被優先刪除。
總之,咱們須要定一個規則。
redis默認支持如下策略:
# MAXMEMORY POLICY: how Redis will select what to remove when maxmemory # is reached. You can select among five behaviors: # # volatile-lru -> remove the key with an expire set using an LRU algorithm # allkeys-lru -> remove any key accordingly to the LRU algorithm # volatile-random -> remove a random key with an expire set # allkeys-random -> remove a random key, any key # volatile-ttl -> remove the key with the nearest expire time (minor TTL) # noeviction -> don't expire at all, just return an error on write operations # # Note: with any of the above policies, Redis will return an error on write # operations, when there are not suitable keys for eviction. # # At the date of writing this commands are: set setnx setex append # incr decr rpush lpush rpushx lpushx linsert lset rpoplpush sadd # sinter sinterstore sunion sunionstore sdiff sdiffstore zadd zincrby # zunionstore zinterstore hset hsetnx hmset hincrby incrby decrby # getset mset msetnx exec sort # # The default is: # # maxmemory-policy noeviction maxmemory-policy allkeys-lru
針對設置了過時時間的,使用lru算法 # volatile-lru -> remove the key with an expire set using an LRU algorithm 針對所有key,使用lru算法 # allkeys-lru -> remove any key accordingly to the LRU algorithm 針對設置了過時時間的,隨機刪 # volatile-random -> remove a random key with an expire set 針對所有key,隨機刪 # allkeys-random -> remove a random key, any key 針對設置了過時時間的,立刻要過時的,刪掉 # volatile-ttl -> remove the key with the nearest expire time (minor TTL) 不過時,不能寫了,就報錯 # noeviction -> don't expire at all, just return an error on write operations
在以下結構體中,定義了以下字段:
struct redisServer { ... unsigned long long maxmemory; /* Max number of memory bytes to use */ int maxmemory_policy; /* Policy for key eviction */ int maxmemory_samples; /* Pricision of random sampling */ ... }
當咱們在配置文件中,進入以下配置時,該結構體中幾個字段的值以下:
maxmemory 3mb maxmemory-policy allkeys-lru # maxmemory-samples 5 這個取了默認值
maxmemory_policy爲3,是由於枚舉值爲3:
#define REDIS_MAXMEMORY_VOLATILE_LRU 0 #define REDIS_MAXMEMORY_VOLATILE_TTL 1 #define REDIS_MAXMEMORY_VOLATILE_RANDOM 2 #define REDIS_MAXMEMORY_ALLKEYS_LRU 3 #define REDIS_MAXMEMORY_ALLKEYS_RANDOM 4 #define REDIS_MAXMEMORY_NO_EVICTION 5 #define REDIS_DEFAULT_MAXMEMORY_POLICY REDIS_MAXMEMORY_NO_EVICTION
在處理命令的時候,會調用中的
redis.c processCommand int processCommand(redisClient *c) { /* The QUIT command is handled separately. Normal command procs will * go through checking for replication and QUIT will cause trouble * when FORCE_REPLICATION is enabled and would be implemented in * a regular command proc. */ // 特別處理 quit 命令 void *commandName = c->argv[0]->ptr; redisLog(REDIS_NOTICE, "The server is now processing %s", commandName); if (!strcasecmp(c->argv[0]->ptr, "quit")) { addReply(c, shared.ok); c->flags |= REDIS_CLOSE_AFTER_REPLY; return REDIS_ERR; } /* Now lookup the command and check ASAP about trivial error conditions * such as wrong arity, bad command name and so forth. */ // 1 查找命令,並進行命令合法性檢查,以及命令參數個數檢查 c->cmd = c->lastcmd = lookupCommand(c->argv[0]->ptr); if (!c->cmd) { // 沒找到指定的命令 flagTransaction(c); addReplyErrorFormat(c, "unknown command '%s'", (char *) c->argv[0]->ptr); return REDIS_OK; } /* Check if the user is authenticated */ //2 檢查認證信息 if (server.requirepass && !c->authenticated && c->cmd->proc != authCommand) { flagTransaction(c); addReply(c, shared.noautherr); return REDIS_OK; } /* If cluster is enabled perform the cluster redirection here. * * 3 若是開啓了集羣模式,那麼在這裏進行轉向操做。 * * However we don't perform the redirection if: * * 不過,若是有如下狀況出現,那麼節點不進行轉向: * * 1) The sender of this command is our master. * 命令的發送者是本節點的主節點 * * 2) The command has no key arguments. * 命令沒有 key 參數 */ if (server.cluster_enabled && !(c->flags & REDIS_MASTER) && !(c->cmd->getkeys_proc == NULL && c->cmd->firstkey == 0)) { int hashslot; // 集羣已下線 if (server.cluster->state != REDIS_CLUSTER_OK) { flagTransaction(c); addReplySds(c, sdsnew("-CLUSTERDOWN The cluster is down. Use CLUSTER INFO for more information\r\n")); return REDIS_OK; // 集羣運做正常 } else { int error_code; clusterNode *n = getNodeByQuery(c, c->cmd, c->argv, c->argc, &hashslot, &error_code); // 不能執行多鍵處理命令 if (n == NULL) { flagTransaction(c); if (error_code == REDIS_CLUSTER_REDIR_CROSS_SLOT) { addReplySds(c, sdsnew("-CROSSSLOT Keys in request don't hash to the same slot\r\n")); } else if (error_code == REDIS_CLUSTER_REDIR_UNSTABLE) { /* The request spawns mutliple keys in the same slot, * but the slot is not "stable" currently as there is * a migration or import in progress. */ addReplySds(c, sdsnew("-TRYAGAIN Multiple keys request during rehashing of slot\r\n")); } else { redisPanic("getNodeByQuery() unknown error."); } return REDIS_OK; //3.1 命令針對的槽和鍵不是本節點處理的,進行轉向 } else if (n != server.cluster->myself) { flagTransaction(c); // -<ASK or MOVED> <slot> <ip>:<port> // 例如 -ASK 10086 127.0.0.1:12345 addReplySds(c, sdscatprintf(sdsempty(), "-%s %d %s:%d\r\n", (error_code == REDIS_CLUSTER_REDIR_ASK) ? "ASK" : "MOVED", hashslot, n->ip, n->port)); return REDIS_OK; } // 若是執行到這裏,說明鍵 key 所在的槽由本節點處理 // 或者客戶端執行的是無參數命令 } } /* Handle the maxmemory directive. * * First we try to free some memory if possible (if there are volatile * keys in the dataset). If there are not the only thing we can do * is returning an error. */ //4 若是設置了最大內存,那麼檢查內存是否超過限制,並作相應的操做 if (server.maxmemory) { //4.1 若是內存已超過限制,那麼嘗試經過刪除過時鍵來釋放內存 int retval = freeMemoryIfNeeded(); // 若是即將要執行的命令可能佔用大量內存(REDIS_CMD_DENYOOM) // 而且前面的內存釋放失敗的話 // 那麼向客戶端返回內存錯誤 if ((c->cmd->flags & REDIS_CMD_DENYOOM) && retval == REDIS_ERR) { flagTransaction(c); addReply(c, shared.oomerr); return REDIS_OK; } } ....
接下來,深刻4.1處:
int freeMemoryIfNeeded(void) { size_t mem_used, mem_tofree, mem_freed; int slaves = listLength(server.slaves); /* Remove the size of slaves output buffers and AOF buffer from the * count of used memory. */ // 計算出 Redis 目前佔用的內存總數,但有兩個方面的內存不會計算在內: // 1)從服務器的輸出緩衝區的內存 // 2)AOF 緩衝區的內存 mem_used = zmalloc_used_memory(); if (slaves) { ... } if (server.aof_state != REDIS_AOF_OFF) { mem_used -= sdslen(server.aof_buf); mem_used -= aofRewriteBufferSize(); } /* Check if we are over the memory limit. */ //1 若是目前使用的內存大小比設置的 maxmemory 要小,那麼無須執行進一步操做 if (mem_used <= server.maxmemory) return REDIS_OK; //2 若是佔用內存比 maxmemory 要大,可是 maxmemory 策略爲不淘汰,那麼直接返回 if (server.maxmemory_policy == REDIS_MAXMEMORY_NO_EVICTION) return REDIS_ERR; /* We need to free memory, but policy forbids. */ /* Compute how much memory we need to free. */ // 3 計算須要釋放多少字節的內存 mem_tofree = mem_used - server.maxmemory; // 初始化已釋放內存的字節數爲 0 mem_freed = 0; // 根據 maxmemory 策略, //4 遍歷字典,釋放內存並記錄被釋放內存的字節數 while (mem_freed < mem_tofree) { int j, k, keys_freed = 0; // 遍歷全部字典 for (j = 0; j < server.dbnum; j++) { long bestval = 0; /* just to prevent warning */ sds bestkey = NULL; dictEntry *de; redisDb *db = server.db + j; dict *dict; if (server.maxmemory_policy == REDIS_MAXMEMORY_ALLKEYS_LRU || server.maxmemory_policy == REDIS_MAXMEMORY_ALLKEYS_RANDOM) { // 若是策略是 allkeys-lru 或者 allkeys-random //5 那麼淘汰的目標爲全部數據庫鍵 dict = server.db[j].dict; } else { // 若是策略是 volatile-lru 、 volatile-random 或者 volatile-ttl //6 那麼淘汰的目標爲帶過時時間的數據庫鍵 dict = server.db[j].expires; } /* volatile-random and allkeys-random policy */ // 若是使用的是隨機策略,那麼從目標字典中隨機選出鍵 if (server.maxmemory_policy == REDIS_MAXMEMORY_ALLKEYS_RANDOM || server.maxmemory_policy == REDIS_MAXMEMORY_VOLATILE_RANDOM) { de = dictGetRandomKey(dict); bestkey = dictGetKey(de); } /* volatile-lru and allkeys-lru policy */ //7 else if (server.maxmemory_policy == REDIS_MAXMEMORY_ALLKEYS_LRU || server.maxmemory_policy == REDIS_MAXMEMORY_VOLATILE_LRU) { struct evictionPoolEntry *pool = db->eviction_pool; while (bestkey == NULL) { // 8 evictionPoolPopulate(dict, db->dict, db->eviction_pool); /* Go backward from best to worst element to evict. */ for (k = REDIS_EVICTION_POOL_SIZE - 1; k >= 0; k--) { if (pool[k].key == NULL) continue; // 8.1 de = dictFind(dict, pool[k].key); /* 8.2 Remove the entry from the pool. */ sdsfree(pool[k].key); /* Shift all elements on its right to left. */ memmove(pool + k, pool + k + 1, sizeof(pool[0]) * (REDIS_EVICTION_POOL_SIZE - k - 1)); /* Clear the element on the right which is empty * since we shifted one position to the left. */ pool[REDIS_EVICTION_POOL_SIZE - 1].key = NULL; pool[REDIS_EVICTION_POOL_SIZE - 1].idle = 0; /* If the key exists, is our pick. Otherwise it is * a ghost and we need to try the next element. */ // 8.3 if (de) { bestkey = dictGetKey(de); break; } else { /* Ghost... */ continue; } } } } /* volatile-ttl */ // 策略爲 volatile-ttl ,從一集 sample 鍵中選出過時時間距離當前時間最接近的鍵 else if (server.maxmemory_policy == REDIS_MAXMEMORY_VOLATILE_TTL) { ... } /* Finally remove the selected key. */ // 8.4 刪除被選中的鍵 if (bestkey) { long long delta; robj *keyobj = createStringObject(bestkey, sdslen(bestkey)); propagateExpire(db, keyobj); /* We compute the amount of memory freed by dbDelete() alone. * It is possible that actually the memory needed to propagate * the DEL in AOF and replication link is greater than the one * we are freeing removing the key, but we can't account for * that otherwise we would never exit the loop. * * AOF and Output buffer memory will be freed eventually so * we only care about memory used by the key space. */ // 計算刪除鍵所釋放的內存數量 delta = (long long) zmalloc_used_memory(); dbDelete(db, keyobj); delta -= (long long) zmalloc_used_memory(); mem_freed += delta; // 對淘汰鍵的計數器增一 server.stat_evictedkeys++; notifyKeyspaceEvent(REDIS_NOTIFY_EVICTED, "evicted", keyobj, db->id); decrRefCount(keyobj); keys_freed++; ... } } if (!keys_freed) return REDIS_ERR; /* nothing to free... */ } return REDIS_OK; }
1處,若是目前使用的內存大小比設置的 maxmemory 要小,那麼無須執行進一步操做
2處,若是佔用內存比 maxmemory 要大,可是 maxmemory 策略爲不淘汰,那麼直接返回
3處,計算須要釋放多少字節的內存
4處,遍歷字典,釋放內存並記錄被釋放內存的字節數
5處,若是策略是 allkeys-lru 或者 allkeys-random 那麼淘汰的目標爲全部數據庫鍵
6處,若是策略是 volatile-lru 、 volatile-random 或者 volatile-ttl ,那麼淘汰的目標爲帶過時時間的數據庫鍵
7處,若是使用的是 LRU 策略, 那麼從 sample 鍵中選出 IDLE 時間最長的那個鍵
8處,調用evictionPoolPopulate,該函數在下面講解,該函數的功能是,傳入一個鏈表,即這裏的db->eviction_pool,而後在函數內部,隨機找出n個key,放入傳入的鏈表中,並按照空閒時間排序,空閒最久的,放到最後。
當該函數,返回後,db->eviction_pool這個鏈表裏就存放了咱們要淘汰的key。
8.1處,找到這個key,這個key,在後邊會被刪除
8.2處,下面這一段,從db->eviction_pool將這個已經處理了的key刪掉
8.3處,若是這個key,是存在的,則跳出循環,在後面8.4處,會被刪除
前面咱們看到,在7處,若是爲lru策略,則會進入8處的函數:
evictionPoolPopulate。
該函數的名稱爲:填充(populate)驅逐(eviction)對象池(pool)。驅逐的意思,就是如今達到了maxmemory,沒辦法,只能開始刪除掉一部分元素,來騰空間了,否則新的put類型的命令,根本沒辦法執行。
該方法的大概思路是,使用lru的時候,隨機找n個key,相似於抽樣,而後放到一個鏈表,根據空閒時間排序。
具體看看該方法的實現:
void evictionPoolPopulate(dict *sampledict, dict *keydict, struct evictionPoolEntry *pool) {
其中,傳入的第三個參數,是要被填充的對象,在c語言中,習慣傳入一個入參,而後在函數內部填充或者修改入參對象的屬性。
該屬性,就是前面說的那個鏈表,用來存放收集的隨機的元素,該鏈表中節點的結構以下:
struct evictionPoolEntry { unsigned long long idle; /* Object idle time. */ sds key; /* Key name. */ };
該結構共2個字段,一個存儲key,一個存儲空閒時間。
該鏈表中,共maxmemory-samples個元素,會按照idle時間長短排序,idle時間長的在鏈表尾部,(假設頭在左,尾在右)。
void evictionPoolPopulate(dict *sampledict, dict *keydict, struct evictionPoolEntry *pool) { int j, k, count; dictEntry *_samples[EVICTION_SAMPLES_ARRAY_SIZE]; dictEntry **samples; /* Try to use a static buffer: this function is a big hit... * Note: it was actually measured that this helps. */ if (server.maxmemory_samples <= EVICTION_SAMPLES_ARRAY_SIZE) { samples = _samples; } else { samples = zmalloc(sizeof(samples[0]) * server.maxmemory_samples); } /* 1 Use bulk get by default. */ count = dictGetRandomKeys(sampledict, samples, server.maxmemory_samples); // 2 for (j = 0; j < count; j++) { unsigned long long idle; sds key; robj *o; dictEntry *de; de = samples[j]; key = dictGetKey(de); /* If the dictionary we are sampling from is not the main * dictionary (but the expires one) we need to lookup the key * again in the key dictionary to obtain the value object. */ if (sampledict != keydict) de = dictFind(keydict, key); // 3 o = dictGetVal(de); // 4 idle = estimateObjectIdleTime(o); /* 5 Insert the element inside the pool. * First, find the first empty bucket or the first populated * bucket that has an idle time smaller than our idle time. */ k = 0; while (k < REDIS_EVICTION_POOL_SIZE && pool[k].key && pool[k].idle < idle) k++; ... // 6 pool[k].key = sdsdup(key); pool[k].idle = idle; } if (samples != _samples) zfree(samples); }
1處,獲取 server.maxmemory_samples
個key,這裏是隨機獲取的,(dictGetRandomKeys),這個值,默認值爲5,放到samples中
2處,遍歷返回來的samples
3處,調用以下宏,獲取val
he的類型爲dictEntry:
/* * 哈希表節點 */ typedef struct dictEntry { // 鍵 void *key; // 值 union { // 1 void *val; uint64_t u64; int64_t s64; } v; // 指向下個哈希表節點,造成鏈表 struct dictEntry *next; } dictEntry;
因此,這裏去
Copy robj *o; o = dictGetVal(de);
實際就是獲取其v屬性中的val,(1處):
robj *o; o = dictGetVal(de);
咱們上面3處,看到,獲取的o的類型爲robj。咱們如今看看怎麼計算對象的空閒時長:
/* Given an object returns the min number of milliseconds the object was never * requested, using an approximated LRU algorithm. */ unsigned long long estimateObjectIdleTime(robj *o) { //4.1 獲取系統的當前時間 unsigned long long lruclock = LRU_CLOCK(); // 4.2 if (lruclock >= o->lru) { // 4.3 return (lruclock - o->lru) * REDIS_LRU_CLOCK_RESOLUTION; } else { return (lruclock + (REDIS_LRU_CLOCK_MAX - o->lru)) * REDIS_LRU_CLOCK_RESOLUTION; } }
這裏,4.1處,獲取系統的當前時間;
4.2處,若是系統時間,大於對象的lru時間
4.3處,則用系統時間減去對象的lru時間,再乘以單位,換算爲毫秒,最終返回的單位,爲毫秒(能夠看註釋。)
#define REDIS_LRU_CLOCK_RESOLUTION 1000 /* LRU clock resolution in ms */
5處,這裏拿當前元素,和pool中已經放進去的元素,從第0個開始比較,若是當前元素的idle時長,大於pool中指針0指向的元素,則和pool中索引1的元素比較;直到條件不知足爲止。
這句話意思就是,相似於冒泡,把當前元素一直日後冒,直到idle時長小於被比較的元素爲止。
通過上面的處理後,鏈表中存放了所有的抽樣元素,且ide時間最長的,在最右邊。
前面4處,說到,用系統的當前時間,減去對象的lru時間。
你們看看對象的結構體
typedef struct redisObject { // 類型 unsigned type:4; // 編碼 unsigned encoding:4; //1 對象最後一次被訪問的時間 unsigned lru:REDIS_LRU_BITS; /* lru time (relative to server.lruclock) */ // 引用計數 int refcount; // 指向實際值的指針 void *ptr; } robj;
上面1處,lru屬性,就是用來存儲這個。
robj *createObject(int type, void *ptr) { robj *o = zmalloc(sizeof(*o)); o->type = type; o->encoding = REDIS_ENCODING_RAW; o->ptr = ptr; o->refcount = 1; /*1 Set the LRU to the current lruclock (minutes resolution). */ o->lru = LRU_CLOCK(); return o; }
1處便是。
robj *createEmbeddedStringObject(char *ptr, size_t len) { robj *o = zmalloc(sizeof(robj)+sizeof(struct sdshdr)+len+1); struct sdshdr *sh = (void*)(o+1); o->type = REDIS_STRING; o->encoding = REDIS_ENCODING_EMBSTR; o->ptr = sh+1; o->refcount = 1; // 1 o->lru = LRU_CLOCK(); sh->len = len; sh->free = 0; if (ptr) { memcpy(sh->buf,ptr,len); sh->buf[len] = '\0'; } else { memset(sh->buf,0,len+1); } return o; }
1處便是。
robj *lookupKey(redisDb *db, robj *key) { // 查找鍵空間 dictEntry *de = dictFind(db->dict,key->ptr); // 節點存在 if (de) { // 取出值 robj *val = dictGetVal(de); /* Update the access time for the ageing algorithm. * Don't do it if we have a saving child, as this will trigger * a copy on write madness. */ // 更新時間信息(只在不存在子進程時執行,防止破壞 copy-on-write 機制) if (server.rdb_child_pid == -1 && server.aof_child_pid == -1) // 1 val->lru = LRU_CLOCK(); // 返回值 return val; } else { // 節點不存在 return NULL; } }
1處便是,包括get、set等各類操做,都會刷新該時間。
仔細看下面的堆棧,set的,get同理:
你們有沒有更清楚一些呢?
總的來講,就是,設置了max-memory後,達到該內存限制後,會在處理命令時,檢查是否要進行內存淘汰;若是要淘汰,則根據maxmemory-policy的策略來。
隨機選擇maxmemory-sample個元素,按照空閒時間排序,拉鍊表;挨個挨個清除。