Redis Server.c & AOF & RDB

initServerConfig()

• Read from Config. Provides default values for the members that can be configured by the user via the redis.conf config file.
• Set LRU Clock via (mstime()/LRU_CLOCK_RESOLUTION) & LRU_CLOCK_MAX;.
• Set RDB save rules via appendServerSaveParams().

initSentinel()

• Initiate the server in sentinel mode if necessary.

initServer()

• Create shared objects via createSharedObjects().

  • Keywords -ERR, +OK, etc.
  • Popular commands lpush, rpush, etc.
  • 0-9999 shared integers.

• Increase the open file limit according to server.maxclients+CONFIG_MIN_RESERVED_FDS via adjustOpenFilesLimit().

• Create eventLoop via aeCreateEventLoop(server.maxclients+CONFIG_FDSET_INCR).

• Listen to TCP port via anetTcpServer().

• Listen to unix socket via anetUnixServer().

• Initiate all server.dbnum DBs, each with five tables.

  • dict
  • expires
  • blocking_keys
  • ready_keys
  • watched_keys

• Initiate evictionPool which is an array of evictionPoolEntry via evictionPoolAlloc().

  evict.c
  #define EVPOOL_SIZE 16
  #define EVPOOL_CACHED_SDS_SIZE 255
  struct evictionPoolEntry {
      unsigned long long idle;    /* Object idle time (inverse frequency for LFU) */
      sds key;                    /* Key name. */
      sds cached;                 /* Cached SDS object for key name. */
      int dbid;                   /* Key DB number. */
  };
  複製代碼

• Attach serviceCron to eventLoop via aeCreateTimeEvent(server.el, 1, serverCron, NULL, NULL).

• Attach acceptTcpHandler to eventLoop via aeCreateFileEvent(server.el, server.ipfd[j], AE_READABLE, acceptTcpHandler,NULL).

• Attach acceptUnixHandler to eventLoop via aeCreateFileEvent(server.el,server.sofd,AE_READABLE, acceptUnixHandler,NULL).

• Open AOF file if AOF is enabled.

loadDataFromDisk()

• If AOF is enabled, load only from AOF via loadAppendOnlyFile() in aof.c.
  1. Create a fake client to inject all the commands stored in AOF file.
  2. Load the AOF FILE and check intergrity that a AOF file must start with 5 characters: REDIS.
  3. The remaining file is just commands in REPL format. e.g: *5\r\n$5\r\nRPUSH\r\n$7\r\nNUMBERS\r\n$3\r\nONE\r\n$3\r\nTWO\r\n$5\r\nTHREE\r\n.
  4. Create a char array buffer and read the AOF file until EOF or malformat occurs.
  5. Buf[0] must be '*'. Buf[1] must be an integer, which is argc of the command.
  6. For i in argc, Buf[0] must be '$'; Buf[1] must be a number(could be long), which is the size of argv[i]. Read Buf[1] length of data from Buf and save it into argv[i]
  7. argv[0] is the command name. Search it in CommandTable via lookupCommand(argv[0]->ptr).
  8. Execute the command via cmd->proc(fakeClient)
• If AOF is not enabled or encounter exception in reading AOF, load RDB via rdbLoad() in rdb.c
  1. Open RDB file as a rio stream
  2. Read the first 9 characters from stream. First 5 must match REDIS, following 4 is the RDB_VERSION. If version is too old, RDB recovery can't be used.
  3. Get current LRUClock to handle expired data from RDB file.
  4. Read next byte from the stream, and map this byte to OP_CODE. See below for OP_CODE details.
  5. If OP_CODE = RDB_OPCODE_SELECTDB, store current db number based on the value after.
  6. If OP_CODE = RDB_OPCODE_RESIZEDB. expand the db size based on the value after.
  7. If OP_CODE in RDB_OPCODE_EXPIRETIME | RDB_OPCODE_EXPIRETIME_MS, store key expiration info based on the value after.
  8. If OP_CODE in RDB_OPCODE_FREQ | RDB_OPCODE_IDLE, store key eviction score based on the value after.
  9. If OP_CODE = RDB_OPCODE_EOF, jump out of the loop.
  10. If OP_CODE in RDB_TYPE_* family, read the next string as the key of a key-value pair and the following chunk of data as the value. Chunk's size and format depend on which RDB_TYPE it is.
  11. Set the key and value pair in the correct db with correct expire info and correct eviction score. Go back to step 4.
  12. If RDB_VERSION > 5 and CRC checksum is enabled on RDB, read the last part of the stream after RDB_OPCODE_EOF. Abort if mismatch.
• RDB file format:

  RDB = |REDIS|RDB_VERSION|DB_SECTION|...|DB_SECTION|EOF|CRC_CHECKSUM|
  DB_SECTION = |RDB_OPCODE_SELECTDB|DB_NUM|RDB_OPCODE_RESIZEDB|DB_SIZE|KEY_VALUE|...|KEY_VALUE|
  KEY_VALUE = |OPTIONAL_KEY_EXPIRE_INFO|OPTIONAL_KEY_LRU_LFU_INFO|RDB_TYPE|KEY|VALUE|
  OPTIONAL_KEY_EXPIRE_INFO = |RDB_OPCODE_EXPIRETIME|EXPIRE_IN_SEC| or |RDB_OPCODE_EXPIRETIME_MS|EXPIRE_IN_MS|
  OPTIONAL_KEY_LRU_LFU_INFO = |RDB_OPCODE_FREQ|LFU_SCORE| or |RDB_OPCODE_IDLE|LRU_SCORE|
  RDB_TYPE = |RDB_TYPE_STRING|RDB_TYPE_LIST|RDB_TYPE_SET|RDB_TYPE_ZSET|RDB_TYPE_HASH|RDB_TYPE_HASH|RDB_TYPE_ZSET_2|
  複製代碼

  • OP_CODE in RDB, each one byte:
    • RDB_OPCODE_EOF
      This means the end of RDB file, exit.
    • RDB_OPCODE_SELECTDB
      This means the following int64 is a db number. Read the value and store it such that Redis will write to the correct DB later on.
    • RDB_OPCODE_RESIZEDB
      This means the following int64 is a db size. Read the value and expand the corresponding DB's db->dict and db->expire to avoid unnecessary rehash during key injection.
    • RDB_OPCODE_EXPIRETIME
      This means the following int32 is an expiration time of a key in second. Read the value and store it such that Redis will set the key with correct expiration time.
    • RDB_OPCODE_EXPIRETIME_MS
      This means the following int64 is an expiration time of a key in millisecond. Read the value and store it such that Redis will set the key with correct expiration time.
    • RDB_OPCODE_FREQ
      This means the following byte is an LFU score of a key in sacle of 0-255. Read the value and store it such that Redis could handle the key correctly during eviction if server.maxmemory_policy & MAXMEMORY_FLAG_LFU.
    • RDB_OPCODE_IDLE
      This means the following int64 is an LRU score of a key. Read the value and store it such that Redis could handle the key correctly during eviction if server.maxmemory_policy & MAXMEMORY_FLAG_LRU.

aeMain()

• This is an infinite while loop until !eventLoop->stop is false.
• From all the timing event, find the nearest one and the calculate the time delta between next fire time and now.
• Call I/O multiplexing api with timeout of the delta from previous step via aeApiPoll(eventLoop, tvp). API being used is platform dependent.
• If any of events got returned, process them one by one. The processing order of a ready event is based on fe->mask & AE_BARRIER. If AE_BARRIER is set, process write event first. Else, read event first.
• Process time events via processTimeEvents(eventLoop).