mysql內存管理

 1 內存管理結構

 mysql有本身的內存申請和釋放機制

 mysql層有mem_root

 innodb層有mem_heap,mem_pool,buf_pool

 它們的結構圖以下

 

 

 

2 mem_root

mem_root是mysql層的動態內存管理

 

typedef struct st_used_mem
{                   /* struct for once_alloc (block) */
  struct st_used_mem *next;       /* Next block in use */
  unsigned int    left;           /* memory left in block  */
  unsigned int    size;           /* size of block */
} USED_MEM;

typedef struct st_mem_root
{
  USED_MEM *free;                  /* blocks with free memory in it */
  USED_MEM *used;                  /* blocks almost without free memory */
  USED_MEM *pre_alloc;             /* preallocated block */
  /* if block have less memory it will be put in 'used' list */
  size_t min_malloc;
  size_t block_size;               /* initial block size */
  unsigned int block_num;          /* allocated blocks counter */
  /* 
     first free block in queue test counter (if it exceed 
     MAX_BLOCK_USAGE_BEFORE_DROP block will be dropped in 'used' list)
  */
  unsigned int first_block_usage;

  void (*error_handler)(void);
} MEM_ROOT;

mem_root主要由兩個鏈表組成free和used。

free：存放有空閒的block，其中的block可能部分被使用。

used：存放已使用的block，其中block可能有少許空閒.

block_size：初始塊大小,後面分配的塊可能比block_size大。

pre_alloc：一直指向初始化時分配的block，改block開始在free鏈表中，隨着內存的不斷申請，改block可能會存在於used鏈表中

 

2.1 init_alloc_root

初始化一個block放入free鏈表，pre_alloc指向這個鏈表

2.2 alloc_root

首先從free鏈表中查找，若是有空間合適的block，則直接使用，若該block剩餘空間小於min_malloc，則會放入used鏈表頭部。若是沒有空間合適的block，則新分配block放入free鏈表尾部。

2.3 reset_root_defaults

1沒有初始化的mem_root不須要reset

知足if (!mem_root->pre_alloc || mem_root->pre_alloc->size != size)才reset

 

2 只釋放徹底沒有使用的空閒塊。

 

2.4 free_root

MY_MARK_BLOCKS_FREE:釋放全部塊

MY_KEEP_PREALLOC：只保留pre_alloc塊

 

3 mem_pool

內存池。innodb早期本身實現的一套夥伴算法內存管理系統，由參數innodb_use_sys_malloc 控制開啓和關閉，默認關閉，即直接從操做系統申請和釋放內存.參見http://docs.oracle.com/cd/E17952_01/refman-5.5-en/innodb-performance-use_sys_malloc.html

mem_pool_t

 

/** Data structure for a memory pool. The space is allocated using the buddy

algorithm, where free list i contains areas of size 2 to power i. */

struct mem_pool_t{

    byte*       buf;        /*!< memory pool */

    ulint       size;       /*!< memory common pool size */

    ulint       reserved;   /*!< amount of currently allocated

                    memory */

    ib_mutex_t      mutex;      /*!< mutex protecting this struct */

    UT_LIST_BASE_NODE_T(mem_area_t)

            free_list[64];  /*!< lists of free memory areas: an

                    area is put to the list whose number

                    is the 2-logarithm of the area size */

};

/** Memory area header */

struct mem_area_t{

    ulint       size_and_free;  /*!< memory area size is obtained by

                    anding with ~MEM_AREA_FREE; area in

                    a free list if ANDing with

                    MEM_AREA_FREE results in nonzero */

    UT_LIST_NODE_T(mem_area_t)

            free_list;  /*!< free list node */

};

buf:一片連續的內存區域

mem_area_t：內存塊, size_and_free標記內存塊的大小和是否已使用。

free_list[64]： 64個桶，每一個桶編號爲0,1,2,3,…,i;每一個桶依次存放2ⁱ大小的內存塊。

 

3.1初始化

mem_pool_create

在服務啓動時調用一次

如初始化一個19字節的內存池

19=100011=2⁴+2¹+2⁰

 3.2 mem_area_alloc

void*

mem_area_alloc(

/*===========*/

    ulint*      psize,  /*!< in: requested size in bytes; for optimum

                space usage, the size should be a power of 2

                minus MEM_AREA_EXTRA_SIZE;

                out: allocated size in bytes (greater than

                or equal to the requested size) */

    mem_pool_t* pool)

從pool中申請內存塊，psize<=2ⁱ,i爲知足此條件的最小值。當第i箇中存在空閒時，直接取出。當第i箇中不存在空閒塊時，從i+1個桶中切割一半到第i個桶中，這是一個遞歸的過程。具體實如今mem_pool_fill_free_list；

以申請大小爲3的內存爲例，（爲了說明方便，這裏不考慮塊中MEM_AREA_EXTRA_SIZE的空間）

申請的獲得大小爲4的塊，這是內存池中的結構以下

 

3.3 mem_pool_free

釋放一個塊到pool中，將塊放入對應大小的桶i中，若是桶i中存在本身的夥伴（和本身相鄰的內存塊，可能在左邊也可能在右邊），則合併放入i+1的桶中，此過程遞歸，直到沒有能夠合併的夥伴。

 

找本身夥伴的函數以下

mem_area_t*

mem_area_get_buddy(

/*===============*/

    mem_area_t* area,   /*!< in: memory area */

    ulint       size,   /*!< in: memory area size */

    mem_pool_t* pool)   /*!< in: memory pool */

{

    mem_area_t* buddy;

 

    ut_ad(size != 0);

 

    if (((((byte*) area) - pool->buf) % (2 * size)) == 0) {

 

        /* The buddy is in a higher address */

 

        buddy = (mem_area_t*)(((byte*) area) + size);

 

        if ((((byte*) buddy) - pool->buf) + size > pool->size) {

 

            /* The buddy is not wholly contained in the pool:

            there is no buddy */

 

            buddy = NULL;

        }

    } else {

        /* The buddy is in a lower address; NOTE that area cannot

        be at the pool lower end, because then we would end up to

        the upper branch in this if-clause: the remainder would be

        0 */

 

        buddy = (mem_area_t*)(((byte*) area) - size);

    }

 

    return(buddy);

}

 

找x的夥伴：

1 首先肯定x的夥伴在高位仍是在低位，x左邊的塊大小都>=x的大小，且x左邊的塊大小=2ⁱ*sizeof(x) (i>0);。

若是x的夥伴是y,則(c-a)%(2*sizeof(x))=sizeof(x) != 0

若是x的夥伴是z,則(c-a)%(2*sizeof(x))= 0

  

須要說明的是：在桶中的塊都是空閒塊。

 

以釋放剛申請得的大小爲4的塊爲例，釋放後，內存結構以下，回到了初始狀態。

 

3.4 mem_pool_free

在服務關閉時調用一次

釋放buf和mem_pool結構便可

mem_pool_free(

/*==========*/

    mem_pool_t* pool)   /*!< in, own: memory pool */

{

    ut_free(pool->buf);

    ut_free(pool);

}

 

 

4 buf_pool

 

塊/頁緩存池。

4.1buf_pool_init

Innodb_buffer_pool_instances指定了bp的個數，innodb_buffer_pool_size指定全部bp的總大小。

buf_pool_init在服務啓動時調用一次

 bp初始化主要是申請內存，並初始化free鏈表。每一個buf_page_t的大小爲innodb_page_size.

UT_LIST_BASE_NODE_T(buf_page_t) free;

4.2 buf_block_alloc

從free鏈表頭部摘出

不放入LRU鏈表

4.3 buf_block_free

放回free鏈表頭部

 

4.4 buffer_pool_free

在服務關閉時調用一次

5 mem_heap

/** The info structure stored at the beginning of a heap block */

struct mem_block_info_t {

    ulint   magic_n;/* magic number for debugging */

    char    file_name[8];/* file name where the mem heap was created */

    ulint   line;   /*!< line number where the mem heap was created */

    UT_LIST_BASE_NODE_T(mem_block_t) base; /* In the first block in the

            the list this is the base node of the list of blocks;

            in subsequent blocks this is undefined */

    UT_LIST_NODE_T(mem_block_t) list; /* This contains pointers to next

            and prev in the list. The first block allocated

            to the heap is also the first block in this list,

            though it also contains the base node of the list. */

    ulint   len;    /*!< physical length of this block in bytes */

    ulint   total_size; /*!< physical length in bytes of all blocks

            in the heap. This is defined only in the base

            node and is set to ULINT_UNDEFINED in others. */

    ulint   type;   /*!< type of heap: MEM_HEAP_DYNAMIC, or

            MEM_HEAP_BUF possibly ORed to MEM_HEAP_BTR_SEARCH */

    ulint   free;   /*!< offset in bytes of the first free position for

            user data in the block */

    ulint   start;  /*!< the value of the struct field 'free' at the

            creation of the block */

#ifndef UNIV_HOTBACKUP

    void*   free_block;

            /* if the MEM_HEAP_BTR_SEARCH bit is set in type,

            and this is the heap root, this can contain an

            allocated buffer frame, which can be appended as a

            free block to the heap, if we need more space;

            otherwise, this is NULL */

    void*   buf_block;

            /* if this block has been allocated from the buffer

            pool, this contains the buf_block_t handle;

            otherwise, this is NULL */

#endif /* !UNIV_HOTBACKUP */

#ifdef MEM_PERIODIC_CHECK

    UT_LIST_NODE_T(mem_block_t) mem_block_list;

            /* List of all mem blocks allocated; protected

            by the mem_comm_pool mutex */

#endif

}; 

 innodb絕大多數內存申請和釋放都是在mem_heap上進行的

 mem_heap其實是一個內存塊鏈表，內存塊大小依次增加，至少是兩倍的增加。

 

5.1 mem_heap_create

初始內存塊鏈表，根據參數初始化內存塊，此時base鏈表只有一個內存塊

   

 if (heap && heap->magic_n != MEM_BLOCK_MAGIC_N) {

        mem_analyze_corruption(heap);

    }

 

    /* In dynamic allocation, calculate the size: block header + data. */

    len = MEM_BLOCK_HEADER_SIZE + MEM_SPACE_NEEDED(n);

 

#ifndef UNIV_HOTBACKUP

    if (type == MEM_HEAP_DYNAMIC || len < UNIV_PAGE_SIZE / 2) {

 

        ut_ad(type == MEM_HEAP_DYNAMIC || n <= MEM_MAX_ALLOC_IN_BUF);

 

        block = static_cast<mem_block_t*>(

            mem_area_alloc(&len, mem_comm_pool));

    } else {

        len = UNIV_PAGE_SIZE;

 

        if ((type & MEM_HEAP_BTR_SEARCH) && heap) {

            /* We cannot allocate the block from the

            buffer pool, but must get the free block from

            the heap header free block field */

 

            buf_block = static_cast<buf_block_t*>(heap->free_block);

            heap->free_block = NULL;

 

            if (UNIV_UNLIKELY(!buf_block)) {

 

                return(NULL);

            }

        } else {

            buf_block = buf_block_alloc(NULL);

        }

 

        block = (mem_block_t*) buf_block->frame;

    }

 

    ut_ad(block);

    block->buf_block = buf_block;

    block->free_block = NULL;

#else /* !UNIV_HOTBACKUP */

    len = MEM_BLOCK_HEADER_SIZE + MEM_SPACE_NEEDED(n);

    block = ut_malloc(len);

    ut_ad(block);

#endif /* !UNIV_HOTBACKUP */

 

 以上代碼能夠看出

Mem_heap的內存來源能夠有三種

1 mem_pool 的mem_area_alloc

2 buffer_pool的buf_block_alloc

3 ut_malloc ->malloc

 

5.2 mem_heap_alloc

從base鏈表最後一個內存塊分配內存，不夠則新分配一個較大內存塊，放入base鏈表的最後。

5.3 mem_heap_free_heap_top

void

mem_heap_free_heap_top(

/*===================*/

    mem_heap_t* heap,   /*!< in: heap from which to free */

    byte*       old_top)/*!< in: pointer to old top of heap */

 

 釋放base鏈表從old_top到最後的內存塊，old_top不釋放

5.4 mem_heap_free_heap_top

釋放base鏈表最後一個內存塊

5.5 mem_heap_free

釋放base鏈表全部內存塊