給 Magento 2 添加緩存層的分析與嘗試

時間  2019-11-12
標籤 magento 添加 緩存 分析 嘗試 简体版
原文   原文鏈接

雖然黑色星期五有驚無險的過去了, 可是 Magento 2 社區版沒法讀寫分離這個限制, 始終是懸在整個網站上的一把利劍。php

我以前嘗試過給 Magento 2 寫一個 MySQL 讀寫分離的插件, 在深刻研究了 Magento 2 的數據庫訪問層後, 發現經過一個簡單的插件, 想作到讀寫分離基本上是不可能的。Magento 2 社區版讀寫數據庫的邏輯裏, 混雜着大量的 Magento 1的代碼和邏輯, 沒法在修改少許代碼的前提下作到讀寫分離, 後來忙着作網站上的各類需求， 因而讀寫分離就擱置了。sql

此次黑五, 整個項目的性能瓶頸就是 MySQL, 流量上來以後, 應用服務器負載基本保持不變, 而數據庫服務器負載卻翻了3倍多, 並且是在數據庫服務器提早升級了硬件配置的基礎上。因此我以爲 Magento 2 的數據庫層必需要優化一下, 既然無法作讀寫分離, 那能不能加個緩存層呢？將絕大多數讀取操做轉移到緩存層去, 理論上數據庫的負載會相應降低。數據庫

要想改的代碼最少, 就得找對地方。 Magento 2 的數據庫 Adapter 是 Magento\Framework\DB\Adapter\Pdo\Mysql 類, 該類繼承自 Zend_Db_Adapter_Abstract數組

全部獲取數據的方法以下:緩存

Zend_Db_Adapter_Abstract::fetchAll($sql, $bind = array(), $fetchMode = null)

Zend_Db_Adapter_Abstract::fetchAssoc($sql, $bind = array())

Zend_Db_Adapter_Abstract::fetchCol($sql, $bind = array())

Zend_Db_Adapter_Abstract::fetchPairs($sql, $bind = array())

Zend_Db_Adapter_Abstract::fetchOne($sql, $bind = array())

Zend_Db_Adapter_Abstract::fetchRow($sql, $bind = array(), $fetchMode = null)

其中, fetchAll() 和 fetchRow() 是用的最多的兩個。服務器

下面以 fetchRow() 爲例, 分析該方案的可行性以及實現方法。ide

/**
 * Fetches the first row of the SQL result.
 * Uses the current fetchMode for the adapter.
 *
 * @param string|Zend_Db_Select $sql An SQL SELECT statement.
 * @param mixed $bind Data to bind into SELECT placeholders.
 * @param mixed                 $fetchMode Override current fetch mode.
 * @return mixed Array, object, or scalar depending on fetch mode.
 */
public function fetchRow($sql, $bind = array(), $fetchMode = null)

經過解析 $sql 對象和 $bind 數組， 能夠獲得精確的、格式化的數據， 包含
1. 數據庫表名
2. 字段鍵值對性能

經過這些數據，能夠構建緩存的鍵(key)和標籤(tag)， 例如：
$cacheKey = table_name::主鍵鍵值對
或者
$cacheKey = table_name::惟一鍵索引鍵值對測試

$cacheTags = [
table_name,
table_name::主鍵鍵值對
table_name::惟一鍵索引鍵值對組1,
table_name::惟一鍵索引鍵值對組2,
…
]fetch

cacheTags 的做用是給緩存分類, 方便後續清理。

有了 $cacheKey, $cacheTags 以後, 就能夠將數據庫查詢的結果保存到緩存中去;

下次再有查詢過來, 先在緩存中查找有無對應的數據, 若是有就直接返回給數據調用方了;

那麼若是數據更新了呢？

數據更新分爲三種: 1. UPDATE, 2. INSERT, 3 DELETE

對於 UPDATE:

/**
 * Updates table rows with specified data based on a WHERE clause.
 *
 * @param  mixed        $table The table to update.
 * @param  array        $bind  Column-value pairs.
 * @param  mixed        $where UPDATE WHERE clause(s).
 * @return int          The number of affected rows.
 * @throws Zend_Db_Adapter_Exception
 */
public function update($table, array $bind, $where = '')

update() 方法接收 3 個參數， 分別是 table_name, 待更新數據鍵值對, where 條件子句。
剛纔咱們在構建 $cacheTags 時， 分別有 table_name、table_name::主鍵鍵值對、table_name::惟一鍵索引鍵值對, table_name 是現成的， 其他兩種tag 須要從 where 子句中解析。 經過解析，最壞狀況是 where 子句未解析到任何鍵值對, 最好狀況是解析到了全部 filed 鍵值對。最壞狀況下， 須要清除 table_name 下的全部緩存數據， 而最好狀況下, 只須要清除一條緩存數據。

對於 INSERT:

/**
 * Inserts a table row with specified data.
 *
 * @param mixed $table The table to insert data into.
 * @param array $bind Column-value pairs.
 * @return int The number of affected rows.
 * @throws Zend_Db_Adapter_Exception
 */
public function insert($table, array $bind)

insert() 方法接收 2 個參數， 分別是 table_name， 待插入數據鍵值對。 因爲新插入的數據根本不存在與緩存中， 因此不須要對緩存進行操做

對於 DELETE:

/**
 * Deletes table rows based on a WHERE clause.
 *
 * @param  mixed        $table The table to update.
 * @param  mixed        $where DELETE WHERE clause(s).
 * @return int          The number of affected rows.
 */
public function delete($table, $where = '')

delete() 方法接收 2 個參數, table_name 和 where 子句， 假如能從 where 子句中解析到主鍵鍵值對 或 惟一鍵索引鍵值對, 就只須要清除一條緩存記錄， 不然須要清除該 table_name 下的全部緩存記錄。

優化效果:
我暫時只是用 ab 測試了 Magento 2 的購物車:

ab -C PHPSESSID=acmsj8q8ld1tvdo77lm5t0dr9b -n 40 -c 5  http://localhost/checkout/cart/

沒有緩存的時候：
test-No-Cache-1:

Requests per second:    1.79 [#/sec] (mean)
Time per request:       2786.478 [ms] (mean)
Time per request:       557.296 [ms] (mean, across all concurrent requests)

Percentage of the requests served within a certain time (ms)
  50%    756
  66%   2064
  75%   5635
  80%   6150
  90%   7632
  95%   8530
  98%   8563
  99%   8563
 100%   8563 (longest request)
 
MySQL 進程的 CPU 佔用率保持在 20% ~ 24%

test-No-Cache-2:

Requests per second:    1.84 [#/sec] (mean)
Time per request:       2720.852 [ms] (mean)
Time per request:       544.170 [ms] (mean, across all concurrent requests)

Percentage of the requests served within a certain time (ms)
  50%    586
  66%   1523
  75%   4036
  80%   5667
  90%  10228
  95%  11621
  98%  12098
  99%  12098
 100%  12098 (longest request)
 
MySQL 進程的 CPU 佔用率保持在 20% ~ 24%

有緩存的時候：
test-With-Cache-1:

Requests per second:    1.99 [#/sec] (mean)
Time per request:       2509.273 [ms] (mean)
Time per request:       501.854 [ms] (mean, across all concurrent requests)

Percentage of the requests served within a certain time (ms)
  50%    489
  66%    511
  75%    574
  80%    637
  90%  19073
  95%  19553
  98%  20063
  99%  20063
 100%  20063 (longest request)
 
MySQL 進程的 CPU 佔用率保持在 5% 左右

test-With-Cache-2:

Requests per second:    2.10 [#/sec] (mean)
Time per request:       2384.145 [ms] (mean)
Time per request:       476.829 [ms] (mean, across all concurrent requests)

Percentage of the requests served within a certain time (ms)
  50%    465
  66%    472
  75%    565
  80%    620
  90%   9509
  95%  18374
  98%  18588
  99%  18588
 100%  18588 (longest request)

MySQL 進程的 CPU 佔用率保持在 5% ~ 7 %

經過上面兩組數據的對比， 很明顯 MySQL 的 CPU 佔用率有了大幅度降低(從 20% 降低到 5%), 可見增長一個緩存層對下降 MySQL 負載是有效果的。

可是有一個小問題, 在不使用緩存的狀況下, Percentage of the requests served within a certain time 這個值，在 90% 這個點以後, 表現要比有緩存的狀況好, 我猜是大量 unserialize() 操做形成 CPU 資源不夠致使響應緩慢。

通過修改後的 vendor/magento/framework/DB/Adapter/Pdo/Mysql.php:

class Mysql extends \Zend_Db_Adapter_Pdo_Mysql implements AdapterInterface
{

    protected $_cache;


    public function fetchAll($sql, $bind = array(), $fetchMode = null)
    {
        if ($sql instanceof \Zend_Db_Select) {
            /** @var array $from */
            $from = $sql->getPart('from');
            $tableName = current($from)['tableName'];
            $cacheKey = 'FETCH_ALL::' . $tableName . '::' . md5((string)$sql);
            $cache = $this->getCache();
            $data = $cache->load($cacheKey);
            if ($data === false) {
                $data = parent::fetchAll($sql, $bind, $fetchMode);
                $cache->save(serialize($data), $cacheKey, ['FETCH_ALL::' . $tableName], 3600);
            } else {
                $data = @unserialize($data);
            }
        } else {
            $data = parent::fetchAll($sql, $bind, $fetchMode);
        }
        return $data;
    }

    public function fetchRow($sql, $bind = [], $fetchMode = null)
    {
        $cacheIdentifiers = $this->resolveSql($sql, $bind);
        if ($cacheIdentifiers !== false) {
            $cache = $this->getCache()->getFrontend();
            $data = $cache->load($cacheIdentifiers['cacheKey']);

            if ($data === false) {
                $data = parent::fetchRow($sql, $bind, $fetchMode);
                if ($data) {
                    $cache->save(serialize($data), $cacheIdentifiers['cacheKey'], $cacheIdentifiers['cacheTags'], 3600);
                }
            } else {
                $data = @unserialize($data);
            }
        } else {
            $data = parent::fetchRow($sql, $bind, $fetchMode);
        }
        return $data;
    }

    public function update($table, array $bind, $where = '')
    {
        parent::update($table, $bind, $where);
        $cacheKey = $this->resolveUpdate($table, $bind, $where);
        if ($cacheKey === false) {
            $cacheKey = $table;
        }
        $this->getCache()->clean([$cacheKey, 'FETCH_ALL::' . $table]);
    }

    /**
     * @return \Magento\Framework\App\CacheInterface
     */
    private function getCache()
    {
        if ($this->_cache === null) {
            $objectManager = \Magento\Framework\App\ObjectManager::getInstance();
            $this->_cache = $objectManager->get(\Magento\Framework\App\CacheInterface::class);
        }
        return $this->_cache;
    }

    /**
     * @param string|\Zend_Db_Select $sql An SQL SELECT statement.
     * @param mixed $bind Data to bind into SELECT placeholders.
     * @return array
     */
    protected function resolveSql($sql, $bind = array())
    {
        $result = false;
        if ($sql instanceof \Zend_Db_Select) {
            try {
                /** @var array $from */
                $from = $sql->getPart('from');
                $tableName = current($from)['tableName'];
                $where = $sql->getPart('where');

                foreach ($this->getIndexFields($tableName) as $indexFields) {
                    $kv = $this->getKv($indexFields, $where, $bind);
                    if ($kv !== false) {
                        $cacheKey = $tableName . '::' . implode('|', $kv);
                        $cacheTags = [
                            $tableName,
                            $cacheKey
                        ];
                        $result = ['cacheKey' => $cacheKey, 'cacheTags' => $cacheTags];
                    }
                }
            }catch (\Zend_Db_Select_Exception $e) {

            }
        }
        return $result;
    }

    protected function resolveUpdate($tableName, array $bind, $where = '')
    {
        $cacheKey = false;
        if (is_string($where)) {
            $where = [$where];
        }
        foreach ($this->getIndexFields($tableName) as $indexFields) {
            $kv = $this->getKv($indexFields, $where, $bind);
            if ($kv !== false) {
                $cacheKey = $tableName . '::' . implode('|', $kv);
            }
        }
        return $cacheKey;
    }

    protected function getIndexFields($tableName)
    {
        $indexes = $this->getIndexList($tableName);

        $indexFields = [];
        foreach ($indexes as $data) {
            if ($data['INDEX_TYPE'] == 'primary') {
                $indexFields[] = $data['COLUMNS_LIST'];
            } elseif ($data['INDEX_TYPE'] == 'unique') {
                $indexFields[] = $data['COLUMNS_LIST'];
            }
        }
        return $indexFields;
    }

    protected function getKv($fields, $where, $bind)
    {
        $found = true;
        $kv = [];
        foreach ($fields as $field) {
            $_found = false;

            if (isset($bind[':' . $field])) {   // 在 bind 數組中查找 filed value
                $kv[$field] = $field . '=' .$bind[':' . $field];
                $_found = true;
            } elseif (is_array($where)) {
                foreach ($where as $case) { // 遍歷 where 條件子句， 查找 filed value
                    $matches = [];
                    $preg = sprintf('#%s.*=(.*)#', $field);
                    $_result = preg_match($preg, $case, $matches);
                    if ($_result) {
                        $kv[$field] = $field . '=' .trim($matches[1], ' \')');
                        $_found = true;
                    }
                }
            }

            if (!$_found) { // 其中任一 field 沒找到，
                $found = false;
                break;
            }
        }
        return $found ? $kv : false;
    }
}
相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程