基於Redis緩存幾十萬條記錄的快速模糊檢索的功能實現（c#）

在開發一套大型的信息系統中，發現不少功能須要按公司查詢，各個模塊在實現時都是直接查數據庫進行模糊檢索，雖然對錶進行向各個應用的庫中進行了同步，可是在使用中發現，模糊檢索公司時仍是比較卡，原始的查詢數據庫實現方法：

            var organizeManager = new BaseOrganizeManager(DbHelperFactory.GetHelper(BaseSystemInfo.BusinessDbType, BaseSystemInfo.BusinessDbConnection));
            if (string.IsNullOrEmpty(key))
            {
                return null;
            }
            key = DbLogic.SqlSafe(key);
            var where = "(" + BaseOrganizeEntity.FieldFullName + " LIKE'%" + key
                + "%' OR " + BaseOrganizeEntity.FieldCode + " LIKE '%" + key + "%' OR "
                + BaseOrganizeEntity.FieldSimpleSpelling + " LIKE '%" + key + "%' OR "
                + BaseOrganizeEntity.FieldQuickQuery + " LIKE '%" + key + "%') AND "
                + BaseOrganizeEntity.FieldDeletionStateCode + " = 0 ";
            var items = organizeManager.GetList2<BaseOrganizeEntity>(where, 20, " Id desc");
            
            if (returnId)
            {
                returnList = items.Select(t => new SuggestEntity(t.FullName, t.Id)).ToList();
                returnList = items.Select(t => new SuggestEntity(t.FullName + " " + t.Code, t.Id)).ToList();
            }
            else
            {
                if (showCode)
                {
                    returnList = items.Select(t => new SuggestEntity(t.FullName, t.Code)).ToList();
                }
                else
                {
                    returnList = items.Select(t => new SuggestEntity(t.FullName, t.FullName)).ToList();
                }
            }
            return returnList;

爲了提升用戶體驗，對公司的模糊檢索使用了Redis緩存，按照如下原則：

1：讀取最少的數據；
2：網絡傳輸最少的數據；
3：全部的可能性都預先緩存；
4：緩存過時後的搜索；
5：數據庫的讀取壓力減小；
6：緩存是否重複；
7：緩存最少的內容，佔用最少的內存；
8：全部的應用共享一份緩存數據；

下面來開始具體實現

一、Redis緩存輔助類建立：

    public sealed partial class PooledRedisHelper
    {
        // 數據庫
        public static int InitialDb = 0;
        private static PooledRedisClientManager instance = null;
        public static PooledRedisClientManager Instance
        {
            get
            {
                if (instance == null)
                {
                    instance = new PooledRedisClientManager(new string[] { BaseSystemInfo.RedisHosts });
                }

                return instance;
            }
        }

        static PooledRedisHelper()
        {
        }

        public static IRedisClient GetClient()
        {
            return Instance.GetClient();
        }
    }

二、緩存數據預熱，具體實現時天天緩存更新一次便可（爲了更精準找到檢索的內容，加入到緩存時按公司名字的順序緩存）

        public static void CachePreheatingSpelling()
        {
            BaseOrganizeManager organizeManager = new Business.BaseOrganizeManager();
            organizeManager.SelectFields = BaseOrganizeEntity.FieldId
                + ", " + BaseOrganizeEntity.FieldCode
                + ", " + BaseOrganizeEntity.FieldFullName;
            List<KeyValuePair<string, object>> parameters = new List<KeyValuePair<string, object>>();
            parameters.Add(new KeyValuePair<string, object>(BaseOrganizeEntity.FieldDeletionStateCode, 0));
            using (var redisClient = PooledRedisHelper.GetClient())
            {
                //using (IDataReader dataReader = organizeManager.ExecuteReader(parameters, BaseOrganizeEntity.FieldId))
                using (IDataReader dataReader = organizeManager.ExecuteReader(parameters, BaseOrganizeEntity.FieldFullName))
                {
                    while (dataReader.Read())
                    {
                        string id = dataReader[BaseOrganizeEntity.FieldId].ToString();
                        string code = dataReader[BaseOrganizeEntity.FieldCode].ToString();
                        string fullName = dataReader[BaseOrganizeEntity.FieldFullName].ToString();
                        string simpleSpelling = dataReader[BaseOrganizeEntity.FieldSimpleSpelling].ToString();
                        string quickQuery = dataReader[BaseOrganizeEntity.FieldQuickQuery].ToString();

                        string organize = id + ";" + code + ";" + fullName;
                        string key = string.Empty;
                        for (int i = 1; i <= code.Length; i++)
                        {
                            key = code.Substring(0, i).ToLower();
                            redisClient.AddItemToSortedSet(key, organize, double.Parse(id));
                            redisClient.ExpireEntryAt(key, DateTime.Now.AddDays(1));
                        }
                        for (int i = 1; i <= fullName.Length; i++)
                        {
                            key = fullName.Substring(0, i).ToLower();
                            redisClient.AddItemToSortedSet(key, organize, double.Parse(id));
                            redisClient.ExpireEntryAt(key, DateTime.Now.AddDays(1));
                        }
                        for (int i = 1; i <= simpleSpelling.Length; i++)
                        {
                            key = simpleSpelling.Substring(0, i).ToLower();
                            redisClient.AddItemToSortedSet(key, organize, double.Parse(id));
                            redisClient.ExpireEntryAt(key, DateTime.Now.AddDays(1));
                        }
                        for (int i = 1; i <= quickQuery.Length; i++)
                        {
                            key = quickQuery.Substring(0, i).ToLower();
                            redisClient.AddItemToSortedSet(key, organize, double.Parse(id));
                            redisClient.ExpireEntryAt(key, DateTime.Now.AddDays(1));
                        }
                    }
                }
            }
        }

緩存中的數據是怎樣的呢，從上面代碼中咱們能夠看出，涉及檢索公司的全部可能性（按名稱、拼音、Code檢索）組合都進行了緩存，讀取時直接按Key取數據，比緩存整張表再查詢速度要快不少。

下圖是緩存數據的局部截圖，這是測試環境緩存的數據，一共166440條記錄。

三、模糊檢索Redis緩存公司數據方法，檢索時按Key取數據（ List<string> list = redisClient.GetRangeFromSortedList(key, 0, topLimit);）

        /// <summary>
        /// Redis中檢索公司
        /// </summary>
        /// <param name="key"></param>
        /// <param name="returnId"></param>
        /// <param name="showCode"></param>
        /// <param name="topLimit"></param>
        /// <returns></returns>
        public static List<KeyValuePair<string, string>> GetOrganizesByKey(string key, bool returnId = true, bool showCode = false, int topLimit = 20)
        {
            List<KeyValuePair<string, string>> result = new List<KeyValuePair<string, string>>();
            using (var redisClient = PooledRedisHelper.GetClient())
            {
                List<string> list = redisClient.GetRangeFromSortedList(key, 0, topLimit);
                if (list != null)
                {
                    for (int i = 0; i < list.Count; i++)
                    {
                        string[] organize = list[i].Split(';');
                        string id = organize[0];
                        string code = organize[1];
                        string fullName = organize[2];
                        if (returnId)
                        {
                            if (showCode)
                            {
                                result.Add(new KeyValuePair<string, string>(id, fullName + " " + code));
                            }
                            else
                            {
                                result.Add(new KeyValuePair<string, string>(id, fullName));
                            }
                        }
                        else
                        {
                            if (showCode)
                            {
                                result.Add(new KeyValuePair<string, string>(code, fullName + " " + code));
                            }
                            else
                            {
                                result.Add(new KeyValuePair<string, string>(code, fullName));
                            }
                        }
                        
                    }
                }
            }

            return result;
        }

四、前端在應用時，直接調用底層這個方法，再封裝成選擇下拉框須要的數據便可，如：

        /// <summary>
        /// 公司檢索 從redis中查詢
        /// </summary>
        /// <param name="key"></param>
        /// <param name="category"></param>
        /// <param name="userInfo"></param>
        /// <param name="showCode"></param>
        /// <param name="returnId"></param>
        /// <param name="topLimit"></param>
        /// <returns></returns>
        public List<SuggestEntity> GetOrganizesByKey(string key, string category, BaseUserInfo userInfo, bool returnId = true, bool showCode = false, int topLimit = 100)
        {
            List<SuggestEntity> returnList = new List<SuggestEntity>();
            List<KeyValuePair<string, string>> list = BaseOrganizeManager.GetOrganizesByKey(key, returnId, showCode, 100);
            foreach (var organize in list)
            {
                returnList.Add(new SuggestEntity(organize.Value, organize.Key));
            }
            return returnList;
        }

前端模糊檢索時，渲染選擇的效果

 

使用這種方式後，比之前檢索速度，效率都快了不少，用戶體驗也好了。