算法-DFA算法-敏感詞過濾算法（OC、Swift、Python）

時間 2019-11-15

標籤算法 dfa 敏感過濾 swift python 欄目 Swift 简体版

原文原文鏈接

前言

前段時間，公司的IM SDK想作敏感詞過濾，可是後端的小夥伴《比較忙》，在開產品需求會的時候想把敏感詞過濾放到前端，讓iOS、安卓本身搞，可是前端小夥伴寫了一個方法來檢測一段文本，耗時【一兩秒】鍾並且比較耗CPU，這樣確定不行的，最後後端小夥伴妥協了，把敏感詞過濾放到後端了。前端

通常的思路多是遍歷敏感詞庫，而後把一段文字的敏感詞過濾掉，可是針對比較大的詞庫時（好比咱們的敏感詞庫10萬），這樣很是耗時和耗內存，在電腦上還能跑跑，可是在手機上分分鐘鐘被系統殺死掉，這樣確定是不行的，這裏就用到一種DFA算法。node

可是使用了DFA算法，十萬的敏感詞庫過濾一句話只須要【0.434510秒】！python

2019-10-23 14:34:08.230918+0800 DFAFilterDemo[4728:4650502] message == 小明罵小王是個王八蛋，小王罵小明是個王八羔子！
2019-10-23 14:34:08.232972+0800 DFAFilterDemo[4728:4650502] result == 小明罵小王是個***，小王罵小明是個王八羔子！
2019-10-23 14:34:08.316380+0800 DFAFilterDemo[4728:4650502] 總共耗時: 0.434510

DFA算法

簡介

何謂DFA，它的全稱是Deterministic Finite Automaton，即肯定有窮自動機；其特徵爲：有一個有限狀態集合和一些從一個狀態通向另外一個狀態的邊，每條邊上標記有一個符號，其中一個狀態是初態，某些狀態是終態。但不一樣於不肯定的有限自動機，DFA中不會有從同一狀態出發的兩條邊標誌有相同的符號；DFA算法的核心是創建了以敏感詞爲基礎的許多敏感詞樹。git

描述

咱們先把敏感詞庫拆分解析成一個」敏感詞樹「，咱們以敏感詞」王八蛋「和」王八羔子「爲例：

拆成的敏感詞樹以下：
github

代碼

OC代碼

//
//  DFAFilter.m
//  DFAFilterDemo
//
//  Created by 張福傑 on 2019/10/22.
//  Copyright © 2019 張福傑. All rights reserved.
//

#import "DFAFilter.h"

@interface DFAFilter ()

@property (nonatomic,strong) NSMutableDictionary *keyword_chains;
@property (nonatomic,  copy) NSString *delimit;

@end

@implementation DFAFilter

- (instancetype)init{
    if(self == [super init]){
        _delimit = @"\x00";
    }
    return self;
}

/// 讀取解析敏感詞
- (void)parseSensitiveWords:(NSString *)path{
    if(path == nil){
        path = [[NSBundle mainBundle] pathForResource:@"sensitive_words" ofType:@"txt"];
    }
    NSString *content = [[NSString alloc] initWithContentsOfFile:path encoding:NSUTF8StringEncoding error:nil];
    NSArray *keyWordList = [content componentsSeparatedByString:@","];
    for (NSString *keyword in keyWordList) {
        [self addSensitiveWords:keyword];
    }
    
    NSLog(@"keyword_chains == %@",self.keyword_chains);
}

- (void)addSensitiveWords:(NSString *)keyword{
    keyword = keyword.lowercaseString;
    keyword = [keyword stringByTrimmingCharactersInSet:[NSCharacterSet whitespaceAndNewlineCharacterSet]];
    
    NSMutableDictionary *node = self.keyword_chains;
    for (int i = 0; i < keyword.length; i ++) {
        NSString *word = [keyword substringWithRange:NSMakeRange(i, 1)];
        if (node[word] == nil) {
            node[word] = [NSMutableDictionary dictionary];
        }
        node = node[word];
    }
    //敏感詞最後一個字符標識
    [node setValue:@0 forKey:self.delimit];
}

- (NSString *)filterSensitiveWords:(NSString *)message replaceKey:(NSString *)replaceKey{
    replaceKey = replaceKey == nil ? @"*" : replaceKey;
    message = message.lowercaseString;
    NSMutableArray *retArray = [[NSMutableArray alloc] init];
    NSInteger start = 0;
    while (start < message.length) {
        NSMutableDictionary *level = self.keyword_chains.mutableCopy;
        NSInteger step_ins = 0;
        NSString *message_chars = [message substringWithRange:NSMakeRange(start, message.length - start)];
        for(int i = 0; i < message_chars.length; i++){
            NSString *chars_i = [message_chars substringWithRange:NSMakeRange(i, 1)];
            if([level.allKeys containsObject:chars_i]){
                step_ins += 1;
                NSDictionary *level_char_dict = level[chars_i];
                if(![level_char_dict.allKeys containsObject:self.delimit]){
                    level = level_char_dict.mutableCopy;
                }else{
                    NSMutableString *ret_str = [[NSMutableString alloc] init];
                    for(int i = 0; i < step_ins; i++){
                         [ret_str appendString:replaceKey];
                    }
                    [retArray addObject:ret_str];
                    start += (step_ins - 1);
                    break;
                }
            }else{
                [retArray addObject:[NSString stringWithFormat:@"%C",[message characterAtIndex:start]]];
                break;
            }
        }
        start ++;
    }
    return [retArray componentsJoinedByString:@""];
}

- (NSMutableDictionary *)keyword_chains{
    if(_keyword_chains == nil){
        _keyword_chains = [[NSMutableDictionary alloc] initWithDictionary:@{}];
    }
    return _keyword_chains;
}

@end

Swift代碼

//
//  DFAFilter.swift
//  DFAFilterDemo
//
//  Created by 張福傑 on 2019/10/23.
//  Copyright © 2019 張福傑. All rights reserved.
//

import UIKit

class DFAFilter: NSObject {
    lazy var keyword_chains: NSMutableDictionary = {
        let dict = NSMutableDictionary()
        return dict
    }()
    
    lazy var delimit: String = "\\x00";
    
    /// 讀取敏感詞
    func parseSensitiveWords() -> Void {
        let path = Bundle.main.path(forResource: "sensitive_words", ofType: "txt");
        let url = URL(fileURLWithPath: path!)
        do {
            let data = try Data(contentsOf: url)
            let content: String = String(data: data, encoding: String.Encoding.utf8)!
            let keyWordList = content.components(separatedBy: ",")
            for keyword in keyWordList {
                addSensitiveWords(keyword)
            }
            
        } catch let error as Error? {
            print(error?.localizedDescription as Any)
        }
    }
    
    /// 添加敏感詞到敏感詞樹
    func addSensitiveWords(_ keyword: String) -> Void {
        let keyword: String = keyword.lowercased().trimmingCharacters(in: .whitespacesAndNewlines)
        var node = self.keyword_chains
        if keyword.count <= 0{
            return
        }
        
        for index in 0...keyword.count - 1 {
            let index0 = keyword.index(keyword.startIndex, offsetBy: index)
            let index1 = keyword.index(keyword.startIndex, offsetBy: index + 1)
            let word = keyword[index0..<index1]
            if node[word] == nil{
                node[word] = NSMutableDictionary()
            }
            node = node[word] as! NSMutableDictionary
        }
        node[self.delimit] = 0
    }
    
    /// 開始過濾敏感詞
    func filterSensitiveWords(_ message: String, replaceKey: String) -> String {
        let replaceKey = replaceKey.count > 0 ? replaceKey : "*"
        let message = message.lowercased()
        let retArray: NSMutableArray = NSMutableArray()
        var start = 0
        while start < message.count {
            var level: NSMutableDictionary = self.keyword_chains.mutableCopy() as! NSMutableDictionary
            var step_ins = 0
            let message_chars = getChar(message, startIndex: start, endIndex: message.count)
            for index in 0...message_chars.count - 1 {
                let chars_i = getChar(message_chars, startIndex: index, endIndex: index + 1)
                if level[chars_i] != nil{
                    step_ins += 1
                    let level_char_dict: NSDictionary = level[chars_i] as! NSDictionary
                    if level_char_dict[self.delimit] == nil{
                        level = level_char_dict.mutableCopy() as! NSMutableDictionary
                    }else{
                        var ret_str = ""
                        for _ in 0...step_ins - 1 {
                            ret_str += replaceKey
                        }
                        retArray.add(ret_str)
                        start += (step_ins - 1)
                        break
                    }
                }else{
                    let word = getChar(message, startIndex: start, endIndex: start + 1)
                    retArray.add(word)
                    break
                }
            }
            start += 1
        }

        return retArray.componentsJoined(by: "")
    }
    
    func getChar(_ message: String, startIndex: NSInteger, endIndex: NSInteger) -> String {
        let index0 = message.index(message.startIndex, offsetBy: startIndex)
        let index1 = message.index(message.startIndex, offsetBy: endIndex)
        let word = message[index0..<index1]
        return String(word)
    }

}

Python代碼

# -*- coding: utf-8 -*-
# @Author: zhangfujie
# @Date:   2019/10/22
# @Last Modified by:   zhangfujie
# @Last Modified time: 2019/10/22
# @ ---------- DFA過濾算 ---------- 
import time
time1 = time.time()

class DFAFilter(object):
    """DFA過濾算法"""
    def __init__(self):
        super(DFAFilter, self).__init__()
        self.keyword_chains = {}
        self.delimit = '\x00'

    # 讀取解析敏感詞
    def parseSensitiveWords(self, path):
        ropen = open(path,'r')
        text = ropen.read()
        keyWordList = text.split(',')
        for keyword in keyWordList:
            self.addSensitiveWords(str(keyword).strip())

    # 生成敏感詞樹
    def addSensitiveWords(self, keyword):
        keyword = keyword.lower()
        chars = keyword.strip()
        if not chars:
            return
        level = self.keyword_chains
        for i in range(len(chars)):
            if chars[i] in level:
                level = level[chars[i]]
            else:
                if not isinstance(level, dict):
                    break
                for j in range(i, len(chars)):
                    level[chars[j]] = {}

                    last_level, last_char = level, chars[j]

                    level = level[chars[j]]
                last_level[last_char] = {self.delimit: 0}
                break
            if i == len(chars) - 1:
                level[self.delimit] = 0

    # 過濾敏感詞
    def filterSensitiveWords(self, message, repl="*"):
        message = message.lower()
        ret = []
        start = 0
        while start < len(message):
            level = self.keyword_chains
            step_ins = 0
            message_chars = message[start:]
            for char in message_chars:
                if char in level:
                    step_ins += 1
                    if self.delimit not in level[char]:
                        level = level[char]
                    else:
                        ret.append(repl * step_ins)
                        start += step_ins - 1
                        break
                else:
                    ret.append(message[start])
                    break
            start += 1

        return ''.join(ret)


if __name__ == "__main__":
    gfw = DFAFilter()
    gfw.parseSensitiveWords('shieldwords.txt')
    text = "小明罵小王是個王八蛋，小王罵小明是個王八羔子！"
    result = gfw.filterSensitiveWords(text)
    print(result)
    time2 = time.time()
    print('總共耗時:' + str(time2 - time1) + 's')