算法-DFA算法-敏感詞過濾算法（OC、Swift、Python）
前言
前段時間，公司的IM SDK想作敏感詞過濾，可是後端的小夥伴《比較忙》，在開產品需求會的時候想把敏感詞過濾放到前端，讓iOS、安卓本身搞，可是前端小夥伴寫了一個方法來檢測一段文本，耗時【一兩秒】鍾並且比較耗CPU，這樣確定不行的，最後後端小夥伴妥協了，把敏感詞過濾放到後端了。前端
通常的思路多是遍歷敏感詞庫，而後把一段文字的敏感詞過濾掉，可是針對比較大的詞庫時（好比咱們的敏感詞庫10萬），這樣很是耗時和耗內存，在電腦上還能跑跑，可是在手機上分分鐘鐘被系統殺死掉，這樣確定是不行的，這裏就用到一種DFA算法。node
可是使用了DFA算法，十萬的敏感詞庫過濾一句話只須要【0.434510秒】！python
2019-10-23 14:34:08.230918+0800 DFAFilterDemo[4728:4650502] message == 小明罵小王是個王八蛋，小王罵小明是個王八羔子！
2019-10-23 14:34:08.232972+0800 DFAFilterDemo[4728:4650502] result == 小明罵小王是個***，小王罵小明是個王八羔子！
2019-10-23 14:34:08.316380+0800 DFAFilterDemo[4728:4650502] 總共耗時: 0.434510
複製代碼
DFA算法
簡介
何謂DFA，它的全稱是Deterministic Finite Automaton，即肯定有窮自動機；其特徵爲：有一個有限狀態集合和一些從一個狀態通向另外一個狀態的邊，每條邊上標記有一個符號，其中一個狀態是初態，某些狀態是終態。但不一樣於不肯定的有限自動機，DFA中不會有從同一狀態出發的兩條邊標誌有相同的符號；DFA算法的核心是創建了以敏感詞爲基礎的許多敏感詞樹。git
描述
咱們先把敏感詞庫拆分解析成一個」敏感詞樹「，咱們以敏感詞」王八蛋「和」王八羔子「爲例： github
代碼
OC代碼
//
// DFAFilter.m
// DFAFilterDemo
//
// Created by 張福傑 on 2019/10/22.
// Copyright © 2019 張福傑. All rights reserved.
//
#import "DFAFilter.h"
@interface DFAFilter ()
@property (nonatomic,strong) NSMutableDictionary *keyword_chains;
@property (nonatomic, copy) NSString *delimit;
@end
@implementation DFAFilter
- (instancetype)init{
if(self == [super init]){
_delimit = @"\x00";
}
return self;
}
/// 讀取解析敏感詞
- (void)parseSensitiveWords:(NSString *)path{
if(path == nil){
path = [[NSBundle mainBundle] pathForResource:@"sensitive_words" ofType:@"txt"];
}
NSString *content = [[NSString alloc] initWithContentsOfFile:path encoding:NSUTF8StringEncoding error:nil];
NSArray *keyWordList = [content componentsSeparatedByString:@","];
for (NSString *keyword in keyWordList) {
[self addSensitiveWords:keyword];
}
NSLog(@"keyword_chains == %@",self.keyword_chains);
}
- (void)addSensitiveWords:(NSString *)keyword{
keyword = keyword.lowercaseString;
keyword = [keyword stringByTrimmingCharactersInSet:[NSCharacterSet whitespaceAndNewlineCharacterSet]];
NSMutableDictionary *node = self.keyword_chains;
for (int i = 0; i < keyword.length; i ++) {
NSString *word = [keyword substringWithRange:NSMakeRange(i, 1)];
if (node[word] == nil) {
node[word] = [NSMutableDictionary dictionary];
}
node = node[word];
}
//敏感詞最後一個字符標識
[node setValue:@0 forKey:self.delimit];
}
- (NSString *)filterSensitiveWords:(NSString *)message replaceKey:(NSString *)replaceKey{
replaceKey = replaceKey == nil ? @"*" : replaceKey;
message = message.lowercaseString;
NSMutableArray *retArray = [[NSMutableArray alloc] init];
NSInteger start = 0;
while (start < message.length) {
NSMutableDictionary *level = self.keyword_chains.mutableCopy;
NSInteger step_ins = 0;
NSString *message_chars = [message substringWithRange:NSMakeRange(start, message.length - start)];
for(int i = 0; i < message_chars.length; i++){
NSString *chars_i = [message_chars substringWithRange:NSMakeRange(i, 1)];
if([level.allKeys containsObject:chars_i]){
step_ins += 1;
NSDictionary *level_char_dict = level[chars_i];
if(![level_char_dict.allKeys containsObject:self.delimit]){
level = level_char_dict.mutableCopy;
}else{
NSMutableString *ret_str = [[NSMutableString alloc] init];
for(int i = 0; i < step_ins; i++){
[ret_str appendString:replaceKey];
}
[retArray addObject:ret_str];
start += (step_ins - 1);
break;
}
}else{
[retArray addObject:[NSString stringWithFormat:@"%C",[message characterAtIndex:start]]];
break;
}
}
start ++;
}
return [retArray componentsJoinedByString:@""];
}
- (NSMutableDictionary *)keyword_chains{
if(_keyword_chains == nil){
_keyword_chains = [[NSMutableDictionary alloc] initWithDictionary:@{}];
}
return _keyword_chains;
}
@end
複製代碼
Swift代碼
//
// DFAFilter.swift
// DFAFilterDemo
//
// Created by 張福傑 on 2019/10/23.
// Copyright © 2019 張福傑. All rights reserved.
//
import UIKit
class DFAFilter: NSObject {
lazy var keyword_chains: NSMutableDictionary = {
let dict = NSMutableDictionary()
return dict
}()
lazy var delimit: String = "\\x00";
/// 讀取敏感詞
func parseSensitiveWords() -> Void {
let path = Bundle.main.path(forResource: "sensitive_words", ofType: "txt");
let url = URL(fileURLWithPath: path!)
do {
let data = try Data(contentsOf: url)
let content: String = String(data: data, encoding: String.Encoding.utf8)!
let keyWordList = content.components(separatedBy: ",")
for keyword in keyWordList {
addSensitiveWords(keyword)
}
} catch let error as Error? {
print(error?.localizedDescription as Any)
}
}
/// 添加敏感詞到敏感詞樹
func addSensitiveWords(_ keyword: String) -> Void {
let keyword: String = keyword.lowercased().trimmingCharacters(in: .whitespacesAndNewlines)
var node = self.keyword_chains
if keyword.count <= 0{
return
}
for index in 0...keyword.count - 1 {
let index0 = keyword.index(keyword.startIndex, offsetBy: index)
let index1 = keyword.index(keyword.startIndex, offsetBy: index + 1)
let word = keyword[index0..<index1]
if node[word] == nil{
node[word] = NSMutableDictionary()
}
node = node[word] as! NSMutableDictionary
}
node[self.delimit] = 0
}
/// 開始過濾敏感詞
func filterSensitiveWords(_ message: String, replaceKey: String) -> String {
let replaceKey = replaceKey.count > 0 ? replaceKey : "*"
let message = message.lowercased()
let retArray: NSMutableArray = NSMutableArray()
var start = 0
while start < message.count {
var level: NSMutableDictionary = self.keyword_chains.mutableCopy() as! NSMutableDictionary
var step_ins = 0
let message_chars = getChar(message, startIndex: start, endIndex: message.count)
for index in 0...message_chars.count - 1 {
let chars_i = getChar(message_chars, startIndex: index, endIndex: index + 1)
if level[chars_i] != nil{
step_ins += 1
let level_char_dict: NSDictionary = level[chars_i] as! NSDictionary
if level_char_dict[self.delimit] == nil{
level = level_char_dict.mutableCopy() as! NSMutableDictionary
}else{
var ret_str = ""
for _ in 0...step_ins - 1 {
ret_str += replaceKey
}
retArray.add(ret_str)
start += (step_ins - 1)
break
}
}else{
let word = getChar(message, startIndex: start, endIndex: start + 1)
retArray.add(word)
break
}
}
start += 1
}
return retArray.componentsJoined(by: "")
}
func getChar(_ message: String, startIndex: NSInteger, endIndex: NSInteger) -> String {
let index0 = message.index(message.startIndex, offsetBy: startIndex)
let index1 = message.index(message.startIndex, offsetBy: endIndex)
let word = message[index0..<index1]
return String(word)
}
}
複製代碼
Python代碼
# -*- coding: utf-8 -*-
# @Author: zhangfujie
# @Date: 2019/10/22
# @Last Modified by: zhangfujie
# @Last Modified time: 2019/10/22
# @ ---------- DFA過濾算 ----------
import time
time1 = time.time()
class DFAFilter(object):
"""DFA過濾算法"""
def __init__(self):
super(DFAFilter, self).__init__()
self.keyword_chains = {}
self.delimit = '\x00'
# 讀取解析敏感詞
def parseSensitiveWords(self, path):
ropen = open(path,'r')
text = ropen.read()
keyWordList = text.split(',')
for keyword in keyWordList:
self.addSensitiveWords(str(keyword).strip())
# 生成敏感詞樹
def addSensitiveWords(self, keyword):
keyword = keyword.lower()
chars = keyword.strip()
if not chars:
return
level = self.keyword_chains
for i in range(len(chars)):
if chars[i] in level:
level = level[chars[i]]
else:
if not isinstance(level, dict):
break
for j in range(i, len(chars)):
level[chars[j]] = {}
last_level, last_char = level, chars[j]
level = level[chars[j]]
last_level[last_char] = {self.delimit: 0}
break
if i == len(chars) - 1:
level[self.delimit] = 0
# 過濾敏感詞
def filterSensitiveWords(self, message, repl="*"):
message = message.lower()
ret = []
start = 0
while start < len(message):
level = self.keyword_chains
step_ins = 0
message_chars = message[start:]
for char in message_chars:
if char in level:
step_ins += 1
if self.delimit not in level[char]:
level = level[char]
else:
ret.append(repl * step_ins)
start += step_ins - 1
break
else:
ret.append(message[start])
break
start += 1
return ''.join(ret)
if __name__ == "__main__":
gfw = DFAFilter()
gfw.parseSensitiveWords('shieldwords.txt')
text = "小明罵小王是個王八蛋，小王罵小明是個王八羔子！"
result = gfw.filterSensitiveWords(text)
print(result)
time2 = time.time()
print('總共耗時:' + str(time2 - time1) + 's')
複製代碼
結束語
demo下載地址: gitee.com/zfj1128/DFA… 過往大佬喜歡的給個小星星吧！算法
歡迎各位大神提出寶貴的意見和建議，也歡迎你們進羣交流365152048！swift
| CSDN博客 | zfj1128.blog.csdn.net |
|---|---|
| GITEE主頁 | gitee.com/zfj1128 |
| GITHUB主頁 | github.com/zfjsyqk |
相關文章
- 1. 算法-DFA算法-敏感詞過濾算法（OC、Swift、Python）
- 2. DFA算法實現敏感詞過濾
- 3. 敏感詞過濾-DFA算法
- 4. Java過濾敏感詞算法--DFA算法
- 5. 敏感詞過濾算法
- 6. Java實現敏感詞過濾 - DFA算法
- 7. 基於DFA算法實現的敏感詞過濾
- 8. DFA算法過濾敏感詞，替換爲*
- 9. java實現敏感詞過濾（DFA算法）
- 10. Java利用DFA算法實現敏感詞過濾
- 更多相關文章...
- • Swift 運算符 - Swift 教程
- • PHP 運算符 - PHP教程
- • 算法總結-廣度優先算法
- • 算法總結-深度優先算法
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
相關文章