02-NLP-02-樸素貝葉斯與應用

時間 2019-12-02

標籤 nlp 樸素貝葉應用简体版

原文原文鏈接

樸素貝葉斯與應用

貝葉斯理論簡單回顧

在咱們有一大堆樣本（包含特徵和類別）的時候，咱們很是容易經過統計獲得 $p (特征 | 类别)$ html

你們又都很熟悉下述公式：python

p (x) p (y | x) = p (y) p (x | y)

因此作一個小小的變換git

p (特 徵) p (類 別 | 特 徵) = p (類 別) p (特 徵

p (類 別 | 特 徵) = p ( 類 別 ) p ( 特 徵 | 類

獨立假設

看起來很簡單，但實際上，你的特徵多是不少維的windows

p (f e a t u r e s | c l a s s) = p (f 0, f 1, \dots, f n | c)

就算是2個維度吧，能夠簡單寫成app

p (f 0, f 1 | c) = p (f 1 | c, f 0) p (f 0 | c)

這時候咱們加一個特別牛逼的假設：特徵之間是獨立的。這樣就獲得了dom

p (f 0, f 1 | c) = p (f 1 | c) p (f 0 | c)

其實也就是：機器學習

p (f 0, f 1, \dots, f n | c) = Π n i p (f i | c)

貝葉斯分類器

OK，回到機器學習，其實咱們就是對每一個類別計算一個機率 $p (c_{i})$ 函數

p (c l a s s i | f 0, f 1, \dots, f n) = p ( c l a s s i ) p (

文本分類問題

下面咱們來看一個文本分類問題，經典的新聞主題分類，用樸素貝葉斯怎麼作。學習

In [2]:

 
      #coding: utf-8
import os
import time
import random
import jieba  #處理中文
#import nltk  #處理英文
import sklearn
from sklearn.naive_bayes import MultinomialNB
import numpy as np
import pylab as pl
import matplotlib.pyplot as plt 
     

In [4]:

 
      #粗暴的詞去重
def make_word_set(words_file):
    words_set = set()   #利用集合來遍歷訓練集，收集全部出現的詞
    with open(words_file, 'r') as fp:
        for line in fp.readlines():
            word = line.strip().decode("utf-8")
            if len(word)>0 and word not in words_set: # 去重
                words_set.add(word)
    return words_set 
     

In [5]:

 
      # 文本處理，也就是樣本生成過程
#搜狗的數據集是將每個類別的若干個文本都放在了子文件夾中

def text_processing(folder_path, test_size=0.2):
    folder_list = os.listdir(folder_path)
    data_list = []
    class_list = []

    # 遍歷文件夾，並將每一個文件夾中內容讀取出來
    for folder in folder_list:
        new_folder_path = os.path.join(folder_path, folder)
        files = os.listdir(new_folder_path)
        # 讀取文件
        j = 1
        for file in files:
            if j > 100: # 怕內存爆掉，只取100個樣本文件，你能夠註釋掉取完
                break
            with open(os.path.join(new_folder_path, file), 'r') as fp:
               raw = fp.read()
            ## 是的，隨處可見的jieba中文分詞
            jieba.enable_parallel(4) # 開啓並行分詞模式，參數爲並行進程數，不支持windows
            word_cut = jieba.cut(raw, cut_all=False) # cut進行分詞，精確模式，返回的結構是一個可迭代的genertor
            word_list = list(word_cut) # genertor轉化爲list，每一個詞unicode格式
            jieba.disable_parallel() # 關閉並行分詞模式
            
            data_list.append(word_list) #訓練集list（數據列表）
            class_list.append(folder.decode('utf-8')) #類別列表
            j += 1
    
    ## 粗暴地劃分訓練集和測試集：取前80%的做爲訓練集，20%做爲測試集
    data_class_list = zip(data_list, class_list)
    random.shuffle(data_class_list)
    index = int(len(data_class_list)*test_size)+1
    train_list = data_class_list[index:]
    test_list = data_class_list[:index]
    train_data_list, train_class_list = zip(*train_list)
    test_data_list, test_class_list = zip(*test_list)
    
    #其實能夠用sklearn自帶的部分作
    #train_data_list, test_data_list, train_class_list, test_class_list = sklearn.cross_validation.train_test_split(data_list, class_list, test_size=test_size)
    

    # 統計詞頻放入all_words_dict
    #手動劃分，遍歷全部詞，遇到了就把它對應的頻次加1，
    #便於統計完成以後，在字典中將比較重要的詞放在比較前面，這樣在後續處理的時候能夠不用到全部的詞只用部分出現頻次較高的詞
    all_words_dict = {}
    for word_list in train_data_list:
        for word in word_list:
            if all_words_dict.has_key(word):
                all_words_dict[word] += 1
            else:
                all_words_dict[word] = 1

    # key函數利用詞頻進行降序排序
    all_words_tuple_list = sorted(all_words_dict.items(), key=lambda f:f[1], reverse=True) # 內建函數sorted參數需爲list
    all_words_list = list(zip(*all_words_tuple_list)[0])

    return all_words_list, train_data_list, test_data_list, train_class_list, test_class_list 
     

In [6]:

 
      def words_dict(all_words_list, deleteN, stopwords_set=set()):
    # 選取特徵詞，從上述all_words_dict列表中抽取1000
    feature_words = []
    n = 1
    for t in range(deleteN, len(all_words_list), 1):
        if n > 1000: # feature_words的維度1000
            break
            
        if not all_words_list[t].isdigit() and all_words_list[t] not in stopwords_set and 1<len(all_words_list[t])<5:
            feature_words.append(all_words_list[t])
            n += 1
    return feature_words 
     

In [7]:

 
      # 文本特徵
def text_features(train_data_list, test_data_list, feature_words, flag='nltk'):
    def text_features(text, feature_words):
        text_words = set(text)
        ## -----------------------------------------------------------------------------------
        if flag == 'nltk':
            ## nltk特徵 dict
            features = {word:1 if word in text_words else 0 for word in feature_words}
        elif flag == 'sklearn':
            ## sklearn特徵 list，若是在詞袋中出現就記錄1，最後獲得一個很長條的列表
　　　　　　　　#對每一個樣本都生成這樣一個列表來表示每一個詞的出現與否
            features = [1 if word in text_words else 0 for word in feature_words]
        else:
            features = []
        ## -----------------------------------------------------------------------------------
        return features
    train_feature_list = [text_features(text, feature_words) for text in train_data_list]
    test_feature_list = [text_features(text, feature_words) for text in test_data_list]
    return train_feature_list, test_feature_list       
     

In [8]:

# 分類，同時輸出準確率等
def text_classifier(train_feature_list, test_feature_list, train_class_list, test_class_list, flag='nltk'):
    ## -----------------------------------------------------------------------------------
    if flag == 'nltk':
        ## 使用nltk分類器
        train_flist = zip(train_feature_list, train_class_list)
        test_flist = zip(test_feature_list, test_class_list)
        classifier = nltk.classify.NaiveBayesClassifier.train(train_flist)
        test_accuracy = nltk.classify.accuracy(classifier, test_flist)
    elif flag == 'sklearn':
        ## sklearn分類器
        classifier = MultinomialNB().fit(train_feature_list, train_class_list)
        test_accuracy = classifier.score(test_feature_list, test_class_list)
    else:
        test_accuracy = []
    return test_accuracy

In [13]:

print "start"

## 文本預處理
folder_path = './Database/SogouC/Sample'   #因爲這裏取得只是一個樣本數據，不是完整的語料庫，因此準確類不是很高，只有50-70%的準確率
all_words_list, train_data_list, test_data_list, train_class_list, test_class_list = text_processing(folder_path, test_size=0.2)

# 生成stopwords_set
stopwords_file = './stopwords_cn.txt'
stopwords_set = make_word_set(stopwords_file)

## 文本特徵提取和分類
# flag = 'nltk'
flag = 'sklearn'
deleteNs = range(0, 1000, 20)
test_accuracy_list = []
for deleteN in deleteNs:
    # feature_words = words_dict(all_words_list, deleteN)
    feature_words = words_dict(all_words_list, deleteN, stopwords_set)
    train_feature_list, test_feature_list = text_features(train_data_list, test_data_list, feature_words, flag)
    test_accuracy = text_classifier(train_feature_list, test_feature_list, train_class_list, test_class_list, flag)
    test_accuracy_list.append(test_accuracy)
print test_accuracy_list

# 結果評價
#plt.figure()
plt.plot(deleteNs, test_accuracy_list)
plt.title('Relationship of deleteNs and test_accuracy')
plt.xlabel('deleteNs')
plt.ylabel('test_accuracy')
plt.show()
#plt.savefig('result.png')

print "finished"

start
[0.63157894736842102, 0.63157894736842102, 0.63157894736842102, 0.57894736842105265, 0.63157894736842102, 0.57894736842105265, 0.57894736842105265, 0.57894736842105265, 0.57894736842105265, 0.57894736842105265, 0.63157894736842102, 0.63157894736842102, 0.57894736842105265, 0.57894736842105265, 0.57894736842105265, 0.57894736842105265, 0.57894736842105265, 0.57894736842105265, 0.57894736842105265, 0.57894736842105265, 0.57894736842105265, 0.63157894736842102, 0.68421052631578949, 0.63157894736842102, 0.63157894736842102, 0.57894736842105265, 0.52631578947368418, 0.63157894736842102, 0.63157894736842102, 0.57894736842105265, 0.57894736842105265, 0.57894736842105265, 0.57894736842105265, 0.63157894736842102, 0.57894736842105265, 0.68421052631578949, 0.57894736842105265, 0.63157894736842102, 0.63157894736842102, 0.63157894736842102, 0.63157894736842102, 0.63157894736842102, 0.68421052631578949, 0.63157894736842102, 0.57894736842105265, 0.57894736842105265, 0.57894736842105265, 0.63157894736842102, 0.63157894736842102, 0.63157894736842102] finished

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。