python天然語言處理——學習筆記：Chapter3糾錯

2017-12-06更新：不少代碼執行結果與書中不一致，是由於python的版本不一致。若是發現有問題，能夠參考英文版：

http://www.nltk.org/book/

第三章，P87有一段處理html的代碼：

>>>raw = nltk.clean_html(html)
>>>tokens = nltk.word_tokenize(raw)
>>>tokens

但是咱們執行會有以下錯誤：

>>> raw = nltk.clean_html(html)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/nltk/util.py", line 356, in clean_html
    raise NotImplementedError ("To remove HTML markup, use BeautifulSoup's get_text() function")
NotImplementedError: To remove HTML markup, use BeautifulSoup's get_text() function

根據官方網站：介紹http://www.nltk.org/_modules/nltk/util.html
def clean_html(html):
raise NotImplementedError ("To remove HTML markup, use BeautifulSoup's get_text() function")
[docs]def clean_url(url):
raise NotImplementedError ("To remove HTML markup, use BeautifulSoup's get_text() function")
網站：http://stackoverflow.com/questions/10524387/beautifulsoup-get-text-does-not-strip-all-tags-and-javascript介紹：
之後的版本，彷佛不支持clean_html()和clean_url()這兩個函數
Support for clean_html and clean_url will be dropped for future versions of nltk. Please use BeautifulSoup for now...it's very unfortunate.
有關處理HTML 的內容，能夠使用http://www.crummy.com/software/BeautifulSoup/上的Beautiful Soup 軟件包。

安裝：sudo pip install beautifulsoup4

以後替換書上的代碼：

from __future__ import division
import nltk, re, pprint
from urllib import urlopen
from bs4 import BeautifulSoup


def read_html():
    url = "http://news.bbc.co.uk/2/hi/health/2284783.stm"
    html = urlopen(url).read()
    soup = BeautifulSoup(html)
    text = soup.get_text()
    print text
    tokens = nltk.word_tokenize(text)
    print tokens


def main():
    read_html()


if __name__ == '__main__':
    main()

上述腳本文件能夠獨立運行，運行結果與書上一致