天然語言處理----處理原始文本

時間 2019-11-26

標籤天然語言處理原始文本简体版

原文原文鏈接

本文主要介紹編程訪問網絡文本的幾種方式。html

1. 訪問網絡資源python

>>> from urllib import urlopen
>>> url='http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.astype.html'
>>> raw=urlopen(url).read()
>>> type(raw)
<type 'str'>
>>> len(raw)
16429
>>> raw[:75]
'\n\n<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"\n  "http://'

View Code

若是Python沒法正確自動檢測出Internet代理，可使用下面方法手動指定。正則表達式

>>> proxies={'http': 'http://www.someproxy.com:3128'}
>>> raw=urlopen(url, proxies=proxies).read（）

2. 訪問博客編程

在Universal Feed Parser的第三方python庫的幫助下，能夠訪問博客的內容。網絡

>>> import feedparser
>>> llog=feedparser.parse('http://weibo.com/ttarticle/p/show?id=2309404116343489194022')
>>> llog.keys()
['feed', 'status', 'version', 'encoding', 'bozo', 'headers', 'href', 'namespaces', 'entries', 'bozo_exception']
>>> type(llog['feed'])
<class 'feedparser.FeedParserDict'>
>>> llog['feed'].keys()
['meta', 'summary']
>>> llog['feed']['meta']
{'content': u'text/html; charset=gb2312', 'http-equiv': u'Content-type'}
>>> llog['feed']['summary']
u'<span id="message"></span>\n\n&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;&amp;'

View Code

3. 處理htmlide

通常有三種方式：正則匹配， nltk.clean_html(), BeautifulSoup. 正則表達式比較繁瑣，而nltk.clean_html（）如今已經不支持了，比較簡單經常使用的是用BeautifulSoup包。ui

from bs4 import BeautifulSoup

html_doc=''' 
    <html><head><title>The Document's story</title></head>
    <html><head><title>The Dormouse's story</title></head>
    <body>
    <p class="title"><b>The Dormouse's story</b></p>

    <p class="story">Once upon a time there were three little sisters; and their names were
    <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
    <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
    <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
        and they lived at the bottom of a well.</p>

    <p class="story">...</p>
    </body></html>
'''
soup = BeautifulSoup(html_doc, 'html.parser')
content=soup.get_text()
print content

運行結果以下：url

runfile('D:/my project/e_book/XXMLV-2/4.Python_代碼/test.py', wdir='D:/my project/e_book/XXMLV-2/4.Python_代碼')

The Document's story
The Dormouse's story

The Dormouse's story
Once upon a time there were three little sisters; and their names were
    Elsie,
    Lacie and
    Tillie;
        and they lived at the bottom of a well.
...

相關標籤/搜索

每日一句

每一个你不满意的现在，都有一个你没有努力的曾经。