萌新向Python數據分析及數據挖掘 第二章 pandas 第一節 pandas使用基礎Q&A 1-15
這是油管上的一個帥哥的網課地址以下 https://www.youtube.com/watch?v=yzIMircGU5I&list=PL5-da3qGB5ICCsgW1MxlZ0Hq8LL5U3u9yhtml
Python pandas Q&A video series by Data School
YouTube playlist and GitHub repository
Table of contents
- What is pandas?
- How do I read a tabular data file into pandas?
- How do I select a pandas Series from a DataFrame?
- Why do some pandas commands end with parentheses (and others don't)?
- How do I rename columns in a pandas DataFrame?
- How do I remove columns from a pandas DataFrame?
- How do I sort a pandas DataFrame or a Series?
- How do I filter rows of a pandas DataFrame by column value?
- How do I apply multiple filter criteria to a pandas DataFrame?
- Your pandas questions answered!
- How do I use the "axis" parameter in pandas?
- How do I use string methods in pandas?
- How do I change the data type of a pandas Series?
- When should I use a "groupby" in pandas?
- How do I explore a pandas Series?
- How do I handle missing values in pandas?
- What do I need to know about the pandas index? (Part 1)
- What do I need to know about the pandas index? (Part 2)
- How do I select multiple rows and columns from a pandas DataFrame?
- When should I use the "inplace" parameter in pandas?
- How do I make my pandas DataFrame smaller and faster?
- How do I use pandas with scikit-learn to create Kaggle submissions?
- More of your pandas questions answered!
- How do I create dummy variables in pandas?
- How do I work with dates and times in pandas?
- How do I find and remove duplicate rows in pandas?
- How do I avoid a SettingWithCopyWarning in pandas?
- How do I change display options in pandas?
- How do I create a pandas DataFrame from another object?
- How do I apply a function to a pandas Series or DataFrame?
In [1]:
# 傳統方式
import pandas as pd
1. What is pandas? (video)
2. How do I read a tabular data file into pandas? (video)
In [2]:
# 直接從URL中讀取Chipotle訂單的數據集，並將結果存儲在數據庫中
url1 = "https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/chipotle.tsv"
#定義地址
orders =pd.read_table(url1)#使用read_table()打開
In [3]:
# 檢查前5行
orders.head()
Out[3]:
Documentation for read_tablepython
In [4]:
# 讀取電影評論員的數據集（修改read_table的默認參數值）
user_cols = ['user_id','age','gender','occupation','zipcode']#定義列名
url2 = "https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/u.user"
#定義地址
#users=pd.read_table(url2,sep='|',header=None,names= user_clos,skiprows=2,skipfooter=3)
users = pd.read_table('http://bit.ly/movieusers', sep='|', header=None, names=user_cols)
#加入參數sep 分隔符，header 頭部 標題，names 列名
In [5]:
# 檢查前5行
users.head()
Out[5]:
[Back to top]git
3. How do I select a pandas Series from a DataFrame? (video)
In [6]:
# 將UFO報告的數據集讀入DataFrame
url3 = "https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/ufo.csv"#定義列名
ufo = pd.read_table(url3, sep=',')
In [7]:
# #用read_table打開csv文件，區別是 read_csv直接是用逗號隔開
ufo = pd.read_csv(url3)
In [8]:
# 檢查前5行
ufo.head()
Out[8]:
In [9]:
# #用括號法查看Series
ufo['City']
# #用點法查看Series，要注意 名字裏面有空格或者是python專有字符的時候不能用，可是方便輸入
ufo.City
Out[9]:
括號表示法老是有效，而點表示法有侷限性：github
- 若是系列名稱中有空格，則點符號不起做用
- 若是系列與DataFrame方法或屬性（如'head'或'shape'）具備相同的名稱，則點符號不起做用
- 點符號不能用於定義新series的名
In [10]:
# #這裏的拼接也不能用點的方法
ufo['Location'] = ufo.City + ', ' + ufo.State
ufo.head()
Out[10]:
[Back to top]正則表達式
4. Why do some pandas commands end with parentheses (and others don't)? (video)
In [11]:
# 將頂級IMDb電影的數據集讀入DataFrame
url4="https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/imdb_1000.csv"
movies = pd.read_csv(url4)
#方法以括號結尾，而屬性則沒有：數據庫
In [12]:
# 示例方法：顯示前5行
movies.head()
Out[12]:
In [13]:
#示例方法：計算摘要統計信息
movies.describe()
Out[13]:
In [14]:
movies.describe(include=['object'])
Out[14]:
In [15]:
# 示例屬性：行數和列數
movies.shape
Out[15]:
In [16]:
# 示例屬性：每列的數據類型
movies.dtypes
Out[16]:
In [17]:
# 使用describe方法的可選參數來僅彙總'object'列
movies.describe(include=['object'])
Out[17]:
Documentation for describeapi
[Back to top]app
5. How do I rename columns in a pandas DataFrame? (video)
In [18]:
# 檢查列名稱
ufo.columns
Out[18]:
In [19]:
# 使用'rename'方法重命名其中兩列
ufo.rename(columns={'Colors Reported':'Colors_Reported', 'Shape Reported':'Shape_Reported'}, inplace=True)
ufo.columns
Out[19]:
Documentation for renameide
In [20]:
# 經過覆蓋'columns'屬性替換全部列名
ufo = pd.read_table(url3, sep=',')
ufo_cols = ['city', 'colors reported', 'shape reported', 'state', 'time']
ufo.columns = ufo_cols
ufo.columns
Out[20]:
In [21]:
# 使用'names'參數替換文件讀取過程當中的列名
ufo = pd.read_csv(url3, header=0, names=ufo_cols)
ufo.columns
Out[21]:
Documentation for read_csv函數
In [22]:
ufo.columns = ufo.columns.str.replace(' ', '_') #如何批量修改替換使得列名無空格
ufo.columns
Out[22]:
Documentation for str.replace
6. How do I remove columns from a pandas DataFrame? (video)
In [35]:
ufo = pd.read_table(url3, sep=',')
ufo.head()
Out[35]:
In [37]:
# #axis=1 是縱向，inplace = True：不建立新的對象，直接對原始對象進行修改；
ufo.drop('Colors Reported', axis=1, inplace=True)
ufo.head()
Out[37]:
Documentation for drop
In [38]:
# 一次刪除多個列
ufo.drop(['City', 'State'], axis=1, inplace=True)
ufo.head()
Out[38]:
In [39]:
# 一次刪除多行（axis = 0表示行）
ufo.drop([0, 1], axis=0, inplace=True)
ufo.head()
#刪除4行 按標籤，axis=0 是橫向，默認爲橫向，但建議寫出來
Out[39]:
7. How do I sort a pandas DataFrame or a Series? (video)
In [40]:
movies.head()
Out[40]:
#注意：如下任何排序方法都不會影響基礎數據。 （換句話說，排序是暫時的）。
In [41]:
#排序單個Series
movies.title.sort_values().head()
Out[41]:
In [42]:
# #排序單個Series 倒序
movies.title.sort_values(ascending=False).head()
Out[42]:
Documentation for sort_values for a Series. (Prior to version 0.17, use order instead.)
In [43]:
# #以單個Series排序DataFrame
movies.sort_values('title').head()
Out[43]:
In [44]:
# 改成按降序排序
movies.sort_values('title', ascending=False).head()
Out[44]:
Documentation for sort_values for a DataFrame. (Prior to version 0.17, use sort instead.)
In [45]:
# 首先按'content_rating'，而後按duration'排序DataFrame
movies.sort_values(['content_rating', 'duration']).head()
Out[45]:
Summary of changes to the sorting API in pandas 0.17
8. How do I filter rows of a pandas DataFrame by column value? (video)
In [46]:
movies.head()
Out[46]:
In [47]:
# 檢查行數和列數
movies.shape
Out[47]:
##目標：過濾DataFrame行，僅顯示「持續時間」至少爲200分鐘的電影
In [48]:
#
#先展現一個比較複雜的方法，用一個for循環制造一個和原數據同樣行數，判斷每一行是否符合條件，列表元素均爲boolean
#建立一個列表，其中每一個元素引用一個DataFrame行：若是行知足條件，則返回true，不然返回False
booleans = []
for length in movies.duration:
if length >= 200:
booleans.append(True)
else:
booleans.append(False)
In [49]:
# 確認列表與DataFrame的長度相同
len(booleans)
Out[49]:
In [50]:
# 檢查前五個列表元素
booleans[0:5]
Out[50]:
In [51]:
# 將列表轉換爲Series
is_long = pd.Series(booleans)
is_long.head()
Out[51]:
In [52]:
# 使用帶有布爾Series的括號表示法告訴DataFrame movies[is_long]要顯示哪些行
movies[is_long]
Out[52]:
In [53]:
# 簡化上面的步驟：不須要編寫for循環來建立is_long'
is_long = movies.duration >= 200
movies[is_long]#運用這種寫法，pandas就知道，按照這個series去篩選
# 或等效地，將其寫在一行（無需建立'is_long'對象）
movies[movies.duration >= 200]
Out[53]:
In [54]:
# 從過濾後的DataFrame中選擇「流派」系列
movies[movies.duration >= 200].genre
# 或者等效地，使用'loc'方法
movies.loc[movies.duration >= 200, 'genre']
Out[54]:
Documentation for loc
9. How do I apply multiple filter criteria to a pandas DataFrame? (video)
In [55]:
# read a dataset of top-rated IMDb movies into a DataFrame
movies = pd.read_csv('http://bit.ly/imdbratings')
movies.head()
Out[55]:
In [56]:
# 過濾DataFrame僅顯示「持續時間」至少爲200分鐘的電影
movies[movies.duration >= 200]
Out[56]:
理解邏輯運算符：
and：僅當運算符的兩邊都爲True時才爲真or：若是運算符的任何一側爲True，則爲真
In [57]:
print(True and True)
print(True and False)
print(False and False)
In [58]:
print(True or True)
print(True or False)
print(False or False)
在pandas中指定多個過濾條件的規則：
使用＆而不是和 使用|而不是或 在每一個條件周圍添加括號以指定評估順序
Goal: Further filter the DataFrame of long movies (duration >= 200) to only show movies which also have a 'genre' of 'Drama'
In [59]:
# 使用'＆'運算符指定兩個條件都是必需的
movies[(movies.duration >=200) & (movies.genre == 'Drama')]
Out[59]:
In [60]:
# I不正確：使用'|'運算符會展現長或戲劇的電影
movies[(movies.duration >=200) | (movies.genre == 'Drama')].head()
Out[60]:
##過濾原始數據框以顯示「類型」爲「犯罪」或「戲劇」或「動做」的電影
In [61]:
# 使用'|'運算符指定行能夠匹配三個條件中的任何一個
movies[(movies.genre == 'Crime') | (movies.genre == 'Drama') | (movies.genre == 'Action')].head(10)
# 用isin等效
movies[movies.genre.isin(['Crime', 'Drama', 'Action'])].head(10)
Out[61]:
Documentation for isin
10. Your pandas questions answered! (video)
Question: When reading from a file, how do I read in only a subset of the columns?
In [62]:
url3 = "https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/ufo.csv"#定義列名
ufo = pd.read_csv(url3)#用read_csv打開csv文件
ufo.columns
Out[62]:
In [63]:
# 列名篩選
ufo = pd.read_csv(url3, usecols=['City', 'State'])
# 用位置切片等效
ufo = pd.read_csv(url3, usecols=[0, 4])
ufo.columns
Out[63]:
Question: When reading from a file, how do I read in only a subset of the rows?
In [64]:
# 讀3行數據
ufo = pd.read_csv(url3, nrows=3)
ufo
Out[64]:
Documentation for read_csv
Question: How do I iterate through a Series?
In [65]:
# Series可直接迭代（如列表）
for c in ufo.City:
print(c)
Question: How do I iterate through a DataFrame?
In [66]:
# 可使用各類方法迭代DataFrame
for index, row in ufo.iterrows():
print(index, row.City, row.State)
Documentation for iterrows
Question: How do I drop all non-numeric columns from a DataFrame?
In [67]:
# 將酒精消耗數據集讀入DataFrame，並檢查數據類型
url7= 'https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/drinks.csv'
drinks = pd.read_csv(url7)
drinks.dtypes
Out[67]:
In [68]:
# 僅包含DataFrame中的數字列
import numpy as np
drinks.select_dtypes(include=[np.number]).dtypes
Out[68]:
Documentation for select_dtypes
Question: How do I know whether I should pass an argument as a string or a list?
In [69]:
# 描述全部數字列
drinks.describe()
Out[69]:
In [70]:
# 傳遞字符串'all'來描述全部列
drinks.describe(include='all')
Out[70]:
In [71]:
# 傳遞數據類型列表以僅描述多個類型
drinks.describe(include=['object', 'float64'])
Out[71]:
In [72]:
# 即便您只想描述單個數據類型，也要傳遞一個列表
drinks.describe(include=['object'])
Out[72]:
Documentation for describe
11. How do I use the "axis" parameter in pandas? (video)
In [73]:
url7= 'https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/drinks.csv'
drinks = pd.read_csv(url7)
drinks.head()
Out[73]:
In [74]:
# drop a column (temporarily)
drinks.drop('continent', axis=1).head()
Out[74]:
Documentation for drop
In [75]:
# 刪除一列（暫時）
drinks.drop(2, axis=0).head()
Out[75]:
使用axis參數引用行或列時：
axis 0表示行 axis 1指的是列
In [76]:
# 計算每一個數字列的平均值
drinks.mean()
# 或等效地，明確指定軸
drinks.mean(axis=0)
Out[76]:
Documentation for mean
In [77]:
# 計算每一行的平均值
drinks.mean(axis=1).head()
Out[77]:
使用axis參數執行數學運算時：
- *axis0 *表示操做應「向下移動」行軸
- *axis1 *表示操做應「移過」列軸
In [78]:
# 'index' 等效 axis 0
drinks.mean(axis='index')
Out[78]:
In [79]:
# 'columns' 等效 axis 1
drinks.mean(axis='columns').head()
Out[79]:
12. How do I use string methods in pandas? (video)
In [80]:
url1 = "https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/chipotle.tsv"
#定義地址
orders =pd.read_table(url1)#使用read_table()打開
orders.head()
Out[80]:
In [81]:
# 在Python中訪問字符串方法的經常使用方法
'hello'.upper()
Out[81]:
In [82]:
# spandas Series 的字符串方法經過'str'訪問
orders.item_name.str.upper().head()
Out[82]:
In [83]:
# string方法'contains'檢查子字符串並返回一個布爾Series
orders.item_name.str.contains('Chicken').head()
Out[83]:
In [84]:
# 布爾Series篩選DataFrame
orders[orders.item_name.str.contains('Chicken')].head()
Out[84]:
In [85]:
# 字符串方法能夠連接在一塊兒
orders.choice_description.str.replace('[', '').str.replace(']', '').head()
Out[85]:
In [86]:
# 許多pandas字符串方法支持正則表達式
orders.choice_description.str.replace('[\[\]]', '').head()
Out[86]:
String handling section of the pandas API reference
13. How do I change the data type of a pandas Series? (video)
In [87]:
url7= 'https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/drinks.csv'
drinks = pd.read_csv(url7)
drinks.head()
Out[87]:
In [88]:
# 檢查每一個系列的數據類型
drinks.dtypes
Out[88]:
In [89]:
# 更改現有系列的數據類型
drinks['beer_servings'] = drinks.beer_servings.astype(float)
drinks.dtypes
Out[89]:
Documentation for astype
In [90]:
# 或者，在讀取文件時更改系列的數據類型
drinks = pd.read_csv(url7, dtype={'beer_servings':float})
drinks.dtypes
Out[90]:
In [91]:
orders = pd.read_table(url1)
orders.head()
Out[91]:
In [92]:
# 檢查每一個系列的數據類型
orders.dtypes
Out[92]:
In [93]:
# 將字符串轉換爲數字以進行數學運算
orders.item_price.str.replace('$', '').astype(float).mean()
Out[93]:
In [94]:
# 字符串方法'contains'檢查子字符串並返回一個布爾系列
orders.item_name.str.contains('Chicken').head()
Out[94]:
In [95]:
# 將布爾系列轉換爲整數（False = 0，True = 1）
orders.item_name.str.contains('Chicken').astype(int).head()
Out[95]:
14. When should I use a "groupby" in pandas? (video)
In [96]:
drinks = pd.read_csv(url7)
drinks.head()
Out[96]:
In [97]:
# 計算整個數據集中的平均beer_servings
drinks.beer_servings.mean()
Out[97]:
In [98]:
# 計算非洲國家的平均beer_servings
drinks[drinks.continent=='Africa'].beer_servings.mean()
Out[98]:
In [99]:
#計算每一個大陸的平均beer_servings
drinks.groupby('continent').beer_servings.mean()
Out[99]:
Documentation for groupby
In [100]:
# 其餘聚合函數（例如'max'）也能夠與groupby一塊兒使用
drinks.groupby('continent').beer_servings.max()
Out[100]:
In [101]:
# 多個聚合函數能夠同時應用
drinks.groupby('continent').beer_servings.agg(['count', 'mean', 'min', 'max'])
Out[101]:
Documentation for agg
In [102]:
# 不指定列，就會算出全部數值列
drinks.groupby('continent').mean()
Out[102]:
In [103]:
# 容許繪圖出如今jupyter notebook中
%matplotlib inline
In [104]:
# 直接在上面的DataFrame的並排條形圖
drinks.groupby('continent').mean().plot(kind='bar')
Out[104]:
Documentation for plot
15. How do I explore a pandas Series? (video)
In [105]:
# read a dataset of top-rated IMDb movies into a DataFrame
url4="https://raw.githubusercontent.com/justmarkham/pandas-videos/master/data/imdb_1000.csv"
movies = pd.read_csv(url4)
movies.head()
Out[105]:
In [106]:
# 檢查數據類型
movies.dtypes
Out[106]:
探索非數字系列
In [107]:
# 計算最多見值的非空值，惟一值和頻率
movies.genre.describe()
Out[107]:
Documentation for describe
In [108]:
# 數Series中每一個值發生的次數
movies.genre.value_counts()
Out[108]:
Documentation for value_counts
In [109]:
# 顯示百分比而不是原始計數
movies.genre.value_counts(normalize=True)
Out[109]:
In [110]:
# '輸出的是一個Series
type(movies.genre.value_counts())
Out[110]:
In [111]:
# 可使用Series方法
movies.genre.value_counts().head()
Out[111]:
In [112]:
# 顯示Series中惟一值
movies.genre.unique()
Out[112]:
In [113]:
#數Series中惟一值的數量
movies.genre.nunique()
Out[113]:
In [114]:
# 兩個Series的交叉列表
pd.crosstab(movies.genre, movies.content_rating)
Out[114]:
Documentation for crosstab
探索數字系列：
In [115]:
# 計算各類彙總統計
movies.duration.describe()
Out[115]:
In [116]:
# 許多統計數據都是做爲Series方法實現的
movies.duration.mean()
Out[116]:
Documentation for mean
In [117]:
# 'value_counts' 主要用於分類數據，而不是數字數據
movies.duration.value_counts().head()
Out[117]:
In [118]:
# 容許繪圖出如今jupyter notebook中
%matplotlib inline
In [119]:
# 'duration'Series的直方圖（顯示數值變量的分佈）
movies.duration.plot(kind='hist')
Out[119]:
In [120]:
# 'genre'Series'value_counts'的條形圖
movies.genre.value_counts().plot(kind='bar')
Out[120]:
Documentation for plot
原文出處：https://www.cnblogs.com/romannista/p/10659805.html
相關文章
- 1. [Python數據挖掘]第2章、Python數據分析簡介
- 2. Python數據分析與挖掘實戰-----第1章 數據挖掘基礎
- 3. 萌新向Python數據分析及數據挖掘 第一章 Python基礎 第七節 用戶輸入和while循環...
- 4. 《數據分析與挖掘 第三章 數據探索》
- 5. 數據分析：pandas 基礎
- 6. Python數據挖掘基礎(三)：Pandas基礎
- 7. 數據挖掘 第二章：數據
- 8. 《Python數據分析與挖掘實戰》第15章——文本挖掘
- 9. python數據分析與挖掘（二十七）--- Pandas量化--股票基礎知識
- 10. 第一章 Pandas基礎
- 更多相關文章...
- • 第一個Hibernate程序 - Hibernate教程
- • 第一個MyBatis程序 - MyBatis教程
- • Flink 數據傳輸及反壓詳解
- • TiDB 在摩拜單車在線數據業務的應用和實踐
相關標籤/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
歡迎關注本站公眾號,獲取更多信息
相關文章