用Python處理csv文件的一些小玩意兒

時間 2019-12-05

原文原文鏈接

Python CSV Toolkit

整理了一些我的在利用python處理csv文件時常常用到的一些自定義方法，放在這裏主要方便本身查閱，也能夠給其餘人作參考python

輸出CSV文件某列的匹配/不匹配的記錄

主要用於從csv文件中抽取出匹配特定列的特定字段集合的記錄，好比現有這麼一個csv文件（表格化後）code

name	age	sex
Danny	24	male
Daisy	23	female
Lancelot	23	unknown
Lydia	21	female
...	...	...

須要輸出其中age爲23的記錄到新的csv文件，則咱們能夠先把23這麼個關鍵詞用一個列表收集起來，而後經過下列代碼從csv文件中找出全部符合條件的記錄並輸出排序

import sys
import csv

# try to fix '_csv.Error: field larger than field limit (131072)'
csv.field_size_limit(sys.maxint)

# write to common csv file with delimiter ','
# output the rows with matched id in id_list to a new csv file
def csv_match(id_list,key,input_file,output_file):
    with open(input_file, 'rb') as f:
        reader = csv.DictReader(f)
        rows = [row for row in reader if row[key] in set(id_list)]

    header = rows[0].keys()
    with open(output_file, 'w') as f:
        f.write(','.join(header))
        f.write('\n')
        for data in rows:
            f.write(",".join(data[h] for h in header))
            f.write('\n')

調用的時候：

lst=['23']
csv_match(lst,'age','in.csv','out.csv')

key爲須要匹配的列名，另外咱們也能夠提取不符合該條件的記錄，‘取個反’就好了

# output the rows with not matched id in id_list to a new csv file
def csv_not_match(id_list, key, input_file, output_file):
    with open(input_file, 'rb') as f:
        reader = csv.DictReader(f)
        rows = [row for row in reader if not row[key] in set(id_list)]

    header = rows[0].keys()
    with open(output_file, 'w') as f:
        f.write(','.join(header))
        f.write('\n')
        for data in rows:
            f.write(",".join(data[h] for h in header))
            f.write('\n')

對於須要判斷csv文件中多個列的值的狀況，只需修改對應的判別條件和傳入參數狀況便可

# output the rows with matched key1 or key2 in refer_list to a new csv file
# @params
# refer_list: the list referred to
# key,key2: column name of csv file to check the value in the refer_list or not
def csv_match2(refer_list, key1, key2, input_file, output_file):
    with open(input_file, 'rb') as f:
        reader = csv.DictReader(f)
        rows = [row for row in reader if (row[key1] in set(refer_list)) or (row[key2] in set(refer_list))]

    header = rows[0].keys()
    with open(output_file, 'w') as f:
        f.write(','.join(header))
        f.write('\n')
        for data in rows:
            f.write(",".join(data[h] for h in header))
            f.write('\n')

調整csv文件的列的順序

有時候咱們輸出的或者拿到的csv文件的列的順序不夠‘人性化’，爲了讓咱們看起來更加直觀，更舒服一點，咱們能夠按照咱們的須要調整列的順序

import csv
# reorder the column of the csv file to what you want
def csv_reorder(in_file, out_file,lst_order):
    with open(in_file, 'rb') as infile, open(out_file, 'wb') as outfile:
        fieldnames=lst_order
        writer = csv.DictWriter(outfile, fieldnames=fieldnames)
        writer.writeheader()
        for row in csv.DictReader(infile):
            writer.writerow(row)

其中lst_order爲咱們須要的列名順序，用list存儲，舉個例子

season_id,league_name,league_size
2003,scottish-premiership,12
2016,1-hnl,10
2004,alka-superligaen,12
2006,allsvenskan,14
1992,premier-league,22
...

如今咱們想調整他的順序，按照league_name,season_id,league_size的順序從新組合一下
則調用

lst_order = ['league_name','season_id','league_size']
csv_reorder('leagues_size.csv', 'leagues_size_new.csv', lst_order)

獲得結果

league_name,season_id,league_size
scottish-premiership,2003,12
1-hnl,2016,10
alka-superligaen,2004,12
allsvenskan,2006,14
premier-league,1992,22
...

CSV轉換器

這個主要是用來進行csv和python的一些內置的容器例如list,dict之類的轉換，包括一些特殊的多級字典，或者是嵌套列表的字典等等，這裏只是把他們打個包放在一塊兒，具體的能夠參照我以前寫的一篇文章

import csv

#---------------------------------------------------csv <--> dict--------------------------------------------

# convert csv file to dict
# @params:
# key/value: the column of original csv file to set as the key and value of dict
def csv2dict(in_file,key,value):
    new_dict = {}
    with open(in_file, 'rb') as f:
        reader = csv.reader(f, delimiter=',')
        fieldnames = next(reader)
        reader = csv.DictReader(f, fieldnames=fieldnames, delimiter=',')
        for row in reader:
            new_dict[row[key]] = row[value]
    return new_dict


# convert csv file to dict(key-value pairs each row)
# default: set row[0] as key and row[1] as value of the dict
def row_csv2dict(csv_file):
    dict_club={}
    with open(csv_file)as f:
        reader=csv.reader(f,delimiter=',')
        for row in reader:
            dict_club[row[0]]=row[1]
    return dict_club

# write dict to csv file
# write each key/value pair on a separate row
def dict2csv(dict, file):
    with open(file, 'wb') as f:
        w = csv.writer(f)
        # write each key/value pair on a separate row
        w.writerows(dict.items())

# write dict to csv file
# write all keys on one row and all values on the next
def dict2csv2(dict, file):
    with open(file, 'wb') as f:
        w = csv.writer(f)
        # write all keys on one row and all values on the next
        w.writerow(dict.keys())
        w.writerow(dict.values())

# build a dict of list like {key:[...element of lst_inner_value...]}
# key is certain column name of csv file
# the lst_inner_value is a list of specific column name of csv file
def build_list_dict(source_file, key, lst_inner_value):
    new_dict = {}
    with open(source_file, 'rb')as csv_file:
        data = csv.DictReader(csv_file, delimiter=",")
        for row in data:
            for element in lst_inner_value:
                new_dict.setdefault(row[key], []).append(row[element])
    return new_dict
# sample:
# test_club=build_list_dict('test_info.csv','season',['move from','move to'])
# print test_club


# build specific nested dict from csv files
# @params:
#   source_file
#   outer_key:the outer level key of nested dict
#   inner_key:the inner level key of nested dict,and rest key-value will be store as the value of inner key
def build_level2_dict(source_file,outer_key,inner_key):
    new_dict = {}
    with open(source_file, 'rb')as csv_file:
        reader = csv.reader(csv_file, delimiter=',')
        fieldnames = next(reader)
        inner_keyset=fieldnames
        inner_keyset.remove(outer_key)
        inner_keyset.remove(inner_key)
        csv_file.seek(0)
        data = csv.DictReader(csv_file, delimiter=",")
        for row in data:
            item = new_dict.get(row[outer_key], dict())
            item[row[inner_key]] = {k: row[k] for k in inner_keyset}
            new_dict[row[outer_key]] = item
    return new_dict

# build specific nested dict from csv files
# @params:
#   source_file
#   outer_key:the outer level key of nested dict
#   inner_key:the inner level key of nested dict
#   inner_value:set the inner value for the inner key
def build_level2_dict2(source_file,outer_key,inner_key,inner_value):
    new_dict = {}
    with open(source_file, 'rb')as csv_file:
        data = csv.DictReader(csv_file, delimiter=",")
        for row in data:
            item = new_dict.get(row[outer_key], dict())
            item[row[inner_key]] = row[inner_value]
            new_dict[row[outer_key]] = item
    return new_dict

# build specific nested dict from csv files
# @params:
#   source_file
#   outer_key:the outer level key of nested dict
#   lst_inner_value: a list of column name,for circumstance that the inner value of the same outer_key are not distinct
#   {outer_key:[{pairs of lst_inner_value}]}
def build_level2_dict3(source_file,outer_key,lst_inner_value):
    new_dict = {}
    with open(source_file, 'rb')as csv_file:
        data = csv.DictReader(csv_file, delimiter=",")
        for row in data:
            new_dict.setdefault(row[outer_key], []).append({k: row[k] for k in lst_inner_value})
    return new_dict

# build specific nested dict from csv files
# @params:
#   source_file
#   outer_key:the outer level key of nested dict
#   lst_inner_value: a list of column name,for circumstance that the inner value of the same outer_key are not distinct
#   {outer_key:{key of lst_inner_value:[...value of lst_inner_value...]}}
def build_level2_dict4(source_file,outer_key,lst_inner_value):
    new_dict = {}
    with open(source_file, 'rb')as csv_file:
        data = csv.DictReader(csv_file, delimiter=",")
        for row in data:
            # print row
            item = new_dict.get(row[outer_key], dict())
            # item.setdefault('move from',[]).append(row['move from'])
            # item.setdefault('move to', []).append(row['move to'])
            for element in lst_inner_value:
                item.setdefault(element, []).append(row[element])
            new_dict[row[outer_key]] = item
    return new_dict

# build specific nested dict from csv files
# @params:
#   source_file
#   outer_key:the outer level key of nested dict
#   lst_inner_key:a list of column name
#   lst_inner_value: a list of column name,for circumstance that the inner value of the same lst_inner_key are not distinct
#   {outer_key:{lst_inner_key:[...lst_inner_value...]}}
def build_list_dict2(source_file,outer_key,lst_inner_key,lst_inner_value):
    new_dict = {}
    with open(source_file, 'rb')as csv_file:
        data = csv.DictReader(csv_file, delimiter=",")
        for row in data:
            # print row
            item = new_dict.get(row[outer_key], dict())
            item.setdefault(row[lst_inner_key], []).append(row[lst_inner_value])
            new_dict[row[outer_key]] = item
    return new_dict

# dct=build_list_dict2('test_info.csv','season','move from','move to')

# build specific nested dict from csv files
# a dict like {outer_key:{inner_key1:{inner_key2:{rest_key:rest_value...}}}}
# the params are extract from the csv column name as you like
def build_level3_dict(source_file,outer_key,inner_key1,inner_key2):
    new_dict = {}
    with open(source_file, 'rb')as csv_file:
        reader = csv.reader(csv_file, delimiter=',')
        fieldnames = next(reader)
        inner_keyset=fieldnames
        inner_keyset.remove(outer_key)
        inner_keyset.remove(inner_key1)
        inner_keyset.remove(inner_key2)
        csv_file.seek(0)
        data = csv.DictReader(csv_file, delimiter=",")
        for row in data:
            item = new_dict.get(row[outer_key], dict())
            sub_item = item.get(row[inner_key1], dict())
            sub_item[row[inner_key2]] = {k: row[k] for k in inner_keyset}
            item[row[inner_key1]] = sub_item
            new_dict[row[outer_key]] = item
    return new_dict

# build specific nested dict from csv files
# a dict like {outer_key:{inner_key1:{inner_key2:inner_value}}}
# the params are extract from the csv column name as you like
def build_level3_dict2(source_file,outer_key,inner_key1,inner_key2,inner_value):
    new_dict = {}
    with open(source_file, 'rb')as csv_file:
        data = csv.DictReader(csv_file, delimiter=",")
        for row in data:
            item = new_dict.get(row[outer_key], dict())
            sub_item = item.get(row[inner_key1], dict())
            sub_item[row[inner_key2]] = row[inner_value]
            item[row[inner_key1]] = sub_item
            new_dict[row[outer_key]] = item
    return new_dict
   

# build specific nested dict from csv files
# a dict like {outer_key:{inner_key1:{inner_key2:[inner_value]}}}
# for multiple inner_value with the same inner_key2,thus gather them in a list
# the params are extract from the csv column name as you like
def build_level3_dict3(source_file,outer_key,inner_key1,inner_key2,inner_value):
    new_dict = {}
    with open(source_file, 'rb')as csv_file:
        data = csv.DictReader(csv_file, delimiter=",")
        for row in data:
            item = new_dict.get(row[outer_key], dict())
            sub_item = item.get(row[inner_key1], dict())
            sub_item.setdefault(row[inner_key2], []).append(row[inner_value])
            item[row[inner_key1]] = sub_item
            new_dict[row[outer_key]] = item
    return new_dict

#----------------------------------------------------------------------------------------------------------

#---------------------------------------------------csv <--> list--------------------------------------------

def list2csv(list, file):
# def list2csv(list):
#     wr = csv.writer(open(file, 'wb'), quoting=csv.QUOTE_ALL)
    wr=open(file,'w')
    for word in list:
        # print ''.join(word)
        # wr.writerow([word])
        wr.write(word+'\n')
        # wr.writerow(str.split(word,'"')[0])
        # print [word]

# test_list = ['United States', 'China', 'America', 'England']

# list2csv(test_list,'small_test.csv')

# write nested list of dict to csv
def nestedlist2csv(list, out_file):
    with open(out_file, 'wb') as f:
        w = csv.writer(f)
        fieldnames=list[0].keys()  # solve the problem to automatically write the header
        w.writerow(fieldnames)
        for row in list:
            w.writerow(row.values())


# my_list = [{'players.vis_name': 'Khazri', 'players.role': 'Midfielder', 'players.country': 'Tunisia',
#             'players.last_name': 'Khazri', 'players.player_id': '989', 'players.first_name': 'Wahbi',
#             'players.date_of_birth': '08/02/1991', 'players.team': 'Bordeaux'},
#            {'players.vis_name': 'Khazri', 'players.role': 'Midfielder', 'players.country': 'Tunisia',
#             'players.last_name': 'Khazri', 'players.player_id': '989', 'players.first_name': 'Wahbi',
#             'players.date_of_birth': '08/02/1991', 'players.team': 'Sunderland'},
#            {'players.vis_name': 'Lewis Baker', 'players.role': 'Midfielder', 'players.country': 'England',
#             'players.last_name': 'Baker', 'players.player_id': '9574', 'players.first_name': 'Lewis',
#             'players.date_of_birth': '25/04/1995', 'players.team': 'Vitesse'}
#            ]

# nestedlist2csv(my_list, 'dict2csv_test.csv')



# collect and convert the first column of csv file to list
def csv2list(csv_file):
    lst = []
    with open(csv_file, 'rb')as f:
        reader = csv.reader(f, delimiter=',')
        for row in reader:
            lst.append(row[0])
    return list(set(lst))
#----------------------------------------------------------------------------------------------------------

抽取特定列

抽取特定列的全部值並存儲於列表
根據下標抽取特定列到某個新的csv文件

抽取特定列的全部值並存儲於列表

獲取某列原始的數據並保存爲列表

# get certain column value of csv(for common csv file(','))
def get_origin_column_value(file, column_name):
    with open(file, 'rb') as f:
        role_list = []
        reader = csv.reader(f, delimiter=',')
        fieldnames = next(reader)
        reader = csv.DictReader(f, fieldnames=fieldnames, delimiter=',')
        for row in reader:
            role_list.append(row[column_name])
        return role_list

對於某些有特殊須要的能夠直接修改代碼，好比對原始的列的值進行除重和排序後獲取，以下

# get certain column value of csv(for common csv file(',')),and judge if it's repeated
def get_column_value2(file, column_name):
    with open(file, 'rb') as f:
        role_list = []
        reader = csv.reader(f, delimiter=',')
        fieldnames = next(reader)
        reader = csv.DictReader(f, fieldnames=fieldnames, delimiter=',')
        for row in reader:
            role_list.append(row[column_name])
        role_set = set(role_list)
        return sorted(list(role_set))

根據下標抽取特定列到某個新的csv文件

import csv
# extract certain column from csv file according to the column#
def column_extract(file_in,file_out,index):
    with open(file_in,'r') as f_in:
        with open(file_out,'w') as f_out:
            for line in f_in:
                f_out.write(line.split(',')[index])
                f_out.write('\n') # comment if a new line already exists

除去徹底重複的記錄

# eliminated the completely repeated record in repeated file for further analysis
def eliminate_repeated_row(in_file,out_file):
    with open(in_file,'rb') as in_file,open(out_file,'wb')as out_file:
        seen=set()
        for line in in_file:
            # print line
            if line in seen:continue

            seen.add(line)
            out_file.write(line)

對csv文件按照某一列排序

# sort the csv file by certain column to put the similar record together for further analysis
def sort_csv_byColumn(in_file, out_file,column_name):
    with open(in_file, 'rb') as f:
        reader = csv.reader(f, delimiter=',')
        fieldnames = next(reader)
        reader = csv.DictReader(f, fieldnames=fieldnames, delimiter=',')
        sorted_list = sorted(reader, key=lambda row: row[column_name], reverse=True)
        # print sorted_list
        csv_converter.nestedlist2csv(sorted_list, out_file)

例如咱們按照league_name排序(注意這裏調用了csv轉換器中的方法將列表的字典轉換爲csv文件)

sort_csv_byColumn('leagues_size.csv','ordered_leagues_size.csv','league_name')

獲得結果

season_id,league_name,league_size
2016,ykkonen,9
2003,ykkonen,14
2005,ykkonen,14
2006,ykkonen,14
2007,ykkonen,14
2010,ykkonen,13
2011,ykkonen,10
2009,ykkonen,14
2008,ykkonen,14
2012,ykkonen,10
2013,ykkonen,10
2014,ykkonen,10
2015,ykkonen,10
2016,wiener-stadtliga,16
1988,wiener-stadtliga,16
1993,wiener-stadtliga,16
1994,wiener-stadtliga,16
1995,wiener-stadtliga,16
1996,wiener-stadtliga,16
1997,wiener-stadtliga,16
1998,wiener-stadtliga,16

若是咱們按league_size排序

sort_csv_byColumn('leagues_size.csv',
                    'orderedbysize_leagues_size.csv','league_size')

獲得結果

season_id,league_name,league_size
2008,virsliga,9
2010,virsliga,9
2012,a-lyga,9
2012,a-pojat-sm-sarja,9
2013,a-pojat-sm-sarja,9
1953,salzburger-liga,9
2010,3-lig-grup-1,9
2013,armenian-first-league,9
2016,ykkonen,9
2014,stirling-sports-premiership,9
2014,hong-kong-premier-league,9
2015,hong-kong-premier-league,9
1996,s-league,9
2015,s-league,9
2013,united-football-league,9
2016,i-league,9

鍵值互換

csv文件每一條記錄其實能夠看做是一個字典，有時csv文件裏有不一樣的鍵對應同一個值的狀況，咱們想講記錄反轉一下，即讓值做爲鍵，對應的鍵做爲值

# return a dict with the same value in original as new key and keys as value
def dict_same_value(original_dict):
    new_dict={}
    for k,v in original_dict.iteritems():
        new_dict.setdefault(v,[]).append(k)
    return new_dict

最後歡迎你們fork關於這個的github上的repository，一塊兒豐富更多好玩的功能~

更新日誌一、2016-12-18 修復了從csv文件中獲取特定的列的值保存爲集合的問題，而是存儲爲原始的列表二、2016-12-22 改進了csv轉換器中的構建二級字典的方法，使其變得更加靈活三、2016年12月24日14:57:48 在csv轉換器部分加入三級字典構造的參照方法四、2017年1月9日11:28:45 在csv轉換器部分，三級字典構造中，加入了最內部存儲值爲列表的構造方法五、2017年1月16日10:43:41 在csv轉換器部分，加入了構造列表字典的方法以及構造特殊的二級字典（內部爲列表）的方法六、2017年2月9日10:58:17 在csv轉換器部分，加入了新的構造特殊的二級字典（內部爲列表）的方法七、2017年2月10日11:21:45 在csv轉換器部分，改進了簡單的csv文件轉換爲字典的方法，此外在Csv_Match部分，加入了匹配判斷多個列對應的元素條件的方法