簡單的作法:java
須要在python中獲取大文件(數十萬行)的行數。python
def file_len(fname): with open(fname) as f: for i, l in enumerate(f): pass return i + 1
有效的方法(緩衝區讀取策略):segmentfault
首先看下運行的結果:數組
mapcount : 0.471799945831 simplecount : 0.634400033951 bufcount : 0.468800067902 opcount : 0.602999973297
所以,對於Windows/Python2.6來講,緩衝區讀取策略彷佛是最快的。app
如下是代碼:dom
from __future__ import with_statement import time import mmap import random from collections import defaultdict def mapcount(filename): f = open(filename, "r+") buf = mmap.mmap(f.fileno(), 0) lines = 0 readline = buf.readline while readline(): lines += 1 return lines def simplecount(filename): lines = 0 for line in open(filename): lines += 1 return lines def bufcount(filename): f = open(filename) lines = 0 buf_size = 1024 * 1024 read_f = f.read # loop optimization buf = read_f(buf_size) while buf: lines += buf.count('\n') buf = read_f(buf_size) return lines def opcount(fname): with open(fname) as f: for i, l in enumerate(f): pass return i + 1 counts = defaultdict(list) for i in range(5): for func in [mapcount, simplecount, bufcount, opcount]: start_time = time.time() assert func("big_file.txt") == 1209138 counts[func].append(time.time() - start_time) for key, vals in counts.items(): print key.__name__, ":", sum(vals) / float(len(vals))
以上就是如何高效地獲取文件行數的詳細內容,但願對你有所幫助。oop
閱讀原文:如何高效地獲取文件行數spa
相關文章:
java中什麼是序列化
如何刪除 list 中指定 index 的元素
如何輸出到stderr
c語言數組中以列優先對嗎?
java中如何定義方法code