Python開發【筆記】:從海量文件的目錄中獲取文件名--方法性能對比

Python獲取文件名的方法性能對比

 

前言:日常在python中從文件夾中獲取文件名的簡單方法   os.system('ll /data/')   可是當文件夾中含有巨量文件時,這種方式徹底是行不通的;python

 

在/dd目錄中生成了近6百萬個文件,接下來看看不一樣方法之間的性能對比  快速生成文件的shell腳本  shell

for i in $(seq 1 1000000);do echo text >>$i.txt;done

  

 

 一、系統命令 ls -l性能

# 系統命令 ls -l

import time
import subprocess

start = time.time()
result = subprocess.Popen('ls -l /dd/', stdout=subprocess.PIPE,shell=True)

for file in result.stdout:
    pass
print(time.time()-start)

# 直接卡死

  

二、glob 模塊spa

# glob 模塊

import glob
import time


start = time.time()
result = glob.glob("/dd/*")
for file in result:
    pass
print(time.time()-start)

# 49.60481119155884

  

三、os.walk 模塊blog

# os.walk 模塊

import os
import time

start = time.time()
for root, dirs, files in os.walk("/dd/", topdown=False):
        pass
print(time.time()-start)

# 8.906772375106812

  

四、os.scandir 模塊排序

# os.scandir 模塊

import os
import time

start = time.time()
path = os.scandir("/dd/")
for i in path:
    pass
print(time.time()-start)

# 4.118424415588379

  

五、shell find命令pdo

# shell find命令

import time
import subprocess

start = time.time()
result = subprocess.Popen('find /dd/', stdout=subprocess.PIPE,shell=True)

for file in result.stdout:
    pass
print(time.time()-start)

# 6.205533027648926

  

六、shell ls -1 -f 命令 不進行排序class

# shell ls -1  -f 命令

import time
import subprocess

start = time.time()
result = subprocess.Popen('ls -1 -f /dd/', stdout=subprocess.PIPE,shell=True)

for file in result.stdout:
    pass
print(time.time()-start)

# 3.3476643562316895

  

七、os.listdirimport

# os.listdir

import os
import time


start = time.time()
result = os.listdir('/dd')
for file in result:
    pass
print(time.time()-start)

# 2.6720399856567383
相關文章
相關標籤/搜索