如何用django批量導入數據之xml

django後臺批量導入數據

在生產環境中，每每數據不是幾條或者幾百條，那麼舉個例子，將公司全部員工員工號或者賬號密碼導入進後臺，那就不建議你去後臺一條條記錄去添加了

如何從xml中批量導入svn記錄

第一步：
爲數據創建模型

@python_2_unicode_compatible
class SVNLog(models.Model):

    vision = models.IntegerField(verbose_name=u"修訂版本", blank=False, null=False,)
    author = models.CharField(verbose_name=u"做者", max_length=60, blank=True, null=True)
    date = models.DateTimeField(verbose_name=u"修訂時間",null=True )
    msg = models.TextField(verbose_name=u"註釋消息", blank=False, null=False, default=u"")
    paths = models.TextField(verbose_name=u"影響的文件", blank=False, null=False, default=u"")
    created_time = models.DateTimeField(verbose_name=u"建立時間", auto_now_add=True, )
    update_time = models.DateTimeField(verbose_name=u"修改時間", auto_now=True, )

    class Meta:
        ordering = ['revision']

    def __str__(self):
        return u'r%s' % (self.revision or u"", )

既然創建好了模型，那咱們再去創建接受咱們xml文件的models

@python_2_unicode_compatible
class ImportLogFile(models.Model):

    LogFile = models.FileField(upload_to='LogFile')
    FileName = models.CharField(max_length=50, verbose_name=u'文件名')

    class Meta:
        ordering = ['FileName']

    def __str__(self):
        return self.FileName

ok,以上代碼咱們定義好了數據和上傳文件的model
同步數據庫
python manage.py makemigrations
python manage.py migrate

接着咱們去修改admin.py 讓咱們能夠從後臺上傳文件，

class ImportLogAdmin(admin.ModelAdmin):

    list_display = ('LogFile','FileName',)
    list_filter = ['FileName',]

    def save_model(self, request, obj, form, change):

        re = super(YDImportLogAdmin,self).save_model(request, obj, form, change)
        update_svn_log(self, request, obj, change)
        return re

注意上面代碼裏的save_model,這裏纔是關鍵，在這裏我重寫了ModelAdmin裏的save_model方法
由於咱們要把上傳文件，讀取文件，解析文件，操做數據庫合爲一步來操做，你們能夠打開debug，在上傳文件的時候，返回參數的obj裏包括了文件上傳的路徑，這個路徑也是下一步咱們操做解析文件的關鍵，好了咱們在這個app文件夾下新建一個utils.py 用來操做咱們操做文件和數據庫的工具類，爲了簡單我寫成了函數以下
先貼一下咱們要測試的xml文件

<?xml version="1.0" encoding="UTF-8"?>
<log>
<logentry
   revision="2">
<author>qwert</author>
<date>2016-09-27T07:16:37.396449Z</date>
<paths>
<path
   action="A"
   prop-mods="false"
   text-mods="true"
   kind="file">/aaa/README
    </path>
</paths>
<msg>20160927 151630</msg>
</logentry>


<logentry
   revision="1">
<author>VisualSVN Server</author>
<date>2016-09-20T05:03:12.861315Z</date>
<paths>
<path
   action="A"
   prop-mods="false"
   text-mods="false"
   kind="dir">/branches</path>
<path
   action="A"
   prop-mods="false"
   text-mods="false"
   kind="dir">/tags</path>
<path
   action="A"
   prop-mods="false"
   text-mods="false"
   kind="dir">/trunk</path>
</paths>
<msg>hello word</msg>
</logentry>
</log>

輸出結果格式

r2 | qwer | 2016-09-27 15:16:37 +0800 (二, 27  9 2016) | 1 line
Changed paths:
   A /xxx/README

20160927 151630
------------------------------------------------------------------------
r1 | VisualSVN Server | 2016-09-20 13:03:12 +0800 (二, 20  9 2016) | 1 line
Changed paths:
   A /branches
   A /tags
   A /trunk

Initial structure.

from .models import SVNLog
import xmltodict
def update_svn_log(self, request, obj, change):

    headers = ['r','a','d','m','p']
    filepath = obj.LogFile.path
    xmlfile = xmltodict.parse(open(filepath, 'r'))
    xml_logentry = xml.get('log').get('logentry')
    info_list = []
    pathlist = []
    sql_insert_list = []
    sql_update_list = []
    for j in xml:
        data_dict = {}
        # get path
        paths = j.get('paths').get('path')
        if isinstance(paths,list):
            for path in paths:
                action = path.get('@action')
                pathtext = path.get('#text')
                pathtext = action + ' ' + pathtext
                pathlist.append(pathtext)
                
            _filelist = u'\n'.join(pathlist)
            _paths = u"Changed paths:\n {}".format(_filelist)
            print _paths
        else:
            _filelist = paths.get('@action') + ' ' + paths.get('#text')
            _paths = u"Changed paths:\n {}".format(_filelist)
            print _paths
        # get revision
        vision = j.get('@vision')
        # get auth
        author = j.get('author')
        #get date
        date = j.get('date')
        #get msg
        msg = j.get('msg')

        data_dict[headers[0]] = int(vision)
        data_dict[headers[1]] = author
        data_dict[headers[2]] = date
        data_dict[headers[3]] = msg
        data_dict[headers[4]] = _paths
        info_list.append(data_dict)

    _svnlog = SVNLog.objects.filter().order_by('-vision').first()
    _last_version = _svnlog.vision if _svnlog else 0

    for value in info_list:
        vision = value['r']
        author = value['a']
        date = value['d']
        msg = value['m']
        paths = value['p']
        print vision,author
        _svnlog = YDSVNLog.objects.filter().order_by('-revision').first()
        _last_version = _svnlog.revision if _svnlog else 0
        if vision > _last_version:
            sql_insert_list.append(SVNLog(revision=revision, author=author, date=date, msg = msg, paths = paths))
        else:
            sql_update_list.append(SVNLog(revision=revision, author=author, date=date, msg = msg, paths = paths))

    SVNLog.objects.bulk_create(sql_insert_list)
    SVNLog.objects.bulk_create(sql_update_list)

咱們使用的xmltodict這個第三方庫來解析xml，他把內容解析成了高效率的orderdict類型，就是有序列的字典

這個xml中比較複雜的是那個paths裏的path，由於這個xml中包含兩個元素，第一個元素的path只含有一個path，第二個元素中的paths包含有三個path，所以咱們在解析獲取的時候須要判斷一下

paths = j.get('paths').get('path')
if isinstance(paths,list):
    pass

咱們判斷這個path是否是一個list類型的，若是是，那咱們就按照list的方式來處理，若是不是，那咱們就按單個的方式來處理,獲取以後按照輸出結果格式處理下結果
而後獲取其餘內容

revision = j.get('@vision')
# get auth
author = j.get('author')
#get date
date = j.get('date')
#get msg
msg = j.get('msg')

最後咱們將獲取到的元素存在字典裏
在循環中判斷當前的版本號和數據庫中的版本號，
若是比原來的小，那麼咱們執行更新操做，反之執行插入操做

最後使用了bulk_create來操做數據庫，這樣避免了循環中每次都進行數據庫操做形成的資源浪費

做者微信號：T_V_T_