使用 shell / python 進行sql的excel報表導出

時間 2019-12-05

標籤使用 shell python 進行 sql excel 報表導出欄目 Unix 简体版

原文原文鏈接

　　若是要求你進行一個表數據的導出，若是使用shell的話，很容易作到，即執行一下 select 語句就能夠拿到返回結果了！html

以下：python

/usr/bin/mysql -u"${username}" -p"${password}" --host=${host} -D"${database}" <  ${sql_script_path} > ${export_data_full_path1};

　　如上執行完成以後，數據就被導出到 export_data_full_path1 指定的文件位置去了。mysql

　　若是想要使用 excel 格式來打開，有一個很簡單的方法，即把後綴名命名爲： .xls 就能夠了。惟一的缺點是，此時你可能看到一個提示，即：文件名後綴與具體的格式不匹配等等！可是你仍然能夠正常打開！sql

　　可是對於有中文一類的導出操做，則又是，另外一番景象了！不過咱們能夠經過一個簡單的編碼轉換來解決這個問題！shell

iconv -futf8 -tgb2312 -o ${export_data_full_path1} ${export_data_full_path1};

　　轉換以後，即便有中文也能正常查看了！數據庫

　　至於使用 shell 進行發送郵件，則也是簡單的一比！bash

echo "yesterday report infomation, FYI ." | mail -s "$yesterday_format report" -a ${export_data_full_path1} ${mail_receivers}

完整的彙總數據sql 加發送郵件 shell ，區區幾行代碼以下：服務器

#!/bin/bash

sql_script_path="/scripts/report/report.sql";

# data sync target configs
host="127.0.0.1";
username="test";
password='123456';
database='test';

export_data_path_prefix="/data/report/export_datas";
mail_receivers="test@163.com";

yesterday_format=`date '+%Y-%m-%d' -d '-1 day'`;
export_data_full_path1="${export_data_path_prefix}/${yesterday_format}-Report.xls";

/usr/bin/mysql -u"${username}" -p"${password}" --host=${host} -D"${database}" <  ${sql_script_path} > ${export_data_full_path1};

iconv -futf8 -tgb2312 -o ${export_data_full_path1} ${export_data_full_path1};


# todo: send mail begin
echo '$yesterday_format report infomation, FYI .' | mail -s "$yesterday_format report" -a ${export_data_full_path1} ${mail_receivers} 

echo `date '+%Y-%m-%d %H:%I:%S'` "over!"

如上，完成了使用 shell 進行彙總數據，並導出 excel 發送郵件的功能了！優缺點以下：app

　　優勢： 1. 簡單！dom

　　缺點： 1. 不能支持複雜格式的指定！ 2. 須要安裝 mysql-client 包！

因此，針對上面問題，尤爲是不支持複雜格式的指定問題，註定咱們須要一個更完善的方式來實現！即便用 python 等一類強大語言支持！

　　一樣，咱們對這類操做有兩大要求： 1. 簡單； 2. 支持複雜操做；很明顯，python 在這種場景下很駕輕就熟！

　　咱們先來看一下完整實現：

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# need install components: openpyxl, pymysql
from openpyxl import Workbook
import pymysql
import time
import sys
from email.mime.application import MIMEApplication
from email.mime.multipart import MIMEMultipart
from email.header import Header
from email.utils import formataddr
import smtplib
import random

class MysqlClient:
    # create connection to mysql
    def __init__(self, db_host, user, password, database, port=10000, authMechanism="PLAIN"):
        self.conn = pymysql.connect(host=db_host,
                                  port=port,
                                  user=user,
                                  password=password,
                                  database=database,
                                  )
    def query(self, sql):
        with self.conn.cursor() as cursor:
            cursor.execute(sql)
            return cursor.fetchall()
    def close(self):
        self.conn.close()


class MailClient:
    # initial account info
    def __init__(self, sender_account, sender_pass, report_date):
        self.sender_account = sender_account
        self.sender_pass = sender_pass
        self.report_date = report_date
    def send_mail(self, receivers, subject, msg_content, xls_file_path):
        try:
          msg = MIMEMultipart()
          attach = MIMEApplication(open(xls_file_path, 'rb').read())
          attach.add_header('Content-Disposition', 'attachment', filename=self.report_date + '-Reprot.xlsx')
          msg.attach(attach)
          body = MIMEText(msg_content, 'html', 'utf-8')
          msg.attach(body)
          msg['From'] = self.sender_account
          msg['To'] = ','.join(receivers)
          msg['Subject'] = Header(subject, 'utf-8')
          server = smtplib.SMTP_SSL("smtp.exmail.qq.com", 465)
          server.login(mail_sender, sender_pass)
          server.sendmail(mail_sender, receivers, msg.as_string())
          server.quit()
          print(self.report_date + '-----success')
        except Exception as e:
          print(self.report_date + '-----failed')
          print(e)

def create_xsls(info_report, xls_file_path):
  mysql_client = MysqlClient(db_host='127.0.0.1', port=3306, user='test', password='test', database='testdb')
  wb = Workbook()
  inx_rep = 0
  for report in info_report: 
    result = mysql_client.query(report.get('report_sql'))
    if inx_rep == 0:
      ws = wb.worksheets[0]
      ws.title = report.get('rep_name')
    else:
      ws = wb.create_sheet(title=report.get('rep_name'))
    titleidx = 0
    titlenames = report.get('rep_title')
    for titlename in titlenames.split(","):
      if chr(ord('A') + titleidx) <= 'Z':
        title_name = chr(ord('A') + titleidx)
      else:
        title_name = "A" + chr(ord('A') + titleidx - 26)

      ws[title_name + "1"] = titlename
      titleidx = titleidx + 1
    row_num = 2
    for line in result:
      titleidx = 0
      for titlename in titlenames.split(","):
        if chr(ord('A') + titleidx) <= 'Z':
          title_name = chr(ord('A') + titleidx)
        else:
          title_name = "A" + chr(ord('A') + titleidx - 26)

        ws[title_name + str(row_num)] = line[titleidx]
        titleidx = titleidx + 1
      row_num = row_num + 1
    inx_rep = inx_rep + 1  
  wb.save(xls_file_path)


def main(argv):
    if(argv[1] == 'user_reg'):
      test_sql = '''SELECT DATE(create_time) AS 日期, user_flag AS 用戶類型,
        FROM t_user s
        WHERE create_time>='2019-01-01' AND repay_time<=DATE_ADD(NOW(),INTERVAL 1 DAY)
        GROUP BY DATE(create_time),user_flag
        LIMIT 1000;
        '''
      info_report = [{"rep_name":"測試工單","rep_title":"日期,用戶類型,註冊來源","report_sql":test_sql}]
      receivers = ['test2@163.com']
    else:
      pass;
    
    report_date = time.strftime('%Y%m%d', time.localtime())
    rep_rand = random.randint(100, 999)
    xls_file_path = '/mnt/data/report/data/' + argv[1] + '-' + report_date + '-Reprot' + str(rep_rand) + '.xlsx';
    create_xsls(info_report, xls_file_path)
    mail_client = MailClient(sender_account='test@163.cn', sender_pass='1234567', report_date=report_date)
    mail_client.send_mail(receivers, subject='測試主題', msg_content='測試body', xls_file_path=xls_file_path)

if __name__ == "__main__":
    main(sys.argv)