功能齊全、效率一流的免費開源數據庫導入導出工具（c#開發，支持SQL server、SQLite、ACCESS三種數據庫），每個月藉此處理數據5G以上

軟件名：DataPie

功能：支持SQL server、SQLite、ACCESS數據庫的導入、導出、存儲過程調用，支持EXCEL200七、EXCEL200三、ACCESS200七、 CSV文件導入數據庫，支持EXCEL、CSV、ZIP、ACCESS文件方式導出，支持數據拆分導出及自定義SQL查詢與導出。

開發背景：做者從事財務管理工做，主要是出具集團的內部財務報表，隨着公司精細化管理的需求，管理報表的數據量急速增加， 依賴EXCEL加工處理數據已經變得極爲困難，所以團隊全面轉向關係數據庫進行數據處理，爲減小財務人員使用數據庫的難度，所以專門針對財務報表覈算須要，開發了該工具。目前，我月度報表處理的數據量超過5G，最大的單次運算量記錄接近千萬，該工具主要發揮的做用就是將收集到的數據， 導入SQL SERVER數據庫，進行報表運算，而且輸出各種財務報表，對於幾十萬級的數據輸入、輸出基本上可以輕鬆應付。

源碼下載地址：https://github.com/yfl8910/DataPie

 

軟件界面及主要代碼：

1.導入界面，支持EXCEL200七、EXCEL200三、ACCESS200七、 CSV文件導入數據庫，支持CSV文件夾總體導入相應表。財務工做的數據源較多，有業務提供的excel數據，也有從系統中取出的csv、excel或者text文檔，爲知足多種數據源的輸入，因此多種數據源導入數據庫。目前SQL SERVER導入總體效率較高，ACCESS下，excel文件導入效率最高（幾十萬的數據量能夠很快導入），其餘方式效率還有待提高。

涉及核心代碼以下

  public static DataTable GetDataTableFromFile(string path, string tname)
        {
            string ace = "Microsoft.ACE.OLEDB.12.0";
            string jet = "Microsoft.Jet.OLEDB.4.0";
            string xl2007 = "Excel 12.0 Xml";
            string xl2003 = "Excel 8.0";
            string imex = "IMEX=1";
            string hdr = "Yes";
            string conn = "Provider={0};Data Source={1};Extended Properties=\"{2};HDR={3};{4}\";";
            string select = "";
            string ext = Path.GetExtension(path);
            OleDbDataAdapter oda;
            DataTable dt = new DataTable(tname);
            switch (ext.ToLower())
            {
                case ".xlsx":
                    conn = String.Format(conn, ace, Path.GetFullPath(path), xl2007, hdr, imex);
                    select = string.Format("SELECT * FROM [{0}$]", tname);
                    break;
                case ".xls":
                    conn = String.Format(conn, jet, Path.GetFullPath(path), xl2003, hdr, imex);
                    select = string.Format("SELECT * FROM [{0}$]", tname);
                    break;
                case ".accdb":
                    conn = String.Format("Provider=Microsoft.ACE.OLEDB.12.0;Data Source= {0};Persist Security Info=False;", path);
                    select = string.Format("SELECT * FROM [{0}]", tname);
                    break;
                case ".csv":
                    conn = String.Format(conn, ace, path.Substring(0, path.LastIndexOf('\\')), "text;Excel 12.0", hdr, imex);
                    select = string.Format("SELECT * FROM [{0}]", Path.GetFileName(path));
                    break;
                default:
                    throw new Exception("File Not Supported!");
            }
            OleDbConnection con = new OleDbConnection(conn);
            con.Open();
            oda = new OleDbDataAdapter(select, con);
            oda.Fill(dt);
            con.Close();
            return dt;
        }

 

 

2.數據導出界面，支持EXCEL200七、CSV、ACCESS、zip壓縮文件、分拆導出幾個功能。我目前從事的崗位，報表總體數據量接近千萬級，其中單表須要導出分發到業務手中的數據量最大也超過了百萬級，目前達到百萬級數據量的表導出，通常採用CSV文件或者壓縮包的形式，或者分多個表導出。

涉及核心代碼。excel導出主要源碼 ：

  public static void SaveExcel(string FileName, string sql, string SheetName)
        {

            FileInfo newFile = new FileInfo(FileName);
            if (newFile.Exists)
            {
                newFile.Delete();
                newFile = new FileInfo(FileName);
            }
            using (ExcelPackage package = new ExcelPackage(newFile))
            {
                try
                {
                    ExcelWorksheet ws = package.Workbook.Worksheets.Add(SheetName);

                    IDataReader reader = DBConfig.db.DBProvider.ExecuteReader(sql);
                    ws.Cells["A1"].LoadFromDataReader(reader, true);
                }
                catch (Exception ex)
                {
                    throw ex;
                }
                package.Save();
            }
         
        }

        /// <summary>
        /// 工做簿中添加新的SHEET
        /// </summary>
        public static bool SaveExcel(ExcelPackage package, string sql, string SheetName)
        {

            try
            {
                ExcelWorksheet ws = package.Workbook.Worksheets.Add(SheetName);
                IDataReader reader = DBConfig.db.DBProvider.ExecuteReader(sql);
                ws.Cells["A1"].LoadFromDataReader(reader, true);

                return true;
            }
            catch (Exception ex)
            {
                throw ex;
            }
        }
        /// <summary>
        /// 單表格導出到一個EXCEL工做簿
        /// </summary>
        public static int ExportExcel(string FileName, string sql, string SheetName)
        {
            if (FileName != null)
            {
                Stopwatch watch = Stopwatch.StartNew();
                watch.Start();

                SaveExcel(FileName, sql, SheetName);
                watch.Stop();
                return Convert.ToInt32(watch.ElapsedMilliseconds / 1000);
            }

            return -1;
        }

        public static async Task<int> ExportExcelAsync(string FileName, string sql, string SheetName)
        {

            return await Task.Run(
                () => { return ExportExcel(FileName,sql,SheetName); }
                );

        }

        /// <summary>
        /// 多表格導出到一個EXCEL工做簿
        /// </summary>
        public static int ExportExcel(string[] TabelNameArray, string filename)
        {
            if (filename != null)
            {
                Stopwatch watch = Stopwatch.StartNew();
                watch.Start();

                FileInfo newFile = new FileInfo(filename);
                if (newFile.Exists)
                {
                    newFile.Delete();
                    newFile = new FileInfo(filename);
                }


                using (ExcelPackage package = new ExcelPackage(newFile))
                {
                    for (int i = 0; i < TabelNameArray.Length; i++)
                    {
                        string sql = "select * from  [" + TabelNameArray[i] + "]";
                        IDataReader reader = DBConfig.db.DBProvider.ExecuteReader(sql);
                        SaveExcel(package, sql, TabelNameArray[i]);

                    }
                    package.Save();
                }
                watch.Stop();
                return Convert.ToInt32(watch.ElapsedMilliseconds / 1000);
            }

            return -1;
        }

        public static  async Task<int> ExportExcelAsync(string[] TabelNameArray, string filename) {

            return await Task.Run(
                () => { return ExportExcel( TabelNameArray,filename); }      
                );

        }

        /// <summary>
        /// 分拆導出
        /// </summary>
        public static int ExportExcel(string[] TabelNameArray, string filename, string[] whereSQLArr)
        {
            if (filename != null)
            {
                Stopwatch watch = Stopwatch.StartNew();
                watch.Start();
                FileInfo file = new FileInfo(filename);
                int wherecount = whereSQLArr.Length;
                int count = TabelNameArray.Length;
                for (int i = 0; i < wherecount; i++)
                {
                    string s = filename.Substring(0, filename.LastIndexOf("\\"));
                    StringBuilder newfileName = new StringBuilder(s);
                    int index = whereSQLArr[i].LastIndexOf("=");
                    string sp = whereSQLArr[i].Substring(index + 2, whereSQLArr[i].Length - index - 3);
                    newfileName.Append("\\" + file.Name.Substring(0, file.Name.LastIndexOf(".")) + "_" + sp + ".xlsx");
                    FileInfo newFile = new FileInfo(newfileName.ToString());
                    if (newFile.Exists)
                    {
                        newFile.Delete();
                        newFile = new FileInfo(newfileName.ToString());
                    }

                    for (int j = 0; j < count; j++)
                    {
                        string sql = "select * from [" + TabelNameArray[j] + "]" + whereSQLArr[i];
                        IDataReader reader = DBConfig.db.DBProvider.ExecuteReader(sql);
                        SaveExcel(newfileName.ToString(), sql, TabelNameArray[j]);

                    }


                }

                watch.Stop();
                return Convert.ToInt32(watch.ElapsedMilliseconds / 1000);
            }

            return -1;
        }

        public static async Task<int> ExportExcelAsync(string[] TabelNameArray, string filename, string[] whereSQLArr)
        {

            return await Task.Run(
                () => { return ExportExcel(TabelNameArray,filename,whereSQLArr); }
                );

        }

csv導出主要源碼 （其餘代碼請參github上的源碼）：

  private static void WriteHeader(IDataReader reader, StreamWriter sw)
        {
            for (int i = 0; i < reader.FieldCount; i++)
            {
                if (i > 0)
                    sw.Write(',');
                sw.Write(reader.GetName(i));
            }
            sw.Write(Environment.NewLine);
        }
        private static void WriteContent(IDataReader reader, StreamWriter sw)
        {
            for (int i = 0; i < reader.FieldCount; i++)
            {
                if (i > 0)
                    sw.Write(',');
                String v = reader[i].ToString();
                if (v.Contains(',') || v.Contains('\n') || v.Contains('\r') || v.Contains('"'))
                {
                    sw.Write('"');
                    sw.Write(v.Replace("\"", "\"\""));
                    sw.Write('"');
                }
                else
                {
                    sw.Write(v);
                }
            }
            sw.Write(Environment.NewLine);
        }

        public static int SaveCsv(IDataReader reader, string filename)
        {
            Stopwatch watch = Stopwatch.StartNew();
            watch.Start();

            StreamWriter sw = new StreamWriter(filename, false, Encoding.GetEncoding("gb2312"));
            WriteHeader(reader, sw);

            while (reader.Read())
            {
                WriteContent(reader, sw);
            }

            sw.Flush();
            sw.Close();//釋放資源

            watch.Stop();
            return Convert.ToInt32(watch.ElapsedMilliseconds / 1000);
        }
        public static async Task<int> ExportCsvAsync(IDataReader reader, string filename)
        {
            return await Task.Run( () => { return SaveCsv( reader,  filename);} );
        }

        public static StreamWriter GetStreamWriter(string filename, int outCount)
        {
            string s = filename.Substring(0, filename.LastIndexOf("."));
            StringBuilder newfileName = new StringBuilder(s);
            newfileName.Append(outCount + 1 + ".csv");
            FileInfo newFile = new FileInfo(newfileName.ToString());
            if (newFile.Exists)
            {
                newFile.Delete();
                newFile = new FileInfo(newfileName.ToString());
            }
            StreamWriter sw = new StreamWriter(newfileName.ToString(), false, Encoding.GetEncoding("gb2312"));
            return sw;
        }
    
        public static int SaveCsv(IDataReader reader, string filename, int pagesize)
        {
            int innerCount = 0, outCount = 0;
            Stopwatch watch = Stopwatch.StartNew();
            watch.Start();
            StreamWriter sw = GetStreamWriter(filename, outCount);
            WriteHeader(reader, sw);
            while (reader.Read())
            {
                if (innerCount < pagesize)
                {
                    WriteContent(reader, sw);
                    innerCount++;
                }
                else
                {
                    innerCount = 0;
                    outCount++;
                    sw.Flush();
                    sw.Close();
                    sw = GetStreamWriter(filename, outCount);
                    WriteHeader(reader, sw);
                    WriteContent(reader, sw);
                    innerCount++;
                }
            }
            sw.Flush();
            sw.Close();
            watch.Stop();
            return Convert.ToInt32(watch.ElapsedMilliseconds / 1000);

        }
        public static async Task<int> ExportCsvAsync(IDataReader reader, string filename, int pagesize)
        {
            return await Task.Run(() => { return SaveCsv(reader, filename,pagesize); });
        }

 

3.設置拆分導出條件。財務工做中，涉及向業務單位分發各自的財務數據，若是採用手工篩選，而後分發效率較低，因此開發此功能，快速實現數據的拆分發送。

 

4.自定義SQL代碼導出。經過雙擊左邊的表名便可自動生成導出SQL腳本，點擊添加條件，還能夠增長導出的條件。該工具在數據覈查時，使用比較方便，雖然SQL server數據庫也自帶相應的功能，可是導出功能缺沒法知足個人需求，因此開發一個方便本身使用的界面仍是頗有必要的。

 

 

其餘各項功能，及代碼請參考github上的源碼。