簡單實現TCP下的大文件高效傳輸

時間  2019-12-10
標籤 簡單 實現 tcp 文件 高效 傳輸 欄目 系統網絡 简体版
原文   原文鏈接

在TCP下進行大文件傳輸不象小文件那樣直接打包個BUFFER發送出去,由於文件比較大因此不可能把文件讀到一個BUFFER發送出去.主要有些文件的大小多是1G,2G或更大,分配這麼大的BUFFER對內存來講顯然是不現實的事情;針對服務端的設計來講就更須要嚴緊些,BUFFER大小的限制也是變得很重要.下面介紹使用Beetle簡單地實現大文件在TCP的傳輸應用.服務器

協議制定

既然須要把文件分塊來處理,那在TCP傳輸的過程須要制定一些協議來規範數據有效性,數據協議主要有三個:告訴服務器須要上傳文件,文件塊上傳和返回每一個環節處理的結果.ide

1)上傳文件指令測試

public class Upload:ObjectMessage
    {
        public string FileMD5
        {
            get;
            set;
        }

        public string FileName
        {
            get;
            set;
        }

        public long FileSize
        {
            get;
            set;
        }

        public override void FromProtocolData(HttpData httpbase)
        {
            FileName = httpbase[CONSTVALUE.HEADER_NAME];
            FileMD5 = httpbase[CONSTVALUE.HEADER_MD5];
            FileSize = long.Parse(httpbase[CONSTVALUE.HEADER_FILESIZE]);
        }

        protected override void OnDisposed()
        {
           
        }

        protected override void OnToProtocolData(HttpData httpbase)
        {
            httpbase.Command = CONSTVALUE.COMMAND_UPLOAD;
            httpbase[CONSTVALUE.HEADER_MD5] = FileMD5;
            httpbase[CONSTVALUE.HEADER_NAME] = FileName;
            httpbase[CONSTVALUE.HEADER_FILESIZE] = FileSize.ToString();
        }
    }
View Code

2)上傳文件塊指令ui

public class UploadData:ObjectMessage
    {

        public string FileMD5
        {
            get;
            set;
        }

        public Beetle.ByteArraySegment Data
        {
            get;
            set;
        }
        
        public override void FromProtocolData(HttpData httpbase)
        {
            FileMD5 = httpbase[CONSTVALUE.HEADER_MD5];
            Data = httpbase.Content;
        }

        protected override void OnDisposed()
        {
            if (Data != null)
            {
                FileTransferPackage.BufferPool.Push(Data);
                Data = null;
            }
        }

        protected override void OnToProtocolData(HttpData httpbase)
        {
            httpbase.Command = CONSTVALUE.COMMAND_UPLOAD_DATA;
            httpbase[CONSTVALUE.HEADER_MD5] = FileMD5;
            httpbase.Content = Data;
        }
    }
View Code

3)返回值指令spa

public class Result :ObjectMessage
    {
        public string FileMD5
        {
            get;
            set;
        }

        public bool Error
        {
            get;
            set;
        }

        public string ErrorDetail
        {
            get;
            set;
        }

        public override void FromProtocolData(HttpData httpbase)
        {
            ErrorDetail = httpbase[CONSTVALUE.HEADER_STATUS_DETAIL];
            Error = httpbase[CONSTVALUE.HEADER_STATUS] == CONSTVALUE.VALUE_SUCCESS;
            FileMD5 = httpbase[CONSTVALUE.HEADER_MD5];
        }

        protected override void OnDisposed()
        {

        }

        protected override void OnToProtocolData(HttpData httpbase)
        {
            httpbase.Command = CONSTVALUE.COMMAND_RESULT;
            if (Error)
            {
                httpbase[CONSTVALUE.HEADER_STATUS] = CONSTVALUE.VALUE_SUCCESS;
            }
            else
            {
                httpbase[CONSTVALUE.HEADER_STATUS] = CONSTVALUE.VALUE_ERROR;
            }
            httpbase[CONSTVALUE.HEADER_STATUS_DETAIL] = ErrorDetail;
            httpbase[CONSTVALUE.HEADER_MD5] = FileMD5;
        }
    }
View Code

ObjectMessage是Beetle一個簡化HTTP協議的擴展對象,它提供自定義Header和Body等功能..net

文件讀寫器

既然須要處理文件塊,那提供一些簡單的文件塊讀取和寫入方法是比較重要的.它不只從設計解決功能的偶合度,還能夠方便從此的利用.設計

1)UploadReader code

public class UploadReader : IDisposable
    {
        public UploadReader(string file)
        {

            mStream = System.IO.File.OpenRead(file);
            StringBuilder sb = new StringBuilder();
            MD5 md5Hasher = MD5.Create();
            foreach (Byte b in md5Hasher.ComputeHash(mStream))
                sb.Append(b.ToString("x2").ToLower());
            FileMD5 = sb.ToString();
            mStream.Position = 0;
            FileSize = mStream.Length;
            FileName = System.IO.Path.GetFileName(file);
                
        }

        private System.IO.FileStream mStream = null;

        public string FileName
        {
            get;
            set;
        }

        public long LastReadLength
        {
            get;
            set;
        }

        public long ReadLength
        {
            get;
            set;
        }

        public long FileSize
        {
            get;
            set;
        }

        public string FileMD5
        {
            get;
            set;
        }

        public bool Completed
        {
            get
            {
                return mStream != null && ReadLength == mStream.Length;
            }
        }

        public void Close()
        {
            if (mStream != null)
            {
                mStream.Close();
                mStream.Dispose();
            }
        }

        public void Reset()
        {
            mStream.Position = 0;
            LastReadLength = 0;
            ReadLength = 0;
        }

        public void Read(ByteArraySegment segment)
        {
            int loads = mStream.Read(segment.Array, 0, FileTransferPackage.BUFFER_SIZE);
            segment.SetInfo(0, loads);
            ReadLength += loads;
        }

        public void Dispose()
        {
            mStream.Dispose();
        }

        public override string ToString()
        {
            string value= string.Format("{0}(MD5:{4})\r\n\r\n[{1}/{2}({3}/秒)]",FileName,ReadLength,FileSize,ReadLength-LastReadLength,FileMD5);
            if (!Completed)
            {
                LastReadLength = ReadLength;
            }
            return value;
        }
    }
View Code

UploadReader的功能主要是把文件流讀取到指定大小的Buffer中,並提供方法獲取當前的讀取狀況orm

2)UploadWriter對象

public class UploadWriter
    {
        public UploadWriter(string rootPath, string filename,string fileMD5,long size)
        {
            mFullName = rootPath + filename;
            FileName = filename;
            FileMD5 = fileMD5;
            Size = size;
        }

        private string mFullName;

        private System.IO.FileStream mStream;

        public System.IO.FileStream Stream
        {
            get
            {
                if (mStream == null)
                {
                    mStream = System.IO.File.Create(mFullName+".up");
                }
                return mStream;
            }
        }

        public long WriteLength
        {
            get;
            set;
        }

        public long LastWriteLength
        {
            get;
            set;
        }

        public long Size
        {
            get;
            set;
        }

        public string FileName
        {
            get;
            set;
        }

        public string FileMD5
        {
            get;
            set;
        }

        public bool Write(ByteArraySegment segment)
        {
            Stream.Write(segment.Array, 0, segment.Count);
            WriteLength += segment.Count;
            Stream.Flush();
            if (WriteLength == Size)
            {
                Stream.Close();
                if (System.IO.File.Exists(mFullName))
                    System.IO.File.Delete(mFullName);
                System.IO.File.Move(mFullName + ".up", mFullName);
                return true;
            }
            return false;
        }
    }
View Code

UploadWriter的功能主要是把文件寫入到臨時文件中,寫入完成後再更改相應的名稱,爲了方便查詢一樣也提供了一些寫入狀況信息.

服務端代碼

 若是有了解過Beetle的服務端制定的話,那服務端的實現是很是簡單的,只須要寫一個對象承繼ServerBase並實現數據接收方法處理便可以,接收的數據會會自動轉換成以前定義的消息對象,而服務端內部處理的細節是徹底不用關心.

protected override void OnMessageReceive(Beetle.PacketRecieveMessagerArgs e)
        {
            base.OnMessageReceive(e);
            if (e.Message is Protocol.Upload)
            {
                OnUpload(e.Channel, e.Message as Protocol.Upload);
            }
            else if (e.Message is Protocol.UploadData)
            {
                OnUploadData(e.Channel, e.Message as Protocol.UploadData);
            }
        }

        private Protocol.Result GetErrorResult(string detail)
        {
            Protocol.Result result = new Protocol.Result();
            result.Error = true;
            result.ErrorDetail = detail;
            return result;
        }

        private void OnUpload(Beetle.TcpChannel channel, Protocol.Upload e)
        {
            Protocol.Result result;
            if (mTask[e.FileMD5] != null)
            { 
                result = GetErrorResult( "該文件正在上傳任務中!");
                channel.Send(result);
                return;
            }
            UploadWriter writer = new UploadWriter(mRootPath, e.FileName, e.FileMD5, e.FileSize);
            lock (mTask)
            {
                mTask[e.FileMD5] = writer;
            }
            result = new Protocol.Result();
            channel.Send(result);
        }

        private void OnUploadData(Beetle.TcpChannel channel, Protocol.UploadData e)
        {
            using (e)
            {
                Protocol.Result result;
                UploadWriter writer = (UploadWriter)mTask[e.FileMD5];
                if (writer == null)
                {
                    result = GetErrorResult("上傳任務不存在!");
                    channel.Send(result);
                    return;
                }
                if (writer.Write(e.Data))
                {
                    lock (mTask)
                    {
                        mTask.Remove(e.FileMD5);
                    }
                }
                result = new Protocol.Result();
                result.FileMD5 = writer.FileMD5;
                channel.Send(result);
            }
        }

當接收到客戶求上傳請求後會創建對應MD5的文件寫入器,後面文件塊的上傳寫入相關對象便可.

客戶端代碼

 Beetle對於Client的支持也是很是簡單方便,只須要定義一個TcpChannel直接發送定義的對象消息並獲取服務器端返回的消息便可.

 1 private void OnUpload(object state)
 2         {
 3             Lib.UploadReader reader = (Lib.UploadReader)state;
 4             try
 5             {
 6                 IsUpload = true;
 7                 Lib.Protocol.Upload upload = new Lib.Protocol.Upload();
 8                 upload.FileMD5 = reader.FileMD5;
 9                 upload.FileName = reader.FileName;
10                 upload.FileSize = reader.FileSize;
11                 Lib.Protocol.Result result = mClient.Send<Lib.Protocol.Result>(upload);
12                 if (result.Error)
13                 {
14                     mLastError = result.ErrorDetail;
15                     return;
16                 }
17                 while (!reader.Completed)
18                 {
19                     mLastError = "文件上傳中...";
20                     Lib.Protocol.UploadData data = new Lib.Protocol.UploadData();
21                     data.Data = Lib.FileTransferPackage.BufferPool.Pop();
22                     data.FileMD5 = reader.FileMD5;
23                     reader.Read(data.Data);
24                     result = mClient.Send<Lib.Protocol.Result>(data);
25                     if (result.Error)
26                     {
27                         mLastError = result.ErrorDetail;
28                         return;
29                     }
30                 }
31                 mLastError = "文件上傳完成!";
32                 
33             }
34             catch (Exception e_)
35             {
36                 mLastError = e_.Message;
37             }
38             mReader.Reset();
39             IsUpload = false;
40                
41         }
View Code

整個過程只須要一個方法卻可完成,首先把須要上傳的文件信息發送到服務器,當服務器確認後不停地把文件塊信息輸送到服務端便可.

使用測試

下載代碼

FileTransfer.Lib.rar (644.64 kb)

相關文章
相關標籤/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公眾號
   歡迎關注本站公眾號,獲取更多信息
聯繫我們 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程