Swoole大數據量傳輸解決方案

Swoole大數據量傳輸解決方案

問題：要不壓縮的狀況下，在Swoole TCP中傳輸大量數據length>1M
首先用node的mockjs生成一個大於1M的文件

var jsonfile = require('jsonfile')
	var file = './data.json'
	let fs = require("fs");
	var Mock = require('mockjs')
	
	function injectContentTofileUntil(fileSizeInMega) {
	    let obj = jsonfile.readFileSync(file)
	    console.log(obj)
	    if(!obj){
	        obj=[]
	    }
	
	    for (let index = 0; index < 1000; index++) {
	        obj.push({
	            name: Mock.Random.cname(),
	            country: Mock.Random.county(true),
	        })
	    }
	
	    jsonfile.writeFileSync(file, obj)
	    let fileSizeInMegabytes = getFileSizeInMega(file)
	    if (fileSizeInMegabytes >= fileSizeInMega) {
	        return true
	    } else {
	        return false
	    }
	}
	
	function getFileSizeInMega(filePath) {
	    let stats = fs.statSync(file)
	    let fileSizeInBytes = stats.size
	    let fileSizeInMegabytes = fileSizeInBytes / 1000000.0
	    return fileSizeInMegabytes
	}
	
	while (!injectContentTofileUntil(1)) {
	    console.log(getFileSizeInMega(file));
	}

該文件爲大於1M的無序文件，在不使用壓縮的狀況下傳輸該文件到server端。

1.正常起server和client傳輸

Server.php

error_reporting(0);
	$client = new swoole_client(SWOOLE_SOCK_TCP, SWOOLE_SOCK_ASYNC);
	$filePath = realpath(__DIR__ . '/./jstools/data.json');
	$fileContent = file_get_contents($filePath);
	
	$client->on("connect", function ($cli) {
	    global $fileContent;
	    $cli->send($fileContent);
	});
	$client->on("receive", function ($cli, $data) {
	    echo "received: {$data}\n";
	});
	$client->on("error", function ($cli) {
	    echo "connect failed\n";
	});
	$client->on("close", function ($cli) {
	    echo "connection close\n";
	});
	$client->connect("127.0.0.1", 8091, 0.5);

Client.php

$server = new swoole_server("0.0.0.0", 8081);
	$server->on('connect', function ($server, $fd){
	    echo "connection open: {$fd}\n";
	});
	$server->on('receive', function ($server, $fd, $reactor_id, $data) {
	    echo "received:" . $data;
	    $server->close($fd);
	    file_put_contents('./response.json', $data);
	});
	$server->on('close', function ($server, $fd) {
	    echo "connection close: {$fd}\n";
	});
	$server->start();

執行命令
php server.php
php client.php
報錯:
WARNING swReactor_write (ERROR 1201): data is too large, cannot exceed buffer size.
緣由:
buffer不足溢出

2.修改buffer不足問題

設置client最大buffer:
"socketbuffersize"=>210241024,

報錯:
2018-05-18 17:46:09 *6146.0 ERROR swWorker_discard_data (ERROR 1007): 1received the wrong data8180 bytes from socket#1

緣由:
當swoole接收到知足buffer大小的內容時會自動結束傳輸，已經把數據給消費者，可是數據並無傳輸完成，此時有幾種方案解決該問題：

• 1.壓縮
• 2.告訴swoole到達buffer大小時還沒傳輸結束，本文只討論此方案

3.實現數據傳輸切分

server設置：

'package_eof' => "\r\n\r\n",
	'open_eof_split' => true,

client設置：

'package_eof' => "\r\n\r\n",

能夠正常傳輸。

4.此方案優缺點：

優勢：能夠正常傳輸任意大小數據
缺點：須要手動給流增長eof
字段含義請自行查閱swoole文檔。

代碼地址：https://gitee.com/sharpcx/swoole_big_str