Netty學習筆記（一）：接收nodejs模擬表單上傳的文件

時間 2019-11-09

標籤 netty 學習筆記接收 nodejs 模擬表單上傳文件欄目 Netty 简体版

原文原文鏈接

很久不寫博客了，也很久不寫代碼了，這兩天臨時趕上一個事情，以爲不難，加上以爲手有些生，就動手作了一下，結果趕上了很多坑，有新坑，有老坑，痛苦無比，如今總算差很少了，趕忙記錄下來，但願之後再也不重複這種痛苦。javascript

事情很簡單，用nodejs模擬表單提交，上傳文件到netty服務器。php

一、netty的參考資料不少，目前有netty3,netty4兩個版本，netty5出到alpha 2版本，不知道怎麼的，就不更新了，官網也註明不支持了，因此我採用的是netty4.1.19版，目前最新的。
參考的資料大體以下html

1）http://netty.io/wiki/index.html，官方的文檔，都寫的很經典，值得學習，裏面的例子snoop對我幫助很大java

2）https://www.programcreek.com/，一個示例代碼的網站。node

netty的代碼基本都是照抄第二個網站的內容，具體地址是https://www.programcreek.com/java-api-examples/index.php?source_dir=netty4.0.27Learn-master/example/src/main/java/io/netty/example/http/upload/HttpUploadServer.java。共有三個文件，HttpUploadServer.java，HttpUploadServerHandler.java，HttpUploadServerInitializer.java
二、nodejs自己比較簡單，但也花了很多時間研究。上傳文件可選的組件也不少，有form-data，request甚至官方的API，使用起來都不復雜，原本選擇的是form-data，可是用起來也遇到了很多問題，最終使用的仍是request，request使用起來很是簡單，我主要參考了以下內容。
1）http://www.open-open.com/lib/view/open1435301679966.html 中文的，介紹的比較詳細。
2）https://github.com/request/request 這是官方網站，內容最全，最權威。git

三、詳細環境
1）Windows 10專業版
2）Spring Tool Suite 3.9.1，其實用eclipse也能夠
3）Netty 4.1.19
4）Nodejs 8.9.3
四、目標
1）Netty程序
a）同時支持post、get方法。
b）將cookie、get參數和post參數保存到map裏，若是是文件上傳，則將其保存到臨時目錄，返回web地址，供客戶訪問。
2）nodejs
a）同時支持get、post方法。
b）能夠設置cookie，由於上傳文件確定是須要登陸的，sessionID通常是保存在cookie裏面。
五、預期思路
1）先解決netty的服務端問題，客戶端先用瀏覽器測試。
2）再解決nodejs的問題。
六、解決過程和踩的坑
1）Netty
a）Netty編程自己不難，可是相對來講要底層一些，若是常常作web開發的人，可能容易困惑，但熟悉一下就行了。
通常來講，netty服務端程序分爲三個程序，以下
Server：啓動線程，保定端口。
Initializer：初始化流處理器，即將接收到的字節流先進行編碼，造成對象，供後續解碼器處理，咱們須要關注的東西很少，在這個程序裏，咱們拿到手的已是解析好的http對象了，只要按照咱們的思路處理就能夠了。
Handler：是咱們本身的邏輯，在這個例子裏就是解析對象，造成map，將文件保存到磁盤上而已。
b）首先是pom文件，以下github

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>io-netty</groupId>
	<artifactId>io-netty-example</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<packaging>jar</packaging>

	<name>io-netty-example</name>
	<url>http://maven.apache.org</url>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
	</properties>

	<dependencies>
		<!-- https://mvnrepository.com/artifact/io.netty/netty-codec-http -->
		<dependency>
			<groupId>io.netty</groupId>
			<artifactId>netty-all</artifactId>
			<version>4.1.19.Final</version>
		</dependency>
		<dependency>
			<groupId>com.alibaba</groupId>
			<artifactId>fastjson</artifactId>
			<version>1.2.44</version>
		</dependency>
		<dependency>
			<groupId>junit</groupId>
			<artifactId>junit</artifactId>
			<version>3.8.1</version>
			<scope>test</scope>
		</dependency>
	</dependencies>
</project>

c）Server很是簡單，代碼也很少，以下web

package io.netty.example.http.upload; import io.netty.bootstrap.ServerBootstrap; import io.netty.channel.Channel; import io.netty.channel.EventLoopGroup; import io.netty.channel.nio.NioEventLoopGroup; import io.netty.channel.socket.nio.NioServerSocketChannel; import io.netty.handler.logging.LogLevel; import io.netty.handler.logging.LoggingHandler; import io.netty.handler.ssl.SslContext; import io.netty.handler.ssl.util.SelfSignedCertificate; /** * A HTTP server showing how to use the HTTP multipart package for file uploads and decoding post data. */ 
public final class HttpUploadServer { static final boolean SSL = System.getProperty("ssl") != null; static final int PORT = Integer.parseInt(System.getProperty("port", SSL? "8443" : "8090")); public static void main(String[] args) throws Exception { // Configure SSL. 
        final SslContext sslCtx; if (SSL) { SelfSignedCertificate ssc = new SelfSignedCertificate(); sslCtx = SslContext.newServerContext(ssc.certificate(), ssc.privateKey()); } else { sslCtx = null; } EventLoopGroup bossGroup = new NioEventLoopGroup(1); EventLoopGroup workerGroup = new NioEventLoopGroup(); try { ServerBootstrap b = new ServerBootstrap(); b.group(bossGroup, workerGroup); b.channel(NioServerSocketChannel.class); b.handler(new LoggingHandler(LogLevel.INFO)); b.childHandler(new HttpUploadServerInitializer(sslCtx)); //調用Initializer Channel ch = b.bind(PORT).sync().channel(); System.err.println("Open your web browser and navigate to " + (SSL? "https" : "http") + "://127.0.0.1:" + PORT + '/'); ch.closeFuture().sync(); } finally { bossGroup.shutdownGracefully(); workerGroup.shutdownGracefully(); } } }

d）Initializer代碼，須要注意的是流處理器，express

/* * Copyright 2012 The Netty Project * * The Netty Project licenses this file to you under the Apache License, * version 2.0 (the "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at: * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the * License for the specific language governing permissions and limitations * under the License. */
package io.netty.example.http.upload; import io.netty.channel.ChannelInitializer; import io.netty.channel.ChannelPipeline; import io.netty.channel.socket.SocketChannel; import io.netty.handler.codec.http.HttpContentCompressor; import io.netty.handler.codec.http.HttpObjectAggregator; import io.netty.handler.codec.http.HttpRequestDecoder; import io.netty.handler.codec.http.HttpResponseEncoder; import io.netty.handler.ssl.SslContext; public class HttpUploadServerInitializer extends ChannelInitializer<SocketChannel> { private final SslContext sslCtx; public HttpUploadServerInitializer(SslContext sslCtx) { this.sslCtx = sslCtx; } @Override public void initChannel(SocketChannel ch) { ChannelPipeline pipeline = ch.pipeline(); if (sslCtx != null) { pipeline.addLast(sslCtx.newHandler(ch.alloc())); } pipeline.addLast(new HttpRequestDecoder()); //處理Request // Uncomment the following line if you don't want to handle HttpChunks. //pipeline.addLast(new HttpObjectAggregator(1048576)); //將對象組裝爲FullHttpRequest
        pipeline.addLast(new HttpResponseEncoder()); //處理Response // Remove the following line if you don't want automatic content compression.
        pipeline.addLast(new HttpContentCompressor()); //壓縮 pipeline.addLast(new HttpUploadServerHandler()); } }

這裏須要注意一點，採用HttpRequestDecoder處理器，會將一個Request對象解析成三個對象HttpRequest、HttpCotent、LastHttpContent，這三個對象大體是這樣的，HttpRequest是地址信息和頭部信息，其中包括get方式傳送的參數和cookie信息；HttpContent是消息體，即Body部分，即post方式form提交的內容；LastHttpContent則是消息體的末尾，即提示消息體結束，也就是整個請求結束。apache

可是須要注意的是，使用HttpObjectAggregator處理器，能夠將Request對象處理爲FullRequest，但我測試了一下，不知道爲何，居然卡死了，因此只好用這種笨辦法，之後研究一下，此次先這樣吧。

e）Handler的代碼有些長，不過仍是貼出來吧。

/* * Copyright 2012 The Netty Project * * The Netty Project licenses this file to you under the Apache License, * version 2.0 (the "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at: * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the * License for the specific language governing permissions and limitations * under the License. */
package io.netty.example.http.upload; import io.netty.buffer.ByteBuf; import io.netty.channel.Channel; import io.netty.channel.ChannelFuture; import io.netty.channel.ChannelFutureListener; import io.netty.channel.ChannelHandlerContext; import io.netty.channel.SimpleChannelInboundHandler; import io.netty.handler.codec.http.Cookie; import io.netty.handler.codec.http.CookieDecoder; import io.netty.handler.codec.http.DefaultFullHttpResponse; import io.netty.handler.codec.http.FullHttpResponse; import io.netty.handler.codec.http.HttpContent; import io.netty.handler.codec.http.HttpHeaders; import io.netty.handler.codec.http.HttpMethod; import io.netty.handler.codec.http.HttpObject; import io.netty.handler.codec.http.HttpRequest; import io.netty.handler.codec.http.HttpResponseStatus; import io.netty.handler.codec.http.HttpVersion; import io.netty.handler.codec.http.LastHttpContent; import io.netty.handler.codec.http.QueryStringDecoder; import io.netty.handler.codec.http.ServerCookieEncoder; import io.netty.handler.codec.http.multipart.Attribute; import io.netty.handler.codec.http.multipart.DefaultHttpDataFactory; import io.netty.handler.codec.http.multipart.DiskAttribute; import io.netty.handler.codec.http.multipart.DiskFileUpload; import io.netty.handler.codec.http.multipart.FileUpload; import io.netty.handler.codec.http.multipart.HttpDataFactory; import io.netty.handler.codec.http.multipart.HttpPostRequestDecoder; import io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.EndOfDataDecoderException; import io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.ErrorDataDecoderException; import io.netty.handler.codec.http.multipart.InterfaceHttpData; import io.netty.handler.codec.http.multipart.InterfaceHttpData.HttpDataType; import io.netty.util.CharsetUtil; import java.io.File; import java.io.IOException; import java.net.URI; import java.nio.file.Files; import java.nio.file.Paths; import java.util.Collections; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Map.Entry; import java.util.Set; import java.util.UUID; import java.util.logging.Level; import java.util.logging.Logger; import com.alibaba.fastjson.JSON; import static io.netty.buffer.Unpooled.*; import static io.netty.handler.codec.http.HttpHeaders.Names.*; public class HttpUploadServerHandler extends SimpleChannelInboundHandler<HttpObject> { private static final Logger logger = Logger.getLogger(HttpUploadServerHandler.class.getName()); private HttpRequest request; private boolean readingChunks; private final StringBuilder responseContent = new StringBuilder(); private static final HttpDataFactory factory = new DefaultHttpDataFactory(DefaultHttpDataFactory.MINSIZE); // Disk // if // size // exceed

    private HttpPostRequestDecoder decoder; //////     private String tempPath = "d:/upload/"; //文件保存目錄 private String url_path = "http://localhost/upload/"; //文件臨時web目錄 private String errorJson; private Map<String, Object> mparams = new HashMap<>(); //將參數保存到map裏面 static { DiskFileUpload.deleteOnExitTemporaryFile = true; // should delete file // on exit (in normal // exit)
        DiskFileUpload.baseDirectory = null; // system temp directory
        DiskAttribute.deleteOnExitTemporaryFile = true; // should delete file on // exit (in normal exit)
        DiskAttribute.baseDirectory = null; // system temp directory
 } @Override public void channelUnregistered(ChannelHandlerContext ctx) throws Exception { if (decoder != null) { decoder.cleanFiles(); } } 
 //處理輸入對象，會執行三次，分別是HttpRequest、HttpContent、LastHttpContent @Override public void channelRead0(ChannelHandlerContext ctx, HttpObject msg) throws Exception { if (msg instanceof HttpRequest) { HttpRequest request = this.request = (HttpRequest) msg; URI uri = new URI(request.getUri()); if (!uri.getPath().equals("/formpostmultipart")) { errorJson = "{code:-1}"; writeError(ctx, errorJson); return; } // new getMethod // for (Entry<String, String> entry : request.headers()) { // responseContent.append("HEADER: " + entry.getKey() + '=' + entry.getValue() + // "\r\n"); // } // new getMethod
            Set<Cookie> cookies; String value = request.headers().get(COOKIE); if (value == null) { cookies = Collections.emptySet(); } else { cookies = CookieDecoder.decode(value); } for (Cookie cookie : cookies) { mparams.put(cookie.getName(), cookie.getValue()); } // add
 System.out.println(JSON.toJSONString(mparams)); QueryStringDecoder decoderQuery = new QueryStringDecoder(request.getUri()); Map<String, List<String>> uriAttributes = decoderQuery.parameters(); // add
 mparams.putAll(uriAttributes); System.out.println(JSON.toJSONString(mparams)); // for (Entry<String, List<String>> attr: uriAttributes.entrySet()) { // for (String attrVal: attr.getValue()) { // responseContent.append("URI: " + attr.getKey() + '=' + attrVal + "\r\n"); // } // } // responseContent.append("\r\n\r\n");

            if (request.getMethod().equals(HttpMethod.GET)) { // GET Method: should not try to create a HttpPostRequestDecoder // So stop here // responseContent.append("\r\n\r\nEND OF GET CONTENT\r\n"); // Not now: LastHttpContent will be sent writeResponse(ctx.channel());
                return; } try { decoder = new HttpPostRequestDecoder(factory, request); } catch (ErrorDataDecoderException e1) { e1.printStackTrace(); responseContent.append(e1.getMessage()); writeResponse(ctx.channel()); ctx.channel().close(); errorJson = "{code:-2}"; writeError(ctx, errorJson); return; } readingChunks = HttpHeaders.isTransferEncodingChunked(request); // responseContent.append("Is Chunked: " + readingChunks + "\r\n"); // responseContent.append("IsMultipart: " + decoder.isMultipart() + "\r\n");
            if (readingChunks) { // Chunk version // responseContent.append("Chunks: ");
                readingChunks = true; } } // check if the decoder was constructed before // if not it handles the form get
        if (decoder != null) { if (msg instanceof HttpContent) { // New chunk is received
                HttpContent chunk = (HttpContent) msg; try { decoder.offer(chunk); } catch (ErrorDataDecoderException e1) { e1.printStackTrace(); // responseContent.append(e1.getMessage());
 writeResponse(ctx.channel()); ctx.channel().close(); errorJson = "{code:-3}"; writeError(ctx, errorJson); return; } // responseContent.append('o'); // example of reading chunk by chunk (minimize memory usage due to // Factory)
 readHttpDataChunkByChunk(ctx); // example of reading only if at the end
                if (chunk instanceof LastHttpContent) { writeResponse(ctx.channel()); readingChunks = false; reset(); } } } else { writeResponse(ctx.channel()); } } private void reset() { request = null; // destroy the decoder to release all resources
 decoder.destroy(); decoder = null; } /** * Example of reading request by chunk and getting values from chunk to chunk * * @throws IOException */
 //處理post數據
    private void readHttpDataChunkByChunk(ChannelHandlerContext ctx) throws IOException { try { while (decoder.hasNext()) { InterfaceHttpData data = decoder.next(); if (data != null) { try { // new value
 writeHttpData(ctx, data); } finally { data.release(); } } } } catch (EndOfDataDecoderException e1) { // end // responseContent.append("\r\n\r\nEND OF CONTENT CHUNK BY CHUNK\r\n\r\n");
            mparams.put("code", "-2"); } } //解析post屬性，保存文件，寫入map private void writeHttpData(ChannelHandlerContext ctx, InterfaceHttpData data) throws IOException { if (data.getHttpDataType() == HttpDataType.Attribute) { Attribute attribute = (Attribute) data; String value; try { value = attribute.getValue(); } catch (IOException e1) { // Error while reading data from File, only print name and error
 e1.printStackTrace(); // responseContent.append("\r\nBODY Attribute: " + // attribute.getHttpDataType().name() + ": " // + attribute.getName() + " Error while reading value: " + e1.getMessage() + // "\r\n");
                errorJson = "{code:-4}"; writeError(ctx, errorJson); return; } mparams.put(attribute.getName(), attribute.getValue()); System.out.println(JSON.toJSONString(mparams)); } else { if (data.getHttpDataType() == HttpDataType.FileUpload) { FileUpload fileUpload = (FileUpload) data; if (fileUpload.isCompleted()) { System.out.println(fileUpload.length()); if (fileUpload.length() > 0) { String orign_name = fileUpload.getFilename(); String file_name = UUID.randomUUID() + "."
                                + orign_name.substring(orign_name.lastIndexOf(".") + 1); fileUpload.renameTo(new File(tempPath + file_name)); mparams.put(data.getName(), url_path + file_name); System.out.println(JSON.toJSONString(mparams)); } } else { errorJson = "{code:-5}"; writeError(ctx, errorJson); } } } } //寫入response，返回給客戶 private void writeResponse(Channel channel) { // Convert the response content to a ChannelBuffer.
        ByteBuf buf = copiedBuffer(JSON.toJSONString(mparams), CharsetUtil.UTF_8); responseContent.setLength(0); // Decide whether to close the connection or not.
        boolean close = HttpHeaders.Values.CLOSE.equalsIgnoreCase(request.headers().get(CONNECTION)) || request.getProtocolVersion().equals(HttpVersion.HTTP_1_0) && !HttpHeaders.Values.KEEP_ALIVE.equalsIgnoreCase(request.headers().get(CONNECTION)); // Build the response object.
        FullHttpResponse response = new DefaultFullHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK, buf); response.headers().set(CONTENT_TYPE, "text/plain; charset=UTF-8"); if (!close) { // There's no need to add 'Content-Length' header // if this is the last response.
 response.headers().set(CONTENT_LENGTH, buf.readableBytes()); } Set<Cookie> cookies; String value = request.headers().get(COOKIE); if (value == null) { cookies = Collections.emptySet(); } else { cookies = CookieDecoder.decode(value); } if (!cookies.isEmpty()) { // Reset the cookies if necessary.
            for (Cookie cookie : cookies) { response.headers().add(SET_COOKIE, ServerCookieEncoder.encode(cookie)); } } // Write the response.
        ChannelFuture future = channel.writeAndFlush(response); // Close the connection after the write operation is done if necessary.
        if (close) { future.addListener(ChannelFutureListener.CLOSE); } } //返回錯誤信息，也是寫入response private void writeError(ChannelHandlerContext ctx, String errorJson) { ByteBuf buf = copiedBuffer(errorJson, CharsetUtil.UTF_8); // Build the response object.
        FullHttpResponse response = new DefaultFullHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.OK, buf); response.headers().set(CONTENT_TYPE, "text/html; charset=UTF-8"); response.headers().set(CONTENT_LENGTH, buf.readableBytes()); // Write the response.
 ctx.channel().writeAndFlush(response); } @Override public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception { logger.log(Level.WARNING, responseContent.toString(), cause); ctx.channel().close(); } }

雖然代碼不少，可是最須要注意的只有四個方法：

channelRead0(ChannelHandlerContext ctx, HttpObject msg)：處理輸入內容，會執行三次，分別是HttpRequest、HttpContent、LastHttpContent，依次處理。

readHttpDataChunkByChunk(ChannelHandlerContext ctx)：解析HttpContent時調用，即消息體時，具體執行過程在函數writeHttpData中

writeResponse(Channel channel)：寫入response，這裏調用了fastjson將map轉換爲json字符串。

f）上傳的html文件

<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<title>HTML5的標題</title>
</head>
<body>
    <form action="http://127.0.0.1:8090/formpostmultipart?a=張三&b=李四" method="post" enctype="multipart/form-data">
       <input type="text" name="name" value="shiyq"/>
       <br>
       <input type="text" name="name" value="歷史地理"/>
       <br>
       <input type=file name="file"/>
       <br>
       <input type="submit" value="上傳"/>
    </form>
</body>
</html>

g）啓動HttpUploadServer，而後在瀏覽器裏訪問upload.html，返回結果以下

{
    "name": "歷史地理",
    "a": [
        "張三"
    ],
    "b": [
        "李四"
    ],
    "file": "http://localhost/upload/13d45df8-d6c7-4a7a-8f21-0251efeca240.png"
}

這裏要注意的是，地址欄傳遞的參數是個數組，即參數名能夠重複，form裏面的值不能夠，只能是一個。

2）NodeJS

nodejs相對要簡單一些，可是也更讓人困惑，主要遇到了兩個問題。

a）請求地址包含中文的狀況，這個實際上是個老問題，很容易解決，可是卻卡住了半天，看來好久不寫程序就是不行啊。最後的解決辦法就是進行url編碼。

b）cookie設置的問題，form-data模塊沒有說明cookie設置的問題，官方API的request也言之不詳，幸虧request寫的比較明白，可是默認還不開啓，須要設置，仍是老話，三天不寫程序手就生了。

c）環境很是簡單，只須要安裝request模塊就能夠了，命令爲npm install request，儘可能不要裝在全局，在個人Windows 10上出現找不到模塊的現象，最後安裝到當前目錄才解決，最後的代碼以下

var fs = require('fs'); var request = require('request').defaults({jar:true}); //不要忘記npm install request,不要忘記設置jar:true,不然沒法設置cookied var file_path="D:/Documents/IMG_20170427_121431.jpg"
var formData = { name:"路送雙", code:"tom", my_file:fs.createReadStream(file_path) } var url = encodeURI("http://localhost:8090/formpostmultipart?a=王二&a=張三&b=李四");//對中文編碼 var j = request.jar(); var cookie = request.cookie('key1=value1'); var cookie1 = request.cookie('key2=value2'); j.setCookie(cookie, url); j.setCookie(cookie1, url); request.post({url:url, jar:j, formData: formData}, function optionalCallback(err, httpResponse, body) { if (err) { return console.error('upload failed:', err); } console.log( body); });

須要注意cookie的設置，不只須要設置jar屬性爲true，還須要調用屢次setCookie，還須要在request.post中指定參數，挺麻煩的。

d）返回結果以下

{ "key1": "value1", "key2": "value2", "a": [ "王二", "張三" ], "b": [ "李四" ], "code": "tom", "my_file": "http://localhost/upload/8d8e2f9f-7513-4844-9614-0d7fb7a33a6e.jpg", "name": "路送雙" }