以前實現了使用流來說http和ftp的文件下載到本地,也實現了將本地文件上傳到hdfs上,那如今就能夠作到將
ftp和http的文件轉移到hdfs上了,而不用先將ftp和http的文件拷貝到本地再上傳到hdfs上了。其實這個東西的原理
很簡單,就是使用流,將ftp或http的文件讀入到流中,而後將流中的內容傳送到hdfs上,這樣子就不用讓數據存到
本地的硬盤上了,只是讓內存來完成這個轉移的過程,但願這個工具,可以幫到有這樣需求的同窗~
這裏先附上以前的幾個工具的連接:java
代碼以下:segmentfault
import java.io.InputStream; import java.io.OutputStream; import java.io.IOException; public class FileTrans { private String head = ""; private String hostname = ""; private String FilePath = ""; private String hdfsFilePath = ""; private HDFSUtil hdfsutil = null; private FtpClient ftp; private HttpUtil http; public void setFilePath(String FilePath){ this.FilePath = FilePath; } public String getFilePath(String FilePath){ return this.FilePath; } public void sethdfsFilePath(String hdfsFilePath){ this.hdfsFilePath = hdfsFilePath; } public String gethdfsFilePath(String hdfsFilePath){ return this.hdfsFilePath; } public void setHostName(String hostname){ this.hostname = hostname; } public String getHostName(){ return this.hostname; } public void setHead(String head){ this.head = head; } public String getHead(){ return this.head; } public FileTrans(String head, String hostname, String filepath, String hdfsnode,String hdfsFilepath){ this.head = head; this.hostname = hostname; this.FilePath = filepath; this.hdfsFilePath = hdfsFilepath; if (head.equals("ftp") && hostname != ""){ this.ftp = new FtpClient(this.hostname); } if ((head.equals("http") || head .equals("https")) && hostname != ""){ String httpurl = head + "://" + hostname + "/" + filepath; this.http = new HttpUtil(httpurl); } if (hdfsnode != ""){ this.hdfsutil = new HDFSUtil(hdfsnode); } this.hdfsutil.setHdfsPath(this.hdfsFilePath); this.hdfsutil.setFilePath(hdfsutil.getHdfsNode()+hdfsutil.getHdfsPath()); this.hdfsutil.setHadoopSite("./hadoop-site.xml"); this.hdfsutil.setHadoopDefault("./hadoop-default.xml"); this.hdfsutil.setConfigure(false); } public static void main(String[] args) throws IOException{ String head = ""; String hostname = ""; String filepath = ""; String hdfsfilepath = ""; String hdfsnode = ""; String localpath = ""; InputStream inStream = null; int samplelines = 0; try{ head = args[0]; //遠端服務器類型,http仍是ftp hostname = args[1]; //遠端服務器hostname filepath = args[2]; //遠端文件路徑 hdfsnode = args[3]; //hdfs的機器名,不帶hdfs開頭 hdfsfilepath = args[4]; //hdfs的文件路徑 localpath = args[5]; //若是須要在本地保存一份的話,輸入本地的路徑,不保存,傳入空格或者samplelines傳入0 samplelines = Integer.parseInt(args[6]); //保存在本地的話,保存前N行,若是不保存,填0 }catch (Exception e){ System.out.println("[FileTrans]:input args error!"); e.printStackTrace(); } FileTrans filetrans = new FileTrans(head, hostname, filepath, hdfsnode,hdfsfilepath); if (filetrans == null){ System.out.println("filetrans null"); return; } if (filetrans.ftp == null && head.equals("ftp")){ System.out.println("filetrans ftp null"); return; } if (filetrans.http == null && (head.equals("http") || head.equals("https"))){ System.out.println("filetrans ftp null"); return; } try{ if (head.equals("ftp")){ inStream = filetrans.ftp.getStream(filepath); if (samplelines > 0){ filetrans.ftp.writeStream(inStream, localpath, samplelines); } } else{ inStream = filetrans.http.getStream(head + "://" + hostname + "/" + filepath); if (samplelines > 0){ filetrans.http.downLoad(head + "://" + hostname + "/" + filepath, localpath, samplelines); } } filetrans.hdfsutil.upLoad(inStream, filetrans.hdfsutil.getFilePath()); if (head == "ftp"){ filetrans.ftp.disconnect(); } }catch (IOException e){ System.out.println("[FileTrans]: file trans failed!"); e.printStackTrace(); } System.out.println("[FileTrans]: file trans success!"); } }
編譯有問題的話,在hadoop工具的那篇文章中有提到,能夠參考
注:最好將其餘三個工具的文件放在同一個目錄下,若是不放在一塊兒,那麼請自行引用服務器
這個工具既能夠將ftp或者http轉移到hdfs,也能將前N行保存到本地,進行分析工具