HDFS目標大文件夾下文件索引創建

時間 2019-11-13

原文原文鏈接

1、索引命令java

　　一、非分佈式，單機linux

1 　hadoop jar /usr/lib/hadoop/lib/hadoop-lzo.jar com.hadoop.compression.lzo.LzoIndexer ${path}

　　二、分佈式：任務提交到yarn，須要使用集羣資源　　shell

1 hadoop jar /usr/lib/hadoop/lib/hadoop-lzo.jar com.hadoop.compression.lzo.DistributedLzoIndexer  ${path}

　　注：可能遇到的狀況，在某些須要隊列或者有keyberos認證的機器，　DistributedLzoIndexer 可能不會那麼理想，須要添加隊列或者加上keyberos參數。app

2、執行linux shell命令java參考模板分佈式

 1 public class ExecCMD {
 2 
 3     private static final Logger log = LoggerFactory.getLogger(ExecCMD.class);
 4 
 5     public static String exec(String cmd) {
 6         try {
 7             String[] cmdA = { "/bin/sh", "-c", cmd };
 8             Process process = Runtime.getRuntime().exec(cmdA);
 9             LineNumberReader br = new LineNumberReader(new InputStreamReader(process.getInputStream()));
10             StringBuffer buff = new StringBuffer();
11             String line;
12             while ((line = br.readLine()) != null) {
13                 buff.append(line).append("\n");
14             }
15             String result = buff.toString();
16             if (!cmd.isEmpty())
17                 log.info("cmd executed, result: " + result);
18             return result;
19         } catch (Exception e) {
20             log.error("Failed to exec cmd: " + cmd, e);
21         }
22         return null;
23     }
24 }