Hadoop MapReduce Task Log 沒法查看syslog問題

現象：

因爲多個map task共用一個JVM，因此只輸出了一組log文件

datanode01:/data/hadoop-x.x.x/logs/userlogs$ ls -R .: attempt_201211220735_0001_m_000000_0  attempt_201211220735_0001_m_000002_0  attempt_201211220735_0001_m_000005_0 attempt_201211220735_0001_m_000001_0  attempt_201211220735_0001_m_000003_0 ./attempt_201211220735_0001_m_000000_0: log.index ./attempt_201211220735_0001_m_000001_0: log.index ./attempt_201211220735_0001_m_000002_0: log.index  stderr  stdout  syslog

 

經過http://xxxxxxxx:50060/tasklog?attemptid= attempt_201211220735_0001_m_000000_0 獲取task的日誌時，會出現syslog沒法獲取

 

緣由：

1.TaskLogServlet.doGet()方法

 

if (filter == null) {
        printTaskLog(response, out, attemptId,start, end, plainText,
                     TaskLog.LogName.STDOUT,isCleanup);
        printTaskLog(response, out, attemptId,start, end, plainText,
                     TaskLog.LogName.STDERR,isCleanup);
        if(haveTaskLog(attemptId, isCleanup, TaskLog.LogName.SYSLOG)) {
          printTaskLog(response, out,attemptId, start, end, plainText,
              TaskLog.LogName.SYSLOG,isCleanup);
        }
        if(haveTaskLog(attemptId, isCleanup, TaskLog.LogName.DEBUGOUT)) {
          printTaskLog(response, out,attemptId, start, end, plainText,
                       TaskLog.LogName.DEBUGOUT, isCleanup);
        }
        if(haveTaskLog(attemptId, isCleanup, TaskLog.LogName.PROFILE)) {
          printTaskLog(response, out,attemptId, start, end, plainText,
                       TaskLog.LogName.PROFILE,isCleanup);
        }
      } else {
        printTaskLog(response, out, attemptId,start, end, plainText, filter,
                     isCleanup);
     }

 

 

嘗試將filter=SYSLOG參數加上，能夠訪問到syslog,但去掉就不行。

看了代碼多了一行

haveTaskLog(attemptId, isCleanup,TaskLog.LogName.SYSLOG)

判斷，跟進代碼發現，檢查的是原來

attempt_201211220735_0001_m_000000_0目錄下是否有syslog文件？

而不是從log.index找location看是否有syslog文件，一個bug出現了！

 

2.TaskLogServlet. printTaskLog方法

獲取日誌文件時會從log.index讀取。

 

InputStreamtaskLogReader =
       new TaskLog.Reader(taskId,filter, start, end, isCleanup);
TaskLog.Reader
public Reader(TaskAttemptIDtaskid, LogName kind,
                  long start,long end, boolean isCleanup) throwsIOException {
      // find the right log file
      Map<LogName, LogFileDetail>allFilesDetails =
         getAllLogsFileDetails(taskid, isCleanup);
 
static Map<LogName, LogFileDetail> getAllLogsFileDetails(
      TaskAttemptID taskid, booleanisCleanup) throws IOException {
 
    Map<LogName, LogFileDetail>allLogsFileDetails =
        newHashMap<LogName, LogFileDetail>();
 
    File indexFile = getIndexFile(taskid,isCleanup);
    BufferedReader fis;
    try {
      fis = newBufferedReader(new InputStreamReader(
        SecureIOUtils.openForRead(indexFile,obtainLogDirOwner(taskid))));
    } catch(FileNotFoundException ex) {
      LOG.warn("Index file for the log of " + taskid + " does not exist.");
 
      //Assume no task reuse is used and files exist on attemptdir
      StringBuffer input = newStringBuffer();
      input.append(LogFileDetail.LOCATION
                     + getAttemptDir(taskid,isCleanup) + "\n");
      for(LogName logName : LOGS_TRACKED_BY_INDEX_FILES) {
        input.append(logName + ":0 -1\n");
      }
      fis = newBufferedReader(new StringReader(input.toString()));
}
………………….

 

 

問題解決：

相似getAllLogsFileDetails同樣，先從log.index獲取日誌目錄獲取logdir，

 

 	private boolean haveTaskLog(TaskAttemptID taskId, boolean isCleanup,
			TaskLog.LogName type) throws IOException {
		File f = TaskLog.getTaskLogFile(taskId, isCleanup, type);
		if (f.exists() && f.canRead()) {
			return true;
		} else {
			File indexFile = TaskLog.getIndexFile(taskId, isCleanup);
			if (!indexFile.exists()) {
				return false;
			}


			BufferedReader fis;
			try {
				fis = new BufferedReader(new InputStreamReader(
						SecureIOUtils.openForRead(indexFile,
								TaskLog.obtainLogDirOwner(taskId))));
			} catch (FileNotFoundException ex) {
				LOG.warn("Index file for the log of " + taskId
						+ " does not exist.");


				// Assume no task reuse is used and files exist on attemptdir
				StringBuffer input = new StringBuffer();
				input.append(LogFileDetail.LOCATION
						+ TaskLog.getAttemptDir(taskId, isCleanup) + "\n");
				for (LogName logName : TaskLog.LOGS_TRACKED_BY_INDEX_FILES) {
					input.append(logName + ":0 -1\n");
				}
				fis = new BufferedReader(new StringReader(input.toString()));
			}


			try {
				String str = fis.readLine();
				if (str == null) { // thefile doesn't have anything
					throw new IOException("Index file for the log of " + taskId
							+ "is empty.");
				}
				String loc = str.substring(str.indexOf(LogFileDetail.LOCATION)
						+ LogFileDetail.LOCATION.length());
				File tf = new File(loc, type.toString());
				return tf.exists() && tf.canRead();


			} finally {
				if (fis != null)
					fis.close();
			}
		}


	}

 

 

從logdir中判斷是否有syslog。

 

Workaround:

查詢時加入在url上加入filter=SYSLOG就能夠看到,不須要修改代碼。