hadoop MapReduce 的DBOutputFormat 使用體會

 

試試在DBOutputFormat 的Reducer 階段有幾十萬的記錄要插入,就會報錯內存溢出。java

能夠嘗試 在mapred-site.xml 裏面修改  orm


<property>
<name>mapred.child.java.opts</name>
<value>-Xmx512m</value>
<description>Java opts for the task tracker child processes.
The following symbol, if present, will be interpolated: @taskid@ is replaced
by current TaskID. Any other occurrences of '@' will go unchanged.
For example, to enable verbose gc logging to a file named for the taskid in
/tmp and to set the heap maximum to be a gigabyte, pass a 'value' of:
-Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc

The configuration variable mapred.child.ulimit can be used to control the
maximum virtual memory of the child processes.
</description>
</property>xml

相關文章
相關標籤/搜索