sparksql parquet 分區推斷Partition Discovery

網上找的大部分資料都很舊,最後翻了下文檔只找到了說明

大概意思是1.6以後若是想要使用分區推斷就要設置數據源的basePath,所以代碼以下

java

 1 public class ParitionInfer {  2     private static SparkConf conf = new SparkConf().setAppName("partitioninfer");  3     private static JavaSparkContext jsc = new JavaSparkContext(conf);  4     private static SparkSession session = new SparkSession(jsc.sc());  5 
 6     public static void main(String[] args) {  7 
 8         // 加載數據源
 9         Dataset<Row> dataset = session.read().option("basePath", "hdfs://hadoop002:9000/spark_sql/users/") 10                 .parquet("hdfs://hadoop002:9000/spark_sql/users/gender=male/country=us/users.parquet"); 11 
12  dataset.printSchema(); 13  dataset.show(); 14 
15  session.stop(); 16  jsc.close(); 17  } 18 }

(注意basePath與實際的parquet文件的路徑,basePath是分區推斷列以前的路徑)

scala

 1 object PartitionInfer {  2   def main(args: Array[String]): Unit = {  3     val conf = new SparkConf().setAppName("partitioninfer").setMaster("local")  4     val sc = new SparkContext(conf)  5     val sqlContext = new SQLContext(sc)  6 
 7     val df = sqlContext.read.option("basePath", "f:/").parquet("f:/gender=male/country=us/users.parquet")  8 
 9  df.printSchema() 10  df.show() 11  } 12 }

scala的版本用的本地路徑,測試發現依然能夠實現分區推斷