SparkPlanner — Default Query Planner (with no Hive Support)
SparkPlanner is a concrete QueryPlanner (indirectly through extending SparkStrategies) that allows for plugging in a collection of additional SparkStrategy transformations.
| SparkStrategy | Description |
|---|---|
|
|
|
|
|
SparkPlanner requires a SparkContext, a SQLConf, and a collection of Strategy objects (as extraStrategies) when created.
SparkPlanner defines numPartitions method that returns the value of spark.sql.shuffle.partitions for the number of partitions to use for joins and aggregations. It is later used in BasicOperators strategy with RepartitionByExpression logical operator.
The required strategies collection uses extraStrategies extension point (defined as the argument to the constructor) and the predefined collection of Strategy objects.
collectPlaceholders required method returns a collection of PlanLater and the corresponding logical plans.
prunePlans required method does nothing, i.e. it returns what it gets directly.
|
Note
|
The order of the
|