#database/olap #arrow # Overview - 分为 scheduler 和 executor 两类进程。 - SQL 和 DataFrame 查询可以通过 Python 和 Rust 提交。 - SQL 查询也可以通过 Arrow Flight SQL JDBC driver 提交。 - scheduler 有一个 web ui 来观察运行状态。 - client、scheduler、executor 之间都使用 [Flight SQL Protocol](https://arrow.apache.org/blog/2022/02/16/introducing-arrow-flight-sql/)通信。 ------- # Deployment ## Scheduler - REST API： |API|Method|Description| |---|---|---| |/api/jobs|GET|Get a list of jobs that have been submitted to the cluster.| |/api/job/{job_id}|GET|Get a summary of a submitted job.| |/api/job/{job_id}/dot|GET|Produce a query plan in DOT (graphviz) format.| |/api/job/{job_id}|PATCH|Cancel a currently running job| -------- # Configuration |key|type|default|description| |---|---|---|---| |ballista.job.name|Utf8|N/A|Sets the job name that will appear in the web user interface for any submitted jobs.| |**ballista.shuffle.partitions**|UInt16|16|Sets the default number of partitions to create when repartitioning query stages.| |ballista.batch.size|UInt16|8192|Sets the default batch size.| |ballista.repartition.joins|Boolean|true|When set to true, Ballista will repartition data using the join keys to execute joins in parallel using the provided `ballista.shuffle.partitions` level.| |ballista.repartition.aggregations|Boolean|true|When set to true, Ballista will repartition data using the aggregate keys to execute aggregates in parallel using the provided `ballista.shuffle.partitions` level.| |ballista.repartition.windows|Boolean|true|When set to true, Ballista will repartition data using the partition keys to execute window functions in parallel using the provided `ballista.shuffle.partitions` level.| |ballista.parquet.pruning|Boolean|true|Determines whether Parquet pruning should be enabled or not.| |ballista.with_information_schema|Boolean|true|Determines whether the `information_schema` should be created in the context. This is necessary for supporting DDL commands such as `SHOW TABLES`.| |ballista.plugin_dir|Boolean|true|Specified a path for plugin files. Dynamic library files in this directory will be loaded when scheduler state initializes.| ---- # Tuning Guide ## Partitions and Parallelism - Ballista currently treats **each file within a table as a single partition** (in the future, Ballista will support splitting files into partitions but this is not implemented yet). - Executor 目前还不支持 spill-to-disk ## Viewing Query Plans and Metrics - 下载 dot 格式的查询计划图 ```sh http://localhost:50050/api/job/{job_id}/dot ``` - 使用 graphviz 转换成图片： ```sh dot -Tpng query.dot > query.png ```