Skip to content

Commit

Permalink
Add section
Browse files Browse the repository at this point in the history
  • Loading branch information
Fokko committed Jul 7, 2023
1 parent 2b4224e commit eade9e0
Showing 1 changed file with 10 additions and 4 deletions.
14 changes: 10 additions & 4 deletions website/docs/docs/core/connect-data-platform/spark-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,9 +119,7 @@ your_profile_name:
port: [port] # default 443
user: [user]
server_side_parameters:
# cluster configuration parameters, otherwise applied via `SET` statements
# for example:
# "spark.databricks.delta.schema.autoMerge.enabled": True
"spark.driver.memory": "4g"
```
</File>
Expand All @@ -148,6 +146,8 @@ your_profile_name:
auth: [e.g. KERBEROS]
kerberos_service_name: [e.g. hive]
use_ssl: [true|false] # value of hive.server2.use.SSL, default false
server_side_parameters:
"spark.driver.memory": "4g"
```

</File>
Expand Down Expand Up @@ -176,6 +176,8 @@ your_profile_name:
user: [user]
connect_timeout: 60 # default 10
connect_retries: 5 # default 0
server_side_parameters:
"spark.driver.memory": "4g"
```

</File>
Expand All @@ -202,7 +204,7 @@ your_profile_name:
schema: [database/schema name]
host: NA # not used, but required by `dbt-core`
server_side_parameters:
"spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
"spark.driver.memory": "4g"
```
</File>
Expand Down Expand Up @@ -231,6 +233,10 @@ connect_retries: 3

</VersionBlock>

### Server side configuration

Spark can be customized using [Application Properties](https://spark.apache.org/docs/latest/configuration.html). Using these properties the execution can be customized, for example, to allocate more memory to the driver process. Also, the Spark SQL runtime can be set through these properties. For example, this allows the user to [set a Spark catalogs](https://spark.apache.org/docs/latest/configuration.html#spark-sql).

## Caveats

### Usage with EMR
Expand Down

0 comments on commit eade9e0

Please sign in to comment.