Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update model SQLs for Teradata #6286

Merged
merged 4 commits into from
Oct 17, 2024
Merged

Update model SQLs for Teradata #6286

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
eacd73c
Update model SQL for Teradata
Oct 14, 2024
eaf89df
Update website/docs/guides/teradata-qs.md
adamtwo Oct 16, 2024
01c4822
Update website/docs/guides/teradata-qs.md
adamtwo Oct 16, 2024
cf92a0a
Merge branch 'current' into update-model-sql-for-teradata
adamtwo Oct 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
224 changes: 167 additions & 57 deletions website/docs/guides/teradata-qs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,52 +59,50 @@

:::

1. Use your preferred SQL IDE editor to create two databases: `jaffle_shop` and `stripe`:
1. Use your preferred SQL IDE editor to create the database, `jaffle_shop`:

```sql
CREATE DATABASE jaffle_shop AS PERM = 1e9;
CREATE DATABASE stripe AS PERM = 1e9;
```

2. In the databases `jaffle_shop` and `stripe`, create three foreign tables and reference the respective csv files located in object storage:

```sql
CREATE FOREIGN TABLE jaffle_shop.customers (
id integer,
first_name varchar (100),
last_name varchar (100)
)
USING (
LOCATION ('/s3/dbt-tutorial-public.s3.amazonaws.com/jaffle_shop_customers.csv')
)
NO PRIMARY INDEX;

CREATE FOREIGN TABLE jaffle_shop.orders (
id integer,
user_id integer,
order_date date,
status varchar(100)
)
USING (
LOCATION ('/s3/dbt-tutorial-public.s3.amazonaws.com/jaffle_shop_orders.csv')
)
NO PRIMARY INDEX;

CREATE FOREIGN TABLE stripe.payment (
id integer,
orderid integer,
paymentmethod varchar (100),
status varchar (100),
amount integer,
created date
)
USING (
LOCATION ('/s3/dbt-tutorial-public.s3.amazonaws.com/stripe_payments.csv')
)
NO PRIMARY INDEX;
```

## Connect dbt cloud to Teradata
2. In `jaffle_shop` database, create three foreign tables and reference the respective csv files located in object storage:

```sql
CREATE FOREIGN TABLE jaffle_shop.customers (
id integer,
first_name varchar (100),
last_name varchar (100),
email varchar (100)
)
USING (
LOCATION ('/gs/storage.googleapis.com/clearscape_analytics_demo_data/dbt/rawcustomers.csv')
)
NO PRIMARY INDEX;

CREATE FOREIGN TABLE jaffle_shop.orders (
id integer,
user_id integer,
order_date date,
status varchar(100)
)
USING (
LOCATION ('/gs/storage.googleapis.com/clearscape_analytics_demo_data/dbt/raw_orders.csv')
)
NO PRIMARY INDEX;

CREATE FOREIGN TABLE jaffle_shop.payments (
id integer,
orderid integer,
paymentmethod varchar (100),
amount integer
)
USING (
LOCATION ('/gs/storage.googleapis.com/clearscape_analytics_demo_data/dbt/raw_payments.csv')
)
NO PRIMARY INDEX;
```

## Connect dbt Cloud to Teradata

Check warning on line 105 in website/docs/guides/teradata-qs.md

View workflow job for this annotation

GitHub Actions / Vale linting 

[vale] reported by reviewdog 🐶
[custom.SentenceCaseHeaders] 'Connect dbt Cloud to Teradata' should use sentence-style capitalization. Try '' instead.

Raw Output:
{"message": "[custom.SentenceCaseHeaders] 'Connect dbt Cloud to Teradata' should use sentence-style capitalization. Try '' instead.", "location": {"path": "website/docs/guides/teradata-qs.md", "range": {"start": {"line": 105, "column": 4}}}, "severity": "WARNING"}

1. Create a new project in dbt Cloud. From **Account settings** (using the gear menu in the top right corner), click **New Project**.
2. Enter a project name and click **Continue**.
Expand Down Expand Up @@ -136,12 +134,10 @@
1. Click **Start developing in the IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse.
2. Above the file tree to the left, click **Initialize your project** to build out your folder structure with example models.
3. Make your initial commit by clicking **Commit and sync**. Use the commit message `initial commit` to create the first commit to your managed repo. Once you’ve created the commit, you can open a branch to add new dbt code.
4. You can now directly query data from your warehouse and execute `dbt run`. You can try this out now:
- Click **Create new file**, add this query to the new file, and click **Save as** to save the new file:
```sql
select * from jaffle_shop.customers
```
- In the command line bar at the bottom, enter `dbt run` and click **Enter**. You should see a `dbt run succeeded` message.

## Delete the example models

<Snippet path="quickstarts/delete-example-models" />

## Build your first model

Expand All @@ -153,7 +149,7 @@
Name the new branch `add-customers-model`.

1. Click the **...** next to the `models` directory, then select **Create file**.
2. Name the file `customers.sql`, then click **Create**.
2. Name the file `bi_customers.sql`, then click **Create**.
3. Copy the following query into the file and click **Save**.

```sql
Expand Down Expand Up @@ -208,7 +204,7 @@

from customers

left join customer_orders using (customer_id)
left join customer_orders on customers.customer_id = customer_orders.customer_id

)

Expand All @@ -222,19 +218,75 @@

## Change the way your model is materialized

<Snippet path="quickstarts/change-way-model-materialized" />
One of the most powerful features of dbt is that you can change the way a model is materialized in your warehouse, simply by changing a configuration value. You can change things between tables and views by changing a keyword rather than writing the data definition language (DDL) to do this behind the scenes.

## Delete the example models
By default, everything gets created as a view. You can override that at the directory level so everything in that directory will materialize to a different materialization.

<Snippet path="quickstarts/delete-example-models" />
1. Edit your `dbt_project.yml` file.
- Update your project `name` to:
<File name='dbt_project.yml'>

```yaml
name: 'jaffle_shop'
```

</File>
- Configure `jaffle_shop` so everything in it will be materialized as a table; and configure `example` so everything in it will be materialized as a view. Update your `models` config block to:

<File name='dbt_project.yml'>

```yaml
models:
jaffle_shop:
+materialized: table
example:
+materialized: view
```

</File>
- Click **Save**.

Check warning on line 247 in website/docs/guides/teradata-qs.md

View workflow job for this annotation

GitHub Actions / Vale linting 

[vale] reported by reviewdog 🐶
[custom.UIElements] UI elements like 'Save' should be bold.

Raw Output:
{"message": "[custom.UIElements] UI elements like 'Save' should be bold.", "location": {"path": "website/docs/guides/teradata-qs.md", "range": {"start": {"line": 247, "column": 15}}}, "severity": "WARNING"}

2. Enter the `dbt run` command. Your `bi_customers` model should now be built as a table!
:::info
To do this, dbt had to first run a `drop view` statement (or API call on BigQuery), then a `create table as` statement.
:::

3. Edit `models/bi_customers.sql` to override the `dbt_project.yml` for the `customers` model only by adding the following snippet to the top, and click **Save**:

Check warning on line 254 in website/docs/guides/teradata-qs.md

View workflow job for this annotation

GitHub Actions / Vale linting 

[vale] reported by reviewdog 🐶
[custom.UIElements] UI elements like 'Save' should be bold.

Raw Output:
{"message": "[custom.UIElements] UI elements like 'Save' should be bold.", "location": {"path": "website/docs/guides/teradata-qs.md", "range": {"start": {"line": 254, "column": 157}}}, "severity": "WARNING"}

<File name='models/bi_customers.sql'>

```sql
{{
config(
materialized='view'
)
}}

with customers as (

select
id as customer_id
...

)

```

</File>

4. Enter the `dbt run` command. Your model, `bi_customers`, should now build as a view.

### FAQs

Check warning on line 279 in website/docs/guides/teradata-qs.md

View workflow job for this annotation

GitHub Actions / Vale linting 

[vale] reported by reviewdog 🐶
[custom.SentenceCaseHeaders] 'FAQs' should use sentence-style capitalization. Try '' instead.

Raw Output:
{"message": "[custom.SentenceCaseHeaders] 'FAQs' should use sentence-style capitalization. Try '' instead.", "location": {"path": "website/docs/guides/teradata-qs.md", "range": {"start": {"line": 279, "column": 5}}}, "severity": "WARNING"}

<FAQ path="Models/available-materializations" />
<FAQ path="Project/which-materialization" />
<FAQ path="Models/available-configurations" />

## Build models on top of other models

<Snippet path="quickstarts/intro-build-models-atop-other-models" />

1. Create a new SQL file, `models/stg_customers.sql`, with the SQL from the `customers` CTE in your original query.
2. Create a second new SQL file, `models/stg_orders.sql`, with the SQL from the `orders` CTE in your original query.

<File name='models/stg_customers.sql'>

```sql
Expand All @@ -248,6 +300,7 @@

</File>

2. Create a second new SQL file, `models/stg_orders.sql`, with the SQL from the `orders` CTE in your original query.
<File name='models/stg_orders.sql'>

```sql
Expand All @@ -262,9 +315,9 @@

</File>

3. Edit the SQL in your `models/customers.sql` file as follows:
3. Edit the SQL in your `models/bi_customers.sql` file as follows:

<File name='models/customers.sql'>
<File name='models/bi_customers.sql'>

```sql
with customers as (
Expand Down Expand Up @@ -397,4 +450,61 @@

<Snippet path="quickstarts/test-and-document-your-project" />

<Snippet path="quickstarts/schedule-a-job" />
## Commit your changes

Now that you've built your customer model, you need to commit the changes you made to the project so that the repository has your latest code.

**If you edited directly in the protected primary branch:**<br />
1. Click the **Commit and sync git** button. This action prepares your changes for commit.
2. A modal titled **Commit to a new branch** will appear.
3. In the modal window, name your new branch `add-customers-model`. This branches off from your primary branch with your new changes.
4. Add a commit message, such as "Add customers model, tests, docs" and and commit your changes.

Check warning on line 461 in website/docs/guides/teradata-qs.md

View workflow job for this annotation

GitHub Actions / Vale linting 

[vale] reported by reviewdog 🐶
[custom.Repitition] 'and' is repeated!

Raw Output:
{"message": "[custom.Repitition] 'and' is repeated!", "location": {"path": "website/docs/guides/teradata-qs.md", "range": {"start": {"line": 461, "column": 69}}}, "severity": "WARNING"}
5. Click **Merge this branch to main** to add these changes to the main branch on your repo.


**If you created a new branch before editing:**<br />
1. Since you already branched out of the primary protected branch, go to **Version Control** on the left.
2. Click **Commit and sync** to add a message.
3. Add a commit message, such as "Add customers model, tests, docs."
4. Click **Merge this branch to main** to add these changes to the main branch on your repo.

## Deploy dbt

Use dbt Cloud's Scheduler to deploy your production jobs confidently and build observability into your processes. You'll learn to create a deployment environment and run a job in the following steps.

### Create a deployment environment

1. In the upper left, select **Deploy**, then click **Environments**.
2. Click **Create Environment**.
3. In the **Name** field, write the name of your deployment environment. For example, "Production."
4. In the **dbt Version** field, select the latest version from the dropdown.
5. Under **Deployment connection**, enter the name of the dataset you want to use as the target, such as "Analytics". This will allow dbt to build and work with that dataset. For some data warehouses, the target dataset may be referred to as a "schema".
6. Click **Save**.

Check warning on line 482 in website/docs/guides/teradata-qs.md

View workflow job for this annotation

GitHub Actions / Vale linting 

[vale] reported by reviewdog 🐶
[custom.UIElements] UI elements like 'Save' should be bold.

Raw Output:
{"message": "[custom.UIElements] UI elements like 'Save' should be bold.", "location": {"path": "website/docs/guides/teradata-qs.md", "range": {"start": {"line": 482, "column": 12}}}, "severity": "WARNING"}

### Create and run a job

Jobs are a set of dbt commands that you want to run on a schedule. For example, `dbt build`.

As the `jaffle_shop` business gains more customers, and those customers create more orders, you will see more records added to your source data. Because you materialized the `bi_customers` model as a table, you'll need to periodically rebuild your table to ensure that the data stays up-to-date. This update will happen when you run a job.

1. After creating your deployment environment, you should be directed to the page for a new environment. If not, select **Deploy** in the upper left, then click **Jobs**.
2. Click **Create one** and provide a name, for example, "Production run", and link to the Environment you just created.
3. Scroll down to the **Execution Settings** section.
4. Under **Commands**, add this command as part of your job if you don't see it:
* `dbt build`
5. Select the **Generate docs on run** checkbox to automatically [generate updated project docs](/docs/collaborate/build-and-view-your-docs) each time your job runs.
6. For this exercise, do _not_ set a schedule for your project to run &mdash; while your organization's project should run regularly, there's no need to run this example project on a schedule. Scheduling a job is sometimes referred to as _deploying a project_.
7. Select **Save**, then click **Run now** to run your job.

Check warning on line 497 in website/docs/guides/teradata-qs.md

View workflow job for this annotation

GitHub Actions / Vale linting 

[vale] reported by reviewdog 🐶
[custom.UIElements] UI elements like 'Save' should be bold.

Raw Output:
{"message": "[custom.UIElements] UI elements like 'Save' should be bold.", "location": {"path": "website/docs/guides/teradata-qs.md", "range": {"start": {"line": 497, "column": 13}}}, "severity": "WARNING"}
8. Click the run and watch its progress under "Run history."
9. Once the run is complete, click **View Documentation** to see the docs for your project.


Congratulations 🎉! You've just deployed your first dbt project!


#### FAQs

Check warning on line 505 in website/docs/guides/teradata-qs.md

View workflow job for this annotation

GitHub Actions / Vale linting 

[vale] reported by reviewdog 🐶
[custom.SentenceCaseHeaders] 'FAQs' should use sentence-style capitalization. Try '' instead.

Raw Output:
{"message": "[custom.SentenceCaseHeaders] 'FAQs' should use sentence-style capitalization. Try '' instead.", "location": {"path": "website/docs/guides/teradata-qs.md", "range": {"start": {"line": 505, "column": 6}}}, "severity": "WARNING"}

<FAQ path="Runs/failed-prod-run" />



Loading