Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions quickstarts/_partials/tabsDBT.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,20 +5,20 @@ import TabItem from '@theme/TabItem';
<TabItem value="Windows" label="Windows" default>
Run in Powershell:
```bash
python -m venv env
.\env\Scripts\activate
uv venv .venv
.venv\Scripts\activate
```
</TabItem>
<TabItem value="MacOS" label="MacOS">
<TabItem value="macOS" label="macOS">
```bash
python3 -m venv env
source env/bin/activate
uv venv .venv
source .venv/bin/activate
```
</TabItem>
<TabItem value="Linux" label="Linux">
```bash
python3 -m venv env
source env/bin/activate
uv venv .venv
source .venv/bin/activate
```
</TabItem>
</Tabs>
Expand Down
Binary file added quickstarts/images/dbt-cloud/1-create-project.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added quickstarts/images/dbt-cloud/10-deploy-job.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added quickstarts/images/dbt-cloud/11-run-job.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added quickstarts/images/dbt-cloud/12-final-output.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added quickstarts/images/dbt-cloud/2.3-settings.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added quickstarts/images/dbt-cloud/3-set-up-repo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added quickstarts/images/dbt-cloud/4-environment.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added quickstarts/images/dbt-cloud/5-env-setting.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added quickstarts/images/dbt-cloud/6-assign-profile.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added quickstarts/images/dbt-cloud/7-add-new-con.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added quickstarts/images/dbt-cloud/9-create-job.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed quickstarts/images/dbt-cloud/dbt-cloud1.png
Binary file not shown.
Binary file removed quickstarts/images/dbt-cloud/dbt-cloud10.png
Binary file not shown.
Binary file removed quickstarts/images/dbt-cloud/dbt-cloud11.png
Binary file not shown.
Binary file removed quickstarts/images/dbt-cloud/dbt-cloud12.png
Binary file not shown.
Binary file removed quickstarts/images/dbt-cloud/dbt-cloud13.png
Binary file not shown.
Binary file removed quickstarts/images/dbt-cloud/dbt-cloud14.png
Binary file not shown.
Binary file removed quickstarts/images/dbt-cloud/dbt-cloud15.png
Binary file not shown.
Binary file removed quickstarts/images/dbt-cloud/dbt-cloud2.png
Binary file not shown.
Binary file removed quickstarts/images/dbt-cloud/dbt-cloud3.png
Binary file not shown.
Binary file removed quickstarts/images/dbt-cloud/dbt-cloud4.png
Binary file not shown.
Binary file removed quickstarts/images/dbt-cloud/dbt-cloud5.png
Diff not rendered.
Binary file removed quickstarts/images/dbt-cloud/dbt-cloud6.png
Diff not rendered.
Binary file removed quickstarts/images/dbt-cloud/dbt-cloud7.png
Diff not rendered.
Binary file removed quickstarts/images/dbt-cloud/dbt-cloud8.png
Diff not rendered.
Binary file removed quickstarts/images/dbt-cloud/dbt-cloud9.png
Diff not rendered.
103 changes: 60 additions & 43 deletions quickstarts/manage-data/create-parquet-files-in-object-storage.md
Original file line number Diff line number Diff line change
@@ -1,40 +1,41 @@
---
sidebar_position: 12
id: create-parquet-files-in-object-storage
author: Obed Vega
email: obed.vega@teradata.com
page_last_update: August 2nd, 2022
description: Teradata Vantage Native Object Storage - read and write from/to object storage, unified SQL interface for Vantage and object storage.
keywords: [data warehouses, compute storage separation, Teradata, vantage, cloud data platform, object storage, business intelligence, enterprise analytics, parquet, create parquet files]
author: Obed Vega, Vidhan Bhonsle
email: developer.relations@teradata.com
page_last_update: July 6th, 2026
description: Use Teradata Native Object Storage to read from and write to object storage using a unified SQL interface for Teradata and object storage.
keywords: [data warehouses, compute storage separation, Teradata, cloud data platform, object storage, business intelligence, enterprise analytics, parquet, create parquet files]
---

import UseCase from '../_partials/use-trials.mdx';
import CommunityLink from '../_partials/community_link.mdx';

# Create Parquet files in object storage

## Overview
Native Object Storage (NOS) is a Vantage feature that allows you to query data stored in files such as CSV, JSON, and Parquet format datasets.
These datasets are located on external S3-compatible object storage such as AWS S3, Google GCS, Azure Blob or on-prem implementations.
It's useful in scenarios where you want to explore data without building a data pipeline to bring it into Vantage. This tutorial demonstrates how to export data from Vantage to object storage using the Parquet file format.
Native Object Storage (NOS) is a feature of Teradata that allows you to query data stored in external object storage, including CSV, JSON, and Parquet files.

These datasets can be located in external object storage such as Amazon S3, Google Cloud Storage, Azure Blob Storage, or on-premises object storage.

NOS is useful when you want to explore external data without first building a data pipeline to load it into Teradata. This tutorial demonstrates how to export data from Teradata to object storage in Parquet format.

## Prerequisites

You need access to a Teradata Vantage instance. NOS is enabled in all Vantage editions from Vantage Express through Developer, DYI to Vantage as a Service starting from version 17.10.
You need access to a Teradata instance. NOS is available in Teradata editions starting from version 17.10.

:::info
This tutorial is based on s3 aws object storage. You will need your own s3 bucket with write permissions to complete the tutorial.
This tutorial uses Amazon S3 object storage. To complete the tutorial, you need an S3 bucket with write permissions.
:::

<UseCase />

## Create a Parquet file with WRITE_NOS function

`WRITE_NOS` allows you to extract selected or all columns from a database table or from derived results and write to external object storage, such as Amazon S3, Azure Blob storage, Azure Data Lake Storage Gen2, and Google Cloud Storage. This functionality stores data in Parquet format.
`WRITE_NOS` allows you to extract selected or all columns from a database table or query results and write them to external object storage, such as Amazon S3, Azure Blob Storage, Azure Data Lake Storage Gen2, and Google Cloud Storage. This functionality writes data in Parquet format.

You can find more documentation about `WRITE_NOS` functionality in the [NOS documentation](https://docs.teradata.com/r/Teradata-VantageTM-Native-Object-Store-Getting-Started-Guide/June-2022/Writing-Data-to-External-Object-Store).
You can find more documentation about the `WRITE_NOS` functionality in the [NOS documentation](https://docs.teradata.com/r/Enterprise_IntelliFlex_VMware/Native-Object-Store-Getting-Started-Guide/Writing-Data-to-External-Object-Store).

You will need access to a database where you can execute `WRITE_NOS` function. If you don't have such a database, run the following commands:
You need access to a database where you can execute the `WRITE_NOS` function. If you don't have such a database, run the following commands:

``` sql
CREATE USER db AS PERM=10e7, PASSWORD=db;
Expand All @@ -45,11 +46,11 @@ GRANT EXECUTE FUNCTION on TD_SYSFNLIB.WRITE_NOS to db;
```

:::note
If you would like to learn more about setting up users and their privileges, checkout the [NOS documentation](https://docs.teradata.com/r/Teradata-VantageTM-Native-Object-Store-Getting-Started-Guide/June-2022/Setting-Up-Access/Setting-Access-Privileges).
If you would like to learn more about setting up users and their privileges, check out the [NOS documentation](https://docs.teradata.com/r/Teradata-VantageTM-Native-Object-Store-Getting-Started-Guide/June-2022/Setting-Up-Access/Setting-Access-Privileges).
:::


1. Let's first create a table on your Teradata Vantage instance:
1. First create a table on your Teradata instance:

```sql
CREATE SET TABLE db.parquet_table ,FALLBACK ,
Expand Down Expand Up @@ -82,17 +83,17 @@ column1 column2 column3
3 22/01/03 3.30
```

3. Create the parquet file with `WRITE_NOS`. Don't forget to replace `<BUCKET_NAME>` with the name of your s3 bucket. Also,replace `<YOUR-ACCESS-KEY-ID>` and `<YOUR-SECRET-ACCESS-KEY>` with your access key and secret.
3. Create the Parquet file with `WRITE_NOS`. Replace `<BUCKET_NAME>` with the name of your S3 bucket. Also, replace `<YOUR-ACCESS-KEY-ID>` and `<YOUR-SECRET-ACCESS-KEY>` with your access key and secret.

:::note
Check your cloud provider docs how to create credentials to access object storage. For example, for AWS check out [How do I create an AWS access key?](https://aws.amazon.com/premiumsupport/knowledge-center/create-access-key/)
Check your cloud provider documentation to learn how to create credentials to access object storage. For example, for AWS check out [How do I create an AWS access key?](https://aws.amazon.com/premiumsupport/knowledge-center/create-access-key/)
:::

```sql
SELECT * FROM WRITE_NOS (
ON ( SELECT * FROM db.parquet_table)
USING
LOCATION('/s3/<BUCKET_NAME>.s3.amazonaws.com/parquet_file_on_NOS.parquet')
LOCATION('/s3/<BUCKET_NAME>.s3.amazonaws.com/parquet_file_on_NOS/')
AUTHORIZATION('{"ACCESS_ID":"<YOUR-ACCESS-KEY-ID>",
"ACCESS_KEY":"<YOUR-SECRET-ACCESS-KEY>"}')
STOREDAS('PARQUET')
Expand All @@ -103,48 +104,64 @@ INCLUDE_HASHBY('TRUE')
) as d;
```

Now you have created a parquet file in your object storage bucket. Now to easily query your file you need to follow step number 4.
:::note
If you are using temporary AWS credentials, include the session token in the `AUTHORIZATION` string:

```sql
AUTHORIZATION('{"ACCESS_ID":"<YOUR-ACCESS-KEY-ID>",
"ACCESS_KEY":"<YOUR-SECRET-ACCESS-KEY>",
"SESSION_TOKEN":"<YOUR-SESSION-TOKEN>"}')
```
:::

Now you have created a Parquet files in your object storage bucket. To query the files, follow step 4.

4. Create an authorization object. Replace `<YOUR-ACCESS-KEY-ID>` and `<YOUR-SECRET-ACCESS-KEY>` with your access key and secret:

4. Create a NOS-backed foreign table. Don't forget to replace `<BUCKET_NAME>` with the name of your s3 bucket. Also,replace `<YOUR-ACCESS-KEY-ID>` and `<YOUR-SECRET-ACCESS-KEY>` with your access key and secret:
```sql
CREATE MULTISET FOREIGN TABLE db.parquet_table_to_read_file_on_NOS
, EXTERNAL SECURITY DEFINER TRUSTED CEPH_AUTH,
CREATE AUTHORIZATION MyAuthObj
USER '<YOUR-ACCESS-KEY-ID>'
PASSWORD '<YOUR-SECRET-ACCESS-KEY>';
```

5. Create a NOS-backed foreign table. Replace `<BUCKET_NAME>` with the name of your S3 bucket:

```sql
CREATE MULTISET FOREIGN TABLE parquet_table_to_read_file_on_NOS,
EXTERNAL SECURITY MyAuthObj,
MAP = TD_MAP1
(
Location VARCHAR(2048) CHARACTER SET UNICODE CASESPECIFIC
, col1 SMALLINT
, col2 DATE
, col3 DECIMAL(10,2)

Location VARCHAR(2048) CHARACTER SET UNICODE CASESPECIFIC,
column1 SMALLINT,
column2 DATE,
column3 DECIMAL(10,2)
)
USING (
LOCATION ('/s3/<BUCKET_NAME>.s3.amazonaws.com/parquet_file_on_NOS.parquet')
AUTHORIZATION('{"ACCESS_ID":"<YOUR-ACCESS-KEY-ID>",
"ACCESS_KEY":"<YOUR-SECRET-ACCESS-KEY>"}')
LOCATION ('/s3/<BUCKET_NAME>.s3.amazonaws.com/parquet_file_on_NOS/')
STOREDAS ('PARQUET')
)NO PRIMARY INDEX;
)
NO PRIMARY INDEX;
```

5. Now you are ready to Query your parquet file on NOS, let's try the following query:
6. Query the Parquet files on NOS:

```sql
SELECT col1, col2, col3 FROM db.parquet_table_to_read_file_on_NOS;
SELECT column1, column2, column3 FROM parquet_table_to_read_file_on_NOS;
```

The data returned from the query should look something like this:

```sql
col1 col2 col3
------ -------- ------------
1 22/01/01 1.10
2 22/01/02 2.20
3 22/01/03 3.30
column1 column2 column3
------- -------- ------------
1 22/01/01 1.10
2 22/01/02 2.20
3 22/01/03 3.30
```

## Summary

In this tutorial we have learned how to export data from Vantage to a parquet file on object storage using Native Object Storage (NOS). NOS supports reading and importing data stored in CSV, JSON and Parquet formats. NOS can also export data from Vantage to object storage.
In this tutorial, you learned how to export data from Teradata to object storage in Parquet format using Native Object Storage (NOS). NOS supports reading data stored in CSV, JSON, and Parquet formats. NOS can also export data from Teradata to object storage.

## Further reading
* [Teradata Vantage™ - Writing Data to External Object Store](https://docs.teradata.com/r/Teradata-VantageTM-Native-Object-Store-Getting-Started-Guide/June-2022/Writing-Data-to-External-Object-Store)

<CommunityLink />
* [Native Object Storage - Writing Data to External Object Store](https://docs.teradata.com/r/Teradata-VantageTM-Native-Object-Store-Getting-Started-Guide/June-2022/Writing-Data-to-External-Object-Store)
Loading
Loading