This repository has been archived by the owner on Jan 29, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 51
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
f7acca7
commit 1731de7
Showing
5 changed files
with
30 additions
and
30 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,7 +5,7 @@ Tiered storage in Aiven for ClickHouse® | |
|
|
||
| Aiven for ClickHouse® tiered storage is a :doc:`limited availability feature </docs/platform/concepts/beta_services>`. If you're interested in trying out this feature, contact the sales team at `[email protected] <mailto:[email protected]>`_. | ||
|
|
||
| Discover the tiered storage capability in Aiven for ClickHouse®. Learn how it works and explore its use cases. Check why you might need it and what benefits you get using it. | ||
| Discover the tiered storage capability in Aiven for ClickHouse®. Learn how it works and explore its use cases. Check out why you might need it and what benefits you get using it. | ||
|
|
||
| Overview | ||
| -------- | ||
|
|
@@ -23,14 +23,14 @@ Object storage - the second tier | |
| Why use it | ||
| ---------- | ||
|
|
||
| By :doc:`enabling </docs/products/clickhouse/howto/enable-tiered-storage>` and properly :doc:`configuring </docs/products/clickhouse/howto/configure-tiered-storage>` the tiered storage feature in Aiven for ClickHouse, you can use storage resources efficiently and, therefore, significantly reduce costs of storing data ingested into an Aiven for ClickHouse instance. | ||
| By :doc:`enabling </docs/products/clickhouse/howto/enable-tiered-storage>` and properly :doc:`configuring </docs/products/clickhouse/howto/configure-tiered-storage>` the tiered storage feature in Aiven for ClickHouse, you can use storage resources efficiently and, therefore, significantly reduce storage costs of your Aiven for ClickHouse instance. | ||
|
|
||
| How it works | ||
| ------------ | ||
|
|
||
| With the tiered storage feature :doc:`enabled </docs/products/clickhouse/howto/enable-tiered-storage>`, by default Aiven for ClickHouse stores data on your SSD until it reaches 80% of its capacity. After exceeding this size-based threshold, the data is stored in the object storage. | ||
| After you :doc:`enable </docs/products/clickhouse/howto/enable-tiered-storage>` the tiered storage feature, Aiven for ClickHouse by default stores data on SSD until it reaches 80% of its capacity. After exceeding this size-based threshold, data is stored in object storage. | ||
|
|
||
| Optionally, you can :doc:`configure the time-based threshold </docs/products/clickhouse/howto/configure-tiered-storage>` for the tiered storage feature. Based on the time-based threshold, the data is moved from your SSD to the object storage after a specified time period. | ||
| Optionally, you can :doc:`configure the time-based threshold </docs/products/clickhouse/howto/configure-tiered-storage>` for your storage. Based on the time-based threshold, the data is moved from your SSD to object storage after a specified time period. | ||
|
|
||
| .. mermaid:: | ||
|
|
||
|
|
@@ -55,7 +55,7 @@ Optionally, you can :doc:`configure the time-based threshold </docs/products/cli | |
| Typical use case | ||
| ---------------- | ||
|
|
||
| In your Aiven for ClickHouse service, there is a significant amount of data that is there for a while and is hardly ever accessed. It's stored on SSD and, thus, high-priced. You decide to :doc:`enable </docs/products/clickhouse/howto/enable-tiered-storage>` the tiered storage feature for your service to make your data storage more efficient and reduce the costs. For that purpose, you contact the sales team at `[email protected] <mailto:[email protected]>`_ to have it enabled on your project, and you :doc:`enable </docs/products/clickhouse/howto/enable-tiered-storage>` the feature on particular tables. You :doc:`configure </docs/products/clickhouse/howto/configure-tiered-storage>` the time-based threshold for controlling how your data is stored in the two layers. | ||
| In your Aiven for ClickHouse service, there is a significant amount of data that is there for a while and is hardly ever accessed. It's stored on SSD and, thus, high-priced. You decide to :doc:`enable </docs/products/clickhouse/howto/enable-tiered-storage>` tiered storage for your service to make your data storage more efficient and reduce the costs. For that purpose, you contact the sales team at `[email protected] <mailto:[email protected]>`_ to have it enabled on your project, and you :doc:`enable </docs/products/clickhouse/howto/enable-tiered-storage>` the feature on tables you want to optimize. You :doc:`configure </docs/products/clickhouse/howto/configure-tiered-storage>` the time-based threshold to control how your data is stored between the two layers. | ||
|
|
||
| .. _tiered-storage-limitations: | ||
|
|
||
|
|
@@ -66,7 +66,7 @@ Limitations | |
|
|
||
| .. tip:: | ||
|
|
||
| As a workaround, you can create a new table (without enabling the tiered storage feature) and copy the data from the original table (with the tiered storage feature :doc:`enabled </docs/products/clickhouse/howto/enable-tiered-storage>`) to the new table. As soon as the data is copied to the new table, you can remove the original table. | ||
| As a workaround, you can create a new table (without enabling tiered storage on it) and copy the data from the original table (with the tiered storage feature :doc:`enabled </docs/products/clickhouse/howto/enable-tiered-storage>`) to the new table. As soon as the data is copied to the new table, you can remove the original table. | ||
|
|
||
| * With the tiered storage feature :doc:`enabled </docs/products/clickhouse/howto/enable-tiered-storage>`, it's not possible to connect to an external existing object storage or cloud storage bucket. | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,12 +5,12 @@ Check data distribution between storage devices in Aiven for ClickHouse®'s tier | |
|
|
||
| Aiven for ClickHouse® tiered storage is a :doc:`limited availability feature </docs/platform/concepts/beta_services>`. If you're interested in trying out this feature, contact the sales team at `[email protected] <mailto:[email protected]>`_. | ||
|
|
||
| Verify how your data is distributed between the two layers of your tiered storage: SSD and object storage. | ||
| Monitor how your data is distributed between the two layers of your tiered storage: SSD and object storage. | ||
|
|
||
| About checking data distribution | ||
| -------------------------------- | ||
|
|
||
| If you have the tiered storage feature :doc:`enabled </docs/products/clickhouse/howto/enable-tiered-storage>` on your project, your data in Aiven for ClickHouse is distributed between two storage devices (tiers). You can learn on what storage devices specific databases and tables are stored. You can also preview their total sizes as well as part counts, minimum part sizes, median part sizes, and maximum part sizes. | ||
| If you have the tiered storage feature :doc:`enabled </docs/products/clickhouse/howto/enable-tiered-storage>` on your project, your data in Aiven for ClickHouse is distributed between two storage devices (tiers). You can check on what storage devices your databases and tables are stored. You can also preview their total sizes as well as part counts, minimum part sizes, median part sizes, and maximum part sizes. | ||
|
|
||
| Prerequisites | ||
| ------------- | ||
|
|
@@ -22,9 +22,9 @@ Prerequisites | |
| Check data distribution in Aiven Console | ||
| ---------------------------------------- | ||
|
|
||
| You can use `Aiven Console <https://console.aiven.io/>`_ to check if the tiered storage is enabled on your service and, if so, how much storage has been used for each type of the disks (the local SSD and the remote object storage) by particular tables. | ||
| You can use `Aiven Console <https://console.aiven.io/>`_ to check if tiered storage is enabled on your service and, if it is, how much storage is used on each tier (local SSD and remote object storage) for particular tables. | ||
|
|
||
| To check the tiered storage's status information, go to `Aiven Console <https://console.aiven.io/>`_ > your Aiven for ClickHouse service > the **Databases and tables** page > your database > your table > **View details** > **Storage details**. | ||
| To access tiered storage's status information, go to `Aiven Console <https://console.aiven.io/>`_ > your Aiven for ClickHouse service > the **Databases and tables** page > your database > your table > **View details** > **Storage details**. | ||
|
|
||
| Run a data distribution check with the ClickHouse client (CLI) | ||
| -------------------------------------------------------------- | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,19 +5,19 @@ Configure data retention thresholds in Aiven for ClickHouse®'s tiered storage | |
|
|
||
| Aiven for ClickHouse® tiered storage is a :doc:`limited availability feature </docs/platform/concepts/beta_services>`. If you're interested in trying out this feature, contact the sales team at `[email protected] <mailto:[email protected]>`_. | ||
|
|
||
| Learn to control how your data is distributed between storage devices in the tiered storage of an Aiven for ClickHouse service. Check out how to configure tables so that your data is automatically written either to SSD or the object storage as needed. | ||
| Learn to control how your data is distributed between storage devices in the tiered storage of an Aiven for ClickHouse service. Check out how to configure tables so that your data is automatically written either to SSD or object storage as needed. | ||
|
|
||
| About data retention control | ||
| ---------------------------- | ||
|
|
||
| If you have the tiered storage feature :doc:`enabled </docs/products/clickhouse/howto/enable-tiered-storage>` on your Aiven for ClickHouse service, your data is distributed between two storage devices (tiers). The data is stored either on SSD or in the object storage, depending on whether and how you configure this behavior. If you don't, by default the data is moved from SSD to the object storage when SSD reaches 80% of its capacity (default size-based data retention policy). | ||
| If you have the tiered storage feature :doc:`enabled </docs/products/clickhouse/howto/enable-tiered-storage>` on your Aiven for ClickHouse service, your data is distributed between two storage devices (tiers). The data is stored either on SSD or in object storage, depending on whether and how you configure this behavior. By default, data is moved from SSD to object storage when SSD reaches 80% of its capacity (default size-based data retention policy). | ||
|
|
||
| You may want to change this default data distribution behavior by :ref:`configuring your table's schema by adding a TTL (time-to-live) clause <time-based-retention-config>`. Such a configuration allows ignoring the SSD-capacity threshold and moving the data from SSD to the object storage based on how long the data is there on your SSD. | ||
| You may want to change this default data distribution behavior by :ref:`configuring your table's schema by adding a TTL (time-to-live) clause <time-based-retention-config>`. Such a configuration allows ignoring the SSD-capacity threshold and moving the data from SSD to object storage based on how long the data is there on your SSD. | ||
|
|
||
| To enable this time-based data distribution mechanism, you can set up a retention policy (threshold) on a table level by using the TTL clause. For data retention control purposes, the TTL clause uses the following: | ||
|
|
||
| * Data item of the `Date` or `DateTime` type as a reference point in time | ||
| * INTERVAL clause as a time period to elapse between the reference point and the data transfer to the object storage | ||
| * INTERVAL clause as a time period to elapse between the reference point and the data transfer to object storage | ||
|
|
||
| Prerequisites | ||
| ------------- | ||
|
|
@@ -76,7 +76,7 @@ Change an already configured TTL in an existing table by using the ALTER TABLE M | |
| .. topic:: Result | ||
|
|
||
| You have your time-based data retention policy set up. Now when a piece of data is there on your SSD for a specified time period, it's moved to the object storage, regardless of how much of the SSD capacity is still available. | ||
| You have your time-based data retention policy set up. From now on, when data is on your SSD longer than a specified time period, it's moved to object storage, regardless of how much of SSD capacity is still available. | ||
|
|
||
| What's next | ||
| ----------- | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,13 +5,13 @@ Enable tiered storage in Aiven for ClickHouse® | |
|
|
||
| Aiven for ClickHouse® tiered storage is a :doc:`limited availability feature </docs/platform/concepts/beta_services>`. If you're interested in trying out this feature, contact the sales team at `[email protected] <mailto:[email protected]>`_. | ||
|
|
||
| Learn how to enable the tiered storage feature on your project and activate it for specific tables. | ||
| To check what the tiered storage is, how it works, and why use it, see :doc:`Tiered storage in Aiven for ClickHouse® </docs/products/clickhouse/concepts/clickhouse-tiered-storage>`. | ||
| Find out how to enable the tiered storage feature on your project and activate it for specific tables. | ||
| To learn what tiered storage is, how it works, and why use it, see :doc:`Tiered storage in Aiven for ClickHouse® </docs/products/clickhouse/concepts/clickhouse-tiered-storage>`. | ||
|
|
||
| About enabling tiered storage | ||
| ----------------------------- | ||
|
|
||
| To use the tiered storage feature, you need to enable it on the project level by contacting the sales team at `[email protected] <mailto:[email protected]>`_ and set it up on the table level using SQL (via CLI, for example). | ||
| To use the tiered storage feature, you need to enable it at project level by contacting the sales team at `[email protected] <mailto:[email protected]>`_ and set it up at table level using SQL (via CLI, for example). | ||
|
|
||
| Limitations | ||
| ''''''''''' | ||
|
|
@@ -20,14 +20,14 @@ Limitations | |
|
|
||
| .. tip:: | ||
|
|
||
| As a workaround, you can create a new table (without enabling the tiered storage) and copy the data from the original table (with the tiered storage :doc:`enabled </docs/products/clickhouse/howto/enable-tiered-storage>`) to the new table. As soon as the data is copied to the new table, you can remove the original table. | ||
| As a workaround, you can create a new table (without enabling tiered storage on it) and copy the data from the original table (with the tiered storage :doc:`enabled </docs/products/clickhouse/howto/enable-tiered-storage>`) to the new table. As soon as the data is copied to the new table, you can remove the original table. | ||
|
|
||
| * With the tiered storage feature :doc:`enabled </docs/products/clickhouse/howto/enable-tiered-storage>`, it's not possible to connect to an external existing object storage or cloud storage bucket. | ||
|
|
||
| Tools | ||
| ''''' | ||
|
|
||
| To enable the tiered storage, use SQL and an SQL client (for example, the ClickHouse CLI client). | ||
| To enable tiered storage, use SQL and an SQL client (for example, the ClickHouse CLI client). | ||
|
|
||
| Prerequisites | ||
| ------------- | ||
|
|
@@ -39,16 +39,16 @@ Prerequisites | |
| Enable tiered storage on a project | ||
| ---------------------------------- | ||
|
|
||
| To enable the tiered storage feature on your project, request it from the sales team at `[email protected] <mailto:[email protected]>`_. | ||
| To enable tiered storage on your project, request it from the sales team at `[email protected] <mailto:[email protected]>`_. | ||
|
|
||
| .. topic:: Result | ||
|
|
||
| Your project supports the tiered storage feature, and you can enable the feature on tables for your Aiven for ClickHouse services. | ||
| Your project now supports tiered storage, and you can enable it for each table of your Aiven for ClickHouse services. | ||
|
|
||
| Enable tiered storage on a table | ||
| -------------------------------- | ||
|
|
||
| When you have the tiered storage feature enabled on your project, you can move on to enabling it on your tables, both new and existing ones. | ||
| When you have tiered storage enabled on your project, you can move on to enabling it on your tables, both new and existing ones. | ||
|
|
||
| 1. :doc:`Connect to your Aiven for ClickHouse service </docs/products/clickhouse/howto/list-connect-to-service>` using, for example, the ClickHouse client (CLI). | ||
|
|
||
|
|
@@ -60,9 +60,9 @@ When you have the tiered storage feature enabled on your project, you can move o | |
| .. topic:: Result | ||
|
|
||
| The tiered storage is enabled on your table and data in this table is now distributed between two tiers: SSD and object storage. | ||
| Tiered storage is enabled on your table and data in this table is now distributed between two tiers: SSD and object storage. | ||
|
|
||
| You can check if the tiered storage is now supported (**Active** / **Inactive**) on your table in `Aiven Console <https://console.aiven.io/>`_ > **Databases & Tables** > **Databases lists** > Your database > Your table > the **Tiered storage** column. | ||
| You can check if tiered storage is now supported (**Active** / **Inactive**) on your table in `Aiven Console <https://console.aiven.io/>`_ > **Databases & Tables** > **Databases lists** > Your database > Your table > the **Tiered storage** column. | ||
|
|
||
| What's next | ||
| ----------- | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,24 +5,24 @@ Transfer data between storage devices in Aiven for ClickHouse®'s tiered storage | |
|
|
||
| Aiven for ClickHouse® tiered storage is a :doc:`limited availability feature </docs/platform/concepts/beta_services>`. If you're interested in trying out this feature, contact the sales team at `[email protected] <mailto:[email protected]>`_. | ||
|
|
||
| Check out this article for instructions on transferring data outside and inside your SSD for an Aiven for ClickHouse® service. | ||
| Check out this article for instructions on transferring data from and to SSD for an Aiven for ClickHouse® service. | ||
|
|
||
| About moving data between storage devices | ||
| ----------------------------------------- | ||
|
|
||
| You may want to :doc:`enable </docs/products/clickhouse/howto/enable-tiered-storage>` the tiered storage feature, move your data from SSD to the object storage, and size down your SSD, which allows changing a service plan to one with less SSD capacity. Also, you may want to move your data from the object storage back to your SSD. Both the operations can be performed using SQL statements on the table level. | ||
| After :doc:`enabling </docs/products/clickhouse/howto/enable-tiered-storage>` the tiered storage feature, you can move your data from SSD to object storage. Next, you may want to size down your SSD by selecting a service plan with less SSD capacity. Later, you can move your data from object storage back to your SSD if needed. Both operations can be performed using SQL statements against your tables directly. | ||
|
|
||
| Prerequisites | ||
| ------------- | ||
|
|
||
| * Aiven organization | ||
| * Tiered storage feature :doc:`enabled </docs/products/clickhouse/howto/enable-tiered-storage>` on the project level | ||
| * Tiered storage feature :doc:`enabled </docs/products/clickhouse/howto/enable-tiered-storage>` at project level | ||
| * Command line tool (:doc:`ClickHouse client </docs/products/clickhouse/howto/connect-with-clickhouse-cli>`) | ||
|
|
||
| Transfer data from SSD to object storage | ||
| ---------------------------------------- | ||
|
|
||
| If you :doc:`enable </docs/products/clickhouse/howto/enable-tiered-storage>` the tiered storage feature on your table, by default your data is moved from SSD to the object storage as soon as the SSD reaches 80% of its capacity. You can also :doc:`configure your tiered storage </docs/products/clickhouse/howto/configure-tiered-storage>` so that data is moved to the object storage at a specific time. | ||
| If you :doc:`enable </docs/products/clickhouse/howto/enable-tiered-storage>` the tiered storage feature on your table, by default your data is moved from SSD to object storage as soon as the SSD reaches 80% of its capacity. You can also :doc:`configure your tiered storage </docs/products/clickhouse/howto/configure-tiered-storage>` so that data is moved to object storage at a specific time. | ||
|
|
||
| 1. :doc:`Connect to your Aiven for ClickHouse service </docs/products/clickhouse/howto/list-connect-to-service>` using, for example, the ClickHouse client (CLI). | ||
|
|
||
|
|
@@ -34,7 +34,7 @@ If you :doc:`enable </docs/products/clickhouse/howto/enable-tiered-storage>` the | |
| .. topic:: Result | ||
|
|
||
| Now, with the tiered storage feature :doc:`enabled </docs/products/clickhouse/howto/enable-tiered-storage>`, your data is moved from SSD to the object storage when the SSD reaches 80% of its capacity. | ||
| Now, with the tiered storage feature :doc:`enabled </docs/products/clickhouse/howto/enable-tiered-storage>`, your data is moved from SSD to object storage when the SSD reaches 80% of its capacity. | ||
|
|
||
| Transfer data from object storage to SSD | ||
| ---------------------------------------- | ||
|
|
||