From 82a0d97d07c43d4ff2005a07785aace7f2fd18ab Mon Sep 17 00:00:00 2001 From: Dorota Wojcik Date: Tue, 23 May 2023 15:01:03 +0200 Subject: [PATCH 01/17] draft --- _toc.yml | 13 +++ docs/products/clickhouse/concepts.rst | 4 + .../concepts/clickhouse-tiered-storage.rst | 78 ++++++++++++++++ docs/products/clickhouse/howto.rst | 7 ++ .../howto/check-data-tiered-storage.rst | 69 ++++++++++++++ .../howto/configure-tiered-storage.rst | 92 +++++++++++++++++++ .../howto/enable-tiered-storage.rst | 83 +++++++++++++++++ .../clickhouse/howto/list-tiered-storage.rst | 20 ++++ .../howto/transfer-data-tiered-storage.rst | 68 ++++++++++++++ 9 files changed, 434 insertions(+) create mode 100644 docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst create mode 100644 docs/products/clickhouse/howto/check-data-tiered-storage.rst create mode 100644 docs/products/clickhouse/howto/configure-tiered-storage.rst create mode 100644 docs/products/clickhouse/howto/enable-tiered-storage.rst create mode 100644 docs/products/clickhouse/howto/list-tiered-storage.rst create mode 100644 docs/products/clickhouse/howto/transfer-data-tiered-storage.rst diff --git a/_toc.yml b/_toc.yml index c828787b35..12106fd892 100644 --- a/_toc.yml +++ b/_toc.yml @@ -728,6 +728,8 @@ entries: title: Strings - file: docs/products/clickhouse/concepts/federated-queries title: Federated queries + - file: docs/products/clickhouse/concepts/clickhouse-tiered-storage + title: Tiered storage - file: docs/products/clickhouse/howto title: HowTo entries: @@ -791,6 +793,17 @@ entries: title: Connect services via integration databases - file: docs/products/clickhouse/howto/connect-with-jdbc title: Connect to external DBs with JDBC + - file: docs/products/clickhouse/howto/list-tiered-storage + title: Tiered storage + entries: + - file: docs/products/clickhouse/howto/enable-tiered-storage + title: Enable tiered storage + - file: docs/products/clickhouse/howto/configure-tiered-storage + title: Configure tiered storage + - file: docs/products/clickhouse/howto/check-data-tiered-storage + title: Check tiered storage status + - file: docs/products/clickhouse/howto/transfer-data-tiered-storage + title: Transfer data in tiered storage - file: docs/products/clickhouse/reference title: Reference entries: diff --git a/docs/products/clickhouse/concepts.rst b/docs/products/clickhouse/concepts.rst index 50c63c5dd7..87979d8aa1 100644 --- a/docs/products/clickhouse/concepts.rst +++ b/docs/products/clickhouse/concepts.rst @@ -22,6 +22,10 @@ Aiven service :shadow: md :margin: 2 2 0 0 + .. grid-item-card:: :doc:`Tiered storage in Aiven for ClickHouse® ` + :shadow: md + :margin: 2 2 0 0 + General ------- diff --git a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst new file mode 100644 index 0000000000..1da8438aa9 --- /dev/null +++ b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst @@ -0,0 +1,78 @@ +Tiered storage in Aiven for ClickHouse® +======================================= + +Discover the tiered storage capability in Aiven for ClickHouse®. Learn how it works and explore its use cases. Check why you might need it and what benefits you get using it. + +Overview +-------- + +The tiered storage feature introduces a method of organizing and storing data in two tiers for improved efficiency and cost optimization. The data is automatically moved to an appropriate tier based on your database's local disk usage. On top of this default data allocation mechanism, you can control the tier your data is stored in using custom data retention periods. + +The tiered storage in Aiven for ClickHouse consists of the following two layers: + +* SSD - so called *hot storage*, the higher tier for fresh, frequently-accessed and valuable data, a fast but costly storage device +* Object storage - so called *cold storage*, Amazon S3, the lower tier for older, less valuable or rarely-accessed data, a slower but more affordable storage device + +Why use it +---------- + +By :doc:`enabling ` and properly :doc:`configuring ` the tiered storage in Aiven for ClickHouse, you can use storage resources efficiently and, therefore, significantly reduce costs of storing data ingested into an Aiven for ClickHouse instance. + +How it works +------------ + +With the tiered storage :doc:`enabled `, by default Aiven for ClickHouse stores data on your SSD until it reaches 80% of its capacity. After exceeding this size-based threshold, the data is stored in the object storage. in the object storage depending on the following. + +Optionally, you can :doc:`configure the time-based threshold ` in the tiered storage. Based on the time-based threshold, the data is moved for your SSD to the object storage after a specified time period. + +Regardless of what data retention control mechanism you choose, you receive a notification when a threshold is reached. + +.. mermaid:: + + sequenceDiagram + Application->>+SSD (hot data): writing data + SSD (hot data)->>S3 (cold data): moving data based
on storage policies + par Application to SSD (hot data) + Application-->>SSD (hot data): querying data + and Application to S3 (cold data) + Application-->>S3 (cold data): querying data + end + alt if stored in S3 + S3 (cold data)->>Application: reading data + else if stored in SSD + SSD (hot data)->>Application: reading data + end + +.. note:: + + Backups are taken for data that resides both on SSD and in object storage. + +Typical use case +---------------- + +In your Aiven for ClickHouse service, there is a significant amount of data that is there for a while and is hardly ever accessed. It's stored on SSD and, thus, high-priced. You decide to :doc:`enable ` the tiered storage for your service to make your data storage more efficient and reduce the costs. For that purpose, you select a plan for your service that supports the tiered storage and you :doc:`enable ` the feature on particular tables. You :doc:`configure the time-based threshold ` the time-based threshold for controlling how your data is stored in the two layers. + +.. _tiered-storage-limitations: + +Limitations +----------- + +* When :doc:`enabled `, the tiered storage feature cannot be deactivated. + + .. tip:: + + As a workaround, you can create a new table (without enabling the tiered storage) and copy the data from the original table (with the tiered storage :doc:`enabled `) to the new table. As soon as the data is copied to the new table, you can remove the original table. + +* With the tiered storage feature :doc:`enabled `, it's not possible to connect to an external existing S3 or cloud storage bucket. + +What's next +----------- + +* :doc:`Enable tiered storage in Aiven for ClickHouse ` +* :doc:`Configure data retention thresholds for tiered storage ` + +Related reading +--------------- + +* :doc:`Check data volume distribution between different disks ` +* :doc:`Transfer data between SSD and object storage ` diff --git a/docs/products/clickhouse/howto.rst b/docs/products/clickhouse/howto.rst index fae927cc43..61dba82c63 100644 --- a/docs/products/clickhouse/howto.rst +++ b/docs/products/clickhouse/howto.rst @@ -41,3 +41,10 @@ Aiven for ClickHouse® how-tos - :doc:`Connect to external database via JDBC ` - :doc:`Manage Aiven for ClickHouse® data service integrations ` - :doc:`Manage Aiven for ClickHouse® integration databases ` + +.. dropdown:: Tiered storage + + - :doc:`Enable tiered storage in Aiven for ClickHouse® ` + - :doc:`Configure tiered storage in Aiven for ClickHouse® ` + - :doc:`Check data distribution in tiered storage for Aiven for ClickHouse® ` + - :doc:`Transfer data between storage devices in Aiven for ClickHouse® ` diff --git a/docs/products/clickhouse/howto/check-data-tiered-storage.rst b/docs/products/clickhouse/howto/check-data-tiered-storage.rst new file mode 100644 index 0000000000..0ac6310cbd --- /dev/null +++ b/docs/products/clickhouse/howto/check-data-tiered-storage.rst @@ -0,0 +1,69 @@ +Check data distribution between storage devices in Aiven for ClickHouse®'s tiered storage |beta| +================================================================================================ + +Verify how your data is distributed between the two layers of your tiered storage: SSD and object storage. + +About checking data distribution +-------------------------------- + +If you have the tiered storage feature :doc:`enabled ` on your Aiven for ClickHouse service, your data is distributed between two storage devices (tiers). You can learn on what storage devices specific databases and tables are stored. You can also preview their total sizes as well as part counts, minimum part sizes, median part sizes, and maximum part sizes. + +Prerequisites +------------- + +* Aiven account +* Tiered storage feature :doc:`enabled ` on an Aiven for ClickHouse service level and on a table level +* Command line tool (:doc:`ClickHouse client `) + +Run a data distribution check +----------------------------- + +1. :doc:`Connect to your Aiven for ClickHouse service ` using, for example, the ClickHouse client (CLI). +2. Run the following query: + + .. code-block:: bash + + SELECT + database, + table, + disk_name, + formatReadableSize(sum(data_compressed_bytes)) AS total_size, + count(*) AS parts_count, + formatReadableSize(min(data_compressed_bytes)) AS min_part_size, + formatReadableSize(median(data_compressed_bytes)) AS median_part_size, + formatReadableSize(max(data_compressed_bytes)) AS max_part_size + FROM system.parts + GROUP BY + database, + table, + disk_name + ORDER BY + database ASC, + table ASC, + disk_name ASC + + You can expect to receive the following output: + + .. code-block:: bash + + ┌─database─┬─table─────┬─disk_name─┬─total_size─┬─parts_count─┬─min_part_size─┬─median_part_size─┬─max_part_size─┐ + │ datasets │ hits_v1 │ default │ 1.20 GiB │ 6 │ 33.65 MiB │ 238.69 MiB │ 253.18 MiB │ + │ datasets │ visits_v1 │ S3 │ 536.69 MiB │ 5 │ 44.61 MiB │ 57.90 MiB │ 317.19 MiB │ + │ system │ query_log │ default │ 75.85 MiB │ 102 │ 7.51 KiB │ 12.36 KiB │ 1.55 MiB │ + └──────────┴───────────┴───────────┴────────────┴─────────────┴───────────────┴──────────────────┴───────────────┘ + +.. topic:: Result + + The query returns a table with data distribution details for all databases and tables that belong to your service: storage device they use, their total sizes as well as parts' counts and sizing. + +What's next +----------- + +* :doc:`Transfer data between SSD and object storage ` +* :doc:`Configure data retention thresholds for tiered storage ` + +Related reading +--------------- + +* :doc:`About tiered storage in Aiven for ClickHouse ` +* :doc:`Enable tiered storage in Aiven for ClickHouse ` diff --git a/docs/products/clickhouse/howto/configure-tiered-storage.rst b/docs/products/clickhouse/howto/configure-tiered-storage.rst new file mode 100644 index 0000000000..49091fdaaa --- /dev/null +++ b/docs/products/clickhouse/howto/configure-tiered-storage.rst @@ -0,0 +1,92 @@ +Configure data retention thresholds in Aiven for ClickHouse®'s tiered storage |beta| +==================================================================================== + +Learn to control how your data is distributed between storage devices in the tiered storage of an Aiven for ClickHosue service. Check out how to configure tables so that your data is automatically written either to SSD or the object storage as needed. + +About data retention control +---------------------------- + +If you have the tiered storage feature :doc:`enabled ` on your Aiven for ClickHouse service, your data is distributed between two storage devices (tiers). The data is stored either on SSD or in the object storage, depending on whether and how you configure this behavior. If you don't, by default the data is moved from SSD to the object storage when SSD reaches 80% of its capacity (default size-based data retention policy). + +You may want to change this default data distribution behavior and configure your table's design so that all the data that is there on your SSD for over a specific period of time is moved to the object storage, regardless of how much of the SSD capacity is still available. + +To enable this time-based data distribution mechanism, you can set up a retention policy (threshold) on a table level by using the TTL (time-to-live) clause. You can include the TTL clause after a column definition and/ or at the end of the table definition. For data retention control purposes, the TTL clause uses the following: + +* Data item of the `Date` or `DateTime` type as a reference point in time +* INTERVAL clause as a time period to elapse between the reference point and the data transfer to the object storage + +Prerequisites +------------- + +* Aiven account +* Tiered storage feature :doc:`enabled ` on an Aiven for ClickHouse service level and on a table level +* Command line tool (:doc:`ClickHouse client `) + +Configure time-based data retention +----------------------------------- + +1. :doc:`Connect to your Aiven for ClickHouse service ` using, for example, the ClickHouse client (CLI). +2. Select a database for operations you intend to perform. + + .. code-block:: bash + + USE database-name + +Add TTL to a new table +'''''''''''''''''''''' + +Create a new table with the ``storage_policy`` setting set to ``tiered`` (to :doc:`enabled ` the feature) and TTL (time-to-live) configured to add a time-based data retention threshold on the table. + +.. code-block:: shell + + CREATE TABLE example_table ( + SearchDate Date, + SearchID UInt64, + SearchPhrase String + ) + ENGINE = MergeTree + ORDER BY (SearchDate, SearchID) + PARTITION BY toYYYYMM(SearchDate) + TTL SearchDate + INTERVAL 1 WEEK TO VOLUME 'tiered' + SETTINGS storage_policy = 'tiered'; + +Add TTL to an existing table +'''''''''''''''''''''''''''' + +Add TTL to a column of an existing table. + +.. code-block:: shell + + ALTER TABLE example_table + MODIFY COLUMN + c String TTL d + INTERVAL 1 WEEK; + +Update TTL to an existing table +''''''''''''''''''''''''''''''' + +Change an already configured TTL of a column of an existing table. + +.. code-block:: shell + + ALTER TABLE example_table + MODIFY COLUMN + c String TTL d + INTERVAL 1 DAY; + +.. topic:: Result + + You have your time-based data retention policy set up. Now when a piece of data is there on your SSD for a specified time period, it's moved to the object storage, regardless of how much of the SSD capacity is still available. + +What's next +----------- + +* :doc:`Check data volume distribution between different disks ` + +Related reading +--------------- + +* :doc:`About tiered storage in Aiven for ClickHouse ` +* :doc:`Enable tiered storage in Aiven for ClickHouse ` +* :doc:`Transfer data between SSD and object storage ` +* `Manage Data with TTL (Time-to-live) `_ +* `Create table statement, TTL documentation `_ +* `MergeTree - column TTL `_ diff --git a/docs/products/clickhouse/howto/enable-tiered-storage.rst b/docs/products/clickhouse/howto/enable-tiered-storage.rst new file mode 100644 index 0000000000..9726047865 --- /dev/null +++ b/docs/products/clickhouse/howto/enable-tiered-storage.rst @@ -0,0 +1,83 @@ +Enable tiered storage in Aiven for ClickHouse® |beta| +===================================================== + +Learn how to enable the tiered storage feature on your Aiven for CliockHouse® service and activate it for specific tables. + +.. seealso:: + + Check what the tiered storage is, how it works, and why use it in :doc:`Tiered storage in Aiven for ClickHouse® `. + +About enabling tiered storage +----------------------------- + +You can enable the tiered storage feature in Aiven for ClickHouse either for a new service or for an existing one. +To enable the feature, you need to activate it on a service level in Aiven Console and, next, set it up on a table level, which is done using CLI (SQL). + +Limitations +''''''''''' + +See :ref:`tiered storage limitations ` for restrictions that apply to the feature for Aiven for ClickHouse. + +Tools +''''' + +To enable the tiered storage, you need to use the following: + +* `Aiven Console `_ +* CLI (SQL) + +Prerequisites +------------- + +* Aiven account +* Access to the `Aiven Console `_ +* Command line tool (:doc:`ClickHouse client `) +* All maintenance updates applied (check in the **Overview** tab of your service in Aiven Console) + +Enable tiered storage on a service +---------------------------------- + +Using the `Aiven Console `_, you can enable the tiered storage both on new and existing services. + +New service +''''''''''' + +You can enable the tiered storage on your new service while creating this service. For a general information on how to add a new service using Aiven Console, see :doc:`Create a new service `. One additional step you need to take to enable the tiered storage in the **Create new service** view of Aiven Console is selecting the **Enable Tiered Storage** toggle before you move on to selecting a cloud provider. + +Existing service +'''''''''''''''' + +To enable the tiered storage on your existing service, log in to `Aiven Console `_, navigate to the **Overview** tab of your service, and select **Enable Tiered Storage** at the top of the page. + +.. topic:: Result + + Your service has the tiered storage feature enabled, which is visible in the **Overview** tab as the **Tiered Storage** section added just below the **Connection information** area. + +Enable tiered storage on a table +-------------------------------- + +When you have the tiered storage feature enabled on your service, you can move on to enabling it on your tables, both new and existing ones. + +1. :doc:`Connect to your Aiven for ClickHouse service ` using, for example, the ClickHouse client (CLI). + +2. To activate the tiered storage feature on a specific table, set ``storage_policy`` to ``tiered`` on this table by executing the following SQL statement: + + .. code-block:: bash + + ALTER TABLE database-name.table-name SET storage_policy=”tiered” + +.. topic:: Result + + The tiered storage is enabled on your table and data in this table is now distributed between two tiers: SSD and object storage. + +What's next +----------- + +* :doc:`Configure data retention thresholds for tiered storage ` +* :doc:`Check data volume distribution between different disks ` + +Related reading +--------------- + +* :doc:`About tiered storage in Aiven for ClickHouse ` +* :doc:`Transfer data between SSD and object storage ` diff --git a/docs/products/clickhouse/howto/list-tiered-storage.rst b/docs/products/clickhouse/howto/list-tiered-storage.rst new file mode 100644 index 0000000000..62b9350254 --- /dev/null +++ b/docs/products/clickhouse/howto/list-tiered-storage.rst @@ -0,0 +1,20 @@ +Tiered storage +============== + +.. grid:: 1 2 2 2 + + .. grid-item-card:: :doc:`Enable tiered storage for Aiven for ClickHouse® ` + :shadow: md + :margin: 2 2 0 0 + + .. grid-item-card:: :doc:`Configure tiered storage for Aiven for ClickHouse® ` + :shadow: md + :margin: 2 2 0 0 + + .. grid-item-card:: :doc:`Check data distribution between storage devices in Aiven for ClickHouse® ` + :shadow: md + :margin: 2 2 0 0 + + .. grid-item-card:: :doc:`Transfer data between storage devices in Aiven for ClickHouse® ` + :shadow: md + :margin: 2 2 0 0 diff --git a/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst new file mode 100644 index 0000000000..43e93c2eb2 --- /dev/null +++ b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst @@ -0,0 +1,68 @@ +Transfer data between storage devices in Aiven for ClickHouse®'s tiered storage |beta| +====================================================================================== + +Check out this article for instructions on transferring data outside and inside your SSD for an Aiven for ClickHouse® service. + +About moving data between storage devices +----------------------------------------- + +You may want to :doc:`enable ` the tiered storage feature, move your data from SSD to the object storage, and size down your SSD, which allows changing a service plan to one with less SSD capacity. Also, you may want to move your date from the object storage back to your SSD. Both the operations can be performed using SQL statements on a table level. + +Prerequisites +------------- + +* Aiven account +* Tiered storage feature :doc:`enabled ` on an Aiven for ClickHouse service level +* Command line tool (:doc:`ClickHouse client `) + +Transfer data from SSD to object storage +---------------------------------------- + +If you :doc:`enable ` the tiered storage feature on your table, by default your data is moved from SSD to the object storage as soon as the SSD reaches 80% of its capacity. You can also :doc:`configure your tiered storage ` so that data is moved to the object storage at a specific time. + +1. :doc:`Connect to your Aiven for ClickHouse service ` using, for example, the ClickHouse client (CLI). + +2. Run the following query: + + .. code-block:: bash + + ALTER TABLE database-name.tablename SET storage_policy=”tiered” + +.. topic:: Result + + Now, with the tiered storage feature :doc:`enabled `, your data is moved from SSD to the object storage when the SSD reaches 80% of its capacity. + +Transfer data from object storage to SSD +---------------------------------------- + +Use the MOVE statement `MOVE PARTITION|PART `_ to transfer data to your SSD. + +1. :doc:`Connect to your Aiven for ClickHouse service ` using, for example, the ClickHouse client (CLI). + +2. Select a database for operations you intend to perform. + + .. code-block:: bash + + USE database-name + +3. Run the following query: + + .. code-block:: bash + + ALTER TABLE table_name MOVE PARTITION partition_expr TO VOLUME 'disk_name' + +.. topic:: Result + + Your data has been moved to the SSD. + +What's next +----------- + +* :doc:`Check data distribution between SSD and object storage ` +* :doc:`Configure data retention thresholds for tiered storage ` + +Related reading +--------------- + +* :doc:`About tiered storage in Aiven for ClickHouse ` +* :doc:`Enable tiered storage in Aiven for ClickHouse ` From 6db340a5606d3f33737eba724a966dd2fcefa0cb Mon Sep 17 00:00:00 2001 From: Dorota Wojcik Date: Thu, 1 Jun 2023 17:10:37 +0200 Subject: [PATCH 02/17] removing S3, cold and hot storage instances --- .../concepts/clickhouse-tiered-storage.rst | 27 ++++++++++--------- 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst index 1da8438aa9..630c0dfc78 100644 --- a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst +++ b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst @@ -10,8 +10,11 @@ The tiered storage feature introduces a method of organizing and storing data in The tiered storage in Aiven for ClickHouse consists of the following two layers: -* SSD - so called *hot storage*, the higher tier for fresh, frequently-accessed and valuable data, a fast but costly storage device -* Object storage - so called *cold storage*, Amazon S3, the lower tier for older, less valuable or rarely-accessed data, a slower but more affordable storage device +SSD - the first tier + For fresh, frequently-accessed and valuable data, a fast but costly storage device + +Object storage - the second tier + For older, less valuable or rarely-accessed data, a slower but more affordable storage device Why use it ---------- @@ -30,17 +33,17 @@ Regardless of what data retention control mechanism you choose, you receive a no .. mermaid:: sequenceDiagram - Application->>+SSD (hot data): writing data - SSD (hot data)->>S3 (cold data): moving data based
on storage policies - par Application to SSD (hot data) - Application-->>SSD (hot data): querying data - and Application to S3 (cold data) - Application-->>S3 (cold data): querying data + Application->>+SSD: writing data + SSD->>Object storage: moving data based
on storage policies + par Application to SSD + Application-->>SSD: querying data + and Application to Object storage + Application-->>Object storage: querying data end - alt if stored in S3 - S3 (cold data)->>Application: reading data + alt if stored in Object storage + Object storage->>Application: reading data else if stored in SSD - SSD (hot data)->>Application: reading data + SSD->>Application: reading data end .. note:: @@ -63,7 +66,7 @@ Limitations As a workaround, you can create a new table (without enabling the tiered storage) and copy the data from the original table (with the tiered storage :doc:`enabled `) to the new table. As soon as the data is copied to the new table, you can remove the original table. -* With the tiered storage feature :doc:`enabled `, it's not possible to connect to an external existing S3 or cloud storage bucket. +* With the tiered storage feature :doc:`enabled `, it's not possible to connect to an external existing object storage or cloud storage bucket. What's next ----------- From 7e96da450d72ebde3d28c713f9a4fb3df663fbb1 Mon Sep 17 00:00:00 2001 From: Dorota Wojcik Date: Fri, 9 Jun 2023 14:16:53 +0200 Subject: [PATCH 03/17] Aris's comments --- .../concepts/clickhouse-tiered-storage.rst | 4 +--- .../howto/configure-tiered-storage.rst | 18 ++++++++---------- .../clickhouse/howto/enable-tiered-storage.rst | 6 +++--- .../howto/transfer-data-tiered-storage.rst | 2 +- 4 files changed, 13 insertions(+), 17 deletions(-) diff --git a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst index 630c0dfc78..5e9b978c70 100644 --- a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst +++ b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst @@ -24,12 +24,10 @@ By :doc:`enabling ` and p How it works ------------ -With the tiered storage :doc:`enabled `, by default Aiven for ClickHouse stores data on your SSD until it reaches 80% of its capacity. After exceeding this size-based threshold, the data is stored in the object storage. in the object storage depending on the following. +With the tiered storage :doc:`enabled `, by default Aiven for ClickHouse stores data on your SSD until it reaches 80% of its capacity. After exceeding this size-based threshold, the data is stored in the object storage. Optionally, you can :doc:`configure the time-based threshold ` in the tiered storage. Based on the time-based threshold, the data is moved for your SSD to the object storage after a specified time period. -Regardless of what data retention control mechanism you choose, you receive a notification when a threshold is reached. - .. mermaid:: sequenceDiagram diff --git a/docs/products/clickhouse/howto/configure-tiered-storage.rst b/docs/products/clickhouse/howto/configure-tiered-storage.rst index 49091fdaaa..c37e768972 100644 --- a/docs/products/clickhouse/howto/configure-tiered-storage.rst +++ b/docs/products/clickhouse/howto/configure-tiered-storage.rst @@ -8,9 +8,9 @@ About data retention control If you have the tiered storage feature :doc:`enabled ` on your Aiven for ClickHouse service, your data is distributed between two storage devices (tiers). The data is stored either on SSD or in the object storage, depending on whether and how you configure this behavior. If you don't, by default the data is moved from SSD to the object storage when SSD reaches 80% of its capacity (default size-based data retention policy). -You may want to change this default data distribution behavior and configure your table's design so that all the data that is there on your SSD for over a specific period of time is moved to the object storage, regardless of how much of the SSD capacity is still available. +You may want to change this default data distribution behavior by :ref:`configuring your table's schema by adding a TTL (time-to-live) clause `. Such a configuration allows ignoring the SSD-capacity threshold and moving the data from SSD to the object storage based on how long the data is there on your SSD. -To enable this time-based data distribution mechanism, you can set up a retention policy (threshold) on a table level by using the TTL (time-to-live) clause. You can include the TTL clause after a column definition and/ or at the end of the table definition. For data retention control purposes, the TTL clause uses the following: +To enable this time-based data distribution mechanism, you can set up a retention policy (threshold) on a table level by using the TTL clause. For data retention control purposes, the TTL clause uses the following: * Data item of the `Date` or `DateTime` type as a reference point in time * INTERVAL clause as a time period to elapse between the reference point and the data transfer to the object storage @@ -22,6 +22,8 @@ Prerequisites * Tiered storage feature :doc:`enabled ` on an Aiven for ClickHouse service level and on a table level * Command line tool (:doc:`ClickHouse client `) +.. _time-based-retention-config: + Configure time-based data retention ----------------------------------- @@ -53,24 +55,20 @@ Create a new table with the ``storage_policy`` setting set to ``tiered`` (to :do Add TTL to an existing table '''''''''''''''''''''''''''' -Add TTL to a column of an existing table. +Use the MODIFY TTL clause: .. code-block:: shell - ALTER TABLE example_table - MODIFY COLUMN - c String TTL d + INTERVAL 1 WEEK; + ALTER TABLE database_name.table_name MODIFY TTL ttl_expression; Update TTL to an existing table ''''''''''''''''''''''''''''''' -Change an already configured TTL of a column of an existing table. +Change an already configured TTL in an existing table by using the ALTER TABLE MODIFY TTL clause: .. code-block:: shell - ALTER TABLE example_table - MODIFY COLUMN - c String TTL d + INTERVAL 1 DAY; + ALTER TABLE database_name.table_name MODIFY TTL ttl_expression; .. topic:: Result diff --git a/docs/products/clickhouse/howto/enable-tiered-storage.rst b/docs/products/clickhouse/howto/enable-tiered-storage.rst index 9726047865..be61b13ac1 100644 --- a/docs/products/clickhouse/howto/enable-tiered-storage.rst +++ b/docs/products/clickhouse/howto/enable-tiered-storage.rst @@ -1,7 +1,7 @@ Enable tiered storage in Aiven for ClickHouse® |beta| ===================================================== -Learn how to enable the tiered storage feature on your Aiven for CliockHouse® service and activate it for specific tables. +Learn how to enable the tiered storage feature on your Aiven for ClickHouse® service and activate it for specific tables. .. seealso:: @@ -11,7 +11,7 @@ About enabling tiered storage ----------------------------- You can enable the tiered storage feature in Aiven for ClickHouse either for a new service or for an existing one. -To enable the feature, you need to activate it on a service level in Aiven Console and, next, set it up on a table level, which is done using CLI (SQL). +To enable the feature, you need to activate it on a service level in Aiven Console and, next, set it up on a table level, which is done using SQL (via CLI, for example). Limitations ''''''''''' @@ -24,7 +24,7 @@ Tools To enable the tiered storage, you need to use the following: * `Aiven Console `_ -* CLI (SQL) +* SQL and an SQL client (for example, ClickHouse CLI client) Prerequisites ------------- diff --git a/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst index 43e93c2eb2..b7099befba 100644 --- a/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst +++ b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst @@ -49,7 +49,7 @@ Use the MOVE statement `MOVE PARTITION|PART Date: Tue, 20 Jun 2023 13:07:32 +0200 Subject: [PATCH 04/17] adding limited access notes --- .../clickhouse/concepts/clickhouse-tiered-storage.rst | 4 ++++ .../clickhouse/howto/configure-tiered-storage.rst | 8 ++++++-- docs/products/clickhouse/howto/enable-tiered-storage.rst | 9 +++++---- .../clickhouse/howto/transfer-data-tiered-storage.rst | 8 ++++++-- 4 files changed, 21 insertions(+), 8 deletions(-) diff --git a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst index 5e9b978c70..7c40becb09 100644 --- a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst +++ b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst @@ -3,6 +3,10 @@ Tiered storage in Aiven for ClickHouse® Discover the tiered storage capability in Aiven for ClickHouse®. Learn how it works and explore its use cases. Check why you might need it and what benefits you get using it. +.. topic:: Feature availability + + Currently, the access to this feature is limited. Contact `sales@Aiven.io `_ for details. + Overview -------- diff --git a/docs/products/clickhouse/howto/configure-tiered-storage.rst b/docs/products/clickhouse/howto/configure-tiered-storage.rst index c37e768972..0aba3ef72a 100644 --- a/docs/products/clickhouse/howto/configure-tiered-storage.rst +++ b/docs/products/clickhouse/howto/configure-tiered-storage.rst @@ -1,8 +1,12 @@ -Configure data retention thresholds in Aiven for ClickHouse®'s tiered storage |beta| -==================================================================================== +Configure data retention thresholds in Aiven for ClickHouse®'s tiered storage +============================================================================= Learn to control how your data is distributed between storage devices in the tiered storage of an Aiven for ClickHosue service. Check out how to configure tables so that your data is automatically written either to SSD or the object storage as needed. +.. topic:: Feature availability + + Currently, the access to this feature is limited. Contact `sales@Aiven.io `_ for details. + About data retention control ---------------------------- diff --git a/docs/products/clickhouse/howto/enable-tiered-storage.rst b/docs/products/clickhouse/howto/enable-tiered-storage.rst index be61b13ac1..cd64983e03 100644 --- a/docs/products/clickhouse/howto/enable-tiered-storage.rst +++ b/docs/products/clickhouse/howto/enable-tiered-storage.rst @@ -1,11 +1,12 @@ -Enable tiered storage in Aiven for ClickHouse® |beta| -===================================================== +Enable tiered storage in Aiven for ClickHouse® +============================================== Learn how to enable the tiered storage feature on your Aiven for ClickHouse® service and activate it for specific tables. +To check what the tiered storage is, how it works, and why use it, see :doc:`Tiered storage in Aiven for ClickHouse® `. -.. seealso:: +.. topic:: Feature availability - Check what the tiered storage is, how it works, and why use it in :doc:`Tiered storage in Aiven for ClickHouse® `. + Currently, the access to this feature is limited. Contact `sales@Aiven.io `_ for details. About enabling tiered storage ----------------------------- diff --git a/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst index b7099befba..4b98192683 100644 --- a/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst +++ b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst @@ -1,8 +1,12 @@ -Transfer data between storage devices in Aiven for ClickHouse®'s tiered storage |beta| -====================================================================================== +Transfer data between storage devices in Aiven for ClickHouse®'s tiered storage +=============================================================================== Check out this article for instructions on transferring data outside and inside your SSD for an Aiven for ClickHouse® service. +.. topic:: Feature availability + + Currently, the access to this feature is limited. Contact `sales@Aiven.io `_ for details. + About moving data between storage devices ----------------------------------------- From 05087ab25593d47979587824cc0c8cc614e346d3 Mon Sep 17 00:00:00 2001 From: Dorota Wojcik Date: Tue, 20 Jun 2023 13:09:41 +0200 Subject: [PATCH 05/17] adding limited access notes --- .../clickhouse/howto/check-data-tiered-storage.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/products/clickhouse/howto/check-data-tiered-storage.rst b/docs/products/clickhouse/howto/check-data-tiered-storage.rst index 0ac6310cbd..9beced46f4 100644 --- a/docs/products/clickhouse/howto/check-data-tiered-storage.rst +++ b/docs/products/clickhouse/howto/check-data-tiered-storage.rst @@ -1,8 +1,12 @@ -Check data distribution between storage devices in Aiven for ClickHouse®'s tiered storage |beta| -================================================================================================ +Check data distribution between storage devices in Aiven for ClickHouse®'s tiered storage +========================================================================================= Verify how your data is distributed between the two layers of your tiered storage: SSD and object storage. +.. topic:: Feature availability + + Currently, the access to this feature is limited. Contact `sales@Aiven.io `_ for details. + About checking data distribution -------------------------------- From a84f79980d8569b699922ada2b9b6d2ef7ba3584 Mon Sep 17 00:00:00 2001 From: Dorota Wojcik Date: Mon, 24 Jul 2023 17:30:04 +0200 Subject: [PATCH 06/17] Aris's comments --- .../clickhouse/concepts/clickhouse-tiered-storage.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst index 7c40becb09..5baa7dd9c3 100644 --- a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst +++ b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst @@ -15,10 +15,10 @@ The tiered storage feature introduces a method of organizing and storing data in The tiered storage in Aiven for ClickHouse consists of the following two layers: SSD - the first tier - For fresh, frequently-accessed and valuable data, a fast but costly storage device + Fast storage device with limited capacity, better suited for fresh and frequently queried data, relatively costly to use Object storage - the second tier - For older, less valuable or rarely-accessed data, a slower but more affordable storage device + Affordable storage device with unlimited capability, better suited for historical and more rarely queried data, relatively slower Why use it ---------- From d6fca9f87a64b61fe41cca0467570d853ac69412 Mon Sep 17 00:00:00 2001 From: Dorota Wojcik Date: Thu, 27 Jul 2023 14:26:51 +0200 Subject: [PATCH 07/17] feature availability naming updated --- .../clickhouse/concepts/clickhouse-tiered-storage.rst | 6 +++--- .../clickhouse/howto/check-data-tiered-storage.rst | 6 +++--- .../clickhouse/howto/configure-tiered-storage.rst | 6 +++--- docs/products/clickhouse/howto/enable-tiered-storage.rst | 8 ++++---- .../clickhouse/howto/transfer-data-tiered-storage.rst | 6 +++--- 5 files changed, 16 insertions(+), 16 deletions(-) diff --git a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst index 5baa7dd9c3..a20a9e6fb1 100644 --- a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst +++ b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst @@ -1,11 +1,11 @@ Tiered storage in Aiven for ClickHouse® ======================================= -Discover the tiered storage capability in Aiven for ClickHouse®. Learn how it works and explore its use cases. Check why you might need it and what benefits you get using it. +.. important:: -.. topic:: Feature availability +Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. - Currently, the access to this feature is limited. Contact `sales@Aiven.io `_ for details. +Discover the tiered storage capability in Aiven for ClickHouse®. Learn how it works and explore its use cases. Check why you might need it and what benefits you get using it. Overview -------- diff --git a/docs/products/clickhouse/howto/check-data-tiered-storage.rst b/docs/products/clickhouse/howto/check-data-tiered-storage.rst index 9beced46f4..a59babbca8 100644 --- a/docs/products/clickhouse/howto/check-data-tiered-storage.rst +++ b/docs/products/clickhouse/howto/check-data-tiered-storage.rst @@ -1,11 +1,11 @@ Check data distribution between storage devices in Aiven for ClickHouse®'s tiered storage ========================================================================================= -Verify how your data is distributed between the two layers of your tiered storage: SSD and object storage. +.. important:: -.. topic:: Feature availability +Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. - Currently, the access to this feature is limited. Contact `sales@Aiven.io `_ for details. +Verify how your data is distributed between the two layers of your tiered storage: SSD and object storage. About checking data distribution -------------------------------- diff --git a/docs/products/clickhouse/howto/configure-tiered-storage.rst b/docs/products/clickhouse/howto/configure-tiered-storage.rst index 0aba3ef72a..83c97b9cd5 100644 --- a/docs/products/clickhouse/howto/configure-tiered-storage.rst +++ b/docs/products/clickhouse/howto/configure-tiered-storage.rst @@ -1,11 +1,11 @@ Configure data retention thresholds in Aiven for ClickHouse®'s tiered storage ============================================================================= -Learn to control how your data is distributed between storage devices in the tiered storage of an Aiven for ClickHosue service. Check out how to configure tables so that your data is automatically written either to SSD or the object storage as needed. +.. important:: -.. topic:: Feature availability +Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. - Currently, the access to this feature is limited. Contact `sales@Aiven.io `_ for details. +Learn to control how your data is distributed between storage devices in the tiered storage of an Aiven for ClickHosue service. Check out how to configure tables so that your data is automatically written either to SSD or the object storage as needed. About data retention control ---------------------------- diff --git a/docs/products/clickhouse/howto/enable-tiered-storage.rst b/docs/products/clickhouse/howto/enable-tiered-storage.rst index cd64983e03..fbcfcd1550 100644 --- a/docs/products/clickhouse/howto/enable-tiered-storage.rst +++ b/docs/products/clickhouse/howto/enable-tiered-storage.rst @@ -1,12 +1,12 @@ Enable tiered storage in Aiven for ClickHouse® ============================================== -Learn how to enable the tiered storage feature on your Aiven for ClickHouse® service and activate it for specific tables. -To check what the tiered storage is, how it works, and why use it, see :doc:`Tiered storage in Aiven for ClickHouse® `. +.. important:: -.. topic:: Feature availability +Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. - Currently, the access to this feature is limited. Contact `sales@Aiven.io `_ for details. +Learn how to enable the tiered storage feature on your Aiven for ClickHouse® service and activate it for specific tables. +To check what the tiered storage is, how it works, and why use it, see :doc:`Tiered storage in Aiven for ClickHouse® `. About enabling tiered storage ----------------------------- diff --git a/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst index 4b98192683..6e9275b1f8 100644 --- a/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst +++ b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst @@ -1,11 +1,11 @@ Transfer data between storage devices in Aiven for ClickHouse®'s tiered storage =============================================================================== -Check out this article for instructions on transferring data outside and inside your SSD for an Aiven for ClickHouse® service. +.. important:: -.. topic:: Feature availability +Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. - Currently, the access to this feature is limited. Contact `sales@Aiven.io `_ for details. +Check out this article for instructions on transferring data outside and inside your SSD for an Aiven for ClickHouse® service. About moving data between storage devices ----------------------------------------- From c3de6e5fbb548b3d675a90ea3243f08b62517fdf Mon Sep 17 00:00:00 2001 From: Dorota Wojcik Date: Thu, 27 Jul 2023 14:50:49 +0200 Subject: [PATCH 08/17] fix vale --- docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst | 2 +- docs/products/clickhouse/howto/check-data-tiered-storage.rst | 2 +- docs/products/clickhouse/howto/configure-tiered-storage.rst | 2 +- docs/products/clickhouse/howto/enable-tiered-storage.rst | 2 +- docs/products/clickhouse/howto/transfer-data-tiered-storage.rst | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst index a20a9e6fb1..abe55e74bc 100644 --- a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst +++ b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst @@ -3,7 +3,7 @@ Tiered storage in Aiven for ClickHouse® .. important:: -Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. + Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. Discover the tiered storage capability in Aiven for ClickHouse®. Learn how it works and explore its use cases. Check why you might need it and what benefits you get using it. diff --git a/docs/products/clickhouse/howto/check-data-tiered-storage.rst b/docs/products/clickhouse/howto/check-data-tiered-storage.rst index a59babbca8..ec8da0ef78 100644 --- a/docs/products/clickhouse/howto/check-data-tiered-storage.rst +++ b/docs/products/clickhouse/howto/check-data-tiered-storage.rst @@ -3,7 +3,7 @@ Check data distribution between storage devices in Aiven for ClickHouse®'s tier .. important:: -Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. + Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. Verify how your data is distributed between the two layers of your tiered storage: SSD and object storage. diff --git a/docs/products/clickhouse/howto/configure-tiered-storage.rst b/docs/products/clickhouse/howto/configure-tiered-storage.rst index 83c97b9cd5..2767e25fe9 100644 --- a/docs/products/clickhouse/howto/configure-tiered-storage.rst +++ b/docs/products/clickhouse/howto/configure-tiered-storage.rst @@ -3,7 +3,7 @@ Configure data retention thresholds in Aiven for ClickHouse®'s tiered storage .. important:: -Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. + Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. Learn to control how your data is distributed between storage devices in the tiered storage of an Aiven for ClickHosue service. Check out how to configure tables so that your data is automatically written either to SSD or the object storage as needed. diff --git a/docs/products/clickhouse/howto/enable-tiered-storage.rst b/docs/products/clickhouse/howto/enable-tiered-storage.rst index fbcfcd1550..6080c498ad 100644 --- a/docs/products/clickhouse/howto/enable-tiered-storage.rst +++ b/docs/products/clickhouse/howto/enable-tiered-storage.rst @@ -3,7 +3,7 @@ Enable tiered storage in Aiven for ClickHouse® .. important:: -Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. + Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. Learn how to enable the tiered storage feature on your Aiven for ClickHouse® service and activate it for specific tables. To check what the tiered storage is, how it works, and why use it, see :doc:`Tiered storage in Aiven for ClickHouse® `. diff --git a/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst index 6e9275b1f8..2d498200cf 100644 --- a/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst +++ b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst @@ -3,7 +3,7 @@ Transfer data between storage devices in Aiven for ClickHouse®'s tiered storage .. important:: -Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. + Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. Check out this article for instructions on transferring data outside and inside your SSD for an Aiven for ClickHouse® service. From 13060fb7d8579d56604bed388b242d88286fa78a Mon Sep 17 00:00:00 2001 From: Dorota Wojcik Date: Tue, 1 Aug 2023 13:44:42 +0200 Subject: [PATCH 09/17] Ben's comments --- .../clickhouse/concepts/clickhouse-tiered-storage.rst | 6 +++--- .../products/clickhouse/howto/check-data-tiered-storage.rst | 2 +- docs/products/clickhouse/howto/configure-tiered-storage.rst | 4 ++-- docs/products/clickhouse/howto/enable-tiered-storage.rst | 2 +- docs/products/clickhouse/howto/list-tiered-storage.rst | 4 ++++ .../clickhouse/howto/transfer-data-tiered-storage.rst | 2 +- 6 files changed, 12 insertions(+), 8 deletions(-) diff --git a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst index abe55e74bc..48dbba7256 100644 --- a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst +++ b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst @@ -3,7 +3,7 @@ Tiered storage in Aiven for ClickHouse® .. important:: - Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. + Aiven for ClickHouse® tiered storage is a :doc:`limited availability feature `. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. Discover the tiered storage capability in Aiven for ClickHouse®. Learn how it works and explore its use cases. Check why you might need it and what benefits you get using it. @@ -30,7 +30,7 @@ How it works With the tiered storage :doc:`enabled `, by default Aiven for ClickHouse stores data on your SSD until it reaches 80% of its capacity. After exceeding this size-based threshold, the data is stored in the object storage. -Optionally, you can :doc:`configure the time-based threshold ` in the tiered storage. Based on the time-based threshold, the data is moved for your SSD to the object storage after a specified time period. +Optionally, you can :doc:`configure the time-based threshold ` in the tiered storage. Based on the time-based threshold, the data is moved from your SSD to the object storage after a specified time period. .. mermaid:: @@ -55,7 +55,7 @@ Optionally, you can :doc:`configure the time-based threshold ` the tiered storage for your service to make your data storage more efficient and reduce the costs. For that purpose, you select a plan for your service that supports the tiered storage and you :doc:`enable ` the feature on particular tables. You :doc:`configure the time-based threshold ` the time-based threshold for controlling how your data is stored in the two layers. +In your Aiven for ClickHouse service, there is a significant amount of data that is there for a while and is hardly ever accessed. It's stored on SSD and, thus, high-priced. You decide to :doc:`enable ` the tiered storage for your service to make your data storage more efficient and reduce the costs. For that purpose, you select a plan for your service that supports the tiered storage and you :doc:`enable ` the feature on particular tables. You :doc:`configure ` the time-based threshold for controlling how your data is stored in the two layers. .. _tiered-storage-limitations: diff --git a/docs/products/clickhouse/howto/check-data-tiered-storage.rst b/docs/products/clickhouse/howto/check-data-tiered-storage.rst index ec8da0ef78..507cd52c51 100644 --- a/docs/products/clickhouse/howto/check-data-tiered-storage.rst +++ b/docs/products/clickhouse/howto/check-data-tiered-storage.rst @@ -3,7 +3,7 @@ Check data distribution between storage devices in Aiven for ClickHouse®'s tier .. important:: - Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. + Aiven for ClickHouse® tiered storage is a :doc:`limited availability feature `. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. Verify how your data is distributed between the two layers of your tiered storage: SSD and object storage. diff --git a/docs/products/clickhouse/howto/configure-tiered-storage.rst b/docs/products/clickhouse/howto/configure-tiered-storage.rst index 2767e25fe9..ea020f6f5f 100644 --- a/docs/products/clickhouse/howto/configure-tiered-storage.rst +++ b/docs/products/clickhouse/howto/configure-tiered-storage.rst @@ -3,9 +3,9 @@ Configure data retention thresholds in Aiven for ClickHouse®'s tiered storage .. important:: - Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. + Aiven for ClickHouse® tiered storage is a :doc:`limited availability feature `. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. -Learn to control how your data is distributed between storage devices in the tiered storage of an Aiven for ClickHosue service. Check out how to configure tables so that your data is automatically written either to SSD or the object storage as needed. +Learn to control how your data is distributed between storage devices in the tiered storage of an Aiven for ClickHouse service. Check out how to configure tables so that your data is automatically written either to SSD or the object storage as needed. About data retention control ---------------------------- diff --git a/docs/products/clickhouse/howto/enable-tiered-storage.rst b/docs/products/clickhouse/howto/enable-tiered-storage.rst index 6080c498ad..a8fa291e4f 100644 --- a/docs/products/clickhouse/howto/enable-tiered-storage.rst +++ b/docs/products/clickhouse/howto/enable-tiered-storage.rst @@ -3,7 +3,7 @@ Enable tiered storage in Aiven for ClickHouse® .. important:: - Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. + Aiven for ClickHouse® tiered storage is a :doc:`limited availability feature `. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. Learn how to enable the tiered storage feature on your Aiven for ClickHouse® service and activate it for specific tables. To check what the tiered storage is, how it works, and why use it, see :doc:`Tiered storage in Aiven for ClickHouse® `. diff --git a/docs/products/clickhouse/howto/list-tiered-storage.rst b/docs/products/clickhouse/howto/list-tiered-storage.rst index 62b9350254..8c2bc31cf4 100644 --- a/docs/products/clickhouse/howto/list-tiered-storage.rst +++ b/docs/products/clickhouse/howto/list-tiered-storage.rst @@ -1,6 +1,10 @@ Tiered storage ============== +.. important:: + + Aiven for ClickHouse® tiered storage is a :doc:`limited availability feature `. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. + .. grid:: 1 2 2 2 .. grid-item-card:: :doc:`Enable tiered storage for Aiven for ClickHouse® ` diff --git a/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst index 2d498200cf..03547bc4a3 100644 --- a/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst +++ b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst @@ -3,7 +3,7 @@ Transfer data between storage devices in Aiven for ClickHouse®'s tiered storage .. important:: - Aiven for ClickHouse® tiered storage is a limited availability feature. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. + Aiven for ClickHouse® tiered storage is a :doc:`limited availability feature `. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. Check out this article for instructions on transferring data outside and inside your SSD for an Aiven for ClickHouse® service. From b78764b1dfe429e07c2db9f4e339f29fb8dc85ac Mon Sep 17 00:00:00 2001 From: Dorota Wojcik Date: Tue, 8 Aug 2023 13:58:52 +0200 Subject: [PATCH 10/17] update agains the ui and ux designs --- .../howto/check-data-tiered-storage.rst | 14 +++++++--- .../howto/configure-tiered-storage.rst | 2 +- .../howto/enable-tiered-storage.rst | 27 +++++++++++++------ 3 files changed, 31 insertions(+), 12 deletions(-) diff --git a/docs/products/clickhouse/howto/check-data-tiered-storage.rst b/docs/products/clickhouse/howto/check-data-tiered-storage.rst index 507cd52c51..bb65bc5889 100644 --- a/docs/products/clickhouse/howto/check-data-tiered-storage.rst +++ b/docs/products/clickhouse/howto/check-data-tiered-storage.rst @@ -15,12 +15,20 @@ If you have the tiered storage feature :doc:`enabled `_ * Tiered storage feature :doc:`enabled ` on an Aiven for ClickHouse service level and on a table level * Command line tool (:doc:`ClickHouse client `) -Run a data distribution check ------------------------------ +Check data distribution in Aiven Console +---------------------------------------- + +You can use `Aiven Console `_ to check if the tiered storage is enabled on your service and, if so, how much storage has been used for each type of the disks (the local SSD and the remote object storage) by particular tables. + +To check the tiered storage's status information, go to `Aiven Console `_ > your Aiven for ClickHouse service > the **Overview** page > the **Tiered storage** section. + +Run a data distribution check with the ClickHouse client (CLI) +-------------------------------------------------------------- 1. :doc:`Connect to your Aiven for ClickHouse service ` using, for example, the ClickHouse client (CLI). 2. Run the following query: diff --git a/docs/products/clickhouse/howto/configure-tiered-storage.rst b/docs/products/clickhouse/howto/configure-tiered-storage.rst index ea020f6f5f..12ef05dc5e 100644 --- a/docs/products/clickhouse/howto/configure-tiered-storage.rst +++ b/docs/products/clickhouse/howto/configure-tiered-storage.rst @@ -22,7 +22,7 @@ To enable this time-based data distribution mechanism, you can set up a retentio Prerequisites ------------- -* Aiven account +* Aiven organization * Tiered storage feature :doc:`enabled ` on an Aiven for ClickHouse service level and on a table level * Command line tool (:doc:`ClickHouse client `) diff --git a/docs/products/clickhouse/howto/enable-tiered-storage.rst b/docs/products/clickhouse/howto/enable-tiered-storage.rst index a8fa291e4f..91e71778bd 100644 --- a/docs/products/clickhouse/howto/enable-tiered-storage.rst +++ b/docs/products/clickhouse/howto/enable-tiered-storage.rst @@ -17,7 +17,13 @@ To enable the feature, you need to activate it on a service level in Aiven Conso Limitations ''''''''''' -See :ref:`tiered storage limitations ` for restrictions that apply to the feature for Aiven for ClickHouse. +* When :doc:`enabled `, the tiered storage feature cannot be deactivated. + + .. tip:: + + As a workaround, you can create a new table (without enabling the tiered storage) and copy the data from the original table (with the tiered storage :doc:`enabled `) to the new table. As soon as the data is copied to the new table, you can remove the original table. + +* With the tiered storage feature :doc:`enabled `, it's not possible to connect to an external existing object storage or cloud storage bucket. Tools ''''' @@ -30,10 +36,9 @@ To enable the tiered storage, you need to use the following: Prerequisites ------------- -* Aiven account -* Access to the `Aiven Console `_ -* Command line tool (:doc:`ClickHouse client `) -* All maintenance updates applied (check in the **Overview** tab of your service in Aiven Console) +* You have an Aiven organization and access to the `Aiven Console `_. +* You have a command line tool (:doc:`ClickHouse client `) installed. +* All maintenance updates are applied on your service (check on the **Overview** page of your service in Aiven Console). Enable tiered storage on a service ---------------------------------- @@ -43,16 +48,20 @@ Using the `Aiven Console `_, you can enable the tiere New service ''''''''''' -You can enable the tiered storage on your new service while creating this service. For a general information on how to add a new service using Aiven Console, see :doc:`Create a new service `. One additional step you need to take to enable the tiered storage in the **Create new service** view of Aiven Console is selecting the **Enable Tiered Storage** toggle before you move on to selecting a cloud provider. +.. tip:: + + For information on how to add a new service using Aiven Console, see :doc:`Create a new service `. + +While creating a new service in `Aiven Console `_, you can enable the tiered storage feature on this service in the **Create ClickHouse® service** view by selecting the **Enable Tiered Storage** toggle. Existing service '''''''''''''''' -To enable the tiered storage on your existing service, log in to `Aiven Console `_, navigate to the **Overview** tab of your service, and select **Enable Tiered Storage** at the top of the page. +To enable the tiered storage on your existing service, log in to `Aiven Console `_, navigate to the **Overview** page of your service, and select the **Enable Tiered Storage** toggle. .. topic:: Result - Your service has the tiered storage feature enabled, which is visible in the **Overview** tab as the **Tiered Storage** section added just below the **Connection information** area. + Your service has the tiered storage feature enabled, which is visible on the **Overview** page in the **Tiered Storage** section. Enable tiered storage on a table -------------------------------- @@ -71,6 +80,8 @@ When you have the tiered storage feature enabled on your service, you can move o The tiered storage is enabled on your table and data in this table is now distributed between two tiers: SSD and object storage. + You can check if the tiered storage is now supported (**Active** / **Not active**) on your table in `Aiven Console `_ > **Databases & Tables** > **Databases lists** > Your database > Your table > the **Tiered storage** column. + What's next ----------- From bfde857af776558deda94f0568a36ea9c0060781 Mon Sep 17 00:00:00 2001 From: Dorota Wojcik Date: Thu, 17 Aug 2023 16:29:10 +0200 Subject: [PATCH 11/17] updates for new enablement method --- .../concepts/clickhouse-tiered-storage.rst | 2 +- .../howto/enable-tiered-storage.rst | 34 +++++-------------- 2 files changed, 9 insertions(+), 27 deletions(-) diff --git a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst index 48dbba7256..e68632313f 100644 --- a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst +++ b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst @@ -55,7 +55,7 @@ Optionally, you can :doc:`configure the time-based threshold ` the tiered storage for your service to make your data storage more efficient and reduce the costs. For that purpose, you select a plan for your service that supports the tiered storage and you :doc:`enable ` the feature on particular tables. You :doc:`configure ` the time-based threshold for controlling how your data is stored in the two layers. +In your Aiven for ClickHouse service, there is a significant amount of data that is there for a while and is hardly ever accessed. It's stored on SSD and, thus, high-priced. You decide to :doc:`enable ` the tiered storage for your service to make your data storage more efficient and reduce the costs. For that purpose, you contact the sales team at `sales@Aiven.io `_ to have it enabled on your project, and you :doc:`enable ` the feature on particular tables. You :doc:`configure ` the time-based threshold for controlling how your data is stored in the two layers. .. _tiered-storage-limitations: diff --git a/docs/products/clickhouse/howto/enable-tiered-storage.rst b/docs/products/clickhouse/howto/enable-tiered-storage.rst index 91e71778bd..5b6b8c11c3 100644 --- a/docs/products/clickhouse/howto/enable-tiered-storage.rst +++ b/docs/products/clickhouse/howto/enable-tiered-storage.rst @@ -5,14 +5,13 @@ Enable tiered storage in Aiven for ClickHouse® Aiven for ClickHouse® tiered storage is a :doc:`limited availability feature `. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. -Learn how to enable the tiered storage feature on your Aiven for ClickHouse® service and activate it for specific tables. +Learn how to enable the tiered storage feature on your project and activate it for specific tables. To check what the tiered storage is, how it works, and why use it, see :doc:`Tiered storage in Aiven for ClickHouse® `. About enabling tiered storage ----------------------------- -You can enable the tiered storage feature in Aiven for ClickHouse either for a new service or for an existing one. -To enable the feature, you need to activate it on a service level in Aiven Console and, next, set it up on a table level, which is done using SQL (via CLI, for example). +To use the tiered storage feature, you need to enable it on the project level by contacting the sales team at `sales@Aiven.io `_ and set it up on the table level using SQL (via CLI, for example). Limitations ''''''''''' @@ -28,45 +27,28 @@ Limitations Tools ''''' -To enable the tiered storage, you need to use the following: - -* `Aiven Console `_ -* SQL and an SQL client (for example, ClickHouse CLI client) +To enable the tiered storage, use SQL and an SQL client (for example, the ClickHouse CLI client). Prerequisites ------------- -* You have an Aiven organization and access to the `Aiven Console `_. +* You have an Aiven organization and at least one project. * You have a command line tool (:doc:`ClickHouse client `) installed. * All maintenance updates are applied on your service (check on the **Overview** page of your service in Aiven Console). -Enable tiered storage on a service +Enable tiered storage on a project ---------------------------------- -Using the `Aiven Console `_, you can enable the tiered storage both on new and existing services. - -New service -''''''''''' - -.. tip:: - - For information on how to add a new service using Aiven Console, see :doc:`Create a new service `. - -While creating a new service in `Aiven Console `_, you can enable the tiered storage feature on this service in the **Create ClickHouse® service** view by selecting the **Enable Tiered Storage** toggle. - -Existing service -'''''''''''''''' - -To enable the tiered storage on your existing service, log in to `Aiven Console `_, navigate to the **Overview** page of your service, and select the **Enable Tiered Storage** toggle. +To enable the tiered storage feature on your project, request it from the sales team at `sales@Aiven.io `_. .. topic:: Result - Your service has the tiered storage feature enabled, which is visible on the **Overview** page in the **Tiered Storage** section. + Your project supports the tiered storage feature, and you can enable the feature on tables for your Aiven for ClickHouse services. Enable tiered storage on a table -------------------------------- -When you have the tiered storage feature enabled on your service, you can move on to enabling it on your tables, both new and existing ones. +When you have the tiered storage feature enabled on your project, you can move on to enabling it on your tables, both new and existing ones. 1. :doc:`Connect to your Aiven for ClickHouse service ` using, for example, the ClickHouse client (CLI). From 51b2b9536849ee2161df0578a058367d7be32a31 Mon Sep 17 00:00:00 2001 From: Dorota Wojcik Date: Thu, 17 Aug 2023 17:00:53 +0200 Subject: [PATCH 12/17] Ben's and John's comments --- .../clickhouse/concepts/clickhouse-tiered-storage.rst | 10 +++++----- .../clickhouse/howto/check-data-tiered-storage.rst | 7 +++---- 2 files changed, 8 insertions(+), 9 deletions(-) diff --git a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst index e68632313f..c4d36d1446 100644 --- a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst +++ b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst @@ -23,14 +23,14 @@ Object storage - the second tier Why use it ---------- -By :doc:`enabling ` and properly :doc:`configuring ` the tiered storage in Aiven for ClickHouse, you can use storage resources efficiently and, therefore, significantly reduce costs of storing data ingested into an Aiven for ClickHouse instance. +By :doc:`enabling ` and properly :doc:`configuring ` the tiered storage feature in Aiven for ClickHouse, you can use storage resources efficiently and, therefore, significantly reduce costs of storing data ingested into an Aiven for ClickHouse instance. How it works ------------ -With the tiered storage :doc:`enabled `, by default Aiven for ClickHouse stores data on your SSD until it reaches 80% of its capacity. After exceeding this size-based threshold, the data is stored in the object storage. +With the tiered storage feature :doc:`enabled `, by default Aiven for ClickHouse stores data on your SSD until it reaches 80% of its capacity. After exceeding this size-based threshold, the data is stored in the object storage. -Optionally, you can :doc:`configure the time-based threshold ` in the tiered storage. Based on the time-based threshold, the data is moved from your SSD to the object storage after a specified time period. +Optionally, you can :doc:`configure the time-based threshold ` for the tiered storage feature. Based on the time-based threshold, the data is moved from your SSD to the object storage after a specified time period. .. mermaid:: @@ -55,7 +55,7 @@ Optionally, you can :doc:`configure the time-based threshold ` the tiered storage for your service to make your data storage more efficient and reduce the costs. For that purpose, you contact the sales team at `sales@Aiven.io `_ to have it enabled on your project, and you :doc:`enable ` the feature on particular tables. You :doc:`configure ` the time-based threshold for controlling how your data is stored in the two layers. +In your Aiven for ClickHouse service, there is a significant amount of data that is there for a while and is hardly ever accessed. It's stored on SSD and, thus, high-priced. You decide to :doc:`enable ` the tiered storage feature for your service to make your data storage more efficient and reduce the costs. For that purpose, you contact the sales team at `sales@Aiven.io `_ to have it enabled on your project, and you :doc:`enable ` the feature on particular tables. You :doc:`configure ` the time-based threshold for controlling how your data is stored in the two layers. .. _tiered-storage-limitations: @@ -66,7 +66,7 @@ Limitations .. tip:: - As a workaround, you can create a new table (without enabling the tiered storage) and copy the data from the original table (with the tiered storage :doc:`enabled `) to the new table. As soon as the data is copied to the new table, you can remove the original table. + As a workaround, you can create a new table (without enabling the tiered storage feature) and copy the data from the original table (with the tiered storage feature :doc:`enabled `) to the new table. As soon as the data is copied to the new table, you can remove the original table. * With the tiered storage feature :doc:`enabled `, it's not possible to connect to an external existing object storage or cloud storage bucket. diff --git a/docs/products/clickhouse/howto/check-data-tiered-storage.rst b/docs/products/clickhouse/howto/check-data-tiered-storage.rst index bb65bc5889..11761bb56e 100644 --- a/docs/products/clickhouse/howto/check-data-tiered-storage.rst +++ b/docs/products/clickhouse/howto/check-data-tiered-storage.rst @@ -15,9 +15,8 @@ If you have the tiered storage feature :doc:`enabled `_ -* Tiered storage feature :doc:`enabled ` on an Aiven for ClickHouse service level and on a table level +* Tiered storage feature :doc:`enabled ` * Command line tool (:doc:`ClickHouse client `) Check data distribution in Aiven Console @@ -25,7 +24,7 @@ Check data distribution in Aiven Console You can use `Aiven Console `_ to check if the tiered storage is enabled on your service and, if so, how much storage has been used for each type of the disks (the local SSD and the remote object storage) by particular tables. -To check the tiered storage's status information, go to `Aiven Console `_ > your Aiven for ClickHouse service > the **Overview** page > the **Tiered storage** section. +To check the tiered storage's status information, go to `Aiven Console `_ > your Aiven for ClickHouse service > the **Databases and tables** page > your database > your table > **View details** > **Storage details**. Run a data distribution check with the ClickHouse client (CLI) -------------------------------------------------------------- @@ -66,7 +65,7 @@ Run a data distribution check with the ClickHouse client (CLI) .. topic:: Result - The query returns a table with data distribution details for all databases and tables that belong to your service: storage device they use, their total sizes as well as parts' counts and sizing. + The query returns a table with data distribution details for all databases and tables that belong to your service: the storage device they use, their total sizes as well as parts counts and sizing. What's next ----------- From 9184641bf045c3716d91244a9f379bb99dab435b Mon Sep 17 00:00:00 2001 From: Dorota Wojcik Date: Fri, 18 Aug 2023 15:04:33 +0200 Subject: [PATCH 13/17] editorial fixes --- .../products/clickhouse/howto/check-data-tiered-storage.rst | 2 +- docs/products/clickhouse/howto/configure-tiered-storage.rst | 2 +- .../clickhouse/howto/transfer-data-tiered-storage.rst | 6 +++--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/products/clickhouse/howto/check-data-tiered-storage.rst b/docs/products/clickhouse/howto/check-data-tiered-storage.rst index 11761bb56e..68e6ed35d6 100644 --- a/docs/products/clickhouse/howto/check-data-tiered-storage.rst +++ b/docs/products/clickhouse/howto/check-data-tiered-storage.rst @@ -10,7 +10,7 @@ Verify how your data is distributed between the two layers of your tiered storag About checking data distribution -------------------------------- -If you have the tiered storage feature :doc:`enabled ` on your Aiven for ClickHouse service, your data is distributed between two storage devices (tiers). You can learn on what storage devices specific databases and tables are stored. You can also preview their total sizes as well as part counts, minimum part sizes, median part sizes, and maximum part sizes. +If you have the tiered storage feature :doc:`enabled ` on your project, your data in Aiven for ClickHouse is distributed between two storage devices (tiers). You can learn on what storage devices specific databases and tables are stored. You can also preview their total sizes as well as part counts, minimum part sizes, median part sizes, and maximum part sizes. Prerequisites ------------- diff --git a/docs/products/clickhouse/howto/configure-tiered-storage.rst b/docs/products/clickhouse/howto/configure-tiered-storage.rst index 12ef05dc5e..a73e9d78fe 100644 --- a/docs/products/clickhouse/howto/configure-tiered-storage.rst +++ b/docs/products/clickhouse/howto/configure-tiered-storage.rst @@ -23,7 +23,7 @@ Prerequisites ------------- * Aiven organization -* Tiered storage feature :doc:`enabled ` on an Aiven for ClickHouse service level and on a table level +* Tiered storage feature :doc:`enabled ` on the project level and on the table level * Command line tool (:doc:`ClickHouse client `) .. _time-based-retention-config: diff --git a/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst index 03547bc4a3..bd66aeac82 100644 --- a/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst +++ b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst @@ -10,13 +10,13 @@ Check out this article for instructions on transferring data outside and inside About moving data between storage devices ----------------------------------------- -You may want to :doc:`enable ` the tiered storage feature, move your data from SSD to the object storage, and size down your SSD, which allows changing a service plan to one with less SSD capacity. Also, you may want to move your date from the object storage back to your SSD. Both the operations can be performed using SQL statements on a table level. +You may want to :doc:`enable ` the tiered storage feature, move your data from SSD to the object storage, and size down your SSD, which allows changing a service plan to one with less SSD capacity. Also, you may want to move your data from the object storage back to your SSD. Both the operations can be performed using SQL statements on the table level. Prerequisites ------------- -* Aiven account -* Tiered storage feature :doc:`enabled ` on an Aiven for ClickHouse service level +* Aiven organization +* Tiered storage feature :doc:`enabled ` on the project level * Command line tool (:doc:`ClickHouse client `) Transfer data from SSD to object storage From 86397949bd43a7cd6a045a29ae21aacd21954553 Mon Sep 17 00:00:00 2001 From: Dorota Wojcik Date: Wed, 6 Sep 2023 14:17:15 +0200 Subject: [PATCH 14/17] update the enabling command --- docs/products/clickhouse/howto/enable-tiered-storage.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/products/clickhouse/howto/enable-tiered-storage.rst b/docs/products/clickhouse/howto/enable-tiered-storage.rst index 5b6b8c11c3..2c90add035 100644 --- a/docs/products/clickhouse/howto/enable-tiered-storage.rst +++ b/docs/products/clickhouse/howto/enable-tiered-storage.rst @@ -56,7 +56,7 @@ When you have the tiered storage feature enabled on your project, you can move o .. code-block:: bash - ALTER TABLE database-name.table-name SET storage_policy=”tiered” + ALTER TABLE database-name.table-name MODIFY SETTING storage_policy = 'tiered' .. topic:: Result From f7acca711f01f7d0e21b1c31ba5f29b7c8ddcfdd Mon Sep 17 00:00:00 2001 From: Dorota Wojcik Date: Fri, 8 Sep 2023 13:54:00 +0200 Subject: [PATCH 15/17] updating status terminology to align with the copy --- docs/products/clickhouse/howto/enable-tiered-storage.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/products/clickhouse/howto/enable-tiered-storage.rst b/docs/products/clickhouse/howto/enable-tiered-storage.rst index 2c90add035..0b789f9d99 100644 --- a/docs/products/clickhouse/howto/enable-tiered-storage.rst +++ b/docs/products/clickhouse/howto/enable-tiered-storage.rst @@ -62,7 +62,7 @@ When you have the tiered storage feature enabled on your project, you can move o The tiered storage is enabled on your table and data in this table is now distributed between two tiers: SSD and object storage. - You can check if the tiered storage is now supported (**Active** / **Not active**) on your table in `Aiven Console `_ > **Databases & Tables** > **Databases lists** > Your database > Your table > the **Tiered storage** column. + You can check if the tiered storage is now supported (**Active** / **Inactive**) on your table in `Aiven Console `_ > **Databases & Tables** > **Databases lists** > Your database > Your table > the **Tiered storage** column. What's next ----------- From 1731de7aa5f1e57e08daa52a13636bce273255c5 Mon Sep 17 00:00:00 2001 From: Dorota Wojcik Date: Sun, 17 Sep 2023 18:18:08 +0200 Subject: [PATCH 16/17] Arthur's comments --- .../concepts/clickhouse-tiered-storage.rst | 12 +++++------ .../howto/check-data-tiered-storage.rst | 8 ++++---- .../howto/configure-tiered-storage.rst | 10 +++++----- .../howto/enable-tiered-storage.rst | 20 +++++++++---------- .../howto/transfer-data-tiered-storage.rst | 10 +++++----- 5 files changed, 30 insertions(+), 30 deletions(-) diff --git a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst index c4d36d1446..2d5becf515 100644 --- a/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst +++ b/docs/products/clickhouse/concepts/clickhouse-tiered-storage.rst @@ -5,7 +5,7 @@ Tiered storage in Aiven for ClickHouse® Aiven for ClickHouse® tiered storage is a :doc:`limited availability feature `. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. -Discover the tiered storage capability in Aiven for ClickHouse®. Learn how it works and explore its use cases. Check why you might need it and what benefits you get using it. +Discover the tiered storage capability in Aiven for ClickHouse®. Learn how it works and explore its use cases. Check out why you might need it and what benefits you get using it. Overview -------- @@ -23,14 +23,14 @@ Object storage - the second tier Why use it ---------- -By :doc:`enabling ` and properly :doc:`configuring ` the tiered storage feature in Aiven for ClickHouse, you can use storage resources efficiently and, therefore, significantly reduce costs of storing data ingested into an Aiven for ClickHouse instance. +By :doc:`enabling ` and properly :doc:`configuring ` the tiered storage feature in Aiven for ClickHouse, you can use storage resources efficiently and, therefore, significantly reduce storage costs of your Aiven for ClickHouse instance. How it works ------------ -With the tiered storage feature :doc:`enabled `, by default Aiven for ClickHouse stores data on your SSD until it reaches 80% of its capacity. After exceeding this size-based threshold, the data is stored in the object storage. +After you :doc:`enable ` the tiered storage feature, Aiven for ClickHouse by default stores data on SSD until it reaches 80% of its capacity. After exceeding this size-based threshold, data is stored in object storage. -Optionally, you can :doc:`configure the time-based threshold ` for the tiered storage feature. Based on the time-based threshold, the data is moved from your SSD to the object storage after a specified time period. +Optionally, you can :doc:`configure the time-based threshold ` for your storage. Based on the time-based threshold, the data is moved from your SSD to object storage after a specified time period. .. mermaid:: @@ -55,7 +55,7 @@ Optionally, you can :doc:`configure the time-based threshold ` the tiered storage feature for your service to make your data storage more efficient and reduce the costs. For that purpose, you contact the sales team at `sales@Aiven.io `_ to have it enabled on your project, and you :doc:`enable ` the feature on particular tables. You :doc:`configure ` the time-based threshold for controlling how your data is stored in the two layers. +In your Aiven for ClickHouse service, there is a significant amount of data that is there for a while and is hardly ever accessed. It's stored on SSD and, thus, high-priced. You decide to :doc:`enable ` tiered storage for your service to make your data storage more efficient and reduce the costs. For that purpose, you contact the sales team at `sales@Aiven.io `_ to have it enabled on your project, and you :doc:`enable ` the feature on tables you want to optimize. You :doc:`configure ` the time-based threshold to control how your data is stored between the two layers. .. _tiered-storage-limitations: @@ -66,7 +66,7 @@ Limitations .. tip:: - As a workaround, you can create a new table (without enabling the tiered storage feature) and copy the data from the original table (with the tiered storage feature :doc:`enabled `) to the new table. As soon as the data is copied to the new table, you can remove the original table. + As a workaround, you can create a new table (without enabling tiered storage on it) and copy the data from the original table (with the tiered storage feature :doc:`enabled `) to the new table. As soon as the data is copied to the new table, you can remove the original table. * With the tiered storage feature :doc:`enabled `, it's not possible to connect to an external existing object storage or cloud storage bucket. diff --git a/docs/products/clickhouse/howto/check-data-tiered-storage.rst b/docs/products/clickhouse/howto/check-data-tiered-storage.rst index 68e6ed35d6..fa031fa39c 100644 --- a/docs/products/clickhouse/howto/check-data-tiered-storage.rst +++ b/docs/products/clickhouse/howto/check-data-tiered-storage.rst @@ -5,12 +5,12 @@ Check data distribution between storage devices in Aiven for ClickHouse®'s tier Aiven for ClickHouse® tiered storage is a :doc:`limited availability feature `. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. -Verify how your data is distributed between the two layers of your tiered storage: SSD and object storage. +Monitor how your data is distributed between the two layers of your tiered storage: SSD and object storage. About checking data distribution -------------------------------- -If you have the tiered storage feature :doc:`enabled ` on your project, your data in Aiven for ClickHouse is distributed between two storage devices (tiers). You can learn on what storage devices specific databases and tables are stored. You can also preview their total sizes as well as part counts, minimum part sizes, median part sizes, and maximum part sizes. +If you have the tiered storage feature :doc:`enabled ` on your project, your data in Aiven for ClickHouse is distributed between two storage devices (tiers). You can check on what storage devices your databases and tables are stored. You can also preview their total sizes as well as part counts, minimum part sizes, median part sizes, and maximum part sizes. Prerequisites ------------- @@ -22,9 +22,9 @@ Prerequisites Check data distribution in Aiven Console ---------------------------------------- -You can use `Aiven Console `_ to check if the tiered storage is enabled on your service and, if so, how much storage has been used for each type of the disks (the local SSD and the remote object storage) by particular tables. +You can use `Aiven Console `_ to check if tiered storage is enabled on your service and, if it is, how much storage is used on each tier (local SSD and remote object storage) for particular tables. -To check the tiered storage's status information, go to `Aiven Console `_ > your Aiven for ClickHouse service > the **Databases and tables** page > your database > your table > **View details** > **Storage details**. +To access tiered storage's status information, go to `Aiven Console `_ > your Aiven for ClickHouse service > the **Databases and tables** page > your database > your table > **View details** > **Storage details**. Run a data distribution check with the ClickHouse client (CLI) -------------------------------------------------------------- diff --git a/docs/products/clickhouse/howto/configure-tiered-storage.rst b/docs/products/clickhouse/howto/configure-tiered-storage.rst index a73e9d78fe..76c68580d3 100644 --- a/docs/products/clickhouse/howto/configure-tiered-storage.rst +++ b/docs/products/clickhouse/howto/configure-tiered-storage.rst @@ -5,19 +5,19 @@ Configure data retention thresholds in Aiven for ClickHouse®'s tiered storage Aiven for ClickHouse® tiered storage is a :doc:`limited availability feature `. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. -Learn to control how your data is distributed between storage devices in the tiered storage of an Aiven for ClickHouse service. Check out how to configure tables so that your data is automatically written either to SSD or the object storage as needed. +Learn to control how your data is distributed between storage devices in the tiered storage of an Aiven for ClickHouse service. Check out how to configure tables so that your data is automatically written either to SSD or object storage as needed. About data retention control ---------------------------- -If you have the tiered storage feature :doc:`enabled ` on your Aiven for ClickHouse service, your data is distributed between two storage devices (tiers). The data is stored either on SSD or in the object storage, depending on whether and how you configure this behavior. If you don't, by default the data is moved from SSD to the object storage when SSD reaches 80% of its capacity (default size-based data retention policy). +If you have the tiered storage feature :doc:`enabled ` on your Aiven for ClickHouse service, your data is distributed between two storage devices (tiers). The data is stored either on SSD or in object storage, depending on whether and how you configure this behavior. By default, data is moved from SSD to object storage when SSD reaches 80% of its capacity (default size-based data retention policy). -You may want to change this default data distribution behavior by :ref:`configuring your table's schema by adding a TTL (time-to-live) clause `. Such a configuration allows ignoring the SSD-capacity threshold and moving the data from SSD to the object storage based on how long the data is there on your SSD. +You may want to change this default data distribution behavior by :ref:`configuring your table's schema by adding a TTL (time-to-live) clause `. Such a configuration allows ignoring the SSD-capacity threshold and moving the data from SSD to object storage based on how long the data is there on your SSD. To enable this time-based data distribution mechanism, you can set up a retention policy (threshold) on a table level by using the TTL clause. For data retention control purposes, the TTL clause uses the following: * Data item of the `Date` or `DateTime` type as a reference point in time -* INTERVAL clause as a time period to elapse between the reference point and the data transfer to the object storage +* INTERVAL clause as a time period to elapse between the reference point and the data transfer to object storage Prerequisites ------------- @@ -76,7 +76,7 @@ Change an already configured TTL in an existing table by using the ALTER TABLE M .. topic:: Result - You have your time-based data retention policy set up. Now when a piece of data is there on your SSD for a specified time period, it's moved to the object storage, regardless of how much of the SSD capacity is still available. + You have your time-based data retention policy set up. From now on, when data is on your SSD longer than a specified time period, it's moved to object storage, regardless of how much of SSD capacity is still available. What's next ----------- diff --git a/docs/products/clickhouse/howto/enable-tiered-storage.rst b/docs/products/clickhouse/howto/enable-tiered-storage.rst index 0b789f9d99..4440528b62 100644 --- a/docs/products/clickhouse/howto/enable-tiered-storage.rst +++ b/docs/products/clickhouse/howto/enable-tiered-storage.rst @@ -5,13 +5,13 @@ Enable tiered storage in Aiven for ClickHouse® Aiven for ClickHouse® tiered storage is a :doc:`limited availability feature `. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. -Learn how to enable the tiered storage feature on your project and activate it for specific tables. -To check what the tiered storage is, how it works, and why use it, see :doc:`Tiered storage in Aiven for ClickHouse® `. +Find out how to enable the tiered storage feature on your project and activate it for specific tables. +To learn what tiered storage is, how it works, and why use it, see :doc:`Tiered storage in Aiven for ClickHouse® `. About enabling tiered storage ----------------------------- -To use the tiered storage feature, you need to enable it on the project level by contacting the sales team at `sales@Aiven.io `_ and set it up on the table level using SQL (via CLI, for example). +To use the tiered storage feature, you need to enable it at project level by contacting the sales team at `sales@Aiven.io `_ and set it up at table level using SQL (via CLI, for example). Limitations ''''''''''' @@ -20,14 +20,14 @@ Limitations .. tip:: - As a workaround, you can create a new table (without enabling the tiered storage) and copy the data from the original table (with the tiered storage :doc:`enabled `) to the new table. As soon as the data is copied to the new table, you can remove the original table. + As a workaround, you can create a new table (without enabling tiered storage on it) and copy the data from the original table (with the tiered storage :doc:`enabled `) to the new table. As soon as the data is copied to the new table, you can remove the original table. * With the tiered storage feature :doc:`enabled `, it's not possible to connect to an external existing object storage or cloud storage bucket. Tools ''''' -To enable the tiered storage, use SQL and an SQL client (for example, the ClickHouse CLI client). +To enable tiered storage, use SQL and an SQL client (for example, the ClickHouse CLI client). Prerequisites ------------- @@ -39,16 +39,16 @@ Prerequisites Enable tiered storage on a project ---------------------------------- -To enable the tiered storage feature on your project, request it from the sales team at `sales@Aiven.io `_. +To enable tiered storage on your project, request it from the sales team at `sales@Aiven.io `_. .. topic:: Result - Your project supports the tiered storage feature, and you can enable the feature on tables for your Aiven for ClickHouse services. + Your project now supports tiered storage, and you can enable it for each table of your Aiven for ClickHouse services. Enable tiered storage on a table -------------------------------- -When you have the tiered storage feature enabled on your project, you can move on to enabling it on your tables, both new and existing ones. +When you have tiered storage enabled on your project, you can move on to enabling it on your tables, both new and existing ones. 1. :doc:`Connect to your Aiven for ClickHouse service ` using, for example, the ClickHouse client (CLI). @@ -60,9 +60,9 @@ When you have the tiered storage feature enabled on your project, you can move o .. topic:: Result - The tiered storage is enabled on your table and data in this table is now distributed between two tiers: SSD and object storage. + Tiered storage is enabled on your table and data in this table is now distributed between two tiers: SSD and object storage. - You can check if the tiered storage is now supported (**Active** / **Inactive**) on your table in `Aiven Console `_ > **Databases & Tables** > **Databases lists** > Your database > Your table > the **Tiered storage** column. + You can check if tiered storage is now supported (**Active** / **Inactive**) on your table in `Aiven Console `_ > **Databases & Tables** > **Databases lists** > Your database > Your table > the **Tiered storage** column. What's next ----------- diff --git a/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst index bd66aeac82..b5d98cdf53 100644 --- a/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst +++ b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst @@ -5,24 +5,24 @@ Transfer data between storage devices in Aiven for ClickHouse®'s tiered storage Aiven for ClickHouse® tiered storage is a :doc:`limited availability feature `. If you're interested in trying out this feature, contact the sales team at `sales@Aiven.io `_. -Check out this article for instructions on transferring data outside and inside your SSD for an Aiven for ClickHouse® service. +Check out this article for instructions on transferring data from and to SSD for an Aiven for ClickHouse® service. About moving data between storage devices ----------------------------------------- -You may want to :doc:`enable ` the tiered storage feature, move your data from SSD to the object storage, and size down your SSD, which allows changing a service plan to one with less SSD capacity. Also, you may want to move your data from the object storage back to your SSD. Both the operations can be performed using SQL statements on the table level. +After :doc:`enabling ` the tiered storage feature, you can move your data from SSD to object storage. Next, you may want to size down your SSD by selecting a service plan with less SSD capacity. Later, you can move your data from object storage back to your SSD if needed. Both operations can be performed using SQL statements against your tables directly. Prerequisites ------------- * Aiven organization -* Tiered storage feature :doc:`enabled ` on the project level +* Tiered storage feature :doc:`enabled ` at project level * Command line tool (:doc:`ClickHouse client `) Transfer data from SSD to object storage ---------------------------------------- -If you :doc:`enable ` the tiered storage feature on your table, by default your data is moved from SSD to the object storage as soon as the SSD reaches 80% of its capacity. You can also :doc:`configure your tiered storage ` so that data is moved to the object storage at a specific time. +If you :doc:`enable ` the tiered storage feature on your table, by default your data is moved from SSD to object storage as soon as the SSD reaches 80% of its capacity. You can also :doc:`configure your tiered storage ` so that data is moved to object storage at a specific time. 1. :doc:`Connect to your Aiven for ClickHouse service ` using, for example, the ClickHouse client (CLI). @@ -34,7 +34,7 @@ If you :doc:`enable ` the .. topic:: Result - Now, with the tiered storage feature :doc:`enabled `, your data is moved from SSD to the object storage when the SSD reaches 80% of its capacity. + Now, with the tiered storage feature :doc:`enabled `, your data is moved from SSD to object storage when the SSD reaches 80% of its capacity. Transfer data from object storage to SSD ---------------------------------------- From 75a8cc33e5152e476edbbab6ba496f1e3cc36922 Mon Sep 17 00:00:00 2001 From: Dorota Wojcik Date: Tue, 19 Sep 2023 07:07:52 +0200 Subject: [PATCH 17/17] Aris's comments --- docs/products/clickhouse/howto/configure-tiered-storage.rst | 2 +- docs/products/clickhouse/howto/transfer-data-tiered-storage.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/products/clickhouse/howto/configure-tiered-storage.rst b/docs/products/clickhouse/howto/configure-tiered-storage.rst index 76c68580d3..ec71794824 100644 --- a/docs/products/clickhouse/howto/configure-tiered-storage.rst +++ b/docs/products/clickhouse/howto/configure-tiered-storage.rst @@ -41,7 +41,7 @@ Configure time-based data retention Add TTL to a new table '''''''''''''''''''''' -Create a new table with the ``storage_policy`` setting set to ``tiered`` (to :doc:`enabled ` the feature) and TTL (time-to-live) configured to add a time-based data retention threshold on the table. +Create a new table with the ``storage_policy`` setting set to ``tiered`` (to :doc:`enable ` the feature) and TTL (time-to-live) configured to add a time-based data retention threshold on the table. .. code-block:: shell diff --git a/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst index b5d98cdf53..113cbb6adf 100644 --- a/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst +++ b/docs/products/clickhouse/howto/transfer-data-tiered-storage.rst @@ -30,7 +30,7 @@ If you :doc:`enable ` the .. code-block:: bash - ALTER TABLE database-name.tablename SET storage_policy=”tiered” + ALTER TABLE database-name.tablename MODIFY SETTING storage_policy = 'tiered' .. topic:: Result