-
@@ -96,8 +94,11 @@
- [`lagging_ranges_threshold`]({% link {{ page.version.version }}/create-changefeed.md %}#lagging-ranges-threshold), which is the amount of time that a range checkpoint needs to be in the past to be considered lagging.
- [`lagging_ranges_polling_interval`]({% link {{ page.version.version }}/create-changefeed.md %}#lagging-ranges-polling-interval), which is the frequency at which lagging ranges are polled and the metric is updated.
"alteredSettings": {
- "cluster.organization": "\u003credacted\u003e", "kv.lease.expiration_leases_only.enabled": "false",
- "kv.rangefeed.enabled": "true"
+ "cluster.organization": "\u003credacted\u003e",
+ "cluster.secret": "\u003credacted\u003e",
+ "kv.rangefeed.enabled": "true",
+ "server.oidc_authentication.client_id": "\u003credacted\u003e",
+ "server.oidc_authentication.client_secret": "\u003credacted\u003e",
}
Security updates
+ +- All cluster settings that accept strings are now fully redacted when transmitted as part of CockroachDB's diagnostics telemetry. The transmitted payload includes a record of modified cluster settings and their values when they are not strings. If you previously applied the mitigations in [Technical Advisory 133479]({% link advisories/a133479.md %}), you can safely turn on diagnostic reporting via the `diagnostics.reporting.enabled` cluster setting without leaking sensitive cluster settings values. [#134063][#134063] + +General changes
+ +- `COCKROACH_SKIP_ENABLING_DIAGNOSTIC_REPORTING` is no longer mentioned in the `cockroach demo` command. [#134083][#134083] + +Bug fixes
+ +- Fixed a bug where CockroachDB could encounter an internal error `interface conversion: coldata.Column is` in an edge case. The bug was present in v22.2.13+, v23.1.9+, and v23.2+. [#133758][#133758] +- Fixed a bug that caused incorrect `NOT NULL` constraint violation errors on `UPSERT` and `INSERT ... ON CONFLICT ... DO UPDATE` statements when those statements updated an existing row and a subset of columns that did not include a `NOT NULL` column of the table. This bug had been present since at least v20.1.0. [#133824][#133824] +- Fixed an unhandled error that could occur when using `REVOKE ... ON SEQUENCE FROM ... user` on an object that is not a sequence. [#133706][#133706] +- Addressed a panic that could occur inside `CREATE TABLE AS` if sequence builtin expressions had invalid function overloads. [#133866][#133866] +- Previously, when executing queries with index / lookup joins where ordering needed to be maintained, CockroachDB's behavior could lead to increased query latency, potentially by several orders of magnitude. This bug was introduced in v22.2, and is now fixed. [#134363][#134363] +- Fixed a bug where `DROP CASCADE` would occasionally panic with an `un-dropped backref` message on partitioned tables. [#134524][#134524] +- Reduced the duration of partitions in the gossip network when a node crashes. This eliminates false positives in the `ranges.unavailable` metric. [#134603][#134603] + +[#133706]: https://github.com/cockroachdb/cockroach/pull/133706 +[#133758]: https://github.com/cockroachdb/cockroach/pull/133758 +[#133824]: https://github.com/cockroachdb/cockroach/pull/133824 +[#133866]: https://github.com/cockroachdb/cockroach/pull/133866 +[#134063]: https://github.com/cockroachdb/cockroach/pull/134063 +[#134083]: https://github.com/cockroachdb/cockroach/pull/134083 +[#134363]: https://github.com/cockroachdb/cockroach/pull/134363 +[#134524]: https://github.com/cockroachdb/cockroach/pull/134524 +[#134603]: https://github.com/cockroachdb/cockroach/pull/134603 +[#134649]: https://github.com/cockroachdb/cockroach/pull/134649 +[154e9f0e0]: https://github.com/cockroachdb/cockroach/commit/154e9f0e0 diff --git a/src/current/_includes/releases/v23.2/v23.2.13.md b/src/current/_includes/releases/v23.2/v23.2.13.md index 28ce188820c..26f4a561790 100644 --- a/src/current/_includes/releases/v23.2/v23.2.13.md +++ b/src/current/_includes/releases/v23.2/v23.2.13.md @@ -16,7 +16,7 @@ Release Date: October 17, 2024 [#130664][#130664] -- The new [metric]({% link v23.2/metrics.md %}) `changefeed.total_ranges` allows observation of the number of changes that are watched by a changefeed aggregator. It uses the same polling interval as `changefeed.lagging_ranges`, which is controlled by the changefeed option `lagging_ranges_polling_interval`. [#130984][#130984] +- The new [metric]({% link v23.2/metrics.md %}) `changefeed.total_ranges` allows observation of the number of ranges that are watched by a changefeed aggregator. It uses the same polling interval as `changefeed.lagging_ranges`, which is controlled by the changefeed option `lagging_ranges_polling_interval`. [#130984][#130984] - The following groups of [metrics]({% link v23.2/metrics.md %}) and [logs]({% link v23.2/logging.md %}) have been renamed to include the buffer they are associated with. The previous metrics are still maintained for backward compatibility. - `changefeed.buffer_entries.*` - `changefeed.buffer_entries_mem.*` @@ -81,6 +81,7 @@ Release Date: October 17, 2024 [#130664]: https://github.com/cockroachdb/cockroach/pull/130664 [#130790]: https://github.com/cockroachdb/cockroach/pull/130790 [#130919]: https://github.com/cockroachdb/cockroach/pull/130919 +[#130984]: https://github.com/cockroachdb/cockroach/pull/130984 [#130988]: https://github.com/cockroachdb/cockroach/pull/130988 [#131065]: https://github.com/cockroachdb/cockroach/pull/131065 [#131128]: https://github.com/cockroachdb/cockroach/pull/131128 diff --git a/src/current/_includes/releases/v23.2/v23.2.16.md b/src/current/_includes/releases/v23.2/v23.2.16.md index 5793f589357..6e205c889b3 100644 --- a/src/current/_includes/releases/v23.2/v23.2.16.md +++ b/src/current/_includes/releases/v23.2/v23.2.16.md @@ -1,6 +1,6 @@ ## v23.2.16 -Release Date: November 15, 2024 +Release Date: November 18, 2024 {% include releases/new-release-downloads-docker-image.md release=include.release %} diff --git a/src/current/_includes/releases/v23.2/v23.2.17.md b/src/current/_includes/releases/v23.2/v23.2.17.md new file mode 100644 index 00000000000..29855bc2638 --- /dev/null +++ b/src/current/_includes/releases/v23.2/v23.2.17.md @@ -0,0 +1,63 @@ +## v23.2.17 + +Release Date: December 12, 2024 + +{% include releases/new-release-downloads-docker-image.md release=include.release %} + +Security updates
+ +- All cluster settings that accept strings are now fully redacted when transmitted as part of Cockroach Labs' diagnostics telemetry. The payload includes a record of modified cluster settings and their values when they are not strings. If you previously applied the mitigations in Technical Advisory 133479, you can safely set the value of cluster setting `server.redact_sensitive_settings.enabled` to `false` and turn on diagnostic reporting via the `diagnostics.reporting.enabled` cluster setting without leaking sensitive cluster setting values. [#134015][#134015] + +General changes
+ +- `COCKROACH_SKIP_ENABLING_DIAGNOSTIC_REPORTING` is no longer mentioned in the `cockroach demo` command. [#134085][#134085] + +{{ site.data.products.enterprise }} edition changes
+ +- Added `system.users` to the list of system tables that changefeeds protect with protected timestamps (PTS). This table is required for CDC queries. [#134233][#134233] + +Operational changes
+ +- Added a new cluster setting `ui.database_locality_metadata.enabled`, which allows operators to disable loading extended database and table region information in the DB Console's Databases and Table Details pages. This information can cause significant CPU load on large clusters with many ranges. Versions of this page from v24.3 onwards do not have this problem. If you require this data, you can use the `SHOW RANGES FROM {DATABASE| TABLE}` query via SQL to compute on-demand. [#134093][#134093] + +Bug fixes
+ +- Previously, CockroachDB could encounter an internal error of the form `interface conversion: coldata.Column is` in an edge case. This is now fixed. The bug was present in versions v22.2.13 and later, v23.1.9 and later, and v23.2 and later. [#133759][#133759] +- Fixed a bug that caused incorrect `NOT NULL` constraint violation errors on `UPSERT` and `INSERT ... ON CONFLICT ... DO UPDATE` statements when those statements updated an existing row and a subset of columns that did not include a `NOT NULL` column of the table. This bug had been present since at least v20.1.0. [#133823][#133823] +- Fixed an unhandled error that could occur when using `REVOKE ... ON SEQUENCE ... FROM user` on an object that was not a sequence. [#133707][#133707] +- Addressed a panic that could occur inside `CREATE TABLE AS` that occurred if sequence builtin expressions had invalid function overloads. [#133867][#133867] +- String constants can now be compared against collated strings. [#134114][#134114] +- Previously, when executing queries with index or lookup joins when the ordering needed to be maintained, CockroachDB in some cases could get into a pathological state which would lead to increased query latency, possibly by several orders of magnitude. This bug was introduced in v22.2 and is now fixed. [#134364][#134364] +- Addressed a bug with `DROP CASCADE` that would occasionally panic with an `un-dropped backref` message on partitioned tables. [#134523][#134523] +- Reduced the duration of partitions in the gossip network when a node crashes. This eliminates false positives in the `ranges.unavailable` metric. [#134602][#134602] +- An error message is no longer returned when a non-admin user runs `DROP ROLE IF EXISTS` on a user that does not exist. [#134967][#134967] +- Fixed a bug that could cause incorrect query results when the optimizer planned a lookup join on an index containing a column of type `CHAR(N)`, `VARCHAR(N)`, `BIT(N)`, `VARBIT(N)`, or `DECIMAL(M, N)`, and the query held that column constant to a single value (e.g., with an equality filter). [#135113][#135113] +- Fixed an unhandled error that would occur if `DROP SCHEMA` was executed on the `public` schema of the `system` database, or on an internal schema like `pg_catalog` or `information_schema`. [#135196][#135196] +- Fixed a bug that caused incorrect evaluation of some binary expressions involving `CHAR(N)` values and untyped string literals with trailing whitespace characters. For example, the expression `'f'::CHAR = 'f '` now correctly evaluates to `true`. [#135691][#135691] + +Performance improvements
+ +- CockroachDB now avoids loading unnecessary file blocks shortly after a rebalance in a rare case. [#134526][#134526] [#135303][#135303] [#135577][#135577] +- Reduced the write-amplification impact of rebalances by splitting snapshot sstable files into smaller ones before ingesting them into Pebble. [#134526][#134526] [#135303][#135303] [#135577][#135577] + +[#133707]: https://github.com/cockroachdb/cockroach/pull/133707 +[#133759]: https://github.com/cockroachdb/cockroach/pull/133759 +[#133823]: https://github.com/cockroachdb/cockroach/pull/133823 +[#133867]: https://github.com/cockroachdb/cockroach/pull/133867 +[#134015]: https://github.com/cockroachdb/cockroach/pull/134015 +[#134085]: https://github.com/cockroachdb/cockroach/pull/134085 +[#134093]: https://github.com/cockroachdb/cockroach/pull/134093 +[#134114]: https://github.com/cockroachdb/cockroach/pull/134114 +[#134233]: https://github.com/cockroachdb/cockroach/pull/134233 +[#134364]: https://github.com/cockroachdb/cockroach/pull/134364 +[#134523]: https://github.com/cockroachdb/cockroach/pull/134523 +[#134526]: https://github.com/cockroachdb/cockroach/pull/134526 +[#134602]: https://github.com/cockroachdb/cockroach/pull/134602 +[#134647]: https://github.com/cockroachdb/cockroach/pull/134647 +[#134967]: https://github.com/cockroachdb/cockroach/pull/134967 +[#135113]: https://github.com/cockroachdb/cockroach/pull/135113 +[#135196]: https://github.com/cockroachdb/cockroach/pull/135196 +[#135303]: https://github.com/cockroachdb/cockroach/pull/135303 +[#135577]: https://github.com/cockroachdb/cockroach/pull/135577 +[#135691]: https://github.com/cockroachdb/cockroach/pull/135691 +[#136006]: https://github.com/cockroachdb/cockroach/pull/136006 diff --git a/src/current/_includes/releases/v24.1/v24.1.7.md b/src/current/_includes/releases/v24.1/v24.1.7.md index 9517b2ec9bb..f0d339f9c81 100644 --- a/src/current/_includes/releases/v24.1/v24.1.7.md +++ b/src/current/_includes/releases/v24.1/v24.1.7.md @@ -1,6 +1,6 @@ ## v24.1.7 -Release Date: November 14, 2024 +Release Date: November 18, 2024 {% include releases/new-release-downloads-docker-image.md release=include.release %} diff --git a/src/current/_includes/releases/v24.1/v24.1.8.md b/src/current/_includes/releases/v24.1/v24.1.8.md new file mode 100644 index 00000000000..a6b427cf04b --- /dev/null +++ b/src/current/_includes/releases/v24.1/v24.1.8.md @@ -0,0 +1,73 @@ +## v24.1.8 + +Release Date: December 12, 2024 + +{% include releases/new-release-downloads-docker-image.md release=include.release %} + +Security updates
+ +- All cluster settings that accept strings are now fully redacted when transmitted as part of our diagnostics telemetry. The transmitted payload includes a record of modified cluster settings and their values when they are not strings. If you previously applied the mitigations in [Technical Advisory 133479]({% link advisories/a133479.md %}), you can safely set the value of cluster setting `server.redact_sensitive_settings.enabled` to false and turn on diagnostic reporting via the `diagnostics.reporting.enabled` cluster setting without leaking sensitive cluster settings values. [#134016][#134016] + +General changes
+ +- `COCKROACH_SKIP_ENABLING_DIAGNOSTIC_REPORTING` is no longer mentioned in the `cockroach demo` command. [#134087][#134087] +- Added `system.users` to the list of system tables that changefeeds protect with protected timestamps (PTS). This table is required for change data capture queries. [#134836][#134836] +- Added changefeed support for the `mvcc_timestamp` option with the `avro` format. If both options are specified, the `avro` schema includes an `mvcc_timestamp` metadata field and emits the row's MVCC timestamp with the row data. [#136482][#136482] + +Operational changes
+ +- To prevent unnecessary queuing in admission control CPU queues, the `goschedstats.always_use_short_sample_period.enabled` setting should be set to `true` for any production cluster. [#133583][#133583] +- A new cluster setting `ui.database_locality_metadata.enabled`, when set to `true`, disables loading database and table region information in the DB Console Database and Table pages. This information can cause significant CPU load on large clusters with many ranges. The Database and Table pages in v24.3 onwards do not have this problem. If you require this data, use the `SHOW RANGES FROM {DATABASE|TABLE}` SQL statement to compute this information on-demand. [#134094][#134094] +- The row-level TTL job now will periodically log progress by showing the number of table spans that have been processed so far. [#135179][#135179] + +Bug fixes
+ +- Fixed a bug where CockroachDB could encounter an internal error `interface conversion: coldata.Column is` in an edge case. The bug was present in v22.2.13+, v23.1.9+, and v23.2+. [#133760][#133760] +- Fixed an unhandled error that could occur when using `REVOKE ... ON SEQUENCE ... FROM user` on an object that is not a sequence. [#133708][#133708] +- Fixed a bug that caused incorrect `NOT NULL` constraint violation errors on `UPSERT` and `INSERT ... ON CONFLICT ... DO UPDATE` statements when those statements updated an existing row and a subset of columns that did not include a `NOT NULL` column of the table. This bug had been present since at least v20.1.0. [#133822][#133822] +- Addressed a panic that could occur inside `CREATE TABLE AS` if sequence builtin expressions had invalid function overloads. [#133868][#133868] +- String constants can now be compared against collated strings. [#134105][#134105] +- Previously, when executing queries with index / lookup joins where ordering needed to be maintained, CockroachDB's behavior could lead to increased query latency, potentially by several orders of magnitude. This bug was introduced in v22.2, and is now fixed. [#134365][#134365] +- Fixed a bug where `DISCARD ALL` statements were counted under the `sql.ddl.count` metric. Now these statements will be counted under the `sql.misc.count` metric. [#134508][#134508] +- Fixed a bug where `DROP CASCADE` would occasionally panic with an `un-dropped backref` message on partitioned tables. [#134522][#134522] +- Reduced the duration of partitions in the gossip network when a node crashes. This eliminates false positives in the `ranges.unavailable` metric. [#134601][#134601] +- As a non-admin user who runs `DROP ROLE IF EXISTS` on a user that does not exist, you no longer get an error message. [#134968][#134968] +- Fixed a bug that caused quotes around the name of a routine to be dropped when the routine was called within another routine. This could prevent the correct routine name from being resolved if the nested routine name was case-sensitive. The bug has existed since v24.1, when nested routines were introduced. [#134001][#134001] +- Fixed a bug that could cause incorrect query results when the optimizer planned a lookup join on an index containing a column of type `CHAR(N)`, `VARCHAR(N)`, `BIT(N)`, `VARBIT(N)`, or `DECIMAL(M, N)`, and the query held that column constant to a single value (for example, with an equality filter). [#135111][#135111] +- Fixed an unhandled error that would occur if `DROP SCHEMA` was executed on the `public` schema of the `system` database, or on an internal schema, such as `pg_catalog` or `information_schema`. [#135195][#135195] +- Fixed a bug that caused incorrect evaluation of some binary expressions involving `CHAR(N)` values and untyped string literals with trailing whitespace characters. For example, the expression `'f'::CHAR = 'f '` now correctly evaluates to `true`. [#135690][#135690] +- `CREATE SCHEMA` now returns the correct error if a the schema name is missing. [#135926][#135926] + +Performance improvements
+ +- Unnecessary block loads of SSTable files are now avoided in some rare cases after a replica rebalance. [#134541][#134541] + +[#133583]: https://github.com/cockroachdb/cockroach/pull/133583 +[#133708]: https://github.com/cockroachdb/cockroach/pull/133708 +[#133760]: https://github.com/cockroachdb/cockroach/pull/133760 +[#133822]: https://github.com/cockroachdb/cockroach/pull/133822 +[#133868]: https://github.com/cockroachdb/cockroach/pull/133868 +[#134001]: https://github.com/cockroachdb/cockroach/pull/134001 +[#134016]: https://github.com/cockroachdb/cockroach/pull/134016 +[#134087]: https://github.com/cockroachdb/cockroach/pull/134087 +[#134094]: https://github.com/cockroachdb/cockroach/pull/134094 +[#134100]: https://github.com/cockroachdb/cockroach/pull/134100 +[#134105]: https://github.com/cockroachdb/cockroach/pull/134105 +[#134365]: https://github.com/cockroachdb/cockroach/pull/134365 +[#134446]: https://github.com/cockroachdb/cockroach/pull/134446 +[#134508]: https://github.com/cockroachdb/cockroach/pull/134508 +[#134522]: https://github.com/cockroachdb/cockroach/pull/134522 +[#134541]: https://github.com/cockroachdb/cockroach/pull/134541 +[#134601]: https://github.com/cockroachdb/cockroach/pull/134601 +[#134648]: https://github.com/cockroachdb/cockroach/pull/134648 +[#134731]: https://github.com/cockroachdb/cockroach/pull/134731 +[#134836]: https://github.com/cockroachdb/cockroach/pull/134836 +[#134968]: https://github.com/cockroachdb/cockroach/pull/134968 +[#135111]: https://github.com/cockroachdb/cockroach/pull/135111 +[#135179]: https://github.com/cockroachdb/cockroach/pull/135179 +[#135195]: https://github.com/cockroachdb/cockroach/pull/135195 +[#135614]: https://github.com/cockroachdb/cockroach/pull/135614 +[#135690]: https://github.com/cockroachdb/cockroach/pull/135690 +[#135926]: https://github.com/cockroachdb/cockroach/pull/135926 +[#136008]: https://github.com/cockroachdb/cockroach/pull/136008 +[#136482]: https://github.com/cockroachdb/cockroach/pull/136482 diff --git a/src/current/_includes/releases/v24.2/v24.2.5.md b/src/current/_includes/releases/v24.2/v24.2.5.md index 643d7a89006..844401e23df 100644 --- a/src/current/_includes/releases/v24.2/v24.2.5.md +++ b/src/current/_includes/releases/v24.2/v24.2.5.md @@ -1,6 +1,6 @@ ## v24.2.5 -Release Date: November 14, 2024 +Release Date: November 18, 2024 {% include releases/new-release-downloads-docker-image.md release=include.release %} diff --git a/src/current/_includes/releases/v24.2/v24.2.6.md b/src/current/_includes/releases/v24.2/v24.2.6.md new file mode 100644 index 00000000000..3cedad5d3a0 --- /dev/null +++ b/src/current/_includes/releases/v24.2/v24.2.6.md @@ -0,0 +1,78 @@ +## v24.2.6 + +Release Date: December 12, 2024 + +{% include releases/new-release-downloads-docker-image.md release=include.release %} + +Security updates
+ +- All cluster settings that accept strings are now fully redacted when transmitted as part of diagnostics telemetry. This payload includes a record of modified cluster settings and their values when they are not strings. Customers who previously applied the mitigations in [Technical Advisory 133479]({% link advisories/a133479.md %}) can safely set the value of cluster setting `server.redact_sensitive_settings.enabled` to false and turn on diagnostic reporting via the `diagnostics.reporting.enabled` cluster setting without leaking sensitive cluster settings values. [#134017][#134017] + +General changes
+ +- `COCKROACH_SKIP_ENABLING_DIAGNOSTIC_REPORTING` is no longer mentioned in the `cockroach demo` command. [#134088][#134088] +- Added `system.users` to the list of system tables that changefeeds protect with protected timestamps. This table is required for change data capture queries. [#134837][#134837] + +Operational changes
+ +- The `goschedstats.always_use_short_sample_period.enabled` setting should be set to true for any production cluster, to prevent unnecessary queuing in admission control CPU queues. [#133584][#133584] +- Added a new cluster setting `ui.database_locality_metadata.enabled` that allows operators to disable loading extended database and table region information in the DB Console Database and Table pages. This information can cause significant CPU load on large clusters with many ranges. Versions of this page from v24.3 and later do not have this problem. If customers require this data, they can use the `SHOW RANGES FROM {DATABASE| TABLE}` query via SQL to compute on-demand. [#134095][#134095] +- Row-level TTL jobs now periodically log progress by showing the number of table spans that have been processed so far. [#135170][#135170] + +Bug fixes
+ +- Fixed a bug that caused non-reusable query plans, e.g., plans for DDL and `SHOW ...` statements, to be cached and reused in future executions, possibly causing stale results to be returned. This bug only occurred when `plan_cache_mode` was set to `auto` or `force_generic_plan`, both of which are not currently the default settings. [#133074][#133074] +- Previously, CockroachDB could encounter an internal error of the form `interface conversion: coldata.Column is` in an edge case and this is now fixed. The bug is present in v22.2.13+, v23.1.9+, v23.2+. [#133761][#133761] +- Fixed a bug that caused incorrect `NOT NULL` constraint violation errors on `UPSERT` and `INSERT .. ON CONFLICT .. DO UPDATE` statements when those statements updated an existing row and a subset of columns that did not include a `NOT NULL` column of the table. This bug has been present since at least v20.1.0. [#133821][#133821] +- Fixed an unhandled error that could occur when using `REVOKE ... ON SEQUENCE FROM ... user` on an object that is not a sequence. [#133709][#133709] +- Addressed a panic inside `CREATE TABLE AS` if sequence builtin expressions had invalid function overloads. [#133869][#133869] +- `STRING`constants can now be compared against collated strings. [#134084][#134084] +- When executing queries with index / lookup joins when the ordering needs to be maintained, previously CockroachDB could experience increased query latency, possibly by several orders of magnitude. This bug was introduced in v22.2 and is now fixed. [#134366][#134366] +- Fixed a minor bug where `DISCARD ALL` statements were counted under the `sql.ddl.count` metric. Now these will be counted under the `sql.misc.count` metric. [#134509][#134509] +- Addressed a bug with `DROP CASCADE` that would occasionally panic with an undropped `backref` message on partitioned tables. [#134472][#134472] +- Reduced the duration of partitions in the gossip network when a node crashes in order to eliminate false positives in the `ranges.unavailable` metric. [#134600][#134600] +- Non-`admin` users that run `DROP ROLE IF EXISTS` on a user that does not exist will no longer receive an error message. [#134969][#134969] +- Fixed a bug that caused quotes around the name of a routine to be dropped when it was called within another routine. This could prevent the correct routine from being resolved if the nested routine name was case sensitive. The bug has existed since v24.1, when nested routines were introduced. [#134000][#134000] +- Fixed a bug that could cause incorrect query results when the optimizer planned a lookup join on an index containing a column of type `CHAR(N)`, `VARCHAR(N)`, `BIT(N)`, `VARBIT(N)`, or `DECIMAL(M, N)`, and the query held that column constant to a single value (e.g., with an equality filter). [#135076][#135076] +- Fixed an unhandled error that would occur if `DROP SCHEMA` was executed on the `public` schema of the `system` database, or on an internal schema like `pg_catalog` or `information_schema`. [#135180][#135180] +- Fixed a bug that caused incorrect evaluation of some binary expressions involving `CHAR(N)` values and untyped string literals with trailing whitespace characters. For example, the expression `'f'::CHAR = 'f '` now correctly evaluates to `true`. [#135689][#135689] +- Fixed a bug where `ALTER DATABASE` operations that modify the zone config would hang if an invalid zone config already exists. [#135215][#135215] +- `CREATE SCHEMA` now returns the correct error if a the schema name is missing. [#135927][#135927] +- Using more than one `DECLARE` statment in the definition of a user-defined function now correctly declares additional variables. [#135738][#135738] +- Fixed a bug where CockroachDB would encounter an internal error when evaluating `FETCH ABSOLUTE 0` statements. The bug has been present since v22.1. [#134992][#134992] +- Fixed an issue where corrupted table statistics could cause the `cockroach` process to crash. [#136041][#136041] +- Fixed a bug that causes the optimizer to use stale table statistics after altering an `ENUM` type used in the table. [#136812][#136812] + +[#133074]: https://github.com/cockroachdb/cockroach/pull/133074 +[#133584]: https://github.com/cockroachdb/cockroach/pull/133584 +[#133709]: https://github.com/cockroachdb/cockroach/pull/133709 +[#133761]: https://github.com/cockroachdb/cockroach/pull/133761 +[#133821]: https://github.com/cockroachdb/cockroach/pull/133821 +[#133869]: https://github.com/cockroachdb/cockroach/pull/133869 +[#134000]: https://github.com/cockroachdb/cockroach/pull/134000 +[#134017]: https://github.com/cockroachdb/cockroach/pull/134017 +[#134084]: https://github.com/cockroachdb/cockroach/pull/134084 +[#134088]: https://github.com/cockroachdb/cockroach/pull/134088 +[#134095]: https://github.com/cockroachdb/cockroach/pull/134095 +[#134099]: https://github.com/cockroachdb/cockroach/pull/134099 +[#134366]: https://github.com/cockroachdb/cockroach/pull/134366 +[#134447]: https://github.com/cockroachdb/cockroach/pull/134447 +[#134472]: https://github.com/cockroachdb/cockroach/pull/134472 +[#134509]: https://github.com/cockroachdb/cockroach/pull/134509 +[#134600]: https://github.com/cockroachdb/cockroach/pull/134600 +[#134646]: https://github.com/cockroachdb/cockroach/pull/134646 +[#134730]: https://github.com/cockroachdb/cockroach/pull/134730 +[#134837]: https://github.com/cockroachdb/cockroach/pull/134837 +[#134969]: https://github.com/cockroachdb/cockroach/pull/134969 +[#134992]: https://github.com/cockroachdb/cockroach/pull/134992 +[#135076]: https://github.com/cockroachdb/cockroach/pull/135076 +[#135170]: https://github.com/cockroachdb/cockroach/pull/135170 +[#135180]: https://github.com/cockroachdb/cockroach/pull/135180 +[#135215]: https://github.com/cockroachdb/cockroach/pull/135215 +[#135611]: https://github.com/cockroachdb/cockroach/pull/135611 +[#135689]: https://github.com/cockroachdb/cockroach/pull/135689 +[#135738]: https://github.com/cockroachdb/cockroach/pull/135738 +[#135927]: https://github.com/cockroachdb/cockroach/pull/135927 +[#136010]: https://github.com/cockroachdb/cockroach/pull/136010 +[#136041]: https://github.com/cockroachdb/cockroach/pull/136041 +[#136812]: https://github.com/cockroachdb/cockroach/pull/136812 diff --git a/src/current/_includes/releases/v24.3/v24.3.1.md b/src/current/_includes/releases/v24.3/v24.3.1.md new file mode 100644 index 00000000000..4db85eb32a8 --- /dev/null +++ b/src/current/_includes/releases/v24.3/v24.3.1.md @@ -0,0 +1,78 @@ +## v24.3.1 + +Release Date: December 12, 2024 + +{% include releases/new-release-downloads-docker-image.md release=include.release %} + +SQL language changes
+ +- When triggers fire one another cyclically, the new `recursion_depth_limit` setting now limits the depth of the recursion. By default, the limit is `1000` nested trigger executions. [#135046][#135046] + +Operational changes
+ +- The metrics scrape HTTP endpoint at `/ _status/vars` will now truncate HELP text at the first sentence, reducing the metadata for metrics with large descriptions. Customers can still access these descriptions via our docs. [#135021][#135021] +- The row-level TTL job now periodically updates the progress meter in the jobs introspection interfaces, including `SHOW JOBS` and the Jobs page in the DB console. [#135171][#135171] +- Telemetry delivery is now considered successful even in cases where we experience a network timeout. This will prevent throttling in cases outside an operator's control. [#136481][#136481] + +DB Console changes
+ +- When activating statement diagnostics in the DB Console, users now have the option to produce a redacted bundle as output. This bundle will omit sensitive data. [#134993][#134993] + +Other changes
+ +- Protected timestamp records for changefeeds now include the `system.users` table. This ensures that user information remains available when running CDC queries against historical data. [#134238][#134238] + +Bug fixes
+ +- Fixed a bug that could cause `DELETE` triggers not to fire on cascading delete, and which could cause `INSERT` triggers to match incorrectly in the same scenario. [#134896][#134896] +- Reduced the duration of partitions in the gossip network when a node crashes in order to eliminate false positives in the `ranges.unavailable` metric. [#134480][#134480] +- When a non-admin user runs `DROP ROLE IF EXISTS` on a user that does not exist, an error is no longer returned. [#134970][#134970] +- Fixed a bug that could cause incorrect query results when the optimizer planned a lookup join on an index containing a column of type `CHAR(N)`, `VARCHAR(N)`, `BIT(N)`, `VARBIT(N)`, or `DECIMAL(M, N)`, and the query held that column constant to a single value (e.g. with an equality filter). [#135037][#135037] +- Fixed an unhandled error that would occur if `DROP SCHEMA` was executed on the `public` schema of the `system` database, or on an internal schema like `pg_catalog` or `information_schema`. [#135181][#135181] +- A bug has been fixed that caused incorrect evaluation of some binary expressions involving `CHAR(N)` values and untyped string literals with trailing whitespace characters. For example, the expression `'f'::CHAR = 'f '` now correctly evaluates to `true`. [#134710][#134710] +- Prevent `ALTER DATABASE` operations that modify the zone config from hanging if an invalid zone config already exists. [#135216][#135216] +- `CREATE SCHEMA` now returns the correct error if a schema name is missing. [#135928][#135928] +- `percentile_cont` and `percentile_disc` aggregate functions now support `float4` data type inputs. Previously, these functions would return an error when used with `float4` values. [#135764][#135764] +- `security.certificate.*` metrics now update correctly when certificates are reloaded during node runtime. Previously, these metrics would not reflect changes to certificates after node startup. [#136227][#136227] +- SQL roles created from LDAP groups that contain periods (.) or hyphens (-) in their Common Names (CN) no longer result in authorization failures. [#134942][#134942] +- LDAP authorization now supports partial group mapping, allowing users to authenticate even when some LDAP groups do not have corresponding CockroachDB roles. Previously, authentication would fail if any LDAP group lacked a matching database role. [#135587][#135587] +- Regional by row tables with uniqueness constraints where the region is not part of those uniqueness constraints and which also contain non-unique indices will now have those constraints properly enforced when modified at `READ COMMITTED` isolation. This bug was introduced in v24.3.0. [#137367][#137367] + +Performance improvements
+ +- The `_status/nodes_ui` API no longer returns unnecessary metrics in its response. This decreases the payload size of the API and improves the load time of various DB Console pages and components. [#135209][#135209] +- PL/pgSQL loops now execute up to 3-4x faster through improved optimization, particularly when they contain subqueries. This enhancement improves performance for routines with many iterations or nested operations. [#135648][#135648] + +[#133230]: https://github.com/cockroachdb/cockroach/pull/133230 +[#134238]: https://github.com/cockroachdb/cockroach/pull/134238 +[#134480]: https://github.com/cockroachdb/cockroach/pull/134480 +[#134710]: https://github.com/cockroachdb/cockroach/pull/134710 +[#134729]: https://github.com/cockroachdb/cockroach/pull/134729 +[#134896]: https://github.com/cockroachdb/cockroach/pull/134896 +[#134942]: https://github.com/cockroachdb/cockroach/pull/134942 +[#134970]: https://github.com/cockroachdb/cockroach/pull/134970 +[#134993]: https://github.com/cockroachdb/cockroach/pull/134993 +[#135021]: https://github.com/cockroachdb/cockroach/pull/135021 +[#135037]: https://github.com/cockroachdb/cockroach/pull/135037 +[#135046]: https://github.com/cockroachdb/cockroach/pull/135046 +[#135094]: https://github.com/cockroachdb/cockroach/pull/135094 +[#135120]: https://github.com/cockroachdb/cockroach/pull/135120 +[#135171]: https://github.com/cockroachdb/cockroach/pull/135171 +[#135181]: https://github.com/cockroachdb/cockroach/pull/135181 +[#135209]: https://github.com/cockroachdb/cockroach/pull/135209 +[#135216]: https://github.com/cockroachdb/cockroach/pull/135216 +[#135587]: https://github.com/cockroachdb/cockroach/pull/135587 +[#135648]: https://github.com/cockroachdb/cockroach/pull/135648 +[#135764]: https://github.com/cockroachdb/cockroach/pull/135764 +[#135928]: https://github.com/cockroachdb/cockroach/pull/135928 +[#136011]: https://github.com/cockroachdb/cockroach/pull/136011 +[#136227]: https://github.com/cockroachdb/cockroach/pull/136227 +[#136481]: https://github.com/cockroachdb/cockroach/pull/136481 +[#137367]: https://github.com/cockroachdb/cockroach/pull/137367 +[0d7f6eed3]: https://github.com/cockroachdb/cockroach/commit/0d7f6eed3 +[1f2b1b084]: https://github.com/cockroachdb/cockroach/commit/1f2b1b084 +[3cbd07fbd]: https://github.com/cockroachdb/cockroach/commit/3cbd07fbd +[3f5305a4c]: https://github.com/cockroachdb/cockroach/commit/3f5305a4c +[965dded2a]: https://github.com/cockroachdb/cockroach/commit/965dded2a +[989a49c3f]: https://github.com/cockroachdb/cockroach/commit/989a49c3f +[9951e3e61]: https://github.com/cockroachdb/cockroach/commit/9951e3e61 diff --git a/src/current/_includes/v23.1/cdc/lagging-ranges.md b/src/current/_includes/v23.1/cdc/lagging-ranges.md index 339a5571443..e0061df7d4b 100644 --- a/src/current/_includes/v23.1/cdc/lagging-ranges.md +++ b/src/current/_includes/v23.1/cdc/lagging-ranges.md @@ -1,10 +1,12 @@ -{% include_cached new-in.html version="v23.1.12" %} Use the `changefeed.lagging_ranges` metric to track the number of ranges that are behind in a changefeed. This is calculated based on the [cluster settings]({% link {{ page.version.version }}/cluster-settings.md %}): +{% include_cached new-in.html version="v23.1.12" %} Use the `changefeed.lagging_ranges` metric to track the number of [ranges]({% link {{ page.version.version }}/architecture/overview.md %}#architecture-range) that are behind in a changefeed. This is calculated based on the [cluster settings]({% link {{ page.version.version }}/cluster-settings.md %}): - `changefeed.lagging_ranges_threshold` sets a duration from the present that determines the length of time a range is considered to be lagging behind, which will then track in the [`lagging_ranges`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#using-changefeed-metrics-labels) metric. Note that ranges undergoing an [initial scan]({% link {{ page.version.version }}/create-changefeed.md %}#initial-scan) for longer than the threshold duration are considered to be lagging. Starting a changefeed with an initial scan on a large table will likely increment the metric for each range in the table. As ranges complete the initial scan, the number of ranges lagging behind will decrease. - **Default:** `3m` - `changefeed.lagging_ranges_polling_interval` sets the interval rate for when lagging ranges are checked and the `lagging_ranges` metric is updated. Polling adds latency to the `lagging_ranges` metric being updated. For example, if a range falls behind by 3 minutes, the metric may not update until an additional minute afterward. - **Default:** `1m` +{% include_cached new-in.html version="v23.1.29" %} Use the `changefeed.total_ranges` metric to monitor the number of ranges that are watched by [aggregator processors]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}) participating in the changefeed job. If you're experiencing lagging ranges, `changefeed.total_ranges` may indicate that the number of ranges watched by aggregator processors in the job is unbalanced. You may want to try [pausing]({% link {{ page.version.version }}/pause-job.md %}) the changefeed and then [resuming]({% link {{ page.version.version }}/resume-job.md %}) it, so that the changefeed replans the work in the cluster. `changefeed.total_ranges` shares the same polling interval as the `changefeed.lagging_ranges` metric, which is controlled by the `changefeed.lagging_ranges_polling_interval` cluster setting. + {{site.data.alerts.callout_success}} -You can use the [`metrics_label`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#using-changefeed-metrics-labels) option to track the `lagging_ranges` metric per changefeed. +You can use the [`metrics_label`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#using-changefeed-metrics-labels) option to track the `lagging_ranges` and `total_ranges` metric per changefeed. {{site.data.alerts.end}} \ No newline at end of file diff --git a/src/current/_includes/v23.1/storage/free-up-disk-space.md b/src/current/_includes/v23.1/storage/free-up-disk-space.md index c63b70b766e..e4a6b08a57a 100644 --- a/src/current/_includes/v23.1/storage/free-up-disk-space.md +++ b/src/current/_includes/v23.1/storage/free-up-disk-space.md @@ -1 +1 @@ -For instructions on how to free up disk space as quickly as possible after deleting data, see [How can I free up disk space quickly?]({% link {{ page.version.version }}/operational-faqs.md %}#how-can-i-free-up-disk-space-quickly) +For instructions on how to free up disk space as quickly as possible after dropping a table, see [How can I free up disk space that was used by a dropped table?]({% link {{ page.version.version }}/operational-faqs.md %}#how-can-i-free-up-disk-space-when-dropping-a-table) diff --git a/src/current/_includes/v23.2/cdc/lagging-ranges.md b/src/current/_includes/v23.2/cdc/lagging-ranges.md index b784a93cbfb..35d269bc706 100644 --- a/src/current/_includes/v23.2/cdc/lagging-ranges.md +++ b/src/current/_includes/v23.2/cdc/lagging-ranges.md @@ -1,10 +1,12 @@ -{% include_cached new-in.html version="v23.2" %} Use the `changefeed.lagging_ranges` metric to track the number of ranges that are behind in a changefeed. This is calculated based on the [changefeed options]({% link {{ page.version.version }}/create-changefeed.md %}#options): +{% include_cached new-in.html version="v23.2" %} Use the `changefeed.lagging_ranges` metric to track the number of [ranges]({% link {{ page.version.version }}/architecture/overview.md %}#architecture-range) that are behind in a changefeed. This is calculated based on the [changefeed options]({% link {{ page.version.version }}/create-changefeed.md %}#options): - `lagging_ranges_threshold` sets a duration from the present that determines the length of time a range is considered to be lagging behind, which will then track in the [`lagging_ranges`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#lagging-ranges-metric) metric. Note that ranges undergoing an [initial scan]({% link {{ page.version.version }}/create-changefeed.md %}#initial-scan) for longer than the threshold duration are considered to be lagging. Starting a changefeed with an initial scan on a large table will likely increment the metric for each range in the table. As ranges complete the initial scan, the number of ranges lagging behind will decrease. - **Default:** `3m` - `lagging_ranges_polling_interval` sets the interval rate for when lagging ranges are checked and the `lagging_ranges` metric is updated. Polling adds latency to the `lagging_ranges` metric being updated. For example, if a range falls behind by 3 minutes, the metric may not update until an additional minute afterward. - **Default:** `1m` +{% include_cached new-in.html version="v23.2.13" %} Use the `changefeed.total_ranges` metric to monitor the number of ranges that are watched by [aggregator processors]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}) participating in the changefeed job. If you're experiencing lagging ranges, `changefeed.total_ranges` may indicate that the number of ranges watched by aggregator processors in the job is unbalanced. You may want to try [pausing]({% link {{ page.version.version }}/pause-job.md %}) the changefeed and then [resuming]({% link {{ page.version.version }}/resume-job.md %}) it, so that the changefeed replans the work in the cluster. `changefeed.total_ranges` shares the same polling interval as the `changefeed.lagging_ranges` metric, which is controlled by the `lagging_ranges_polling_interval` option. + {{site.data.alerts.callout_success}} -You can use the [`metrics_label`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#using-changefeed-metrics-labels) option to track the `lagging_ranges` metric per changefeed. +You can use the [`metrics_label`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#using-changefeed-metrics-labels) option to track the `lagging_ranges` and `total_ranges` metric per changefeed. {{site.data.alerts.end}} \ No newline at end of file diff --git a/src/current/_includes/v23.2/known-limitations/physical-cluster-replication.md b/src/current/_includes/v23.2/known-limitations/physical-cluster-replication.md index 0a9fc624cf2..7d0d79a79b5 100644 --- a/src/current/_includes/v23.2/known-limitations/physical-cluster-replication.md +++ b/src/current/_includes/v23.2/known-limitations/physical-cluster-replication.md @@ -1,6 +1,6 @@ - Physical cluster replication is supported only on CockroachDB {{ site.data.products.core }} in new v23.2 clusters. Physical Cluster Replication cannot be enabled on clusters that have been upgraded from a previous version of CockroachDB. - Read queries are not supported on the standby cluster before [cutover]({% link {{ page.version.version }}/cutover-replication.md %}). -- The primary and standby cluster **cannot have different [region topology]({% link {{ page.version.version }}/topology-patterns.md %})**. For example, replicating a multi-region primary cluster to a single-region standby cluster is not supported. Mismatching regions between a multi-region primary and standby cluster is also not supported. +- The primary and standby clusters must have the same [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}). - Cutting back to the primary cluster after a cutover is a manual process. Refer to [Cut back to the primary cluster]({% link {{ page.version.version }}/cutover-replication.md %}#cut-back-to-the-primary-cluster). In addition, after cutover, to continue using physical cluster replication, you must configure it again. - Before cutover to the standby, the standby cluster does not support running [backups]({% link {{ page.version.version }}/backup-and-restore-overview.md %}) or [changefeeds]({% link {{ page.version.version }}/change-data-capture-overview.md %}). - After a cutover, there is no mechanism to stop applications from connecting to the original primary cluster. It is necessary to redirect application traffic manually, such as by using a network load balancer or adjusting DNS records. diff --git a/src/current/_includes/v23.2/storage/free-up-disk-space.md b/src/current/_includes/v23.2/storage/free-up-disk-space.md index c63b70b766e..e4a6b08a57a 100644 --- a/src/current/_includes/v23.2/storage/free-up-disk-space.md +++ b/src/current/_includes/v23.2/storage/free-up-disk-space.md @@ -1 +1 @@ -For instructions on how to free up disk space as quickly as possible after deleting data, see [How can I free up disk space quickly?]({% link {{ page.version.version }}/operational-faqs.md %}#how-can-i-free-up-disk-space-quickly) +For instructions on how to free up disk space as quickly as possible after dropping a table, see [How can I free up disk space that was used by a dropped table?]({% link {{ page.version.version }}/operational-faqs.md %}#how-can-i-free-up-disk-space-when-dropping-a-table) diff --git a/src/current/_includes/v23.2/ui/databases.md b/src/current/_includes/v23.2/ui/databases.md index 94ae7fc0585..1b8ec510206 100644 --- a/src/current/_includes/v23.2/ui/databases.md +++ b/src/current/_includes/v23.2/ui/databases.md @@ -15,7 +15,7 @@ The following information is displayed for each database: {% endif -%} | Tables | The number of tables in the database. | {% if page.cloud != true -%} -| Regions/Nodes | The regions and nodes on which the tables in the database are located. This is not displayed on a single-node cluster. | +| Regions/Nodes | The regions and nodes on which the tables in the database are located. This is not displayed on a single-node cluster.On a multi-node cluster, the display of this information is controlled by the cluster setting [`ui.database_locality_metadata.enabled`](#ui-database_locality_metadata-enabled-cluster-setting) (default `true`). | | Index Recommendations | The number of index recommendations for the database. | {%- else -%} | Regions | The regions where the tables in the database are located. | @@ -26,6 +26,11 @@ Click a **database name** to open the **Tables** page. - Select **View: Tables** in the pulldown menu to display the [Tables view](#tables-view). - Select **View: Grants** in the pulldown menu to display the [Grants view](#grants-view). +{% if page.cloud != true -%} +### `ui.database_locality_metadata.enabled` cluster setting +{% include_cached new-in.html version="v23.2.17" %} Retrieving extended database and table region information can cause significant CPU load on large multi-node clusters with many ranges. You can prevent the retrieval of this data and the associated CPU load by disabling the [`ui.database_locality_metadata.enabled` cluster setting]({{ link_prefix }}cluster-settings.html#setting-ui-database-locality-metadata-enabled). When set to `false`, “No data” will be displayed for region data and replica counts. If you require this data, use the SQL statement [`SHOW RANGES FROM {DATABASE|TABLE}`]({{ link_prefix }}show-ranges.html) to compute this information. +{% endif -%} + ## Search and filter By default, the Databases page shows all databases running on the cluster. By default, the [**Tables** view](#tables-view) and the [**Grants** view](#grants-view) show all tables in a selected database. @@ -63,7 +68,7 @@ The following information is displayed for each table: | Columns | The number of columns in the table. | | Indexes | The number of indexes in the table. | {% if page.cloud != true -%} -| Regions | The regions and nodes on which the table data is stored. This is not displayed on a single-node cluster. | +| Regions | The regions and nodes on which the table data is stored. This is not displayed on a single-node cluster.
On a multi-node cluster, the display of this information is controlled by the cluster setting [`ui.database_locality_metadata.enabled`](#ui-database_locality_metadata-enabled-cluster-setting) (default `true`). | {% else -%} | Regions | The regions where the table data is stored. {% endif -%} @@ -84,14 +89,14 @@ The table details include: {% if page.cloud != true %} - **Size**: The approximate disk size of all replicas of this table on the cluster. -- **Replicas**: The number of [replicas]({{ link_prefix }}architecture/replication-layer.html) of this table on the cluster. +- **Replicas**: The number of [replicas]({{ link_prefix }}architecture/replication-layer.html) of this table on the cluster. On a multi-node cluster, the display of this information is controlled by the cluster setting [`ui.database_locality_metadata.enabled`](#ui-database_locality_metadata-enabled-cluster-setting) (default `true`). - **Ranges**: The number of [ranges]({{ link_prefix }}architecture/glossary.html#architecture-range) in this table. - **% of Live Data**: Percentage of total uncompressed logical data that has not been modified (updated or deleted). - **Table Stats Last Updated**: The last time table statistics were created or updated. {% endif %} - **Auto Stats Collection**: Whether [automatic statistics collection]({{ link_prefix }}cost-based-optimizer.html#table-statistics) is enabled. {% if page.cloud != true %} -- **Regions/Nodes**: The regions and nodes on which the table data is stored. This is not displayed on a single-node cluster. +- **Regions/Nodes**: The regions and nodes on which the table data is stored. This is not displayed on a single-node cluster. On a multi-node cluster, the display of this information is controlled by the cluster setting [`ui.database_locality_metadata.enabled`](#ui-database_locality_metadata-enabled-cluster-setting) (default `true`). {% else %} - **Regions**: The regions where the table data is stored. {% endif %} diff --git a/src/current/_includes/v24.1/cdc/lagging-ranges.md b/src/current/_includes/v24.1/cdc/lagging-ranges.md index 8d0b5eb6c23..17a7e035916 100644 --- a/src/current/_includes/v24.1/cdc/lagging-ranges.md +++ b/src/current/_includes/v24.1/cdc/lagging-ranges.md @@ -1,10 +1,12 @@ -Use the `changefeed.lagging_ranges` metric to track the number of ranges that are behind in a changefeed. This is calculated based on the [changefeed options]({% link {{ page.version.version }}/create-changefeed.md %}#options): +Use the `changefeed.lagging_ranges` metric to track the number of [ranges]({% link {{ page.version.version }}/architecture/overview.md %}#range) that are behind in a changefeed. This is calculated based on the [changefeed options]({% link {{ page.version.version }}/create-changefeed.md %}#options): - `lagging_ranges_threshold` sets a duration from the present that determines the length of time a range is considered to be lagging behind, which will then track in the [`lagging_ranges`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#lagging-ranges-metric) metric. Note that ranges undergoing an [initial scan]({% link {{ page.version.version }}/create-changefeed.md %}#initial-scan) for longer than the threshold duration are considered to be lagging. Starting a changefeed with an initial scan on a large table will likely increment the metric for each range in the table. As ranges complete the initial scan, the number of ranges lagging behind will decrease. - **Default:** `3m` - `lagging_ranges_polling_interval` sets the interval rate for when lagging ranges are checked and the `lagging_ranges` metric is updated. Polling adds latency to the `lagging_ranges` metric being updated. For example, if a range falls behind by 3 minutes, the metric may not update until an additional minute afterward. - **Default:** `1m` +{% include_cached new-in.html version="v24.1.6" %} Use the `changefeed.total_ranges` metric to monitor the number of ranges that are watched by [aggregator processors]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}) participating in the changefeed job. If you're experiencing lagging ranges, `changefeed.total_ranges` may indicate that the number of ranges watched by aggregator processors in the job is unbalanced. You may want to try [pausing]({% link {{ page.version.version }}/pause-job.md %}) the changefeed and then [resuming]({% link {{ page.version.version }}/resume-job.md %}) it, so that the changefeed replans the work in the cluster. `changefeed.total_ranges` shares the same polling interval as the `changefeed.lagging_ranges` metric, which is controlled by the `lagging_ranges_polling_interval` option. + {{site.data.alerts.callout_success}} -You can use the [`metrics_label`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#using-changefeed-metrics-labels) option to track the `lagging_ranges` metric per changefeed. +You can use the [`metrics_label`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#using-changefeed-metrics-labels) option to track the `lagging_ranges` and `total_ranges` metric per changefeed. {{site.data.alerts.end}} \ No newline at end of file diff --git a/src/current/_includes/v24.1/known-limitations/physical-cluster-replication.md b/src/current/_includes/v24.1/known-limitations/physical-cluster-replication.md index f914c7eced2..a6c7edf2f32 100644 --- a/src/current/_includes/v24.1/known-limitations/physical-cluster-replication.md +++ b/src/current/_includes/v24.1/known-limitations/physical-cluster-replication.md @@ -1,6 +1,6 @@ - Physical cluster replication is supported only on CockroachDB {{ site.data.products.core }} in new clusters on v23.2 or above. Physical Cluster Replication cannot be enabled on clusters that have been upgraded from a previous version of CockroachDB. - Read queries are not supported on the standby cluster before [cutover]({% link {{ page.version.version }}/cutover-replication.md %}). -- The primary and standby cluster **cannot have different [region topology]({% link {{ page.version.version }}/topology-patterns.md %})**. For example, replicating a multi-region primary cluster to a single-region standby cluster is not supported. Mismatching regions between a multi-region primary and standby cluster is also not supported. +- The primary and standby clusters must have the same [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}). - Cutting back to the primary cluster after a cutover is a manual process. Refer to [Cut back to the primary cluster]({% link {{ page.version.version }}/cutover-replication.md %}#cut-back-to-the-primary-cluster). In addition, after cutover, to continue using physical cluster replication, you must configure it again. - Before cutover to the standby, the standby cluster does not support running [backups]({% link {{ page.version.version }}/backup-and-restore-overview.md %}) or [changefeeds]({% link {{ page.version.version }}/change-data-capture-overview.md %}). - Large data imports, such as those produced by [`RESTORE`]({% link {{ page.version.version }}/restore.md %}) or [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}), may dramatically increase [replication lag]({% link {{ page.version.version }}/physical-cluster-replication-technical-overview.md %}#cutover-and-promotion-process). diff --git a/src/current/_includes/v24.1/storage/free-up-disk-space.md b/src/current/_includes/v24.1/storage/free-up-disk-space.md index c63b70b766e..e4a6b08a57a 100644 --- a/src/current/_includes/v24.1/storage/free-up-disk-space.md +++ b/src/current/_includes/v24.1/storage/free-up-disk-space.md @@ -1 +1 @@ -For instructions on how to free up disk space as quickly as possible after deleting data, see [How can I free up disk space quickly?]({% link {{ page.version.version }}/operational-faqs.md %}#how-can-i-free-up-disk-space-quickly) +For instructions on how to free up disk space as quickly as possible after dropping a table, see [How can I free up disk space that was used by a dropped table?]({% link {{ page.version.version }}/operational-faqs.md %}#how-can-i-free-up-disk-space-when-dropping-a-table) diff --git a/src/current/_includes/v24.1/ui/databases.md b/src/current/_includes/v24.1/ui/databases.md index 94ae7fc0585..529b2b11efb 100644 --- a/src/current/_includes/v24.1/ui/databases.md +++ b/src/current/_includes/v24.1/ui/databases.md @@ -15,7 +15,7 @@ The following information is displayed for each database: {% endif -%} | Tables | The number of tables in the database. | {% if page.cloud != true -%} -| Regions/Nodes | The regions and nodes on which the tables in the database are located. This is not displayed on a single-node cluster. | +| Regions/Nodes | The regions and nodes on which the tables in the database are located. This is not displayed on a single-node cluster.
On a multi-node cluster, the display of this information is controlled by the cluster setting [`ui.database_locality_metadata.enabled`](#ui-database_locality_metadata-enabled-cluster-setting) (default `true`). | | Index Recommendations | The number of index recommendations for the database. | {%- else -%} | Regions | The regions where the tables in the database are located. | @@ -26,6 +26,11 @@ Click a **database name** to open the **Tables** page. - Select **View: Tables** in the pulldown menu to display the [Tables view](#tables-view). - Select **View: Grants** in the pulldown menu to display the [Grants view](#grants-view). +{% if page.cloud != true -%} +### `ui.database_locality_metadata.enabled` cluster setting +{% include_cached new-in.html version="v24.1.8" %} Retrieving extended database and table region information can cause significant CPU load on large multi-node clusters with many ranges. You can prevent the retrieval of this data and the associated CPU load by disabling the [`ui.database_locality_metadata.enabled` cluster setting]({{ link_prefix }}cluster-settings.html#setting-ui-database-locality-metadata-enabled). When set to `false`, “No data” will be displayed for region data and replica counts. If you require this data, use the SQL statement [`SHOW RANGES FROM {DATABASE|TABLE}`]({{ link_prefix }}show-ranges.html) to compute this information. +{% endif -%} + ## Search and filter By default, the Databases page shows all databases running on the cluster. By default, the [**Tables** view](#tables-view) and the [**Grants** view](#grants-view) show all tables in a selected database. @@ -63,7 +68,7 @@ The following information is displayed for each table: | Columns | The number of columns in the table. | | Indexes | The number of indexes in the table. | {% if page.cloud != true -%} -| Regions | The regions and nodes on which the table data is stored. This is not displayed on a single-node cluster. | +| Regions | The regions and nodes on which the table data is stored. This is not displayed on a single-node cluster.
On a multi-node cluster, the display of this information is controlled by the cluster setting [`ui.database_locality_metadata.enabled`](#ui-database_locality_metadata-enabled-cluster-setting) (default `true`). | {% else -%} | Regions | The regions where the table data is stored. {% endif -%} @@ -84,14 +89,14 @@ The table details include: {% if page.cloud != true %} - **Size**: The approximate disk size of all replicas of this table on the cluster. -- **Replicas**: The number of [replicas]({{ link_prefix }}architecture/replication-layer.html) of this table on the cluster. +- **Replicas**: The number of [replicas]({{ link_prefix }}architecture/replication-layer.html) of this table on the cluster. On a multi-node cluster, the display of this information is controlled by the cluster setting [`ui.database_locality_metadata.enabled`](#ui-database_locality_metadata-enabled-cluster-setting) (default `true`). - **Ranges**: The number of [ranges]({{ link_prefix }}architecture/glossary.html#architecture-range) in this table. - **% of Live Data**: Percentage of total uncompressed logical data that has not been modified (updated or deleted). - **Table Stats Last Updated**: The last time table statistics were created or updated. {% endif %} - **Auto Stats Collection**: Whether [automatic statistics collection]({{ link_prefix }}cost-based-optimizer.html#table-statistics) is enabled. {% if page.cloud != true %} -- **Regions/Nodes**: The regions and nodes on which the table data is stored. This is not displayed on a single-node cluster. +- **Regions/Nodes**: The regions and nodes on which the table data is stored. This is not displayed on a single-node cluster. On a multi-node cluster, the display of this information is controlled by the cluster setting [`ui.database_locality_metadata.enabled`](#ui-database_locality_metadata-enabled-cluster-setting) (default `true`). {% else %} - **Regions**: The regions where the table data is stored. {% endif %} diff --git a/src/current/_includes/v24.2/cdc/lagging-ranges.md b/src/current/_includes/v24.2/cdc/lagging-ranges.md index 8d0b5eb6c23..eb22275849a 100644 --- a/src/current/_includes/v24.2/cdc/lagging-ranges.md +++ b/src/current/_includes/v24.2/cdc/lagging-ranges.md @@ -1,10 +1,12 @@ -Use the `changefeed.lagging_ranges` metric to track the number of ranges that are behind in a changefeed. This is calculated based on the [changefeed options]({% link {{ page.version.version }}/create-changefeed.md %}#options): +Use the `changefeed.lagging_ranges` metric to track the number of [ranges]({% link {{ page.version.version }}/architecture/overview.md %}#architecture-range) that are behind in a changefeed. This is calculated based on the [changefeed options]({% link {{ page.version.version }}/create-changefeed.md %}#options): - `lagging_ranges_threshold` sets a duration from the present that determines the length of time a range is considered to be lagging behind, which will then track in the [`lagging_ranges`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#lagging-ranges-metric) metric. Note that ranges undergoing an [initial scan]({% link {{ page.version.version }}/create-changefeed.md %}#initial-scan) for longer than the threshold duration are considered to be lagging. Starting a changefeed with an initial scan on a large table will likely increment the metric for each range in the table. As ranges complete the initial scan, the number of ranges lagging behind will decrease. - **Default:** `3m` - `lagging_ranges_polling_interval` sets the interval rate for when lagging ranges are checked and the `lagging_ranges` metric is updated. Polling adds latency to the `lagging_ranges` metric being updated. For example, if a range falls behind by 3 minutes, the metric may not update until an additional minute afterward. - **Default:** `1m` +{% include_cached new-in.html version="v24.2.4" %} Use the `changefeed.total_ranges` metric to monitor the number of ranges that are watched by [aggregator processors]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}) participating in the changefeed job. If you're experiencing lagging ranges, `changefeed.total_ranges` may indicate that the number of ranges watched by aggregator processors in the job is unbalanced. You may want to try [pausing]({% link {{ page.version.version }}/pause-job.md %}) the changefeed and then [resuming]({% link {{ page.version.version }}/resume-job.md %}) it, so that the changefeed replans the work in the cluster. `changefeed.total_ranges` shares the same polling interval as the `changefeed.lagging_ranges` metric, which is controlled by the `lagging_ranges_polling_interval` option. + {{site.data.alerts.callout_success}} -You can use the [`metrics_label`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#using-changefeed-metrics-labels) option to track the `lagging_ranges` metric per changefeed. +You can use the [`metrics_label`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#using-changefeed-metrics-labels) option to track the `lagging_ranges` and `total_ranges` metric per changefeed. {{site.data.alerts.end}} \ No newline at end of file diff --git a/src/current/_includes/v24.2/known-limitations/pcr-scheduled-changefeeds.md b/src/current/_includes/v24.2/known-limitations/pcr-scheduled-changefeeds.md deleted file mode 100644 index 31fbf83187c..00000000000 --- a/src/current/_includes/v24.2/known-limitations/pcr-scheduled-changefeeds.md +++ /dev/null @@ -1 +0,0 @@ -After the [cutover process]({% link {{ page.version.version }}/cutover-replication.md %}) for [physical cluster replication]({% link {{ page.version.version }}/physical-cluster-replication-overview.md %}), [scheduled changefeeds]({% link {{ page.version.version }}/create-schedule-for-changefeed.md %}) will continue on the promoted cluster. You will need to manage [pausing]({% link {{ page.version.version }}/pause-schedules.md %}) or [canceling]({% link {{ page.version.version }}/drop-schedules.md %}) the schedule on the promoted standby cluster to avoid two clusters running the same changefeed to one sink. [#123776](https://github.com/cockroachdb/cockroach/issues/123776) \ No newline at end of file diff --git a/src/current/_includes/v24.2/known-limitations/physical-cluster-replication.md b/src/current/_includes/v24.2/known-limitations/physical-cluster-replication.md index f914c7eced2..a6c7edf2f32 100644 --- a/src/current/_includes/v24.2/known-limitations/physical-cluster-replication.md +++ b/src/current/_includes/v24.2/known-limitations/physical-cluster-replication.md @@ -1,6 +1,6 @@ - Physical cluster replication is supported only on CockroachDB {{ site.data.products.core }} in new clusters on v23.2 or above. Physical Cluster Replication cannot be enabled on clusters that have been upgraded from a previous version of CockroachDB. - Read queries are not supported on the standby cluster before [cutover]({% link {{ page.version.version }}/cutover-replication.md %}). -- The primary and standby cluster **cannot have different [region topology]({% link {{ page.version.version }}/topology-patterns.md %})**. For example, replicating a multi-region primary cluster to a single-region standby cluster is not supported. Mismatching regions between a multi-region primary and standby cluster is also not supported. +- The primary and standby clusters must have the same [zone configurations]({% link {{ page.version.version }}/configure-replication-zones.md %}). - Cutting back to the primary cluster after a cutover is a manual process. Refer to [Cut back to the primary cluster]({% link {{ page.version.version }}/cutover-replication.md %}#cut-back-to-the-primary-cluster). In addition, after cutover, to continue using physical cluster replication, you must configure it again. - Before cutover to the standby, the standby cluster does not support running [backups]({% link {{ page.version.version }}/backup-and-restore-overview.md %}) or [changefeeds]({% link {{ page.version.version }}/change-data-capture-overview.md %}). - Large data imports, such as those produced by [`RESTORE`]({% link {{ page.version.version }}/restore.md %}) or [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}), may dramatically increase [replication lag]({% link {{ page.version.version }}/physical-cluster-replication-technical-overview.md %}#cutover-and-promotion-process). diff --git a/src/current/_includes/v24.2/storage/free-up-disk-space.md b/src/current/_includes/v24.2/storage/free-up-disk-space.md index c63b70b766e..e4a6b08a57a 100644 --- a/src/current/_includes/v24.2/storage/free-up-disk-space.md +++ b/src/current/_includes/v24.2/storage/free-up-disk-space.md @@ -1 +1 @@ -For instructions on how to free up disk space as quickly as possible after deleting data, see [How can I free up disk space quickly?]({% link {{ page.version.version }}/operational-faqs.md %}#how-can-i-free-up-disk-space-quickly) +For instructions on how to free up disk space as quickly as possible after dropping a table, see [How can I free up disk space that was used by a dropped table?]({% link {{ page.version.version }}/operational-faqs.md %}#how-can-i-free-up-disk-space-when-dropping-a-table) diff --git a/src/current/_includes/v24.2/ui/databases.md b/src/current/_includes/v24.2/ui/databases.md index 94ae7fc0585..0465cb16860 100644 --- a/src/current/_includes/v24.2/ui/databases.md +++ b/src/current/_includes/v24.2/ui/databases.md @@ -15,7 +15,7 @@ The following information is displayed for each database: {% endif -%} | Tables | The number of tables in the database. | {% if page.cloud != true -%} -| Regions/Nodes | The regions and nodes on which the tables in the database are located. This is not displayed on a single-node cluster. | +| Regions/Nodes | The regions and nodes on which the tables in the database are located. This is not displayed on a single-node cluster.
On a multi-node cluster, the display of this information is controlled by the cluster setting [`ui.database_locality_metadata.enabled`](#ui-database_locality_metadata-enabled-cluster-setting) (default `true`). | | Index Recommendations | The number of index recommendations for the database. | {%- else -%} | Regions | The regions where the tables in the database are located. | @@ -26,6 +26,11 @@ Click a **database name** to open the **Tables** page. - Select **View: Tables** in the pulldown menu to display the [Tables view](#tables-view). - Select **View: Grants** in the pulldown menu to display the [Grants view](#grants-view). +{% if page.cloud != true -%} +### `ui.database_locality_metadata.enabled` cluster setting +{% include_cached new-in.html version="v24.2.6" %} Retrieving extended database and table region information can cause significant CPU load on large multi-node clusters with many ranges. You can prevent the retrieval of this data and the associated CPU load by disabling the [`ui.database_locality_metadata.enabled` cluster setting]({{ link_prefix }}cluster-settings.html#setting-ui-database-locality-metadata-enabled). When set to `false`, “No data” will be displayed for region data and replica counts. If you require this data, use the SQL statement [`SHOW RANGES FROM {DATABASE|TABLE}`]({{ link_prefix }}show-ranges.html) to compute this information. +{% endif -%} + ## Search and filter By default, the Databases page shows all databases running on the cluster. By default, the [**Tables** view](#tables-view) and the [**Grants** view](#grants-view) show all tables in a selected database. @@ -63,7 +68,7 @@ The following information is displayed for each table: | Columns | The number of columns in the table. | | Indexes | The number of indexes in the table. | {% if page.cloud != true -%} -| Regions | The regions and nodes on which the table data is stored. This is not displayed on a single-node cluster. | +| Regions | The regions and nodes on which the table data is stored. This is not displayed on a single-node cluster.
On a multi-node cluster, the display of this information is controlled by the cluster setting [`ui.database_locality_metadata.enabled`](#ui-database_locality_metadata-enabled-cluster-setting) (default `true`). | {% else -%} | Regions | The regions where the table data is stored. {% endif -%} @@ -84,14 +89,14 @@ The table details include: {% if page.cloud != true %} - **Size**: The approximate disk size of all replicas of this table on the cluster. -- **Replicas**: The number of [replicas]({{ link_prefix }}architecture/replication-layer.html) of this table on the cluster. +- **Replicas**: The number of [replicas]({{ link_prefix }}architecture/replication-layer.html) of this table on the cluster. On a multi-node cluster, the display of this information is controlled by the cluster setting [`ui.database_locality_metadata.enabled`](#ui-database_locality_metadata-enabled-cluster-setting) (default `true`). - **Ranges**: The number of [ranges]({{ link_prefix }}architecture/glossary.html#architecture-range) in this table. - **% of Live Data**: Percentage of total uncompressed logical data that has not been modified (updated or deleted). - **Table Stats Last Updated**: The last time table statistics were created or updated. {% endif %} - **Auto Stats Collection**: Whether [automatic statistics collection]({{ link_prefix }}cost-based-optimizer.html#table-statistics) is enabled. {% if page.cloud != true %} -- **Regions/Nodes**: The regions and nodes on which the table data is stored. This is not displayed on a single-node cluster. +- **Regions/Nodes**: The regions and nodes on which the table data is stored. This is not displayed on a single-node cluster. On a multi-node cluster, the display of this information is controlled by the cluster setting [`ui.database_locality_metadata.enabled`](#ui-database_locality_metadata-enabled-cluster-setting) (default `true`). {% else %} - **Regions**: The regions where the table data is stored. {% endif %} diff --git a/src/current/_includes/v24.3/backups/backup-to-deprec.md b/src/current/_includes/v24.3/backups/backup-to-deprec.md deleted file mode 100644 index 77a2b6cae18..00000000000 --- a/src/current/_includes/v24.3/backups/backup-to-deprec.md +++ /dev/null @@ -1,7 +0,0 @@ -{{site.data.alerts.callout_danger}} -The `BACKUP ... TO` and `RESTORE ... FROM` syntax is **deprecated** as of v22.1 and will be removed in a future release. - -We recommend using the `BACKUP ... INTO {collectionURI}` syntax, which creates or adds to a [backup collection]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}#backup-collections) in your storage location. For restoring backups, we recommend using `RESTORE FROM {backup} IN {collectionURI}` with `{backup}` being [`LATEST`]({% link {{ page.version.version }}/restore.md %}#restore-the-most-recent-full-or-incremental-backup) or a specific [subdirectory]({% link {{ page.version.version }}/restore.md %}#subdir-param). - -For guidance on the syntax for backups and restores, see the [`BACKUP`]({% link {{ page.version.version }}/backup.md %}#examples) and [`RESTORE`]({% link {{ page.version.version }}/restore.md %}#examples) examples. -{{site.data.alerts.end}} diff --git a/src/current/_includes/v24.3/backups/old-syntax-removed.md b/src/current/_includes/v24.3/backups/old-syntax-removed.md new file mode 100644 index 00000000000..7052fe0d3af --- /dev/null +++ b/src/current/_includes/v24.3/backups/old-syntax-removed.md @@ -0,0 +1,5 @@ +{{site.data.alerts.callout_danger}} +The `BACKUP ... TO` and `RESTORE ... FROM {storage_uri}` syntax has been removed from CockroachDB v24.3 and later. + +For details on the syntax to run `BACKUP` and `RESTORE`, refer to the {% if page.name == "backup.md" %} [backup](#examples) {% else %} [backup]({% link {{ page.version.version }}/backup.md %}#examples) {% endif %} and {% if page.name == "restore.md" %} [restore](#examples) {% else %} [restore]({% link {{ page.version.version }}/restore.md %}#examples) {% endif %} examples. +{{site.data.alerts.end}} \ No newline at end of file diff --git a/src/current/_includes/v24.3/backups/recommend-backups-for-upgrade.md b/src/current/_includes/v24.3/backups/recommend-backups-for-upgrade.md index 375f4489914..2ef075abf9c 100644 --- a/src/current/_includes/v24.3/backups/recommend-backups-for-upgrade.md +++ b/src/current/_includes/v24.3/backups/recommend-backups-for-upgrade.md @@ -3,7 +3,4 @@ When upgrading to a major release, you can optionally [take a self-managed backup]({% link cockroachcloud/take-and-restore-self-managed-backups.md %}) of your cluster to your own cloud storage, as an extra layer of protection in case the upgrade leads to issues. {% else %} -CockroachDB is designed with high fault tolerance. However, taking regular backups of your data is an operational best practice for [disaster recovery]({% link {{ page.version.version }}/disaster-recovery-planning.md %}) planning. - -We recommend that you enable [managed backups]({% link cockroachcloud/managed-backups.md %}#managed-backup-settings) and confirm that the cluster is backed up before beginning a major-version upgrade. This provides an extra layer of protection in case the upgrade leads to issues. -{% endif %} +CockroachDB is designed with high fault tolerance. However, taking regular backups of your data is an operational best practice for [disaster recovery]({% link {{ page.version.version }}/disaster-recovery-planning.md %}) planning.{% endif %} diff --git a/src/current/_includes/v24.3/cdc/lagging-ranges.md b/src/current/_includes/v24.3/cdc/lagging-ranges.md index 8d0b5eb6c23..45180baa57f 100644 --- a/src/current/_includes/v24.3/cdc/lagging-ranges.md +++ b/src/current/_includes/v24.3/cdc/lagging-ranges.md @@ -1,10 +1,12 @@ -Use the `changefeed.lagging_ranges` metric to track the number of ranges that are behind in a changefeed. This is calculated based on the [changefeed options]({% link {{ page.version.version }}/create-changefeed.md %}#options): +Use the `changefeed.lagging_ranges` metric to track the number of [ranges]({% link {{ page.version.version }}/architecture/overview.md %}#range) that are behind in a changefeed. This is calculated based on the [changefeed options]({% link {{ page.version.version }}/create-changefeed.md %}#options): - `lagging_ranges_threshold` sets a duration from the present that determines the length of time a range is considered to be lagging behind, which will then track in the [`lagging_ranges`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#lagging-ranges-metric) metric. Note that ranges undergoing an [initial scan]({% link {{ page.version.version }}/create-changefeed.md %}#initial-scan) for longer than the threshold duration are considered to be lagging. Starting a changefeed with an initial scan on a large table will likely increment the metric for each range in the table. As ranges complete the initial scan, the number of ranges lagging behind will decrease. - **Default:** `3m` - `lagging_ranges_polling_interval` sets the interval rate for when lagging ranges are checked and the `lagging_ranges` metric is updated. Polling adds latency to the `lagging_ranges` metric being updated. For example, if a range falls behind by 3 minutes, the metric may not update until an additional minute afterward. - **Default:** `1m` +{% include_cached new-in.html version="v24.3" %} Use the `changefeed.total_ranges` metric to monitor the number of ranges that are watched by [aggregator processors]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}) participating in the changefeed job. If you're experiencing lagging ranges, `changefeed.total_ranges` may indicate that the number of ranges watched by aggregator processors in the job is unbalanced. You may want to try [pausing]({% link {{ page.version.version }}/pause-job.md %}) the changefeed and then [resuming]({% link {{ page.version.version }}/resume-job.md %}) it, so that the changefeed replans the work in the cluster. `changefeed.total_ranges` shares the same polling interval as the `changefeed.lagging_ranges` metric, which is controlled by the `lagging_ranges_polling_interval` option. + {{site.data.alerts.callout_success}} -You can use the [`metrics_label`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#using-changefeed-metrics-labels) option to track the `lagging_ranges` metric per changefeed. +You can use the [`metrics_label`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#using-changefeed-metrics-labels) option to track the `lagging_ranges` and `total_ranges` metric per changefeed. {{site.data.alerts.end}} \ No newline at end of file diff --git a/src/current/_includes/v24.3/child-metrics-table.md b/src/current/_includes/v24.3/child-metrics-table.md index 05d57c55453..2b1a16f092a 100644 --- a/src/current/_includes/v24.3/child-metrics-table.md +++ b/src/current/_includes/v24.3/child-metrics-table.md @@ -7,7 +7,7 @@ Following is a list of the metrics that have child metrics:
| CockroachDB Metric Name | +Description | +Type | +Unit | +
{{ m.metric_id }} |
+ {% comment %} Use the value from the metrics-list, if any, followed by the value in the available-metrics-not-in-metrics-list, if any. {% endcomment %}
+ {{ metrics-list[0].description }}{{ m.description }} | +{{ metrics-list[0].type }}{{ m.type }} | +{{ metrics-list[0].unit }}{{ m.unit }} | +
-
+
+
Usage-based billing only
Provisioned compute, usage-based storage, data transfer, backups, and Change Data Capture (CDC)
Provisioned compute, usage-based storage, data transfer, backups, and Change data capture (CDC)
Provisioned compute, storage, and IOPS with usage-based billing for data transfer, backups, and CDC
Managed Backups storage rates vary per cloud provider, region, and backup frequency.
Self-Managed Backups fee incurred per GiB transferred to your own object storage.
Managed Backups storage rates vary per cloud provider, region, backup frequency, and whether security add-on is enabled.
Self-Managed Backups fee incurred per GiB transferred to your own object storage. Rate varies depending on whether security add-on is enabled.
Managed backups storage rates vary per cloud provider, region, and backup frequency.
Self-managed backups fee incurred per GiB transferred to your own object storage.
Managed backups storage rates vary per cloud provider, region, backup frequency, and whether security add-on is enabled.
Self-managed backups fee incurred per GiB transferred to your own object storage. Rate varies depending on whether security add-on is enabled.
General updates
+ +- CockroachDB {{ site.data.products.cloud }} is now available as a pay-as-you-go offering on the Google Cloud Marketplace. This allows Google Cloud customers to pay for CockroachDB {{ site.data.products.cloud }} charges via their Google Cloud accounts, with no up-front commitments. For more detail, refer to: + - [CockroachDB (pay-as-you-go)](https://console.cloud.google.com/marketplace/product/cockroachlabs/cockroachdb-pay-as-you-go) on the Google Cloud Marketplace. + - [Subscribe through the Google Cloud Marketplace]({% link cockroachcloud/billing-management.md %}?filters=gcp#subscribe-through-aws-marketplace) in the CockroachDB {{ site.data.products.cloud }} documentation. + ## December 1, 2024 As of December 1, 2024, [updated pricing](https://www.cockroachlabs.com/pricing/new/) that was recently [announced](https://www.cockroachlabs.com/blog/improved-cockroachdb-cloud-pricing/) for CockroachDB Cloud is now in effect for all customers except those with annual or multi-year contracts that began prior to December 1, 2024. For those customers, the updated pricing, including new usage-based costs, goes into effect upon contract renewal. Prior to renewal, line items for usage of data transfer, backups, and changefeeds are displayed in the [Billing](https://cockroachlabs.cloud/billing) interface and on invoices with a $0 charge, while showing actual usage metrics to help estimate future costs. @@ -112,7 +120,7 @@ In addition, this release includes the following features: - CockroachDB {{ site.data.products.cloud }} is now available as a pay-as-you-go offering on the AWS Marketplace. This allows AWS customers to pay for CockroachDB Cloud charges via their AWS accounts, with no up-front commitments. For more detail, refer to: - [CockroachDB (pay-as-you-go)](https://aws.amazon.com/marketplace/pp/prodview-n3xpypxea63du) on AWS Marketplace. - - [Subscribe through AWS Marketplace]({% link cockroachcloud/billing-management.md %}?filters=marketplace#subscribe-through-aws-marketplace) in the CockroachDB {{ site.data.products.cloud }} documentation. + - [Subscribe through AWS Marketplace]({% link cockroachcloud/billing-management.md %}?filters=aws#subscribe-through-aws-marketplace) in the CockroachDB {{ site.data.products.cloud }} documentation. ## July 18, 2024 diff --git a/src/current/v22.2/begin-transaction.md b/src/current/v22.2/begin-transaction.md index d071eea8165..3e9966642f4 100644 --- a/src/current/v22.2/begin-transaction.md +++ b/src/current/v22.2/begin-transaction.md @@ -15,7 +15,7 @@ When using transactions, your application should include logic to [retry transac ## Synopsis
-{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/begin.html %}
+{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/legacy_begin.html %}
## Required privileges
diff --git a/src/current/v23.1/begin-transaction.md b/src/current/v23.1/begin-transaction.md
index 685a395a2f1..b2cbef9a048 100644
--- a/src/current/v23.1/begin-transaction.md
+++ b/src/current/v23.1/begin-transaction.md
@@ -7,15 +7,14 @@ docs_area: reference.sql
The `BEGIN` [statement]({% link {{ page.version.version }}/sql-statements.md %}) initiates a [transaction]({% link {{ page.version.version }}/transactions.md %}), which either successfully executes all of the statements it contains or none at all.
-{{site.data.alerts.callout_danger}}
-When using transactions, your application should include logic to [retry transactions]({% link {{ page.version.version }}/transactions.md %}#transaction-retries) that are aborted to break a dependency cycle between concurrent transactions.
+{{site.data.alerts.callout_info}}
+When running under the default [`SERIALIZABLE`]({% link {{ page.version.version }}/demo-serializable.md %}) isolation level, your application should [use a retry loop to handle transaction retry errors]({% link {{ page.version.version }}/query-behavior-troubleshooting.md %}#transaction-retry-errors) that can occur under [contention]({{ link_prefix }}performance-best-practices-overview.html#transaction-contention).
{{site.data.alerts.end}}
-
## Synopsis
-{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/begin.html %}
+{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/legacy_begin.html %}
## Required privileges
diff --git a/src/current/v23.1/create-changefeed.md b/src/current/v23.1/create-changefeed.md
index 224833977f5..1a1aeaea3b3 100644
--- a/src/current/v23.1/create-changefeed.md
+++ b/src/current/v23.1/create-changefeed.md
@@ -170,7 +170,7 @@ Option | Value | Description
New in v23.1: `key_column` | `'column'` | Override the key used in [message metadata]({% link {{ page.version.version }}/changefeed-messages.md %}). This changes the key hashed to determine downstream partitions. In sinks that support partitioning by message, CockroachDB uses the [32-bit FNV-1a](https://wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function) hashing algorithm to determine which partition to send to.**Note:** `key_column` does not preserve ordering of messages from CockroachDB to the downstream sink, therefore you must also include the [`unordered`](#unordered) option in your changefeed creation statement. It does not affect per-key [ordering guarantees]({% link {{ page.version.version }}/changefeed-messages.md %}#ordering-and-delivery-guarantees) or the output of [`key_in_value`](#key-in-value).
See the [Define a key to determine the changefeed sink partition](#define-a-key-to-determine-the-changefeed-sink-partition) example. `key_in_value` | N/A | Add a primary key array to the emitted message. This makes the [primary key]({% link {{ page.version.version }}/primary-key.md %}) of a deleted row recoverable in sinks where each message has a value but not a key (most have a key and value in each message). `key_in_value` is automatically used for [cloud storage sinks]({% link {{ page.version.version }}/changefeed-sinks.md %}#cloud-storage-sink), [webhook sinks]({% link {{ page.version.version }}/changefeed-sinks.md %}#webhook-sink), and [GC Pub/Sub sinks]({% link {{ page.version.version }}/changefeed-sinks.md %}#google-cloud-pub-sub). `metrics_label` | [`STRING`]({% link {{ page.version.version }}/string.md %}) | Define a metrics label to which the metrics for one or multiple changefeeds increment. All changefeeds also have their metrics aggregated.
The maximum length of a label is 128 bytes. There is a limit of 1024 unique labels.
`WITH metrics_label=label_name`
For more detail on usage and considerations, see [Using changefeed metrics labels]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#using-changefeed-metrics-labels). -`min_checkpoint_frequency` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Controls how often nodes flush their progress to the [coordinating changefeed node]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}). Changefeeds will wait for at least the specified duration before a flush to the sink. This can help you control the flush frequency of higher latency sinks to achieve better throughput. However, more frequent checkpointing can increase CPU usage. If this is set to `0s`, a node will flush messages as long as the high-water mark has increased for the ranges that particular node is processing. If a changefeed is resumed, then `min_checkpoint_frequency` is the amount of time that changefeed will need to catch up. That is, it could emit [duplicate messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) during this time.
**Note:** [`resolved`](#resolved-option) messages will not be emitted more frequently than the configured `min_checkpoint_frequency` (but may be emitted less frequently). If you require `resolved` messages more frequently than `30s`, you must configure `min_checkpoint_frequency` to at least the desired `resolved` message frequency. For more details, refer to [Resolved message frequency]({% link {{ page.version.version }}/changefeed-messages.md %}#resolved-timestamp-frequency).
**Default:** `30s` +`min_checkpoint_frequency` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Controls how often a node's changefeed [aggregator]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}) will flush their progress to the [coordinating changefeed node]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}). A node's changefeed aggregator will wait at least the specified duration between sending progress updates for the ranges it is watching to the coordinator. This can help you control the flush frequency of higher latency sinks to achieve better throughput. However, more frequent checkpointing can increase CPU usage. If this is set to `0s`, a node will flush messages as long as the high-water mark has increased for the ranges that particular node is processing. If a changefeed is resumed, then `min_checkpoint_frequency` is the amount of time that changefeed will need to catch up. That is, it could emit [duplicate messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) during this time.
**Note:** [`resolved`](#resolved-option) messages will not be emitted more frequently than the configured `min_checkpoint_frequency` (but may be emitted less frequently). If you require `resolved` messages more frequently than `30s`, you must configure `min_checkpoint_frequency` to at least the desired `resolved` message frequency. For more details, refer to [Resolved message frequency]({% link {{ page.version.version }}/changefeed-messages.md %}#resolved-timestamp-frequency).
**Default:** `30s` `mvcc_timestamp` | N/A | Include the [MVCC]({% link {{ page.version.version }}/architecture/storage-layer.md %}#mvcc) timestamp for each emitted row in a changefeed. With the `mvcc_timestamp` option, each emitted row will always contain its MVCC timestamp, even during the changefeed's initial backfill. `on_error` | `pause` / `fail` | Use `on_error=pause` to pause the changefeed when encountering **non**-retryable errors. `on_error=pause` will pause the changefeed instead of sending it into a terminal failure state. **Note:** Retryable errors will continue to be retried with this option specified.
Use with [`protect_data_from_gc_on_pause`](#protect-pause) to protect changes from [garbage collection]({% link {{ page.version.version }}/configure-replication-zones.md %}#gc-ttlseconds).
If a changefeed with `on_error=pause` is running when a watched table is [truncated]({% link {{ page.version.version }}/truncate.md %}), the changefeed will pause but will not be able to resume reads from that table. Using [`ALTER CHANGEFEED`]({% link {{ page.version.version }}/alter-changefeed.md %}) to drop the table from the changefeed and then [resuming the job]({% link {{ page.version.version }}/resume-job.md %}) will work, but you cannot add the same table to the changefeed again. Instead, you will need to [create a new changefeed](#start-a-new-changefeed-where-another-ended) for that table.
Default: `on_error=fail` `protect_data_from_gc_on_pause` | N/A | When a [changefeed is paused]({% link {{ page.version.version }}/pause-job.md %}), ensure that the data needed to [resume the changefeed]({% link {{ page.version.version }}/resume-job.md %}) is not garbage collected. If `protect_data_from_gc_on_pause` is **unset**, pausing the changefeed will release the existing protected timestamp records. It is also important to note that pausing and adding `protect_data_from_gc_on_pause` to a changefeed will not protect data if the [garbage collection]({% link {{ page.version.version }}/configure-replication-zones.md %}#gc-ttlseconds) window has already passed.
Use with [`on_error=pause`](#on-error) to protect changes from garbage collection when encountering non-retryable errors.
See [Garbage collection and changefeeds]({% link {{ page.version.version }}/changefeed-messages.md %}#garbage-collection-and-changefeeds) for more detail on protecting changefeed data.
**Note:** If you use this option, changefeeds that are left paused for long periods of time can prevent garbage collection. Use with the [`gc_protect_expires_after`](#gc-protect-expire) option to set a limit for protected data and for how long a changefeed will remain paused. diff --git a/src/current/v23.1/how-does-an-enterprise-changefeed-work.md b/src/current/v23.1/how-does-an-enterprise-changefeed-work.md index 0d657ca2e6d..98952ad89da 100644 --- a/src/current/v23.1/how-does-an-enterprise-changefeed-work.md +++ b/src/current/v23.1/how-does-an-enterprise-changefeed-work.md @@ -11,7 +11,7 @@ When an {{ site.data.products.enterprise }} changefeed is started on a node, tha {% include {{ page.version.version }}/cdc/work-distribution-setting.md %} {{site.data.alerts.end}} -Each node uses its aggregator processors to send back checkpoint progress to the coordinator, which gathers this information to update the high-water mark timestamp. The high-water mark acts as a checkpoint for the changefeed’s job progress, and guarantees that all changes before (or at) the timestamp have been emitted. In the unlikely event that the changefeed’s coordinating node were to fail during the job, that role will move to a different node and the changefeed will restart from the last checkpoint. If restarted, the changefeed may [re-emit messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) starting at the high-water mark time to the current time. Refer to [Ordering Guarantees]({% link {{ page.version.version }}/changefeed-messages.md %}#ordering-and-delivery-guarantees) for detail on CockroachDB's at-least-once-delivery-guarantee and how per-key message ordering is applied. +Each node uses its _aggregator processors_ to send back checkpoint progress to the coordinator, which gathers this information to update the _high-water mark timestamp_. The high-water mark acts as a checkpoint for the changefeed’s job progress, and guarantees that all changes before (or at) the timestamp have been emitted. In the unlikely event that the changefeed’s coordinating node were to fail during the job, that role will move to a different node and the changefeed will restart from the last checkpoint. If restarted, the changefeed may [re-emit messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) starting at the high-water mark time to the current time. Refer to [Ordering Guarantees]({% link {{ page.version.version }}/changefeed-messages.md %}#ordering-and-delivery-guarantees) for detail on CockroachDB's at-least-once-delivery-guarantee and how per-key message ordering is applied.
diff --git a/src/current/v23.1/monitor-and-debug-changefeeds.md b/src/current/v23.1/monitor-and-debug-changefeeds.md
index 8801fa611cd..fab5c562f75 100644
--- a/src/current/v23.1/monitor-and-debug-changefeeds.md
+++ b/src/current/v23.1/monitor-and-debug-changefeeds.md
@@ -156,6 +156,7 @@ changefeed_emitted_bytes{scope="vehicles"} 183557
`error_retries` | Total retryable errors encountered by changefeeds. | Errors
`backfill_pending_ranges` | Number of [ranges]({% link {{ page.version.version }}/architecture/overview.md %}#architecture-range) in an ongoing backfill that are yet to be fully emitted. | Ranges
`message_size_hist` | Distribution in the size of emitted messages. | Bytes
+New in v23.1.29: `total_ranges` | Total number of ranges that are watched by [aggregator processors]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}) participating in the changefeed job. `changefeed.total_ranges` shares the same polling interval as the [`changefeed.lagging_ranges`](#lagging-ranges-metric) metric, which is controlled by the `changefeed.lagging_ranges_polling_interval` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}). For more details, refer to [Lagging ranges](#lagging-ranges).
### Monitoring and measuring changefeed latency
diff --git a/src/current/v23.1/operational-faqs.md b/src/current/v23.1/operational-faqs.md
index b84fff484ca..56c0c9ff40b 100644
--- a/src/current/v23.1/operational-faqs.md
+++ b/src/current/v23.1/operational-faqs.md
@@ -81,9 +81,13 @@ When MVCC garbage is deleted by garbage collection, the data is still not yet ph
{% include {{page.version.version}}/storage/free-up-disk-space.md %}
-## How can I free up disk space quickly?
+## How can I free up disk space when dropping a table?
-If you've noticed that [your disk space is not freeing up quickly enough after deleting data](#why-is-my-disk-usage-not-decreasing-after-deleting-data), you can take the following steps to free up disk space more quickly. This example assumes a table `t`.
+If you've noticed that [your disk space is not freeing up quickly enough after dropping a table](#why-is-my-disk-usage-not-decreasing-after-deleting-data), you can take the following steps to free up disk space more quickly the next time you drop a table. This example assumes a table `t` exists.
+
+{{site.data.alerts.callout_info}}
+The procedure shown here only works if you get the range IDs from the table **before** running [`DROP TABLE`]({% link {{ page.version.version }}/drop-table.md %}). If you are in an emergency situation due to running out of disk, see [What happens when a node runs out of disk space?](#what-happens-when-a-node-runs-out-of-disk-space)
+{{site.data.alerts.end}}
1. Lower the [`gc.ttlseconds` parameter]({% link {{ page.version.version }}/configure-replication-zones.md %}#gc-ttlseconds) to 10 minutes.
diff --git a/src/current/v23.2/begin-transaction.md b/src/current/v23.2/begin-transaction.md
index 8c159dac361..6c50b135e95 100644
--- a/src/current/v23.2/begin-transaction.md
+++ b/src/current/v23.2/begin-transaction.md
@@ -7,15 +7,14 @@ docs_area: reference.sql
The `BEGIN` [statement]({% link {{ page.version.version }}/sql-statements.md %}) initiates a [transaction]({% link {{ page.version.version }}/transactions.md %}), which either successfully executes all of the statements it contains or none at all.
-{{site.data.alerts.callout_danger}}
-When using transactions, your application should include logic to [retry transactions]({% link {{ page.version.version }}/transactions.md %}#transaction-retries) that are aborted to break a dependency cycle between concurrent transactions.
+{{site.data.alerts.callout_info}}
+When running under the default [`SERIALIZABLE`]({% link {{ page.version.version }}/demo-serializable.md %}) isolation level, your application should [use a retry loop to handle transaction retry errors]({% link {{ page.version.version }}/query-behavior-troubleshooting.md %}#transaction-retry-errors) that can occur under [contention]({{ link_prefix }}performance-best-practices-overview.html#transaction-contention).
{{site.data.alerts.end}}
-
## Synopsis
-{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/begin.html %}
+{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/legacy_begin.html %}
## Required privileges
diff --git a/src/current/v23.2/create-and-configure-changefeeds.md b/src/current/v23.2/create-and-configure-changefeeds.md
index 2cb7b8a6120..496a810426b 100644
--- a/src/current/v23.2/create-and-configure-changefeeds.md
+++ b/src/current/v23.2/create-and-configure-changefeeds.md
@@ -17,6 +17,7 @@ This page describes:
1. Enable rangefeeds on CockroachDB {{ site.data.products.advanced }} and CockroachDB {{ site.data.products.core }}. Refer to [Enable rangefeeds](#enable-rangefeeds) for instructions.
1. Decide on whether you will run an {{ site.data.products.enterprise }} or basic changefeed. Refer to the [Overview]({% link {{ page.version.version }}/change-data-capture-overview.md %}) page for a comparative capability table.
1. Plan the number of changefeeds versus the number of tables to include in a single changefeed for your cluster. {% include {{ page.version.version }}/cdc/changefeed-number-limit.md %} Refer to [System resources and running changefeeds](#system-resources-and-running-changefeeds) and [Recommendations for the number of target tables](#recommendations-for-the-number-of-target-tables).
+ - {% include common/cdc-cloud-costs-link.md %}
1. Consider whether your {{ site.data.products.enterprise }} [changefeed use case](#create) would be better served by [change data capture queries]({% link {{ page.version.version }}/cdc-queries.md %}) that can filter data on a single table. CDC queries can improve the efficiency of changefeeds because the job will not need to encode as much change data.
1. Read the [Considerations](#considerations) section that provides information on changefeed interactions that could affect how you configure or run your changefeed.
@@ -68,7 +69,7 @@ To maintain a high number of changefeeds in your cluster:
### Considerations
- If you require [`resolved`]({% link {{ page.version.version }}/create-changefeed.md %}#resolved-option) message frequency under `30s`, then you **must** set the [`min_checkpoint_frequency`]({% link {{ page.version.version }}/create-changefeed.md %}#min-checkpoint-frequency) option to at least the desired `resolved` frequency.
-- Many DDL queries (including [`TRUNCATE`]({% link {{ page.version.version }}/truncate.md %}), [`DROP TABLE`]({% link {{ page.version.version }}/drop-table.md %}), and queries that add a column family) will cause errors on a changefeed watching the affected tables. You will need to [start a new changefeed]({% link {{ page.version.version }}/create-changefeed.md %}#start-a-new-changefeed-where-another-ended). If a table is truncated that a changefeed with `on_error='pause'` is watching, you will also need to start a new changefeed. See change data capture [Known Limitations]({% link {{ page.version.version }}/change-data-capture-overview.md %}) for more detail.
+- Many DDL queries (including [`TRUNCATE`]({% link {{ page.version.version }}/truncate.md %}), [`DROP TABLE`]({% link {{ page.version.version }}/drop-table.md %}), and queries that add a column family) will cause errors on a changefeed watching the affected tables. You will need to [start a new changefeed]({% link {{ page.version.version }}/create-changefeed.md %}#start-a-new-changefeed-where-another-ended). If a table is truncated that a changefeed with `on_error='pause'` is watching, you will also need to start a new changefeed. Refer to the change data capture [Known Limitations](#known-limitations) for more detail.
- Partial or intermittent sink unavailability may impact changefeed stability. If a sink is unavailable, messages can't send, which means that a changefeed's high-water mark timestamp is at risk of falling behind the cluster's [garbage collection window]({% link {{ page.version.version }}/configure-replication-zones.md %}#replication-zone-variables). Throughput and latency can be affected once the sink is available again. However, [ordering guarantees]({% link {{ page.version.version }}/changefeed-messages.md %}#ordering-and-delivery-guarantees) will still hold for as long as a changefeed [remains active]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#monitor-a-changefeed).
- When an [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}) statement is run, any current changefeed jobs targeting that table will fail.
- After you [restore from a full-cluster backup]({% link {{ page.version.version }}/restore.md %}#full-cluster), changefeed jobs will **not** resume on the new cluster. It is necessary to manually create the changefeeds following the full-cluster restore.
diff --git a/src/current/v23.2/create-changefeed.md b/src/current/v23.2/create-changefeed.md
index 01ea40b326a..fe6a9a5341c 100644
--- a/src/current/v23.2/create-changefeed.md
+++ b/src/current/v23.2/create-changefeed.md
@@ -176,7 +176,7 @@ Option | Value | Description
New in v23.2: `lagging_ranges_threshold` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Set a duration from the present that determines the length of time a range is considered to be lagging behind, which will then track in the [`lagging_ranges`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#lagging-ranges-metric) metric. Note that ranges undergoing an [initial scan](#initial-scan) for longer than the threshold duration are considered to be lagging. Starting a changefeed with an initial scan on a large table will likely increment the metric for each range in the table. As ranges complete the initial scan, the number of ranges lagging behind will decrease.**Default:** `3m` New in v23.2: `lagging_ranges_polling_interval` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Set the interval rate for when lagging ranges are checked and the `lagging_ranges` metric is updated. Polling adds latency to the `lagging_ranges` metric being updated. For example, if a range falls behind by 3 minutes, the metric may not update until an additional minute afterward.
**Default:** `1m` `metrics_label` | [`STRING`]({% link {{ page.version.version }}/string.md %}) | Define a metrics label to which the metrics for one or multiple changefeeds increment. All changefeeds also have their metrics aggregated.
The maximum length of a label is 128 bytes. There is a limit of 1024 unique labels.
`WITH metrics_label=label_name`
For more detail on usage and considerations, see [Using changefeed metrics labels]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#using-changefeed-metrics-labels). -`min_checkpoint_frequency` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Controls how often nodes flush their progress to the [coordinating changefeed node]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}). Changefeeds will wait for at least the specified duration before a flush to the sink. This can help you control the flush frequency of higher latency sinks to achieve better throughput. However, more frequent checkpointing can increase CPU usage. If this is set to `0s`, a node will flush messages as long as the high-water mark has increased for the ranges that particular node is processing. If a changefeed is resumed, then `min_checkpoint_frequency` is the amount of time that changefeed will need to catch up. That is, it could emit [duplicate messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) during this time.
**Note:** [`resolved`](#resolved-option) messages will not be emitted more frequently than the configured `min_checkpoint_frequency` (but may be emitted less frequently). If you require `resolved` messages more frequently than `30s`, you must configure `min_checkpoint_frequency` to at least the desired `resolved` message frequency. For more details, refer to [Resolved message frequency]({% link {{ page.version.version }}/changefeed-messages.md %}#resolved-timestamp-frequency).
**Default:** `30s` +`min_checkpoint_frequency` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Controls how often a node's changefeed [aggregator]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}) will flush their progress to the [coordinating changefeed node]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}). A node's changefeed aggregator will wait at least the specified duration between sending progress updates for the ranges it is watching to the coordinator. This can help you control the flush frequency of higher latency sinks to achieve better throughput. However, more frequent checkpointing can increase CPU usage. If this is set to `0s`, a node will flush messages as long as the high-water mark has increased for the ranges that particular node is processing. If a changefeed is resumed, then `min_checkpoint_frequency` is the amount of time that changefeed will need to catch up. That is, it could emit [duplicate messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) during this time.
**Note:** [`resolved`](#resolved-option) messages will not be emitted more frequently than the configured `min_checkpoint_frequency` (but may be emitted less frequently). If you require `resolved` messages more frequently than `30s`, you must configure `min_checkpoint_frequency` to at least the desired `resolved` message frequency. For more details, refer to [Resolved message frequency]({% link {{ page.version.version }}/changefeed-messages.md %}#resolved-timestamp-frequency).
**Default:** `30s` `mvcc_timestamp` | N/A | Include the [MVCC]({% link {{ page.version.version }}/architecture/storage-layer.md %}#mvcc) timestamp for each emitted row in a changefeed. With the `mvcc_timestamp` option, each emitted row will always contain its MVCC timestamp, even during the changefeed's initial backfill. `on_error` | `pause` / `fail` | Use `on_error=pause` to pause the changefeed when encountering **non**-retryable errors. `on_error=pause` will pause the changefeed instead of sending it into a terminal failure state. **Note:** Retryable errors will continue to be retried with this option specified.
Use with [`protect_data_from_gc_on_pause`](#protect-pause) to protect changes from [garbage collection]({% link {{ page.version.version }}/configure-replication-zones.md %}#gc-ttlseconds).
If a changefeed with `on_error=pause` is running when a watched table is [truncated]({% link {{ page.version.version }}/truncate.md %}), the changefeed will pause but will not be able to resume reads from that table. Using [`ALTER CHANGEFEED`]({% link {{ page.version.version }}/alter-changefeed.md %}) to drop the table from the changefeed and then [resuming the job]({% link {{ page.version.version }}/resume-job.md %}) will work, but you cannot add the same table to the changefeed again. Instead, you will need to [create a new changefeed](#start-a-new-changefeed-where-another-ended) for that table.
Default: `on_error=fail` `protect_data_from_gc_on_pause` | N/A | This option is deprecated as of v23.2 and will be removed in a future release.
When a [changefeed is paused]({% link {{ page.version.version }}/pause-job.md %}), ensure that the data needed to [resume the changefeed]({% link {{ page.version.version }}/resume-job.md %}) is not garbage collected. If `protect_data_from_gc_on_pause` is **unset**, pausing the changefeed will release the existing protected timestamp records. It is also important to note that pausing and adding `protect_data_from_gc_on_pause` to a changefeed will not protect data if the [garbage collection]({% link {{ page.version.version }}/configure-replication-zones.md %}#gc-ttlseconds) window has already passed.
Use with [`on_error=pause`](#on-error) to protect changes from garbage collection when encountering non-retryable errors.
Refer to [Protect Changefeed Data from Garbage Collection]({% link {{ page.version.version }}/protect-changefeed-data.md %}) for more detail on protecting changefeed data.
**Note:** If you use this option, changefeeds that are left paused for long periods of time can prevent garbage collection. Use with the [`gc_protect_expires_after`](#gc-protect-expire) option to set a limit for protected data and for how long a changefeed will remain paused. diff --git a/src/current/v23.2/how-does-an-enterprise-changefeed-work.md b/src/current/v23.2/how-does-an-enterprise-changefeed-work.md index 676a45c9c10..7083eaf91eb 100644 --- a/src/current/v23.2/how-does-an-enterprise-changefeed-work.md +++ b/src/current/v23.2/how-does-an-enterprise-changefeed-work.md @@ -11,7 +11,7 @@ When an {{ site.data.products.enterprise }} changefeed is started on a node, tha {% include {{ page.version.version }}/cdc/work-distribution-setting.md %} {{site.data.alerts.end}} -Each node uses its aggregator processors to send back checkpoint progress to the coordinator, which gathers this information to update the high-water mark timestamp. The high-water mark acts as a checkpoint for the changefeed’s job progress, and guarantees that all changes before (or at) the timestamp have been emitted. In the unlikely event that the changefeed’s coordinating node were to fail during the job, that role will move to a different node and the changefeed will restart from the last checkpoint. If restarted, the changefeed may [re-emit messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) starting at the high-water mark time to the current time. Refer to [Ordering Guarantees]({% link {{ page.version.version }}/changefeed-messages.md %}#ordering-and-delivery-guarantees) for detail on CockroachDB's at-least-once-delivery-guarantee and how per-key message ordering is applied. +Each node uses its _aggregator processors_ to send back checkpoint progress to the coordinator, which gathers this information to update the _high-water mark timestamp_. The high-water mark acts as a checkpoint for the changefeed’s job progress, and guarantees that all changes before (or at) the timestamp have been emitted. In the unlikely event that the changefeed’s coordinating node were to fail during the job, that role will move to a different node and the changefeed will restart from the last checkpoint. If restarted, the changefeed may [re-emit messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) starting at the high-water mark time to the current time. Refer to [Ordering Guarantees]({% link {{ page.version.version }}/changefeed-messages.md %}#ordering-and-delivery-guarantees) for detail on CockroachDB's at-least-once-delivery-guarantee and how per-key message ordering is applied.
diff --git a/src/current/v23.2/monitor-and-debug-changefeeds.md b/src/current/v23.2/monitor-and-debug-changefeeds.md
index d01a020ec8d..0e3b515696e 100644
--- a/src/current/v23.2/monitor-and-debug-changefeeds.md
+++ b/src/current/v23.2/monitor-and-debug-changefeeds.md
@@ -156,6 +156,7 @@ changefeed_emitted_bytes{scope="vehicles"} 183557
`error_retries` | Total retryable errors encountered by changefeeds. | Errors
`backfill_pending_ranges` | Number of [ranges]({% link {{ page.version.version }}/architecture/overview.md %}#architecture-range) in an ongoing backfill that are yet to be fully emitted. | Ranges
`message_size_hist` | Distribution in the size of emitted messages. | Bytes
+New in v23.2.13: `total_ranges` | Total number of ranges that are watched by [aggregator processors]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}) participating in the changefeed job. `changefeed.total_ranges` shares the same polling interval as the [`changefeed.lagging_ranges`](#lagging-ranges-metric) metric, which is controlled by the `lagging_ranges_polling_interval` option. For more details, refer to [Lagging ranges](#lagging-ranges).
### Monitoring and measuring changefeed latency
diff --git a/src/current/v23.2/operational-faqs.md b/src/current/v23.2/operational-faqs.md
index 6fa24aa8336..41bc09654b7 100644
--- a/src/current/v23.2/operational-faqs.md
+++ b/src/current/v23.2/operational-faqs.md
@@ -81,9 +81,13 @@ When MVCC garbage is deleted by garbage collection, the data is still not yet ph
{% include {{page.version.version}}/storage/free-up-disk-space.md %}
-## How can I free up disk space quickly?
+## How can I free up disk space when dropping a table?
-If you've noticed that [your disk space is not freeing up quickly enough after deleting data](#why-is-my-disk-usage-not-decreasing-after-deleting-data), you can take the following steps to free up disk space more quickly. This example assumes a table `t`.
+If you've noticed that [your disk space is not freeing up quickly enough after dropping a table](#why-is-my-disk-usage-not-decreasing-after-deleting-data), you can take the following steps to free up disk space more quickly the next time you drop a table. This example assumes a table `t` exists.
+
+{{site.data.alerts.callout_info}}
+The procedure shown here only works if you get the range IDs from the table **before** running [`DROP TABLE`]({% link {{ page.version.version }}/drop-table.md %}). If you are in an emergency situation due to running out of disk, see [What happens when a node runs out of disk space?](#what-happens-when-a-node-runs-out-of-disk-space)
+{{site.data.alerts.end}}
1. Lower the [`gc.ttlseconds` parameter]({% link {{ page.version.version }}/configure-replication-zones.md %}#gc-ttlseconds) to 10 minutes.
diff --git a/src/current/v24.1/begin-transaction.md b/src/current/v24.1/begin-transaction.md
index 8c159dac361..6c50b135e95 100644
--- a/src/current/v24.1/begin-transaction.md
+++ b/src/current/v24.1/begin-transaction.md
@@ -7,15 +7,14 @@ docs_area: reference.sql
The `BEGIN` [statement]({% link {{ page.version.version }}/sql-statements.md %}) initiates a [transaction]({% link {{ page.version.version }}/transactions.md %}), which either successfully executes all of the statements it contains or none at all.
-{{site.data.alerts.callout_danger}}
-When using transactions, your application should include logic to [retry transactions]({% link {{ page.version.version }}/transactions.md %}#transaction-retries) that are aborted to break a dependency cycle between concurrent transactions.
+{{site.data.alerts.callout_info}}
+When running under the default [`SERIALIZABLE`]({% link {{ page.version.version }}/demo-serializable.md %}) isolation level, your application should [use a retry loop to handle transaction retry errors]({% link {{ page.version.version }}/query-behavior-troubleshooting.md %}#transaction-retry-errors) that can occur under [contention]({{ link_prefix }}performance-best-practices-overview.html#transaction-contention).
{{site.data.alerts.end}}
-
## Synopsis
-{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/begin.html %}
+{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/legacy_begin.html %}
## Required privileges
diff --git a/src/current/v24.1/create-and-configure-changefeeds.md b/src/current/v24.1/create-and-configure-changefeeds.md
index ac4e4f08067..d7b0d5ad034 100644
--- a/src/current/v24.1/create-and-configure-changefeeds.md
+++ b/src/current/v24.1/create-and-configure-changefeeds.md
@@ -17,6 +17,7 @@ This page describes:
1. Enable rangefeeds on CockroachDB {{ site.data.products.advanced }} and CockroachDB {{ site.data.products.core }}. Refer to [Enable rangefeeds](#enable-rangefeeds) for instructions.
1. Decide on whether you will run an {{ site.data.products.enterprise }} or basic changefeed. Refer to the [Overview]({% link {{ page.version.version }}/change-data-capture-overview.md %}) page for a comparative capability table.
1. Plan the number of changefeeds versus the number of tables to include in a single changefeed for your cluster. {% include {{ page.version.version }}/cdc/changefeed-number-limit.md %} Refer to [System resources and running changefeeds](#system-resources-and-running-changefeeds) and [Recommendations for the number of target tables](#recommendations-for-the-number-of-target-tables).
+ - {% include common/cdc-cloud-costs-link.md %}
1. Consider whether your {{ site.data.products.enterprise }} [changefeed use case](#create) would be better served by [change data capture queries]({% link {{ page.version.version }}/cdc-queries.md %}) that can filter data on a single table. CDC queries can improve the efficiency of changefeeds because the job will not need to encode as much change data.
1. Read the [Considerations](#considerations) section that provides information on changefeed interactions that could affect how you configure or run your changefeed.
@@ -68,7 +69,7 @@ To maintain a high number of changefeeds in your cluster:
### Considerations
- If you require [`resolved`]({% link {{ page.version.version }}/create-changefeed.md %}#resolved) message frequency under `30s`, then you **must** set the [`min_checkpoint_frequency`]({% link {{ page.version.version }}/create-changefeed.md %}#min-checkpoint-frequency) option to at least the desired `resolved` frequency.
-- Many DDL queries (including [`TRUNCATE`]({% link {{ page.version.version }}/truncate.md %}), [`DROP TABLE`]({% link {{ page.version.version }}/drop-table.md %}), and queries that add a column family) will cause errors on a changefeed watching the affected tables. You will need to [start a new changefeed]({% link {{ page.version.version }}/create-changefeed.md %}#start-a-new-changefeed-where-another-ended). If a table is truncated that a changefeed with `on_error='pause'` is watching, you will also need to start a new changefeed. See change data capture [Known Limitations]({% link {{ page.version.version }}/change-data-capture-overview.md %}) for more detail.
+- Many DDL queries (including [`TRUNCATE`]({% link {{ page.version.version }}/truncate.md %}), [`DROP TABLE`]({% link {{ page.version.version }}/drop-table.md %}), and queries that add a column family) will cause errors on a changefeed watching the affected tables. You will need to [start a new changefeed]({% link {{ page.version.version }}/create-changefeed.md %}#start-a-new-changefeed-where-another-ended). If a table is truncated that a changefeed with `on_error='pause'` is watching, you will also need to start a new changefeed. Refer to the change data capture [Known Limitations](#known-limitations) for more detail.
- Partial or intermittent sink unavailability may impact changefeed stability. If a sink is unavailable, messages can't send, which means that a changefeed's high-water mark timestamp is at risk of falling behind the cluster's [garbage collection window]({% link {{ page.version.version }}/configure-replication-zones.md %}#replication-zone-variables). Throughput and latency can be affected once the sink is available again. However, [ordering guarantees]({% link {{ page.version.version }}/changefeed-messages.md %}#ordering-and-delivery-guarantees) will still hold for as long as a changefeed [remains active]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#monitor-a-changefeed).
- When an [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}) statement is run, any current changefeed jobs targeting that table will fail.
- After you [restore from a full-cluster backup]({% link {{ page.version.version }}/restore.md %}#full-cluster), changefeed jobs will **not** resume on the new cluster. It is necessary to manually create the changefeeds following the full-cluster restore.
diff --git a/src/current/v24.1/create-changefeed.md b/src/current/v24.1/create-changefeed.md
index 69df27f6043..407f07d457d 100644
--- a/src/current/v24.1/create-changefeed.md
+++ b/src/current/v24.1/create-changefeed.md
@@ -194,7 +194,7 @@ Option | Value | Description
`lagging_ranges_threshold` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Set a duration from the present that determines the length of time a range is considered to be lagging behind, which will then track in the [`lagging_ranges`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#lagging-ranges-metric) metric. Note that ranges undergoing an [initial scan](#initial-scan) for longer than the threshold duration are considered to be lagging. Starting a changefeed with an initial scan on a large table will likely increment the metric for each range in the table. As ranges complete the initial scan, the number of ranges lagging behind will decrease.**Default:** `3m` `lagging_ranges_polling_interval` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Set the interval rate for when lagging ranges are checked and the `lagging_ranges` metric is updated. Polling adds latency to the `lagging_ranges` metric being updated. For example, if a range falls behind by 3 minutes, the metric may not update until an additional minute afterward.
**Default:** `1m` `metrics_label` | [`STRING`]({% link {{ page.version.version }}/string.md %}) | Define a metrics label to which the metrics for one or multiple changefeeds increment. All changefeeds also have their metrics aggregated.
The maximum length of a label is 128 bytes. There is a limit of 1024 unique labels.
`WITH metrics_label=label_name`
For more detail on usage and considerations, see [Using changefeed metrics labels]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#using-changefeed-metrics-labels). -`min_checkpoint_frequency` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Controls how often nodes flush their progress to the [coordinating changefeed node]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}). Changefeeds will wait for at least the specified duration before a flush to the sink. This can help you control the flush frequency of higher latency sinks to achieve better throughput. However, more frequent checkpointing can increase CPU usage. If this is set to `0s`, a node will flush messages as long as the high-water mark has increased for the ranges that particular node is processing. If a changefeed is resumed, then `min_checkpoint_frequency` is the amount of time that changefeed will need to catch up. That is, it could emit [duplicate messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) during this time.
**Note:** [`resolved`](#resolved) messages will not be emitted more frequently than the configured `min_checkpoint_frequency` (but may be emitted less frequently). If you require `resolved` messages more frequently than `30s`, you must configure `min_checkpoint_frequency` to at least the desired `resolved` message frequency. For more details, refer to [Resolved message frequency]({% link {{ page.version.version }}/changefeed-messages.md %}#resolved-timestamp-frequency).
**Default:** `30s` +`min_checkpoint_frequency` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Controls how often a node's changefeed [aggregator]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}) will flush their progress to the [coordinating changefeed node]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}). A node's changefeed aggregator will wait at least the specified duration between sending progress updates for the ranges it is watching to the coordinator. This can help you control the flush frequency of higher latency sinks to achieve better throughput. However, more frequent checkpointing can increase CPU usage. If this is set to `0s`, a node will flush messages as long as the high-water mark has increased for the ranges that particular node is processing. If a changefeed is resumed, then `min_checkpoint_frequency` is the amount of time that changefeed will need to catch up. That is, it could emit [duplicate messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) during this time.
**Note:** [`resolved`](#resolved) messages will not be emitted more frequently than the configured `min_checkpoint_frequency` (but may be emitted less frequently). If you require `resolved` messages more frequently than `30s`, you must configure `min_checkpoint_frequency` to at least the desired `resolved` message frequency. For more details, refer to [Resolved message frequency]({% link {{ page.version.version }}/changefeed-messages.md %}#resolved-timestamp-frequency).
**Default:** `30s` `mvcc_timestamp` | N/A | Include the [MVCC]({% link {{ page.version.version }}/architecture/storage-layer.md %}#mvcc) timestamp for each emitted row in a changefeed. With the `mvcc_timestamp` option, each emitted row will always contain its MVCC timestamp, even during the changefeed's initial backfill. `on_error` | `pause` / `fail` | Use `on_error=pause` to pause the changefeed when encountering **non**-retryable errors. `on_error=pause` will pause the changefeed instead of sending it into a terminal failure state. **Note:** Retryable errors will continue to be retried with this option specified.
Use with [`protect_data_from_gc_on_pause`](#protect-data-from-gc-on-pause) to protect changes from [garbage collection]({% link {{ page.version.version }}/configure-replication-zones.md %}#gc-ttlseconds).
If a changefeed with `on_error=pause` is running when a watched table is [truncated]({% link {{ page.version.version }}/truncate.md %}), the changefeed will pause but will not be able to resume reads from that table. Using [`ALTER CHANGEFEED`]({% link {{ page.version.version }}/alter-changefeed.md %}) to drop the table from the changefeed and then [resuming the job]({% link {{ page.version.version }}/resume-job.md %}) will work, but you cannot add the same table to the changefeed again. Instead, you will need to [create a new changefeed](#start-a-new-changefeed-where-another-ended) for that table.
Default: `on_error=fail` `protect_data_from_gc_on_pause` | N/A | This option is deprecated as of v23.2 and will be removed in a future release.
When a [changefeed is paused]({% link {{ page.version.version }}/pause-job.md %}), ensure that the data needed to [resume the changefeed]({% link {{ page.version.version }}/resume-job.md %}) is not garbage collected. If `protect_data_from_gc_on_pause` is **unset**, pausing the changefeed will release the existing protected timestamp records. It is also important to note that pausing and adding `protect_data_from_gc_on_pause` to a changefeed will not protect data if the [garbage collection]({% link {{ page.version.version }}/configure-replication-zones.md %}#gc-ttlseconds) window has already passed.
Use with [`on_error=pause`](#on-error) to protect changes from garbage collection when encountering non-retryable errors.
Refer to [Protect Changefeed Data from Garbage Collection]({% link {{ page.version.version }}/protect-changefeed-data.md %}) for more detail on protecting changefeed data.
**Note:** If you use this option, changefeeds that are left paused for long periods of time can prevent garbage collection. Use with the [`gc_protect_expires_after`](#gc-protect-expires-after) option to set a limit for protected data and for how long a changefeed will remain paused. diff --git a/src/current/v24.1/how-does-an-enterprise-changefeed-work.md b/src/current/v24.1/how-does-an-enterprise-changefeed-work.md index 06eec389fa2..f2ade9c3535 100644 --- a/src/current/v24.1/how-does-an-enterprise-changefeed-work.md +++ b/src/current/v24.1/how-does-an-enterprise-changefeed-work.md @@ -7,7 +7,7 @@ docs_area: stream_data When an {{ site.data.products.enterprise }} changefeed is started on a node, that node becomes the _coordinator_ for the changefeed job (**Node 2** in the diagram). The coordinator node acts as an administrator: keeping track of all other nodes during job execution and the changefeed work as it completes. The changefeed job will run across nodes in the cluster to access changed data in the watched table. The job will evenly distribute changefeed work across the cluster by assigning it to any [replica]({% link {{ page.version.version }}/architecture/replication-layer.md %}) for a particular range, which determines the node that will emit the changefeed data. If a [locality filter]({% link {{ page.version.version }}/changefeeds-in-multi-region-deployments.md %}#run-a-changefeed-job-by-locality) is specified, work is distributed to a node from those that match the locality filter and has the most locality tiers in common with a node that has a replica. -Each node uses its aggregator processors to send back checkpoint progress to the coordinator, which gathers this information to update the high-water mark timestamp. The high-water mark acts as a checkpoint for the changefeed’s job progress, and guarantees that all changes before (or at) the timestamp have been emitted. In the unlikely event that the changefeed’s coordinating node were to fail during the job, that role will move to a different node and the changefeed will restart from the last checkpoint. If restarted, the changefeed may [re-emit messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) starting at the high-water mark time to the current time. Refer to [Ordering Guarantees]({% link {{ page.version.version }}/changefeed-messages.md %}#ordering-and-delivery-guarantees) for detail on CockroachDB's at-least-once-delivery-guarantee and how per-key message ordering is applied. +Each node uses its _aggregator processors_ to send back checkpoint progress to the coordinator, which gathers this information to update the _high-water mark timestamp_. The high-water mark acts as a checkpoint for the changefeed’s job progress, and guarantees that all changes before (or at) the timestamp have been emitted. In the unlikely event that the changefeed’s coordinating node were to fail during the job, that role will move to a different node and the changefeed will restart from the last checkpoint. If restarted, the changefeed may [re-emit messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) starting at the high-water mark time to the current time. Refer to [Ordering Guarantees]({% link {{ page.version.version }}/changefeed-messages.md %}#ordering-and-delivery-guarantees) for detail on CockroachDB's at-least-once-delivery-guarantee and how per-key message ordering is applied.
diff --git a/src/current/v24.1/monitor-and-debug-changefeeds.md b/src/current/v24.1/monitor-and-debug-changefeeds.md
index 44e0de3aab3..caf985a5b9b 100644
--- a/src/current/v24.1/monitor-and-debug-changefeeds.md
+++ b/src/current/v24.1/monitor-and-debug-changefeeds.md
@@ -153,6 +153,7 @@ changefeed_emitted_bytes{scope="vehicles"} 183557
`changefeed.message_size_hist` | Distribution in the size of emitted messages. | Bytes | Histogram
`changefeed.running` | Number of currently running changefeeds, including sinkless changefeeds. | Changefeeds | Gauge
`changefeed.sink_batch_hist_nanos` | Time messages spend batched in the sink buffer before being flushed and acknowledged. | Nanoseconds | Histogram
+New in v24.1.6: `changefeed.total_ranges` | Total number of ranges that are watched by [aggregator processors]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}) participating in the changefeed job. `changefeed.total_ranges` shares the same polling interval as the [`changefeed.lagging_ranges`](#lagging-ranges-metric) metric, which is controlled by the `lagging_ranges_polling_interval` option. For more details, refer to [Lagging ranges](#lagging-ranges).
### Monitoring and measuring changefeed latency
diff --git a/src/current/v24.1/operational-faqs.md b/src/current/v24.1/operational-faqs.md
index d482b34322d..d132ffcded5 100644
--- a/src/current/v24.1/operational-faqs.md
+++ b/src/current/v24.1/operational-faqs.md
@@ -81,9 +81,13 @@ When MVCC garbage is deleted by garbage collection, the data is still not yet ph
{% include {{page.version.version}}/storage/free-up-disk-space.md %}
-## How can I free up disk space quickly?
+## How can I free up disk space when dropping a table?
-If you've noticed that [your disk space is not freeing up quickly enough after deleting data](#why-is-my-disk-usage-not-decreasing-after-deleting-data), you can take the following steps to free up disk space more quickly. This example assumes a table `t`.
+If you've noticed that [your disk space is not freeing up quickly enough after dropping a table](#why-is-my-disk-usage-not-decreasing-after-deleting-data), you can take the following steps to free up disk space more quickly the next time you drop a table. This example assumes a table `t` exists.
+
+{{site.data.alerts.callout_info}}
+The procedure shown here only works if you get the range IDs from the table **before** running [`DROP TABLE`]({% link {{ page.version.version }}/drop-table.md %}). If you are in an emergency situation due to running out of disk, see [What happens when a node runs out of disk space?](#what-happens-when-a-node-runs-out-of-disk-space)
+{{site.data.alerts.end}}
1. Lower the [`gc.ttlseconds` parameter]({% link {{ page.version.version }}/configure-replication-zones.md %}#gc-ttlseconds) to 10 minutes.
diff --git a/src/current/v24.1/transactions.md b/src/current/v24.1/transactions.md
index 81dd2e35914..a3d1fcd091c 100644
--- a/src/current/v24.1/transactions.md
+++ b/src/current/v24.1/transactions.md
@@ -212,9 +212,11 @@ CockroachDB uses slightly different isolation levels than [ANSI SQL isolation le
#### Aliases
-`SNAPSHOT`, `READ UNCOMMITTED`, `READ COMMITTED`, and `REPEATABLE READ` are aliases for `SERIALIZABLE`.
+`SNAPSHOT` and `REPEATABLE READ` are aliases for `SERIALIZABLE`.
-If [`READ COMMITTED` isolation is enabled]({% link {{ page.version.version }}/read-committed.md %}#enable-read-committed-isolation) using the `sql.txn.read_committed_isolation.enabled` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}), `READ COMMITTED` is no longer an alias for `SERIALIZABLE`, and `READ UNCOMMITTED` becomes an alias for `READ COMMITTED`.
+`READ UNCOMMITTED` is an alias for `READ COMMITTED`.
+
+If `READ COMMITTED` isolation is disabled using the `sql.txn.read_committed_isolation.enabled` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}#setting-sql-txn-read-committed-isolation-enabled), `READ COMMITTED` and `READ UNCOMMITTED` become aliases for `SERIALIZABLE`.
#### Comparison
diff --git a/src/current/v24.2/begin-transaction.md b/src/current/v24.2/begin-transaction.md
index 8c159dac361..6c50b135e95 100644
--- a/src/current/v24.2/begin-transaction.md
+++ b/src/current/v24.2/begin-transaction.md
@@ -7,15 +7,14 @@ docs_area: reference.sql
The `BEGIN` [statement]({% link {{ page.version.version }}/sql-statements.md %}) initiates a [transaction]({% link {{ page.version.version }}/transactions.md %}), which either successfully executes all of the statements it contains or none at all.
-{{site.data.alerts.callout_danger}}
-When using transactions, your application should include logic to [retry transactions]({% link {{ page.version.version }}/transactions.md %}#transaction-retries) that are aborted to break a dependency cycle between concurrent transactions.
+{{site.data.alerts.callout_info}}
+When running under the default [`SERIALIZABLE`]({% link {{ page.version.version }}/demo-serializable.md %}) isolation level, your application should [use a retry loop to handle transaction retry errors]({% link {{ page.version.version }}/query-behavior-troubleshooting.md %}#transaction-retry-errors) that can occur under [contention]({{ link_prefix }}performance-best-practices-overview.html#transaction-contention).
{{site.data.alerts.end}}
-
## Synopsis
-{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/begin.html %}
+{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/legacy_begin.html %}
## Required privileges
diff --git a/src/current/v24.2/create-and-configure-changefeeds.md b/src/current/v24.2/create-and-configure-changefeeds.md
index 201c023e128..f61991da802 100644
--- a/src/current/v24.2/create-and-configure-changefeeds.md
+++ b/src/current/v24.2/create-and-configure-changefeeds.md
@@ -17,6 +17,7 @@ This page describes:
1. Enable rangefeeds on CockroachDB {{ site.data.products.advanced }} and CockroachDB {{ site.data.products.core }}. Refer to [Enable rangefeeds](#enable-rangefeeds) for instructions.
1. Decide on whether you will run an {{ site.data.products.enterprise }} or basic changefeed. Refer to the [Overview]({% link {{ page.version.version }}/change-data-capture-overview.md %}) page for a comparative capability table.
1. Plan the number of changefeeds versus the number of tables to include in a single changefeed for your cluster. {% include {{ page.version.version }}/cdc/changefeed-number-limit.md %} Refer to [System resources and running changefeeds](#system-resources-and-running-changefeeds) and [Recommendations for the number of target tables](#recommendations-for-the-number-of-target-tables).
+ - {% include common/cdc-cloud-costs-link.md %}
1. Consider whether your {{ site.data.products.enterprise }} [changefeed use case](#create) would be better served by [change data capture queries]({% link {{ page.version.version }}/cdc-queries.md %}) that can filter data on a single table. CDC queries can improve the efficiency of changefeeds because the job will not need to encode as much change data.
1. Read the [Considerations](#considerations) section that provides information on changefeed interactions that could affect how you configure or run your changefeed.
@@ -68,7 +69,7 @@ To maintain a high number of changefeeds in your cluster:
### Considerations
- If you require [`resolved`]({% link {{ page.version.version }}/create-changefeed.md %}#resolved) message frequency under `30s`, then you **must** set the [`min_checkpoint_frequency`]({% link {{ page.version.version }}/create-changefeed.md %}#min-checkpoint-frequency) option to at least the desired `resolved` frequency.
-- Many DDL queries (including [`TRUNCATE`]({% link {{ page.version.version }}/truncate.md %}), [`DROP TABLE`]({% link {{ page.version.version }}/drop-table.md %}), and queries that add a column family) will cause errors on a changefeed watching the affected tables. You will need to [start a new changefeed]({% link {{ page.version.version }}/create-changefeed.md %}#start-a-new-changefeed-where-another-ended). If a table is truncated that a changefeed with `on_error='pause'` is watching, you will also need to start a new changefeed. See change data capture [Known Limitations]({% link {{ page.version.version }}/change-data-capture-overview.md %}) for more detail.
+- Many DDL queries (including [`TRUNCATE`]({% link {{ page.version.version }}/truncate.md %}), [`DROP TABLE`]({% link {{ page.version.version }}/drop-table.md %}), and queries that add a column family) will cause errors on a changefeed watching the affected tables. You will need to [start a new changefeed]({% link {{ page.version.version }}/create-changefeed.md %}#start-a-new-changefeed-where-another-ended). If a table is truncated that a changefeed with `on_error='pause'` is watching, you will also need to start a new changefeed. Refer to the change data capture [Known Limitations](#known-limitations) for more detail.
- Partial or intermittent sink unavailability may impact changefeed stability. If a sink is unavailable, messages can't send, which means that a changefeed's high-water mark timestamp is at risk of falling behind the cluster's [garbage collection window]({% link {{ page.version.version }}/configure-replication-zones.md %}#replication-zone-variables). Throughput and latency can be affected once the sink is available again. However, [ordering guarantees]({% link {{ page.version.version }}/changefeed-messages.md %}#ordering-and-delivery-guarantees) will still hold for as long as a changefeed [remains active]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#monitor-a-changefeed).
- When an [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}) statement is run, any current changefeed jobs targeting that table will fail.
- After you [restore from a full-cluster backup]({% link {{ page.version.version }}/restore.md %}#full-cluster), changefeed jobs will **not** resume on the new cluster. It is necessary to manually create the changefeeds following the full-cluster restore.
@@ -186,7 +187,6 @@ For more information, see [`EXPERIMENTAL CHANGEFEED FOR`]({% link {{ page.versio
## Known limitations
{% include {{ page.version.version }}/known-limitations/cdc.md %}
-- {% include {{ page.version.version }}/known-limitations/pcr-scheduled-changefeeds.md %}
- {% include {{ page.version.version }}/known-limitations/alter-changefeed-cdc-queries.md %}
- {% include {{ page.version.version }}/known-limitations/cdc-queries-column-families.md %}
- {% include {{ page.version.version }}/known-limitations/changefeed-column-family-message.md %}
diff --git a/src/current/v24.2/create-changefeed.md b/src/current/v24.2/create-changefeed.md
index 2b9068356dc..b5f69549a9e 100644
--- a/src/current/v24.2/create-changefeed.md
+++ b/src/current/v24.2/create-changefeed.md
@@ -134,7 +134,7 @@ Option | Value | Description
`lagging_ranges_threshold` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Set a duration from the present that determines the length of time a range is considered to be lagging behind, which will then track in the [`lagging_ranges`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#lagging-ranges-metric) metric. Note that ranges undergoing an [initial scan](#initial-scan) for longer than the threshold duration are considered to be lagging. Starting a changefeed with an initial scan on a large table will likely increment the metric for each range in the table. As ranges complete the initial scan, the number of ranges lagging behind will decrease.**Default:** `3m` `lagging_ranges_polling_interval` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Set the interval rate for when lagging ranges are checked and the `lagging_ranges` metric is updated. Polling adds latency to the `lagging_ranges` metric being updated. For example, if a range falls behind by 3 minutes, the metric may not update until an additional minute afterward.
**Default:** `1m` `metrics_label` | [`STRING`]({% link {{ page.version.version }}/string.md %}) | Define a metrics label to which the metrics for one or multiple changefeeds increment. All changefeeds also have their metrics aggregated.
The maximum length of a label is 128 bytes. There is a limit of 1024 unique labels.
`WITH metrics_label=label_name`
For more detail on usage and considerations, see [Using changefeed metrics labels]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#using-changefeed-metrics-labels). -`min_checkpoint_frequency` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Controls how often nodes flush their progress to the [coordinating changefeed node]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}). Changefeeds will wait for at least the specified duration before a flush to the sink. This can help you control the flush frequency of higher latency sinks to achieve better throughput. However, more frequent checkpointing can increase CPU usage. If this is set to `0s`, a node will flush messages as long as the high-water mark has increased for the ranges that particular node is processing. If a changefeed is resumed, then `min_checkpoint_frequency` is the amount of time that changefeed will need to catch up. That is, it could emit [duplicate messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) during this time.
**Note:** [`resolved`](#resolved) messages will not be emitted more frequently than the configured `min_checkpoint_frequency` (but may be emitted less frequently). If you require `resolved` messages more frequently than `30s`, you must configure `min_checkpoint_frequency` to at least the desired `resolved` message frequency. For more details, refer to [Resolved message frequency]({% link {{ page.version.version }}/changefeed-messages.md %}#resolved-timestamp-frequency).
**Default:** `30s` +`min_checkpoint_frequency` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Controls how often a node's changefeed [aggregator]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}) will flush their progress to the [coordinating changefeed node]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}). A node's changefeed aggregator will wait at least the specified duration between sending progress updates for the ranges it is watching to the coordinator. This can help you control the flush frequency of higher latency sinks to achieve better throughput. However, more frequent checkpointing can increase CPU usage. If this is set to `0s`, a node will flush messages as long as the high-water mark has increased for the ranges that particular node is processing. If a changefeed is resumed, then `min_checkpoint_frequency` is the amount of time that changefeed will need to catch up. That is, it could emit [duplicate messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) during this time.
**Note:** [`resolved`](#resolved) messages will not be emitted more frequently than the configured `min_checkpoint_frequency` (but may be emitted less frequently). If you require `resolved` messages more frequently than `30s`, you must configure `min_checkpoint_frequency` to at least the desired `resolved` message frequency. For more details, refer to [Resolved message frequency]({% link {{ page.version.version }}/changefeed-messages.md %}#resolved-timestamp-frequency).
**Default:** `30s` `mvcc_timestamp` | N/A | Include the [MVCC]({% link {{ page.version.version }}/architecture/storage-layer.md %}#mvcc) timestamp for each emitted row in a changefeed. With the `mvcc_timestamp` option, each emitted row will always contain its MVCC timestamp, even during the changefeed's initial backfill. `on_error` | `pause` / `fail` | Use `on_error=pause` to pause the changefeed when encountering **non**-retryable errors. `on_error=pause` will pause the changefeed instead of sending it into a terminal failure state. **Note:** Retryable errors will continue to be retried with this option specified.
Use with [`protect_data_from_gc_on_pause`](#protect-data-from-gc-on-pause) to protect changes from [garbage collection]({% link {{ page.version.version }}/configure-replication-zones.md %}#gc-ttlseconds).
If a changefeed with `on_error=pause` is running when a watched table is [truncated]({% link {{ page.version.version }}/truncate.md %}), the changefeed will pause but will not be able to resume reads from that table. Using [`ALTER CHANGEFEED`]({% link {{ page.version.version }}/alter-changefeed.md %}) to drop the table from the changefeed and then [resuming the job]({% link {{ page.version.version }}/resume-job.md %}) will work, but you cannot add the same table to the changefeed again. Instead, you will need to [create a new changefeed](#start-a-new-changefeed-where-another-ended) for that table.
Default: `on_error=fail` `protect_data_from_gc_on_pause` | N/A | This option is deprecated as of v23.2 and will be removed in a future release.
When a [changefeed is paused]({% link {{ page.version.version }}/pause-job.md %}), ensure that the data needed to [resume the changefeed]({% link {{ page.version.version }}/resume-job.md %}) is not garbage collected. If `protect_data_from_gc_on_pause` is **unset**, pausing the changefeed will release the existing protected timestamp records. It is also important to note that pausing and adding `protect_data_from_gc_on_pause` to a changefeed will not protect data if the [garbage collection]({% link {{ page.version.version }}/configure-replication-zones.md %}#gc-ttlseconds) window has already passed.
Use with [`on_error=pause`](#on-error) to protect changes from garbage collection when encountering non-retryable errors.
Refer to [Protect Changefeed Data from Garbage Collection]({% link {{ page.version.version }}/protect-changefeed-data.md %}) for more detail on protecting changefeed data.
**Note:** If you use this option, changefeeds that are left paused for long periods of time can prevent garbage collection. Use with the [`gc_protect_expires_after`](#gc-protect-expires-after) option to set a limit for protected data and for how long a changefeed will remain paused. diff --git a/src/current/v24.2/create-schedule-for-changefeed.md b/src/current/v24.2/create-schedule-for-changefeed.md index 6902e2cc135..6c3730172dc 100644 --- a/src/current/v24.2/create-schedule-for-changefeed.md +++ b/src/current/v24.2/create-schedule-for-changefeed.md @@ -58,6 +58,10 @@ Option | Value | Description `on_execution_failure` | `retry` / `reschedule` / `pause` | Determine how the schedule handles an error.
`retry`: Retry the changefeed immediately.
`reschedule`: Reschedule the changefeed based on the `RECURRING` expression.
`pause`: Pause the schedule. This requires that you [resume the schedule]({% link {{ page.version.version }}/resume-schedules.md %}) manually.
**Default:** `reschedule` `on_previous_running` | `start` / `skip` / `wait` | Control whether the changefeed schedule should start a changefeed if the previous scheduled changefeed is still running.
`start`: Start the new changefeed anyway, even if the previous one is running.
`skip`: Skip the new changefeed and run the next changefeed based on the `RECURRING` expression.
`wait`: Wait for the previous changefeed to complete.
**Default:** `wait` +{{site.data.alerts.callout_info}} +{% include_cached new-in.html version="v24.2" %} To avoid multiple clusters running the same schedule concurrently, changefeed schedules will [pause]({% link {{ page.version.version }}/pause-schedules.md %}) when [restored]({% link {{ page.version.version }}/restore.md %}) onto a different cluster or after [physical cluster replication]({% link {{ page.version.version }}/cutover-replication.md %}) has completed. +{{site.data.alerts.end}} + ## Examples Before running any of the examples in this section, it is necessary to enable the `kv.rangefeed.enabled` cluster setting. If you are working on a CockroachDB {{ site.data.products.standard }} or {{ site.data.products.basic }} cluster, this cluster setting is enabled by default. diff --git a/src/current/v24.2/cutover-replication.md b/src/current/v24.2/cutover-replication.md index 08a6b44e058..a70e693bf17 100644 --- a/src/current/v24.2/cutover-replication.md +++ b/src/current/v24.2/cutover-replication.md @@ -167,17 +167,21 @@ During a replication stream, jobs running on the primary cluster will replicate [Backup schedules]({% link {{ page.version.version }}/manage-a-backup-schedule.md %}) will pause after cutover on the promoted cluster. Take the following steps to resume jobs: 1. Verify that there are no other schedules running backups to the same [collection of backups]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}#backup-collections), i.e., the schedule that was running on the original primary cluster. -1. Resume the backup schedule on the promoted cluster. +1. [Resume]({% link {{ page.version.version }}/resume-schedules.md %}) the backup schedule on the promoted cluster. {{site.data.alerts.callout_info}} -If your backup schedule was created on a cluster in v23.1 or earlier, it will **not** pause automatically on the promoted cluster after cutover. In this case, you must pause the schedule manually on the promoted cluster and then take the outlined steps. +If your backup schedule was created on a cluster in v23.1 or earlier, it will **not** pause automatically on the promoted cluster after cutover. In this case, you must [pause]({% link {{ page.version.version }}/pause-schedules.md %}) the schedule manually on the promoted cluster and then take the outlined steps. {{site.data.alerts.end}} ### Changefeeds [Changefeeds]({% link {{ page.version.version }}/change-data-capture-overview.md %}) will fail on the promoted cluster immediately after cutover to avoid two clusters running the same changefeed to one sink. We recommend that you recreate changefeeds on the promoted cluster. -[Scheduled changefeeds]({% link {{ page.version.version }}/create-schedule-for-changefeed.md %}) will continue on the promoted cluster. You will need to manage [pausing]({% link {{ page.version.version }}/pause-schedules.md %}) or [canceling]({% link {{ page.version.version }}/drop-schedules.md %}) the schedule on the promoted standby cluster to avoid two clusters running the same changefeed to one sink. +{% include_cached new-in.html version="v24.2" %} To avoid multiple clusters running the same schedule concurrently, [changefeed schedules]({% link {{ page.version.version }}/create-schedule-for-changefeed.md %}) will [pause]({% link {{ page.version.version }}/pause-schedules.md %}) after physical cluster replication has completed. + +{{site.data.alerts.callout_info}} +If your changefeed schedule was created on a cluster in v24.1 or earlier, it will **not** pause automatically on the promoted cluster after cutover. In this case, you will need to manage [pausing]({% link {{ page.version.version }}/pause-schedules.md %}) or [canceling]({% link {{ page.version.version }}/drop-schedules.md %}) the schedule on the promoted standby cluster to avoid two clusters running the same changefeed to one sink. +{{site.data.alerts.end}} ## Cut back to the primary cluster diff --git a/src/current/v24.2/how-does-an-enterprise-changefeed-work.md b/src/current/v24.2/how-does-an-enterprise-changefeed-work.md index 43fb7ea3fe0..566f6a15e48 100644 --- a/src/current/v24.2/how-does-an-enterprise-changefeed-work.md +++ b/src/current/v24.2/how-does-an-enterprise-changefeed-work.md @@ -7,7 +7,7 @@ docs_area: stream_data When an {{ site.data.products.enterprise }} changefeed is started on a node, that node becomes the _coordinator_ for the changefeed job (**Node 2** in the diagram). The coordinator node acts as an administrator: keeping track of all other nodes during job execution and the changefeed work as it completes. The changefeed job will run across nodes in the cluster to access changed data in the watched table. The job will evenly distribute changefeed work across the cluster by assigning it to any [replica]({% link {{ page.version.version }}/architecture/replication-layer.md %}) for a particular range, which determines the node that will emit the changefeed data. If a [locality filter]({% link {{ page.version.version }}/changefeeds-in-multi-region-deployments.md %}#run-a-changefeed-job-by-locality) is specified, work is distributed to a node from those that match the locality filter and has the most locality tiers in common with a node that has a replica. -Each node uses its aggregator processors to send back checkpoint progress to the coordinator, which gathers this information to update the high-water mark timestamp. The high-water mark acts as a checkpoint for the changefeed’s job progress, and guarantees that all changes before (or at) the timestamp have been emitted. In the unlikely event that the changefeed’s coordinating node were to fail during the job, that role will move to a different node and the changefeed will restart from the last checkpoint. If restarted, the changefeed may [re-emit messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) starting at the high-water mark time to the current time. Refer to [Ordering Guarantees]({% link {{ page.version.version }}/changefeed-messages.md %}#ordering-and-delivery-guarantees) for detail on CockroachDB's at-least-once-delivery-guarantee and how per-key message ordering is applied. +Each node uses its _aggregator processors_ to send back checkpoint progress to the coordinator, which gathers this information to update the _high-water mark timestamp_. The high-water mark acts as a checkpoint for the changefeed’s job progress, and guarantees that all changes before (or at) the timestamp have been emitted. In the unlikely event that the changefeed’s coordinating node were to fail during the job, that role will move to a different node and the changefeed will restart from the last checkpoint. If restarted, the changefeed may [re-emit messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) starting at the high-water mark time to the current time. Refer to [Ordering Guarantees]({% link {{ page.version.version }}/changefeed-messages.md %}#ordering-and-delivery-guarantees) for detail on CockroachDB's at-least-once-delivery-guarantee and how per-key message ordering is applied.
diff --git a/src/current/v24.2/known-limitations.md b/src/current/v24.2/known-limitations.md
index d7dccc618a5..b6c00e689f2 100644
--- a/src/current/v24.2/known-limitations.md
+++ b/src/current/v24.2/known-limitations.md
@@ -454,7 +454,6 @@ Accessing the DB Console for a secure cluster now requires login information (i.
#### Physical cluster replication
{% include {{ page.version.version }}/known-limitations/physical-cluster-replication.md %}
-- {% include {{ page.version.version }}/known-limitations/pcr-scheduled-changefeeds.md %}
- {% include {{ page.version.version }}/known-limitations/cutover-stop-application.md %}
#### `RESTORE` limitations
@@ -478,7 +477,6 @@ As a workaround, take a cluster backup instead, as the `system.comments` table i
Change data capture (CDC) provides efficient, distributed, row-level changefeeds into Apache Kafka for downstream processing such as reporting, caching, or full-text indexing. It has the following known limitations:
{% include {{ page.version.version }}/known-limitations/cdc.md %}
-- {% include {{ page.version.version }}/known-limitations/pcr-scheduled-changefeeds.md %}
{% include {{ page.version.version }}/known-limitations/cdc-queries.md %}
- {% include {{ page.version.version }}/known-limitations/cdc-queries-column-families.md %}
- {% include {{ page.version.version }}/known-limitations/changefeed-column-family-message.md %}
diff --git a/src/current/v24.2/monitor-and-debug-changefeeds.md b/src/current/v24.2/monitor-and-debug-changefeeds.md
index eb17ffe4f72..64d58d7174a 100644
--- a/src/current/v24.2/monitor-and-debug-changefeeds.md
+++ b/src/current/v24.2/monitor-and-debug-changefeeds.md
@@ -153,6 +153,7 @@ changefeed_emitted_bytes{scope="vehicles"} 183557
`changefeed.message_size_hist` | Distribution in the size of emitted messages. | Bytes | Histogram
`changefeed.running` | Number of currently running changefeeds, including sinkless changefeeds. | Changefeeds | Gauge
`changefeed.sink_batch_hist_nanos` | Time messages spend batched in the sink buffer before being flushed and acknowledged. | Nanoseconds | Histogram
+New in v24.2.4: `changefeed.total_ranges` | Total number of ranges that are watched by [aggregator processors]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}) participating in the changefeed job. `changefeed.total_ranges` shares the same polling interval as the [`changefeed.lagging_ranges`](#lagging-ranges-metric) metric, which is controlled by the `lagging_ranges_polling_interval` option. For more details, refer to [Lagging ranges](#lagging-ranges).
### Monitoring and measuring changefeed latency
diff --git a/src/current/v24.2/operational-faqs.md b/src/current/v24.2/operational-faqs.md
index 10c04d37472..7a073b9c0dc 100644
--- a/src/current/v24.2/operational-faqs.md
+++ b/src/current/v24.2/operational-faqs.md
@@ -81,9 +81,13 @@ When MVCC garbage is deleted by garbage collection, the data is still not yet ph
{% include {{page.version.version}}/storage/free-up-disk-space.md %}
-## How can I free up disk space quickly?
+## How can I free up disk space when dropping a table?
-If you've noticed that [your disk space is not freeing up quickly enough after deleting data](#why-is-my-disk-usage-not-decreasing-after-deleting-data), you can take the following steps to free up disk space more quickly. This example assumes a table `t`.
+If you've noticed that [your disk space is not freeing up quickly enough after dropping a table](#why-is-my-disk-usage-not-decreasing-after-deleting-data), you can take the following steps to free up disk space more quickly the next time you drop a table. This example assumes a table `t` exists.
+
+{{site.data.alerts.callout_info}}
+The procedure shown here only works if you get the range IDs from the table **before** running [`DROP TABLE`]({% link {{ page.version.version }}/drop-table.md %}). If you are in an emergency situation due to running out of disk, see [What happens when a node runs out of disk space?](#what-happens-when-a-node-runs-out-of-disk-space)
+{{site.data.alerts.end}}
1. Lower the [`gc.ttlseconds` parameter]({% link {{ page.version.version }}/configure-replication-zones.md %}#gc-ttlseconds) to 10 minutes.
diff --git a/src/current/v24.2/physical-cluster-replication-overview.md b/src/current/v24.2/physical-cluster-replication-overview.md
index c0b699faf1d..41259a7b312 100644
--- a/src/current/v24.2/physical-cluster-replication-overview.md
+++ b/src/current/v24.2/physical-cluster-replication-overview.md
@@ -35,7 +35,6 @@ You can use PCR in a disaster recovery plan to:
## Known limitations
{% include {{ page.version.version }}/known-limitations/physical-cluster-replication.md %}
-- {% include {{ page.version.version }}/known-limitations/pcr-scheduled-changefeeds.md %}
- {% include {{ page.version.version }}/known-limitations/cutover-stop-application.md %}
## Performance
diff --git a/src/current/v24.2/transactions.md b/src/current/v24.2/transactions.md
index 81dd2e35914..a3d1fcd091c 100644
--- a/src/current/v24.2/transactions.md
+++ b/src/current/v24.2/transactions.md
@@ -212,9 +212,11 @@ CockroachDB uses slightly different isolation levels than [ANSI SQL isolation le
#### Aliases
-`SNAPSHOT`, `READ UNCOMMITTED`, `READ COMMITTED`, and `REPEATABLE READ` are aliases for `SERIALIZABLE`.
+`SNAPSHOT` and `REPEATABLE READ` are aliases for `SERIALIZABLE`.
-If [`READ COMMITTED` isolation is enabled]({% link {{ page.version.version }}/read-committed.md %}#enable-read-committed-isolation) using the `sql.txn.read_committed_isolation.enabled` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}), `READ COMMITTED` is no longer an alias for `SERIALIZABLE`, and `READ UNCOMMITTED` becomes an alias for `READ COMMITTED`.
+`READ UNCOMMITTED` is an alias for `READ COMMITTED`.
+
+If `READ COMMITTED` isolation is disabled using the `sql.txn.read_committed_isolation.enabled` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}#setting-sql-txn-read-committed-isolation-enabled), `READ COMMITTED` and `READ UNCOMMITTED` become aliases for `SERIALIZABLE`.
#### Comparison
diff --git a/src/current/v24.3/backup.md b/src/current/v24.3/backup.md
index b5b6f8cceef..7d5b004a881 100644
--- a/src/current/v24.3/backup.md
+++ b/src/current/v24.3/backup.md
@@ -28,7 +28,7 @@ To view the contents of an backup created with the `BACKUP` statement, use [`SHO
{% include {{ page.version.version }}/backups/scheduled-backups-tip.md %}
-{% include {{ page.version.version }}/backups/backup-to-deprec.md %}
+{% include {{ page.version.version }}/backups/old-syntax-removed.md %}
## Considerations
@@ -236,11 +236,7 @@ Per our guidance in the [Performance](#performance) section, we recommend starti
If you need to limit the control specific users have over your storage buckets, see [Assume role authentication]({% link {{ page.version.version }}/cloud-storage-authentication.md %}) for setup instructions.
-{{site.data.alerts.callout_info}}
-The `BACKUP ... TO` syntax is **deprecated** as of v22.1 and will be removed in a future release.
-
-Cockroach Labs recommends using the `BACKUP ... INTO {collectionURI}` syntax shown in the following examples.
-{{site.data.alerts.end}}
+{% include {{ page.version.version }}/backups/old-syntax-removed.md %}
### Back up a cluster
diff --git a/src/current/v24.3/begin-transaction.md b/src/current/v24.3/begin-transaction.md
index 8c159dac361..6c50b135e95 100644
--- a/src/current/v24.3/begin-transaction.md
+++ b/src/current/v24.3/begin-transaction.md
@@ -7,15 +7,14 @@ docs_area: reference.sql
The `BEGIN` [statement]({% link {{ page.version.version }}/sql-statements.md %}) initiates a [transaction]({% link {{ page.version.version }}/transactions.md %}), which either successfully executes all of the statements it contains or none at all.
-{{site.data.alerts.callout_danger}}
-When using transactions, your application should include logic to [retry transactions]({% link {{ page.version.version }}/transactions.md %}#transaction-retries) that are aborted to break a dependency cycle between concurrent transactions.
+{{site.data.alerts.callout_info}}
+When running under the default [`SERIALIZABLE`]({% link {{ page.version.version }}/demo-serializable.md %}) isolation level, your application should [use a retry loop to handle transaction retry errors]({% link {{ page.version.version }}/query-behavior-troubleshooting.md %}#transaction-retry-errors) that can occur under [contention]({{ link_prefix }}performance-best-practices-overview.html#transaction-contention).
{{site.data.alerts.end}}
-
## Synopsis
-{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/begin.html %}
+{% remote_include https://raw.githubusercontent.com/cockroachdb/generated-diagrams/{{ page.release_info.crdb_branch_name }}/grammar_svg/legacy_begin.html %}
## Required privileges
diff --git a/src/current/v24.3/change-data-capture-overview.md b/src/current/v24.3/change-data-capture-overview.md
index 7997456e941..64fa3ea148b 100644
--- a/src/current/v24.3/change-data-capture-overview.md
+++ b/src/current/v24.3/change-data-capture-overview.md
@@ -15,7 +15,7 @@ For example, you might want to:
- Export a snaphot of tables to backfill new applications.
- Send updates to data stores for machine learning models.
-The main feature of CockroachDB CDC is the _changefeed_, which targets an allowlist of tables, or "watched tables".
+{% include common/define-watched-cdc.md %}
## Stream row-level changes with changefeeds
diff --git a/src/current/v24.3/changefeed-best-practices.md b/src/current/v24.3/changefeed-best-practices.md
index 391018ffabe..3e61a629a48 100644
--- a/src/current/v24.3/changefeed-best-practices.md
+++ b/src/current/v24.3/changefeed-best-practices.md
@@ -6,6 +6,10 @@ toc: true
This page describes best practices to consider when starting changefeeds on a CockroachDB cluster. We recommend referring to this information while planning your cluster's changefeeds and following the links in each of the sections for more details on a topic.
+{{site.data.alerts.callout_info}}
+To help in planning your cluster's changefeeds on CockroachDB {{ site.data.products.cloud }} clusters, refer to the [Understand CockroachDB Cloud Costs]({% link cockroachcloud/costs.md %}) page for detail on how CDC is billed monthly based on usage.
+{{site.data.alerts.end}}
+
## Plan the number of watched tables for a single changefeed
When creating a changefeed, it's important to consider the number of changefeeds versus the number of tables to include in a single changefeed:
diff --git a/src/current/v24.3/child-metrics.md b/src/current/v24.3/child-metrics.md
index 3618ec7d5ad..8fee7bd1a6a 100644
--- a/src/current/v24.3/child-metrics.md
+++ b/src/current/v24.3/child-metrics.md
@@ -110,6 +110,41 @@ changefeed_error_retries{node_id="1",scope="office_dogs"} 0
{% assign feature = "changefeed" %}
{% include {{ page.version.version }}/child-metrics-table.md %}
+## Clusters with logical data replication jobs
+
+When child metrics is enabled and [logical data replication (LDR) jobs with metrics labels]({% link {{ page.version.version }}/logical-data-replication-monitoring.md %}#metrics-labels) are created on the cluster, the `logical_replication_*_by_label` metrics are exported per LDR metric label. The `label` may have the values set using the `label` option. The cardinality increases with the number of LDR metric labels.
+
+For example, when you create two LDR jobs with the metrics labels `ldr_job1` and `ldr_job2`, the metrics `logical_replication_*_by_label` export child metrics with a `label` for `ldr_job1` and `ldr_job2`.
+
+~~~
+# HELP logical_replication_replicated_time_by_label Replicated time of the logical replication stream by label
+# TYPE logical_replication_replicated_time_by_label gauge
+logical_replication_replicated_time_by_label{label="ldr_job2",node_id="2"} 1.73411035e+09
+logical_replication_replicated_time_by_label{label="ldr_job1",node_id="2"} 1.73411035e+09
+# HELP logical_replication_catchup_ranges_by_label Source side ranges undergoing catch up scans
+# TYPE logical_replication_catchup_ranges_by_label gauge
+logical_replication_catchup_ranges_by_label{label="ldr_job1",node_id="2"} 0
+logical_replication_catchup_ranges_by_label{label="ldr_job2",node_id="2"} 0
+# HELP logical_replication_scanning_ranges_by_label Source side ranges undergoing an initial scan
+# TYPE logical_replication_scanning_ranges_by_label gauge
+logical_replication_scanning_ranges_by_label{label="ldr_job1",node_id="2"} 0
+logical_replication_scanning_ranges_by_label{label="ldr_job2",node_id="2"} 0
+~~~
+
+Note that the `logical_replication_*` metrics without the `_by_label` suffix may be `inaccurate with multiple LDR jobs`.
+
+~~~
+# HELP logical_replication_catchup_ranges Source side ranges undergoing catch up scans (inaccurate with multiple LDR jobs)
+# TYPE logical_replication_catchup_ranges gauge
+logical_replication_catchup_ranges{node_id="2"} 0
+# HELP logical_replication_scanning_ranges Source side ranges undergoing an initial scan (inaccurate with multiple LDR jobs)
+# TYPE logical_replication_scanning_ranges gauge
+logical_replication_scanning_ranges{node_id="2"} 0
+~~~
+
+{% assign feature = "ldr" %}
+{% include {{ page.version.version }}/child-metrics-table.md %}
+
## Clusters with row-level TTL jobs
When child metrics is enabled and [row-level TTL jobs]({% link {{ page.version.version }}/row-level-ttl.md %}) are created on the cluster with the [`ttl_label_metrics` storage parameter enabled]({% link {{ page.version.version }}/row-level-ttl.md %}#ttl-metrics), the `jobs.row_level_ttl.*` metrics are exported per TTL job with `ttl_label_metrics` enabled with a label for `relation`. The value of the `relation` label may have the format: `{database}_{schema}_{table}_{primary key}`. The cardinality increases with the number of TTL jobs with `ttl_label_metrics` enabled. An aggregated metric is also included.
diff --git a/src/current/v24.3/create-and-configure-changefeeds.md b/src/current/v24.3/create-and-configure-changefeeds.md
index 494c1653c58..4662b2451ba 100644
--- a/src/current/v24.3/create-and-configure-changefeeds.md
+++ b/src/current/v24.3/create-and-configure-changefeeds.md
@@ -17,6 +17,7 @@ This page describes:
1. Enable rangefeeds on CockroachDB {{ site.data.products.advanced }} and CockroachDB {{ site.data.products.core }}. Refer to [Enable rangefeeds](#enable-rangefeeds) for instructions.
1. Decide on whether you will run an {{ site.data.products.enterprise }} or basic changefeed. Refer to the [Overview]({% link {{ page.version.version }}/change-data-capture-overview.md %}) page for a comparative capability table.
1. Plan the number of changefeeds versus the number of tables to include in a single changefeed for your cluster. {% include {{ page.version.version }}/cdc/changefeed-number-limit.md %} Refer to [System resources and running changefeeds]({% link {{ page.version.version }}/changefeed-best-practices.md %}#maintain-system-resources-and-running-changefeeds) and [Recommendations for the number of target tables]({% link {{ page.version.version }}/changefeed-best-practices.md %}#plan-the-number-of-watched-tables-for-a-single-changefeed).
+ - {% include common/cdc-cloud-costs-link.md %}
1. Consider whether your {{ site.data.products.enterprise }} [changefeed use case](#create) would be better served by [change data capture queries]({% link {{ page.version.version }}/cdc-queries.md %}) that can filter data on a single table. CDC queries can improve the efficiency of changefeeds because the job will not need to encode as much change data.
1. Read the following:
- The [Changefeed Best Practices]({% link {{ page.version.version }}/changefeed-best-practices.md %}) reference for details on planning changefeeds, monitoring basics, and schema changes.
@@ -46,7 +47,7 @@ For further detail on performance-related configuration, refer to the [Advanced
### Considerations
- If you require [`resolved`]({% link {{ page.version.version }}/create-changefeed.md %}#resolved) message frequency under `30s`, then you **must** set the [`min_checkpoint_frequency`]({% link {{ page.version.version }}/create-changefeed.md %}#min-checkpoint-frequency) option to at least the desired `resolved` frequency.
-- Many DDL queries (including [`TRUNCATE`]({% link {{ page.version.version }}/truncate.md %}), [`DROP TABLE`]({% link {{ page.version.version }}/drop-table.md %}), and queries that add a column family) will cause errors on a changefeed watching the affected tables. You will need to [start a new changefeed]({% link {{ page.version.version }}/create-changefeed.md %}#start-a-new-changefeed-where-another-ended). If a table is truncated that a changefeed with `on_error='pause'` is watching, you will also need to start a new changefeed. See change data capture [Known Limitations]({% link {{ page.version.version }}/change-data-capture-overview.md %}) for more detail.
+- Many DDL queries (including [`TRUNCATE`]({% link {{ page.version.version }}/truncate.md %}), [`DROP TABLE`]({% link {{ page.version.version }}/drop-table.md %}), and queries that add a column family) will cause errors on a changefeed watching the affected tables. You will need to [start a new changefeed]({% link {{ page.version.version }}/create-changefeed.md %}#start-a-new-changefeed-where-another-ended). If a table is truncated that a changefeed with `on_error='pause'` is watching, you will also need to start a new changefeed. Refer to the change data capture [Known Limitations](#known-limitations) for more detail.
- Partial or intermittent sink unavailability may impact changefeed stability. If a sink is unavailable, messages can't send, which means that a changefeed's high-water mark timestamp is at risk of falling behind the cluster's [garbage collection window]({% link {{ page.version.version }}/configure-replication-zones.md %}#replication-zone-variables). Throughput and latency can be affected once the sink is available again. However, [ordering guarantees]({% link {{ page.version.version }}/changefeed-messages.md %}#ordering-and-delivery-guarantees) will still hold for as long as a changefeed [remains active]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#monitor-a-changefeed).
- When an [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}) statement is run, any current changefeed jobs targeting that table will fail.
- After you [restore from a full-cluster backup]({% link {{ page.version.version }}/restore.md %}#full-cluster), changefeed jobs will **not** resume on the new cluster. It is necessary to manually create the changefeeds following the full-cluster restore.
@@ -164,7 +165,6 @@ For more information, see [`EXPERIMENTAL CHANGEFEED FOR`]({% link {{ page.versio
## Known limitations
{% include {{ page.version.version }}/known-limitations/cdc.md %}
-- {% include {{ page.version.version }}/known-limitations/pcr-scheduled-changefeeds.md %}
- {% include {{ page.version.version }}/known-limitations/alter-changefeed-cdc-queries.md %}
- {% include {{ page.version.version }}/known-limitations/cdc-queries-column-families.md %}
- {% include {{ page.version.version }}/known-limitations/changefeed-column-family-message.md %}
diff --git a/src/current/v24.3/create-changefeed.md b/src/current/v24.3/create-changefeed.md
index 9a90e244e7c..591f470f445 100644
--- a/src/current/v24.3/create-changefeed.md
+++ b/src/current/v24.3/create-changefeed.md
@@ -134,7 +134,7 @@ Option | Value | Description
`lagging_ranges_threshold` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Set a duration from the present that determines the length of time a range is considered to be lagging behind, which will then track in the [`lagging_ranges`]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#lagging-ranges-metric) metric. Note that ranges undergoing an [initial scan](#initial-scan) for longer than the threshold duration are considered to be lagging. Starting a changefeed with an initial scan on a large table will likely increment the metric for each range in the table. As ranges complete the initial scan, the number of ranges lagging behind will decrease.**Default:** `3m` `lagging_ranges_polling_interval` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Set the interval rate for when lagging ranges are checked and the `lagging_ranges` metric is updated. Polling adds latency to the `lagging_ranges` metric being updated. For example, if a range falls behind by 3 minutes, the metric may not update until an additional minute afterward.
**Default:** `1m` `metrics_label` | [`STRING`]({% link {{ page.version.version }}/string.md %}) | Define a metrics label to which the metrics for one or multiple changefeeds increment. All changefeeds also have their metrics aggregated.
The maximum length of a label is 128 bytes. There is a limit of 1024 unique labels.
`WITH metrics_label=label_name`
For more detail on usage and considerations, see [Using changefeed metrics labels]({% link {{ page.version.version }}/monitor-and-debug-changefeeds.md %}#using-changefeed-metrics-labels). -`min_checkpoint_frequency` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Controls how often nodes flush their progress to the [coordinating changefeed node]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}). Changefeeds will wait for at least the specified duration before a flush to the sink. This can help you control the flush frequency of higher latency sinks to achieve better throughput. However, more frequent checkpointing can increase CPU usage. If this is set to `0s`, a node will flush messages as long as the high-water mark has increased for the ranges that particular node is processing. If a changefeed is resumed, then `min_checkpoint_frequency` is the amount of time that changefeed will need to catch up. That is, it could emit [duplicate messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) during this time.
**Note:** [`resolved`](#resolved) messages will not be emitted more frequently than the configured `min_checkpoint_frequency` (but may be emitted less frequently). If you require `resolved` messages more frequently than `30s`, you must configure `min_checkpoint_frequency` to at least the desired `resolved` message frequency. For more details, refer to [Resolved message frequency]({% link {{ page.version.version }}/changefeed-messages.md %}#resolved-timestamp-frequency).
**Default:** `30s` +`min_checkpoint_frequency` | [Duration string](https://pkg.go.dev/time#ParseDuration) | Controls how often a node's changefeed [aggregator]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}) will flush their progress to the [coordinating changefeed node]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}). A node's changefeed aggregator will wait at least the specified duration between sending progress updates for the ranges it is watching to the coordinator. This can help you control the flush frequency of higher latency sinks to achieve better throughput. However, more frequent checkpointing can increase CPU usage. If this is set to `0s`, a node will flush messages as long as the high-water mark has increased for the ranges that particular node is processing. If a changefeed is resumed, then `min_checkpoint_frequency` is the amount of time that changefeed will need to catch up. That is, it could emit [duplicate messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) during this time.
**Note:** [`resolved`](#resolved) messages will not be emitted more frequently than the configured `min_checkpoint_frequency` (but may be emitted less frequently). If you require `resolved` messages more frequently than `30s`, you must configure `min_checkpoint_frequency` to at least the desired `resolved` message frequency. For more details, refer to [Resolved message frequency]({% link {{ page.version.version }}/changefeed-messages.md %}#resolved-timestamp-frequency).
**Default:** `30s` `mvcc_timestamp` | N/A | Include the [MVCC]({% link {{ page.version.version }}/architecture/storage-layer.md %}#mvcc) timestamp for each emitted row in a changefeed. With the `mvcc_timestamp` option, each emitted row will always contain its MVCC timestamp, even during the changefeed's initial backfill. `on_error` | `pause` / `fail` | Use `on_error=pause` to pause the changefeed when encountering **non**-retryable errors. `on_error=pause` will pause the changefeed instead of sending it into a terminal failure state. **Note:** Retryable errors will continue to be retried with this option specified.
Use with [`protect_data_from_gc_on_pause`](#protect-data-from-gc-on-pause) to protect changes from [garbage collection]({% link {{ page.version.version }}/configure-replication-zones.md %}#gc-ttlseconds).
If a changefeed with `on_error=pause` is running when a watched table is [truncated]({% link {{ page.version.version }}/truncate.md %}), the changefeed will pause but will not be able to resume reads from that table. Using [`ALTER CHANGEFEED`]({% link {{ page.version.version }}/alter-changefeed.md %}) to drop the table from the changefeed and then [resuming the job]({% link {{ page.version.version }}/resume-job.md %}) will work, but you cannot add the same table to the changefeed again. Instead, you will need to [create a new changefeed](#start-a-new-changefeed-where-another-ended) for that table.
Default: `on_error=fail` `protect_data_from_gc_on_pause` | N/A | This option is deprecated as of v23.2 and will be removed in a future release.
When a [changefeed is paused]({% link {{ page.version.version }}/pause-job.md %}), ensure that the data needed to [resume the changefeed]({% link {{ page.version.version }}/resume-job.md %}) is not garbage collected. If `protect_data_from_gc_on_pause` is **unset**, pausing the changefeed will release the existing protected timestamp records. It is also important to note that pausing and adding `protect_data_from_gc_on_pause` to a changefeed will not protect data if the [garbage collection]({% link {{ page.version.version }}/configure-replication-zones.md %}#gc-ttlseconds) window has already passed.
Use with [`on_error=pause`](#on-error) to protect changes from garbage collection when encountering non-retryable errors.
Refer to [Protect Changefeed Data from Garbage Collection]({% link {{ page.version.version }}/protect-changefeed-data.md %}) for more detail on protecting changefeed data.
**Note:** If you use this option, changefeeds that are left paused for long periods of time can prevent garbage collection. Use with the [`gc_protect_expires_after`](#gc-protect-expires-after) option to set a limit for protected data and for how long a changefeed will remain paused. diff --git a/src/current/v24.3/create-logical-replication-stream.md b/src/current/v24.3/create-logical-replication-stream.md index f09afad5917..3e7fb81a8e6 100644 --- a/src/current/v24.3/create-logical-replication-stream.md +++ b/src/current/v24.3/create-logical-replication-stream.md @@ -48,26 +48,26 @@ Parameter | Description Option | Description -------+------------ `cursor` | Emits any changes after the specified timestamp. LDR will not perform an initial backfill with the `cursor` option, it will stream any changes after the specified timestamp. The LDR job will encounter an error if you specify a `cursor` timestamp that is before the configured [garbage collection]({% link {{ page.version.version }}/architecture/storage-layer.md %}#garbage-collection) window for that table. **Warning:** Apply the `cursor` option carefully to LDR streams. Using a timestamp in error could cause data loss. -`discard` | Ignore [TTL deletes]({% link {{ page.version.version }}/row-level-ttl.md %}) in an LDR stream. Use `discard = ttl-deletes`. **Note**: To ignore row-level TTL deletes in an LDR stream, it is necessary to set the [`ttl_disable_changefeed_replication`]({% link {{ page.version.version }}/row-level-ttl.md %}#ttl-storage-parameters) storage parameter on the source table. Refer to the [Ignore row-level TTL deletes](#ignore-row-level-ttl-deletes) example. +`discard` | ([**Unidirectional LDR only**]({% link {{ page.version.version }}/logical-data-replication-overview.md %}#use-cases)) Ignore [TTL deletes]({% link {{ page.version.version }}/row-level-ttl.md %}) in an LDR stream with `discard = ttl-deletes`. **Note**: To ignore row-level TTL deletes in an LDR stream, it is necessary to set the [`ttl_disable_changefeed_replication`]({% link {{ page.version.version }}/row-level-ttl.md %}#ttl-storage-parameters) storage parameter on the source table. Refer to the [Ignore row-level TTL deletes](#ignore-row-level-ttl-deletes) example. `label` | Tracks LDR metrics at the job level. Add a user-specified string with `label`. Refer to [Metrics labels]({% link {{ page.version.version }}/logical-data-replication-monitoring.md %}#metrics-labels). -`mode` | Determines how LDR replicates the data to the destination cluster. Possible values: `immediate`, `validated`. For more details refer to [LDR modes](#ldr-modes). +`mode` | Determines how LDR replicates the data to the destination cluster. Possible values: `immediate`, `validated`. For more details, refer to [LDR modes](#ldr-modes). ## LDR modes _Modes_ determine how LDR replicates the data to the destination cluster. There are two modes: -- `immediate` (default): Attempts to replicate the changed row directly into the destination table, without re-running constraint validations. It does not support writing into tables with [foreign key]({% link {{ page.version.version }}/foreign-key.md %}) constraints. -- `validated`: Attempts to apply the write in a similar way to a user-run query, which would re-run all constraint validations or triggers relevant to the destination table(s). It will potentially reject the change if it fails rather than writing it, which will cause the row change to enter the DLQ instead. +- `immediate` (default): {% include {{ page.version.version }}/ldr/immediate-description.md %} +- `validated`: {% include {{ page.version.version }}/ldr/validated-description.md %} ## Bidirectional LDR -_Bidirectional_ LDR consists of two clusters with two LDR jobs running in opposite directions between the clusters. If you're setting up bidirectional LDR, both clusters will act as a source and a destination in the respective LDR jobs. +_Bidirectional_ LDR consists of two clusters with two LDR jobs running in opposite directions between the clusters. If you're setting up [bidirectional LDR]({% link {{ page.version.version }}/logical-data-replication-overview.md %}#use-cases), both clusters will act as a source and a destination in the respective LDR jobs. LDR supports starting with two empty tables, or one non-empty table. LDR does **not** support starting with two non-empty tables. When you set up bidirectional LDR, if you're starting with one non-empty table, start the first LDR job from empty to non-empty table. Therefore, you would run `CREATE LOGICAL REPLICATION STREAM` from the destination cluster where the non-empty table exists. ## Examples -To start LDR, you must run the `CREATE LOGICAL REPLICATION STREAM` statement from the **destination** cluster. Use the[ fully qualified table name(s)]({% link {{ page.version.version }}/sql-name-resolution.md %}#how-name-resolution-works) and ensure that the source and destination table names are identical. The following examples show statement usage with different options and use cases. +To start LDR, you must run the `CREATE LOGICAL REPLICATION STREAM` statement from the **destination** cluster. Use the [fully qualified table name(s)]({% link {{ page.version.version }}/sql-name-resolution.md %}#how-name-resolution-works). The following examples show statement usage with different options and use cases. ### Start an LDR stream @@ -89,14 +89,14 @@ CREATE LOGICAL REPLICATION STREAM FROM TABLES ({database.public.table_name},{dat ### Ignore row-level TTL deletes -If you would like to ignore [row-level TTL]({% link {{ page.version.version }}/row-level-ttl.md %}) deletes in the LDR stream, set the [`ttl_disable_changefeed_replication` storage parameter]({% link {{ page.version.version }}/row-level-ttl.md %}#ttl-storage-parameters) on the table. On the **source** cluster, alter the table to set the table storage parameter: +If you would like to ignore [row-level TTL]({% link {{ page.version.version }}/row-level-ttl.md %}) deletes in a **unidirectional** LDR stream, set the [`ttl_disable_changefeed_replication` storage parameter]({% link {{ page.version.version }}/row-level-ttl.md %}#ttl-storage-parameters) on the table. On the **source** cluster, alter the table to set the table storage parameter: {% include_cached copy-clipboard.html %} ~~~ sql ALTER TABLE {table_name} SET (ttl_disable_changefeed_replication = 'true'); ~~~ -When you start LDR on the **destination cluster**, include the `discard = ttl-deletes` option in the statement: +When you start LDR on the **destination cluster**, include the [`discard = ttl-deletes` option](#discard-ttl-deletes-option) in the statement: {% include_cached copy-clipboard.html %} ~~~ sql diff --git a/src/current/v24.3/create-schedule-for-changefeed.md b/src/current/v24.3/create-schedule-for-changefeed.md index 6902e2cc135..5822569b0dc 100644 --- a/src/current/v24.3/create-schedule-for-changefeed.md +++ b/src/current/v24.3/create-schedule-for-changefeed.md @@ -58,6 +58,10 @@ Option | Value | Description `on_execution_failure` | `retry` / `reschedule` / `pause` | Determine how the schedule handles an error.
`retry`: Retry the changefeed immediately.
`reschedule`: Reschedule the changefeed based on the `RECURRING` expression.
`pause`: Pause the schedule. This requires that you [resume the schedule]({% link {{ page.version.version }}/resume-schedules.md %}) manually.
**Default:** `reschedule` `on_previous_running` | `start` / `skip` / `wait` | Control whether the changefeed schedule should start a changefeed if the previous scheduled changefeed is still running.
`start`: Start the new changefeed anyway, even if the previous one is running.
`skip`: Skip the new changefeed and run the next changefeed based on the `RECURRING` expression.
`wait`: Wait for the previous changefeed to complete.
**Default:** `wait` +{{site.data.alerts.callout_info}} +To avoid multiple clusters running the same schedule concurrently, changefeed schedules will [pause]({% link {{ page.version.version }}/pause-schedules.md %}) when [restored]({% link {{ page.version.version }}/restore.md %}) onto a different cluster or after [physical cluster replication]({% link {{ page.version.version }}/failover-replication.md %}) has completed. +{{site.data.alerts.end}} + ## Examples Before running any of the examples in this section, it is necessary to enable the `kv.rangefeed.enabled` cluster setting. If you are working on a CockroachDB {{ site.data.products.standard }} or {{ site.data.products.basic }} cluster, this cluster setting is enabled by default. diff --git a/src/current/v24.3/deploy-cockroachdb-on-aws.md b/src/current/v24.3/deploy-cockroachdb-on-aws.md index 2acc789f23a..58f4e2fe26b 100644 --- a/src/current/v24.3/deploy-cockroachdb-on-aws.md +++ b/src/current/v24.3/deploy-cockroachdb-on-aws.md @@ -9,7 +9,7 @@ docs_area: {% include {{ page.version.version }}/filter-tabs/deploy-crdb-aws.md %} -This page shows you how to manually deploy a secure multi-node CockroachDB cluster on Amazon's AWS EC2 platform, using AWS's managed load balancing service to distribute client traffic. +This page shows you how to manually deploy a multi-node, self-hosted CockroachDB cluster on Amazon's AWS EC2 platform, using AWS's managed load-balancing service to distribute client traffic. After setting up the AWS network, clock synchronization, and load balancing, it should take approximately 20 minutes to complete the deployment. This is based on initializing a three-node CockroachDB cluster in a single AWS region and running our sample workload. @@ -18,7 +18,7 @@ If you are only testing CockroachDB, or you are not concerned with protecting ne {% include cockroachcloud/use-cockroachcloud-instead.md %} {{site.data.alerts.callout_info}} -You need a license to use CockroachDB; obtain a private offer link on the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-ph5bx6fhm4nlq) or see [CockroachDB Pricing](https://www.cockroachlabs.com/pricing/) to learn about custom pricing. +You need a license to use CockroachDB. Refer to the [Licensing FAQ]({% link {{ page.version.version }}/licensing-faqs.md %}) and [CockroachDB Pricing](https://www.cockroachlabs.com/pricing). [Contact us](https://cockroachlabs.com/contact-sales) about custom pricing through AWS Marketplace. {{site.data.alerts.end}} ## Before you begin diff --git a/src/current/v24.3/failover-replication.md b/src/current/v24.3/failover-replication.md index acce31d45c8..394d234fe29 100644 --- a/src/current/v24.3/failover-replication.md +++ b/src/current/v24.3/failover-replication.md @@ -167,17 +167,21 @@ During a replication stream, jobs running on the primary cluster will replicate [Backup schedules]({% link {{ page.version.version }}/manage-a-backup-schedule.md %}) will pause after failover on the promoted cluster. Take the following steps to resume jobs: 1. Verify that there are no other schedules running backups to the same [collection of backups]({% link {{ page.version.version }}/take-full-and-incremental-backups.md %}#backup-collections), i.e., the schedule that was running on the original primary cluster. -1. Resume the backup schedule on the promoted cluster. +1. [Resume]({% link {{ page.version.version }}/resume-schedules.md %}) the backup schedule on the promoted cluster. {{site.data.alerts.callout_info}} -If your backup schedule was created on a cluster in v23.1 or earlier, it will **not** pause automatically on the promoted cluster after failover. In this case, you must pause the schedule manually on the promoted cluster and then take the outlined steps. +If your backup schedule was created on a cluster in v23.1 or earlier, it will **not** pause automatically on the promoted cluster after failover. In this case, you must [pause]({% link {{ page.version.version }}/pause-schedules.md %}) the schedule manually on the promoted cluster and then take the outlined steps. {{site.data.alerts.end}} ### Changefeeds [Changefeeds]({% link {{ page.version.version }}/change-data-capture-overview.md %}) will fail on the promoted cluster immediately after failover to avoid two clusters running the same changefeed to one sink. We recommend that you recreate changefeeds on the promoted cluster. -[Scheduled changefeeds]({% link {{ page.version.version }}/create-schedule-for-changefeed.md %}) will continue on the promoted cluster. You will need to manage [pausing]({% link {{ page.version.version }}/pause-schedules.md %}) or [canceling]({% link {{ page.version.version }}/drop-schedules.md %}) the schedule on the promoted standby cluster to avoid two clusters running the same changefeed to one sink. +To avoid multiple clusters running the same schedule concurrently, [changefeed schedules]({% link {{ page.version.version }}/create-schedule-for-changefeed.md %}) will [pause]({% link {{ page.version.version }}/pause-schedules.md %}) after physical cluster replication has completed. + +{{site.data.alerts.callout_info}} +If your changefeed schedule was created on a cluster in v24.1 or earlier, it will **not** pause automatically on the promoted cluster after failover. In this case, you will need to manage [pausing]({% link {{ page.version.version }}/pause-schedules.md %}) or [canceling]({% link {{ page.version.version }}/drop-schedules.md %}) the schedule on the promoted standby cluster to avoid two clusters running the same changefeed to one sink. +{{site.data.alerts.end}} ## Fail back to the primary cluster diff --git a/src/current/v24.3/how-does-an-enterprise-changefeed-work.md b/src/current/v24.3/how-does-an-enterprise-changefeed-work.md index 43fb7ea3fe0..566f6a15e48 100644 --- a/src/current/v24.3/how-does-an-enterprise-changefeed-work.md +++ b/src/current/v24.3/how-does-an-enterprise-changefeed-work.md @@ -7,7 +7,7 @@ docs_area: stream_data When an {{ site.data.products.enterprise }} changefeed is started on a node, that node becomes the _coordinator_ for the changefeed job (**Node 2** in the diagram). The coordinator node acts as an administrator: keeping track of all other nodes during job execution and the changefeed work as it completes. The changefeed job will run across nodes in the cluster to access changed data in the watched table. The job will evenly distribute changefeed work across the cluster by assigning it to any [replica]({% link {{ page.version.version }}/architecture/replication-layer.md %}) for a particular range, which determines the node that will emit the changefeed data. If a [locality filter]({% link {{ page.version.version }}/changefeeds-in-multi-region-deployments.md %}#run-a-changefeed-job-by-locality) is specified, work is distributed to a node from those that match the locality filter and has the most locality tiers in common with a node that has a replica. -Each node uses its aggregator processors to send back checkpoint progress to the coordinator, which gathers this information to update the high-water mark timestamp. The high-water mark acts as a checkpoint for the changefeed’s job progress, and guarantees that all changes before (or at) the timestamp have been emitted. In the unlikely event that the changefeed’s coordinating node were to fail during the job, that role will move to a different node and the changefeed will restart from the last checkpoint. If restarted, the changefeed may [re-emit messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) starting at the high-water mark time to the current time. Refer to [Ordering Guarantees]({% link {{ page.version.version }}/changefeed-messages.md %}#ordering-and-delivery-guarantees) for detail on CockroachDB's at-least-once-delivery-guarantee and how per-key message ordering is applied. +Each node uses its _aggregator processors_ to send back checkpoint progress to the coordinator, which gathers this information to update the _high-water mark timestamp_. The high-water mark acts as a checkpoint for the changefeed’s job progress, and guarantees that all changes before (or at) the timestamp have been emitted. In the unlikely event that the changefeed’s coordinating node were to fail during the job, that role will move to a different node and the changefeed will restart from the last checkpoint. If restarted, the changefeed may [re-emit messages]({% link {{ page.version.version }}/changefeed-messages.md %}#duplicate-messages) starting at the high-water mark time to the current time. Refer to [Ordering Guarantees]({% link {{ page.version.version }}/changefeed-messages.md %}#ordering-and-delivery-guarantees) for detail on CockroachDB's at-least-once-delivery-guarantee and how per-key message ordering is applied.
diff --git a/src/current/v24.3/intellij-idea.md b/src/current/v24.3/intellij-idea.md
index 78c09b6669c..05772f16b0a 100644
--- a/src/current/v24.3/intellij-idea.md
+++ b/src/current/v24.3/intellij-idea.md
@@ -34,7 +34,7 @@ Users can expect to encounter the following behaviors when using CockroachDB wit
##### [XXUUU] ERROR: could not decorrelate subquery
-
+
Displays once per load of schema.
@@ -42,7 +42,7 @@ Displays once per load of schema.
##### [42883] ERROR: unknown function: pg_function_is_visible() Failed to retrieve
-
+
Display periodically. Does not impact functionality.
@@ -50,7 +50,7 @@ Display periodically. Does not impact functionality.
##### [42703] org.postgresql.util.PSQLException: ERROR: column "n.xmin" does not exist
-
+
Requires setting **Introspect using JDBC metadata** ([details below](#set-cockroachdb-as-a-data-source-in-intellij)).
@@ -58,8 +58,8 @@ Requires setting **Introspect using JDBC metadata** ([details below](#set-cockro
## Set CockroachDB as a data source in IntelliJ
-1. Launch the **Database** tool window. (**View** > **Tool Windows** > **Database**) 
-1. Add a PostgreSQL data source. (**New (+)** > **Data Source** > **PostgreSQL**)
+1. Launch the **Database** tool window. (**View** > **Tool Windows** > **Database**) 
+1. Add a PostgreSQL data source. (**New (+)** > **Data Source** > **PostgreSQL**)
1. On the **General** tab, enter your database's connection string:
Field | Value
@@ -71,10 +71,10 @@ Requires setting **Introspect using JDBC metadata** ([details below](#set-cockro
**Password** | If your cluster uses password authentication, enter the password.
**Driver** | Select or install **PostgreSQL** using a version greater than or equal to 41.1. (Older drivers have not been tested.)
- 
+ 
1. Install or select a **PostgreSQL** driver. We recommend a version greater than or equal to 41.1.
1. If your cluster uses SSL authentication, go to the **SSH/SSL** tab, select **Use SSL** and provide the location of your certificate files.
-1. Go to the **Options** tab, and then select **Introspect using JDBC metadata**.
+1. Go to the **Options** tab, and then select **Introspect using JDBC metadata**.
1. Click **OK**.
You can now use IntelliJ's [database tool window](https://www.jetbrains.com/help/idea/working-with-the-database-tool-window.html) to interact with your CockroachDB cluster.
diff --git a/src/current/v24.3/known-limitations.md b/src/current/v24.3/known-limitations.md
index 792ba38c91f..184ff766a49 100644
--- a/src/current/v24.3/known-limitations.md
+++ b/src/current/v24.3/known-limitations.md
@@ -465,7 +465,6 @@ Accessing the DB Console for a secure cluster now requires login information (i.
#### Physical cluster replication
{% include {{ page.version.version }}/known-limitations/physical-cluster-replication.md %}
-- {% include {{ page.version.version }}/known-limitations/pcr-scheduled-changefeeds.md %}
- {% include {{ page.version.version }}/known-limitations/failover-stop-application.md %}
#### `RESTORE` limitations
@@ -489,7 +488,6 @@ As a workaround, take a cluster backup instead, as the `system.comments` table i
Change data capture (CDC) provides efficient, distributed, row-level changefeeds into Apache Kafka for downstream processing such as reporting, caching, or full-text indexing. It has the following known limitations:
{% include {{ page.version.version }}/known-limitations/cdc.md %}
-- {% include {{ page.version.version }}/known-limitations/pcr-scheduled-changefeeds.md %}
{% include {{ page.version.version }}/known-limitations/cdc-queries.md %}
- {% include {{ page.version.version }}/known-limitations/cdc-queries-column-families.md %}
- {% include {{ page.version.version }}/known-limitations/changefeed-column-family-message.md %}
diff --git a/src/current/v24.3/logical-data-replication-monitoring.md b/src/current/v24.3/logical-data-replication-monitoring.md
index 88c2b919354..240744851fa 100644
--- a/src/current/v24.3/logical-data-replication-monitoring.md
+++ b/src/current/v24.3/logical-data-replication-monitoring.md
@@ -14,7 +14,7 @@ Logical data replication is only supported in CockroachDB {{ site.data.products.
You can monitor [**logical data replication (LDR)**]({% link {{ page.version.version }}/logical-data-replication-overview.md %}) using:
- [`SHOW LOGICAL REPLICATION JOBS`](#sql-shell) in the SQL shell to view a list of LDR jobs on the cluster.
-- The **Logical Data Replication** dashboard on the [DB Console](#db-console) to view metrics at the cluster level. {% comment %}To add link later to dashboard page{% endcomment %}
+- The [**Logical Data Replication** dashboard]({% link {{ page.version.version }}/ui-logical-data-replication-dashboard.md %}) on the [DB Console](#db-console) to view metrics at the cluster level.
- [Prometheus and Alertmanager](#prometheus) to track and alert on LDR metrics.
- Metrics export with [Datadog](#datadog).
- [Metrics labels](#metrics-labels) to view metrics at the job level.
@@ -77,26 +77,24 @@ You can also use [`SHOW JOBS`]({% link {{ page.version.version }}/show-jobs.md %
In the DB Console, you can use:
-- The [**Metrics** dashboard]({% link {{ page.version.version }}/ui-overview-dashboard.md %}) for LDR to view metrics for the job on the destination cluster.
+- The [**Metrics** dashboard]({% link {{ page.version.version }}/ui-logical-data-replication-dashboard.md %}) for LDR to view metrics for the job on the destination cluster.
- The [**Jobs** page]({% link {{ page.version.version }}/ui-jobs-page.md %}) to view the history retention job on the source cluster and the LDR job on the destination cluster
The metrics for LDR in the DB Console metrics are at the **cluster** level. This means that if there are multiple LDR jobs running on a cluster the DB Console will show the average metrics across jobs.
### Metrics dashboard
-You can use the [**Logical Data Replication** dashboard]({% link {{ page.version.version }}/ui-overview-dashboard.md %}) of the destination cluster to monitor the following metric graphs at the **cluster** level:
+You can use the [**Logical Data Replication** dashboard]({% link {{ page.version.version }}/ui-logical-data-replication-dashboard.md %}) of the destination cluster to monitor the following metric graphs at the **cluster** level:
- Replication latency
- Replication lag
- Row updates applied
-- Logical bytes reviewed
+- Logical bytes received
- Batch application processing time: 50th percentile
- Batch application processing time: 99th percentile
- DLQ causes
- Retry queue size
-{% comment %}Dashboard page in the DB Console docs to be added with more information per other dashboards. Link to there from this section.{% endcomment %}
-
To track replicated time, ingested events, and events added to the DLQ at the **job** level, refer to [Metrics labels](#metrics-labels).
### Jobs page
@@ -116,11 +114,11 @@ You can use Prometheus and Alertmanager to track and alert on LDR metrics. Refer
To view metrics at the job level, you can use the `label` option when you start LDR to add a metrics label to the LDR job. This enables [child metric]({% link {{ page.version.version }}/child-metrics.md %}) export, which are Prometheus time series with extra labels. You can track the following metrics for an LDR job with labels:
-- `logical_replication.replicated_time_seconds`
-- `logical_replication.events_ingested`
-- `logical_replication.events_dlqed`
-- `logical_replication.scanning_ranges`
-- `logical_replication.catchup_ranges`
+- `logical_replication.catchup_ranges_by_label`
+- `logical_replication.events_dlqed_by_label`
+- `logical_replication.events_ingested_by_label`
+- `logical_replication.replicated_time_by_label`
+- `logical_replication.scanning_ranges_by_label`
To use metrics labels, ensure you have enabled the child metrics cluster setting:
@@ -138,7 +136,7 @@ ON 'external://{source_external_connection}'
INTO TABLE {database.public.table_name} WITH label=ldr_job;
~~~
-For a full reference on tracking metrics with labels, refer to the [Child Metrics]({% link {{ page.version.version }}/child-metrics.md %}) page.
+For a full reference on tracking metrics with labels, refer to the [Child Metrics]({% link {{ page.version.version }}/child-metrics.md %}#clusters-with-logical-data-replication-jobs) page.
### Datadog
diff --git a/src/current/v24.3/logical-data-replication-overview.md b/src/current/v24.3/logical-data-replication-overview.md
index 69df26a4707..7495991c39f 100644
--- a/src/current/v24.3/logical-data-replication-overview.md
+++ b/src/current/v24.3/logical-data-replication-overview.md
@@ -44,7 +44,7 @@ Isolate critical application workloads from non-critical application workloads.
- **Table-level replication**: When you initiate LDR, it will replicate all of the source table's existing data to the destination table. From then on, LDR will replicate the source table's data to the destination table to achieve eventual consistency.
- **Last write wins conflict resolution**: LDR uses [_last write wins (LWW)_ conflict resolution]({% link {{ page.version.version }}/manage-logical-data-replication.md %}#conflict-resolution), which will use the latest [MVCC]({% link {{ page.version.version }}/architecture/storage-layer.md %}#mvcc) timestamp to resolve a conflict in row insertion.
- **Dead letter queue (DLQ)**: When LDR starts, the job will create a [DLQ table]({% link {{ page.version.version }}/manage-logical-data-replication.md %}#dead-letter-queue-dlq) with each replicating table in order to track unresolved conflicts. You can interact and manage this table like any other SQL table.
-- **Replication modes**: LDR offers different _modes_ that apply data differently during replication, which allows you to consider optimizing for throughput or constraints during replication.
+- **Replication modes**: LDR offers different [_modes_]({% link {{ page.version.version }}/create-logical-replication-stream.md %}#ldr-modes) that apply data differently during replication, which allows you to consider optimizing for throughput or constraints during replication.
- **Monitoring**: To [monitor]({% link {{ page.version.version }}/logical-data-replication-monitoring.md %}) LDR's initial progress, current status, and performance, you can view metrics available in the DB Console, Prometheus, and Metrics Export.
## Get started
diff --git a/src/current/v24.3/manage-logical-data-replication.md b/src/current/v24.3/manage-logical-data-replication.md
index 6e9b47fa1cf..eb43b0a1932 100644
--- a/src/current/v24.3/manage-logical-data-replication.md
+++ b/src/current/v24.3/manage-logical-data-replication.md
@@ -18,11 +18,11 @@ Once you have **logical data replication (LDR)** running, you will need to track
## Conflict resolution
-Conflict resolution in LDR differs depending on whether the conflict occurs at the [KV]({% link {{ page.version.version }}/architecture/storage-layer.md %}) level or the [SQL]({% link {{ page.version.version }}/architecture/sql-layer.md %}) level.
+In LDR, conflicts are detected at both the [KV]({% link {{ page.version.version }}/architecture/storage-layer.md %}) and the [SQL]({% link {{ page.version.version }}/architecture/sql-layer.md %}) level, which determines how LDR resolves the conflict.
### KV level conflicts
-LDR uses _last write wins (LWW)_ conflict resolution based on the [MVCC timestamp]({% link {{ page.version.version }}/architecture/storage-layer.md %}#mvcc) of the replicating write. LDR will resolve conflicts by inserting the row with the latest MVCC timestamp.
+LDR uses _last write wins (LWW)_ conflict resolution based on the [MVCC timestamp]({% link {{ page.version.version }}/architecture/storage-layer.md %}#mvcc) of the replicating write. LDR will resolve conflicts by inserting the row with the latest MVCC timestamp. Conflicts at the KV level are detected in both `immediate` and `validated` mode.
Conflicts at the KV level are detected when there is either:
diff --git a/src/current/v24.3/monitor-and-debug-changefeeds.md b/src/current/v24.3/monitor-and-debug-changefeeds.md
index bcdfd17293e..8ee4a0f5aa8 100644
--- a/src/current/v24.3/monitor-and-debug-changefeeds.md
+++ b/src/current/v24.3/monitor-and-debug-changefeeds.md
@@ -153,6 +153,7 @@ changefeed_emitted_bytes{scope="vehicles"} 183557
`changefeed.message_size_hist` | Distribution in the size of emitted messages. | Bytes | Histogram
`changefeed.running` | Number of currently running changefeeds, including sinkless changefeeds. | Changefeeds | Gauge
`changefeed.sink_batch_hist_nanos` | Time messages spend batched in the sink buffer before being flushed and acknowledged. | Nanoseconds | Histogram
+New in v24.3: `changefeed.total_ranges` | Total number of ranges that are watched by [aggregator processors]({% link {{ page.version.version }}/how-does-an-enterprise-changefeed-work.md %}) participating in the changefeed job. `changefeed.total_ranges` shares the same polling interval as the [`changefeed.lagging_ranges`](#lagging-ranges-metric) metric, which is controlled by the `lagging_ranges_polling_interval` option. For more details, refer to [Lagging ranges](#lagging-ranges).
### Monitoring and measuring changefeed latency
diff --git a/src/current/v24.3/operational-faqs.md b/src/current/v24.3/operational-faqs.md
index 10c04d37472..7a073b9c0dc 100644
--- a/src/current/v24.3/operational-faqs.md
+++ b/src/current/v24.3/operational-faqs.md
@@ -81,9 +81,13 @@ When MVCC garbage is deleted by garbage collection, the data is still not yet ph
{% include {{page.version.version}}/storage/free-up-disk-space.md %}
-## How can I free up disk space quickly?
+## How can I free up disk space when dropping a table?
-If you've noticed that [your disk space is not freeing up quickly enough after deleting data](#why-is-my-disk-usage-not-decreasing-after-deleting-data), you can take the following steps to free up disk space more quickly. This example assumes a table `t`.
+If you've noticed that [your disk space is not freeing up quickly enough after dropping a table](#why-is-my-disk-usage-not-decreasing-after-deleting-data), you can take the following steps to free up disk space more quickly the next time you drop a table. This example assumes a table `t` exists.
+
+{{site.data.alerts.callout_info}}
+The procedure shown here only works if you get the range IDs from the table **before** running [`DROP TABLE`]({% link {{ page.version.version }}/drop-table.md %}). If you are in an emergency situation due to running out of disk, see [What happens when a node runs out of disk space?](#what-happens-when-a-node-runs-out-of-disk-space)
+{{site.data.alerts.end}}
1. Lower the [`gc.ttlseconds` parameter]({% link {{ page.version.version }}/configure-replication-zones.md %}#gc-ttlseconds) to 10 minutes.
diff --git a/src/current/v24.3/physical-cluster-replication-overview.md b/src/current/v24.3/physical-cluster-replication-overview.md
index 8ad79e3d6d0..bd59c819079 100644
--- a/src/current/v24.3/physical-cluster-replication-overview.md
+++ b/src/current/v24.3/physical-cluster-replication-overview.md
@@ -15,6 +15,10 @@ In a disaster recovery scenario, you can [_fail over_]({% link {{ page.version.v
For a list of requirements for PCR, refer to the [Before you begin]({% link {{ page.version.version }}/set-up-physical-cluster-replication.md %}#before-you-begin) section of the [setup tutorial]({% link {{ page.version.version }}/set-up-physical-cluster-replication.md %}).
+{{site.data.alerts.callout_success}}
+Cockroach Labs also has a [logical data replication]({% link {{ page.version.version }}/logical-data-replication-overview.md %}) tool that continuously replicates tables between an active _source_ CockroachDB cluster to an active _destination_ CockroachDB cluster. Both source and destination can receive application reads and writes, and participate in [_bidirectional_]({% link {{ page.version.version }}/logical-data-replication-overview.md %}#use-cases) LDR for eventual consistency in the replicating tables.
+{{site.data.alerts.end}}
+
## Use cases
You can use PCR in a disaster recovery plan to:
@@ -30,7 +34,8 @@ You can use PCR in a disaster recovery plan to:
- **Transactional consistency**: You can fail over to the standby cluster at the [`LATEST` timestamp]({% link {{ page.version.version }}/failover-replication.md %}#fail-over-to-the-most-recent-replicated-time) or a [point of time]({% link {{ page.version.version }}/failover-replication.md %}#fail-over-to-a-point-in-time) in the past or the future. When the failover process completes, the standby cluster will be in a transactionally consistent state as of the point in time you specified.
- **Maintained/improved RPO and RTO**: Depending on workload and deployment configuration, [replication lag]({% link {{ page.version.version }}/physical-cluster-replication-technical-overview.md %}) between the primary and standby is generally in the tens-of-seconds range. The failover process from the primary cluster to the standby should typically happen within five minutes when completing a failover to the latest replicated time using `LATEST`.
- **Failover to a timestamp in the past or the future**: In the case of logical disasters or mistakes, you can [fail over]({% link {{ page.version.version }}/failover-replication.md %}) from the primary to the standby cluster to a timestamp in the past. This means that you can return the standby to a timestamp before the mistake was replicated to the standby. You can also configure the [`WITH RETENTION`]({% link {{ page.version.version }}/alter-virtual-cluster.md %}#set-a-retention-window) option to control how far in the past you can fail over to. Furthermore, you can plan a failover by specifying a timestamp in the future.
-- **Monitoring**: To monitor the replication's initial progress, current status, and performance, you can use metrics available in the [DB Console]({% link {{ page.version.version }}/ui-overview.md %}) and [Prometheus]({% link {{ page.version.version }}/monitor-cockroachdb-with-prometheus.md %}). For more detail, refer to [Physical Cluster Replication Monitoring]({% link {{ page.version.version }}/physical-cluster-replication-monitoring.md %}).
+- {% include_cached new-in.html version="v24.3" %} **Read from standby cluster**: You can configure PCR to allow read queries on the standby cluster. For more details, refer to [Start a PCR stream with read from standby]({% link {{ page.version.version }}/create-virtual-cluster.md %}#start-a-pcr-stream-with-read-from-standby).
+- **Monitoring**: To monitor the replication's initial progress, current status, and performance, you can use metrics available in the [DB Console]({% link {{ page.version.version }}/ui-overview.md %}) and [Prometheus]({% link {{ page.version.version }}/monitor-cockroachdb-with-prometheus.md %}). For more details, refer to [Physical Cluster Replication Monitoring]({% link {{ page.version.version }}/physical-cluster-replication-monitoring.md %}).
{{site.data.alerts.callout_info}}
[Failing over to a timestamp in the past]({% link {{ page.version.version }}/failover-replication.md %}#fail-over-to-a-point-in-time) involves reverting data on the standby cluster. As a result, this type of failover takes longer to complete than failover to the [latest replicated time]({% link {{ page.version.version }}/failover-replication.md %}#fail-over-to-the-most-recent-replicated-time). The increase in failover time will correlate to how much data you are reverting from the standby. For more detail, refer to the [Technical Overview]({% link {{ page.version.version }}/physical-cluster-replication-technical-overview.md %}) page for PCR.
@@ -39,7 +44,6 @@ You can use PCR in a disaster recovery plan to:
## Known limitations
{% include {{ page.version.version }}/known-limitations/physical-cluster-replication.md %}
-- {% include {{ page.version.version }}/known-limitations/pcr-scheduled-changefeeds.md %}
- {% include {{ page.version.version }}/known-limitations/failover-stop-application.md %}
## Performance
diff --git a/src/current/v24.3/restore.md b/src/current/v24.3/restore.md
index 7ac3e82afe9..e8aac279600 100644
--- a/src/current/v24.3/restore.md
+++ b/src/current/v24.3/restore.md
@@ -17,7 +17,7 @@ You can restore:
For details on restoring across versions of CockroachDB, see [Restoring Backups Across Versions]({% link {{ page.version.version }}/restoring-backups-across-versions.md %}).
-{% include {{ page.version.version }}/backups/backup-to-deprec.md %}
+{% include {{ page.version.version }}/backups/old-syntax-removed.md %}
## Considerations
diff --git a/src/current/v24.3/set-up-logical-data-replication.md b/src/current/v24.3/set-up-logical-data-replication.md
index 1aa4ad90cca..f8b9baf6ac1 100644
--- a/src/current/v24.3/set-up-logical-data-replication.md
+++ b/src/current/v24.3/set-up-logical-data-replication.md
@@ -10,7 +10,7 @@ toc: true
Logical data replication is only supported in CockroachDB {{ site.data.products.core }} clusters.
{{site.data.alerts.end}}
-In this tutorial, you will set up **logical data replication (LDR)** streaming data from a source table to a destination table between two CockroachDB clusters. Both clusters are active and can serve traffic. You can apply the outlined steps to create _unidirectional_ LDR from a source table to a destination table (cluster A to cluster B) in one LDR job. Optionally, you can also create _bidirectional_ LDR from cluster B's table to cluster A's table by starting a second LDR job. In a bidirectional setup, each cluster operates as both a source and a destination in separate LDR jobs.
+In this tutorial, you will set up [**logical data replication (LDR)**]({% link {{ page.version.version }}/logical-data-replication-overview.md %}) streaming data from a source table to a destination table between two CockroachDB clusters. Both clusters are active and can serve traffic. You can apply the outlined steps to create _unidirectional_ LDR from a source table to a destination table (cluster A to cluster B) in one LDR job. Optionally, you can also create _bidirectional_ LDR from cluster B's table to cluster A's table by starting a second LDR job. In a bidirectional setup, each cluster operates as both a source and a destination in separate LDR jobs.
For more details on use cases, refer to the [Logical Data Replication Overview]({% link {{ page.version.version }}/logical-data-replication-overview.md %}).
@@ -55,7 +55,7 @@ You cannot use LDR on a table with a schema that contains the following:
For more details, refer to the LDR [Known limitations]({% link {{ page.version.version }}/logical-data-replication-overview.md %}#known-limitations).
-When you run LDR in `immediate` mode, you cannot replicate a table with [foreign key constraints]({% link {{ page.version.version }}/foreign-key.md %}). In `validated` mode, foreign key constraints **must** match. All constraints are enforced at the time of SQL/application write.
+When you run LDR in [`immediate` mode](#modes), you cannot replicate a table with [foreign key constraints]({% link {{ page.version.version }}/foreign-key.md %}). In [`validated` mode](#modes), foreign key constraints **must** match. All constraints are enforced at the time of SQL/application write.
## Step 1. Prepare the cluster
@@ -66,7 +66,7 @@ When you run LDR in `immediate` mode, you cannot replicate a table with [foreign
cockroach sql --url "postgresql://root@{node IP or hostname}:26257?sslmode=verify-full" --certs-dir=certs
~~~
-1. Enable the `kv.rangefeed.enabled` cluster setting on the **source** cluster:
+1. Enable the `kv.rangefeed.enabled` [cluster setting]({% link {{ page.version.version }}/cluster-settings.md %}) on the **source** cluster:
{% include_cached copy-clipboard.html %}
~~~ sql
@@ -85,7 +85,7 @@ When you run LDR in `immediate` mode, you cannot replicate a table with [foreign
GRANT SYSTEM REPLICATION TO {your username};
~~~
- If you need to change the password later, refer to [`ALTER USER`]({% link {{ page.version.version }}/alter-user.md %}). {% comment %}Add link to https://www.cockroachlabs.com/docs/stable/ui-overview#db-console-security-considerations for secure clusters{% endcomment %}
+ If you need to change the password later, refer to [`ALTER USER`]({% link {{ page.version.version }}/alter-user.md %}).
## Step 2. Connect from the destination to the source
@@ -110,21 +110,17 @@ You can use the `cockroach encode-uri` command to generate a connection string c
{% include_cached copy-clipboard.html %}
~~~ sql
- CREATE EXTERNAL CONNECTION {source} AS 'postgresql://{replication user}:{password}@{node IP}:26257?options=-ccluster%3Dsystem&sslinline=true&sslmode=verify-full&sslrootcert=-----BEGIN+CERTIFICATE-----{encoded certificate}-----END+CERTIFICATE-----%0A`;
+ CREATE EXTERNAL CONNECTION {source} AS 'postgresql://{replication user}:{password}@{node IP}:26257?options=-ccluster%3Dsystem&sslinline=true&sslmode=verify-full&sslrootcert=-----BEGIN+CERTIFICATE-----{encoded certificate}-----END+CERTIFICATE-----%0A';
~~~
## Step 3. Start LDR
In this step, you'll start the LDR job from the destination cluster. You can replicate one or multiple tables in a single LDR job. You cannot replicate system tables in LDR, which means that you must manually apply configurations and cluster settings, such as [row-level TTL]({% link {{ page.version.version }}/row-level-ttl.md %}) and user permissions on the destination cluster.
-_Modes_ determine how LDR replicates the data to the destination cluster. There are two modes:
+_Modes_ determine how LDR replicates the data to the destination cluster. There are two modes:
-- `immediate` (default): Attempts to replicate the changed row directly into the destination table, without re-running constraint validations. It does not support writing into tables with [foreign key]({% link {{ page.version.version }}/foreign-key.md %}) constraints.
-- `validated`: Attempts to apply the write in a similar way to a user-run query, which would re-run all constraint validations relevant to the destination table(s). If the change violates foreign key dependencies, unique constraints, or other constraints, the row will be put in the [dead letter queue (DLQ)]({% link {{ page.version.version }}/manage-logical-data-replication.md %}#dead-letter-queue-dlq) instead.
-
-{{site.data.alerts.callout_info}}
-If you would like to ignore TTL deletes in LDR, you can use the `discard = ttl-deletes` option in the `CREATE LOGICAL REPLICATION STREAM` statement. For an example, refer to [Ignore row-level TTL deletes]({% link {{ page.version.version }}/create-logical-replication-stream.md %}#ignore-row-level-ttl-deletes).
-{{site.data.alerts.end}}
+- `immediate` (default): {% include {{ page.version.version }}/ldr/immediate-description.md %}
+- `validated`: {% include {{ page.version.version }}/ldr/validated-description.md %}
1. From the **destination** cluster, start LDR. Use the fully qualified table name for the source and destination tables:
@@ -180,7 +176,7 @@ For a full reference on monitoring LDR, refer to [Logical Data Replication Monit
1. Access the DB Console at `http://{node IP or hostname}:8080` and enter your user's credentials.
1. On the source cluster, navigate to the [**Jobs** page]({% link {{ page.version.version }}/ui-jobs-page.md %}) to view a list of all jobs. Use the job **Type** dropdown and select **Replication Producer**. This will display the history retention job. This will run while the LDR job is active to protect changes to the table from [garbage collection]({% link {{ page.version.version }}/architecture/storage-layer.md %}#garbage-collection) until they have been replicated to the destination cluster.
1. On the destination cluster, use the job **Type** dropdown and select **Logical Replication Ingestion**. This page will display the logical replication stream job. There will be a progress bar in the **Status** column when LDR is replicating a table with existing data. This progress bar shows the status of the initial scan, which backfills the destination table with the existing data.
-1. On the destination cluster, click on **Metrics** in the left-hand navigation menu. Use the **Dashboard** dropdown to select **Logical Data Replication**. This page shows graphs for monitoring LDR.
+1. On the destination cluster, click on **Metrics** in the left-hand navigation menu. Use the **Dashboard** dropdown to select [**Logical Data Replication**]({% link {{ page.version.version }}/ui-logical-data-replication-dashboard.md %}). This page shows graphs for monitoring LDR.
## What's next
diff --git a/src/current/v24.3/show-backup.md b/src/current/v24.3/show-backup.md
index e529a0a05f5..d7ecd78a5d9 100644
--- a/src/current/v24.3/show-backup.md
+++ b/src/current/v24.3/show-backup.md
@@ -8,11 +8,9 @@ docs_area: reference.sql
The `SHOW BACKUP` [statement]({% link {{ page.version.version }}/sql-statements.md %}) lists the contents of a backup created with the [`BACKUP`]({% link {{ page.version.version }}/backup.md %}) statement.
{{site.data.alerts.callout_danger}}
-The `SHOW BACKUP` syntax **without** the `IN` keyword is **deprecated** as of v22.1 and will be removed in a future release.
+The `SHOW BACKUP` syntax **without** the `IN` keyword has been removed from CockroachDB v24.3 and later.
-We recommend using `SHOW BACKUP FROM {subdirectory} IN {collectionURI}` to view backups in a subdirectory at a collection's URI.
-
-For guidance on the syntax for `SHOW BACKUP FROM`, see the [examples](#examples).
+For guidance on the syntax for `SHOW BACKUP FROM`, refer to the [Synopsis](#synopsis) and [examples](#examples) on this page.
{{site.data.alerts.end}}
## Required privileges
diff --git a/src/current/v24.3/sql-feature-support.md b/src/current/v24.3/sql-feature-support.md
index 3ca62dbbc71..df9fa0711d8 100644
--- a/src/current/v24.3/sql-feature-support.md
+++ b/src/current/v24.3/sql-feature-support.md
@@ -1,18 +1,16 @@
---
-title: SQL Feature Support in CockroachDB v24.2
+title: SQL Feature Support in CockroachDB
summary: Summary of CockroachDB's conformance to the SQL standard and which common extensions it supports.
toc: true
keywords: gin, gin index, gin indexes, inverted index, inverted indexes, accelerated index, accelerated indexes
docs_area: reference.sql
---
-Making CockroachDB easy to use is a top priority for us, so we chose to implement SQL. However, even though SQL has a standard, no database implements all of it, nor do any of them have standard implementations of all features.
-
-To understand which standard SQL features we support (as well as common extensions to the standard), use the table below.
+CockroachDB {{ page.version.version }} supports the following standard SQL features and common extensions.
- **Component** lists the components that are commonly considered part of SQL.
- **Supported** shows CockroachDB's level of support for the component.
-- **Type** indicates whether the component is part of the SQL *Standard* or is an *Extension* created by ourselves or others.
+- **Type** indicates whether the component is part of the SQL *Standard* or is an *Extension* created by Cockroach Labs or others.
- **Details** provides greater context about the component.