Skip to content

Commit

Permalink
[DOC-10592] Add details about NUMA
Browse files Browse the repository at this point in the history
  • Loading branch information
mdlinville committed Oct 31, 2024
1 parent ce498c4 commit a69e977
Show file tree
Hide file tree
Showing 5 changed files with 44 additions and 12 deletions.
8 changes: 8 additions & 0 deletions src/current/_includes/common/numa_and_go.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
In a [NUMA (non-uniform memory access) architecture](https://en.wikipedia.org/wiki/Non-uniform_memory_access), the system’s memory is physically distributed across multiple memory banks or "nodes", and each node is assigned to a processor. A processor can access its local memory much faster than non-local memory. This non-uniform memory access can lead to performance differences depending on data location.

CockroachDB is written in Go, which has no support for NUMA scheduling or pinning. To run multiple CockroachDB clusters on a NUMA architecture:

- Assign no more than 32 cores to the node for each instance.
- Ensure that your orchestration framework, process manager, or startup scripts start each CockroachDB in a separate NUMA node.
- Use the `--locality` flag to indicate when multiple CockroachDB instances are running on a given physical host.
- Ensure that each CockroachDB cluster writes to a unique set of storage volumes (block devices).
12 changes: 9 additions & 3 deletions src/current/v23.2/install-cockroachdb-linux.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ See [Release Notes]({% link releases/{{page.version.version}}.md %}) for what's

Use one of the options below to install CockroachDB.

To install a FIPS-compliant CockroachDB binary, refer to [Install a FIPS-compliant build of CockroachDB]({% link {{ page.version.version }}/fips.md %}).

CockroachDB on ARM is <b><a href="{% link {{page.version.version}}/cockroachdb-feature-availability.md %}#feature-availability-phases">Generally Available</a></b> in v23.2.0 and above. For limitations specific to ARM, refer to <a href="#limitations">Limitations</a>.
- To install a FIPS-compliant CockroachDB binary, refer to [Install a FIPS-compliant build of CockroachDB]({% link {{ page.version.version }}/fips.md %}) instead of this page.
- For limitations of CockroachDB on ARM, refer to [ARM limitations](#arm).
- For limitations of CockroachDB on a NUMA architecture, refer to [NUMA limitations](#numa).

<div id="download-the-binary-linux" class="install-option">
<h2 id="install-binary">Download the binary</h2>
Expand Down Expand Up @@ -163,6 +163,8 @@ true

<h2 id="limitations">Limitations</h2>

### ARM

CockroachDB runtimes built for the ARM architecture have the following limitations:

- Floating point operations may yield different results on ARM than on Intel, particularly [Fused Multiply Add (FMA) intrinsics](https://en.wikipedia.org/wiki/Multiply%E2%80%93accumulate_operation#Fused_multiply.E2.80.93add).
Expand All @@ -174,6 +176,10 @@ CockroachDB runtimes built for the ARM architecture have the following limitatio
- In production, Cockroach Labs recommends that all cluster nodes have identical CockroachDB versions, CPU architecture, hardware, and software.
- A mix of Intel and ARM nodes is supported as a temporary transitional state during the migration only. Cockroach Labs recommends that you test and validate your workload ahead of the migration to ensure that the workload and your application work as expected in a cluster with both Intel and ARM nodes, especially with respect to floating-point arithmetic.

### NUMA

{% include_cached common/numa_and_go.md %}

<h2 id="whats-next">What&#39;s next?</h2>

{% include {{ page.version.version }}/misc/install-next-steps.html %}
Expand Down
12 changes: 9 additions & 3 deletions src/current/v24.1/install-cockroachdb-linux.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ docs_area: deploy

Use one of the options below to install CockroachDB. To upgrade an existing cluster, refer to [Upgrade to {{ page.version.version }}]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}).

To install a FIPS-compliant CockroachDB binary, refer to [Install a FIPS-compliant build of CockroachDB]({% link {{ page.version.version }}/fips.md %}).

CockroachDB on ARM is <b><a href="{% link {{page.version.version}}/cockroachdb-feature-availability.md %}#feature-availability-phases">Generally Available</a></b> in v23.2.0 and above. For limitations specific to ARM, refer to <a href="#limitations">Limitations</a>.
- To install a FIPS-compliant CockroachDB binary, refer to [Install a FIPS-compliant build of CockroachDB]({% link {{ page.version.version }}/fips.md %}) instead of this page.
- For limitations of CockroachDB on ARM, refer to [ARM limitations](#arm).
- For limitations of CockroachDB on a NUMA architecture, refer to [NUMA limitations](#numa).

<div id="download-the-binary-linux" class="install-option">
<h2 id="install-binary">Download the binary</h2>
Expand Down Expand Up @@ -163,6 +163,8 @@ true

<h2 id="limitations">Limitations</h2>

### ARM

CockroachDB runtimes built for the ARM architecture have the following limitations:

- Floating point operations may yield different results on ARM than on Intel, particularly [Fused Multiply Add (FMA) intrinsics](https://en.wikipedia.org/wiki/Multiply%E2%80%93accumulate_operation#Fused_multiply.E2.80.93add).
Expand All @@ -174,6 +176,10 @@ CockroachDB runtimes built for the ARM architecture have the following limitatio
- In production, Cockroach Labs recommends that all cluster nodes have identical CockroachDB versions, CPU architecture, hardware, and software.
- A mix of Intel and ARM nodes is supported as a temporary transitional state during the migration only. Cockroach Labs recommends that you test and validate your workload ahead of the migration to ensure that the workload and your application work as expected in a cluster with both Intel and ARM nodes, especially with respect to floating-point arithmetic.

### NUMA

{% include_cached common/numa_and_go.md %}

<h2 id="whats-next">What&#39;s next?</h2>

{% include {{ page.version.version }}/misc/install-next-steps.html %}
Expand Down
12 changes: 9 additions & 3 deletions src/current/v24.2/install-cockroachdb-linux.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ docs_area: deploy

Use one of the options below to install CockroachDB. To upgrade an existing cluster, refer to [Upgrade to {{ page.version.version }}]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}).

To install a FIPS-compliant CockroachDB binary, refer to [Install a FIPS-compliant build of CockroachDB]({% link {{ page.version.version }}/fips.md %}).

CockroachDB on ARM is <b><a href="{% link {{page.version.version}}/cockroachdb-feature-availability.md %}#feature-availability-phases">Generally Available</a></b> in v23.2.0 and above. For limitations specific to ARM, refer to <a href="#limitations">Limitations</a>.
- To install a FIPS-compliant CockroachDB binary, refer to [Install a FIPS-compliant build of CockroachDB]({% link {{ page.version.version }}/fips.md %}) instead of this page.
- For limitations of CockroachDB on ARM, refer to [ARM limitations](#arm).
- For limitations of CockroachDB on a NUMA architecture, refer to [NUMA limitations](#numa).

<div id="download-the-binary-linux" class="install-option">
<h2 id="install-binary">Download the binary</h2>
Expand Down Expand Up @@ -163,6 +163,8 @@ true

<h2 id="limitations">Limitations</h2>

### ARM

CockroachDB runtimes built for the ARM architecture have the following limitations:

- Floating point operations may yield different results on ARM than on Intel, particularly [Fused Multiply Add (FMA) intrinsics](https://en.wikipedia.org/wiki/Multiply%E2%80%93accumulate_operation#Fused_multiply.E2.80.93add).
Expand All @@ -174,6 +176,10 @@ CockroachDB runtimes built for the ARM architecture have the following limitatio
- In production, Cockroach Labs recommends that all cluster nodes have identical CockroachDB versions, CPU architecture, hardware, and software.
- A mix of Intel and ARM nodes is supported as a temporary transitional state during the migration only. Cockroach Labs recommends that you test and validate your workload ahead of the migration to ensure that the workload and your application work as expected in a cluster with both Intel and ARM nodes, especially with respect to floating-point arithmetic.

### NUMA

{% include_cached common/numa_and_go.md %}

<h2 id="whats-next">What&#39;s next?</h2>

{% include {{ page.version.version }}/misc/install-next-steps.html %}
Expand Down
12 changes: 9 additions & 3 deletions src/current/v24.3/install-cockroachdb-linux.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ docs_area: deploy

Use one of the options below to install CockroachDB. To upgrade an existing cluster, refer to [Upgrade to {{ page.version.version }}]({% link {{ page.version.version }}/upgrade-cockroach-version.md %}).

To install a FIPS-compliant CockroachDB binary, refer to [Install a FIPS-compliant build of CockroachDB]({% link {{ page.version.version }}/fips.md %}).

CockroachDB on ARM is <b><a href="{% link {{page.version.version}}/cockroachdb-feature-availability.md %}#feature-availability-phases">Generally Available</a></b> in v23.2.0 and above. For limitations specific to ARM, refer to <a href="#limitations">Limitations</a>.
- To install a FIPS-compliant CockroachDB binary, refer to [Install a FIPS-compliant build of CockroachDB]({% link {{ page.version.version }}/fips.md %}) instead of this page.
- For limitations of CockroachDB on ARM, refer to [ARM limitations](#arm).
- For limitations of CockroachDB on a NUMA architecture, refer to [NUMA limitations](#numa).

<div id="download-the-binary-linux" class="install-option">
<h2 id="install-binary">Download the binary</h2>
Expand Down Expand Up @@ -163,6 +163,8 @@ true

<h2 id="limitations">Limitations</h2>

### ARM

CockroachDB runtimes built for the ARM architecture have the following limitations:

- Floating point operations may yield different results on ARM than on Intel, particularly [Fused Multiply Add (FMA) intrinsics](https://en.wikipedia.org/wiki/Multiply%E2%80%93accumulate_operation#Fused_multiply.E2.80.93add).
Expand All @@ -174,6 +176,10 @@ CockroachDB runtimes built for the ARM architecture have the following limitatio
- In production, Cockroach Labs recommends that all cluster nodes have identical CockroachDB versions, CPU architecture, hardware, and software.
- A mix of Intel and ARM nodes is supported as a temporary transitional state during the migration only. Cockroach Labs recommends that you test and validate your workload ahead of the migration to ensure that the workload and your application work as expected in a cluster with both Intel and ARM nodes, especially with respect to floating-point arithmetic.

### NUMA

{% include_cached common/numa_and_go.md %}

<h2 id="whats-next">What&#39;s next?</h2>

{% include {{ page.version.version }}/misc/install-next-steps.html %}
Expand Down

0 comments on commit a69e977

Please sign in to comment.