Skip to content

Commit

Permalink
Bump test images (#2852)
Browse files Browse the repository at this point in the history
  • Loading branch information
databyjp authored Dec 10, 2024
1 parent 3852110 commit 9fc69da
Show file tree
Hide file tree
Showing 11 changed files with 169 additions and 32 deletions.
35 changes: 35 additions & 0 deletions _includes/code/howto/manage-data.collections.py
Original file line number Diff line number Diff line change
Expand Up @@ -565,6 +565,41 @@

client.close()

# ==============================================
# ===== ALL REPLICATION SETTINGS
# ==============================================

# Connect to a setting with 3 replicas
client = weaviate.connect_to_local(
port=8180 # Port for demo setup with 3 replicas
)

# Clean slate
client.collections.delete("Article")

# START AllReplicationSettings
from weaviate.classes.config import Configure, ReplicationDeletionStrategy

client.collections.create(
"Article",
# highlight-start
replication_config=Configure.replication(
factor=3,
async_enabled=True, # Enable asynchronous repair
deletion_strategy=ReplicationDeletionStrategy.TIME_BASED_RESOLUTION, # Added in v1.28; Set the deletion conflict resolution strategy
)
# highlight-end
)
# END AllReplicationSettings

# Test
collection = client.collections.get("Article")
config = collection.config.get()
assert config.replication_config.async_enabled == True
assert config.replication_config.deletion_strategy == ReplicationDeletionStrategy.TIME_BASED_RESOLUTION

client.close()

# ====================
# ===== SHARDING =====
# ====================
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -151,9 +151,61 @@ For speed, data operations on a tenant occur independently of any tenant activit

## Repairs

When Weaviate detects inconsistent data across replicas, it attempts to repair the out of sync data.
In distributed systems like Weaviate, object replicas can become inconsistent due to any number of reasons - network issues, node failures, or timing conflicts. When Weaviate detects inconsistent data across replicas, it attempts to repair the out of sync data.

Starting in v1.26, Weaviate adds [async replication](#async-replication) to proactively detect inconsistencies. In earlier versions, Weaviate uses a [repair-on-read](#repair-on-read) strategy to repair inconsistencies at read time.
Weaviate uses [async replication](#async-replication), [deletion resolution](#deletion-resolution-strategies) and [repair-on-read](#repair-on-read) strategies to maintain consistency across replicas.

### Async replication

:::info Added in `v1.26`
:::

Async replication runs in the background. It uses a Merkle tree algorithm to monitor and compare the state of nodes within a cluster. If the algorithm identifies an inconsistency, it resyncs the data on the inconsistent node.

Repair-on-read works well with one or two isolated repairs. Async replication is effective in situations where there are many inconsistencies. For example, if an offline node misses a series of updates, async replication quickly restores consistency when the node returns to service.

Async replication supplements the repair-on-read mechanism. If a node becomes inconsistent between sync checks, the repair-on-read mechanism catches the problem at read time.

To activate async replication, set `asyncEnabled` to true in the [`replicationConfig` section of your collection definition](../../manage-data/collections.mdx#replication-settings).

### Deletion resolution strategies

:::info Added in `v1.28`
:::

When an object is present on some replicas but not others, this can be because a creation has not yet been propagated to all replicas, or because a deletion has not yet been propagated to all replicas. It is important to distinguish between these two cases.

Deletion resolution works alongside async replication and repair-on-read to ensure consistent handling of deleted objects across the cluster. For each collection, you can set one of the following deletion resolution strategies:

- `NoAutomatedResolution`
- `DeleteOnConflict`
- `TimeBasedResolution`

#### `NoAutomatedResolution`

This is the default setting, and the only setting available in Weaviate versions prior to `v1.28`. In this mode, Weaviate does not treat deletion conflicts as a special case. If an object is present on some replicas but not others, Weaviate may potentially restore the object on the replicas where it is missing.

#### `DeleteOnConflict`

A deletion conflict in `deleteOnConflict` is always resolved by deleting the object on all replicas.

To do so, Weaviate updates an object as a deleted object on a replica upon receiving a deletion request, rather than removing all traces of the object.

#### `TimeBasedResolution`

A deletion conflict in `timeBasedResolution` is resolved based on the timestamp of the deletion request, in comparison to any subsequent updates to the object such as a creation or an update.

If the deletion request has a timestamp that is later than the timestamp of any subsequent updates, the object is deleted on all replicas. If the deletion request has a timestamp that is earlier than the timestamp of any subsequent updates, the later updates are applied to all replicas.

For example:
- If an object is deleted at timestamp 100 and then recreated at timestamp 90, the recreation wins
- If an object is deleted at timestamp 100 and then recreated at timestamp 110, the deletion wins

#### Choosing a strategy

- Use `NoAutomatedResolution` when you want maximum control and handle conflicts manually
- Use `DeleteOnConflict` when you want to ensure deletions are always honored
- Use `TimeBasedResolution` when you want the most recent operation to take precedence

### Repair-on-read

Expand All @@ -178,19 +230,6 @@ The read repair process also depends on the read and write consistency levels us

Repairs only happen on read, so they do not create a lot of background overhead. While nodes are in an inconsistent state, read operations with consistency level of `ONE` may return stale data.

### Async replication

:::info Added in `v1.26`
:::

Async replication runs in the background. It uses a Merkle tree algorithm to monitor and compare the state of nodes within a cluster. If the algorithm identifies an inconsistency, it resyncs the data on the inconsistent node.

Repair-on-read works well with one or two isolated repairs. Async replication is effective in situations where there are many inconsistencies. For example, if an offline node misses a series of updates, async replication quickly restores consistency when the node returns to service.

Async replication supplements the repair-on-read mechanism. If a node becomes inconsistent between sync checks, the repair-on-read mechanism catches the problem at read time.

To activate async replication, set `asyncEnabled` to true in the [`replicationConfig` section of your collection definition](../../manage-data/collections.mdx#replication-settings).

## Related pages
- [API References | GraphQL | Get | Consistency Levels](../../api/graphql/get.md#consistency-levels)
- [API References | REST | Objects](/developers/weaviate/api/rest#tag/objects)
Expand Down
69 changes: 66 additions & 3 deletions developers/weaviate/manage-data/collections.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -838,11 +838,74 @@ import RaftRFChangeWarning from '/_includes/1-25-replication-factor.mdx';

<RaftRFChangeWarning/>

Configure replication per collection.
Configure replication settings, such as [async replication](../concepts/replication-architecture/consistency.md#async-replication) and [deletion resolution strategy](../concepts/replication-architecture/consistency.md#deletion-resolution-strategies).

import ReplicationConfigWithAsyncRepair from '/_includes/code/configuration/replication-consistency.mdx';
<Tabs groupId="languages">
<TabItem value="py" label="Python Client v4">
<FilteredTextBlock
text={PyCode}
startMarker="# START AllReplicationSettings"
endMarker="# END AllReplicationSettings"
language="py"
docRefs={[
"weaviate.collections.classes.html#weaviate.collections.classes.config.Configure"
]}
/>
</TabItem>

<ReplicationConfigWithAsyncRepair />
<TabItem value="py3" label="Python Client v3">
<FilteredTextBlock
text={PyCodeV3}
startMarker="# START AsyncRepair"
endMarker="# END AsyncRepair"
language="pyv3"
/>
</TabItem>

<TabItem value="js" label="JS/TS Client v3">
<FilteredTextBlock
text={TSCode}
startMarker="// START AsyncRepair"
endMarker="// END AsyncRepair"
language="ts"
/>
</TabItem>

<TabItem value="js2" label="JS/TS Client v2">
<FilteredTextBlock
text={TSCodeLegacy}
startMarker="// START AsyncRepair"
endMarker="// END AsyncRepair"
language="tsv2"
/>
</TabItem>
<TabItem value="curl" label="cURL">

```bash
curl \
-X POST \
-H "Content-Type: application/json" \
-d '{
"class": "Article",
"properties": [
{
"dataType": [
"string"
],
"description": "Title of the article",
"name": "title"
}
],
"replicationConfig": {
"factor": 3,
"asyncEnabled": true
}
}' \
http://localhost:8080/v1/schema
```

</TabItem>
</Tabs>

<details>
<summary>
Expand Down
4 changes: 2 additions & 2 deletions developers/weaviate/model-providers/cohere/embeddings.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,8 @@ Provide the API key to Weaviate using one of the following methods:
endMarker="# END CohereInstantiation"
language="py"
docRefs={[
"connect_to_weaviate_cloud",
"auth.Auth",
"weaviate.html#weaviate.connect_to_weaviate_cloud",
"weaviate.html#weaviate.auth.Auth",
]}
/>
</TabItem>
Expand Down
8 changes: 4 additions & 4 deletions src/components/Documentation/FilteredTextBlock.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,14 @@ const GITHUB_CONFIG = {
const DOC_SYSTEMS = {
py: {
baseUrl:
'https://weaviate-python-client.readthedocs.io/en/stable/weaviate.html',
constructUrl: (baseUrl, ref) => `${baseUrl}#weaviate.${ref}`,
'https://weaviate-python-client.readthedocs.io/en/stable',
constructUrl: (baseUrl, ref) => `${baseUrl}/${ref}`,
icon: '/img/site/logo-py.svg',
},
pyv3: {
baseUrl:
'https://weaviate-python-client.readthedocs.io/en/v3.26.2/weaviate.html',
constructUrl: (baseUrl, ref) => `${baseUrl}#weaviate.${ref}`,
'https://weaviate-python-client.readthedocs.io/en/v3.26.2',
constructUrl: (baseUrl, ref) => `${baseUrl}/${ref}`,
icon: '/img/site/logo-py.svg',
},
ts: {
Expand Down
2 changes: 1 addition & 1 deletion tests/docker-compose-anon-2.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ services:
- '8080'
- --scheme
- http
image: cr.weaviate.io/semitechnologies/weaviate:1.27.5
image: cr.weaviate.io/semitechnologies/weaviate:1.28.0-rc.0
ports:
- 8090:8080
- 50061:50051
Expand Down
2 changes: 1 addition & 1 deletion tests/docker-compose-anon-bind.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ services:
- '8080'
- --scheme
- http
image: cr.weaviate.io/semitechnologies/weaviate:1.27.5
image: cr.weaviate.io/semitechnologies/weaviate:1.28.0-rc.0
ports:
- 8380:8080
- 50351:50051
Expand Down
2 changes: 1 addition & 1 deletion tests/docker-compose-anon-clip.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ services:
- '8080'
- --scheme
- http
image: cr.weaviate.io/semitechnologies/weaviate:1.27.5
image: cr.weaviate.io/semitechnologies/weaviate:1.28.0-rc.0
ports:
- 8280:8080
- 50251:50051
Expand Down
2 changes: 1 addition & 1 deletion tests/docker-compose-anon.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ services:
- '8080'
- --scheme
- http
image: cr.weaviate.io/semitechnologies/weaviate:1.27.5
image: cr.weaviate.io/semitechnologies/weaviate:1.28.0-rc.0
ports:
- 8080:8080
- 50051:50051
Expand Down
6 changes: 3 additions & 3 deletions tests/docker-compose-three-nodes.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ services:
- '8080'
- --scheme
- http
image: cr.weaviate.io/semitechnologies/weaviate:1.27.5
image: cr.weaviate.io/semitechnologies/weaviate:1.28.0-rc.0
restart: on-failure:0
ports:
- "8180:8080"
Expand Down Expand Up @@ -39,7 +39,7 @@ services:
- '8080'
- --scheme
- http
image: cr.weaviate.io/semitechnologies/weaviate:1.27.5
image: cr.weaviate.io/semitechnologies/weaviate:1.28.0-rc.0
restart: on-failure:0
ports:
- "8181:8080"
Expand Down Expand Up @@ -71,7 +71,7 @@ services:
- '8080'
- --scheme
- http
image: cr.weaviate.io/semitechnologies/weaviate:1.27.5
image: cr.weaviate.io/semitechnologies/weaviate:1.28.0-rc.0
restart: on-failure:0
ports:
- "8182:8080"
Expand Down
2 changes: 1 addition & 1 deletion tests/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ services:
- '8080'
- --scheme
- http
image: cr.weaviate.io/semitechnologies/weaviate:1.27.5
image: cr.weaviate.io/semitechnologies/weaviate:1.28.0-rc.0
ports:
- 8099:8080
- 50052:50051
Expand Down

0 comments on commit 9fc69da

Please sign in to comment.