Skip to content

Commit

Permalink
Address comments
Browse files Browse the repository at this point in the history
Signed-off-by: Ben <[email protected]>
  • Loading branch information
Neon-White committed Nov 26, 2024
1 parent dfa3840 commit a18af53
Showing 1 changed file with 32 additions and 40 deletions.
72 changes: 32 additions & 40 deletions doc/bucket-replication.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ In general, a replication policy is a JSON-compliant string which defines an arr
- When a filter with prefix is provided - only objects keys that match the prefix will be replicated

Behind the scenes, bucket replication esstentialy works by comparing object lists. NooBaa lists all objects on the source and target buckets, and checks which objects are missing on the target bucket. It then copies the missing objects from the source to the target bucket (while adhering to any provided rules).

It is possible to accelerate replication by utilizing logs - at the time of writing, AWS S3 server access logging or Azure Monitor. This mechanism allows NooBaa to copy only objects that have been created or modified since the feature was turned on, while the rest replicate in the background. This allows users to get up to speed with recent objects, while the classic replication mechanism catches up with the rest.

## Bucket Class Replication
Expand All @@ -30,32 +31,38 @@ In addition, when the bucketclass is backed by namespacestores, each policy can
- (Necessary on AWS) `{"logs_location": {"logs_bucket": <>}}` - this field should be set to the location of the AWS S3 server access logs

## Examples
Note that the example poicies below can also be saved as files and passed to the NooBaa CLI. In that case, it's necessary to omit the outer single quotes.
### AWS replication policy:

`'{"rules":[{"rule_id":"aws-rule-1", "destination_bucket":"first.bucket", "filter": {"prefix": "a."}}]}'`

### AWS replication policy with log optimization:

`'{"rules":[{"rule_id":"aws-rule-1", "destination_bucket":"first.bucket", "filter": {"prefix": "a."}}], "log_replication_info": {"logs_location": {"logs_bucket": "logsarehere"}}}'`

### Azure replication policy with log optimization, deletion and version sync:

`'{"rules":[{"rule_id":"azure-rule-1", "sync_deletions": true, "sync_versions": false, "destination_bucket":"first.bucket"}], "log_replication_info": {"endpoint_type": "AZURE"}}'`

### Namespace bucketclass creation with replication to first.bucket:

With the NooBaa CLI -
There are two ways to apply a bucket/bucketclass replication policy:

The first is with the NooBaa CLI (requires the policies to be saved as a separate JSON file and passed to the CLI) - for example:
#### Namespace bucketclass creation via the NooBaa CLI with replication to first.bucket:
```shell
noobaa -n app-namespace bucketclass create namespace-bucketclass single bc --resource azure-blob-ns --replication-policy=/path/to/json-file.json
```
/path/to/json-file.json is the path to a JSON file which defines the replication policy, e.g. -
>/path/to/json-file.json is the path to a JSON file which defines the replication policy, e.g. -
```json
{"rules":[{ "rule_id": "rule-1", "destination_bucket": "first.bucket", "filter": {"prefix": "d"}} ]}
```

With a YAML file:
The second is by applying a YAML file containing the policy.
It's also possible to apply a replication policy to OBCs even after their creation (although the same thing is not possible with bucketclasses).
For OBCs, the policy needs to be provided under the `spec.additionalConfig.replicationPolicy` property. For example:

#### ObjectBucketClaim creation via YAML file with replication to first.bucket:
```yaml
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: my-bucket-claim
namespace: appnamespace
spec:
generateBucketName: my-bucket
storageClassName: noobaa.noobaa.io
additionalConfig:
replicationPolicy: '[{ "rule_id": "rule-2", "destination_bucket": "first.bucket", "filter": {"prefix": "bc"}}]'
```
For bucketclasses, the policy should be provided under `spec.replicationPolicy`- for example:
#### Bucketclass creation via YAML file with replication to first.bucket:
```yaml
apiVersion: noobaa.io/v1alpha1
kind: BucketClass
Expand All @@ -70,28 +77,13 @@ spec:
replicationPolicy: '[{ "rule_id": "rule-1", "destination_bucket": "first.bucket", "filter": {"prefix": "ba"}}]'
```

### OBC with specific Replication Policy
A few more rules for example:

Applications that require a bucket to have a specific replication policy can create an OBC and add to the claim
the `spec.additionalConfig.replication-policy` property.

/path/to/json-file.json is the path to a JSON file which defines the replication policy

Example:
#### AWS replication policy:
`'{"rules":[{"rule_id":"aws-rule-1", "destination_bucket":"first.bucket", "filter": {"prefix": "a."}}]}'`

```bash
noobaa obc create my-bucket-claim -n appnamespace --replication-policy /path/to/json-file.json
```
#### AWS replication policy with log optimization:
`'{"rules":[{"rule_id":"aws-rule-1", "destination_bucket":"first.bucket", "filter": {"prefix": "a."}}], "log_replication_info": {"logs_location": {"logs_bucket": "logsarehere"}}}'`

```yaml
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
name: my-bucket-claim
namespace: appnamespace
spec:
generateBucketName: my-bucket
storageClassName: noobaa.noobaa.io
additionalConfig:
replicationPolicy: '[{ "rule_id": "rule-2", "destination_bucket": "first.bucket", "filter": {"prefix": "bc"}}]'
```
#### Azure replication policy with log optimization, deletion and version sync:
`'{"rules":[{"rule_id":"azure-rule-1", "sync_deletions": true, "sync_versions": false, "destination_bucket":"first.bucket"}], "log_replication_info": {"endpoint_type": "AZURE"}}'`

0 comments on commit a18af53

Please sign in to comment.