mirror: Watch does not work for chained/neasted mirrors

[Jakob3xD]

Expected behavior

I have 3 minio servers in different locations. The first server should replicate to the second server and the second server to the third server. This is done via mc mirror --watch test-2 test-1 and mc mirror --watch test-3 test-2. It should mirror all PUT and DELETE requests from the test-3 minio up to the test-1 minio.

Use case:
The "source" minio server following named test-3 is located in EU and the other two minio servers are located in the USA.
To reduce traffic between USA and EU I want the test-2 minio server to be the source of the test-1 minio server.

Actual behavior

PUT request performed to the test-3 minio only gets mirrored to the test-2 minio and not up the test-1 minio.
On the other hand DELETE requests are performed correctly.

Steps to reproduce the behavior

1. Create 3 Minio Servers:

root@minio-test-1:~# docker create --name test-minio --network host -e MINIO_ROOT_USER=root-0423 -e MINIO_ROOT_PASSWORD=xyz -e MINIO_PROMETHEUS_AUTH_TYPE=public -e MINIO_BROWSER=off  minio/minio server --address 10.0.0.2:9000 /data
root@minio-test-2:~# docker create --name test-minio --network host -e MINIO_ROOT_USER=root-0423 -e MINIO_ROOT_PASSWORD=xyz -e MINIO_PROMETHEUS_AUTH_TYPE=public -e MINIO_BROWSER=off  minio/minio server --address 10.0.0.3:9000 /data
root@minio-test-3:~# docker create --name test-minio --network host -e MINIO_ROOT_USER=root-0423 -e MINIO_ROOT_PASSWORD=xyz -e MINIO_PROMETHEUS_AUTH_TYPE=public -e MINIO_BROWSER=off  minio/minio server --address 10.0.0.4:9000 /data

2. Create a minio config

{
    "aliases": {
        "test-1": {
            "accessKey": "root-0423",
            "api": "s3v4",
            "path": "on",
            "secretKey": "xyz",
            "url": "http://10.0.0.2:9000"
        },
        "test-2": {
            "accessKey": "root-0423",
            "api": "s3v4",
            "path": "on",
            "secretKey": "xyz",
            "url": "http://10.0.0.3:9000"
        },
        "test-3": {
            "accessKey": "root-0423",
            "api": "s3v4",
            "path": "on",
            "secretKey": "xyz",
            "url": "http://10.0.0.4:9000"
        }
    },
    "version": "10"
}

3. Create the mirror containers

root@minio-test-1:~# docker create --name minio-mirror -v $PWD/.mc/:/root/.mc/ --network host minio/mc:latest mirror --json --watch --overwrite test-2 test-1
root@minio-test-2:~# docker create --name minio-mirror -v $PWD/.mc/:/root/.mc/ --network host minio/mc:latest mirror --json --watch --overwrite test-3 test-2

4. Create test bucket on minio test-3
5. Copy a file to minio test-3

You can now perform mc ls test-2/test and mc ls test-1/test. The object will be listed on test-2 but not on test-1.
If you copy the file to test-1 by hand and the delete the object on test-3 all servers will be empty.

mc --version

mc version RELEASE.2023-04-12T02-21-51Z (commit-id=1843717c57fb87612469b7610344a7d49d97a497)
Runtime: go1.20.3 linux/amd64
Copyright (c) 2015-2023 MinIO, Inc.
License GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html

System information

minio/mc Docker Image

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.