DST configuration local to snapshots

[Harvie]

Hello,
i have recently changed IP address of my DST_A, but for some reason, local snapshots already have org.znapzend:dst_a property copied localy, which really messed up my plan to simply switch to different ip address. I think this should be inherited, not set localy to each snapshot or dataset. Otherwise it causes problems when something changes regarding the DST configuration. eg. the snapshots were kept indefinitely. that is despite using --cleanOffline, which might be bug on its own.

# zfs get all tank/vps@2023-08-06-020000
...
tank/vps@2023-08-06-020000  org.znapzend:dst_a_synced   1                                       local
tank/vps@2023-08-06-020000  org.znapzend:dst_a          [email protected]:tank/node1/vps  local    <--- WHY IS THIS NOT INHERITED???
tank/vps@2023-08-06-020000  org.znapzend:mbuffer        /usr/bin/mbuffer                        inherited from tank
tank/vps@2023-08-06-020000  org.znapzend:mbuffer_size   1G                                      inherited from tank
tank/vps@2023-08-06-020000  org.znapzend:dst_a_plan     1hours=>15minutes,3days=>1hours         inherited from tank
tank/vps@2023-08-06-020000  org.znapzend:src_plan       1hours=>15minutes,3days=>1hours         inherited from tank
tank/vps@2023-08-06-020000  org.znapzend:post_znap_cmd  off                                     inherited from tank
tank/vps@2023-08-06-020000  org.znapzend:enabled        on                                      inherited from tank
tank/vps@2023-08-06-020000  org.znapzend:recursive      on                                      inherited from tank
tank/vps@2023-08-06-020000  org.znapzend:zend_delay     0                                       inherited from tank
tank/vps@2023-08-06-020000  org.znapzend:tsformat       %Y-%m-%d-%H%M%S                         inherited from tank
tank/vps@2023-08-06-020000  org.znapzend:pre_znap_cmd   off                                     inherited from tank

What do you think about this? is this a bug? how am is supposed to change dst_a without breaking things?

[jimklimov]

Well, they are snapshots - so refer to the values that were valid at the moment of their creation, and should be immutable afterwards. I don't think these are practically used by znapzend or anyone though - only the "local"-ly (not "received", and in many cases not "inherited", etc.) stored attributes in "live" datasets or volumes are what matters.

UPDATE: This statement was disproven, see below.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Harvie]

bump

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Harvie]

bump

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Harvie]

bump

[jimklimov]

Re-reading the issue, and remembering the code (not online now) I think my earlier comment could be wrong.

The new idea is (based on --cleanOffline implication) that these props allow to track which snapshots have been successfully replicated to remote server, rotated media taken from a vault, etc.

In that regard, pinpointing where known copies of these bits really are is better done with snapshot-local props (where available) than with a value du-jour inherited from live dataset.

As for auto-cleaning - did you debug suspected flaws? I don't see logs, analysis, PRs...

[jimklimov]

Ran a few experiments with znapzend --debug --runonce (and some --features specific to the hosts) program sending to a storage host called simply znapzend via ~/.ssh/config:

...
[2024-12-05 10:06:27.65889] [15696] [info] starting work on backupSet rpool
# zfs list -H -r -o name -t filesystem,volume rpool
[2024-12-05 10:06:27.81908] [15696] [debug] sending snapshots from rpool to znapzend:pond/export/DUMP/znapzend/ci-oo/rpool
[2024-12-05 10:06:27.81964] [15696] [debug] Are we sending "--since"? since=="0", skipIntermediates=="0", forbidDestRollback=="0", justCreated=="false"
# zfs list -H -o name -t snapshot -s creation -d 1 rpool
# ssh -o batchMode=yes -o ConnectTimeout=30 znapzend zfs list -H -o name -t snapshot -s creation -d 1 pond/export/DUMP/znapzend/ci-oo/rpool
# zfs send -Lce -I 'rpool@znapzend-auto-2024-12-05T00:00:00Z' 'rpool@znapzend-auto-2024-12-05T10:06:16Z'|ssh -o batchMode=yes -o ConnectTimeout=30 znapzend 'mbuffer -q -s 256k -W 600 -m 1G|zfs recv -u -F pond/export/DUMP/znapzend/ci-oo/rpool'
# zfs list -H -o name -t snapshot rpool@znapzend-auto-2024-12-05T10:06:16Z
# zfs set org.znapzend:dst_0=znapzend:pond/export/DUMP/znapzend/ci-oo/rpool rpool@znapzend-auto-2024-12-05T10:06:16Z
# zfs set org.znapzend:dst_0_synced=1 rpool@znapzend-auto-2024-12-05T10:06:16Z
...

Note here the last two lines, with zfs set - these properties are indeed assigned into the snapshot dataset (so it is possible, I may have posted wrong comments on this earlier). This happens just after the zfs send has completed and indeed marks that a remote copy of this snapshot exists and where.

The snapshot dataset does have some properties inherited from the live dataset by default, as well as these two local to it:

:; zfs get all rpool@znapzend-auto-2024-12-05T10:06:16Z | grep org.znapzend
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:dst_0_synced        1                                                     local
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:dst_0               znapzend:pond/export/DUMP/znapzend/ci-oo/rpool  local
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:src_mbuffer         off                                                   inherited from rpool
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:tsformat            znapzend-auto-%Y-%m-%dT%H:%M:%SZ                      inherited from rpool
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:post_znap_cmd       off                                                   inherited from rpool
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:mbuffer_size        1G                                                    inherited from rpool
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:recursive           on                                                    inherited from rpool
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:pre_znap_cmd        off                                                   inherited from rpool
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:enabled             on                                                    inherited from rpool
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:dst_0_mbuffer       mbuffer                                               inherited from rpool
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:dst_0_mbuffer_size  1G                                                    inherited from rpool
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:zend_delay          0                                                     inherited from rpool
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:src_mbuffer_size    128M                                                  inherited from rpool
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:dst_0_plan          1months=>1weeks,1years=>1months,10years=>6months      inherited from rpool
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:dst_0_pstcmd        off                                                   inherited from rpool
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:dst_0_precmd        off                                                   inherited from rpool
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:src_plan            1months=>1weeks,1years=>1months,10years=>6months      inherited from rpool
rpool@znapzend-auto-2024-12-05T10:06:16Z  org.znapzend:mbuffer             off                                                   inherited from rpool

So at least we have a data point on what happens in practice, when, and sort of why.

I am not ready to elaborate on how/why this "messes up clean-up" - that indeed may be a bug of its own, probably can be tracked here and relevant part of debug log is likely welcome.

As a quick workaround I might suggest defining remote destinations in ~/.ssh/config - keeping the actual IP addresses, host names, users, bespoke SSH keys etc. there, so the wording in your dst remains the same. See details in https://github.com/oetiker/znapzend#running-with-restricted-shell

Anyhow we can not pass ALL of those in a simple dst spec string as a property. And if changes of such string break stuff - more reason to avoid that... not that it isn't a bug to try and fix, or explain why it is not a bug but is by design - e.g. to avoid loss of snapshots not replicated to a known still-alive destination (the bug-fix is for docs then, and maybe for znapzendzetup tool to handle dst reformulations by updating existing local snapshots).

CC @oetiker : WDYT?

[Harvie]

I was running znapzend for couple of years, but then i was forced to change ip adresses of the servers during migration. i think its not that uncommon usecase. But it seems in znapzend there is no easy way to do it without breaking stuff or completely reconfiguring (and resyncing all data).

[oetiker]

hmmm I was not aware that the properties set in the snapshot should have any bearings on anything ...