This guide details replication between two Caelum appliances: the regular copy of your shares to a secondary site, and the failover to disaster recovery (DR) if the main site becomes unavailable. It complements the administration guide (screen by screen) and the task guide (routine operations).
Replication relies on the ZFS engine: Caelum takes a snapshot of the
share and sends it to the remote appliance (zfs send / zfs receive) over an
encrypted SSH channel. After the first full send, subsequent sends are
incremental: only the blocks changed since the previous snapshot transit the
network.
Caelum 0.65.9: unqualified beta. The campaign contains no two-node Caelum cluster acceptance test. Validate failure handling, peer isolation, actual RPO and restored access outside production before relying on these modes. sends, replica checks and failover.
Table of contents#
- Overview
- Declare a replication target
- Pairing (authorize replication)
- Create a replication job
- Frequencies and continuous mode
- Permission fidelity (NTFS ACL)
- Incoming replications and disaster recovery (DR)
- Track status and troubleshoot
- High availability in replication mode ("Option A")
- Pitfalls and limits to know
1. Overview#
The Replication screen of the previous interface (/legacy) is organized
into three blocks:
- Peer appliances: the remote appliances you replicate to (the "where").
- Replication jobs: what you replicate and how often — a job associates a source volume with a target (the "what" and the "when").
- Incoming replications: on the destination side, the list of datasets received from other Caelum appliances, with the Verify replica action.
The Storage → Replication screen of the new console shows the replication state and copy lag (RPO), the jobs (with Send now), the replication destinations, the node's public key (Pair a target, Authorise a key), incoming replication (Mount for recovery), recovery readiness measured on both nodes and the Switch over to this node button.
Jobs define the datasets sent. Continuity mode adds roles and manual promotion; it does not guarantee peer isolation.
Two network channels are used, and it helps to distinguish them:
- The data transfer (ZFS snapshots) goes over SSH (port 22 by default,
configurable), with an
ed25519key specific to the source appliance. - Pairing and configuration replication (S3 target, LDAP users…) go over the remote appliance's HTTPS API (port 443), behind the integrated reverse proxy. It is the remote admin session that authorizes the SSH key on the target side.
2. Declare a replication target#
A target represents the remote appliance that will receive your snapshots.
- In the previous interface (
/legacy), go to Replication → Add appliance. - Fill in:
- Name: a label for the target (e.g.
secondary-site). - Address (IP or DNS name): the address of the remote appliance.
- SSH user: the SSH account used for
zfs receive(galiloby default — only change it if your environment requires it). - SSH port:
22by default.
- Name: a label for the target (e.g.
- Click Test & detect. Caelum checks SSH connectivity and detects the destination pool on the remote appliance (the ZFS target location is deduced automatically from the pool; no path to enter).
- Useful options:
- Network compression: measure its benefit on your data and link; no universal gain is claimed.
- Source IP (optional): forces the source IP/interface of the replication
flow (
ssh -b). Useful if the appliance has a dedicated replication network. Empty = default system route.
- Save.
If Test & detect reports that the target refuses the SSH key, this is normal at first contact: you must pair the two appliances (next section).
3. Pairing (authorize replication)#
At first contact, the remote appliance does not yet know the source appliance's SSH key: it refuses the transfer. Pairing authorizes this key once and for all.
When Test & detect detects this case, a pairing sub-block appears in the modal:
- Enter an administrator account of the remote appliance and its password. These credentials serve only for authorization and are not stored: Caelum opens an HTTPS admin session to the remote appliance (port 443) and passes it the source's SSH public key to authorize.
- Click Authorize & retest.
Once the public key is added on the target side (to its authorized_keys file),
the SSH test passes and the target is ready to receive replications. Pairing is
idempotent: re-authorizing an already-known key does nothing more.
In the new console, Storage → Replication → Pair a target also registers a peer and checks that both channels respond (HTTPS for pairing, SSH for transfer). If the peer's SSH identity changes (peer reinstalled or replaced), the screen flags it and offers Trust the new key — never applied automatically.
Pairing stores no password. Only the source appliance's public key is kept on the destination side. To revoke a source, remove its key from the remote appliance.
4. Create a replication job#
A job associates a volume with a target and sets the frequency of the sends.
- In Replication, under Replication jobs, create a New job.
- Choose:
- Target: one of the targets declared in step 2.
- Volume to replicate: the volume sent in full (snapshot + data) to the target. The volume's ZFS dataset — and therefore all its shares — is replicated. You create one job per volume.
- Frequency: see the next section.
- Save.
The first send is a full (all the volume's data). The subsequent
sends are incremental: Caelum takes a new snapshot and only sends the blocks
changed since the last replicated snapshot (zfs send -i). This strongly reduces
the transferred volume and the duration of each cycle.
To launch a cycle immediately (without waiting for the schedule), use the job's Run now action (or Send now in the new console). To adjust the frequency or enable/disable a job without recreating it, open Edit replication job.
The real throughput to a target can be measured via the Throughput test (transfer of 100 MiB + measurement of latency and of source/target disk writes). It helps you choose a realistic frequency: an incremental cycle must not last longer than the chosen interval.
5. Frequencies and continuous mode#
Transfers remain asynchronous ZFS snapshots, even when scheduled closely together. The configured frequency is not a guaranteed RPO.
Effective RPO depends on the age of the last successfully received snapshot. Slow transfers, failures or an unavailable peer can extend it beyond the configured interval. Check results and snapshots actually received, then choose frequency and retention based on your tests. Promotion remains manual; this is not continuous synchronous replication.
6. Permission fidelity (NTFS ACL)#
Active Directory integration was tested in an SMB/NFS scenario. Distinguish the historical model from “applied permissions” mode: their interaction with system permissions differs. NFS AUTH_SYS is not equivalent to rich NTFS ACLs. Before enabling cross-protocol access, test owner, DENY, inheritance, nested groups and resolved identities. Campaign case T-243 concerns the tested 0.65.7 scenario, not every 0.65.9 mode.
For recovery, check identity resolution and effective access on the destination node. Security descriptors alone do not constitute exhaustive fidelity validation or qualification of all cross-protocol access.
7. Incoming replications and disaster recovery (DR)#
On the destination side, received datasets appear in the Incoming replications block of the Replication view. As long as no failover is requested, these datasets are not served: they simply accumulate the replicated snapshots.
Verify a replica, then mount as DR#
Two actions follow one another on a received dataset (previous interface):
- Verify replica: Caelum clones the last received snapshot and mounts it read-only, attached to a dedicated DR volume. This protects the original replication flow and lets you check the integrity of the data without disrupting anything. First check that the last received replication is recent.
- Stop test: destroys the clone once the check is done (or once the main site is restored).
In the new console, Mount for recovery (Incoming replication block) mounts a received dataset read-only, without promoting the node; the recovery mount can be refreshed onto the latest received snapshot or removed. To actually serve users from the backup site (disaster on the main site), use the failover described in section 9.
The Disaster recovery — import from the source appliance block imports the three tables that ZFS replication does not carry: archived-object pointers, S3 destination, LDAP users and groups.
Document and test your DR procedure cold, outside production. Failover involves organizational choices (read-only vs write on the backup site, rollback once the main site is restored) that depend on your context. Regularly check that the received replications are up to date: an obsolete received dataset means a degraded RPO.
8. Track status and troubleshoot#
- Replication screen (new console): the aggregated replication state and the copy lag (RPO), measured on the receiving node.
- Jobs: each job shows its last result (success/failure, message, date) and its last snapshot sent.
- Recovery readiness (new console): what failover requires, measured on both nodes (AD join, S3 secret, NFS engine, version).
- Replication channel: the screen flags a broken SSH channel ("Sending to the peer fails") even if the jobs look healthy.
A few common causes of failure:
- SSH key not authorized on the target side: redo the pairing (section 3).
- Link saturated or too slow for the chosen frequency: space out the frequency, enable network compression, or use a dedicated source IP. The Throughput test helps with diagnosis.
- Destination pool full: free up space on the target side or reduce retention.
- Different versions: for DR and HA, keep both appliances on compatible versions.
If you get stuck, download the support bundle (Settings → Maintenance or Diagnostics → Support, secrets masked) and see troubleshooting.
9. High availability in replication mode ("Option A")#
Each appliance retains its local pool. Data is copied using asynchronous ZFS snapshots and promotion is manual.
Before promotion:
- Isolate the peer and verify it is no longer serving data; do not rely solely on the interface acknowledgement.
- Check the age of the last received snapshot, covered datasets/jobs and acceptable data loss.
- Check AD identities, S3 secrets, access and archived files on the recovery node.
- Prepare failback in a dedicated validation exercise outside production.
Delivered-workflow limitation: local promotion precedes the remote demotion request. If that request fails, local promotion is not undone. A split-brain risk acknowledgement is neither fencing nor proof that the peer has been excluded.
10. Pitfalls and limits to know#
- S3 secrets: they can be transported and re-encrypted when the exchange channel is valid. Re-enter them only if transfer or adoption fails; verify recall and archiving before resuming service.
- Pools: shared-storage HA covers the primary pool. Replication coverage depends on datasets and jobs actually configured, not a blanket exclusion of additional pools.
- RPO: age of the last successfully received snapshot, which can exceed the configured interval. Replication promotion is manual.
- Fencing: isolate the peer before promotion; the code does not guarantee prior remote demotion. The shared-storage script includes fencing, but the wizard cannot configure it. No qualified transparent failover is promised.
- Permissions and archives: test AD identities and user access; follow restoration and catalog replay limitations for archived files.
- Validation: the campaign does not qualify a two-node Caelum cluster. Do not rely on this mode without failure and failback testing.
Going further#
- Routine operations: task guide.
- Screen by screen: administration guide.
- Solve a problem: troubleshooting and FAQ.