Replication and disaster recovery (DR)

This guide details replication between two Caelum appliances: the regular copy of your shares to a secondary site, and the failover to disaster recovery (DR) if the main site becomes unavailable. It complements the administration guide (screen by screen) and the task guide (routine operations).

Replication relies on the ZFS engine: Caelum takes a snapshot of the share and sends it to the remote appliance (zfs send / zfs receive) over an encrypted SSH channel. After the first full send, subsequent sends are incremental: only the blocks changed since the previous snapshot transit the network.

Table of contents#

1. Overview
2. Declare a replication target
3. Pairing (authorize replication)
4. Create a replication job
5. Frequencies and continuous mode
6. Permission fidelity (NTFS ACL)
7. Incoming replications and disaster recovery (DR)
8. Track status and troubleshoot
9. High availability in replication mode ("Option A")
10. Pitfalls and limits to know

1. Overview#

The Replication view (sidebar) is organized into three blocks:

• Replication targets: the remote appliances you replicate to (the "where").
• Replication jobs: what you replicate and how often — a job associates a source volume with a target (the "what" and the "when").
• Incoming replications: on the destination side, the list of datasets received from other Caelum appliances, with the Verify replica action.

Two network channels are used, and it helps to distinguish them:

• The data transfer (ZFS snapshots) goes over SSH (port 22 by default, configurable), with an ed25519 key specific to the source appliance.
• Pairing and configuration replication (S3 target, LDAP users…) go over the remote appliance's HTTPS API (port 443), behind the integrated reverse proxy. It is the remote admin session that authorizes the SSH key on the target side.

2. Declare a replication target#

A target represents the remote appliance that will receive your snapshots.

1. Go to Replication → New target.
2. Fill in:
   • Name: a label for the target (e.g. secondary-site).
   • Address (IP or DNS name): the address of the remote appliance.
   • SSH user: the SSH account used for zfs receive (galilo by default — only change it if your environment requires it).
   • SSH port: 22 by default.
3. Click Test & detect. Caelum checks SSH connectivity and detects the destination pool on the remote appliance (the ZFS target location is deduced automatically from the pool; no path to enter).
4. Useful options:
   • Network compression (low-bandwidth / WAN link): enables ssh -C during the transfer. Typical gain 2-3× on text data or VM images; little benefit on a 10G LAN (and it uses CPU).
   • Source IP (optional): forces the source IP/interface of the replication flow (ssh -b). Useful if the appliance has a dedicated replication network. Empty = default system route.
5. Save.

If Test & detect reports that the target refuses the SSH key, this is normal at first contact: you must pair the two appliances (next section).

3. Pairing (authorize replication)#

At first contact, the remote appliance does not yet know the source appliance's SSH key: it refuses the transfer. Pairing authorizes this key once and for all.

When Test & detect detects this case, a pairing sub-block appears in the modal:

1. Enter an administrator account of the remote appliance and its password. These credentials serve only for authorization and are not stored: Caelum opens an HTTPS admin session to the remote appliance (port 443) and passes it the source's SSH public key to authorize.
2. Click Authorize & retest.

Once the public key is added on the target side (to its authorized_keys file), the SSH test passes and the target is ready to receive replications. Pairing is idempotent: re-authorizing an already-known key does nothing more.

4. Create a replication job#

A job associates a volume with a target and sets the frequency of the sends.

1. In Replication, under Replication jobs, create a New job.
2. Choose:
   • Target: one of the targets declared in step 2.
   • Volume to replicate: the volume sent in full (snapshot + data) to the target. The volume's ZFS dataset — and therefore all its shares — is replicated. You create one job per volume.
   • Frequency: see the next section.
3. Save.

The first send is a full (all the volume's data). The subsequent sends are incremental: Caelum takes a new snapshot and only sends the blocks changed since the last replicated snapshot (zfs send -i). This strongly reduces the transferred volume and the duration of each cycle.

To launch a cycle immediately (without waiting for the schedule), use the job's Run now action. To adjust the frequency or enable/disable a job without recreating it, open Edit replication job.

5. Frequencies and continuous mode#

A job's frequency ranges from near real-time to weekly:

• Continuous (~15 s — real-time RPO): Caelum replicates continuously (a cycle of about 15 seconds). To sustain this pace, retention is managed automatically (the last 60 snapshots are kept) and the measured RPO is shown live on the dashboard.
• Every 15 minutes, 30 minutes, hourly, every 2 / 4 / 6 / 12 hours.
• Daily (default).
• Weekly.

Choose the frequency according to your RPO (maximum acceptable data loss) and the link's throughput: on a slow WAN, a short interval may not leave time for the incremental cycle to finish. The Throughput test and the dashboard's live RPO guide you.

6. Permission fidelity (NTFS ACL)#

When the source share carries NTFS permissions from an Active Directory domain, Caelum replicates these permissions faithfully (security descriptors: owner, allow/deny ACEs, inheritance) to the destination share. After a DR failover, domain users therefore find the same permissions on the backup site.

This preservation is enabled at the job level for the concerned shares. It requires that the destination appliance is also able to resolve the domain's identities (joined to the same AD) to apply the permissions.

7. Incoming replications and disaster recovery (DR)#

On the destination side, received datasets appear in the Incoming replications block of the Replication view. As long as no failover is requested, these datasets are not served: they simply accumulate the replicated snapshots.

Verify a replica, then mount as DR#

Three actions follow one another on a received dataset:

1. Verify replica (formerly "Mount as DR"): Caelum clones the last received snapshot and mounts it read-only, attached to a dedicated DR volume. This protects the original replication flow and lets you check the integrity of the data without disrupting anything. First check that the last received replication is recent.
2. Mount as DR: recreates the real shares from the replica, to actually serve users from the backup site (case of a disaster on the main site).
3. Stop the test: destroys the clone once the check is done (or once the main site is restored).

The clone can also be refreshed (re-cloned from a more recent received snapshot) from the same view.

8. Track status and troubleshoot#

• Dashboard: a replication widget shows the active flows and, for continuous jobs, the RPO measured live.
• Jobs: each job shows its last result (success/failure, message, date) and its last snapshot sent.
• Tasks view: replication cycles appear in the operations list, with their status.

A few common causes of failure:

• SSH key not authorized on the target side: redo the pairing (section 3).
• Link saturated or too slow for the chosen frequency: space out the frequency, enable network compression, or use a dedicated source IP. The Throughput test helps with diagnosis.
• Destination pool full: free up space on the target side or reduce retention.
• Different versions: for DR and HA, keep both appliances on compatible versions.

If you get stuck, generate a support bundle (Maintenance tab, secrets masked) and see troubleshooting.

9. High availability in replication mode ("Option A")#

The Replication view described above is used to replicate and to test a replica ("Verify replica" / DR). High availability in replication mode — called "Option A" — relies on the same mechanics (each appliance keeps its own local pool, an asynchronous ZFS replication keeps the passive node as a read-only mirror) and adds two things: a data role per node and a one-gesture failover.

• Local role: Primary (serves data read-write) or Secondary (mirror) (receives replication, read-only).
• Failover: the "Fail over (failover)" button promotes the Secondary to Primary (its volumes switch to read-write, its shares are served).
• Automatic reversal: on failover, replication reverses — the new Primary now replicates toward the former one, which demotes itself.

HA Replication Assistant — Option A#

From the High availability tab, choose the Replication mode then follow the 2-step wizard:

1. Peer & replication: designate the peer appliance and the volumes to replicate (the "Replicate to the HA peer" toggle is also offered when creating a volume).
2. Summary & risks: confirm the two mandatory acknowledgements — the RPO (possible data loss = replication interval) and the split-brain risk (two primaries if the network partitions without fencing).

Once Option A is active, a failover is triggered from the High availability tab (the "Fail over (failover)" button).

10. Pitfalls and limits to know#

Going further#

• Routine operations: task guide.
• Screen by screen: administration guide.
• Solve a problem: troubleshooting and FAQ.