Medusa is a command line tool that offers the following features:
Choose and initialize the storage system:
Install Medusa on each Cassandra node:
sudo pip3 install cassandra-medusasudo pip3 install cassandra-medusa[S3]sudo pip3 install cassandra-medusa[GCS]Running the installation using sudo is necessary to have the /usr/local/bin/medusa script created properly.
If your Cassandra servers do not have internet access:
mkdir pip_dependencies && pip download -r requirements.txt -d medusa_dependencies to download the dependencies into a sub directorycp requirements.txt medusa_dependencies/tar -zcf medusa_dependencies.tar.gz medusa_dependencies to compress the dependenciespip install -r medusa_dependencies/requirements.txt --no-index --find-links to install the dependencies on the nodespython setup.py install from the cassandra-medusa source directoryCreate the /etc/medusa directory if it doesn't exist, and create a file named /etc/medusa/medusa.ini with the content of medusa-example.ini.
Modify it to match your requirements:
[cassandra]
;stop_cmd = /etc/init.d/cassandra stop
;start_cmd = /etc/init.d/cassandra start
;config_file = <path to cassandra.yaml. Defaults to /etc/cassandra/cassandra.yaml>
;cql_username = <username>
;cql_password = <password>
;nodetool_username = <my nodetool username>
;nodetool_password = <my nodetool password>
;nodetool_password_file_path = <path to nodetool password file>
;nodetool_host = <host name or IP to use for nodetool>
;nodetool_port = <port number to use for nodetool>
; Command ran to verify if Cassandra is running on a node. Defaults to "nodetool version"
;check_running = nodetool version
[storage]
storage_provider = <Storage system used for backups>
; storage_provider should be either of "local", "google_storage" or the s3_* values from
; https://github.com/apache/libcloud/blob/trunk/libcloud/storage/types.py
; Name of the bucket used for storing backups
bucket_name = cassandra_backups
; JSON key file for service account with access to GCS bucket or AWS credentials file (home-dir/.aws/credentials)
key_file = /etc/medusa/credentials
; Path of the local storage bucket (used only with 'local' storage provider)
;base_path = /path/to/backups
; Any prefix used for multitenancy in the same bucket
;prefix = clusterA
;fqdn = <enforce the name of the local node. Computed automatically if not provided.>
; Number of days before backups are purged. 0 means backups don't get purged by age (default)
max_backup_age = 0
; Number of backups to retain. Older backups will get purged beyond that number. 0 means backups don't get purged by count (default)
max_backup_count = 0
; Both thresholds can be defined for backup purge.
; Used to throttle S3 backups/restores:
transfer_max_bandwidth = 50MB/s
; Max number of downloads/uploads. Not used by the GCS backend.
concurrent_transfers = 1
; Size over which S3 uploads will be using the awscli with multi part uploads. Defaults to 100MB.
multi_part_upload_threshold = 104857600
[monitoring]
;monitoring_provider = <Provider used for sending metrics. Currently either of "ffwd" or "local">
[ssh]
;username = <SSH username to use for restoring clusters>
;key_file = <SSH key for use for restoring clusters. Expected in PEM unencrypted format.>
[checks]
;health_check = <Which ports to check when verifying a node restored properly. Options are 'cql' (default), 'thrift', 'all'.>
;query = <CQL query to run after a restore to verify it went OK>
;expected_rows = <Number of rows expected to be returned when the query runs. Not checked if not specified.>
;expected_result = <Coma separated string representation of values returned by the query. Checks only 1st row returned, and only if specified>
[logging]
; Controls file logging, disabled by default.
; enabled = 0
; file = medusa.log
; level = INFO
; Control the log output format
; format = [%(asctime)s] %(levelname)s: %(message)s
; Size over which log file will rotate
; maxBytes = 20000000
; How many log files to keep
; backupCount = 50$ medusa
Usage: medusa [OPTIONS] COMMAND [ARGS]...
Options:
-v, --verbosity Verbosity
--without-log-timestamp Do not show timestamp in logs
--config-file TEXT Specify config file
--bucket-name TEXT Bucket name
--key-file TEXT GCP credentials key file
--prefix TEXT Prefix for shared storage
--fqdn TEXT Act as another host
--ssh-username TEXT
--ssh-key-file TEXT
--help Show this message and exit.
Commands:
backup Backup Cassandra
build-index Builds indices for all present backups
and...
download Download backup
fetch-tokenmap Backup Cassandra
get-last-complete-cluster-backup
Pints the name of the latest complete
cluster...
list-backups List backups
purge Delete obsolete backups
report-last-backup Find time since last backup and print it
to...
restore-cluster Restore Cassandra cluster
restore-node Restore single Cassandra node
status Show status of backups
verify Verify the integrity of a backup$ medusa backup --help
Usage: medusa backup [OPTIONS]
Backup Cassandra
Options:
--backup-name TEXT Custom name for the backup
--stagger INTEGER Check for staggering initial backups for
duration seconds
--mode [full|differential]
--help Show this message and exit.Once Medusa is setup, you can create a full backup with the following command:
$ medusa backup --backup-name=<name of the backup>In order to perform an differential backup, add the --mode=differential argument to your command:
$ medusa backup --backup-name=<name of the backup> --mode=differentialTo perform cluster wide backups, the command must run on all nodes in the cluster, using the same backup name.
$ medusa list-backups --help
Usage: medusa list-backups [OPTIONS]
List backups
Options:
--show-all / --no-show-all List all backups in the bucket
--help Show this message and exit.List all backups for the current node/cluster:
$ medusa list-backups
2019080507 (started: 2019-08-05 07:07:03, finished: 2019-08-05 08:01:04)
2019080607 (started: 2019-08-06 07:07:04, finished: 2019-08-06 07:59:08)
2019080707 (started: 2019-08-07 07:07:04, finished: 2019-08-07 07:59:55)
2019080807 (started: 2019-08-08 07:07:03, finished: 2019-08-08 07:59:22)
2019080907 (started: 2019-08-09 07:07:04, finished: 2019-08-09 08:00:14)
2019081007 (started: 2019-08-10 07:07:04, finished: 2019-08-10 08:02:41)
2019081107 (started: 2019-08-11 07:07:04, finished: 2019-08-11 08:03:48)
2019081207 (started: 2019-08-12 07:07:04, finished: 2019-08-12 07:59:59)
2019081307 (started: 2019-08-13 07:07:03, finished: Incomplete [179 of 180 nodes])
2019081407 (started: 2019-08-14 07:07:04, finished: 2019-08-14 07:56:44)
2019081507 (started: 2019-08-15 07:07:03, finished: 2019-08-15 07:50:24)When listing backups from a cluster which is not the backed up one, please add the --show-all flag to bypass the filter and display all existing backups from storage.
$ medusa restore-node --help
Usage: medusa restore-node [OPTIONS]
Restore single Cassandra node
Options:
--temp-dir TEXT Directory for temporary storage
--backup-name TEXT Backup name [required]
--in-place Indicates if the restore happens on the node the
backup was done on.
--keep-auth Keep system_auth keyspace as found on the node
--seeds TEXT Nodes to wait for after downloading backup but
before starting C*
--verify / --no-verify Verify that the cluster is operational after the
restore completes,
--keyspace TEXT Restore tables from this keyspace
--table TEXT Restore only this table
--use-sstableloader Use the sstableloader to load the backup into the
cluster
--help Show this message and exit.In order to restore a backup on a single node, run the following command:
$ sudo medusa restore-node --backup-name=<name of the backup>Medusa will need to run with sudo as it will:
The --use-sstableloader flag will be useful for restoring data when the topology doesn't match between the backed up cluster and the restore one.
In this mode, Cassandra will not be stopped and downloaded SSTables will be loaded into Cassandra by the sstableloader. Data already present in the cluster will not be altered.
The --fqdn argument allows to force the node to act on behalf of another backup node. It can take several hostnames separated by commas in order to restore several nodes backup using the sstableloader.
The --keyspace option allows limiting the restore to all tables in the given keyspace. The --table option allows limiting the restore to just one table. The tables must be specified in the keyspace.table format. It is possible to repeat both of the options. Medusa will make an union of everything specified and restore all keyspaces and tables mentioned. The --keyspace option takes precedence, so using --keyspace ks1 and then adding --table ks1.t will not limit the restore to just one table - everything from ks1 will be restored.
$ medusa restore-cluster --help
Usage: medusa restore-cluster [OPTIONS]
Restore Cassandra cluster
Options:
--backup-name TEXT Backup name [required]
--seed-target TEXT seed of the target hosts
--temp-dir TEXT Directory for temporary storage
--host-list TEXT List of nodes to restore with the associated
target host
--keep-auth / --overwrite-auth Keep/overwrite system_auth as found on the
nodes
-y, --bypass-checks Bypasses the security check for restoring a
cluster
--verify / --no-verify Verify that the cluster is operational after
the restore completes,
--keyspace TEXT Restore tables from this keyspace
--table TEXT Restore only this table
--use-sstableloader Use the sstableloader to load the backup
into the cluster
--help Show this message and exit.In this section, we will describe procedures that apply to the following cases:
This method is by far the fastest as it replaces SSTables directly on disk.
In order to restore a backup for a full cluster, in the case where the restored cluster is the exact same as the backed up one:
$ medusa restore-cluster --backup-name=<name of the backup> --seed-target node1.domain.netMedusa will need to run without sudo as it will connect through ssh to all nodes in the cluster in order to perform remote operations. It will, by default, use the current user to connect and rely on agent forwarding for authentication (you must ssh into the server using -A to enable agent forwarding).
The --seed-target node is used to connect to Cassandra and retrieve the current topology of the cluster. This allows Medusa to map each backup to the correct node in the current topology.
The following operations will take place:
restore-node on each node in the clusterIn order to restore a backup of a full cluster but on different servers. This can be used to restore a production cluster data to a staging cluster (with the same number of nodes), or recovering from an outage where previously used hardware cannot be re-used.
$ medusa restore-cluster --backup-name=<name of the backup> --host-list /etc/medusa/restore_mapping.txtThe restore-mapping.txt file will provide the mapping between the backed up cluster nodes and the restore cluster ones. It is expected in the following CSV format: <Is it a seed node?>,<target node>,<source node>
Sample file:
True,new_node1.foo.net,old_node1.foo.net
True,new_node2.foo.net,old_node2.foo.net
False,new_node3.foo.net,old_node3.foo.netMedusa will need to run without sudo as it will connect through ssh to all nodes in the cluster in order to perform remote operations. It will, by default, use the current user to connect and rely on agent forwarding for authentication (you must ssh into the server using -A to enable agent forwarding).
restore-node on each node in the clusterBy default, Medusa will overwrite the system_auth keyspace with the backed up one. If you want to retain the existing system_auth keyspace, you'll have to run restore-cluster with the --keep-auth flag:
$ medusa restore-cluster --backup-name=<name of the backup> --host-list /etc/medusa/restore_mapping.txt --keep-authIn this section, we will describe procedures that apply to the other cases:
This case will be detected automatically by Medusa when checking the topologies, but it can be enforced by adding the --use-sstableloader flag to the restore-cluster command.
This technique allows to restore any backup on any cluster of any size, at the expense of some overhead.
Using this technique, Cassandra will not be stopped on the nodes and the following steps will happen:
The data model will be updated as follows:
Run restore-node on each node in the target cluster, passing a list of backup nodes as --fqdn and the --sstableloader flag
for each specified node in --fqdn:
If your cluster is configured with the default auto_snapshot: true then dropping the tables will trigger a snapshot that will persist their data on disk. Medusa will not clear this snapshot after restore.
The --keyspace and --table options of restore-cluster command work exactly the same way as they do for the restore-node command.
$ medusa verify --help
Usage: medusa verify [OPTIONS]
Verify the integrity of a backup
Options:
--backup-name TEXT Backup name [required]
--help Show this message and exit.Run a health check on a backup, which will verify that:
$ medusa verify --backup-name=2019090503
Validating 2019090503 ...
- Completion: OK!
- Manifest validated: OK!!In case some nodes in the cluster didn't complete the backups, you'll get the following output:
$ medusa verify --backup-name=2019081703
Validating 2019081703 ...
- Completion: Not complete!
- [127.0.0.2] Backup missing
- Manifest validated: OK!!$ medusa purge --help
Usage: medusa purge [OPTIONS]
Delete obsolete backups
Options:
--help Show this message and exit.In order to remove obsolete backups from storage, according to the configured max_backup_age and/or max_backup_count, run:
$ medusa purge
[2019-09-04 13:44:16] INFO: Starting purge
[2019-09-04 13:44:17] INFO: 25 backups are candidate to be purged
[2019-09-04 13:44:17] INFO: Purging backup 2019082513...
[2019-09-04 13:44:17] INFO: Purging backup 2019082514...
[2019-09-04 13:44:18] INFO: Purging backup 2019082515...
[2019-09-04 13:44:18] INFO: Purging backup 2019082516...
[2019-09-04 13:44:19] INFO: Purging backup 2019082517...
[2019-09-04 13:44:19] INFO: Purging backup 2019082518...
[2019-09-04 13:44:19] INFO: Purging backup 2019082519...
[2019-09-04 13:44:20] INFO: Purging backup 2019082520...
[2019-09-04 13:44:20] INFO: Purging backup 2019082521...
[2019-09-04 13:44:20] INFO: Purging backup 2019082522...
[2019-09-04 13:44:21] INFO: Purging backup 2019082523...
[2019-09-04 13:44:21] INFO: Purging backup 2019082600...
[2019-09-04 13:44:21] INFO: Purging backup 2019082601...
[2019-09-04 13:44:22] INFO: Purging backup 2019082602...
[2019-09-04 13:44:22] INFO: Purging backup 2019082603...
[2019-09-04 13:44:23] INFO: Purging backup 2019082604...
[2019-09-04 13:44:23] INFO: Purging backup 2019082605...
[2019-09-04 13:44:23] INFO: Purging backup 2019082606...
[2019-09-04 13:44:24] INFO: Purging backup 2019082607...
[2019-09-04 13:44:24] INFO: Purging backup 2019082608...
[2019-09-04 13:44:24] INFO: Purging backup 2019082609...
[2019-09-04 13:44:25] INFO: Purging backup 2019082610...
[2019-09-04 13:44:25] INFO: Purging backup 2019082611...
[2019-09-04 13:44:25] INFO: Purging backup 2019082612...
[2019-09-04 13:44:26] INFO: Purging backup 2019082613...
[2019-09-04 13:44:26] INFO: Cleaning up orphaned files...
[2019-09-04 13:45:59] INFO: Purged 652 objects with a total size of 3.74 MBSince SSTables and meta files are stored in different places for differential backups, the purge is a two step process:
data directory. All SSTables present in the data directory but absent from all manifests will get deleted in that step.$ medusa status --help
Usage: medusa status [OPTIONS]
Show status of backups
Options:
--backup-name TEXT Backup name [required]
--help Show this message and exit.Outputs a summary of a specific backup status:
$ medusa status --backup-name=2019090503
2019090503
- Started: 2019-09-05 03:53:04, Finished: 2019-09-05 04:49:52
- 32 nodes completed, 0 nodes incomplete, 0 nodes missing
- 163256 files, 12.20 TB$ medusa report-last-backup --help
Usage: medusa report-last-backup [OPTIONS]
Find time since last backup and print it to stdout :return:
Options:
--push-metrics Also push the information via metrics
--help Show this message and exit.This command will display several informations on the latest backup:
$ medusa report-last-backup
[2019-09-04 12:56:15] INFO: Latest node backup finished 18746 seconds ago
[2019-09-04 12:56:18] INFO: Latest complete backup:
[2019-09-04 12:56:18] INFO: - Name: 2019090407
[2019-09-04 12:56:18] INFO: - Finished: 18173 seconds ago
[2019-09-04 12:56:19] INFO: Latest backup:
[2019-09-04 12:56:19] INFO: - Name: 2019090407
[2019-09-04 12:56:19] INFO: - Finished: True
[2019-09-04 12:56:19] INFO: - Details - Node counts
[2019-09-04 12:56:19] INFO: - Complete backup: 180 nodes have completed the backup
[2019-09-04 12:58:47] INFO: - Total size: 94.69 TiB
[2019-09-04 12:58:55] INFO: - Total files: 5168096When used with --push-metrics, Medusa will push completion metrics to the configured monitoring system.
© 2020 Anant Corporation, All Rights Reserved. All logos, trademarks and registered trademarks are the property of their respective owners. .