sbt is installed on your machine, and run
config.yaml for your migration using the template
config.yaml in the repository root. Read the comments throughout carefully.
The Scylla Migrator is built against Spark 2.3.1, so you'll need to run that version on your cluster.
build.sh, copy the jar from
./target/scala-2.11/scylla-migrator-assembly-0.0.1.jar and the
config.yaml you've created to the Spark master server.
Then, run this command on the Spark master server:
spark-submit --class com.scylladb.migrator.Migrator \ --master spark://<spark-master-hostname>:7077 \ --conf spark.scylla.config=<path to config.yaml> \ <path to scylla-migrator-assembly-0.0.1.jar>
This project also includes an entrypoint for comparing the source table and the target table. You can launch it as so (after performing the previous steps):
spark-submit --class com.scylladb.migrator.Validator \ --master spark://<spark-master-hostname>:7077 \ --conf spark.scylla.config=<path to config.yaml> \ <path to scylla-migrator-assembly-0.0.1.jar>
To run in the local Docker-based setup:
First start the environment:
docker-compose up -d
cqlsh in Cassandra's container and create a keyspace and a table with some data:
docker-compose exec cassandra cqlsh <create stuff>
cqlsh in Scylla's container and create the destination keyspace and table with the same schema as the source table:
docker-compose exec scylla cqlsh <create stuff>
config.yamlfile; note the comments throughout.
spark-submit in the master's container to run the job:
docker-compose exec spark-master spark-submit --class com.scylladb.migrator.Migrator \ --master spark://spark-master:7077 \ --conf spark.driver.host=spark-master \ --conf spark.scylla.config=/app/config.yaml \ /jars/scylla-migrator-assembly-0.0.1.jar
spark-master container mounts the
./target/scala-2.11 dir on
/jars and the repository root on
/app. To update the jar with new code, just run
build.sh and then run
© 2020 Anant Corporation, All Rights Reserved. All logos, trademarks and registered trademarks are the property of their respective owners.