Data roundtripping

Overview

Teaching: 15 min
Exercises: 5 min

Questions

• How do I move data into the cloud?

• How do I get my analysis results back to my computer?

Objectives

• Transfer data into a cloud session

• Transfer data out of a cloud session

Moving data is simple

(but not always easy!)

Now that you’re on the cloud, you’ll need data. There are two main places you can get data from: your local machine, or other machines in the cloud, like NCBI.

How you get your data depends on where the data is right now.

Getting data from the cloud

There are two programs that will download data from a remote server to your local (or remote) machine: wget and curl. They were designed to do slightly different tasks by default, so you’ll need to give the programs somewhat different options to get the same behaviour, but they are mostly interchangeable.

• wget is short for “world wide web get”, and it’s basic function is to download web pages or data at a web address.

• cURL is a pun, it is suppose to be read as “see URL”, so it’s basic function is to display webpages or data at a web address.

Which one you need to use mostly depends on your operating system, as most computers will only have one or the other installed by default.

Let’s say you want to download some data from Ensembl. We’re going to download a very small tab-delimited file that just tells us what data is available on the Ensembl bacteria server. Before we can start our download, we need to know whether we’re using curl or wget.

To see which program you have type:

$ which curl
$ which wget

which is a BASH program that looks through everything you have installed, and tells you what folder it is installed to. If it can’t find the program you asked for, it returns nothing, i.e. gives you no results.

On Mac OSX, you’ll likely get the following output:

$ which curl

/usr/bin/curl

$ which wget

$

This output means that you have curl installed, but not wget.

Once you know whether you have curl or wget use one of the following commands to download the file:

$ cd
$ wget ftp://ftp.ensemblgenomes.org/pub/release-37/bacteria/species_EnsemblBacteria.txt

or

$ cd
$ curl -O ftp://ftp.ensemblgenomes.org/pub/release-37/bacteria/species_EnsemblBacteria.txt

Since we wanted to download the file rather than just view it, we used wget without any modifiers. With curl however, we had to use the -O flag, which simultaneously tells curl to download the page instead of showing it to us and specifies that it should save the file using the same name it had on the server: species_EnsemblBacteria.txt

It’s important to note that both curl and wget download to the computer that the command line belongs to. So, if you are logged into AWS on the command line and execute the curl command above in the AWS terminal, the file will be downloaded to your AWS machine, not your local one.

Moving files between your laptop and your instance

What if the data you need is on your local computer, but you need to get it into the cloud? There are also several ways to do this, but it’s always easier to start the transfer locally. This means if you’re using a transfer program, it needs to be installed on your local machine, not on your instance. If you’re typing into a terminal, the terminal should not be logged into your instance, it should be showing your local computer.

These directions are platform specific so please follow the instructions for your system:

Please select the platform you wish to use for the exercises: AWS_UNIX AWS_Windows

> cd Downloads

C:\User\your-pc-username\Downloads> pscp.exe local_file.txt dcuser@ip.address:/home/dcuser/

C:\User\your-pc-username\Downloads> pscp.exe dcuser@ip.address:/home/dcuser/dc_workshop/results/fastqc_untrimmed_reads/SRR097977_fastqc.zip C:\User\your-pc-username\Downloads

scp <file I want to move> <where I want to move it>

$ scp <local file> <AWS instance>

$ scp <AWS instance> <local file>

$  scp local_file.txt dcuser@ip.address:/home/dcuser/

$ find ~ -name *.zip

$ scp dcuser@ip.address:/home/dcuser/dc_workshop/results/fastqc_untrimmed_reads/SRR097977_fastqc.zip ~/Downloads

Key Points

• No matter which way you want to move data, it’s easier to start the transfer from your local machine