AImageLab SRV

Table of Contents

Data Transfer

We support multiple ways to transfer Data in and out of the cluster and the method you choose depends on the amount of Data to transfer and the speed of your connection.
File transfers from a local system can be done through command-line tools such as secure copy (scp), secure ftp (sftp) and rsync. Some command-line tools may be unavailable on Windows, though alternative applications exist. (e.g., Git Bash, FileZilla, Bitvise SSH Client)

Secure Shell Copy

Command line terminal access is provided via the SSH protocol, while command line file transfer is available with sftp (Secure File Transfer Protocol) and scp (Secure Copy).

to/from the Frontends (ailb-login-xx.ing.unimore.it):

  • This would be for simple file transfers that can complete within a 10 CPU-minutes timeframe

from the Compute Nodes:

  • This would be for file transfers that can complete within a 24 hour window (default walltime, some users may have access to QoS that have longer max walltime)
  • This will only work for outbound transfers because compute nodes are not exposed externally
  • You must have an active job with an allocated compute node to get access to the node

to/from the Data Mover service (ailb-data.ing.unimore.it):

  • Ideal solution for heavy file transfers that can require more than 24 hours

Command line SCP

The Secure Copy utility, scp, can send data to and fetch data from a remote server.

In the examples here, replace <path-to-file> with the path of the file you wish to copy, [username] with your username, and <target-path> with the full path to the directory you would like to send the file to.

# Copying files from a local workstation to Frontend Servers
scp <path-to-file> [username]@ailb-login-xxx.ing.unimore.it:<target-path>

Windows users can access scp through PowerShell or Git bash or using a GUI application like FileZilla.

Interactive file transfer with SFTP

The sftp utility is an interactive alternative to scp that allows multiple, bi-directional transfer operations in a single
session. Within an sftp session, a series of domain-specific file system commands can be used to navigate, move, remove, and copy data
between a local system and CCR resources.

sftp [username]@ailb-login-xxx.ing.unimore.it

We can then use various commands to traverse and manipulate both local and remote file systems.

Command Function Example
cd Changes the directory of the remote computer cd remote_directory
lcd Changes the directory of the local computer lcd local_directory
ls Lists the contents of the remote directory ls
lls Lists the contents of the local directory lls
pwd Prints working directory of the remote computer pwd
lpwd Prints working directory of the local computer lpwd
get Copies a file from the remote directory to the local directory get remote_file
put Copies a file from the local directory to the remote directory put local_file
exit Closes the connection to the remote computer and exits the program exit
help Displays application information on using commands help

Rsync

While scp is useful for simple file copy operations, the rsync utility can be used to synchronize files and directories across two locations. This can often lead to efficiencies in repeat-transfer scenarios, as rsync only copies files that are different between the source and target locations (and can even transfer partial files when only part of a file has changed). This can be very useful in reducing the amount of copies you may perform when synchronizing two datasets.

In the examples here, replace <path-to-file> with the path of the file you wish to copy, [username] with your username, and <target-path> with the full path to the directory you would like to send the file to.

# Synchronizing from a local workstation to the cluster
rsync -r <path-to-directory> [username]@ailb-login-xxx.ing.unimore.it:<target-path>

rsync is not available on Windows by default, but may be installed individually or as part of Windows Subsystem for Linux (WSL).

More reading

Last updated: August 13, 2024