Usage from Expanse cluster at SDSC

Terminal

A. Setup on Expanse

Use the foldershare command line client to upload/download data from Expanse's file system.

  1. Login to Expanse and set up your terminal environment as follows
    export MODULEPATH=$MODULEPATH:/expanse/community/seedmelab/modules 
    module load foldershare
    fs="foldershare --host hostname --username YOUR_USERNAME --password  YOUR_PASSWORD"
    # Add above to your ~/.bashrc file for convenience
    
  2. Type "foldershare --help" in a terminal window to get help. 
     foldershare --help
     # Alternatively
     fs --help

B. FolderShare client command set

Many Linux-style file and folder commands are supported, such as:

  • ls to list a folder's contents.
  • stat to get the status of a file or folder.
  • mkdir to make a new folder.
  • rm to remove a file or folder tree.
  • rmdir to remove an empty folder.
  • cp to copy a file or folder.
  • mv to move a file or folder.

Additional commands for data transfer and update:

  • download to download a file or folder.
  • upload to upload a file or folder.
  • update to add or change a file or folder description.

 

C. Command line usage 

Following description provides sample interaction via the command line interface

Create a variable to hold the following information

# Complete step A1 which loads the foldershare module
fs="foldershare --host hostname --username YOUR_USERNAME --password YOUR_PASSWORD"

Create a new folder 

# foldershare mkdir OPTIONS FULL_REMOTE_PATH
$fs mkdir --help
$fs mkdir /test

Upload a local file and folder to remote location and display progress

# foldershare upload OPTIONS LOCAL_PATH FULL_REMOTE_PATH
$fs upload --help
$fs upload -v /expanse/community/seedmelab/data/123.zip /test
$fs upload -v --sync /expanse/community/seedmelab/data/sample-structured-data / 

List contents of remote folder

$fs ls --help
$fs ls /sample-structured-data
$fs ls -l /sample-structured-data
$fs ls -lR /

Download a remote folder and its contents and display progress

# Create temporary directory for downloading content
download_dir=`mktemp -d -t hpcshare-XXXXXX`
# foldershare download OPTIONS FULL_REMOTE_PATH LOCAL_PATH 
$fs download --help
$fs download -v --sync /test $download_dir
# Check download on local path
ls -l $download_dir 

Update description of remote file/folder 

# foldershare update description 'arbitrary text' FULL_REMOTE_PATH
$fs update --help
$fs update description 'Hello world!' /test/123.zip 

Get description of remote file/folder with stat command and different output format

# foldershare --format TYPE command 
# Supported TYPE: auto(default) full key value linux text
$fs --format text stat /test/123.zip 

Copy remote file/folder to another remote location

# foldershare cp OPTIONS /FULL_REMOTE_SOURCE_PATH /FULL_REMOTE_DESTINATION_PATH 
$fs cp --help
# --wait option waits until the copy operation is completed, useful when subsequent commands depend on this operation
$fs cp --wait /test/child /test/subfolder

Move remote folder to another remote location

# foldershare mv OPTIONS /FULL_REMOTE_SOURCE_PATH /FULL_REMOTE_DESTINATION_PATH 
$fs mv --help
# --wait option waits until the copy operation is completed, useful when subsequent commands depend on this operation
$fs mv --wait /test/child /test/subfolder 

Remove remote file/folder 

# foldershare rm OPTIONS /FULL_REMOTE_PATH 
$fs rm --help
$fs rm -rf /test/subfolder