Usage from Expanse cluster at SDSC


A. Setup on Expanse

Use the foldershare command line client to upload/download data from Expanse's file system. The client is already installed on the expanse cluster.

  1. Login to Expanse
  2. Create an authorization file in your home directory
  3. Set up your terminal environment
    export MODULEPATH=$MODULEPATH:/expanse/community/seedmelab/modules 
    module load foldershare
    fs="foldershare --authfile ~/myauth.json"
    # Add above to your ~/.bashrc file for convenience
    # Alternatively login to as follows   
    # foldershare --host hostname --username YOUR_USERNAME --password  YOUR_PASSWORD
  4. Type "foldershare --help" in a terminal window to get help. 
     foldershare --help
     # Alternatively
     fs --help
  5. See help on usage via Python 

B. FolderShare client command set

Many Linux-style file and folder commands are supported, such as:

  • ls to list a folder's contents.
  • stat to get the status of a file or folder.
  • mkdir to make a new folder.
  • rm to remove a file or folder tree.
  • rmdir to remove an empty folder.
  • cp to copy a file or folder.
  • mv to move a file or folder.

Additional commands for data transfer and update:

  • download to download a file or folder.
  • upload to upload a file or folder.
  • update to add or change a file or folder description.


C. Command line usage 

Following description provides sample interaction via the command line interface

Create a variable to hold the following information

# Complete step A1 which loads the foldershare module
fs="foldershare --host hostname --username YOUR_USERNAME --password YOUR_PASSWORD"

Create a new folder 

# foldershare mkdir OPTIONS FULL_REMOTE_PATH
$fs mkdir --help
$fs mkdir /test

Upload a local file and folder to remote location and display progress

$fs upload --help
$fs upload -v /expanse/community/seedmelab/data/ /test
$fs upload -v --sync /expanse/community/seedmelab/data/sample-structured-data / 

List contents of remote folder

$fs ls --help
$fs ls /sample-structured-data
$fs ls -l /sample-structured-data
$fs ls -lR /

Download a remote folder and its contents and display progress

# Create temporary directory for downloading content
download_dir=`mktemp -d -t hpcshare-XXXXXX`
$fs download --help
$fs download -v --sync /test $download_dir
# Check download on local path
ls -l $download_dir 

Update description of remote file/folder 

# foldershare update description 'arbitrary text' FULL_REMOTE_PATH
$fs update --help
$fs update description 'Hello world!' /test/ 

Get description of remote file/folder with stat command and different output format

# foldershare --format TYPE command 
# Supported TYPE: auto(default) full key value linux text
$fs --format text stat /test/ 

Copy remote file/folder to another remote location

$fs cp --help
# --wait option waits until the copy operation is completed, useful when subsequent commands depend on this operation
$fs cp --wait /test/child /test/subfolder

Move remote folder to another remote location

$fs mv --help
# --wait option waits until the copy operation is completed, useful when subsequent commands depend on this operation
$fs mv --wait /test/child /test/subfolder 

Remove remote file/folder 

# foldershare rm OPTIONS /FULL_REMOTE_PATH 
$fs rm --help
$fs rm -rf /test/subfolder