Usage from Expanse cluster at SDSC

A. Setup on Expanse

Use the foldershare command line client to upload/download data from Expanse's file system. The client is already installed on the expanse cluster.

Login to Expanse
Create an authorization file in your home directory

Set up your terminal environment

export MODULEPATH=$MODULEPATH:/expanse/community/seedmelab/modules 
module load foldershare
fs="foldershare --authfile ~/myauth.json"
# Add above to your ~/.bashrc file for convenience
# Alternatively login to hpcshare.sdsc.edu as follows   
# foldershare --host hostname --username YOUR_USERNAME --password  YOUR_PASSWORD

Type "foldershare --help" in a terminal window to get help.
```
 foldershare --help
 # Alternatively
 fs --help
```
See help on usage via Python

B. FolderShare client command set

Many Linux-style file and folder commands are supported, such as:

ls to list a folder's contents.
stat to get the status of a file or folder.
mkdir to make a new folder.
rm to remove a file or folder tree.
rmdir to remove an empty folder.
cp to copy a file or folder.
mv to move a file or folder.

Additional commands for data transfer and update:

download to download a file or folder.
upload to upload a file or folder.
update to add or change a file or folder description.

C. Command line usage

Following description provides sample interaction via the command line interface

Create a variable to hold the following information

# Complete step A1 which loads the foldershare module
fs="foldershare --host hostname --username YOUR_USERNAME --password YOUR_PASSWORD"

Create a new folder

# foldershare mkdir OPTIONS FULL_REMOTE_PATH
$fs mkdir --help
$fs mkdir /test

Upload a local file and folder to remote location and display progress

# foldershare upload OPTIONS LOCAL_PATH FULL_REMOTE_PATH
$fs upload --help
$fs upload -v /expanse/community/seedmelab/data/123.zip /test
$fs upload -v --sync /expanse/community/seedmelab/data/sample-structured-data /

List contents of remote folder

$fs ls --help
$fs ls /sample-structured-data
$fs ls -l /sample-structured-data
$fs ls -lR /

Download a remote folder and its contents and display progress

# Create temporary directory for downloading content
download_dir=`mktemp -d -t hpcshare-XXXXXX`
# foldershare download OPTIONS FULL_REMOTE_PATH LOCAL_PATH 
$fs download --help
$fs download -v --sync /test $download_dir
# Check download on local path
ls -l $download_dir

Update description of remote file/folder

# foldershare update description 'arbitrary text' FULL_REMOTE_PATH
$fs update --help
$fs update description 'Hello world!' /test/123.zip

Get description of remote file/folder with stat command and different output format

# foldershare --format TYPE command 
# Supported TYPE: auto(default) full key value linux text
$fs --format text stat /test/123.zip

Copy remote file/folder to another remote location

# foldershare cp OPTIONS /FULL_REMOTE_SOURCE_PATH /FULL_REMOTE_DESTINATION_PATH 
$fs cp --help
# --wait option waits until the copy operation is completed, useful when subsequent commands depend on this operation
$fs cp --wait /test/child /test/subfolder

Move remote folder to another remote location

# foldershare mv OPTIONS /FULL_REMOTE_SOURCE_PATH /FULL_REMOTE_DESTINATION_PATH 
$fs mv --help
# --wait option waits until the copy operation is completed, useful when subsequent commands depend on this operation
$fs mv --wait /test/child /test/subfolder

Remove remote file/folder

# foldershare rm OPTIONS /FULL_REMOTE_PATH 
$fs rm --help
$fs rm -rf /test/subfolder