A. Setup on Expanse
Use the foldershare command line client to upload/download data from Expanse's file system. The client is already installed on the expanse cluster.
- Login to Expanse
- Create an authorization file in your home directory
- Set up your terminal environment
export MODULEPATH=$MODULEPATH:/expanse/community/seedmelab/modules module load foldershare fs="foldershare --authfile ~/myauth.json" # Add above to your ~/.bashrc file for convenience # Alternatively login to hpcshare.sdsc.edu as follows # foldershare --host hostname --username YOUR_USERNAME --password YOUR_PASSWORD
- Type "foldershare --help" in a terminal window to get help.
foldershare --help # Alternatively fs --help
- See help on usage via Python
B. FolderShare client command set
Many Linux-style file and folder commands are supported, such as:
- ls to list a folder's contents.
- stat to get the status of a file or folder.
- mkdir to make a new folder.
- rm to remove a file or folder tree.
- rmdir to remove an empty folder.
- cp to copy a file or folder.
- mv to move a file or folder.
Additional commands for data transfer and update:
- download to download a file or folder.
- upload to upload a file or folder.
- update to add or change a file or folder description.
C. Command line usage
Following description provides sample interaction via the command line interface
Create a variable to hold the following information
# Complete step A1 which loads the foldershare module fs="foldershare --host hostname --username YOUR_USERNAME --password YOUR_PASSWORD"
Create a new folder
# foldershare mkdir OPTIONS FULL_REMOTE_PATH $fs mkdir --help $fs mkdir /test
Upload a local file and folder to remote location and display progress
# foldershare upload OPTIONS LOCAL_PATH FULL_REMOTE_PATH $fs upload --help $fs upload -v /expanse/community/seedmelab/data/123.zip /test $fs upload -v --sync /expanse/community/seedmelab/data/sample-structured-data /
List contents of remote folder
$fs ls --help $fs ls /sample-structured-data $fs ls -l /sample-structured-data $fs ls -lR /
Download a remote folder and its contents and display progress
# Create temporary directory for downloading content download_dir=`mktemp -d -t hpcshare-XXXXXX` # foldershare download OPTIONS FULL_REMOTE_PATH LOCAL_PATH $fs download --help $fs download -v --sync /test $download_dir # Check download on local path ls -l $download_dir
Update description of remote file/folder
# foldershare update description 'arbitrary text' FULL_REMOTE_PATH $fs update --help $fs update description 'Hello world!' /test/123.zip
Get description of remote file/folder with stat command and different output format
# foldershare --format TYPE command # Supported TYPE: auto(default) full key value linux text $fs --format text stat /test/123.zip
Copy remote file/folder to another remote location
# foldershare cp OPTIONS /FULL_REMOTE_SOURCE_PATH /FULL_REMOTE_DESTINATION_PATH $fs cp --help # --wait option waits until the copy operation is completed, useful when subsequent commands depend on this operation $fs cp --wait /test/child /test/subfolder
Move remote folder to another remote location
# foldershare mv OPTIONS /FULL_REMOTE_SOURCE_PATH /FULL_REMOTE_DESTINATION_PATH $fs mv --help # --wait option waits until the copy operation is completed, useful when subsequent commands depend on this operation $fs mv --wait /test/child /test/subfolder
Remove remote file/folder
# foldershare rm OPTIONS /FULL_REMOTE_PATH $fs rm --help $fs rm -rf /test/subfolder