Gdrive

Contents

1. Summary and Version Information
2. Configuring gdrive
3. Using gdrive
4. Errors/issues/etc
   1. Rate limit errors
   2. Expired or revoked token errors

Summary and Version Information

Notes:
^*: Packages labelled as "available" on an HPC cluster means that it can be used on the compute nodes of that cluster. Even software not listed as available on an HPC cluster is generally available on the login nodes of the cluster (assuming it is available for the appropriate OS version; e.g. RedHat Linux 6 for the two Deepthought clusters). This is due to the fact that the compute nodes do not use AFS and so have copies of the AFS software tree, and so we only install packages as requested. Contact us if you need a version listed as not available on one of the clusters.

In general, you need to prepare your Unix environment to be able to use this software. To do this, either:

• tap TAPFOO

• module load MODFOO

where TAPFOO and MODFOO are one of the tags in the tap and module columns above, respectively. The tap command will print a short usage text (use -q to supress this, this is needed in startup dot files); you can get a similar text with module help MODFOO. For more information on the tap and module commands.

For packages which are libraries which other codes get built against, see the section on compiling codes for more help.

Tap/module commands listed with a version of current will set up for what we considered the most current stable and tested version of the package installed on the system. The exact version is subject to change with little if any notice, and might be platform dependent. Versions labelled new would represent a newer version of the package which is still being tested by users; if stability is not a primary concern you are encouraged to use it. Those with versions listed as old set up for an older version of the package; you should only use this if the newer versions are causing issues. Old versions may be dropped after a while. Again, the exact versions are subject to change with little if any notice.

In general, you can abbreviate the module tags. If no version is given, the default current version is used. For packages with compiler/MPI/etc dependencies, if a compiler module or MPI library was previously loaded, it will try to load the correct build of the package for those packages. If you specify the compiler/MPI dependency, it will attempt to load the compiler/MPI library for you if needed.

Configuring gdrive

NOTE:You might wish to look at the rclone command instead of gdrive for accessing your Google drive. In addition to supporting many Cloud storage providers, it does a better job of keeping within Google data rate limitations.

Before you can use gdrive, you must create a token to allow gdrive access to your Google G Suite drive. To do this:

1. Run module load gdrive if you have not already done so to add it to your path.
2. By default, gdrive will store your token in the ~/.gdrive directory. If you wish to store the token elsewhere (e.g. you use ~/.gdrive for your personal Google drive, and want a different config file for your team drive), you can either add the flags --config PATH_TO_CONFIG_DIR or set the environmental variable GDRIVE_CONFIG_DIR to the new config dir.
3. Run gdrive about. This command will print out an URL and prompt for a verification code. Enter the URL in a browser, authenticate to Google and allow gdrive access to your account, and cut and past the verification code presented in the browser to the gdrive prompt.

BE SURE TO PROTECT YOUR gdrive configruation directory (either ~/.gdrive or whereever you instructed gdrive to use with the --config flag or GDRIVE_CONFIG_DIR environmental variable). Anyone with read access to that directory can access your Google drive as you.

Using gdrive

You will need to module load gdrive to add gdrive to your path if you have not already done so.

gdrive help will provide some usage instructions, but basically you can use gdrive download FILEID to download files from Google drive to the local filesystem and gdrive upload PATH to upload files from the local file system to Google drive.

Errors/issues/etc

Rate limit errors

Gdrive is prone to fail with errors like Failed to upload file: googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded , especially if you are moving large amounts of data as is common among HPC users.

Unfortunately, there is no fix for this in gdrive at the time of writing. Instead, we recommend that users switch to the rclone command. In addition to providing consistent command line support to many Cloud storage providers, it has built in exponential backoff features which will keep the data rate within Google's requirements.

Expired or revoked token errors

If you start getting errors about expired or revoked tokens, you might need to refresh your gdrive token. To do this, just delete the file token_v2.json in your gdrive configuration directory (normally ~/.gdrive unless you set GDRIVE_CONFIG_DIR or are using the --config flag).

The just repeat the gdrive about process about to refresh your token.