Gdrive: Gdrive is a command line utility for interacting with Google Drive

Contents

  1. Overview of package
  2. Overview of package
    1. General usage
  3. Availability of package by cluster
  4. Configuring gdrive
  5. Using gdrive
  6. Errors/issues/etc
    1. Rate limit errors
    2. Expired or revoked token errors

Overview of package

General information about package
Package: Gdrive
Description: Gdrive is a command line utility for interacting with Google Drive
For more information: https://github.com/prasmussen/gdrive
Categories:
License: OpenSource (MIT)

General usage information

Gdrive is a command line utility for interacting with Google Drive.

Start by running: gdrive about

You will be prompted for a verification code. Follow the URL listed in a web browser, authenticating to your google account, and type in the verification code you receive. This will cause a token to be placed in the .gdrive folder in your home directory. PROTECT THIS DIRECTORY --- anyone with access to it can access your google drive as you. Subsequent commands will use that token.

WARNING
*** DEPRECATED ***
The gdrive package/command is deprecated. Please use the rclone package/command instead of gdrive for accessing your Google drive. In addition to having consistent support for many Cloud storage providers including Google drive, we find that rclone supports Google drive better than the gdrive command, in particular with regards to keeping within Google data rate limitations.

Available versions of the package Gdrive, by cluster

This section lists the available versions of the package Gdriveon the different clusters.

Available versions of Gdrive on the Deepthought2 cluster (RHEL6) [DEPRECATED]

Available versions of Gdrive on the Deepthought2 cluster (RHEL6) [DEPRECATED]
Version Module tags CPU(s) optimized for GPU ready?
2.1.0 gdrive/2.1.0 x86_64 N

Configuring gdrive

WARNING
*** DEPRECATED ***
The gdrive package/command is deprecated. Please use the rclone package/command instead of gdrive for accessing your Google drive. In addition to having consistent support for many Cloud storage providers including Google drive, we find that rclone supports Google drive better than the gdrive command, in particular with regards to keeping within Google data rate limitations.

Before you can use gdrive, you must create a token to allow gdrive access to your Google G Suite drive. To do this:

  1. Run module load gdrive if you have not already done so to add it to your path.
  2. By default, gdrive will store your token in the ~/.gdrive directory. If you wish to store the token elsewhere (e.g. you use ~/.gdrive for your personal Google drive, and want a different config file for your team drive), you can either add the flags --config PATH_TO_CONFIG_DIR or set the environmental variable GDRIVE_CONFIG_DIR to the new config dir.
  3. Run gdrive about. This command will print out an URL and prompt for a verification code. Enter the URL in a browser, authenticate to Google and allow gdrive access to your account, and cut and past the verification code presented in the browser to the gdrive prompt.
WARNING
BE SURE TO PROTECT YOUR gdrive configruation directory (either ~/.gdrive or whereever you instructed gdrive to use with the --config flag or GDRIVE_CONFIG_DIR environmental variable). Anyone with read access to that directory can access your Google drive as you.

Using gdrive

You will need to module load gdrive to add gdrive to your path if you have not already done so.

gdrive help will provide some usage instructions, but basically you can use gdrive download FILEID to download files from Google drive to the local filesystem and gdrive upload PATH to upload files from the local file system to Google drive.

Errors/issues/etc

Rate limit errors

Gdrive is prone to fail with errors like Failed to upload file: googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded , especially if you are moving large amounts of data as is common among HPC users.

Unfortunately, there is no fix for this in gdrive at the time of writing. Instead, we recommend that users switch to the rclone command. In addition to providing consistent command line support to many Cloud storage providers, it has built in exponential backoff features which will keep the data rate within Google's requirements.

Expired or revoked token errors

If you start getting errors about expired or revoked tokens, you might need to refresh your gdrive token. To do this, just delete the file token_v2.json in your gdrive configuration directory (normally ~/.gdrive unless you set GDRIVE_CONFIG_DIR or are using the --config flag).

The just repeat the gdrive about process about to refresh your token.