Skip to main content

Upload Script

The upload script provided here can serve as a sample for working with the Data Curation API

Prerequisites​

  1. Node.js must be installed. For more information, refer to the Node.JS site (https://docs.npmjs.com/downloading-and-installing-node-js-and-npm)

Credentials​

The script requires a credentials.json file provided in the same directory. It should be formatted as follows, with the client id and secret provided for each environment.

The environments (e.g. environment1, environment2) must match what is in the upload.js file.

{
    "environment1": {
        "client_id": "",
        "client_secret": ""
    },
    "environment2": {
        "client_id": "",
        "client_secret": ""
    }
}

Script​

The script can be accessed here: upload.js Script

Configuration​

The following values must be provided at the top of the script:

  1. API_URLS for each environment
  2. OAUTH_URLS for each environment

Note that to change the number of environments or their names, they must also be edited in the code itself. Searching for environment1 will locate all areas that need to be edited.

Using the Script​

Format:
node [upload.js] [environment] [input file/folder] [output folder]

Sample:
node .\folder\upload.js environment1 C:\path\to\input C:\path\to\results

Send All Files in a Directory​

node .\src\client\upload.js environment1 C:\path\to\input C:\path\to\results

Send a Single File​

node .\src\client\upload.js environment1 C:\path\to\input\file.jpg C:\path\to\results

Example Console Output​

Upon success, there should be a results file in the output directory for every file sent.

Uploading 1 files to S3
Logged in
sale-img-rotated.bmp - 3eef8d59-bcac-45b5-afcf-fa9d9946da4f
1 of 1 uploads complete in 7.9 seconds
1 of 1 downloads complete in 81 seconds
Total time: 89.1 seconds

If there is an authentication error you will see error messages next to the file name.

Uploading 1 files to S3
Logged in
sale-text.pdf - Error {"Message":"User is not authorized to access this resource with an explicit deny"}
0 of 1 uploads complete in 0.5 seconds
0 of 0 downloads complete in 0 seconds
Total time: 0.8 seconds

If there is an error during the async processing you will need to terminate the client script as it will continue polling the results S3 bucket. There should be a UUID next to each file that can be used to debug the pipeline in AWS.