file_catalog

Store file metadata information in a file catalog

Prerequisites

To get the prerequisites necessary for the file catalog:

pip install -r requirements.txt

Running the server

To start an instance of the server running:

python -m file_catalog

Configuration

All configuration is done using environment variables. To get the list of possible configuration parameters and their defaults, run

python -m file_catalog --show-config-spec

Interface

The primary interface is an HTTP server. TLS and other security hardening mechanisms are handled by a reverse proxy server as for normal web applications.

Browser

Requests to the main url / are browsable like a standard website. They will use javascript to activate the REST API as necessary.

REST API

Requests with urls of the form /api/RESOURCE can access the REST API. Responses are in HAL JSON format.

File-Entry Fields

File-Metadata Schema:

• See types.py

Mandatory Fields:

• uuid (provided by File Catalog)
• logical_name
• locations (with at least one non-empty URL)
• file_size
• checksum.sha512

Route: /api/files

Resource representing the collection of all files in the catalog.

Method: GET

Obtain list of files

REST-Query Parameters

• limit
• start
• logical_name (shortcut parameter)
• directory (shortcut parameter)
• filename (shortcut parameter)
• logical-name-regex (shortcut parameter)
• run_number (shortcut parameter)
• dataset (shortcut parameter)
• event_id (shortcut parameter)
• processing_level (shortcut parameter)
• season (shortcut parameter)
• query
• keys
• all-keys (shortcut parameter)
• max_time_ms

HTTP Response Status Codes

• 200: Response contains collection of file resources
• 400: Bad request (query parameters invalid)
• 429: Too many requests (if server is being hammered)
• 500: Unspecified server error
• 503: Service unavailable (maintenance, etc.)

Method: POST

Create a new file or add a replica

If a file exists and the checksum is the same, a replica is added. If the checksum is different a conflict error is returned.

REST-Body

• See File-Entry Fields

HTTP Response Status Codes

• 200: Replica has been added. Response contains link to file resource
• 201: Response contains link to newly created file resource
• 400: Bad request (metadata failed validation)
• 409: Conflict (if the file-version already exists); includes link to existing file
• 429: Too many requests (if server is being hammered)
• 500: Unspecified server error
• 503: Service unavailable (maintenance, etc.)

Method: DELETE

Not supported

Method: PUT

Not supported

Method: PATCH

Not supported

Route: /api/files/{uuid}

Resource representing the metadata for a file in the file catalog.

Method: GET

Obtain file metadata information

REST-Query Parameters

• None

HTTP Response Status Codes

• 200: Response contains metadata of file resource
• 404: Not Found (file resource does not exist)
• 429: Too many requests (if server is being hammered)
• 500: Unspecified server error
• 503: Service unavailable (maintenance, etc.)

Method: POST

Not supported

Method: DELETE

Delete the metadata for the file

REST-Query Parameters

• None

HTTP Response Status Codes

• 204: No Content (file resource is successfully deleted)
• 404: Not Found (file resource does not exist)
• 429: Too many requests (if server is being hammered)
• 500: Unspecified server error
• 503: Service unavailable (maintenance, etc.)

Method: PUT

Fully update/replace file metadata information

REST-Body

• See File-Entry Fields

HTTP Response Status Codes

• 200: Response indicates metadata of file resource has been updated/replaced
• 404: Not Found (file resource does not exist) + link to “files” resource for POST
• 409: Conflict (if updating an outdated resource - use ETAG hash to compare)
• 429: Too many requests (if server is being hammered)
• 500: Unspecified server error
• 503: Service unavailable (maintenance, etc.)

Method: PATCH

Partially update/replace file metadata information

The JSON provided as body to PATCH need not contain all the keys, only the need to be updated. If a key is provided with a value null, then that key can be removed from the metadata.

REST-Body

• See File-Entry Fields

HTTP Response Status Codes

• 200: Response indicates metadata of file resource has been updated/replaced
• 404: Not Found (file resource does not exist) + link to “files” resource for POST
• 409: Conflict (if updating an outdated resource - use ETAG hash to compare)
• 429: Too many requests (if server is being hammered)
• 500: Unspecified server error
• 503: Service unavailable (maintenance, etc.)

More About REST-Query Parameters

limit

• positive integer; number of results to provide (default: 10000)
• NOTE: The server may honor the limit parameter. In cases where the server does not honor the limit parameter, it should do so by providing fewer resources (limit should be considered the client’s upper limit for the number of resources in the response).

start

• non-negative integer; result at which to start at (default: 0)
• NOTE: the server should honor the start parameter
• TIP: increment start by limit to paginate through many results

query

• MongoDB query; use to specify file-entry fields/ranges; forwarded to MongoDB daemon

keys

• a |-delimited string-list of keys; defines what fields to include in result(s)
• ex: "foo|bar|baz"
• different routes/methods define differing defaults
• NOTE: there is no performance hit for including more fields
• see all-keys

max_time_ms

• non-negative integer OR None; timeout to kill long queries in MILLISECONDS
• overrides the default timeout of 600000 ms (10 minutes)
• None indicates no timeout (this can hang the server -- you have been warned)

Shortcut Parameters: logical-name-regex, logical_name, directory, filename

In decreasing order of precedence...

• logical-name-regex

  • query by regex pattern (at your own risk... performance-wise)
  • equivalent to: query: {"logical_name": {"$regex": p}}

• logical_name

  • equivalent to: query["logical_name"]

• directory

  • query by absolute directory filepath
  • equivalent to: query: {"logical_name": {"$regex": "^/your/path/.*"}}
  • NOTE: a trailing-/ will be inserted if you don't provide one
  • TIP: use in conjunction with filename (ie: /root/dirs/.../filename)

• filename

  • query by filename (no parent-directory path needed)
  • equivalent to: query: {"logical_name": {"$regex": ".*/your-file$"}}
  • NOTE: a leading-/ will be inserted if you don't provide one
  • TIP: use in conjunction with directory (ie: /root/dirs/.../filename)

Shortcut Parameter: run_number

• equivalent to: query["run.run_number"]

Shortcut Parameter: dataset

• equivalent to: query["iceprod.dataset"]

Shortcut Parameter: event_id

• equivalent to: query: {"run.first_event":{"$lte": e}, "run.last_event":{"$gte": e}}

Shortcut Parameter: processing_level

• equivalent to: query["processing_level"]

Shortcut Parameter: season

• equivalent to: query["offline_processing_metadata.season"]

Shortcut Parameter: all-keys

• boolean (True/"True"/"true"/1); include all fields in result(s)
• NOTE: there is no performance hit for including more fields
• TIP: this is preferred over querying /api/files, grabbing the uuid, then querying /api/files/{uuid}

Development

Establishing a development environment

Follow these steps to get a development environment for the File Catalog:

cd ~/projects
git clone [email protected]:WIPACrepo/file_catalog.git
cd file_catalog
./setupenv.sh

MongoDB Instance for Testing

This command will spin up a disposable MongoDB instance using Docker:

docker run \
    --detach \
    --name test-mongo \
    --network=host \
    --rm \
    circleci/mongo:latest-ram

Building a Docker container

The following commands will create a Docker container for the file-catalog:

docker build -t file-catalog:{version} -f Dockerfile .
docker image tag file-catalog:{version} file-catalog:latest

Where {version} is found in file_catalog/__init__py; e.g.:

__version__ = '1.2.0'       # For {version} use: 1.2.0

Pushing Docker containers to local registry in Kubernetes

Here are some commands to get the Docker container pushed to our Docker register in our Kubernetes cluster:

kubectl -n kube-system port-forward $(kubectl get pods --namespace kube-system -l "app=docker-registry,release=docker-registry" -o jsonpath="{.items[0].metadata.name}") 5000:5000 &
docker tag file-catalog:{version} localhost:5000/file-catalog:{version}
docker push localhost:5000/file-catalog:{version}