Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Annotator Trainability #244

Open
wants to merge 28 commits into
base: main
Choose a base branch
from
Open

Add Annotator Trainability #244

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
1f51e6b
Merge pull request #29 from Sage-Bionetworks/develop
tschaffter Oct 10, 2020
0ab43fb
Merge branch 'main' of https://github.com/Sage-Bionetworks/nlp-sandbo…
boyleconnor Jun 15, 2021
04c329d
Merge branch 'main' of https://github.com/Sage-Bionetworks/nlp-sandbo…
boyleconnor Jun 22, 2021
89f97db
Add TrainingStatus Schema
boyleconnor Jun 22, 2021
f332421
Add Training Endpoints to Date, Location, and Name Annotators
boyleconnor Jun 22, 2021
720fc2c
Revert "Add Training Endpoints to Date, Location, and Name Annotators"
boyleconnor Jun 22, 2021
eb58bbe
Add Info re: Trained Annotator to Schema
boyleconnor Jun 23, 2021
833ae72
Add Description to TrainingStatus
boyleconnor Jun 23, 2021
f12ad40
Add Endpoint for Starting Annotator Training
boyleconnor Jun 23, 2021
c484380
Add Training Status URL
boyleconnor Jun 23, 2021
07db3b8
Add Model-Loading Endpoint
boyleconnor Jun 23, 2021
f29ed6f
Refactor bad `schema` Property of Tool
boyleconnor Jun 23, 2021
1032397
Add `trainable` Property to Example `Tool`
boyleconnor Jun 23, 2021
7b684dc
Remove Empty Request & Response Bodies
boyleconnor Jun 23, 2021
e8639ba
Add Training Paths to Annotator Specs
boyleconnor Jun 23, 2021
0c2c61b
Add Training tag to Annotator Merge Exceptions
boyleconnor Jun 23, 2021
f111d13
Add Training Tag to nlpsandbox no-merge List
boyleconnor Jun 23, 2021
0ca3068
Switch TrainingStatus Enum to Kebab Case
boyleconnor Jun 24, 2021
83c510b
Add Dataset URIs to TrainedAnnotatorStatus.yaml
boyleconnor Jun 25, 2021
59f1c6e
Add Dataset URIs to TrainedAnnotatorStatus.yaml
boyleconnor Jun 25, 2021
be6b00b
Annotator -> Tool
boyleconnor Jun 29, 2021
0151f12
Update Tool.yaml
boyleconnor Jul 1, 2021
a4f7ac9
Remove TrainedToolStatus from Tool Info
boyleconnor Jul 1, 2021
26fe354
`Annotator` -> `Tool`
boyleconnor Jul 1, 2021
f1b1d62
Update and rename annotatorTrainingStatus.yaml to toolTrainingStatus.…
boyleconnor Jul 1, 2021
628cd7b
Create trainedToolStatus Endpoint
boyleconnor Jul 1, 2021
24d94fe
Remove Erroneous References to "Annotator"
boyleconnor Jul 3, 2021
4161b41
Rename trainAnnotator to trainTool
boyleconnor Jul 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions openapi/_internal/nlpsandbox/openapi-merge.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,26 +5,31 @@ inputs:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/person-name-annotator.openapi.yaml
operationSelection:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/location-annotator.openapi.yaml
operationSelection:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/id-annotator.openapi.yaml
operationSelection:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/contact-annotator.openapi.yaml
operationSelection:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/phi-deidentifier.openapi.yaml
operationSelection:
excludeTags:
Expand All @@ -35,6 +40,7 @@ inputs:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/data-node.openapi.yaml
operationSelection:
excludeTags:
Expand Down
6 changes: 6 additions & 0 deletions openapi/_internal/tool/openapi-merge.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,26 +5,31 @@ inputs:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/person-name-annotator.openapi.yaml
operationSelection:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/location-annotator.openapi.yaml
operationSelection:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/id-annotator.openapi.yaml
operationSelection:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/contact-annotator.openapi.yaml
operationSelection:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/phi-deidentifier.openapi.yaml
operationSelection:
excludeTags:
Expand All @@ -35,4 +40,5 @@ inputs:
excludeTags:
- Tool
- HealthCheck
- Training
output: openapi.yaml
9 changes: 9 additions & 0 deletions openapi/commons/components/schemas/Tool.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,13 @@ properties:
description: The URL to the homepage of the tool
type: string
format: uri
trainable:
description: Whether the tool implements trainability
type: boolean
default: false
trainedAnnotatorStatus:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The term "annotator" refers to a specific type of tool. Prefer the term "tool" here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not efficient to query the entire Tool object just to know this status. I'd keep Tool for static information, and create a specific endpoint to get training status information with a light response.

description: Information about the trained component of the annotator
$ref: TrainedAnnotatorStatus.yaml
type:
$ref: ../parameters/ToolType.yaml
apiVersion:
Expand All @@ -50,6 +57,7 @@ required:
- url
- type
- apiVersion
- trainable
example:
name: awesome-nlp-tool
version: 1.0.6
Expand All @@ -59,5 +67,6 @@ example:
author: Awesome Team
authorEmail: [email protected]
url: https://example.com
trainable: false
type: nlpsandbox:date-annotator
apiVersion: 1.2.0
9 changes: 9 additions & 0 deletions openapi/commons/components/schemas/TrainedAnnotatorStatus.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
type: object
boyleconnor marked this conversation as resolved.
Show resolved Hide resolved
description: Information about whether the annotator is using a trained model, and which one it is
properties:
loaded:
description: Whether the annotator is using a trained model
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would be more descriptive here. For example, whether the trained model is ready to be used for inference. Also the "loading" status may be more related to inference that "training status". Consider moving this property to an "inference status" schema? What about introducing a ToolStatus - or ToolInferenceStatus - schemas? This "loaded" property could become ToolStatus.initialized = true or ToolStatus.ready = true. Here initialized or ready would be that in general the tool is ready to do what it's meant to do, i.e. generating predictions.

type: boolean
dataset:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does this value come from?

description: Which dataset the annotator's trained model was trained on
type: string
6 changes: 6 additions & 0 deletions openapi/commons/components/schemas/TrainingStatus.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
type: string
description: Whether this annotator instance has been used to generate a trained model
enum:
- not yet started
boyleconnor marked this conversation as resolved.
Show resolved Hide resolved
- ongoing
- finished
17 changes: 17 additions & 0 deletions openapi/commons/paths/annotatorTrainingStatus.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
get:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename toolTrainingStatus or simply trainingStatus since we won't train anything else than tool.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the endpoint of this path will be /tool/trainingStatus so no need to include tool again in this schema name.

Copy link
Contributor Author

@boyleconnor boyleconnor Jun 25, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about /tool/training/status ?

Copy link
Contributor Author

@boyleconnor boyleconnor Jun 25, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other one will be /tool/initialization/ and /tool/initialization/status?

tags:
- Training
summary: Get status of annotator training
description: Get status of the tool's training process (404 if annotator is not trainable)
operationId: getAnnotatorTrainingStatus
responses:
'200':
description: Success
content:
application/json:
schema:
$ref: ../components/schemas/TrainingStatus.yaml
'404':
boyleconnor marked this conversation as resolved.
Show resolved Hide resolved
$ref: ../components/responses/NotFound.yaml
'500':
$ref: ../components/responses/InternalServerError.yaml
18 changes: 18 additions & 0 deletions openapi/commons/paths/loadTrainedAnnotator.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
post:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loading a trained model is a step of an initialization process. What about having a path initializeTraining and initializeInference?

tags:
- Training
summary: Load the results of training
description: Load the results of training (e.g. a machine learning model) into memory from mounted volume
operationId: loadTrainedAnnotator
responses:
'201':
description: Trained annotator loaded
links:
GetTrainingStatus:
operationId: getAnnotatorTrainingStatus
'403':
$ref: ../components/responses/Unauthorized.yaml
'404':
$ref: ../components/responses/NotFound.yaml
'500':
$ref: ../components/responses/InternalServerError.yaml
16 changes: 16 additions & 0 deletions openapi/commons/paths/trainAnnotator.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
post:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Name it simply train.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The endpoint will be /tool/train

tags:
- Training
summary: Initiate annotator training (async)
description: Start training the tool's annotation model
operationId: trainAnnotator
responses:
'201':
description: Training initialized
links:
GetTrainingStatus:
operationId: annotatorTrainingStatus
'404':
$ref: ../components/responses/NotFound.yaml
'500':
$ref: ../components/responses/InternalServerError.yaml
6 changes: 6 additions & 0 deletions openapi/contact-annotator/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,9 @@ paths:
$ref: ../commons/paths/tool.yaml
/tool/dependencies:
$ref: ../commons/paths/dependencies.yaml
/train:
$ref: ../commons/paths/trainAnnotator.yaml
/train/status:
$ref: ../commons/paths/annotatorTrainingStatus.yaml
/train/load:
$ref: ../commons/paths/loadTrainedAnnotator.yaml
8 changes: 7 additions & 1 deletion openapi/covid-symptom-annotator/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,4 +70,10 @@ paths:
/tool:
$ref: ../commons/paths/tool.yaml
/tool/dependencies:
$ref: ../commons/paths/dependencies.yaml
$ref: ../commons/paths/dependencies.yaml
/train:
$ref: ../commons/paths/trainAnnotator.yaml
/train/status:
$ref: ../commons/paths/annotatorTrainingStatus.yaml
/train/load:
$ref: ../commons/paths/loadTrainedAnnotator.yaml
8 changes: 7 additions & 1 deletion openapi/date-annotator/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,4 +71,10 @@ paths:
/tool:
$ref: ../commons/paths/tool.yaml
/tool/dependencies:
$ref: ../commons/paths/dependencies.yaml
$ref: ../commons/paths/dependencies.yaml
/train:
$ref: ../commons/paths/trainAnnotator.yaml
/train/status:
$ref: ../commons/paths/annotatorTrainingStatus.yaml
/train/load:
$ref: ../commons/paths/loadTrainedAnnotator.yaml
8 changes: 7 additions & 1 deletion openapi/id-annotator/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,4 +69,10 @@ paths:
/tool:
$ref: ../commons/paths/tool.yaml
/tool/dependencies:
$ref: ../commons/paths/dependencies.yaml
$ref: ../commons/paths/dependencies.yaml
/train:
$ref: ../commons/paths/trainAnnotator.yaml
/train/status:
$ref: ../commons/paths/annotatorTrainingStatus.yaml
/train/load:
$ref: ../commons/paths/loadTrainedAnnotator.yaml
8 changes: 7 additions & 1 deletion openapi/location-annotator/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,4 +69,10 @@ paths:
/tool:
$ref: ../commons/paths/tool.yaml
/tool/dependencies:
$ref: ../commons/paths/dependencies.yaml
$ref: ../commons/paths/dependencies.yaml
/train:
$ref: ../commons/paths/trainAnnotator.yaml
/train/status:
$ref: ../commons/paths/annotatorTrainingStatus.yaml
/train/load:
$ref: ../commons/paths/loadTrainedAnnotator.yaml
8 changes: 7 additions & 1 deletion openapi/person-name-annotator/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,4 +69,10 @@ paths:
/tool:
$ref: ../commons/paths/tool.yaml
/tool/dependencies:
$ref: ../commons/paths/dependencies.yaml
$ref: ../commons/paths/dependencies.yaml
/train:
$ref: ../commons/paths/trainAnnotator.yaml
/train/status:
$ref: ../commons/paths/annotatorTrainingStatus.yaml
/train/load:
$ref: ../commons/paths/loadTrainedAnnotator.yaml
6 changes: 6 additions & 0 deletions openapi/phi-annotator/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,3 +87,9 @@ paths:
$ref: ../commons/paths/tool.yaml
/tool/dependencies:
$ref: ../commons/paths/dependencies.yaml
/train:
$ref: ../commons/paths/trainAnnotator.yaml
/train/status:
$ref: ../commons/paths/annotatorTrainingStatus.yaml
/train/load:
$ref: ../commons/paths/loadTrainedAnnotator.yaml