Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Annotator Trainability #244

Open
wants to merge 28 commits into
base: main
Choose a base branch
from
Open

Add Annotator Trainability #244

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
1f51e6b
Merge pull request #29 from Sage-Bionetworks/develop
tschaffter Oct 10, 2020
0ab43fb
Merge branch 'main' of https://github.com/Sage-Bionetworks/nlp-sandbo…
boyleconnor Jun 15, 2021
04c329d
Merge branch 'main' of https://github.com/Sage-Bionetworks/nlp-sandbo…
boyleconnor Jun 22, 2021
89f97db
Add TrainingStatus Schema
boyleconnor Jun 22, 2021
f332421
Add Training Endpoints to Date, Location, and Name Annotators
boyleconnor Jun 22, 2021
720fc2c
Revert "Add Training Endpoints to Date, Location, and Name Annotators"
boyleconnor Jun 22, 2021
eb58bbe
Add Info re: Trained Annotator to Schema
boyleconnor Jun 23, 2021
833ae72
Add Description to TrainingStatus
boyleconnor Jun 23, 2021
f12ad40
Add Endpoint for Starting Annotator Training
boyleconnor Jun 23, 2021
c484380
Add Training Status URL
boyleconnor Jun 23, 2021
07db3b8
Add Model-Loading Endpoint
boyleconnor Jun 23, 2021
f29ed6f
Refactor bad `schema` Property of Tool
boyleconnor Jun 23, 2021
1032397
Add `trainable` Property to Example `Tool`
boyleconnor Jun 23, 2021
7b684dc
Remove Empty Request & Response Bodies
boyleconnor Jun 23, 2021
e8639ba
Add Training Paths to Annotator Specs
boyleconnor Jun 23, 2021
0c2c61b
Add Training tag to Annotator Merge Exceptions
boyleconnor Jun 23, 2021
f111d13
Add Training Tag to nlpsandbox no-merge List
boyleconnor Jun 23, 2021
0ca3068
Switch TrainingStatus Enum to Kebab Case
boyleconnor Jun 24, 2021
83c510b
Add Dataset URIs to TrainedAnnotatorStatus.yaml
boyleconnor Jun 25, 2021
59f1c6e
Add Dataset URIs to TrainedAnnotatorStatus.yaml
boyleconnor Jun 25, 2021
be6b00b
Annotator -> Tool
boyleconnor Jun 29, 2021
0151f12
Update Tool.yaml
boyleconnor Jul 1, 2021
a4f7ac9
Remove TrainedToolStatus from Tool Info
boyleconnor Jul 1, 2021
26fe354
`Annotator` -> `Tool`
boyleconnor Jul 1, 2021
f1b1d62
Update and rename annotatorTrainingStatus.yaml to toolTrainingStatus.…
boyleconnor Jul 1, 2021
628cd7b
Create trainedToolStatus Endpoint
boyleconnor Jul 1, 2021
24d94fe
Remove Erroneous References to "Annotator"
boyleconnor Jul 3, 2021
4161b41
Rename trainAnnotator to trainTool
boyleconnor Jul 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions openapi/_internal/nlpsandbox/openapi-merge.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,26 +5,31 @@ inputs:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/person-name-annotator.openapi.yaml
operationSelection:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/location-annotator.openapi.yaml
operationSelection:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/id-annotator.openapi.yaml
operationSelection:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/contact-annotator.openapi.yaml
operationSelection:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/phi-deidentifier.openapi.yaml
operationSelection:
excludeTags:
Expand All @@ -35,6 +40,7 @@ inputs:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/data-node.openapi.yaml
operationSelection:
excludeTags:
Expand Down
6 changes: 6 additions & 0 deletions openapi/_internal/tool/openapi-merge.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,26 +5,31 @@ inputs:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/person-name-annotator.openapi.yaml
operationSelection:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/location-annotator.openapi.yaml
operationSelection:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/id-annotator.openapi.yaml
operationSelection:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/contact-annotator.openapi.yaml
operationSelection:
excludeTags:
- Tool
- HealthCheck
- Training
- inputFile: ../individuals/phi-deidentifier.openapi.yaml
operationSelection:
excludeTags:
Expand All @@ -35,4 +40,5 @@ inputs:
excludeTags:
- Tool
- HealthCheck
- Training
output: openapi.yaml
8 changes: 7 additions & 1 deletion openapi/commons/components/schemas/Tool.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@ properties:
description: The URL to the homepage of the tool
type: string
format: uri
trainable:
description: Whether the tool implements trainability
type: boolean
default: false
type:
$ref: ../parameters/ToolType.yaml
apiVersion:
Expand All @@ -50,6 +54,7 @@ required:
- url
- type
- apiVersion
- trainable
example:
name: awesome-nlp-tool
version: 1.0.6
Expand All @@ -59,5 +64,6 @@ example:
author: Awesome Team
authorEmail: [email protected]
url: https://example.com
trainable: false
type: nlpsandbox:date-annotator
apiVersion: 1.2.0
apiVersion: 1.2.0
14 changes: 14 additions & 0 deletions openapi/commons/components/schemas/TrainedToolStatus.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
type: object
description: Information about whether the tool is using a trained model, and the dataset on which the tool was trained
properties:
loaded:
description: Whether the tool is using a trained model
type: boolean
notesURI:
description: URI to first page of the notes in the dataset
type: string
format: uri
goldStandardAnnotationsURI:
description: URI to first page of gold standard annotations
type: string
format: uri
6 changes: 6 additions & 0 deletions openapi/commons/components/schemas/TrainingStatus.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
type: string
description: Whether this annotator instance has been used to generate a trained model
enum:
- not-yet-started
- ongoing
- finished
18 changes: 18 additions & 0 deletions openapi/commons/paths/loadTrainedAnnotator.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
post:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loading a trained model is a step of an initialization process. What about having a path initializeTraining and initializeInference?

tags:
- Training
summary: Load the results of training
description: Load the results of training (e.g. a machine learning model) into memory from mounted volume
operationId: loadTrainedAnnotator
responses:
'201':
description: Trained annotator loaded
links:
GetTrainingStatus:
operationId: getAnnotatorTrainingStatus
'403':
$ref: ../components/responses/Unauthorized.yaml
'404':
$ref: ../components/responses/NotFound.yaml
'500':
$ref: ../components/responses/InternalServerError.yaml
17 changes: 17 additions & 0 deletions openapi/commons/paths/toolTrainingStatus.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
get:
tags:
- Training
summary: Get status of tool training
description: Get status of the tool's training process (404 if tool is not trainable)
operationId: getToolTrainingStatus
responses:
'200':
description: Success
content:
application/json:
schema:
$ref: ../components/schemas/TrainingStatus.yaml
'404':
$ref: ../components/responses/NotFound.yaml
'500':
$ref: ../components/responses/InternalServerError.yaml
16 changes: 16 additions & 0 deletions openapi/commons/paths/trainTool.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
post:
tags:
- Training
summary: Initiate tool training (async)
description: Start training the tool's model
operationId: trainAnnotator
responses:
'201':
description: Training initialized
links:
GetTrainingStatus:
operationId: toolTrainingStatus
'404':
$ref: ../components/responses/NotFound.yaml
'500':
$ref: ../components/responses/InternalServerError.yaml
17 changes: 17 additions & 0 deletions openapi/commons/paths/trainedToolStatus.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
get:
tags:
- Training
summary: Get information about whether trained model has been loaded
description: Get information about the tool's trained model (404 if tool is not trainable)
operationId: getTrainedToolStatus
responses:
'200':
description: Success
content:
application/json:
schema:
$ref: ../components/schemas/TrainedToolStatus.yaml
'404':
$ref: ../components/responses/NotFound.yaml
'500':
$ref: ../components/responses/InternalServerError.yaml
6 changes: 6 additions & 0 deletions openapi/contact-annotator/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,9 @@ paths:
$ref: ../commons/paths/tool.yaml
/tool/dependencies:
$ref: ../commons/paths/dependencies.yaml
/train:
$ref: ../commons/paths/trainAnnotator.yaml
/train/status:
$ref: ../commons/paths/annotatorTrainingStatus.yaml
/train/load:
$ref: ../commons/paths/loadTrainedAnnotator.yaml
8 changes: 7 additions & 1 deletion openapi/covid-symptom-annotator/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,4 +70,10 @@ paths:
/tool:
$ref: ../commons/paths/tool.yaml
/tool/dependencies:
$ref: ../commons/paths/dependencies.yaml
$ref: ../commons/paths/dependencies.yaml
/train:
$ref: ../commons/paths/trainAnnotator.yaml
/train/status:
$ref: ../commons/paths/annotatorTrainingStatus.yaml
/train/load:
$ref: ../commons/paths/loadTrainedAnnotator.yaml
8 changes: 7 additions & 1 deletion openapi/date-annotator/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,4 +71,10 @@ paths:
/tool:
$ref: ../commons/paths/tool.yaml
/tool/dependencies:
$ref: ../commons/paths/dependencies.yaml
$ref: ../commons/paths/dependencies.yaml
/train:
$ref: ../commons/paths/trainAnnotator.yaml
/train/status:
$ref: ../commons/paths/annotatorTrainingStatus.yaml
/train/load:
$ref: ../commons/paths/loadTrainedAnnotator.yaml
8 changes: 7 additions & 1 deletion openapi/id-annotator/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,4 +69,10 @@ paths:
/tool:
$ref: ../commons/paths/tool.yaml
/tool/dependencies:
$ref: ../commons/paths/dependencies.yaml
$ref: ../commons/paths/dependencies.yaml
/train:
$ref: ../commons/paths/trainAnnotator.yaml
/train/status:
$ref: ../commons/paths/annotatorTrainingStatus.yaml
/train/load:
$ref: ../commons/paths/loadTrainedAnnotator.yaml
8 changes: 7 additions & 1 deletion openapi/location-annotator/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,4 +69,10 @@ paths:
/tool:
$ref: ../commons/paths/tool.yaml
/tool/dependencies:
$ref: ../commons/paths/dependencies.yaml
$ref: ../commons/paths/dependencies.yaml
/train:
$ref: ../commons/paths/trainAnnotator.yaml
/train/status:
$ref: ../commons/paths/annotatorTrainingStatus.yaml
/train/load:
$ref: ../commons/paths/loadTrainedAnnotator.yaml
8 changes: 7 additions & 1 deletion openapi/person-name-annotator/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,4 +69,10 @@ paths:
/tool:
$ref: ../commons/paths/tool.yaml
/tool/dependencies:
$ref: ../commons/paths/dependencies.yaml
$ref: ../commons/paths/dependencies.yaml
/train:
$ref: ../commons/paths/trainAnnotator.yaml
/train/status:
$ref: ../commons/paths/annotatorTrainingStatus.yaml
/train/load:
$ref: ../commons/paths/loadTrainedAnnotator.yaml
6 changes: 6 additions & 0 deletions openapi/phi-annotator/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,3 +87,9 @@ paths:
$ref: ../commons/paths/tool.yaml
/tool/dependencies:
$ref: ../commons/paths/dependencies.yaml
/train:
$ref: ../commons/paths/trainAnnotator.yaml
/train/status:
$ref: ../commons/paths/annotatorTrainingStatus.yaml
/train/load:
$ref: ../commons/paths/loadTrainedAnnotator.yaml