experimental annotation framework for phrase level annotation
Built using Flask, sqlite3, bootstrap and Javascript.
python version >= 3.6 recommended
This package requires flask
git clone http://github.com/atharva-naik/XNLI-annotator
Please follow the steps below for annotation:
Create a new user by clicking on the Create button on the New User tile, shown below, to keep track of your annotations. Immediately after creating your user you will be guided to the main annotation page.
If you are already registered just click on the tile corresponding to your username and you will be guided to a login screen as shown below.
How to use the main annotation page:
- Phrase annotator: Just select a span of text and pick the label from ("Entail","Contradict","Neutral","Unaligned"). Clear P, Clear H can be used t o clear all marked phrases in the premise and hypothesis resepectively. Clear an be used to clear annotations form the selected span. Clear All clears all annotations. Press save to save the annotation.
- Bookmarking: Save a particular annotation that you are doubtful about for later review.
- Navigate: Guides you to the navigation page.
- Meaning: Select a word you, whose meaning you are unsure about. Click the button below, and the meaning will be shown in the "Meaning of selected word" area below.
- Annotation sidebar: It shows the annotation that is currently saved on the server. It can be used to logout, open dashboard and view bookmarked annotations.
- Instructions: Click on the ⓘ to view these instructions.
The navigation pane shows "Pending" or "Complete" beside each sentence-pair/sample. Use it to get a bird's eye view of your progress.
The bookmarked section can be used to revisit annotations you are unsure about.
- Leaderboard: learn about the progress of your co-annotators, and how much progress you haved made.
- Label distribution and other statistics: Total number of annotators, how many annotations you have finished, inter-annotator agreement and pertinent dataset statistics are shown here.
- Export feature: export a jsonl file which is easy to read for both you and your computer (for processing and cleaning).
