Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve usability of inter-annotator agreement computation in status view #22

Open
cfedermann opened this issue Jan 17, 2013 · 1 comment
Assignees

Comments

@cfedermann
Copy link
Owner

Compute inter-annotator agreement (IAA) for the number of annotators or coders (C) which maximises the number of items (I) that have been evaluated by the respective sub set of coders. This avoids showing no IAA scores until all coders have completed the task.

Additionally, there could be checkboxes which can be used to toggle whether a system should be included in IAA computation or not. That way, different scenarios could be tested in a more playful way, from within the status view.

If implemented, the checkbox selection should also be reflected when downloading results data.

@ghost ghost assigned cfedermann Jan 17, 2013
@cfedermann cfedermann modified the milestones: Future release, Useful status view for all tasks May 5, 2015
@cfedermann
Copy link
Owner Author

Will have to clean this up once I get to refactoring eval tasks into parallel modules...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant