Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Baseline fix #1

Open
kudkudak opened this issue Mar 11, 2017 · 2 comments
Open

Baseline fix #1

kudkudak opened this issue Mar 11, 2017 · 2 comments

Comments

@kudkudak
Copy link

kudkudak commented Mar 11, 2017

Turns out it is important to calculate counts only based on tokens of given passage, rather than estimated on all stories. It is important to produce similar results as in https://www.microsoft.com/en-us/research/wp-content/uploads/2016/11/MCTest_EMNLP2013.pdf. Performance changes from

From:

correct
question_type subset
multiple mc160 0.523438
mc500 0.414634
one mc160 0.508929
mc500 0.477941

To:

correct
question_type subset
multiple mc160 0.578125
mc500 0.539634
one mc160 0.678571
mc500 0.544118

Fix is to calculate icounts as _icounts = compute_inverse_counts([tokens]) in SlidingWindow.predict_target

@kudkudak kudkudak changed the title Baseline Baseline fix Mar 11, 2017
@kudkudak
Copy link
Author

Another fix: baseline dist calculation should use float as: closest = np.abs(last_q - last_a) / (float(len(passage)) - 1). I understand stopwords removal is done during data preprocessing

@allanj
Copy link

allanj commented Oct 11, 2017

I couldn't obtain the results you list there after the suggested modification.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants