HypernymySuite is a tool for evaluating some hypernymy detection modules. Its predominant focus is reproducing the results for the following paper.
Stephen Roller, Douwe Kiela, and Maximilian Nickel. 2018. Hearst Patterns Revisited: Automatic Hypernym Detection from Large Text Corpora. ACL. (arXiv)
We hope that open sourcing our evaluation will help facilitate future research.
You can produce results in a JSON format by calling main.py:
python main.py cnt --dset hearst_counts.txt.gz
These results can be made machine readable by piping them into compile_table:
python main.py cnt --dset hearst_counts.txt.gz | python compile_table.py
To generate the full table from the report, you may simply use generate_table.sh:
bash generate_table.sh results.json
Please note that due to licensing concerns, we were not able to release our train/validation/test folds from the paper, so results may differ slightly than those reported.
The module was developed with python3 in mind, and is not tested for python2. Nonetheless, cross-platform compatibility may be possible.
The suite requires several packages you probably already have installed:
numpy, scipy, pandas, scikit-learn and nltk. These can be installed
using pip:
pip install -r requirements.txt
If you've never used nltk before, you'll need to install the wordnet module.
python -c "import nltk; nltk.download('wordnet')"
On OS X, you may need to install coreutils and gnu-sed for the script download_data.sh to run correctly. These can be installed using brew:
brew install coreutils gnu-sed
After installation, you will either need to modify download_data.sh to run gsort and gsed instead of sort and sed, or alternatively add a "gnubin" directory to your PATH from your bashrc:
PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
For more information, see brew info coreutils or brew info gnu-sed.
You can evaluate your own model in two separate ways. The simplest way is simply
to create a copy of example.tsv, and fill in your model's predictions in the sim
column. You must include a prediction for every pair, but you may set the is_oov
column to 1 to ensure it is correctly calculated.
You may then evaluate the model:
python main.py precomputed --dset example.tsv
You can also implement any model by extending the base.HypernymySuiteModel class
and filling in your own implemenation for predict or predict_many.
If you find this code useful for your research, please cite the following paper:
@inproceedings{roller2018hearst
title = {Hearst Patterns Revisited: Automatic Hypernym Detection from Large Text Corpora},
author = {Roller, Stephen and Kiela, Douwe and Nickel, Maximilian},
year = {2018},
booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics},
location = {Melbourne, Australia},
publisher = {Association for Computational Linguistics}
}
This code is licensed under CC-BY-NC4.0.
The data contained in hearst_counts.txt was extracted from a combination of
Wikipedia and Gigaword.
Please see publication for details.