Issue 137 - Fix Hyperscan duplicate editions bug

[mattdahl]

Bug is described in #137. In theory, I think this could affect other tokenizers too, but I only see the problem in HyperscanTokenizer. My proposed solution is simple -- just remove duplicate candidate editions after citation tokens are merged. If there is some reason why those duplicates should be retained, I'm missing it.

[mlissner]

Seems simple enough, but a number of tests are failing. Any idea what's going on there?

[mattdahl]

Not sure what the problem is, they all passed fine on my fork (same commit hash): https://github.com/mattdahl/eyecite/actions/runs/3962150256/. Also not sure why the dependencies (fast_diff_match_patch, exrex) don't load on only some Python versions.

[flooie]

@mattdahl @mlissner - it certainly doesn't appear to be related to your PR. I re-ran tests for python 3.8 for a branch that passed tests yesterday (I think) to see if it would happen on it now fails. So somewhere downstream something has changed.

[mattdahl]

Can this be reviewed/merged now that #140 has fixed the CI problem? (Though note that now another CI problem will need to be fixed first in #143)

[mlissner]

I synced with main and tests are re-running. If they pass, we'll be in business, I think.

[mlissner]

Thanks for bumping this Matt! Merged!