Tokenization Consistency Matters for Generative Models on Extractive NLP Tasks - Findings of EMNLP2023
The code can be found in https://github.com/awslabs/ConsistentTokenization
Paper: https://arxiv.org/abs/2212.09912
For questions, feel free to open an issue and ping me on {email, Twitter, Slack}.