New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

分词器的问题 #84

Open

liguangxue opened this issue Jul 21, 2023 · 0 comments

liguangxue commented Jul 21, 2023

请问一下，分词的时候，为什么要把标点符号和空格都去掉，这样英文就没法分词了吧？还是我用的不对？
text = utils.RemovePunctuation(text)
//移除所有的空格
text = utils.RemoveSpace(text)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment