fix deprecated tokenizer methods, add test cases (#139) #856

Job	Run time
Build on ubuntu-latest using Python 3.9	3m 3s
Build on ubuntu-latest using Python 3.10	5m 3s
Build on ubuntu-latest using Python 3.11	5m 21s
Build on ubuntu-latest using Python 3.12	3m 15s
Build on macos-12 using Python 3.9	3m 8s
Build on macos-12 using Python 3.10	2m 40s
Build on macos-12 using Python 3.11	2m 54s
Build on macos-12 using Python 3.12	2m 31s
Build on windows-latest using Python 3.9	3m 55s
Build on windows-latest using Python 3.10	4m 45s
Build on windows-latest using Python 3.11	4m 40s
Build on windows-latest using Python 3.12	4m 3s
Create Wheel	51s
	46m 9s

Provide feedback