fix deprecated tokenizer methods, add test cases #852

Job	Run time
Build on ubuntu-latest using Python 3.9	3m 4s
Build on ubuntu-latest using Python 3.10	2m 42s
Build on ubuntu-latest using Python 3.11	3m 5s
Build on ubuntu-latest using Python 3.12	2m 51s
Build on macos-12 using Python 3.9	2m 24s
Build on macos-12 using Python 3.10	3m 1s
Build on macos-12 using Python 3.11	2m 34s
Build on macos-12 using Python 3.12	2m 26s
Build on windows-latest using Python 3.9	4m 17s
Build on windows-latest using Python 3.10	4m 8s
Build on windows-latest using Python 3.11	4m 35s
Build on windows-latest using Python 3.12	4m 38s
Create Wheel	55s
	40m 40s

Provide feedback