- Environment Requirements
- Overview
- Dataset
- Preview
- Training
- Results and Accuracy Measures
- Documentation
- References
- Contributors
- Python 3
- Pytorch 2.0.1
- cv2 4.0.1
This machine learning model learns and mimics personal handwriting style. The model is trained on a dataset of handwritten samples of a person and is capable of converting standard fonts into any given person’s personalized handwriting. It generates handwritten text that closely resembles unique writing style for various applications.
We have used IAM Offline Handwriting Dataset consisting of over 20,000 samples of handwriting of over 500 different authors. We have used the handwriting samples of 339 authors for training purpose and 159 for validation purpose. This dataset was used because of its easy availability and diverse handwriting styles that were required for the Handwriting Imitation model.
Handwriting Imitation Model Algorithm
Handwriting Transformers model: Handwriting Transformers model works on generating the handwritten text of the query text based on the input handwriting. Our model has been trained on the vast IAM Dataset to generate handwriting using transformers and GAN. Later we finetune the model weights according to the input handwriting to generate images that match the input
Model architecture: The basic architecture of our GAN model is such that we have used transformer layers in our generator to do the encoding and decoding procedure. First of all we have cnn encoded images as input to the model which are converted to grayscale format and also padded. Subsequently the images are converted to tensor. These images go to the transformer encoder which processes the image descriptors helps in refining and enhancing the features. Then this goes to the transformer decoder layer where the query text is embedded with the input text.The query text is also encoded using the OCR Network encoder. Now this goes to the CNN decoder where the fake image generated by the generator is passed on to the discriminator and based on its feedback generator modifies its weights during the training.
| 800 epochs | 2000 epochs |
| 4000 epochs | 6000 epochs |
| 8000 epochs | 10000 epochs |
See the random handwriting generation model here.
See the personalized handwriting generation model here.
Note: Please change the runtime the runtime to GPU before running all the cells. For personalised handwriting upload your image and change its image path in the mentioned location.
You can check the video for our website. You can try our website for the demo file and try out our features here. Please download the zip file and change some paths.
We have used FID Score as our metrics for the following model. Below are some of our results and their fid scores.
| FID SCORE IN VOCAB:113.253975 | FID SCORE IN VOCAB:105.99905 |
| FID SCORE OUT VOCAB:105.119934 | FID SCORE OUT VOCAB:112.280205 |
See the Handwriting Imitation document for full documentation on implementation of models and the website.
- https://github.com/M4rt1nM4yr/HandwritingImitationDemo/blob/main/HandwritingTransformers
- https://github.com/ganji15/HiGANplus
- https://openaccess.thecvf.com/content/ICCV2021/papers/Bhunia_Handwriting_Transformers_ICCV_2021_paper.pdf
- https://github.com/JaidedAI/EasyOCRtr
- https://huggingface.co/docs/transformers/model_doc/trocr