| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
| ', ' | ', ' | ', ' | ', ' | ', ' |
857 | SVT |IIIT5k
3000 |IC15
1811| SVTP |CUTE80 | Avg | Download | +|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:| +| CPPD Tiny | 97.1 | 94.4 | 96.6 | 86.6 | 88.5 | 90.3 | 92.25 | [en](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_tiny_en_train.tar) | +| CPPD Base | 98.2 | 95.5 | 97.6 | 87.9 | 90.0 | 92.7 | 93.80 | [en](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_en_train.tar)| +| CPPD Base 48*160 | 97.5 | 95.5 | 97.7 | 87.7 | 92.4 | 93.7 | 94.10 | [en](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_48_160_en_train.tar) | + +* Trained on Synth dataset(MJ+ST), Test on Union14M-L benchmark from [U14m](https://github.com/Mountchicken/Union14M/). + +| Model |Curve | Multi-
Oriented |Artistic |Contextless| Salient | Multi-
word | General | Avg | Download | +|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:|:-------:| +| CPPD Tiny | 52.4 | 12.3 | 48.2 | 54.4 | 61.5 | 53.4 | 61.4 | 49.10 | Same as the table above. | +| CPPD Base | 65.5 | 18.6 | 56.0 | 61.9 | 71.0 | 57.5 | 65.8 | 56.63 | Same as the table above. | +| CPPD Base 48*160 | 71.9 | 22.1 | 60.5 | 67.9 | 78.3 | 63.9 | 67.1 | 61.69 | Same as the table above. | + +* Trained on Union14M-L training dataset. + +| Model |IC13
857 | SVT |IIIT5k
3000 |IC15
1811| SVTP |CUTE80 | Avg | Download | +|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:| +| CPPD Base 32*128 | 98.7 | 98.5 | 99.4 | 91.7 | 96.7 | 99.7 | 97.44 | [en](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_u14m_train.tar) | + +| Model |Curve | Multi-
Oriented |Artistic |Contextless| Salient | Multi-
word | General | Avg | Download | +|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:|:-------:| +| CPPD Base 32*128 | 87.5 | 70.7 | 78.2 | 82.9 | 85.5 | 85.4 | 84.3 | 82.08 | Same as the table above. | + +* Chinese dataset from [Chinese Benckmark](https://github.com/FudanVI/benchmarking-chinese-text-recognition). + +| Model | Scene | Web | Document | Handwriting | Avg | Download | +|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:| +| CPPD Base | 74.4 | 76.1 | 98.6 | 55.3 | 76.10 | [ch](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_ch_train.tar) | +| CPPD Base + STN | 78.4 | 79.3 | 98.9 | 57.6 | 78.55 | [ch](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_stn_ch_train.tar) | + +## 2. Environment + +Please refer to ["Environment Preparation"](../../ppocr/environment.en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](../../ppocr/blog/clone.en.md)to clone the project code. + +### Dataset Preparation + +[English dataset download](https://github.com/baudm/parseq) +[Union14M-Benchmark download](https://github.com/Mountchicken/Union14M) +[Chinese dataset download](https://github.com/fudanvi/benchmarking-chinese-text-recognition#download) + +## 3. Model Training / Evaluation / Prediction + +Please refer to [Text Recognition Tutorial](../../ppocr/model_train/recognition.en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. + +### Training + +Specifically, after the data preparation is completed, the training can be started. The training command is as follows: + +```bash linenums="1" +# Single GPU training (long training period, not recommended) +python3 tools/train.py -c configs/rec/rec_svtrnet_cppd_base_en.yml + +# Multi GPU training, specify the gpu number through the --gpus parameter +python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_svtrnet_cppd_base_en.yml +``` + +### Evaluation + +You can download the model files and configuration files provided by `CPPD`: [download link](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_en_train.tar), take `CPPD-B` as an example, using the following command to evaluate: + +```bash linenums="1" +# Download the tar archive containing the model files and configuration files of CPPD-B and extract it +wget https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_en_train.tar && tar xf rec_svtr_cppd_base_en_train.tar +# GPU evaluation +python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c ./rec_svtr_cppd_base_en_train/rec_svtrnet_cppd_base_en.yml -o Global.pretrained_model=./rec_svtr_cppd_base_en_train/best_model +``` + +### Prediction + +```bash linenums="1" +python3 tools/infer_rec.py -c ./rec_svtr_cppd_base_en_train/rec_svtrnet_cppd_base_en.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_svtr_cppd_base_en_train/best_model +``` + +## 4. Inference and Deployment + +### 4.1 Python Inference + +First, the model saved during the CPPD text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_en_train.tar) ), you can use the following command to convert: + +```bash linenums="1" +# export model +# en +python3 tools/export_model.py -c configs/rec/rec_svtrnet_cppd_base_en.yml -o Global.pretrained_model=./rec_svtr_cppd_base_en_train/best_model.pdparams Global.save_inference_dir=./rec_svtr_cppd_base_en_infer +# ch +python3 tools/export_model.py -c configs/rec/rec_svtrnet_cppd_base_ch.yml -o Global.pretrained_model=./rec_svtr_cppd_base_ch_train/best_model.pdparams Global.save_inference_dir=./rec_svtr_cppd_base_ch_infer + +# speed test +# docker image https://hub.docker.com/r/paddlepaddle/paddle/tags/: sudo docker pull paddlepaddle/paddle:2.4.2-gpu-cuda11.2-cudnn8.2-trt8.0 +# install auto_log: pip install https://paddleocr.bj.bcebos.com/libs/auto_log-1.2.0-py3-none-any.whl +# en +python3 tools/infer/predict_rec.py --image_dir='../iiik' --rec_model_dir='./rec_svtr_cppd_base_en_infer/' --rec_algorithm='CPPD' --rec_image_shape='3,32,100' --rec_char_dict_path='./ppocr/utils/ic15_dict.txt' --warmup=True --benchmark=True --rec_batch_num=1 --use_tensorrt=True +# ch +python3 tools/infer/predict_rec.py --image_dir='../iiik' --rec_model_dir='./rec_svtr_cppd_base_ch_infer/' --rec_algorithm='CPPDPadding' --rec_image_shape='3,32,256' --warmup=True --benchmark=True --rec_batch_num=1 --use_tensorrt=True +# stn_ch +python3 tools/infer/predict_rec.py --image_dir='../iiik' --rec_model_dir='./rec_svtr_cppd_base_stn_ch_infer/' --rec_algorithm='CPPD' --rec_image_shape='3,64,256' --warmup=True --benchmark=True --rec_batch_num=1 --use_tensorrt=True +``` + +**Note:** If you are training the model on your own dataset and have modified the dictionary file, please pay attention to modify the `character_dict_path` in the configuration file to the modified dictionary file. + +After the conversion is successful, there are three files in the directory: + +```text linenums="1" +/inference/rec_svtr_cppd_base_en_infer/ + ├── inference.pdiparams + ├── inference.pdiparams.info + └── inference.pdmodel +``` + +### 4.2 C++ Inference + +Not supported + +### 4.3 Serving + +Not supported + +### 4.4 More + +Not supported + +## Citation + +```bibtex +@article{Du2023CPPD, + title = {Context Perception Parallel Decoder for Scene Text Recognition}, + author = {Du, Yongkun and Chen, Zhineng and Jia, Caiyan and Yin, Xiaoting and Li, Chenxia and Du, Yuning and Jiang, Yu-Gang}, + booktitle = {Arxiv}, + year = {2023}, + url = {https://arxiv.org/abs/2307.12270} +} +``` diff --git a/docs/algorithm/text_recognition/algorithm_rec_cppd.md b/docs/algorithm/text_recognition/algorithm_rec_cppd.md new file mode 100644 index 0000000000..561fda146c --- /dev/null +++ b/docs/algorithm/text_recognition/algorithm_rec_cppd.md @@ -0,0 +1,175 @@ +--- +comments: true +--- + +# 场景文本识别算法-CPPD + +## 1. 算法简介 + +论文信息： +> [Context Perception Parallel Decoder for Scene Text Recognition](https://arxiv.org/abs/2307.12270) +> Yongkun Du and Zhineng Chen and Caiyan Jia and Xiaoting Yin and Chenxia Li and Yuning Du and Yu-Gang Jiang + +### CPPD算法简介 + +基于深度学习的场景文本识别模型通常是Encoder-Decoder结构，其中decoder可以分为两种：(1)CTC，(2)Attention-based。目前SOTA模型大多使用Attention-based的decoder，而attention-based可以分为AR和PD两种，一般来说，AR解码器识别精度优于PD，而PD解码速度快于AR，CPPD通过精心设计的CO和CC模块，达到了“AR的精度，PD的速度”的效果。 + +CPPD在场景文本识别公开数据集上的精度(%)和模型文件如下： + +* 英文训练集和测试集来自于[PARSeq](https://github.com/baudm/parseq)。 + +| 模型 |IC13
857 | SVT |IIIT5k
3000 |IC15
1811| SVTP |CUTE80 | Avg | 下载链接 | +|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:| +| CPPD Tiny | 97.1 | 94.4 | 96.6 | 86.6 | 88.5 | 90.3 | 92.25 | [英文](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_tiny_en_train.tar) | +| CPPD Base | 98.2 | 95.5 | 97.6 | 87.9 | 90.0 | 92.7 | 93.80 | [英文](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_en_train.tar)| +| CPPD Base 48*160 | 97.5 | 95.5 | 97.7 | 87.7 | 92.4 | 93.7 | 94.10 | [英文](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_48_160_en_train.tar) | + +* 英文合成数据集(MJ+ST)训练，英文Union14M-L benchmark测试结果[U14m](https://github.com/Mountchicken/Union14M/)。 + +| 模型 |Curve | Multi-
Oriented |Artistic |Contextless| Salient | Multi-
word | General | Avg | 下载链接 | +|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:|:-------:| +| CPPD Tiny | 52.4 | 12.3 | 48.2 | 54.4 | 61.5 | 53.4 | 61.4 | 49.10 | 同上表 | +| CPPD Base | 65.5 | 18.6 | 56.0 | 61.9 | 71.0 | 57.5 | 65.8 | 56.63 | 同上表 | +| CPPD Base 48*160 | 71.9 | 22.1 | 60.5 | 67.9 | 78.3 | 63.9 | 67.1 | 61.69 | 同上表 | + +* Union14M-L 训练集From scratch训练，英文测试结果。 + +| 模型 |IC13
857 | SVT |IIIT5k
3000 |IC15
1811| SVTP |CUTE80 | Avg | 下载链接 | +|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:| +| CPPD Base 32*128 | 98.5 | 97.7 | 99.2 | 90.3 | 94.6 | 98.3 | 96.42 | Coming soon | + +| 模型 |Curve | Multi-
Oriented |Artistic |Contextless| Salient | Multi-
word | General | Avg | 下载链接 | +|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:|:-------:| +| CPPD Base 32*128 | 83.0 | 71.2 | 75.1 | 80.9 | 79.4 | 82.6 | 83.7 | 79.41 | Coming soon | + +* 加载合成数据集预训练模型，Union14M-L 训练集微调训练，英文测试结果。 + +| 模型 |IC13
857 | SVT |IIIT5k
3000 |IC15
1811| SVTP |CUTE80 | Avg | 下载链接 | +|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:| +| CPPD Base 32*128 | 98.7 | 98.5 | 99.4 | 91.7 | 96.7 | 99.7 | 97.44 | [英文](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_u14m_train.tar) | + +| 模型 |Curve | Multi-
Oriented |Artistic |Contextless| Salient | Multi-
word | General | Avg | 下载链接 | +|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:|:-------:| +| CPPD Base 32*128 | 87.5 | 70.7 | 78.2 | 82.9 | 85.5 | 85.4 | 84.3 | 82.08 | 同上表 | + +* 中文训练集和测试集来自于[Chinese Benckmark](https://github.com/FudanVI/benchmarking-chinese-text-recognition)。 + +| 模型 | Scene | Web | Document | Handwriting | Avg | 下载链接 | +|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:| +| CPPD Base | 74.4 | 76.1 | 98.6 | 55.3 | 76.10 | [中文](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_ch_train.tar) | +| CPPD Base + STN | 78.4 | 79.3 | 98.9 | 57.6 | 78.55 | [中文](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_stn_ch_train.tar) | + +## 2. 环境配置 + +请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。 + +## 3. 模型训练、评估、预测 + +### 3.1 模型训练 + +#### 数据集准备 + +[英文数据集下载](https://github.com/baudm/parseq) + +[Union14M-L 下载](https://github.com/Mountchicken/Union14M) + +[中文数据集下载](https://github.com/fudanvi/benchmarking-chinese-text-recognition#download) + +#### 启动训练 + +请参考[文本识别训练教程](../../ppocr/model_train/recognition.md)。PaddleOCR对代码进行了模块化，训练`CPPD`识别模型时需要**更换配置文件**为`CPPD`的[配置文件](../../configs/rec/rec_svtrnet_cppd_base_en.yml)。 + +具体地，在完成数据准备后，便可以启动训练，训练命令如下： + +```bash linenums="1" +#单卡训练（训练周期长，不建议） +python3 tools/train.py -c configs/rec/rec_svtrnet_cppd_base_en.yml + +# 多卡训练，通过--gpus参数指定卡号 +python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_svtrnet_cppd_base_en.yml +``` + +### 3.2 评估 + +可下载`CPPD`提供的模型文件和配置文件：[下载地址](https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_en_train.tar) ，以`CPPD-B`为例，使用如下命令进行评估： + +```bash linenums="1" +# 下载包含CPPD-B的模型文件和配置文件的tar压缩包并解压 +wget https://paddleocr.bj.bcebos.com/CCPD/rec_svtr_cppd_base_en_train.tar && tar xf rec_svtr_cppd_base_en_train.tar +# 注意将pretrained_model的路径设置为本地路径。 +python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c ./rec_svtr_cppd_base_en_train/rec_svtrnet_cppd_base_en.yml -o Global.pretrained_model=./rec_svtr_cppd_base_en_train/best_model +``` + +### 3.3 预测 + +使用如下命令进行单张图片预测： + +```bash linenums="1" +# 注意将pretrained_model的路径设置为本地路径。 +python3 tools/infer_rec.py -c ./rec_svtr_cppd_base_en_train/rec_svtrnet_cppd_base_en.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_svtr_cppd_base_en_train/best_model +# 预测文件夹下所有图像时，可修改infer_img为文件夹，如 Global.infer_img='./doc/imgs_words_en/'。 +``` + +## 4. 推理部署 + +### 4.1 Python推理 + +首先将训练得到best模型，转换成inference model。下面以基于`CPPD-B`，在英文数据集训练的模型为例（[模型和配置文件下载地址](https://paddleocr.bj.bcebos.com/CPPD/rec_svtr_cppd_base_en_train.tar)，可以使用如下命令进行转换： + +**注意：** + +* 如果您是在自己的数据集上训练的模型，并且调整了字典文件，请注意修改配置文件中的`character_dict_path`是否为所正确的字典文件。 + +执行如下命令进行模型导出和推理： + +```bash linenums="1" +# 注意将pretrained_model的路径设置为本地路径。 +# export model +# en +python3 tools/export_model.py -c configs/rec/rec_svtrnet_cppd_base_en.yml -o Global.pretrained_model=./rec_svtr_cppd_base_en_train/best_model.pdparams Global.save_inference_dir=./rec_svtr_cppd_base_en_infer +# ch +python3 tools/export_model.py -c configs/rec/rec_svtrnet_cppd_base_ch.yml -o Global.pretrained_model=./rec_svtr_cppd_base_ch_train/best_model.pdparams Global.save_inference_dir=./rec_svtr_cppd_base_ch_infer + +# speed test +# docker image https://hub.docker.com/r/paddlepaddle/paddle/tags/: sudo docker pull paddlepaddle/paddle:2.4.2-gpu-cuda11.2-cudnn8.2-trt8.0 +# install auto_log: pip install https://paddleocr.bj.bcebos.com/libs/auto_log-1.2.0-py3-none-any.whl +# en +python3 tools/infer/predict_rec.py --image_dir='../iiik' --rec_model_dir='./rec_svtr_cppd_base_en_infer/' --rec_algorithm='CPPD' --rec_image_shape='3,32,100' --rec_char_dict_path='./ppocr/utils/ic15_dict.txt' --warmup=True --benchmark=True --rec_batch_num=1 --use_tensorrt=True +# ch +python3 tools/infer/predict_rec.py --image_dir='../iiik' --rec_model_dir='./rec_svtr_cppd_base_ch_infer/' --rec_algorithm='CPPDPadding' --rec_image_shape='3,32,256' --warmup=True --benchmark=True --rec_batch_num=1 --use_tensorrt=True +# stn_ch +python3 tools/infer/predict_rec.py --image_dir='../iiik' --rec_model_dir='./rec_svtr_cppd_base_stn_ch_infer/' --rec_algorithm='CPPD' --rec_image_shape='3,64,256' --warmup=True --benchmark=True --rec_batch_num=1 --use_tensorrt=True +``` + +导出成功后，在目录下有三个文件： + +``` +/inference/rec_svtr_cppd_base_en_infer/ + ├── inference.pdiparams # 识别inference模型的参数文件 + ├── inference.pdiparams.info # 识别inference模型的参数信息，可忽略 + └── inference.pdmodel # 识别inference模型的program文件 +``` + +### 4.2 C++推理部署 + +由于C++预处理后处理还未支持CPPD，所以暂未支持 + +### 4.3 Serving服务化部署 + +暂不支持 + +### 4.4 更多推理部署 + +暂不支持 + +## 引用 + +```bibtex +@article{Du2023CPPD, + title = {Context Perception Parallel Decoder for Scene Text Recognition}, + author = {Du, Yongkun and Chen, Zhineng and Jia, Caiyan and Yin, Xiaoting and Li, Chenxia and Du, Yuning and Jiang, Yu-Gang}, + booktitle = {Arxiv}, + year = {2023}, + url = {https://arxiv.org/abs/2307.12270} +} +``` diff --git a/docs/algorithm/text_recognition/algorithm_rec_crnn.en.md b/docs/algorithm/text_recognition/algorithm_rec_crnn.en.md new file mode 100644 index 0000000000..db0d989552 --- /dev/null +++ b/docs/algorithm/text_recognition/algorithm_rec_crnn.en.md @@ -0,0 +1,102 @@ +--- +comments: true +--- + +# CRNN + +## 1. Introduction + +Paper: +> [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://arxiv.org/abs/1507.05717) + +> Baoguang Shi, Xiang Bai, Cong Yao + +> IEEE, 2015 + +Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: + +|Model|Backbone|ACC|config|Download link| +| --- | --- | --- | --- | --- | +|---|---|---|---|---| +|CRNN|Resnet34_vd|81.04%|[configs/rec/rec_r34_vd_none_bilstm_ctc.yml](../../configs/rec/rec_r34_vd_none_bilstm_ctc.yml)|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_bilstm_ctc_v2.0_train.tar)| +|CRNN|MobileNetV3|77.95%|[configs/rec/rec_mv3_none_bilstm_ctc.yml](../../configs/rec/rec_mv3_none_bilstm_ctc.yml)|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_none_bilstm_ctc_v2.0_train.tar)| + +## 2. Environment + +Please refer to ["Environment Preparation"](../../ppocr/environment.en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](../../ppocr/blog/clone.en.md)to clone the project code. + +## 3. Model Training / Evaluation / Prediction + +Please refer to [Text Recognition Tutorial](../../ppocr/model_train/recognition.en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. + +### Training + +Specifically, after the data preparation is completed, the training can be started. The training command is as follows: + +```bash linenums="1" +# Single GPU training (long training period, not recommended) +python3 tools/train.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml + +# Multi GPU training, specify the gpu number through the --gpus parameter +python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml +``` + +### Evaluation + +```bash linenums="1" +# GPU evaluation +python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy +``` + +### Prediction + +```bash linenums="1" +# The configuration file used for prediction must match the training +python3 tools/infer_rec.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png +``` + +## 4. Inference and Deployment + +### 4.1 Python Inference + +First, the model saved during the CRNN text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_r31_CRNN_train.tar) ), you can use the following command to convert: + +```bash linenums="1" +python3 tools/export_model.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml -o Global.pretrained_model=./rec_r34_vd_none_bilstm_ctc_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_crnn +``` + +For CRNN text recognition model inference, the following commands can be executed: + +```bash linenums="1" +python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./inference/rec_crnn/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" +``` + +### 4.2 C++ Inference + +With the inference model prepared, refer to the [cpp infer](../../ppocr/infer_deploy/cpp_infer.en.md) tutorial for C++ inference. + +### 4.3 Serving + +With the inference model prepared, refer to the [pdserving](../../ppocr/infer_deploy/paddle_server.en.md) tutorial for service deployment by Paddle Serving. + +### 4.4 More + +More deployment schemes supported for CRNN: + +- Paddle2ONNX: with the inference model prepared, please refer to the [paddle2onnx](../../ppocr/infer_deploy/paddle2onnx.en.md) tutorial. + +## 5. FAQ + +## Citation + +```bibtex +@ARTICLE{7801919, + author={Shi, Baoguang and Bai, Xiang and Yao, Cong}, + journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, + title={An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition}, + year={2017}, + volume={39}, + number={11}, + pages={2298-2304}, + doi={10.1109/TPAMI.2016.2646371}} +``` diff --git a/docs/algorithm/text_recognition/algorithm_rec_crnn.md b/docs/algorithm/text_recognition/algorithm_rec_crnn.md new file mode 100644 index 0000000000..1f572beca5 --- /dev/null +++ b/docs/algorithm/text_recognition/algorithm_rec_crnn.md @@ -0,0 +1,118 @@ +--- +typora-copy-images-to: images +comments: true +--- + +# CRNN + +## 1. 算法简介 + +论文信息： +> [An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition](https://arxiv.org/abs/1507.05717) +> Baoguang Shi, Xiang Bai, Cong Yao +> IEEE, 2015 + +参考[DTRB](https://arxiv.org/abs/1904.01906) 文字识别训练和评估流程，使用MJSynth和SynthText两个文字识别数据集训练，在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估，算法效果如下： + +|模型|骨干网络|Avg Accuracy|配置文件|下载链接| +|---|---|---|---|---| +|CRNN|Resnet34_vd|81.04%|[configs/rec/rec_r34_vd_none_bilstm_ctc.yml](../../configs/rec/rec_r34_vd_none_bilstm_ctc.yml)|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_bilstm_ctc_v2.0_train.tar)| +|CRNN|MobileNetV3|77.95%|[configs/rec/rec_mv3_none_bilstm_ctc.yml](../../configs/rec/rec_mv3_none_bilstm_ctc.yml)|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_none_bilstm_ctc_v2.0_train.tar)| + +## 2. 环境配置 + +请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。 + +## 3. 模型训练、评估、预测 + +请参考[文本识别训练教程](../../ppocr/model_train/recognition.md)。PaddleOCR对代码进行了模块化，训练不同的识别模型只需要**更换配置文件**即可。 + +### 训练 + +在完成数据准备后，便可以启动训练，训练命令如下： + +```bash linenums="1" +# 单卡训练（训练周期长，不建议） +python3 tools/train.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml + +# 多卡训练，通过--gpus参数指定卡号 +python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c rec_r34_vd_none_bilstm_ctc.yml +``` + +### 评估 + +```bash linenums="1" +# GPU 评估， Global.pretrained_model 为待测权重 +python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy +``` + +### 预测 + +```bash linenums="1" +# 预测使用的配置文件必须与训练一致 +python3 tools/infer_rec.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png +``` + +## 4. 推理部署 + +### 4.1 Python推理 + +首先将 CRNN 文本识别训练过程中保存的模型，转换成inference model。以基于Resnet34_vd骨干网络，使用MJSynth和SynthText两个英文文本识别合成数据集训练的[模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_bilstm_ctc_v2.0_train.tar) 为例，可以使用如下命令进行转换： + +```bash linenums="1" +python3 tools/export_model.py -c configs/rec/rec_r34_vd_none_bilstm_ctc.yml -o Global.pretrained_model=./rec_r34_vd_none_bilstm_ctc_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_crnn +``` + +CRNN 文本识别模型推理，可以执行如下命令： + +```bash linenums="1" +python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./inference/rec_crnn/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt" +``` + + + +执行命令后，上面图像的识别结果如下： + +```bash linenums="1" +Predicts of ./doc/imgs_words_en/word_336.png:('super', 0.9999073) +``` + +**注意**：由于上述模型是参考[DTRB](https://arxiv.org/abs/1904.01906)文本识别训练和评估流程，与超轻量级中文识别模型训练有两方面不同： + +- 训练时采用的图像分辨率不同，训练上述模型采用的图像分辨率是[3，32，100]，而中文模型训练时，为了保证长文本的识别效果，训练时采用的图像分辨率是[3, 32, 320]。预测推理程序默认的形状参数是训练中文采用的图像分辨率，即[3, 32, 320]。因此，这里推理上述英文模型时，需要通过参数rec_image_shape设置识别图像的形状。 +- 字符列表，DTRB论文中实验只是针对26个小写英文本母和10个数字进行实验，总共36个字符。所有大小字符都转成了小写字符，不在上面列表的字符都忽略，认为是空格。因此这里没有输入字符字典，而是通过如下命令生成字典.因此在推理时需要设置参数rec_char_dict_path，指定为英文字典"./ppocr/utils/ic15_dict.txt"。 + + ```python linenums="1" + self.character_str = "0123456789abcdefghijklmnopqrstuvwxyz" + dict_character = list(self.character_str) + ``` + +### 4.2 C++推理 + +准备好推理模型后，参考[cpp infer](../../ppocr/infer_deploy/cpp_infer.md)教程进行操作即可。 + +### 4.3 Serving服务化部署 + +准备好推理模型后，参考[pdserving](../../ppocr/infer_deploy/paddle_server.md)教程进行Serving服务化部署，包括Python Serving和C++ Serving两种模式。 + +### 4.4 更多推理部署 + +CRNN模型还支持以下推理部署方式： + +- Paddle2ONNX推理：准备好推理模型后，参考[paddle2onnx](../../ppocr/infer_deploy/paddle2onnx.md)教程操作。 + +## 5. FAQ + +## 引用 + +```bibtex +@ARTICLE{7801919, + author={Shi, Baoguang and Bai, Xiang and Yao, Cong}, + journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, + title={An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition}, + year={2017}, + volume={39}, + number={11}, + pages={2298-2304}, + doi={10.1109/TPAMI.2016.2646371}} +``` diff --git a/docs/algorithm/text_recognition/algorithm_rec_nrtr.en.md b/docs/algorithm/text_recognition/algorithm_rec_nrtr.en.md new file mode 100644 index 0000000000..c56d78945c --- /dev/null +++ b/docs/algorithm/text_recognition/algorithm_rec_nrtr.en.md @@ -0,0 +1,256 @@ +--- +comments: true +--- + +# NRTR + +## 1. Introduction + +Paper: +> [NRTR: A No-Recurrence Sequence-to-Sequence Model For Scene Text Recognition](https://arxiv.org/abs/1806.00926) +> Fenfen Sheng and Zhineng Chen and Bo Xu +> ICDAR, 2019 + +Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: + +|Model|Backbone|config|Acc|Download link| +| --- | --- | --- | --- | --- | +|NRTR|MTB|[rec_mtb_nrtr.yml](../../configs/rec/rec_mtb_nrtr.yml)|84.21%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mtb_nrtr_train.tar)| + +## 2. Environment + +Please refer to ["Environment Preparation"](../../ppocr/environment.en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](../../ppocr/blog/clone.en.md)to clone the project code. + +## 3. Model Training / Evaluation / Prediction + +Please refer to [Text Recognition Tutorial](../../ppocr/model_train/recognition.en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. + +### Training + +Specifically, after the data preparation is completed, the training can be started. The training command is as follows: + +```bash linenums="1" +# Single GPU training (long training period, not recommended) +python3 tools/train.py -c configs/rec/rec_mtb_nrtr.yml + +# Multi GPU training, specify the gpu number through the --gpus parameter +python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_mtb_nrtr.yml +``` + +### Evaluation + +```bash linenums="1" +# GPU evaluation +python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_mtb_nrtr.yml -o Global.pretrained_model={path/to/weights}/best_accuracy +``` + +### Prediction + +```bash linenums="1" +# The configuration file used for prediction must match the training +python3 tools/infer_rec.py -c configs/rec/rec_mtb_nrtr.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_mtb_nrtr_train/best_accuracy +``` + +## 4. Inference and Deployment + +### 4.1 Python Inference + +First, the model saved during the NRTR text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mtb_nrtr_train.tar)) ), you can use the following command to convert: + +```bash linenums="1" +python3 tools/export_model.py -c configs/rec/rec_mtb_nrtr.yml -o Global.pretrained_model=./rec_mtb_nrtr_train/best_accuracy Global.save_inference_dir=./inference/rec_mtb_nrtr +``` + +**Note:** + +- If you are training the model on your own dataset and have modified the dictionary file, please pay attention to modify the `character_dict_path` in the configuration file to the modified dictionary file. +- If you modified the input size during training, please modify the `infer_shape` corresponding to NRTR in the `tools/export_model.py` file. + +After the conversion is successful, there are three files in the directory: + +```text linenums="1" +/inference/rec_mtb_nrtr/ + ├── inference.pdiparams + ├── inference.pdiparams.info + └── inference.pdmodel +``` + +For NRTR text recognition model inference, the following commands can be executed: + +```bash linenums="1" +python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_mtb_nrtr/' --rec_algorithm='NRTR' --rec_image_shape='1,32,100' --rec_char_dict_path='./ppocr/utils/EN_symbol_dict.txt' +``` + + + +After executing the command, the prediction result (recognized text and score) of the image above is printed to the screen, an example is as follows: + +```bash linenums="1" +Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9465042352676392) +``` + +### 4.2 C++ Inference + +Not supported + +### 4.3 Serving + +Not supported + +### 4.4 More + +Not supported + +## 5. FAQ + +1. In the `NRTR` paper, Beam search is used to decode characters, but the speed is slow. Beam search is not used by default here, and greedy search is used to decode characters. + +## 6. Release Note + +1. The release/2.6 version updates the NRTR code structure. The new version of NRTR can load the model parameters of the old version (release/2.5 and before), and you may use the following code to convert the old version model parameters to the new version model parameters: + +
+
+
+2. The new version has a clean code structure and improved inference speed compared with the old version.
+
+## Citation
+
+```bibtex
+@article{Sheng2019NRTR,
+ title = {NRTR: A No-Recurrence Sequence-to-Sequence Model For Scene Text Recognition},
+ author = {Fenfen Sheng and Zhineng Chen and Bo Xu},
+ booktitle = {ICDAR},
+ year = {2019},
+ url = {http://arxiv.org/abs/1806.00926},
+ pages = {781-786}
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_nrtr.md b/docs/algorithm/text_recognition/algorithm_rec_nrtr.md
new file mode 100644
index 0000000000..68bb58b46d
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_nrtr.md
@@ -0,0 +1,272 @@
+---
+typora-copy-images-to: images
+comments: true
+---
+
+# 场景文本识别算法-NRTR
+
+## 1. 算法简介
+
+论文信息：
+> [NRTR: A No-Recurrence Sequence-to-Sequence Model For Scene Text Recognition](https://arxiv.org/abs/1806.00926)
+> Fenfen Sheng and Zhineng Chen and Bo Xu
+> ICDAR, 2019
+
+`NRTR`使用MJSynth和SynthText两个文字识别数据集训练，在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估，算法复现效果如下：
+
+|模型|骨干网络|配置文件|Acc|下载链接|
+| --- | --- | --- | --- | --- |
+|NRTR|MTB|[rec_mtb_nrtr.yml](../../configs/rec/rec_mtb_nrtr.yml)|84.21%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mtb_nrtr_train.tar)|
+
+## 2. 环境配置
+
+请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。
+
+## 3. 模型训练、评估、预测
+
+### 3.1 模型训练
+
+请参考[文本识别训练教程](../../ppocr/model_train/recognition.md)。PaddleOCR对代码进行了模块化，训练`NRTR`识别模型时需要**更换配置文件**为`NRTR`的[配置文件](../../configs/rec/rec_mtb_nrtr.yml)。
+
+#### 启动训练
+
+具体地，在完成数据准备后，便可以启动训练，训练命令如下：
+
+```bash linenums="1"
+#单卡训练（训练周期长，不建议）
+python3 tools/train.py -c configs/rec/rec_mtb_nrtr.yml
+
+# 多卡训练，通过--gpus参数指定卡号
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_mtb_nrtr.yml
+```
+
+### 3.2 评估
+
+可下载已训练完成的模型文件，使用如下命令进行评估：
+
+```bash linenums="1"
+# 注意将pretrained_model的路径设置为本地路径。
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_mtb_nrtr.yml -o Global.pretrained_model=./rec_mtb_nrtr_train/best_accuracy
+```
+
+### 3.3 预测
+
+使用如下命令进行单张图片预测：
+
+```bash linenums="1"
+# 注意将pretrained_model的路径设置为本地路径。
+python3 tools/infer_rec.py -c configs/rec/rec_mtb_nrtr.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_mtb_nrtr_train/best_accuracy
+# 预测文件夹下所有图像时，可修改infer_img为文件夹，如 Global.infer_img='./doc/imgs_words_en/'。
+```
+
+## 4. 推理部署
+
+### 4.1 Python推理
+
+首先将训练得到best模型，转换成inference model。这里以训练完成的模型为例（[模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mtb_nrtr_train.tar) )，可以使用如下命令进行转换：
+
+```bash linenums="1"
+# 注意将pretrained_model的路径设置为本地路径。
+python3 tools/export_model.py -c configs/rec/rec_mtb_nrtr.yml -o Global.pretrained_model=./rec_mtb_nrtr_train/best_accuracy Global.save_inference_dir=./inference/rec_mtb_nrtr/
+```
+
+**注意：**
+
+- 如果您是在自己的数据集上训练的模型，并且调整了字典文件，请注意修改配置文件中的`character_dict_path`是否是所需要的字典文件。
+- 如果您修改了训练时的输入大小，请修改`tools/export_model.py`文件中的对应NRTR的`infer_shape`。
+
+转换成功后，在目录下有三个文件：
+
+```text linenums="1"
+/inference/rec_mtb_nrtr/
+ ├── inference.pdiparams # 识别inference模型的参数文件
+ ├── inference.pdiparams.info # 识别inference模型的参数信息，可忽略
+ └── inference.pdmodel # 识别inference模型的program文件
+```
+
+执行如下命令进行模型推理：
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_mtb_nrtr/' --rec_algorithm='NRTR' --rec_image_shape='1,32,100' --rec_char_dict_path='./ppocr/utils/EN_symbol_dict.txt'
+# 预测文件夹下所有图像时，可修改image_dir为文件夹，如 --image_dir='./doc/imgs_words_en/'。
+```
+
+
+
+执行命令后，上面图像的预测结果（识别的文本和得分）会打印到屏幕上，示例如下：
+
+```bash linenums="1"
+Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9465042352676392)
+```
+
+**注意**：
+
+- 训练上述模型采用的图像分辨率是[1，32，100]，需要通过参数`rec_image_shape`设置为您训练时的识别图像形状。
+- 在推理时需要设置参数`rec_char_dict_path`指定字典，如果您修改了字典，请修改该参数为您的字典文件。
+- 如果您修改了预处理方法，需修改`tools/infer/predict_rec.py`中NRTR的预处理为您的预处理方法。
+
+### 4.2 C++推理部署
+
+由于C++预处理后处理还未支持NRTR，所以暂未支持
+
+### 4.3 Serving服务化部署
+
+暂不支持
+
+### 4.4 更多推理部署
+
+暂不支持
+
+## 5. FAQ
+
+1. `NRTR`论文中使用Beam搜索进行解码字符，但是速度较慢，这里默认未使用Beam搜索，以贪婪搜索进行解码字符。
+
+## 6. 发行公告
+
+1. release/2.6更新NRTR代码结构，新版NRTR可加载旧版(release/2.5及之前)模型参数，使用下面示例代码将旧版模型参数转换为新版模型参数：
+
+ Click to expand
+ + ```python linenums="1" + params = paddle.load('path/' + '.pdparams') # the old version parameters + state_dict = model.state_dict() # the new version model parameters + new_state_dict = {} + + for k1, v1 in state_dict.items(): + + k = k1 + if 'encoder' in k and 'self_attn' in k and 'qkv' in k and 'weight' in k: + + k_para = k[:13] + 'layers.' + k[13:] + q = params[k_para.replace('qkv', 'conv1')].transpose((1, 0, 2, 3)) + k = params[k_para.replace('qkv', 'conv2')].transpose((1, 0, 2, 3)) + v = params[k_para.replace('qkv', 'conv3')].transpose((1, 0, 2, 3)) + + new_state_dict[k1] = np.concatenate([q[:, :, 0, 0], k[:, :, 0, 0], v[:, :, 0, 0]], -1) + + elif 'encoder' in k and 'self_attn' in k and 'qkv' in k and 'bias' in k: + + k_para = k[:13] + 'layers.' + k[13:] + q = params[k_para.replace('qkv', 'conv1')] + k = params[k_para.replace('qkv', 'conv2')] + v = params[k_para.replace('qkv', 'conv3')] + + new_state_dict[k1] = np.concatenate([q, k, v], -1) + + elif 'encoder' in k and 'self_attn' in k and 'out_proj' in k: + + k_para = k[:13] + 'layers.' + k[13:] + new_state_dict[k1] = params[k_para] + + elif 'encoder' in k and 'norm3' in k: + k_para = k[:13] + 'layers.' + k[13:] + new_state_dict[k1] = params[k_para.replace('norm3', 'norm2')] + + elif 'encoder' in k and 'norm1' in k: + k_para = k[:13] + 'layers.' + k[13:] + new_state_dict[k1] = params[k_para] + + + elif 'decoder' in k and 'self_attn' in k and 'qkv' in k and 'weight' in k: + k_para = k[:13] + 'layers.' + k[13:] + q = params[k_para.replace('qkv', 'conv1')].transpose((1, 0, 2, 3)) + k = params[k_para.replace('qkv', 'conv2')].transpose((1, 0, 2, 3)) + v = params[k_para.replace('qkv', 'conv3')].transpose((1, 0, 2, 3)) + new_state_dict[k1] = np.concatenate([q[:, :, 0, 0], k[:, :, 0, 0], v[:, :, 0, 0]], -1) + + elif 'decoder' in k and 'self_attn' in k and 'qkv' in k and 'bias' in k: + k_para = k[:13] + 'layers.' + k[13:] + q = params[k_para.replace('qkv', 'conv1')] + k = params[k_para.replace('qkv', 'conv2')] + v = params[k_para.replace('qkv', 'conv3')] + new_state_dict[k1] = np.concatenate([q, k, v], -1) + + elif 'decoder' in k and 'self_attn' in k and 'out_proj' in k: + + k_para = k[:13] + 'layers.' + k[13:] + new_state_dict[k1] = params[k_para] + + elif 'decoder' in k and 'cross_attn' in k and 'q' in k and 'weight' in k: + k_para = k[:13] + 'layers.' + k[13:] + k_para = k_para.replace('cross_attn', 'multihead_attn') + q = params[k_para.replace('q', 'conv1')].transpose((1, 0, 2, 3)) + new_state_dict[k1] = q[:, :, 0, 0] + + elif 'decoder' in k and 'cross_attn' in k and 'q' in k and 'bias' in k: + k_para = k[:13] + 'layers.' + k[13:] + k_para = k_para.replace('cross_attn', 'multihead_attn') + q = params[k_para.replace('q', 'conv1')] + new_state_dict[k1] = q + + elif 'decoder' in k and 'cross_attn' in k and 'kv' in k and 'weight' in k: + k_para = k[:13] + 'layers.' + k[13:] + k_para = k_para.replace('cross_attn', 'multihead_attn') + k = params[k_para.replace('kv', 'conv2')].transpose((1, 0, 2, 3)) + v = params[k_para.replace('kv', 'conv3')].transpose((1, 0, 2, 3)) + new_state_dict[k1] = np.concatenate([k[:, :, 0, 0], v[:, :, 0, 0]], -1) + + elif 'decoder' in k and 'cross_attn' in k and 'kv' in k and 'bias' in k: + k_para = k[:13] + 'layers.' + k[13:] + k_para = k_para.replace('cross_attn', 'multihead_attn') + k = params[k_para.replace('kv', 'conv2')] + v = params[k_para.replace('kv', 'conv3')] + new_state_dict[k1] = np.concatenate([k, v], -1) + + elif 'decoder' in k and 'cross_attn' in k and 'out_proj' in k: + + k_para = k[:13] + 'layers.' + k[13:] + k_para = k_para.replace('cross_attn', 'multihead_attn') + new_state_dict[k1] = params[k_para] + elif 'decoder' in k and 'norm' in k: + k_para = k[:13] + 'layers.' + k[13:] + new_state_dict[k1] = params[k_para] + elif 'mlp' in k and 'weight' in k: + k_para = k[:13] + 'layers.' + k[13:] + k_para = k_para.replace('fc', 'conv') + k_para = k_para.replace('mlp.', '') + w = params[k_para].transpose((1, 0, 2, 3)) + new_state_dict[k1] = w[:, :, 0, 0] + elif 'mlp' in k and 'bias' in k: + k_para = k[:13] + 'layers.' + k[13:] + k_para = k_para.replace('fc', 'conv') + k_para = k_para.replace('mlp.', '') + w = params[k_para] + new_state_dict[k1] = w + + else: + new_state_dict[k1] = params[k1] + + if list(new_state_dict[k1].shape) != list(v1.shape): + print(k1) + + + for k, v1 in state_dict.items(): + if k not in new_state_dict.keys(): + print(1, k) + elif list(new_state_dict[k].shape) != list(v1.shape): + print(2, k) + + + + model.set_state_dict(new_state_dict) + paddle.save(model.state_dict(), 'nrtrnew_from_old_params.pdparams') + + ``` + +
+
+
+2. 新版相比与旧版，代码结构简洁，推理速度有所提高。
+
+## 引用
+
+```bibtex
+@article{Sheng2019NRTR,
+ title = {NRTR: A No-Recurrence Sequence-to-Sequence Model For Scene Text Recognition},
+ author = {Fenfen Sheng and Zhineng Chen and Bo Xu},
+ booktitle = {ICDAR},
+ year = {2019},
+ url = {http://arxiv.org/abs/1806.00926},
+ pages = {781-786}
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_parseq.en.md b/docs/algorithm/text_recognition/algorithm_rec_parseq.en.md
new file mode 100644
index 0000000000..2a971f88d6
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_parseq.en.md
@@ -0,0 +1,105 @@
+---
+comments: true
+---
+
+# PasreQ
+
+## 1. Introduction
+
+Paper:
+> [Scene Text Recognition with Permuted Autoregressive Sequence Models](https://arxiv.org/abs/2207.06966)
+> Darwin Bautista, Rowel Atienza
+> ECCV, 2021
+
+Using real datasets (real) and synthetic datsets (synth) for training respectively，and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets.
+
+- The real datasets include COCO-Text, RCTW17, Uber-Text, ArT, LSVT, MLT19, ReCTS, TextOCR and OpenVINO datasets.
+- The synthesis datasets include MJSynth and SynthText datasets.
+
+the algorithm reproduction effect is as follows:
+
+|Training Dataset|Model|Backbone|config|Acc|Download link|
+| --- | --- | --- | --- | --- | --- |
+|Synth|ParseQ|VIT|[rec_vit_parseq.yml](../../configs/rec/rec_vit_parseq.yml)|91.24%|[train model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/parseq/rec_vit_parseq_synth.tgz)|
+|Real|ParseQ|VIT|[rec_vit_parseq.yml](../../configs/rec/rec_vit_parseq.yml)|94.74%|[train model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/parseq/rec_vit_parseq_real.tgz)|
+
+## 2. Environment
+
+Please refer to ["Environment Preparation"](../../ppocr/environment.en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](../../ppocr/blog/clone.en.md)to clone the project code.
+
+## 3. Model Training / Evaluation / Prediction
+
+Please refer to [Text Recognition Tutorial](../../ppocr/model_train/recognition.en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**.
+
+### Training
+
+Specifically, after the data preparation is completed, the training can be started. The training command is as follows:
+
+```bash linenums="1"
+# Single GPU training (long training period, not recommended)
+python3 tools/train.py -c configs/rec/rec_vit_parseq.yml
+
+# Multi GPU training, specify the gpu number through the --gpus parameter
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_vit_parseq.yml
+```
+
+### Evaluation
+
+```bash linenums="1"
+# GPU evaluation
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_vit_parseq.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### Prediction
+
+```bash linenums="1"
+# The configuration file used for prediction must match the training
+python3 tools/infer_rec.py -c configs/rec/rec_vit_parseq.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. Inference and Deployment
+
+### 4.1 Python Inference
+
+First, the model saved during the SAR text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.1/parseq/rec_vit_parseq_real.tgz) ), you can use the following command to convert:
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_vit_parseq.yml -o Global.pretrained_model=./rec_vit_parseq_real/best_accuracy Global.save_inference_dir=./inference/rec_parseq
+```
+
+For SAR text recognition model inference, the following commands can be executed:
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_parseq/" --rec_image_shape="3, 32, 128" --rec_algorithm="ParseQ" --rec_char_dict_path="ppocr/utils/dict/parseq_dict.txt" --max_text_length=25 --use_space_char=False
+```
+
+### 4.2 C++ Inference
+
+Not supported
+
+### 4.3 Serving
+
+Not supported
+
+### 4.4 More
+
+Not supported
+
+## 5. FAQ
+
+## Citation
+
+```bibtex
+@InProceedings{bautista2022parseq,
+ title={Scene Text Recognition with Permuted Autoregressive Sequence Models},
+ author={Bautista, Darwin and Atienza, Rowel},
+ booktitle={European Conference on Computer Vision},
+ pages={178--196},
+ month={10},
+ year={2022},
+ publisher={Springer Nature Switzerland},
+ address={Cham},
+ doi={10.1007/978-3-031-19815-1_11},
+ url={https://doi.org/10.1007/978-3-031-19815-1_11}
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_parseq.md b/docs/algorithm/text_recognition/algorithm_rec_parseq.md
new file mode 100644
index 0000000000..73caf946c1
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_parseq.md
@@ -0,0 +1,105 @@
+---
+comments: true
+---
+
+# ParseQ
+
+## 1. 算法简介
+
+论文信息：
+> [Scene Text Recognition with Permuted Autoregressive Sequence Models](https://arxiv.org/abs/2207.06966)
+> Darwin Bautista, Rowel Atienza
+> ECCV, 2021
+
+原论文分别使用真实文本识别数据集(Real)和合成文本识别数据集(Synth)进行训练，在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估。其中：
+
+- 真实文本识别数据集(Real)包含COCO-Text, RCTW17, Uber-Text, ArT, LSVT, MLT19, ReCTS, TextOCR, OpenVINO数据集
+- 合成文本识别数据集(Synth)包含MJSynth和SynthText数据集
+
+在不同数据集上训练的算法的复现效果如下：
+
+|数据集|模型|骨干网络|配置文件|Acc|下载链接|
+| --- | --- | --- | --- | --- | --- |
+|Synth|ParseQ|VIT|[rec_vit_parseq.yml](../../configs/rec/rec_vit_parseq.yml)|91.24%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/parseq/rec_vit_parseq_synth.tgz)|
+|Real|ParseQ|VIT|[rec_vit_parseq.yml](../../configs/rec/rec_vit_parseq.yml)|94.74%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/parseq/rec_vit_parseq_real.tgz)|
+
+## 2. 环境配置
+
+请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。
+
+## 3. 模型训练、评估、预测
+
+请参考[文本识别教程](../../ppocr/model_train/recognition.md)。PaddleOCR对代码进行了模块化，训练不同的识别模型只需要**更换配置文件**即可。
+
+### 训练
+
+具体地，在完成数据准备后，便可以启动训练，训练命令如下：
+
+```bash linenums="1"
+# 单卡训练（训练周期长，不建议）
+python3 tools/train.py -c configs/rec/rec_vit_parseq.yml
+
+# 多卡训练，通过--gpus参数指定卡号
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_vit_parseq.yml
+```
+
+### 评估
+
+```bash linenums="1"
+# GPU 评估， Global.pretrained_model 为待测权重
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_vit_parseq.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### 预测
+
+```bash linenums="1"
+# 预测使用的配置文件必须与训练一致
+python3 tools/infer_rec.py -c configs/rec/rec_vit_parseq.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. 推理部署
+
+### 4.1 Python推理
+
+首先将ParseQ文本识别训练过程中保存的模型，转换成inference model。（ [模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.1/parseq/rec_vit_parseq_real.tgz) )，可以使用如下命令进行转换：
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_vit_parseq.yml -o Global.pretrained_model=./rec_vit_parseq_real/best_accuracy Global.save_inference_dir=./inference/rec_parseq
+```
+
+ParseQ文本识别模型推理，可以执行如下命令：
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_parseq/" --rec_image_shape="3, 32, 128" --rec_algorithm="ParseQ" --rec_char_dict_path="ppocr/utils/dict/parseq_dict.txt" --max_text_length=25 --use_space_char=False
+```
+
+### 4.2 C++推理
+
+由于C++预处理后处理还未支持ParseQ，所以暂未支持
+
+### 4.3 Serving服务化部署
+
+暂不支持
+
+### 4.4 更多推理部署
+
+暂不支持
+
+## 5. FAQ
+
+## 引用
+
+```bibtex
+@InProceedings{bautista2022parseq,
+ title={Scene Text Recognition with Permuted Autoregressive Sequence Models},
+ author={Bautista, Darwin and Atienza, Rowel},
+ booktitle={European Conference on Computer Vision},
+ pages={178--196},
+ month={10},
+ year={2022},
+ publisher={Springer Nature Switzerland},
+ address={Cham},
+ doi={10.1007/978-3-031-19815-1_11},
+ url={https://doi.org/10.1007/978-3-031-19815-1_11}
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_rare.en.md b/docs/algorithm/text_recognition/algorithm_rec_rare.en.md
new file mode 100644
index 0000000000..599ec232f9
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_rare.en.md
@@ -0,0 +1,100 @@
+---
+comments: true
+---
+
+# RARE
+
+## 1. Introduction
+
+Paper information:
+> [Robust Scene Text Recognition with Automatic Rectification](https://arxiv.org/abs/1603.03915v2)
+> Baoguang Shi, Xinggang Wang, Pengyuan Lyu, Cong Yao, Xiang Bai∗
+> CVPR, 2016
+
+Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows:
+
+|Models|Backbone Networks|Configuration Files|Avg Accuracy|Download Links|
+| --- | --- | --- | --- | --- |
+|RARE|Resnet34_vd|[configs/rec/rec_r34_vd_tps_bilstm_att.yml](../../configs/rec/rec_r34_vd_tps_bilstm_att.yml)|83.60%|[training model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_tps_bilstm_att_v2.0_train.tar)|
+|RARE|MobileNetV3|[configs/rec/rec_mv3_tps_bilstm_att.yml](../../configs/rec/rec_mv3_tps_bilstm_att.yml)|82.50%|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_tps_bilstm_att_v2.0_train.tar)|
+
+## 2. Environment
+
+Please refer to [Operating Environment Preparation](../../ppocr/environment.en.md) to configure the PaddleOCR operating environment, and refer to [Project Clone](../../ppocr/blog/clone.en.md)to clone the project code.
+
+## 3. Model Training / Evaluation / Prediction
+
+Please refer to [Text Recognition Training Tutorial](../../ppocr/model_train/recognition.en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. Take the backbone network based on Resnet34_vd as an example:
+
+### 3.1 Training
+
+````bash linenums="1"
+# Single card training (long training period, not recommended)
+python3 tools/train.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml
+# Multi-card training, specify the card number through the --gpus parameter
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml
+````
+
+### 3.2 Evaluation
+
+````bash linenums="1"
+# GPU evaluation, Global.pretrained_model is the model to be evaluated
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+````
+
+### 3.3 Prediction
+
+````bash linenums="1"
+python3 tools/infer_rec.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+````
+
+## 4. Inference
+
+### 4.1 Python Inference
+
+First, convert the model saved during the RARE text recognition training process into an inference model. Take the model trained on the MJSynth and SynthText text recognition datasets based on the Resnet34_vd backbone network as an example ([Model download address](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_tps_bilstm_att_v2.0_train.tar) ), which can be converted using the following command:
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml -o Global.pretrained_model=./rec_r34_vd_tps_bilstm_att_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_rare
+````
+
+RARE text recognition model inference, you can execute the following commands:
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_rare/" --rec_image_shape="3, 32, 100" --rec_char_dict_path= "./ppocr/utils/ic15_dict.txt"
+````
+
+The inference results are as follows:
+
+
+
+````text linenums="1"
+Predicts of doc/imgs_words/en/word_1.png:('joint ', 0.9999969601631165)
+````
+
+### 4.2 C++ Inference
+
+Not currently supported
+
+### 4.3 Serving
+
+Not currently supported
+
+### 4.4 More
+
+The RARE model also supports the following inference deployment methods:
+
+- Paddle2ONNX Inference: After preparing the inference model, refer to the [paddle2onnx](../../ppocr/infer_deploy/paddle2onnx.en.md) tutorial.
+
+## 5. FAQ
+
+## Citation
+
+````bibtex
+@inproceedings{2016Robust,
+ title={Robust Scene Text Recognition with Automatic Rectification},
+ author={ Shi, B. and Wang, X. and Lyu, P. and Cong, Y. and Xiang, B. },
+ booktitle={2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
+ year={2016},
+}
+````
diff --git a/docs/algorithm/text_recognition/algorithm_rec_rare.md b/docs/algorithm/text_recognition/algorithm_rec_rare.md
new file mode 100644
index 0000000000..3464f9c8fe
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_rare.md
@@ -0,0 +1,100 @@
+---
+comments: true
+---
+
+# RARE
+
+## 1. 算法简介
+
+论文信息：
+> [Robust Scene Text Recognition with Automatic Rectification](https://arxiv.org/abs/1603.03915v2)
+> Baoguang Shi, Xinggang Wang, Pengyuan Lyu, Cong Yao, Xiang Bai∗
+> CVPR, 2016
+
+使用MJSynth和SynthText两个文字识别数据集训练，在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估，算法复现效果如下：
+
+|模型|骨干网络|配置文件|Avg Accuracy|下载链接|
+| --- | --- | --- | --- | --- |
+|RARE|Resnet34_vd|[configs/rec/rec_r34_vd_tps_bilstm_att.yml](../../configs/rec/rec_r34_vd_tps_bilstm_att.yml)|83.60%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_tps_bilstm_att_v2.0_train.tar)|
+|RARE|MobileNetV3|[configs/rec/rec_mv3_tps_bilstm_att.yml](../../configs/rec/rec_mv3_tps_bilstm_att.yml)|82.50%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_tps_bilstm_att_v2.0_train.tar)|
+
+## 2. 环境配置
+
+请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。
+
+## 3. 模型训练、评估、预测
+
+请参考[文本识别训练教程](../../ppocr/model_train/recognition.md)。PaddleOCR对代码进行了模块化，训练不同的识别模型只需要**更换配置文件**即可。以基于Resnet34_vd骨干网络为例:
+
+### 3.1 训练
+
+```bash linenums="1"
+# 单卡训练（训练周期长，不建议）
+python3 tools/train.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml
+# 多卡训练，通过--gpus参数指定卡号
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml
+```
+
+### 3.2 评估
+
+```bash linenums="1"
+# GPU评估, Global.pretrained_model为待评估模型
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### 3.3 预测
+
+```bash linenums="1"
+python3 tools/infer_rec.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. 推理部署
+
+### 4.1 Python推理
+
+首先将RARE文本识别训练过程中保存的模型，转换成inference model。以基于Resnet34_vd骨干网络，在MJSynth和SynthText两个文字识别数据集训练得到的模型为例（ [模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_tps_bilstm_att_v2.0_train.tar) )，可以使用如下命令进行转换：
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_r34_vd_tps_bilstm_att.yml -o Global.pretrained_model=./rec_r34_vd_tps_bilstm_att_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_rare
+```
+
+RARE文本识别模型推理，可以执行如下命令：
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_rare/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt"
+```
+
+推理结果如下所示：
+
+
+
+```text linenums="1"
+Predicts of doc/imgs_words/en/word_1.png:('joint ', 0.9999969601631165)
+```
+
+### 4.2 C++推理
+
+暂不支持
+
+### 4.3 Serving服务化部署
+
+暂不支持
+
+### 4.4 更多推理部署
+
+RARE模型还支持以下推理部署方式：
+
+- Paddle2ONNX推理：准备好推理模型后，参考[paddle2onnx](../../ppocr/infer_deploy/paddle2onnx.md)教程操作。
+
+## 5. FAQ
+
+## 引用
+
+```bibtex
+@inproceedings{2016Robust,
+ title={Robust Scene Text Recognition with Automatic Rectification},
+ author={ Shi, B. and Wang, X. and Lyu, P. and Cong, Y. and Xiang, B. },
+ booktitle={2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
+ year={2016},
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_rfl.en.md b/docs/algorithm/text_recognition/algorithm_rec_rfl.en.md
new file mode 100644
index 0000000000..ab586c718b
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_rfl.en.md
@@ -0,0 +1,126 @@
+---
+comments: true
+---
+
+# RFL
+
+## 1. Introduction
+
+Paper:
+> [Reciprocal Feature Learning via Explicit and Implicit Tasks in Scene Text Recognition](https://arxiv.org/abs/2105.06229.pdf)
+> Hui Jiang, Yunlu Xu, Zhanzhan Cheng, Shiliang Pu, Yi Niu, Wenqi Ren, Fei Wu, and Wenming Tan
+> ICDAR, 2021
+
+Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows:
+
+|Model|Backbone|config|Acc|Download link|
+| --- | --- | --- | --- | --- |
+|RFL-CNT|ResNetRFL|[rec_resnet_rfl_visual.yml](../../configs/rec/rec_resnet_rfl_visual.yml)|93.40%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_resnet_rfl_visual_train.tar)|
+|RFL-Att|ResNetRFL|[rec_resnet_rfl_att.yml](../../configs/rec/rec_resnet_rfl_att.yml)|88.63%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_resnet_rfl_att_train.tar)|
+
+## 2. Environment
+
+Please refer to ["Environment Preparation"](../../ppocr/environment.en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](../../ppocr/blog/clone.en.md)to clone the project code.
+
+## 3. Model Training / Evaluation / Prediction
+
+PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**.
+
+### Training
+
+Specifically, after the data preparation is completed, the training can be started. The training command is as follows:
+
+```bash linenums="1"
+#step1:train the CNT branch
+# Single GPU training (long training period, not recommended)
+python3 tools/train.py -c configs/rec/rec_resnet_rfl_visual.yml
+
+# Multi GPU training, specify the gpu number through the --gpus parameter
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_resnet_rfl_visual.yml
+
+#step2:joint training of CNT and Att branches
+# Single GPU training (long training period, not recommended)
+python3 tools/train.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+
+# Multi GPU training, specify the gpu number through the --gpus parameter
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### Evaluation
+
+```bash linenums="1"
+# GPU evaluation
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### Prediction
+
+```bash linenums="1"
+# The configuration file used for prediction must match the training
+python3 tools/infer_rec.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+## 4. Inference and Deployment
+
+### 4.1 Python Inference
+
+First, the model saved during the RFL text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/contribution/rec_resnet_rfl.tar)) ), you can use the following command to convert:
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.save_inference_dir=./inference/rec_resnet_rfl_att
+```
+
+**Note:**
+
+- If you are training the model on your own dataset and have modified the dictionary file, please pay attention to modify the `character_dict_path` in the configuration file to the modified dictionary file.
+- If you modified the input size during training, please modify the `infer_shape` corresponding to NRTR in the `tools/export_model.py` file.
+
+After the conversion is successful, there are three files in the directory:
+
+```text linenums="1"
+/inference/rec_resnet_rfl_att/
+ ├── inference.pdiparams
+ ├── inference.pdiparams.info
+ └── inference.pdmodel
+```
+
+For RFL text recognition model inference, the following commands can be executed:
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_resnet_rfl_att/' --rec_algorithm='RFL' --rec_image_shape='1,32,100'
+```
+
+
+
+After executing the command, the prediction result (recognized text and score) of the image above is printed to the screen, an example is as follows:
+The result is as follows:
+
+```bash linenums="1"
+Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9999927282333374)
+```
+
+### 4.2 C++ Inference
+
+Not supported
+
+### 4.3 Serving
+
+Not supported
+
+### 4.4 More
+
+Not supported
+
+## 5. FAQ
+
+## Citation
+
+```bibtex
+@article{2021Reciprocal,
+ title = {Reciprocal Feature Learning via Explicit and Implicit Tasks in Scene Text Recognition},
+ author = {Jiang, H. and Xu, Y. and Cheng, Z. and Pu, S. and Niu, Y. and Ren, W. and Wu, F. and Tan, W. },
+ booktitle = {ICDAR},
+ year = {2021},
+ url = {https://arxiv.org/abs/2105.06229}
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_rfl.md b/docs/algorithm/text_recognition/algorithm_rec_rfl.md
new file mode 100644
index 0000000000..9ab9bd8c46
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_rfl.md
@@ -0,0 +1,140 @@
+---
+comments: true
+---
+
+# 场景文本识别算法-RFL
+
+## 1. 算法简介
+
+论文信息：
+> [Reciprocal Feature Learning via Explicit and Implicit Tasks in Scene Text Recognition](https://arxiv.org/abs/2105.06229.pdf)
+> Hui Jiang, Yunlu Xu, Zhanzhan Cheng, Shiliang Pu, Yi Niu, Wenqi Ren, Fei Wu, and Wenming Tan
+> ICDAR, 2021
+
+`RFL`使用MJSynth和SynthText两个文字识别数据集训练，在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估，算法复现效果如下：
+
+|模型|骨干网络|配置文件|Acc|下载链接|
+| --- | --- | --- | --- | --- |
+|RFL-CNT|ResNetRFL|[rec_resnet_rfl_visual.yml](../../configs/rec/rec_resnet_rfl_visual.yml)|93.40%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_resnet_rfl_visual_train.tar)|
+|RFL-Att|ResNetRFL|[rec_resnet_rfl_att.yml](../../configs/rec/rec_resnet_rfl_att.yml)|88.63%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_resnet_rfl_att_train.tar)|
+
+## 2. 环境配置
+
+请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。
+
+## 3. 模型训练、评估、预测
+
+### 3.1 模型训练
+
+PaddleOCR对代码进行了模块化，训练`RFL`识别模型时需要**更换配置文件**为`RFL`的[配置文件](../../configs/rec/rec_resnet_rfl_att.yml)。
+
+#### 启动训练
+
+具体地，在完成数据准备后，便可以启动训练，训练命令如下：
+
+```bash linenums="1"
+#step1:训练CNT分支
+#单卡训练（训练周期长，不建议）
+python3 tools/train.py -c configs/rec/rec_resnet_rfl_visual.yml
+
+# 多卡训练，通过--gpus参数指定卡号
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_resnet_rfl_visual.yml
+
+#step2:联合训练CNT和Att分支,注意将pretrained_model的路径设置为本地路径。
+#单卡训练（训练周期长，不建议）
+python3 tools/train.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.pretrained_model=./output/rec/rec_resnet_rfl_visual/best_accuracy
+
+# 多卡训练，通过--gpus参数指定卡号
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.pretrained_model=./output/rec/rec_resnet_rfl_visual/best_accuracy
+```
+
+### 3.2 评估
+
+可下载已训练完成的[模型文件](https://paddleocr.bj.bcebos.com/contribution/rec_resnet_rfl.tar)，使用如下命令进行评估：
+
+```bash linenums="1"
+# 注意将pretrained_model的路径设置为本地路径。
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.pretrained_model=./output/rec/rec_resnet_rfl_att/best_accuracy
+```
+
+### 3.3 预测
+
+使用如下命令进行单张图片预测：
+
+```bash linenums="1"
+# 注意将pretrained_model的路径设置为本地路径。
+python3 tools/infer_rec.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./output/rec/rec_resnet_rfl_att/best_accuracy
+# 预测文件夹下所有图像时，可修改infer_img为文件夹，如 Global.infer_img='./doc/imgs_words_en/'。
+```
+
+## 4. 推理部署
+
+### 4.1 Python推理
+
+首先将训练得到best模型，转换成inference model。这里以训练完成的模型为例（[模型下载地址](https://paddleocr.bj.bcebos.com/contribution/rec_resnet_rfl.tar) )，可以使用如下命令进行转换：
+
+```bash linenums="1"
+# 注意将pretrained_model的路径设置为本地路径。
+python3 tools/export_model.py -c configs/rec/rec_resnet_rfl_att.yml -o Global.pretrained_model=./output/rec/rec_resnet_rfl_att/best_accuracy Global.save_inference_dir=./inference/rec_resnet_rfl_att/
+```
+
+**注意：** 如果您是在自己的数据集上训练的模型，并且调整了字典文件，请注意修改配置文件中的`character_dict_path`是否是所需要的字典文件。
+
+- 如果您修改了训练时的输入大小，请修改`tools/export_model.py`文件中的对应RFL的`infer_shape`。
+
+转换成功后，在目录下有三个文件：
+
+```text linenums="1"
+/inference/rec_resnet_rfl_att/
+ ├── inference.pdiparams # 识别inference模型的参数文件
+ ├── inference.pdiparams.info # 识别inference模型的参数信息，可忽略
+ └── inference.pdmodel # 识别inference模型的program文件
+```
+
+执行如下命令进行模型推理：
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_resnet_rfl_att/' --rec_algorithm='RFL' --rec_image_shape='1,32,100'
+# 预测文件夹下所有图像时，可修改image_dir为文件夹，如 --image_dir='./doc/imgs_words_en/'。
+```
+
+
+
+执行命令后，上面图像的预测结果（识别的文本和得分）会打印到屏幕上，示例如下：
+结果如下：
+
+```bash linenums="1"
+Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9999927282333374)
+```
+
+**注意**：
+
+- 训练上述模型采用的图像分辨率是[1，32，100]，需要通过参数`rec_image_shape`设置为您训练时的识别图像形状。
+- 在推理时需要设置参数`rec_char_dict_path`指定字典，如果您修改了字典，请修改该参数为您的字典文件。
+- 如果您修改了预处理方法，需修改`tools/infer/predict_rec.py`中RFL的预处理为您的预处理方法。
+
+### 4.2 C++推理部署
+
+由于C++预处理后处理还未支持RFL，所以暂未支持
+
+### 4.3 Serving服务化部署
+
+暂不支持
+
+### 4.4 更多推理部署
+
+暂不支持
+
+## 5. FAQ
+
+## 引用
+
+```bibtex
+@article{2021Reciprocal,
+ title = {Reciprocal Feature Learning via Explicit and Implicit Tasks in Scene Text Recognition},
+ author = {Jiang, H. and Xu, Y. and Cheng, Z. and Pu, S. and Niu, Y. and Ren, W. and Wu, F. and Tan, W. },
+ booktitle = {ICDAR},
+ year = {2021},
+ url = {https://arxiv.org/abs/2105.06229}
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_robustscanner.en.md b/docs/algorithm/text_recognition/algorithm_rec_robustscanner.en.md
new file mode 100644
index 0000000000..a4aa3bc7c9
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_robustscanner.en.md
@@ -0,0 +1,96 @@
+---
+comments: true
+---
+
+# RobustScanner
+
+## 1. Introduction
+
+Paper:
+> [RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition](https://arxiv.org/pdf/2007.07542.pdf)
+> Xiaoyu Yue, Zhanghui Kuang, Chenhao Lin, Hongbin Sun, Wayne
+Zhang
+> ECCV, 2020
+
+Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows:
+
+|Model|Backbone|config|Acc|Download link|
+| --- | --- | --- | --- | --- |
+|RobustScanner|ResNet31|[rec_r31_robustscanner.yml](../../configs/rec/rec_r31_robustscanner.yml)|87.77%|[trained model](https://paddleocr.bj.bcebos.com/contribution/rec_r31_robustscanner.tar)|
+
+Note:In addition to using the two text recognition datasets MJSynth and SynthText, [SynthAdd](https://pan.baidu.com/share/init?surl=uV0LtoNmcxbO-0YA7Ch4dg) data (extraction code: 627x), and some real data are used in training, the specific data details can refer to the paper.
+
+## 2. Environment
+
+Please refer to ["Environment Preparation"](../../ppocr/environment.en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](../../ppocr/blog/clone.en.md)to clone the project code.
+
+## 3. Model Training / Evaluation / Prediction
+
+Please refer to [Text Recognition Tutorial](../../ppocr/model_train/recognition.en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**.
+
+### Training
+
+Specifically, after the data preparation is completed, the training can be started. The training command is as follows:
+
+```bash linenums="1"
+# Single GPU training (long training period, not recommended)
+python3 tools/train.py -c configs/rec/rec_r31_robustscanner.yml
+
+# Multi GPU training, specify the gpu number through the --gpus parameter
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r31_robustscanner.yml
+```
+
+### Evaluation
+
+```bash linenums="1"
+# GPU evaluation
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r31_robustscanner.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### Prediction
+
+```bash linenums="1"
+# The configuration file used for prediction must match the training
+python3 tools/infer_rec.py -c configs/rec/rec_r31_robustscanner.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. Inference and Deployment
+
+### 4.1 Python Inference
+
+First, the model saved during the RobustScanner text recognition training process is converted into an inference model. you can use the following command to convert:
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_r31_robustscanner.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.save_inference_dir=./inference/rec_r31_robustscanner
+```
+
+For RobustScanner text recognition model inference, the following commands can be executed:
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_r31_robustscanner/" --rec_image_shape="3, 48, 48, 160" --rec_algorithm="RobustScanner" --rec_char_dict_path="ppocr/utils/dict90.txt" --use_space_char=False
+```
+
+### 4.2 C++ Inference
+
+Not supported
+
+### 4.3 Serving
+
+Not supported
+
+### 4.4 More
+
+Not supported
+
+## 5. FAQ
+
+## Citation
+
+```bibtex
+@article{2020RobustScanner,
+ title={RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition},
+ author={Xiaoyu Yue and Zhanghui Kuang and Chenhao Lin and Hongbin Sun and Wayne Zhang},
+ journal={ECCV2020},
+ year={2020},
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_robustscanner.md b/docs/algorithm/text_recognition/algorithm_rec_robustscanner.md
new file mode 100644
index 0000000000..6abf33ca43
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_robustscanner.md
@@ -0,0 +1,97 @@
+---
+typora-copy-images-to: images
+comments: true
+---
+
+# RobustScanner
+
+## 1. 算法简介
+
+论文信息：
+> [RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition](https://arxiv.org/pdf/2007.07542.pdf)
+> Xiaoyu Yue, Zhanghui Kuang, Chenhao Lin, Hongbin Sun, Wayne
+Zhang
+> ECCV, 2020
+
+使用MJSynth和SynthText两个合成文字识别数据集训练，在IIIT, SVT, IC13, IC15, SVTP, CUTE数据集上进行评估，算法复现效果如下：
+
+|模型|骨干网络|配置文件|Acc|下载链接|
+| --- | --- | --- | --- | --- |
+|RobustScanner|ResNet31|[rec_r31_robustscanner.yml](../../configs/rec/rec_r31_robustscanner.yml)|87.77%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_r31_robustscanner.tar)|
+
+注：除了使用MJSynth和SynthText两个文字识别数据集外，还加入了[SynthAdd](https://pan.baidu.com/share/init?surl=uV0LtoNmcxbO-0YA7Ch4dg)数据（提取码：627x），和部分真实数据，具体数据细节可以参考论文。
+
+## 2. 环境配置
+
+请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。
+
+## 3. 模型训练、评估、预测
+
+请参考[文本识别教程](../../ppocr/model_train/recognition.md)。PaddleOCR对代码进行了模块化，训练不同的识别模型只需要**更换配置文件**即可。
+
+### 训练
+
+具体地，在完成数据准备后，便可以启动训练，训练命令如下：
+
+```bash linenums="1"
+#单卡训练（训练周期长，不建议）
+python3 tools/train.py -c configs/rec/rec_r31_robustscanner.yml
+
+# 多卡训练，通过--gpus参数指定卡号
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r31_robustscanner.yml
+```
+
+### 评估
+
+```bash linenums="1"
+# GPU 评估， Global.pretrained_model 为待测权重
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r31_robustscanner.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### 预测
+
+```bash linenums="1"
+# 预测使用的配置文件必须与训练一致
+python3 tools/infer_rec.py -c configs/rec/rec_r31_robustscanner.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. 推理部署
+
+### 4.1 Python推理
+
+首先将RobustScanner文本识别训练过程中保存的模型，转换成inference model。可以使用如下命令进行转换：
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_r31_robustscanner.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.save_inference_dir=./inference/rec_r31_robustscanner
+```
+
+RobustScanner文本识别模型推理，可以执行如下命令：
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_r31_robustscanner/" --rec_image_shape="3, 48, 48, 160" --rec_algorithm="RobustScanner" --rec_char_dict_path="ppocr/utils/dict90.txt" --use_space_char=False
+```
+
+### 4.2 C++推理
+
+由于C++预处理后处理还未支持RobustScanner，所以暂未支持
+
+### 4.3 Serving服务化部署
+
+暂不支持
+
+### 4.4 更多推理部署
+
+暂不支持
+
+## 5. FAQ
+
+## 引用
+
+```bibtex
+@article{2020RobustScanner,
+ title={RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition},
+ author={Xiaoyu Yue and Zhanghui Kuang and Chenhao Lin and Hongbin Sun and Wayne Zhang},
+ journal={ECCV2020},
+ year={2020},
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_rosetta.en.md b/docs/algorithm/text_recognition/algorithm_rec_rosetta.en.md
new file mode 100644
index 0000000000..a49ad8a802
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_rosetta.en.md
@@ -0,0 +1,100 @@
+---
+comments: true
+---
+
+# Rosetta
+
+## 1. Introduction
+
+Paper information:
+> [Rosetta: Large Scale System for Text Detection and Recognition in Images](https://arxiv.org/abs/1910.05085)
+> Borisyuk F , Gordo A , V Sivakumar
+> KDD, 2018
+
+Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows:
+
+|Models|Backbone Networks|Configuration Files|Avg Accuracy|Download Links|
+| --- | --- | --- | --- | --- |
+|Rosetta|Resnet34_vd|[configs/rec/rec_r34_vd_none_none_ctc.yml](../../configs/rec/rec_r34_vd_none_none_ctc.yml)|79.11%|[training model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_none_ctc_v2.0_train.tar)|
+|Rosetta|MobileNetV3|[configs/rec/rec_mv3_none_none_ctc.yml](../../configs/rec/rec_mv3_none_none_ctc.yml)|75.80%|[training model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_none_none_ctc_v2.0_train.tar)|
+
+## 2. Environment
+
+Please refer to [Operating Environment Preparation](../../ppocr/environment.en.md) to configure the PaddleOCR operating environment, and refer to [Project Clone](../../ppocr/blog/clone.en.md)to clone the project code.
+
+## 3. Model Training / Evaluation / Prediction
+
+Please refer to [Text Recognition Training Tutorial](../../ppocr/model_train/recognition.en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. Take the backbone network based on Resnet34_vd as an example:
+
+### 3.1 Training
+
+```bash linenums="1"
+# Single card training (long training period, not recommended)
+python3 tools/train.py -c configs/rec/rec_r34_vd_none_none_ctc.yml
+# Multi-card training, specify the card number through the --gpus parameter
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r34_vd_none_none_ctc.yml
+```
+
+### 3.2 Evaluation
+
+```bash linenums="1"
+# GPU evaluation, Global.pretrained_model is the model to be evaluated
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r34_vd_none_none_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### 3.3 Prediction
+
+```bash linenums="1"
+python3 tools/infer_rec.py -c configs/rec/rec_r34_vd_none_none_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. Inference and Deployment
+
+### 4.1 Python Inference
+
+First, convert the model saved during the Rosetta text recognition training process into an inference model. Take the model trained on the MJSynth and SynthText text recognition datasets based on the Resnet34_vd backbone network as an example ( [Model download address](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_none_ctc_v2.0_train.tar) ), which can be converted using the following command:
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_r34_vd_none_none_ctc.yml -o Global.pretrained_model=./rec_r34_vd_none_none_ctc_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_rosetta
+```
+
+Rosetta text recognition model inference, you can execute the following commands:
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_rosetta/" --rec_image_shape="3, 32, 100" --rec_char_dict_path= "./ppocr/utils/ic15_dict.txt"
+```
+
+The inference results are as follows:
+
+
+
+```bash linenums="1"
+Predicts of doc/imgs_words/en/word_1.png:('joint', 0.9999982714653015)
+```
+
+### 4.2 C++ Inference
+
+Not currently supported
+
+### 4.3 Serving
+
+Not currently supported
+
+### 4.4 More
+
+The Rosetta model also supports the following inference deployment methods:
+
+- Paddle2ONNX Inference: After preparing the inference model, refer to the [paddle2onnx](../../ppocr/infer_deploy/paddle2onnx.en.md) tutorial.
+
+## 5. FAQ
+
+## Citation
+
+```bibtex
+@inproceedings{2018Rosetta,
+ title={Rosetta: Large Scale System for Text Detection and Recognition in Images},
+ author={ Borisyuk, Fedor and Gordo, Albert and Sivakumar, Viswanath },
+ booktitle={the 24th ACM SIGKDD International Conference},
+ year={2018},
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_rosetta.md b/docs/algorithm/text_recognition/algorithm_rec_rosetta.md
new file mode 100644
index 0000000000..659a904348
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_rosetta.md
@@ -0,0 +1,101 @@
+---
+typora-copy-images-to: images
+comments: true
+---
+
+# Rosetta
+
+## 1. 算法简介
+
+论文信息：
+> [Rosetta: Large Scale System for Text Detection and Recognition in Images](https://arxiv.org/abs/1910.05085)
+> Borisyuk F , Gordo A , V Sivakumar
+> KDD, 2018
+
+使用MJSynth和SynthText两个文字识别数据集训练，在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估， 算法复现效果如下：
+
+|模型|骨干网络|配置文件|Avg Accuracy|下载链接|
+| --- | --- | --- | --- | --- |
+|Rosetta|Resnet34_vd|[configs/rec/rec_r34_vd_none_none_ctc.yml](../../configs/rec/rec_r34_vd_none_none_ctc.yml)|79.11%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_none_ctc_v2.0_train.tar)|
+|Rosetta|MobileNetV3|[configs/rec/rec_mv3_none_none_ctc.yml](../../configs/rec/rec_mv3_none_none_ctc.yml)|75.80%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_none_none_ctc_v2.0_train.tar)|
+
+## 2. 环境配置
+
+请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。
+
+## 3. 模型训练、评估、预测
+
+请参考[文本识别训练教程](../../ppocr/model_train/recognition.md)。PaddleOCR对代码进行了模块化，训练不同的识别模型只需要**更换配置文件**即可。 以基于Resnet34_vd骨干网络为例:
+
+### 3.1 训练
+
+```bash linenums="1"
+# 单卡训练（训练周期长，不建议）
+python3 tools/train.py -c configs/rec/rec_r34_vd_none_none_ctc.yml
+# 多卡训练，通过--gpus参数指定卡号
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r34_vd_none_none_ctc.yml
+```
+
+### 3.2 评估
+
+```bash linenums="1"
+# GPU评估, Global.pretrained_model为待评估模型
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r34_vd_none_none_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### 3.3 预测
+
+```bash linenums="1"
+python3 tools/infer_rec.py -c configs/rec/rec_r34_vd_none_none_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. 推理部署
+
+### 4.1 Python推理
+
+首先将Rosetta文本识别训练过程中保存的模型，转换成inference model。以基于Resnet34_vd骨干网络，在MJSynth和SynthText两个文字识别数据集训练得到的模型为例（ [模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_none_ctc_v2.0_train.tar) )，可以使用如下命令进行转换：
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_r34_vd_none_none_ctc.yml -o Global.pretrained_model=./rec_r34_vd_none_none_ctc_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_rosetta
+```
+
+Rosetta文本识别模型推理，可以执行如下命令：
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_rosetta/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt"
+```
+
+推理结果如下所示：
+
+
+
+```bash linenums="1"
+Predicts of doc/imgs_words/en/word_1.png:('joint', 0.9999982714653015)
+```
+
+### 4.2 C++推理
+
+暂不支持
+
+### 4.3 Serving服务化部署
+
+暂不支持
+
+### 4.4 更多推理部署
+
+Rosetta模型还支持以下推理部署方式：
+
+- Paddle2ONNX推理：准备好推理模型后，参考[paddle2onnx](../../ppocr/infer_deploy/paddle2onnx.md)教程操作。
+
+## 5. FAQ
+
+## 引用
+
+```bibtex
+@inproceedings{2018Rosetta,
+ title={Rosetta: Large Scale System for Text Detection and Recognition in Images},
+ author={ Borisyuk, Fedor and Gordo, Albert and Sivakumar, Viswanath },
+ booktitle={the 24th ACM SIGKDD International Conference},
+ year={2018},
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_sar.en.md b/docs/algorithm/text_recognition/algorithm_rec_sar.en.md
new file mode 100644
index 0000000000..6339d2cb7a
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_sar.en.md
@@ -0,0 +1,96 @@
+---
+comments: true
+---
+
+# SAR
+
+## 1. Introduction
+
+Paper:
+> [Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition](https://arxiv.org/abs/1811.00751)
+> Hui Li, Peng Wang, Chunhua Shen, Guyu Zhang
+> AAAI, 2019
+
+Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows:
+
+|Model|Backbone|config|Acc|Download link|
+| --- | --- | --- | --- | --- |
+|SAR|ResNet31|[rec_r31_sar.yml](../../configs/rec/rec_r31_sar.yml)|87.20%|[train model](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_r31_sar_train.tar)|
+
+Note:In addition to using the two text recognition datasets MJSynth and SynthText, [SynthAdd](https://pan.baidu.com/share/init?surl=uV0LtoNmcxbO-0YA7Ch4dg) data (extraction code: 627x), and some real data are used in training, the specific data details can refer to the paper.
+
+## 2. Environment
+
+Please refer to ["Environment Preparation"](../../ppocr/environment.en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](../../ppocr/blog/clone.en.md)to clone the project code.
+
+## 3. Model Training / Evaluation / Prediction
+
+Please refer to [Text Recognition Tutorial](../../ppocr/model_train/recognition.en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**.
+
+### Training
+
+Specifically, after the data preparation is completed, the training can be started. The training command is as follows:
+
+```bash linenums="1"
+# Single GPU training (long training period, not recommended)
+python3 tools/train.py -c configs/rec/rec_r31_sar.yml
+
+# Multi GPU training, specify the gpu number through the --gpus parameter
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r31_sar.yml
+```
+
+### Evaluation
+
+```bash linenums="1"
+# GPU evaluation
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r31_sar.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### Prediction
+
+```bash linenums="1"
+# The configuration file used for prediction must match the training
+python3 tools/infer_rec.py -c configs/rec/rec_r31_sar.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. Inference and Deployment
+
+### 4.1 Python Inference
+
+First, the model saved during the SAR text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_r31_sar_train.tar) ), you can use the following command to convert:
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_r31_sar.yml -o Global.pretrained_model=./rec_r31_sar_train/best_accuracy Global.save_inference_dir=./inference/rec_sar
+```
+
+For SAR text recognition model inference, the following commands can be executed:
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_sar/" --rec_image_shape="3, 48, 48, 160" --rec_algorithm="SAR" --rec_char_dict_path="ppocr/utils/dict90.txt" --max_text_length=30 --use_space_char=False
+```
+
+### 4.2 C++ Inference
+
+Not supported
+
+### 4.3 Serving
+
+Not supported
+
+### 4.4 More
+
+Not supported
+
+## 5. FAQ
+
+## Citation
+
+```bibtex
+@article{Li2019ShowAA,
+ title={Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition},
+ author={Hui Li and Peng Wang and Chunhua Shen and Guyu Zhang},
+ journal={ArXiv},
+ year={2019},
+ volume={abs/1811.00751}
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_sar.md b/docs/algorithm/text_recognition/algorithm_rec_sar.md
new file mode 100644
index 0000000000..f8a6dcb375
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_sar.md
@@ -0,0 +1,96 @@
+---
+comments: true
+---
+
+# SAR
+
+## 1. 算法简介
+
+论文信息：
+> [Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition](https://arxiv.org/abs/1811.00751)
+> Hui Li, Peng Wang, Chunhua Shen, Guyu Zhang
+> AAAI, 2019
+
+使用MJSynth和SynthText两个文字识别数据集训练，在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估，算法复现效果如下：
+
+|模型|骨干网络|配置文件|Acc|下载链接|
+| --- | --- | --- | --- | --- |
+|SAR|ResNet31|[rec_r31_sar.yml](../../configs/rec/rec_r31_sar.yml)|87.20%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_r31_sar_train.tar)|
+
+注：除了使用MJSynth和SynthText两个文字识别数据集外，还加入了[SynthAdd](https://pan.baidu.com/share/init?surl=uV0LtoNmcxbO-0YA7Ch4dg)数据（提取码：627x），和部分真实数据，具体数据细节可以参考论文。
+
+## 2. 环境配置
+
+请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。
+
+## 3. 模型训练、评估、预测
+
+请参考[文本识别教程](../../ppocr/model_train/recognition.md)。PaddleOCR对代码进行了模块化，训练不同的识别模型只需要**更换配置文件**即可。
+
+### 训练
+
+具体地，在完成数据准备后，便可以启动训练，训练命令如下：
+
+```bash linenums="1"
+#单卡训练（训练周期长，不建议）
+python3 tools/train.py -c configs/rec/rec_r31_sar.yml
+
+# 多卡训练，通过--gpus参数指定卡号
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r31_sar.yml
+```
+
+### 评估
+
+```bash linenums="1"
+# GPU 评估， Global.pretrained_model 为待测权重
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r31_sar.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### 预测
+
+```bash linenums="1"
+# 预测使用的配置文件必须与训练一致
+python3 tools/infer_rec.py -c configs/rec/rec_r31_sar.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. 推理部署
+
+### 4.1 Python推理
+
+首先将SAR文本识别训练过程中保存的模型，转换成inference model。（ [模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_r31_sar_train.tar) )，可以使用如下命令进行转换：
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_r31_sar.yml -o Global.pretrained_model=./rec_r31_sar_train/best_accuracy Global.save_inference_dir=./inference/rec_sar
+```
+
+SAR文本识别模型推理，可以执行如下命令：
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_sar/" --rec_image_shape="3, 48, 48, 160" --rec_algorithm="SAR" --rec_char_dict_path="ppocr/utils/dict90.txt" --max_text_length=30 --use_space_char=False
+```
+
+### 4.2 C++推理
+
+由于C++预处理后处理还未支持SAR，所以暂未支持
+
+### 4.3 Serving服务化部署
+
+暂不支持
+
+### 4.4 更多推理部署
+
+暂不支持
+
+## 5. FAQ
+
+## 引用
+
+```bibtex
+@article{Li2019ShowAA,
+ title={Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition},
+ author={Hui Li and Peng Wang and Chunhua Shen and Guyu Zhang},
+ journal={ArXiv},
+ year={2019},
+ volume={abs/1811.00751}
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_satrn.en.md b/docs/algorithm/text_recognition/algorithm_rec_satrn.en.md
new file mode 100644
index 0000000000..885a3c4a53
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_satrn.en.md
@@ -0,0 +1,93 @@
+---
+comments: true
+---
+
+# SATRN
+
+## 1. Introduction
+
+论文信息：
+> [On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention](https://arxiv.org/abs/1910.04396)
+> Junyeop Lee, Sungrae Park, Jeonghun Baek, Seong Joon Oh, Seonghyeon Kim, Hwalsuk Lee
+> CVPR, 2020
+Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows:
+
+|Model|Backbone|config|Acc|Download link|
+| --- | --- | --- | --- | --- |
+|SATRN|ShallowCNN|88.05%|[configs/rec/rec_satrn.yml](../../configs/rec/rec_satrn.yml)|[训练模型](https://pan.baidu.com/s/10J-Bsd881bimKaclKszlaQ?pwd=lk8a)|
+
+## 2. Environment
+
+Please refer to ["Environment Preparation"](../../ppocr/environment.en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](../../ppocr/blog/clone.en.md)to clone the project code.
+
+## 3. Model Training / Evaluation / Prediction
+
+Please refer to [Text Recognition Tutorial](../../ppocr/model_train/recognition.en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**.
+
+### Training
+
+Specifically, after the data preparation is completed, the training can be started. The training command is as follows:
+
+```bash linenums="1"
+# Single GPU training (long training period, not recommended)
+python3 tools/train.py -c configs/rec/rec_satrn.yml
+# Multi GPU training, specify the gpu number through the --gpus parameter
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_satrn.yml
+```
+
+### Evaluation
+
+```bash linenums="1"
+# GPU evaluation
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_satrn.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### Prediction
+
+```bash linenums="1"
+# The configuration file used for prediction must match the training
+python3 tools/infer_rec.py -c configs/rec/rec_satrn.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. Inference and Deployment
+
+### 4.1 Python Inference
+
+First, the model saved during the SATRN text recognition training process is converted into an inference model. ( [Model download link](https://pan.baidu.com/s/10J-Bsd881bimKaclKszlaQ?pwd=lk8a) ), you can use the following command to convert:
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_satrn.yml -o Global.pretrained_model=./rec_satrn_train/best_accuracy Global.save_inference_dir=./inference/rec_satrn
+```
+
+For SATRN text recognition model inference, the following commands can be executed:
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_satrn/" --rec_image_shape="3, 48, 48, 160" --rec_algorithm="SATRN" --rec_char_dict_path="ppocr/utils/dict90.txt" --max_text_length=30 --use_space_char=False
+```
+
+### 4.2 C++ Inference
+
+Not supported
+
+### 4.3 Serving
+
+Not supported
+
+### 4.4 More
+
+Not supported
+
+## 5. FAQ
+
+## Citation
+
+```bibtex
+@article{lee2019recognizing,
+ title={On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention},
+ author={Junyeop Lee and Sungrae Park and Jeonghun Baek and Seong Joon Oh and Seonghyeon Kim and Hwalsuk Lee},
+ year={2019},
+ eprint={1910.04396},
+ archivePrefix={arXiv},
+ primaryClass={cs.CV}
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_satrn.md b/docs/algorithm/text_recognition/algorithm_rec_satrn.md
new file mode 100644
index 0000000000..52415720d5
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_satrn.md
@@ -0,0 +1,93 @@
+---
+comments: true
+---
+
+# SATRN
+
+## 1. 算法简介
+
+论文信息：
+> [On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention](https://arxiv.org/abs/1910.04396)
+> Junyeop Lee, Sungrae Park, Jeonghun Baek, Seong Joon Oh, Seonghyeon Kim, Hwalsuk Lee
+> CVPR, 2020
+参考[DTRB](https://arxiv.org/abs/1904.01906) 文字识别训练和评估流程，使用MJSynth和SynthText两个文字识别数据集训练，在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估，算法效果如下：
+
+|模型|骨干网络|Avg Accuracy|配置文件|下载链接|
+|---|---|---|---|---|
+|SATRN|ShallowCNN|88.05%|[configs/rec/rec_satrn.yml](../../configs/rec/rec_satrn.yml)|[训练模型](https://pan.baidu.com/s/10J-Bsd881bimKaclKszlaQ?pwd=lk8a)|
+
+## 2. 环境配置
+
+请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。
+
+## 3. 模型训练、评估、预测
+
+请参考[文本识别训练教程](../../ppocr/model_train/recognition.md)。PaddleOCR对代码进行了模块化，训练不同的识别模型只需要**更换配置文件**即可。
+
+### 训练
+
+在完成数据准备后，便可以启动训练，训练命令如下：
+
+```bash linenums="1"
+# 单卡训练（训练周期长，不建议）
+python3 tools/train.py -c configs/rec/rec_satrn.yml
+# 多卡训练，通过--gpus参数指定卡号
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c rec_satrn.yml
+```
+
+### 评估
+
+```bash linenums="1"
+# GPU 评估， Global.pretrained_model 为待测权重
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_satrn.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### 预测
+
+```bash linenums="1"
+# 预测使用的配置文件必须与训练一致
+python3 tools/infer_rec.py -c configs/rec/rec_satrn.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. 推理部署
+
+### 4.1 Python推理
+
+首先将SATRN文本识别训练过程中保存的模型，转换成inference model。（ [模型下载地址](https://pan.baidu.com/s/10J-Bsd881bimKaclKszlaQ?pwd=lk8a) )，可以使用如下命令进行转换：
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_satrn.yml -o Global.pretrained_model=./rec_satrn/best_accuracy Global.save_inference_dir=./inference/rec_satrn
+```
+
+SATRN文本识别模型推理，可以执行如下命令：
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_satrn/" --rec_image_shape="3, 48, 48, 160" --rec_algorithm="SATRN" --rec_char_dict_path="ppocr/utils/dict90.txt" --max_text_length=30 --use_space_char=False
+```
+
+### 4.2 C++推理
+
+由于C++预处理后处理还未支持SATRN，所以暂未支持
+
+### 4.3 Serving服务化部署
+
+暂不支持
+
+### 4.4 更多推理部署
+
+暂不支持
+
+## 5. FAQ
+
+## 引用
+
+```bibtex
+@article{lee2019recognizing,
+ title={On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention},
+ author={Junyeop Lee and Sungrae Park and Jeonghun Baek and Seong Joon Oh and Seonghyeon Kim and Hwalsuk Lee},
+ year={2019},
+ eprint={1910.04396},
+ archivePrefix={arXiv},
+ primaryClass={cs.CV}
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_seed.en.md b/docs/algorithm/text_recognition/algorithm_rec_seed.en.md
new file mode 100644
index 0000000000..feceda0f66
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_seed.en.md
@@ -0,0 +1,92 @@
+---
+comments: true
+---
+
+# SEED
+
+## 1. Introduction
+
+Paper:
+> [SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition](https://arxiv.org/pdf/2005.10977.pdf)
+
+> Qiao, Zhi and Zhou, Yu and Yang, Dongbao and Zhou, Yucan and Wang, Weiping
+
+> CVPR, 2020
+
+Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows:
+
+|Model|Backbone|ACC|config|Download link|
+| --- | --- | --- | --- | --- |
+|SEED|Aster_Resnet| 85.20% | [configs/rec/rec_resnet_stn_bilstm_att.yml](../../configs/rec/rec_resnet_stn_bilstm_att.yml) | [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_resnet_stn_bilstm_att.tar) |
+
+## 2. Environment
+
+Please refer to ["Environment Preparation"](../../ppocr/environment.en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](../../ppocr/blog/clone.en.md)to clone the project code.
+
+## 3. Model Training / Evaluation / Prediction
+
+Please refer to [Text Recognition Tutorial](../../ppocr/model_train/recognition.en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**.
+
+### Training
+
+The SEED model needs to additionally load the [language model](https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.en.300.bin.gz) trained by FastText, and install the fasttext dependencies:
+
+```bash linenums="1"
+python3 -m pip install fasttext==0.9.1
+```
+
+Specifically, after the data preparation is completed, the training can be started. The training command is as follows:
+
+```bash linenums="1"
+# Single GPU training (long training period, not recommended)
+python3 tools/train.py -c configs/rec/rec_resnet_stn_bilstm_att.yml
+
+# Multi GPU training, specify the gpu number through the --gpus parameter
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c rec_resnet_stn_bilstm_att.yml
+```
+
+### Evaluation
+
+```bash linenums="1"
+# GPU evaluation
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_resnet_stn_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### Prediction
+
+```bash linenums="1"
+# The configuration file used for prediction must match the training
+python3 tools/infer_rec.py -c configs/rec/rec_resnet_stn_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. Inference and Deployment
+
+### 4.1 Python Inference
+
+Not support
+
+### 4.2 C++ Inference
+
+Not support
+
+### 4.3 Serving
+
+Not support
+
+### 4.4 More
+
+Not support
+
+## 5. FAQ
+
+## Citation
+
+```bibtex
+@inproceedings{qiao2020seed,
+ title={Seed: Semantics enhanced encoder-decoder framework for scene text recognition},
+ author={Qiao, Zhi and Zhou, Yu and Yang, Dongbao and Zhou, Yucan and Wang, Weiping},
+ booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
+ pages={13528--13537},
+ year={2020}
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_seed.md b/docs/algorithm/text_recognition/algorithm_rec_seed.md
new file mode 100644
index 0000000000..8428f0ad5a
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_seed.md
@@ -0,0 +1,91 @@
+---
+comments: true
+---
+
+# SEED
+
+## 1. 算法简介
+
+论文信息：
+> [SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition](https://arxiv.org/pdf/2005.10977.pdf)
+> Qiao, Zhi and Zhou, Yu and Yang, Dongbao and Zhou, Yucan and Wang, Weiping
+> CVPR, 2020
+
+参考[DTRB](https://arxiv.org/abs/1904.01906) 文字识别训练和评估流程，使用MJSynth和SynthText两个文字识别数据集训练，在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估，算法效果如下：
+
+|模型|骨干网络|Avg Accuracy|配置文件|下载链接|
+|---|---|---|---|---|
+|SEED|Aster_Resnet| 85.20% | [configs/rec/rec_resnet_stn_bilstm_att.yml](../../configs/rec/rec_resnet_stn_bilstm_att.yml) | [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.1/rec/rec_resnet_stn_bilstm_att.tar) |
+
+## 2. 环境配置
+
+请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。
+
+## 3. 模型训练、评估、预测
+
+请参考[文本识别训练教程](../../ppocr/model_train/recognition.md)。PaddleOCR对代码进行了模块化，训练不同的识别模型只需要**更换配置文件**即可。
+
+### 训练
+
+SEED模型需要额外加载FastText训练好的[语言模型](https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.en.300.bin.gz) ,并且安装 fasttext 依赖：
+
+```bash linenums="1"
+python3 -m pip install fasttext==0.9.1
+```
+
+然后，在完成数据准备后，便可以启动训练，训练命令如下：
+
+```bash linenums="1"
+# 单卡训练（训练周期长，不建议）
+python3 tools/train.py -c configs/rec/rec_resnet_stn_bilstm_att.yml
+
+# 多卡训练，通过--gpus参数指定卡号
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c rec_resnet_stn_bilstm_att.yml
+
+```
+
+### 评估
+
+```bash linenums="1"
+# GPU 评估， Global.pretrained_model 为待测权重
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_resnet_stn_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### 预测
+
+```bash linenums="1"
+# 预测使用的配置文件必须与训练一致
+python3 tools/infer_rec.py -c configs/rec/rec_resnet_stn_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. 推理部署
+
+### 4.1 Python推理
+
+coming soon
+
+### 4.2 C++推理
+
+coming soon
+
+### 4.3 Serving服务化部署
+
+coming soon
+
+### 4.4 更多推理部署
+
+coming soon
+
+## 5. FAQ
+
+## 引用
+
+```bibtex
+@inproceedings{qiao2020seed,
+ title={Seed: Semantics enhanced encoder-decoder framework for scene text recognition},
+ author={Qiao, Zhi and Zhou, Yu and Yang, Dongbao and Zhou, Yucan and Wang, Weiping},
+ booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
+ pages={13528--13537},
+ year={2020}
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_spin.en.md b/docs/algorithm/text_recognition/algorithm_rec_spin.en.md
new file mode 100644
index 0000000000..1d9a0ef037
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_spin.en.md
@@ -0,0 +1,93 @@
+---
+comments: true
+---
+
+# SPIN: Structure-Preserving Inner Offset Network for Scene Text Recognition
+
+## 1. Introduction
+
+Paper:
+> [SPIN: Structure-Preserving Inner Offset Network for Scene Text Recognition](https://arxiv.org/abs/2005.13117)
+> Chengwei Zhang, Yunlu Xu, Zhanzhan Cheng, Shiliang Pu, Yi Niu, Fei Wu, Futai Zou
+> AAAI, 2020
+
+Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets. The algorithm reproduction effect is as follows:
+
+|Model|Backbone|config|Acc|Download link|
+| --- | --- | --- | --- | --- |
+|SPIN|ResNet32|[rec_r32_gaspin_bilstm_att.yml](../../configs/rec/rec_r32_gaspin_bilstm_att.yml)|90.00%|[trained model](https://paddleocr.bj.bcebos.com/contribution/rec_r32_gaspin_bilstm_att.tar) |
+
+## 2. Environment
+
+Please refer to ["Environment Preparation"](../../ppocr/environment.en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](../../ppocr/blog/clone.en.md)to clone the project code.
+
+## 3. Model Training / Evaluation / Prediction
+
+Please refer to [Text Recognition Tutorial](../../ppocr/model_train/recognition.en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**.
+
+### Training
+
+Specifically, after the data preparation is completed, the training can be started. The training command is as follows:
+
+```bash linenums="1"
+# Single GPU training (long training period, not recommended)
+python3 tools/train.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml
+
+# Multi GPU training, specify the gpu number through the --gpus parameter
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml
+```
+
+### Evaluation
+
+```bash linenums="1"
+# GPU evaluation
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### Prediction
+
+```bash linenums="1"
+# The configuration file used for prediction must match the training
+python3 tools/infer_rec.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. Inference and Deployment
+
+### 4.1 Python Inference
+
+First, the model saved during the SPIN text recognition training process is converted into an inference model. you can use the following command to convert:
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.save_inference_dir=./inference/rec_r32_gaspin_bilstm_att
+```
+
+For SPIN text recognition model inference, the following commands can be executed:
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_r32_gaspin_bilstm_att/" --rec_image_shape="3, 32, 100" --rec_algorithm="SPIN" --rec_char_dict_path="/ppocr/utils/dict/spin_dict.txt" --use_space_char=False
+```
+
+### 4.2 C++ Inference
+
+Not supported
+
+### 4.3 Serving
+
+Not supported
+
+### 4.4 More
+
+Not supported
+
+## 5. FAQ
+
+## Citation
+
+```bibtex
+@article{2020SPIN,
+ title={SPIN: Structure-Preserving Inner Offset Network for Scene Text Recognition},
+ author={Chengwei Zhang and Yunlu Xu and Zhanzhan Cheng and Shiliang Pu and Yi Niu and Fei Wu and Futai Zou},
+ journal={AAAI2020},
+ year={2020},
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_spin.md b/docs/algorithm/text_recognition/algorithm_rec_spin.md
new file mode 100644
index 0000000000..bab687b696
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_spin.md
@@ -0,0 +1,94 @@
+---
+comments: true
+---
+
+# SPIN: Structure-Preserving Inner Offset Network for Scene Text Recognition
+
+## 1. 算法简介
+
+论文信息：
+> [SPIN: Structure-Preserving Inner Offset Network for Scene Text Recognition](https://arxiv.org/abs/2005.13117)
+> Chengwei Zhang, Yunlu Xu, Zhanzhan Cheng, Shiliang Pu, Yi Niu, Fei Wu, Futai Zou
+> AAAI, 2020
+
+SPIN收录于AAAI2020。主要用于OCR识别任务。在任意形状文本识别中，矫正网络是一种较为常见的前置处理模块，但诸如RARE\ASTER\ESIR等只考虑了空间变换，并没有考虑色度变换。本文提出了一种结构Structure-Preserving Inner Offset Network (SPIN)，可以在色彩空间上进行变换。该模块是可微分的，可以加入到任意识别器中。
+使用MJSynth和SynthText两个合成文字识别数据集训练，在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估，算法复现效果如下：
+
+|模型|骨干网络|配置文件|Acc|下载链接|
+| --- | --- | --- | --- | --- |
+|SPIN|ResNet32|[rec_r32_gaspin_bilstm_att.yml](../../configs/rec/rec_r32_gaspin_bilstm_att.yml)|90.00%|[训练模型](https://paddleocr.bj.bcebos.com/contribution/rec_r32_gaspin_bilstm_att.tar)|
+
+## 2. 环境配置
+
+请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。
+
+## 3. 模型训练、评估、预测
+
+请参考[文本识别教程](../../ppocr/model_train/recognition.md)。PaddleOCR对代码进行了模块化，训练不同的识别模型只需要**更换配置文件**即可。
+
+### 训练
+
+具体地，在完成数据准备后，便可以启动训练，训练命令如下：
+
+```bash linenums="1"
+#单卡训练（训练周期长，不建议）
+python3 tools/train.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml
+
+# 多卡训练，通过--gpus参数指定卡号
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml
+```
+
+### 评估
+
+```bash linenums="1"
+# GPU 评估， Global.pretrained_model 为待测权重
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### 预测
+
+```bash linenums="1"
+# 预测使用的配置文件必须与训练一致
+python3 tools/infer_rec.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. 推理部署
+
+### 4.1 Python推理
+
+首先将SPIN文本识别训练过程中保存的模型，转换成inference model。可以使用如下命令进行转换：
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_r32_gaspin_bilstm_att.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.save_inference_dir=./inference/rec_r32_gaspin_bilstm_att
+```
+
+SPIN文本识别模型推理，可以执行如下命令：
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_r32_gaspin_bilstm_att/" --rec_image_shape="3, 32, 100" --rec_algorithm="SPIN" --rec_char_dict_path="/ppocr/utils/dict/spin_dict.txt" --use_space_char=Falsee
+```
+
+### 4.2 C++推理
+
+由于C++预处理后处理还未支持SPIN，所以暂未支持
+
+### 4.3 Serving服务化部署
+
+暂不支持
+
+### 4.4 更多推理部署
+
+暂不支持
+
+## 5. FAQ
+
+## 引用
+
+```bibtex
+@article{2020SPIN,
+ title={SPIN: Structure-Preserving Inner Offset Network for Scene Text Recognition},
+ author={Chengwei Zhang and Yunlu Xu and Zhanzhan Cheng and Shiliang Pu and Yi Niu and Fei Wu and Futai Zou},
+ journal={AAAI2020},
+ year={2020},
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_srn.en.md b/docs/algorithm/text_recognition/algorithm_rec_srn.en.md
new file mode 100644
index 0000000000..8800103916
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_srn.en.md
@@ -0,0 +1,94 @@
+---
+comments: true
+---
+
+# SRN
+
+## 1. Introduction
+
+Paper:
+> [Towards Accurate Scene Text Recognition with Semantic Reasoning Networks](https://arxiv.org/abs/2003.12294#)
+> Deli Yu, Xuan Li, Chengquan Zhang, Junyu Han, Jingtuo Liu, Errui Ding
+> CVPR,2020
+
+Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows:
+
+|Model|Backbone|config|Acc|Download link|
+| --- | --- | --- | --- | --- |
+|SRN|Resnet50_vd_fpn|[rec_r50_fpn_srn.yml](../../configs/rec/rec_r50_fpn_srn.yml)|86.31%|[train model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r50_vd_srn_train.tar)|
+
+## 2. Environment
+
+Please refer to ["Environment Preparation"](../../ppocr/environment.en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](../../ppocr/blog/clone.en.md)to clone the project code.
+
+## 3. Model Training / Evaluation / Prediction
+
+Please refer to [Text Recognition Tutorial](../../ppocr/model_train/recognition.en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**.
+
+### Training
+
+Specifically, after the data preparation is completed, the training can be started. The training command is as follows:
+
+```bash linenums="1"
+# Single GPU training (long training period, not recommended)
+python3 tools/train.py -c configs/rec/rec_r50_fpn_srn.yml
+
+# Multi GPU training, specify the gpu number through the --gpus parameter
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r50_fpn_srn.yml
+```
+
+### Evaluation
+
+```bash linenums="1"
+# GPU evaluation
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r50_fpn_srn.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### Prediction
+
+```bash linenums="1"
+# The configuration file used for prediction must match the training
+python3 tools/infer_rec.py -c configs/rec/rec_r50_fpn_srn.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. Inference and Deployment
+
+### 4.1 Python Inference
+
+First, the model saved during the SRN text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r50_vd_srn_train.tar) ), you can use the following command to convert:
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_r50_fpn_srn.yml -o Global.pretrained_model=./rec_r50_vd_srn_train/best_accuracy Global.save_inference_dir=./inference/rec_srn
+```
+
+For SRN text recognition model inference, the following commands can be executed:
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_srn/" --rec_image_shape="1,64,256" --rec_char_type="ch" --rec_algorithm="SRN" --rec_char_dict_path="ppocr/utils/ic15_dict.txt" --use_space_char=False
+```
+
+### 4.2 C++ Inference
+
+Not supported
+
+### 4.3 Serving
+
+Not supported
+
+### 4.4 More
+
+Not supported
+
+## 5. FAQ
+
+## Citation
+
+```bibtex
+@article{Yu2020TowardsAS,
+ title={Towards Accurate Scene Text Recognition With Semantic Reasoning Networks},
+ author={Deli Yu and Xuan Li and Chengquan Zhang and Junyu Han and Jingtuo Liu and Errui Ding},
+ journal={2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
+ year={2020},
+ pages={12110-12119}
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_srn.md b/docs/algorithm/text_recognition/algorithm_rec_srn.md
new file mode 100644
index 0000000000..04823bcb96
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_srn.md
@@ -0,0 +1,94 @@
+---
+comments: true
+---
+
+# SRN
+
+## 1. 算法简介
+
+论文信息：
+> [Towards Accurate Scene Text Recognition with Semantic Reasoning Networks](https://arxiv.org/abs/2003.12294#)
+> Deli Yu, Xuan Li, Chengquan Zhang, Junyu Han, Jingtuo Liu, Errui Ding
+> CVPR,2020
+
+使用MJSynth和SynthText两个文字识别数据集训练，在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估，算法复现效果如下：
+
+|模型|骨干网络|配置文件|Acc|下载链接|
+| --- | --- | --- | --- | --- |
+|SRN|Resnet50_vd_fpn|[rec_r50_fpn_srn.yml](../../configs/rec/rec_r50_fpn_srn.yml)|86.31%|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r50_vd_srn_train.tar)|
+
+## 2. 环境配置
+
+请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。
+
+## 3. 模型训练、评估、预测
+
+请参考[文本识别教程](../../ppocr/model_train/recognition.md)。PaddleOCR对代码进行了模块化，训练不同的识别模型只需要**更换配置文件**即可。
+
+### 训练
+
+具体地，在完成数据准备后，便可以启动训练，训练命令如下：
+
+```bash linenums="1"
+#单卡训练（训练周期长，不建议）
+python3 tools/train.py -c configs/rec/rec_r50_fpn_srn.yml
+
+# 多卡训练，通过--gpus参数指定卡号
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r50_fpn_srn.yml
+```
+
+### 评估
+
+```bash linenums="1"
+# GPU 评估， Global.pretrained_model 为待测权重
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r50_fpn_srn.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### 预测
+
+```bash linenums="1"
+# 预测使用的配置文件必须与训练一致
+python3 tools/infer_rec.py -c configs/rec/rec_r50_fpn_srn.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. 推理部署
+
+### 4.1 Python推理
+
+首先将SRN文本识别训练过程中保存的模型，转换成inference model。（ [模型下载地址](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r50_vd_srn_train.tar) )，可以使用如下命令进行转换：
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_r50_fpn_srn.yml -o Global.pretrained_model=./rec_r50_vd_srn_train/best_accuracy Global.save_inference_dir=./inference/rec_srn
+```
+
+SRN文本识别模型推理，可以执行如下命令：
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words/en/word_1.png" --rec_model_dir="./inference/rec_srn/" --rec_image_shape="1,64,256" --rec_algorithm="SRN" --rec_char_dict_path=./ppocr/utils/ic15_dict.txt --use_space_char=False
+```
+
+### 4.2 C++推理
+
+由于C++预处理后处理还未支持SRN，所以暂未支持
+
+### 4.3 Serving服务化部署
+
+暂不支持
+
+### 4.4 更多推理部署
+
+暂不支持
+
+## 5. FAQ
+
+## 引用
+
+```bibtex
+@article{Yu2020TowardsAS,
+ title={Towards Accurate Scene Text Recognition With Semantic Reasoning Networks},
+ author={Deli Yu and Xuan Li and Chengquan Zhang and Junyu Han and Jingtuo Liu and Errui Ding},
+ journal={2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
+ year={2020},
+ pages={12110-12119}
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_starnet.en.md b/docs/algorithm/text_recognition/algorithm_rec_starnet.en.md
new file mode 100644
index 0000000000..a5552064f5
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_starnet.en.md
@@ -0,0 +1,115 @@
+---
+comments: true
+---
+
+# STAR-Net
+
+## 1. Introduction
+
+Paper information:
+> [STAR-Net: a spatial attention residue network for scene text recognition.](http://www.bmva.org/bmvc/2016/papers/paper043/paper043.pdf)
+> Wei Liu, Chaofeng Chen, Kwan-Yee K. Wong, Zhizhong Su and Junyu Han.
+> BMVC, pages 43.1-43.13, 2016
+
+Refer to [DTRB](https://arxiv.org/abs/1904.01906) text Recognition Training and Evaluation Process . Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows:
+
+|Models|Backbone Networks|Avg Accuracy|Configuration Files|Download Links|
+| --- | --- | --- | --- | --- |
+|StarNet|Resnet34_vd|84.44%|[configs/rec/rec_r34_vd_tps_bilstm_ctc.yml](../../configs/rec/rec_r34_vd_tps_bilstm_ctc.yml)|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_tps_bilstm_ctc_v2.0_train.tar)|
+|StarNet|MobileNetV3|81.42%|[configs/rec/rec_mv3_tps_bilstm_ctc.yml](../../configs/rec/rec_mv3_tps_bilstm_ctc.yml)|[trained model](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_tps_bilstm_ctc_v2.0_train.tar)|
+
+## 2. Environment
+
+Please refer to [Operating Environment Preparation](../../ppocr/environment.en.md) to configure the PaddleOCR operating environment, and refer to [Project Clone](../../ppocr/blog/clone.en.md)to clone the project code.
+
+## 3. Model Training / Evaluation / Prediction
+
+Please refer to [Text Recognition Training Tutorial](../../ppocr/model_train/recognition.en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. Take the backbone network based on Resnet34_vd as an example:
+
+### 3.1 Training
+
+After the data preparation is complete, the training can be started. The training command is as follows:
+
+````bash linenums="1"
+# Single card training (long training period, not recommended)
+python3 tools/train.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml # Multi-card training, specify the card number through the --gpus parameter
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c rec_r34_vd_tps_bilstm_ctc.yml
+````
+
+### 3.2 Evaluation
+
+````bash linenums="1"
+# GPU evaluation, Global.pretrained_model is the model to be evaluated
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+````
+
+### 3.3 Prediction
+
+````bash linenums="1"
+# The configuration file used for prediction must match the training
+python3 tools/infer_rec.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+````
+
+## 4. Inference
+
+### 4.1 Python Inference
+
+First, convert the model saved during the STAR-Net text recognition training process into an inference model. Take the model trained on the MJSynth and SynthText text recognition datasets based on the Resnet34_vd backbone network as an example [Model download address]( https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_bilstm_ctc_v2.0_train.tar) , which can be converted using the following command:
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml -o Global.pretrained_model=./rec_r34_vd_tps_bilstm_ctc_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_starnet
+```
+
+STAR-Net text recognition model inference, you can execute the following commands:
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./inference/rec_starnet/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt"
+```
+
+
+
+The inference results are as follows:
+
+```bash linenums="1"
+Predicts of ./doc/imgs_words_en/word_336.png:('super', 0.9999073)
+```
+
+**Attention** Since the above model refers to the [DTRB](https://arxiv.org/abs/1904.01906) text recognition training and evaluation process, it is different from the ultra-lightweight Chinese recognition model training in two aspects:
+
+- The image resolutions used during training are different. The image resolutions used for training the above models are [3, 32, 100], while for Chinese model training, in order to ensure the recognition effect of long texts, the image resolutions used during training are [ 3, 32, 320]. The default shape parameter of the predictive inference program is the image resolution used for training Chinese, i.e. [3, 32, 320]. Therefore, when inferring the above English model here, it is necessary to set the shape of the recognized image through the parameter rec_image_shape.
+
+- Character list, the experiment in the DTRB paper is only for 26 lowercase English letters and 10 numbers, a total of 36 characters. All uppercase and lowercase characters are converted to lowercase characters, and characters not listed above are ignored and considered spaces. Therefore, there is no input character dictionary here, but a dictionary is generated by the following command. Therefore, the parameter rec_char_dict_path needs to be set during inference, which is specified as an English dictionary "./ppocr/utils/ic15_dict.txt".
+
+```python linenums="1"
+self.character_str = "0123456789abcdefghijklmnopqrstuvwxyz"
+dict_character = list(self.character_str)
+```
+
+### 4.2 C++ Inference
+
+After preparing the inference model, refer to the [cpp infer](../../ppocr/infer_deploy/cpp_infer.en.md) tutorial to operate.
+
+### 4.3 Serving
+
+After preparing the inference model, refer to the [pdserving](../../ppocr/infer_deploy/paddle_server.en.md) tutorial for Serving deployment, including two modes: Python Serving and C++ Serving.
+
+### 4.4 More
+
+The STAR-Net model also supports the following inference deployment methods:
+
+- Paddle2ONNX Inference: After preparing the inference model, refer to the [paddle2onnx](../../ppocr/infer_deploy/paddle2onnx.en.md) tutorial.
+
+## 5. FAQ
+
+## Citation
+
+```bibtex
+@inproceedings{liu2016star,
+ title={STAR-Net: a spatial attention residue network for scene text recognition.},
+ author={Liu, Wei and Chen, Chaofeng and Wong, Kwan-Yee K and Su, Zhizhong and Han, Junyu},
+ booktitle={BMVC},
+ volume={2},
+ pages={7},
+ year={2016}
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_starnet.md b/docs/algorithm/text_recognition/algorithm_rec_starnet.md
new file mode 100644
index 0000000000..8f95bdcd6b
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_starnet.md
@@ -0,0 +1,118 @@
+---
+typora-copy-images-to: images
+comments: true
+---
+
+# STAR-Net
+
+## 1. 算法简介
+
+论文信息：
+> [STAR-Net: a spatial attention residue network for scene text recognition.](http://www.bmva.org/bmvc/2016/papers/paper043/paper043.pdf)
+> Wei Liu, Chaofeng Chen, Kwan-Yee K. Wong, Zhizhong Su and Junyu Han.
+> BMVC, pages 43.1-43.13, 2016
+
+参考[DTRB](https://arxiv.org/abs/1904.01906) 文字识别训练和评估流程，使用MJSynth和SynthText两个文字识别数据集训练，在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估，算法效果如下：
+
+|模型|骨干网络|Avg Accuracy|配置文件|下载链接|
+|---|---|---|---|---|
+|StarNet|Resnet34_vd|84.44%|[configs/rec/rec_r34_vd_tps_bilstm_ctc.yml](../../configs/rec/rec_r34_vd_tps_bilstm_ctc.yml)|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_tps_bilstm_ctc_v2.0_train.tar)|
+|StarNet|MobileNetV3|81.42%|[configs/rec/rec_mv3_tps_bilstm_ctc.yml](../../configs/rec/rec_mv3_tps_bilstm_ctc.yml)|[训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_mv3_tps_bilstm_ctc_v2.0_train.tar)|
+
+## 2. 环境配置
+
+请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。
+
+## 3. 模型训练、评估、预测
+
+请参考[文本识别训练教程](../../ppocr/model_train/recognition.md)。PaddleOCR对代码进行了模块化，训练不同的识别模型只需要**更换配置文件**即可。
+
+### 训练
+
+在完成数据准备后，便可以启动训练，训练命令如下：
+
+```bash linenums="1"
+# 单卡训练（训练周期长，不建议）
+python3 tools/train.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml
+
+# 多卡训练，通过--gpus参数指定卡号
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c rec_r34_vd_tps_bilstm_ctc.yml
+```
+
+### 评估
+
+```bash linenums="1"
+# GPU 评估， Global.pretrained_model 为待测权重
+python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy
+```
+
+### 预测
+
+```bash linenums="1"
+# 预测使用的配置文件必须与训练一致
+python3 tools/infer_rec.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml -o Global.pretrained_model={path/to/weights}/best_accuracy Global.infer_img=doc/imgs_words/en/word_1.png
+```
+
+## 4. 推理部署
+
+### 4.1 Python推理
+
+首先将 STAR-Net 文本识别训练过程中保存的模型，转换成inference model。以基于Resnet34_vd骨干网络，使用MJSynth和SynthText两个英文文本识别合成数据集训练的[模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/en/rec_r34_vd_none_bilstm_ctc_v2.0_train.tar) 为例，可以使用如下命令进行转换：
+
+```bash linenums="1"
+python3 tools/export_model.py -c configs/rec/rec_r34_vd_tps_bilstm_ctc.yml -o Global.pretrained_model=./rec_r34_vd_tps_bilstm_ctc_v2.0_train/best_accuracy Global.save_inference_dir=./inference/rec_starnet
+```
+
+STAR-Net 文本识别模型推理，可以执行如下命令：
+
+```bash linenums="1"
+python3 tools/infer/predict_rec.py --image_dir="./doc/imgs_words_en/word_336.png" --rec_model_dir="./inference/rec_starnet/" --rec_image_shape="3, 32, 100" --rec_char_dict_path="./ppocr/utils/ic15_dict.txt"
+```
+
+
+
+执行命令后，上面图像的识别结果如下：
+
+```bash linenums="1"
+Predicts of ./doc/imgs_words_en/word_336.png:('super', 0.9999073)
+```
+
+**注意**：由于上述模型是参考[DTRB](https://arxiv.org/abs/1904.01906)文本识别训练和评估流程，与超轻量级中文识别模型训练有两方面不同：
+
+- 训练时采用的图像分辨率不同，训练上述模型采用的图像分辨率是[3，32，100]，而中文模型训练时，为了保证长文本的识别效果，训练时采用的图像分辨率是[3, 32, 320]。预测推理程序默认的形状参数是训练中文采用的图像分辨率，即[3, 32, 320]。因此，这里推理上述英文模型时，需要通过参数rec_image_shape设置识别图像的形状。
+
+- 字符列表，DTRB论文中实验只是针对26个小写英文本母和10个数字进行实验，总共36个字符。所有大小字符都转成了小写字符，不在上面列表的字符都忽略，认为是空格。因此这里没有输入字符字典，而是通过如下命令生成字典.因此在推理时需要设置参数rec_char_dict_path，指定为英文字典"./ppocr/utils/ic15_dict.txt"。
+
+```python linenums="1"
+self.character_str = "0123456789abcdefghijklmnopqrstuvwxyz"
+dict_character = list(self.character_str)
+```
+
+### 4.2 C++推理
+
+准备好推理模型后，参考[cpp infer](../../ppocr/infer_deploy/cpp_infer.md)教程进行操作即可。
+
+### 4.3 Serving服务化部署
+
+准备好推理模型后，参考[pdserving](../../ppocr/infer_deploy/paddle_server.md)教程进行Serving服务化部署，包括Python Serving和C++ Serving两种模式。
+
+### 4.4 更多推理部署
+
+STAR-Net模型还支持以下推理部署方式：
+
+- Paddle2ONNX推理：准备好推理模型后，参考[paddle2onnx](../../ppocr/infer_deploy/paddle2onnx.md)教程操作。
+
+## 5. FAQ
+
+## 引用
+
+```bibtex
+@inproceedings{liu2016star,
+ title={STAR-Net: a spatial attention residue network for scene text recognition.},
+ author={Liu, Wei and Chen, Chaofeng and Wong, Kwan-Yee K and Su, Zhizhong and Han, Junyu},
+ booktitle={BMVC},
+ volume={2},
+ pages={7},
+ year={2016}
+}
+```
diff --git a/docs/algorithm/text_recognition/algorithm_rec_svtr.en.md b/docs/algorithm/text_recognition/algorithm_rec_svtr.en.md
new file mode 100644
index 0000000000..0080846957
--- /dev/null
+++ b/docs/algorithm/text_recognition/algorithm_rec_svtr.en.md
@@ -0,0 +1,144 @@
+---
+comments: true
+---
+
+# SVTR
+
+## 1. Introduction
+
+Paper:
+> [SVTR: Scene Text Recognition with a Single Visual Model](https://arxiv.org/abs/2205.00159)
+> Yongkun Du and Zhineng Chen and Caiyan Jia Xiaoting Yin and Tianlun Zheng and Chenxia Li and Yuning Du and Yu-Gang Jiang
+> IJCAI, 2022
+
+The accuracy (%) and model files of SVTR on the public dataset of scene text recognition are as follows:
+
+* Chinese dataset from [Chinese Benckmark](https://arxiv.org/abs/2112.15093) , and the Chinese training evaluation strategy of SVTR follows the paper.
+
+| Model |IC13详情
+ + ```python linenums="1" + params = paddle.load('path/' + '.pdparams') # 旧版本参数 + state_dict = model.state_dict() # 新版模型参数 + new_state_dict = {} + + for k1, v1 in state_dict.items(): + + k = k1 + if 'encoder' in k and 'self_attn' in k and 'qkv' in k and 'weight' in k: + + k_para = k[:13] + 'layers.' + k[13:] + q = params[k_para.replace('qkv', 'conv1')].transpose((1, 0, 2, 3)) + k = params[k_para.replace('qkv', 'conv2')].transpose((1, 0, 2, 3)) + v = params[k_para.replace('qkv', 'conv3')].transpose((1, 0, 2, 3)) + + new_state_dict[k1] = np.concatenate([q[:, :, 0, 0], k[:, :, 0, 0], v[:, :, 0, 0]], -1) + + elif 'encoder' in k and 'self_attn' in k and 'qkv' in k and 'bias' in k: + + k_para = k[:13] + 'layers.' + k[13:] + q = params[k_para.replace('qkv', 'conv1')] + k = params[k_para.replace('qkv', 'conv2')] + v = params[k_para.replace('qkv', 'conv3')] + + new_state_dict[k1] = np.concatenate([q, k, v], -1) + + elif 'encoder' in k and 'self_attn' in k and 'out_proj' in k: + + k_para = k[:13] + 'layers.' + k[13:] + new_state_dict[k1] = params[k_para] + + elif 'encoder' in k and 'norm3' in k: + k_para = k[:13] + 'layers.' + k[13:] + new_state_dict[k1] = params[k_para.replace('norm3', 'norm2')] + + elif 'encoder' in k and 'norm1' in k: + k_para = k[:13] + 'layers.' + k[13:] + new_state_dict[k1] = params[k_para] + + + elif 'decoder' in k and 'self_attn' in k and 'qkv' in k and 'weight' in k: + k_para = k[:13] + 'layers.' + k[13:] + q = params[k_para.replace('qkv', 'conv1')].transpose((1, 0, 2, 3)) + k = params[k_para.replace('qkv', 'conv2')].transpose((1, 0, 2, 3)) + v = params[k_para.replace('qkv', 'conv3')].transpose((1, 0, 2, 3)) + new_state_dict[k1] = np.concatenate([q[:, :, 0, 0], k[:, :, 0, 0], v[:, :, 0, 0]], -1) + + elif 'decoder' in k and 'self_attn' in k and 'qkv' in k and 'bias' in k: + k_para = k[:13] + 'layers.' + k[13:] + q = params[k_para.replace('qkv', 'conv1')] + k = params[k_para.replace('qkv', 'conv2')] + v = params[k_para.replace('qkv', 'conv3')] + new_state_dict[k1] = np.concatenate([q, k, v], -1) + + elif 'decoder' in k and 'self_attn' in k and 'out_proj' in k: + + k_para = k[:13] + 'layers.' + k[13:] + new_state_dict[k1] = params[k_para] + + elif 'decoder' in k and 'cross_attn' in k and 'q' in k and 'weight' in k: + k_para = k[:13] + 'layers.' + k[13:] + k_para = k_para.replace('cross_attn', 'multihead_attn') + q = params[k_para.replace('q', 'conv1')].transpose((1, 0, 2, 3)) + new_state_dict[k1] = q[:, :, 0, 0] + + elif 'decoder' in k and 'cross_attn' in k and 'q' in k and 'bias' in k: + k_para = k[:13] + 'layers.' + k[13:] + k_para = k_para.replace('cross_attn', 'multihead_attn') + q = params[k_para.replace('q', 'conv1')] + new_state_dict[k1] = q + + elif 'decoder' in k and 'cross_attn' in k and 'kv' in k and 'weight' in k: + k_para = k[:13] + 'layers.' + k[13:] + k_para = k_para.replace('cross_attn', 'multihead_attn') + k = params[k_para.replace('kv', 'conv2')].transpose((1, 0, 2, 3)) + v = params[k_para.replace('kv', 'conv3')].transpose((1, 0, 2, 3)) + new_state_dict[k1] = np.concatenate([k[:, :, 0, 0], v[:, :, 0, 0]], -1) + + elif 'decoder' in k and 'cross_attn' in k and 'kv' in k and 'bias' in k: + k_para = k[:13] + 'layers.' + k[13:] + k_para = k_para.replace('cross_attn', 'multihead_attn') + k = params[k_para.replace('kv', 'conv2')] + v = params[k_para.replace('kv', 'conv3')] + new_state_dict[k1] = np.concatenate([k, v], -1) + + elif 'decoder' in k and 'cross_attn' in k and 'out_proj' in k: + + k_para = k[:13] + 'layers.' + k[13:] + k_para = k_para.replace('cross_attn', 'multihead_attn') + new_state_dict[k1] = params[k_para] + elif 'decoder' in k and 'norm' in k: + k_para = k[:13] + 'layers.' + k[13:] + new_state_dict[k1] = params[k_para] + elif 'mlp' in k and 'weight' in k: + k_para = k[:13] + 'layers.' + k[13:] + k_para = k_para.replace('fc', 'conv') + k_para = k_para.replace('mlp.', '') + w = params[k_para].transpose((1, 0, 2, 3)) + new_state_dict[k1] = w[:, :, 0, 0] + elif 'mlp' in k and 'bias' in k: + k_para = k[:13] + 'layers.' + k[13:] + k_para = k_para.replace('fc', 'conv') + k_para = k_para.replace('mlp.', '') + w = params[k_para] + new_state_dict[k1] = w + + else: + new_state_dict[k1] = params[k1] + + if list(new_state_dict[k1].shape) != list(v1.shape): + print(k1) + + + for k, v1 in state_dict.items(): + if k not in new_state_dict.keys(): + print(1, k) + elif list(new_state_dict[k].shape) != list(v1.shape): + print(2, k) + + + + model.set_state_dict(new_state_dict) + paddle.save(model.state_dict(), 'nrtrnew_from_old_params.pdparams') + + ``` + +857 | SVT |IIIT5k
3000 |IC15
1811| SVTP |CUTE80 | Avg_6 |IC15
2077 |IC13
1015 |IC03
867|IC03
860|Avg_10 | Chinese
scene_test| Download link | +|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:|:-------:|:-----:|:-----:|:---------------------------------------------:|:-----:|:------:| +| SVTR Tiny | 96.85 | 91.34 | 94.53 | 83.99 | 85.43 | 89.24 | 90.87 | 80.55 | 95.37 | 95.27 | 95.70 | 90.13 | 67.90 | [English](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar) / [Chinese](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_ch_train.tar) | +| SVTR Small | 95.92 | 93.04 | 95.03 | 84.70 | 87.91 | 92.01 | 91.63 | 82.72 | 94.88 | 96.08 | 96.28 | 91.02 | 69.00 | [English](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_small_none_ctc_en_train.tar) / [Chinese](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_small_none_ctc_ch_train.tar) | +| SVTR Base | 97.08 | 91.50 | 96.03 | 85.20 | 89.92 | 91.67 | 92.33 | 83.73 | 95.66 | 95.62 | 95.81 | 91.61 | 71.40 | [English](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_base_none_ctc_en_train.tar) / - | +| SVTR Large | 97.20 | 91.65 | 96.30 | 86.58 | 88.37 | 95.14 | 92.82 | 84.54 | 96.35 | 96.54 | 96.74 | 92.24 | 72.10 | [English](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_large_none_ctc_en_train.tar) / [Chinese](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_large_none_ctc_ch_train.tar) | + +## 2. Environment + +Please refer to ["Environment Preparation"](../../ppocr/environment.en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](../../ppocr/blog/clone.en.md)to clone the project code. + +### Dataset Preparation + +[English dataset download](https://github.com/clovaai/deep-text-recognition-benchmark#download-lmdb-dataset-for-traininig-and-evaluation-from-here) +[Chinese dataset download](https://github.com/fudanvi/benchmarking-chinese-text-recognition#download) + +## 3. Model Training / Evaluation / Prediction + +Please refer to [Text Recognition Tutorial](../../ppocr/model_train/recognition.en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. + +### Training + +Specifically, after the data preparation is completed, the training can be started. The training command is as follows: + +```bash linenums="1" +# Single GPU training (long training period, not recommended) +python3 tools/train.py -c configs/rec/rec_svtrnet.yml + +# Multi GPU training, specify the gpu number through the --gpus parameter +python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_svtrnet.yml +``` + +### Evaluation + +You can download the model files and configuration files provided by `SVTR`: [download link](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar), take `SVTR-T` as an example, using the following command to evaluate: + +```bash linenums="1" +# Download the tar archive containing the model files and configuration files of SVTR-T and extract it +wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar && tar xf rec_svtr_tiny_none_ctc_en_train.tar +# GPU evaluation +python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c ./rec_svtr_tiny_none_ctc_en_train/rec_svtr_tiny_6local_6global_stn_en.yml -o Global.pretrained_model=./rec_svtr_tiny_none_ctc_en_train/best_accuracy +``` + +### Prediction + +```bash linenums="1" +python3 tools/infer_rec.py -c ./rec_svtr_tiny_none_ctc_en_train/rec_svtr_tiny_6local_6global_stn_en.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_svtr_tiny_none_ctc_en_train/best_accuracy +``` + +## 4. Inference and Deployment + +### 4.1 Python Inference + +First, the model saved during the SVTR text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar) ), you can use the following command to convert: + +```bash linenums="1" +python3 tools/export_model.py -c configs/rec/rec_svtrnet.yml -o Global.pretrained_model=./rec_svtr_tiny_none_ctc_en_train/best_accuracy Global.save_inference_dir=./inference/rec_svtr_tiny_stn_en +``` + +**Note:** If you are training the model on your own dataset and have modified the dictionary file, please pay attention to modify the `character_dict_path` in the configuration file to the modified dictionary file. + +After the conversion is successful, there are three files in the directory: + +```text linenums="1" +/inference/rec_svtr_tiny_stn_en/ + ├── inference.pdiparams + ├── inference.pdiparams.info + └── inference.pdmodel +``` + +For SVTR text recognition model inference, the following commands can be executed: + +```bash linenums="1" +python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_svtr_tiny_stn_en/' --rec_algorithm='SVTR' --rec_image_shape='3,64,256' --rec_char_dict_path='./ppocr/utils/ic15_dict.txt' +``` + + + +After executing the command, the prediction result (recognized text and score) of the image above is printed to the screen, an example is as follows: + +```bash linenums="1" +Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9999998807907104) +``` + +### 4.2 C++ Inference + +Not supported + +### 4.3 Serving + +Not supported + +### 4.4 More + +Not supported + +## 5. FAQ + +* 1. Speed situation on CPU and GPU + * Since most of the operators used by `SVTR` are matrix multiplication, in the GPU environment, the speed has an advantage, but in the environment where mkldnn is enabled on the CPU, `SVTR` has no advantage over the optimized convolutional network. +* 2. SVTR model convert to ONNX failed + * Ensure `paddle2onnx` and `onnxruntime` versions are up to date, refer to [SVTR model to onnx step-by-step example](https://github.com/PaddlePaddle/PaddleOCR/issues/7821#issuecomment-) for the convert onnx command. 1271214273). +* 3. SVTR model convert to ONNX is successful but the inference result is incorrect + * The possible reason is that the model parameter `out_char_num` is not set correctly, it should be set to W//4, W//8 or W//12, please refer to [Section 3.3.3 of SVTR, a high-precision Chinese scene text recognition model](https://aistudio.baidu.com/aistudio/) projectdetail/5073182?contributionType=1). +* 4. Optimization of long text recognition + * Refer to [Section 3.3 of SVTR, a high-precision Chinese scene text recognition model](https://aistudio.baidu.com/aistudio/projectdetail/5073182?contributionType=1). +* 5. Notes on the reproduction of the paper results + * Dataset using provided by [ABINet](https://github.com/FangShancheng/ABINet). + * By default, 4 cards of GPUs are used for training, the default Batchsize of a single card is 512, and the total Batchsize is 2048, corresponding to a learning rate of 0.0005. When modifying the Batchsize or changing the number of GPU cards, the learning rate should be modified in equal proportion. +* 6. Exploration Directions for further optimization + * Learning rate adjustment: adjusting to twice the default to keep Batchsize unchanged; or reducing Batchsize to 1/2 the default to keep the learning rate unchanged. + * Data augmentation strategies: optionally `RecConAug` and `RecAug`. + * If STN is not used, `Local` of `mixer` can be replaced by `Conv` and `local_mixer` can all be modified to `[5, 5]`. + * Grid search for optimal `embed_dim`, `depth`, `num_heads` configurations. + * Use the `Post-Normalization strategy`, which is to modify the model configuration `prenorm` to `True`. + +## Citation + +```bibtex +@article{Du2022SVTR, + title = {SVTR: Scene Text Recognition with a Single Visual Model}, + author = {Du, Yongkun and Chen, Zhineng and Jia, Caiyan and Yin, Xiaoting and Zheng, Tianlun and Li, Chenxia and Du, Yuning and Jiang, Yu-Gang}, + booktitle = {IJCAI}, + year = {2022}, + url = {https://arxiv.org/abs/2205.00159} +} +``` diff --git a/docs/algorithm/text_recognition/algorithm_rec_svtr.md b/docs/algorithm/text_recognition/algorithm_rec_svtr.md new file mode 100644 index 0000000000..bb9dafe7cf --- /dev/null +++ b/docs/algorithm/text_recognition/algorithm_rec_svtr.md @@ -0,0 +1,170 @@ +--- +comments: true +--- + +# 场景文本识别算法-SVTR + +## 1. 算法简介 + +论文信息： +> [SVTR: Scene Text Recognition with a Single Visual Model](https://arxiv.org/abs/2205.00159) +> Yongkun Du and Zhineng Chen and Caiyan Jia and Xiaoting Yin and Tianlun Zheng and Chenxia Li and Yuning Du and Yu-Gang Jiang +> IJCAI, 2022 + +场景文本识别旨在将自然图像中的文本转录为数字字符序列，从而传达对场景理解至关重要的高级语义。这项任务由于文本变形、字体、遮挡、杂乱背景等方面的变化具有一定的挑战性。先前的方法为提高识别精度做出了许多工作。然而文本识别器除了准确度外，还因为实际需求需要考虑推理速度等因素。 + +### SVTR算法简介 + +主流的场景文本识别模型通常包含两个模块：用于特征提取的视觉模型和用于文本转录的序列模型。这种架构虽然准确，但复杂且效率较低，限制了在实际场景中的应用。SVTR提出了一种用于场景文本识别的单视觉模型，该模型在patch-wise image tokenization框架内，完全摒弃了序列建模，在精度具有竞争力的前提下，模型参数量更少，速度更快，主要有以下几点贡献： + +1. 首次发现单视觉模型可以达到与视觉语言模型相媲美甚至更高的准确率，并且其具有效率高和适应多语言的优点，在实际应用中很有前景。 +2. SVTR从字符组件的角度出发，逐渐的合并字符组件，自下而上地完成字符的识别。 +3. SVTR引入了局部和全局Mixing，分别用于提取字符组件特征和字符间依赖关系，与多尺度的特征一起，形成多粒度特征描述。 + +SVTR在场景文本识别公开数据集上的精度(%)和模型文件如下： + +* 中文数据集来自于[Chinese Benckmark](https://arxiv.org/abs/2112.15093) ，SVTR的中文训练评估策略遵循该论文。 + +| 模型 |IC13
857 | SVT |IIIT5k
3000 |IC15
1811| SVTP |CUTE80 | Avg_6 |IC15
2077 |IC13
1015 |IC03
867|IC03
860|Avg_10 | Chinese
scene_test| 下载链接 | +|:----------:|:------:|:-----:|:---------:|:------:|:-----:|:-----:|:-----:|:-------:|:-------:|:-----:|:-----:|:------:|:-----:|:-----:| +| SVTR Tiny | 96.85 | 91.34 | 94.53 | 83.99 | 85.43 | 89.24 | 90.87 | 80.55 | 95.37 | 95.27 | 95.70 | 90.13 | 67.90 | [英文](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar) / [中文](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_ch_train.tar) | +| SVTR Small | 95.92 | 93.04 | 95.03 | 84.70 | 87.91 | 92.01 | 91.63 | 82.72 | 94.88 | 96.08 | 96.28 | 91.02 | 69.00 | [英文](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_small_none_ctc_en_train.tar) / [中文](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_small_none_ctc_ch_train.tar) | +| SVTR Base | 97.08 | 91.50 | 96.03 | 85.20 | 89.92 | 91.67 | 92.33 | 83.73 | 95.66 | 95.62 | 95.81 | 91.61 | 71.40 | [英文](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_base_none_ctc_en_train.tar) / - | +| SVTR Large | 97.20 | 91.65 | 96.30 | 86.58 | 88.37 | 95.14 | 92.82 | 84.54 | 96.35 | 96.54 | 96.74 | 92.24 | 72.10 | [英文](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_large_none_ctc_en_train.tar) / [中文](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_large_none_ctc_ch_train.tar) | + +## 2. 环境配置 + +请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。 + +## 3. 模型训练、评估、预测 + +### 3.1 模型训练 + +#### 数据集准备 + +[英文数据集下载](https://github.com/clovaai/deep-text-recognition-benchmark#download-lmdb-dataset-for-traininig-and-evaluation-from-here) +[中文数据集下载](https://github.com/fudanvi/benchmarking-chinese-text-recognition#download) + +#### 启动训练 + +请参考[文本识别训练教程](../../ppocr/model_train/recognition.md)。PaddleOCR对代码进行了模块化，训练`SVTR`识别模型时需要**更换配置文件**为`SVTR`的[配置文件](../../configs/rec/rec_svtrnet.yml)。 + +具体地，在完成数据准备后，便可以启动训练，训练命令如下： + +```bash linenums="1" +#单卡训练（训练周期长，不建议） +python3 tools/train.py -c configs/rec/rec_svtrnet.yml + +# 多卡训练，通过--gpus参数指定卡号 +python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_svtrnet.yml +``` + +### 3.2 评估 + +可下载`SVTR`提供的模型文件和配置文件：[下载地址](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar) ，以`SVTR-T`为例，使用如下命令进行评估： + +```bash linenums="1" +# 下载包含SVTR-T的模型文件和配置文件的tar压缩包并解压 +wget https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar && tar xf rec_svtr_tiny_none_ctc_en_train.tar +# 注意将pretrained_model的路径设置为本地路径。 +python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c ./rec_svtr_tiny_none_ctc_en_train/rec_svtr_tiny_6local_6global_stn_en.yml -o Global.pretrained_model=./rec_svtr_tiny_none_ctc_en_train/best_accuracy +``` + +### 3.3 预测 + +使用如下命令进行单张图片预测： + +```bash linenums="1" +# 注意将pretrained_model的路径设置为本地路径。 +python3 tools/infer_rec.py -c ./rec_svtr_tiny_none_ctc_en_train/rec_svtr_tiny_6local_6global_stn_en.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_svtr_tiny_none_ctc_en_train/best_accuracy +# 预测文件夹下所有图像时，可修改infer_img为文件夹，如 Global.infer_img='./doc/imgs_words_en/'。 +``` + +## 4. 推理部署 + +### 4.1 Python推理 + +首先将训练得到best模型，转换成inference model。下面以`SVTR-T`在英文数据集训练的模型为例（[模型和配置文件下载地址](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/rec_svtr_tiny_none_ctc_en_train.tar) )，可以使用如下命令进行转换： + +```bash linenums="1" +# 注意将pretrained_model的路径设置为本地路径。 +python3 tools/export_model.py -c ./rec_svtr_tiny_none_ctc_en_train/rec_svtr_tiny_6local_6global_stn_en.yml -o Global.pretrained_model=./rec_svtr_tiny_none_ctc_en_train/best_accuracy Global.save_inference_dir=./inference/rec_svtr_tiny_stn_en +``` + +**注意：** 如果您是在自己的数据集上训练的模型，并且调整了字典文件，请注意修改配置文件中的`character_dict_path`是否为所正确的字典文件。 + +转换成功后，在目录下有三个文件： + +```text linenums="1" +/inference/rec_svtr_tiny_stn_en/ + ├── inference.pdiparams # 识别inference模型的参数文件 + ├── inference.pdiparams.info # 识别inference模型的参数信息，可忽略 + └── inference.pdmodel # 识别inference模型的program文件 +``` + +执行如下命令进行模型推理： + +```bash linenums="1" +python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_svtr_tiny_stn_en/' --rec_algorithm='SVTR' --rec_image_shape='3,64,256' --rec_char_dict_path='./ppocr/utils/ic15_dict.txt' +# 预测文件夹下所有图像时，可修改image_dir为文件夹，如 --image_dir='./doc/imgs_words_en/'。 +``` + + + +执行命令后，上面图像的预测结果（识别的文本和得分）会打印到屏幕上，示例如下： +结果如下： + +```bash linenums="1" +Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9999998807907104) +``` + +**注意**： + +* 如果您调整了训练时的输入分辨率，需要通过参数`rec_image_shape`设置为您需要的识别图像形状。 +* 在推理时需要设置参数`rec_char_dict_path`指定字典，如果您修改了字典，请修改该参数为您的字典文件。 +* 如果您修改了预处理方法，需修改`tools/infer/predict_rec.py`中SVTR的预处理为您的预处理方法。 + +### 4.2 C++推理部署 + +由于C++预处理后处理还未支持SVTR，所以暂未支持 + +### 4.3 Serving服务化部署 + +暂不支持 + +### 4.4 更多推理部署 + +暂不支持 + +## 5. FAQ + +* 1. GPU和CPU速度对比 + * 由于`SVTR`使用的算子大多为矩阵相乘，在GPU环境下，速度具有优势，但在CPU开启mkldnn加速环境下，`SVTR`相比于被优化的卷积网络没有优势。 + +* 2. SVTR模型转ONNX失败 + * 保证`paddle2onnx`和`onnxruntime`版本最新，转onnx命令参考[SVTR模型转onnx步骤实例](https://github.com/PaddlePaddle/PaddleOCR/issues/7821#issuecomment-1271214273)。 +* 3. SVTR转ONNX成功但是推理结果不正确 + * 可能的原因模型参数`out_char_num`设置不正确，应设置为W//4、W//8或者W//12，可以参考[高精度中文场景文本识别模型SVTR的3.3.3章节](https://aistudio.baidu.com/aistudio/projectdetail/5073182?contributionType=1)。 +* 4. 长文本识别优化 + * 参考[高精度中文场景文本识别模型SVTR的3.3章节](https://aistudio.baidu.com/aistudio/projectdetail/5073182?contributionType=1)。 +* 5. 论文结果复现注意事项 + * 数据集使用[ABINet](https://github.com/FangShancheng/ABINet)提供的数据集； + * 默认使用4卡GPU训练，单卡Batchsize默认为512，总Batchsize为2048，对应的学习率为0.0005，当修改Batchsize或者改变GPU卡数，学习率应等比例修改。 +* 6. 进一步优化的探索点 + * 学习率调整：可以调整为默认的两倍保持Batchsize不变；或者将Batchsize减小为默认的1/2，保持学习率不变； + * 数据增强策略：可选`RecConAug`和`RecAug`； + * 如果不使用STN时，可以将`mixer`的`Local`替换为`Conv`、`local_mixer`全部修改为`[5, 5]`； + * 网格搜索最优的`embed_dim`、`depth`、`num_heads`配置； + * 使用`后Normalization策略`，即是将模型配置`prenorm`修改为`True`。 + +## 引用 + +```bibtex +@article{Du2022SVTR, + title = {SVTR: Scene Text Recognition with a Single Visual Model}, + author = {Du, Yongkun and Chen, Zhineng and Jia, Caiyan and Yin, Xiaoting and Zheng, Tianlun and Li, Chenxia and Du, Yuning and Jiang, Yu-Gang}, + booktitle = {IJCAI}, + year = {2022}, + url = {https://arxiv.org/abs/2205.00159} +} +``` diff --git a/docs/algorithm/text_recognition/algorithm_rec_svtrv2.md b/docs/algorithm/text_recognition/algorithm_rec_svtrv2.md new file mode 100644 index 0000000000..22f320d3e5 --- /dev/null +++ b/docs/algorithm/text_recognition/algorithm_rec_svtrv2.md @@ -0,0 +1,121 @@ +--- +comments: true +--- + +# 场景文本识别算法-SVTRv2 + +## 1. 算法简介 + +### SVTRv2算法简介 + +[PaddleOCR 算法模型挑战赛 - 赛题一：OCR 端到端识别任务](https://aistudio.baidu.com/competition/detail/1131/0/introduction)排行榜第一算法。主要思路：1、检测和识别模型的Backbone升级为RepSVTR；2、识别教师模型升级为SVTRv2，可识别长文本。 + +## 2. 环境配置 + +请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。 + +## 3. 模型训练、评估、预测 + +### 3.1 模型训练 + +训练命令： + +```bash linenums="1" +#单卡训练（训练周期长，不建议） +python3 tools/train.py -c configs/rec/SVTRv2/rec_repsvtr_gtc.yml + +# 多卡训练，通过--gpus参数指定卡号 +# Rec 学生模型 +python -m paddle.distributed.launch --gpus '0,1,2,3,4,5,6,7' tools/train.py -c configs/rec/SVTRv2/rec_repsvtr_gtc.yml +# Rec 教师模型 +python -m paddle.distributed.launch --gpus '0,1,2,3,4,5,6,7' tools/train.py -c configs/rec/SVTRv2/rec_svtrv2_gtc.yml +# Rec 蒸馏训练 +python -m paddle.distributed.launch --gpus '0,1,2,3,4,5,6,7' tools/train.py -c configs/rec/SVTRv2/rec_svtrv2_gtc_distill.yml +``` + +### 3.2 评估 + +```bash linenums="1" +# 注意将pretrained_model的路径设置为本地路径。 +python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/SVTRv2/rec_repsvtr_gtc.yml -o Global.pretrained_model=output/rec_repsvtr_gtc/best_accuracy +``` + +### 3.3 预测 + +使用如下命令进行单张图片预测： + +```bash linenums="1" +# 注意将pretrained_model的路径设置为本地路径。 +python3 tools/infer_rec.py -c tools/eval.py -c configs/rec/SVTRv2/rec_repsvtr_gtc.yml -o Global.pretrained_model=output/rec_repsvtr_gtc/best_accuracy Global.infer_img='./doc/imgs_words_en/word_10.png' +# 预测文件夹下所有图像时，可修改infer_img为文件夹，如 Global.infer_img='./doc/imgs_words_en/'。 +``` + +## 4. 推理部署 + +### 4.1 Python推理 + +首先将训练得到best模型，转换成inference model，以RepSVTR为例，可以使用如下命令进行转换： + +```bash linenums="1" +# 注意将pretrained_model的路径设置为本地路径。 +python3 tools/export_model.py -c configs/rec/SVTRv2/rec_repsvtr_gtc.yml -o Global.pretrained_model=output/rec_repsvtr_gtc/best_accuracy Global.save_inference_dir=./inference/rec_repsvtr_infer +``` + +**注意：** 如果您是在自己的数据集上训练的模型，并且调整了字典文件，请注意修改配置文件中的`character_dict_path`是否为所正确的字典文件。 + +转换成功后，在目录下有三个文件： + +```text linenums="1" +./inference/rec_repsvtr_infer/ + ├── inference.pdiparams # 识别inference模型的参数文件 + ├── inference.pdiparams.info # 识别inference模型的参数信息，可忽略 + └── inference.pdmodel # 识别inference模型的program文件 +``` + +执行如下命令进行模型推理： + +```bash linenums="1" +python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_repsvtr_infer/' +# 预测文件夹下所有图像时，可修改image_dir为文件夹，如 --image_dir='./doc/imgs_words_en/'。 +``` + + + +执行命令后，上面图像的预测结果（识别的文本和得分）会打印到屏幕上，示例如下： +结果如下： + +```bash linenums="1" +Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9999998807907104) +``` + +**注意**： + +- 如果您调整了训练时的输入分辨率，需要通过参数`rec_image_shape`设置为您需要的识别图像形状。 +- 在推理时需要设置参数`rec_char_dict_path`指定字典，如果您修改了字典，请修改该参数为您的字典文件。 +- 如果您修改了预处理方法，需修改`tools/infer/predict_rec.py`中SVTR的预处理为您的预处理方法。 + +### 4.2 C++推理部署 + +由于C++预处理后处理还未支持SVTRv2 + +### 4.3 Serving服务化部署 + +暂不支持 + +### 4.4 更多推理部署 + +暂不支持 + +## 5. FAQ + +## 引用 + +```bibtex +@article{Du2022SVTR, + title = {SVTR: Scene Text Recognition with a Single Visual Model}, + author = {Du, Yongkun and Chen, Zhineng and Jia, Caiyan and Yin, Xiaoting and Zheng, Tianlun and Li, Chenxia and Du, Yuning and Jiang, Yu-Gang}, + booktitle = {IJCAI}, + year = {2022}, + url = {https://arxiv.org/abs/2205.00159} +} +``` diff --git a/docs/algorithm/text_recognition/algorithm_rec_visionlan.en.md b/docs/algorithm/text_recognition/algorithm_rec_visionlan.en.md new file mode 100644 index 0000000000..99566d6858 --- /dev/null +++ b/docs/algorithm/text_recognition/algorithm_rec_visionlan.en.md @@ -0,0 +1,120 @@ +--- +comments: true +--- + +# VisionLAN + +## 1. Introduction + +Paper: +> [From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network](https://arxiv.org/abs/2108.09661) +> Yuxin Wang, Hongtao Xie, Shancheng Fang, Jing Wang, Shenggao Zhu, Yongdong Zhang +> ICCV, 2021 + +Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: + +|Model|Backbone|config|Acc|Download link| +| --- | --- | --- | --- | --- | +|VisionLAN|ResNet45|[rec_r45_visionlan.yml](../../configs/rec/rec_r45_visionlan.yml)|90.30%|[model link](https://paddleocr.bj.bcebos.com/VisionLAN/rec_r45_visionlan_train.tar)| + +## 2. Environment + +Please refer to ["Environment Preparation"](../../ppocr/environment.en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](../../ppocr/blog/clone.en.md)to clone the project code. + +## 3. Model Training / Evaluation / Prediction + +Please refer to [Text Recognition Tutorial](../../ppocr/model_train/recognition.en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. + +### Training + +Specifically, after the data preparation is completed, the training can be started. The training command is as follows: + +```bash linenums="1" +# Single GPU training (long training period, not recommended) +python3 tools/train.py -c configs/rec/rec_r45_visionlan.yml + +# Multi GPU training, specify the gpu number through the --gpus parameter +python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r45_visionlan.yml +``` + +### Evaluation + +```bash linenums="1" +# GPU evaluation +python3 tools/eval.py -c configs/rec/rec_r45_visionlan.yml -o Global.pretrained_model={path/to/weights}/best_accuracy +``` + +### Prediction + +```bash linenums="1" +# The configuration file used for prediction must match the training +python3 tools/infer_rec.py -c configs/rec/rec_r45_visionlan.yml -o Global.infer_img='./doc/imgs_words/en/word_2.png' Global.pretrained_model=./rec_r45_visionlan_train/best_accuracy +``` + +## 4. Inference and Deployment + +### 4.1 Python Inference + +First, the model saved during the VisionLAN text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/VisionLAN/rec_r45_visionlan_train.tar)) ), you can use the following command to convert: + +```bash linenums="1" +python3 tools/export_model.py -c configs/rec/rec_r45_visionlan.yml -o Global.pretrained_model=./rec_r45_visionlan_train/best_accuracy Global.save_inference_dir=./inference/rec_r45_visionlan/ +``` + +**Note:** + +- If you are training the model on your own dataset and have modified the dictionary file, please pay attention to modify the `character_dict_path` in the configuration file to the modified dictionary file. +- If you modified the input size during training, please modify the `infer_shape` corresponding to VisionLAN in the `tools/export_model.py` file. + +After the conversion is successful, there are three files in the directory: + +```text linenums="1" +./inference/rec_r45_visionlan/ + ├── inference.pdiparams + ├── inference.pdiparams.info + └── inference.pdmodel +``` + +For VisionLAN text recognition model inference, the following commands can be executed: + +```bash linenums="1" +python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words/en/word_2.png' --rec_model_dir='./inference/rec_r45_visionlan/' --rec_algorithm='VisionLAN' --rec_image_shape='3,64,256' --rec_char_dict_path='./ppocr/utils/ic15_dict.txt' --use_space_char=False +``` + + + +After executing the command, the prediction result (recognized text and score) of the image above is printed to the screen, an example is as follows: +The result is as follows: + +```bash linenums="1" +Predicts of ./doc/imgs_words/en/word_2.png:('yourself', 0.9999493) +``` + +### 4.2 C++ Inference + +Not supported + +### 4.3 Serving + +Not supported + +### 4.4 More + +Not supported + +## 5. FAQ + +1. Note that the MJSynth and SynthText datasets come from [VisionLAN repo](https://github.com/wangyuxin87/VisionLAN). +2. We use the pre-trained model provided by the VisionLAN authors for finetune training. The dictionary for the pre-trained model is 'ppocr/utils/ic15_dict.txt'. + +## Citation + +```bibtex +@inproceedings{wang2021two, + title={From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network}, + author={Wang, Yuxin and Xie, Hongtao and Fang, Shancheng and Wang, Jing and Zhu, Shenggao and Zhang, Yongdong}, + booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision}, + pages={14194--14203}, + year={2021} +} +``` diff --git a/docs/algorithm/text_recognition/algorithm_rec_visionlan.md b/docs/algorithm/text_recognition/algorithm_rec_visionlan.md new file mode 100644 index 0000000000..1bb4a4f20f --- /dev/null +++ b/docs/algorithm/text_recognition/algorithm_rec_visionlan.md @@ -0,0 +1,135 @@ +--- +comments: true +--- + +# 场景文本识别算法-VisionLAN + +## 1. 算法简介 + +论文信息： +> [From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network](https://arxiv.org/abs/2108.09661) +> Yuxin Wang, Hongtao Xie, Shancheng Fang, Jing Wang, Shenggao Zhu, Yongdong Zhang +> ICCV, 2021 + +`VisionLAN`使用MJSynth和SynthText两个文字识别数据集训练，在IIIT, SVT, IC13, IC15, SVTP, CUTE数据集上进行评估，算法复现效果如下： + +|模型|骨干网络|配置文件|Acc|下载链接| +| --- | --- | --- | --- | --- | +|VisionLAN|ResNet45|[rec_r45_visionlan.yml](../../configs/rec/rec_r45_visionlan.yml)|90.30%|[预训练、训练模型](https://paddleocr.bj.bcebos.com/VisionLAN/rec_r45_visionlan_train.tar)| + +## 2. 环境配置 + +请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。 + +## 3. 模型训练、评估、预测 + +### 3.1 模型训练 + +请参考[文本识别训练教程](../../ppocr/model_train/recognition.md)。PaddleOCR对代码进行了模块化，训练`VisionLAN`识别模型时需要**更换配置文件**为`VisionLAN`的[配置文件](../../configs/rec/rec_r45_visionlan.yml)。 + +#### 启动训练 + +具体地，在完成数据准备后，便可以启动训练，训练命令如下： + +```bash linenums="1" +#单卡训练（训练周期长，不建议） +python3 tools/train.py -c configs/rec/rec_r45_visionlan.yml + +# 多卡训练，通过--gpus参数指定卡号 +python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_r45_visionlan.yml +``` + +### 3.2 评估 + +可下载已训练完成的模型文件，使用如下命令进行评估： + +```bash linenums="1" +# 注意将pretrained_model的路径设置为本地路径。 +python3 tools/eval.py -c configs/rec/rec_r45_visionlan.yml -o Global.pretrained_model=./rec_r45_visionlan_train/best_accuracy +``` + +### 3.3 预测 + +使用如下命令进行单张图片预测： + +```bash linenums="1" +# 注意将pretrained_model的路径设置为本地路径。 +python3 tools/infer_rec.py -c configs/rec/rec_r45_visionlan.yml -o Global.infer_img='./doc/imgs_words/en/word_2.png' Global.pretrained_model=./rec_r45_visionlan_train/best_accuracy +# 预测文件夹下所有图像时，可修改infer_img为文件夹，如 Global.infer_img='./doc/imgs_words_en/'。 +``` + +## 4. 推理部署 + +### 4.1 Python推理 + +首先将训练得到best模型，转换成inference model。这里以训练完成的模型为例（[模型下载地址](https://paddleocr.bj.bcebos.com/VisionLAN/rec_r45_visionlan_train.tar))，可以使用如下命令进行转换： + +```bash linenums="1" +# 注意将pretrained_model的路径设置为本地路径。 +python3 tools/export_model.py -c configs/rec/rec_r45_visionlan.yml -o Global.pretrained_model=./rec_r45_visionlan_train/best_accuracy Global.save_inference_dir=./inference/rec_r45_visionlan/ +``` + +**注意：** + +- 如果您是在自己的数据集上训练的模型，并且调整了字典文件，请注意修改配置文件中的`character_dict_path`是否是所需要的字典文件。 +- 如果您修改了训练时的输入大小，请修改`tools/export_model.py`文件中的对应VisionLAN的`infer_shape`。 + +转换成功后，在目录下有三个文件： + +```text linenums="1" +./inference/rec_r45_visionlan/ + ├── inference.pdiparams # 识别inference模型的参数文件 + ├── inference.pdiparams.info # 识别inference模型的参数信息，可忽略 + └── inference.pdmodel # 识别inference模型的program文件 +``` + +执行如下命令进行模型推理： + +```bash linenums="1" +python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words/en/word_2.png' --rec_model_dir='./inference/rec_r45_visionlan/' --rec_algorithm='VisionLAN' --rec_image_shape='3,64,256' --rec_char_dict_path='./ppocr/utils/ic15_dict.txt' --use_space_char=False +# 预测文件夹下所有图像时，可修改image_dir为文件夹，如 --image_dir='./doc/imgs_words_en/'。 +``` + + + +执行命令后，上面图像的预测结果（识别的文本和得分）会打印到屏幕上，示例如下： +结果如下： + +```bash linenums="1" +Predicts of ./doc/imgs_words/en/word_2.png:('yourself', 0.9999493) +``` + +**注意**： + +- 训练上述模型采用的图像分辨率是[3，64，256]，需要通过参数`rec_image_shape`设置为您训练时的识别图像形状。 +- 在推理时需要设置参数`rec_char_dict_path`指定字典，如果您修改了字典，请修改该参数为您的字典文件。 +- 如果您修改了预处理方法，需修改`tools/infer/predict_rec.py`中VisionLAN的预处理为您的预处理方法。 + +### 4.2 C++推理部署 + +由于C++预处理后处理还未支持VisionLAN，所以暂未支持 + +### 4.3 Serving服务化部署 + +暂不支持 + +### 4.4 更多推理部署 + +暂不支持 + +## 5. FAQ + +1. MJSynth和SynthText两种数据集来自于[VisionLAN源repo](https://github.com/wangyuxin87/VisionLAN) 。 +2. 我们使用VisionLAN作者提供的预训练模型进行finetune训练，预训练模型配套字典为'ppocr/utils/ic15_dict.txt'。 + +## 引用 + +```bibtex +@inproceedings{wang2021two, + title={From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network}, + author={Wang, Yuxin and Xie, Hongtao and Fang, Shancheng and Wang, Jing and Zhu, Shenggao and Zhang, Yongdong}, + booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision}, + pages={14194--14203}, + year={2021} +} +``` diff --git a/docs/algorithm/text_recognition/algorithm_rec_vitstr.en.md b/docs/algorithm/text_recognition/algorithm_rec_vitstr.en.md new file mode 100644 index 0000000000..9a56504cfa --- /dev/null +++ b/docs/algorithm/text_recognition/algorithm_rec_vitstr.en.md @@ -0,0 +1,119 @@ +--- +comments: true +--- + +# ViTSTR + +## 1. Introduction + +Paper: +> [Vision Transformer for Fast and Efficient Scene Text Recognition](https://arxiv.org/abs/2105.08582) +> Rowel Atienza +> ICDAR, 2021 + +Using MJSynth and SynthText two text recognition datasets for training, and evaluating on IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE datasets, the algorithm reproduction effect is as follows: + +|Model|Backbone|config|Acc|Download link| +| --- | --- | --- | --- | --- | +|ViTSTR|ViTSTR|[rec_vitstr_none_ce.yml](../../configs/rec/rec_vitstr_none_ce.yml)|79.82%|[trained model](https://paddleocr.bj.bcebos.com/rec_vitstr_none_none_train.tar)| + +## 2. Environment + +Please refer to ["Environment Preparation"](../../ppocr/environment.en.md) to configure the PaddleOCR environment, and refer to ["Project Clone"](../../ppocr/blog/clone.en.md)to clone the project code. + +## 3. Model Training / Evaluation / Prediction + +Please refer to [Text Recognition Tutorial](../../ppocr/model_train/recognition.en.md). PaddleOCR modularizes the code, and training different recognition models only requires **changing the configuration file**. + +### Training + +Specifically, after the data preparation is completed, the training can be started. The training command is as follows: + +```bash linenums="1" +# Single GPU training (long training period, not recommended) +python3 tools/train.py -c configs/rec/rec_vitstr_none_ce.yml + +# Multi GPU training, specify the gpu number through the --gpus parameter +python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_vitstr_none_ce.yml +``` + +### Evaluation + +```bash linenums="1" +# GPU evaluation +python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_vitstr_none_ce.yml -o Global.pretrained_model={path/to/weights}/best_accuracy +``` + +### Prediction + +```bash linenums="1" +# The configuration file used for prediction must match the training +python3 tools/infer_rec.py -c configs/rec/rec_vitstr_none_ce.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_vitstr_none_ce_train/best_accuracy +``` + +## 4. Inference and Deployment + +### 4.1 Python Inference + +First, the model saved during the ViTSTR text recognition training process is converted into an inference model. ( [Model download link](https://paddleocr.bj.bcebos.com/rec_vitstr_none_none_train.tar)) ), you can use the following command to convert: + +```bash linenums="1" +python3 tools/export_model.py -c configs/rec/rec_vitstr_none_ce.yml -o Global.pretrained_model=./rec_vitstr_none_ce_train/best_accuracy Global.save_inference_dir=./inference/rec_vitstr +``` + +**Note:** + +- If you are training the model on your own dataset and have modified the dictionary file, please pay attention to modify the `character_dict_path` in the configuration file to the modified dictionary file. +- If you modified the input size during training, please modify the `infer_shape` corresponding to ViTSTR in the `tools/export_model.py` file. + +After the conversion is successful, there are three files in the directory: + +```text linenums="1" +/inference/rec_vitstr/ + ├── inference.pdiparams + ├── inference.pdiparams.info + └── inference.pdmodel +``` + +For ViTSTR text recognition model inference, the following commands can be executed: + +```bash linenums="1" +python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_vitstr/' --rec_algorithm='ViTSTR' --rec_image_shape='1,224,224' --rec_char_dict_path='./ppocr/utils/EN_symbol_dict.txt' +``` + + + +After executing the command, the prediction result (recognized text and score) of the image above is printed to the screen, an example is as follows: +The result is as follows: + +```bash linenums="1" +Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9998350143432617) +``` + +### 4.2 C++ Inference + +Not supported + +### 4.3 Serving + +Not supported + +### 4.4 More + +Not supported + +## 5. FAQ + +1. In the `ViTSTR` paper, using pre-trained weights on ImageNet1k for initial training, we did not use pre-trained weights in training, and the final accuracy did not change or even improved. + +## Citation + +```bibtex +@article{Atienza2021ViTSTR, + title = {Vision Transformer for Fast and Efficient Scene Text Recognition}, + author = {Rowel Atienza}, + booktitle = {ICDAR}, + year = {2021}, + url = {https://arxiv.org/abs/2105.08582} +} +``` diff --git a/docs/algorithm/text_recognition/algorithm_rec_vitstr.md b/docs/algorithm/text_recognition/algorithm_rec_vitstr.md new file mode 100644 index 0000000000..9f2912e353 --- /dev/null +++ b/docs/algorithm/text_recognition/algorithm_rec_vitstr.md @@ -0,0 +1,135 @@ +--- +comments: true +--- + +# 场景文本识别算法-ViTSTR + +## 1. 算法简介 + +论文信息： +> [Vision Transformer for Fast and Efficient Scene Text Recognition](https://arxiv.org/abs/2105.08582) +> Rowel Atienza +> ICDAR, 2021 + +`ViTSTR`使用MJSynth和SynthText两个文字识别数据集训练，在IIIT, SVT, IC03, IC13, IC15, SVTP, CUTE数据集上进行评估，算法复现效果如下： + +|模型|骨干网络|配置文件|Acc|下载链接| +| --- | --- | --- | --- | --- | +|ViTSTR|ViTSTR|[rec_vitstr_none_ce.yml](../../configs/rec/rec_vitstr_none_ce.yml)|79.82%|[训练模型](https://paddleocr.bj.bcebos.com/rec_vitstr_none_ce_train.tar)| + +## 2. 环境配置 + +请先参考[《运行环境准备》](../../ppocr/environment.md)配置PaddleOCR运行环境，参考[《项目克隆》](../../ppocr/blog/clone.md)克隆项目代码。 + +## 3. 模型训练、评估、预测 + +### 3.1 模型训练 + +请参考[文本识别训练教程](../../ppocr/model_train/recognition.md)。PaddleOCR对代码进行了模块化，训练`ViTSTR`识别模型时需要**更换配置文件**为`ViTSTR`的[配置文件](../../configs/rec/rec_vitstr_none_ce.yml)。 + +#### 启动训练 + +具体地，在完成数据准备后，便可以启动训练，训练命令如下： + +```bash linenums="1" +#单卡训练（训练周期长，不建议） +python3 tools/train.py -c configs/rec/rec_vitstr_none_ce.yml + +# 多卡训练，通过--gpus参数指定卡号 +python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_vitstr_none_ce.yml +``` + +### 3.2 评估 + +可下载已训练完成的模型文件，使用如下命令进行评估： + +```bash linenums="1" +# 注意将pretrained_model的路径设置为本地路径。 +python3 -m paddle.distributed.launch --gpus '0' tools/eval.py -c configs/rec/rec_vitstr_none_ce.yml -o Global.pretrained_model=./rec_vitstr_none_ce_train/best_accuracy +``` + +### 3.3 预测 + +使用如下命令进行单张图片预测： + +```bash linenums="1" +# 注意将pretrained_model的路径设置为本地路径。 +python3 tools/infer_rec.py -c configs/rec/rec_vitstr_none_ce.yml -o Global.infer_img='./doc/imgs_words_en/word_10.png' Global.pretrained_model=./rec_vitstr_none_ce_train/best_accuracy +# 预测文件夹下所有图像时，可修改infer_img为文件夹，如 Global.infer_img='./doc/imgs_words_en/'。 +``` + +## 4. 推理部署 + +### 4.1 Python推理 + +首先将训练得到best模型，转换成inference model。这里以训练完成的模型为例（[模型下载地址](https://paddleocr.bj.bcebos.com/rec_vitstr_none_ce_train.tar) )，可以使用如下命令进行转换： + +```bash linenums="1" +# 注意将pretrained_model的路径设置为本地路径。 +python3 tools/export_model.py -c configs/rec/rec_vitstr_none_ce.yml -o Global.pretrained_model=./rec_vitstr_none_ce_train/best_accuracy Global.save_inference_dir=./inference/rec_vitstr/ +``` + +**注意：** + +- 如果您是在自己的数据集上训练的模型，并且调整了字典文件，请注意修改配置文件中的`character_dict_path`是否是所需要的字典文件。 +- 如果您修改了训练时的输入大小，请修改`tools/export_model.py`文件中的对应ViTSTR的`infer_shape`。 + +转换成功后，在目录下有三个文件： + +```text linenums="1" +/inference/rec_vitstr/ + ├── inference.pdiparams # 识别inference模型的参数文件 + ├── inference.pdiparams.info # 识别inference模型的参数信息，可忽略 + └── inference.pdmodel # 识别inference模型的program文件 +``` + +执行如下命令进行模型推理： + +```bash linenums="1" +python3 tools/infer/predict_rec.py --image_dir='./doc/imgs_words_en/word_10.png' --rec_model_dir='./inference/rec_vitstr/' --rec_algorithm='ViTSTR' --rec_image_shape='1,224,224' --rec_char_dict_path='./ppocr/utils/EN_symbol_dict.txt' +# 预测文件夹下所有图像时，可修改image_dir为文件夹，如 --image_dir='./doc/imgs_words_en/'。 +``` + + + +执行命令后，上面图像的预测结果（识别的文本和得分）会打印到屏幕上，示例如下： +结果如下： + +```bash linenums="1" +Predicts of ./doc/imgs_words_en/word_10.png:('pain', 0.9998350143432617) +``` + +**注意**： + +- 训练上述模型采用的图像分辨率是[1，224，224]，需要通过参数`rec_image_shape`设置为您训练时的识别图像形状。 +- 在推理时需要设置参数`rec_char_dict_path`指定字典，如果您修改了字典，请修改该参数为您的字典文件。 +- 如果您修改了预处理方法，需修改`tools/infer/predict_rec.py`中ViTSTR的预处理为您的预处理方法。 + +### 4.2 C++推理部署 + +由于C++预处理后处理还未支持ViTSTR，所以暂未支持 + +### 4.3 Serving服务化部署 + +暂不支持 + +### 4.4 更多推理部署 + +暂不支持 + +## 5. FAQ + +1. 在`ViTSTR`论文中，使用在ImageNet1k上的预训练权重进行初始化训练，我们在训练未采用预训练权重，最终精度没有变化甚至有所提高。 +2. 我们仅仅复现了`ViTSTR`中的tiny版本，如果需要使用small、base版本，可将[ViTSTR源repo](https://github.com/roatienza/deep-text-recognition-benchmark) 中的预训练权重转为Paddle权重使用。 + +## 引用 + +```bibtex +@article{Atienza2021ViTSTR, + title = {Vision Transformer for Fast and Efficient Scene Text Recognition}, + author = {Rowel Atienza}, + booktitle = {ICDAR}, + year = {2021}, + url = {https://arxiv.org/abs/2105.08582} +} +``` diff --git a/docs/algorithm/text_recognition/images/word_1-20240704183926496.png b/docs/algorithm/text_recognition/images/word_1-20240704183926496.png new file mode 100644 index 0000000000..7b915fd6da Binary files /dev/null and b/docs/algorithm/text_recognition/images/word_1-20240704183926496.png differ diff --git a/docs/algorithm/text_recognition/images/word_1-20240704184113913.png b/docs/algorithm/text_recognition/images/word_1-20240704184113913.png new file mode 100644 index 0000000000..7b915fd6da Binary files /dev/null and b/docs/algorithm/text_recognition/images/word_1-20240704184113913.png differ diff --git a/docs/algorithm/text_recognition/images/word_10.png b/docs/algorithm/text_recognition/images/word_10.png new file mode 100644 index 0000000000..07370f757e Binary files /dev/null and b/docs/algorithm/text_recognition/images/word_10.png differ diff --git a/docs/algorithm/text_recognition/images/word_336-20240705082445918.png b/docs/algorithm/text_recognition/images/word_336-20240705082445918.png new file mode 100644 index 0000000000..3bddd294ed Binary files /dev/null and b/docs/algorithm/text_recognition/images/word_336-20240705082445918.png differ diff --git a/docs/algorithm/text_recognition/images/word_336.png b/docs/algorithm/text_recognition/images/word_336.png new file mode 100644 index 0000000000..3bddd294ed Binary files /dev/null and b/docs/algorithm/text_recognition/images/word_336.png differ diff --git "a/docs/applications/PCB\345\255\227\347\254\246\350\257\206\345\210\253.md" "b/docs/applications/PCB\345\255\227\347\254\246\350\257\206\345\210\253.md" new file mode 100644 index 0000000000..d611db7c2e --- /dev/null +++ "b/docs/applications/PCB\345\255\227\347\254\246\350\257\206\345\210\253.md" @@ -0,0 +1,595 @@ +--- +typora-copy-images-to: images +comments: true +--- + +# 基于PP-OCRv3的PCB字符识别 + +## 1. 项目介绍 + +印刷电路板(PCB)是电子产品中的核心器件，对于板件质量的测试与监控是生产中必不可少的环节。在一些场景中，通过PCB中信号灯颜色和文字组合可以定位PCB局部模块质量问题，PCB文字识别中存在如下难点： + +- 裁剪出的PCB图片宽高比例较小 +- 文字区域整体面积也较小 +- 包含垂直、水平多种方向文本 + +针对本场景，PaddleOCR基于全新的PP-OCRv3通过合成数据、微调以及其他场景适配方法完成小字符文本识别任务，满足企业上线要求。PCB检测、识别效果如 **图1** 所示： + + + +注：欢迎在AIStudio领取免费算力体验线上实训，项目链接: [基于PP-OCRv3实现PCB字符识别](https://aistudio.baidu.com/aistudio/projectdetail/4008973) + +## 2. 安装说明 + +下载PaddleOCR源码，安装依赖环境。 + +```python linenums="1" +# 如仍需安装or安装更新，可以执行以下步骤 +git clone https://github.com/PaddlePaddle/PaddleOCR.git +# git clone https://gitee.com/PaddlePaddle/PaddleOCR +``` + +```python linenums="1" +# 安装依赖包 +pip install -r /home/aistudio/PaddleOCR/requirements.txt +``` + +## 3. 数据准备 + +我们通过图片合成工具生成 **图2** 所示的PCB图片，整图只有高25、宽150左右、文字区域高9、宽45左右，包含垂直和水平2种方向的文本： + + + +暂时不开源生成的PCB数据集，但是通过更换背景，通过如下代码生成数据即可： + +``` +cd gen_data +python3 gen.py --num_img=10 +``` + +生成图片参数解释： + +``` +num_img：生成图片数量 +font_min_size、font_max_size：字体最大、最小尺寸 +bg_path：文字区域背景存放路径 +det_bg_path：整图背景存放路径 +fonts_path：字体路径 +corpus_path：语料路径 +output_dir：生成图片存储路径 +``` + +这里生成 **100张** 相同尺寸和文本的图片，如 **图3** 所示，方便大家跑通实验。通过如下代码解压数据集： + + + +```python linenums="1" +tar xf ./data/data148165/dataset.tar -C ./ +``` + +在生成数据集的时需要生成检测和识别训练需求的格式： + +- **文本检测** + +标注文件格式如下，中间用'\t'分隔： + +``` +" 图像文件名 json.dumps编码的图像标注信息" +ch4_test_images/img_61.jpg [{"transcription": "MASA", "points": [[310, 104], [416, 141], [418, 216], [312, 179]]}, {...}] +``` + +json.dumps编码前的图像标注信息是包含多个字典的list，字典中的 `points` 表示文本框的四个点的坐标(x, y)，从左上角的点开始顺时针排列。 `transcription` 表示当前文本框的文字，***当其内容为“###”时，表示该文本框无效，在训练时会跳过。*** + +- **文本识别** + +标注文件的格式如下， txt文件中默认请将图片路径和图片标签用'\t'分割，如用其他方式分割将造成训练报错。 + +``` +" 图像文件名 图像标注信息 " + +train_data/rec/train/word_001.jpg 简单可依赖 +train_data/rec/train/word_002.jpg 用科技让复杂的世界更简单 +... +``` + +## 4. 文本检测 + +选用飞桨OCR开发套件[PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR)中的PP-OCRv3模型进行文本检测和识别。针对检测模型和识别模型，进行了共计9个方面的升级： + +- PP-OCRv3检测模型对PP-OCRv2中的CML协同互学习文本检测蒸馏策略进行了升级，分别针对教师模型和学生模型进行进一步效果优化。其中，在对教师模型优化时，提出了大感受野的PAN结构LK-PAN和引入了DML蒸馏策略；在对学生模型优化时，提出了残差注意力机制的FPN结构RSE-FPN。 + +- PP-OCRv3的识别模块是基于文本识别算法SVTR优化。SVTR不再采用RNN结构，通过引入Transformers结构更加有效地挖掘文本行图像的上下文信息，从而提升文本识别能力。PP-OCRv3通过轻量级文本识别网络SVTR_LCNet、Attention损失指导CTC损失训练策略、挖掘文字上下文信息的数据增广策略TextConAug、TextRotNet自监督预训练模型、UDML联合互学习策略、UIM无标注数据挖掘方案，6个方面进行模型加速和效果提升。 + +更多细节请参考PP-OCRv3[技术报告](../ppocr/blog/PP-OCRv3_introduction.md)。 + +我们使用 **3种方案** 进行检测模型的训练、评估： + +- **PP-OCRv3英文超轻量检测预训练模型直接评估** +- PP-OCRv3英文超轻量检测预训练模型 + **验证集padding**直接评估 +- PP-OCRv3英文超轻量检测预训练模型 + **fine-tune** + +### 4.1 预训练模型直接评估 + +我们首先通过PaddleOCR提供的预训练模型在验证集上进行评估，如果评估指标能满足效果，可以直接使用预训练模型，不再需要训练。 + +使用预训练模型直接评估步骤如下： + +#### 1）下载预训练模型 + +PaddleOCR已经提供了PP-OCR系列模型，部分模型展示如下表所示： + +| 模型简介 | 模型名称 | 推荐场景 | 检测模型 | 方向分类器 | 识别模型 | +| ------- | ----------------------- | --------------- | ---------------- | ------- | --------- | +| 中英文超轻量PP-OCRv3模型（16.2M） | ch_PP-OCRv3_xx | 移动端&服务器端 | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_det_distill_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/chinese/ch_PP-OCRv3_rec_train.tar) | +| 英文超轻量PP-OCRv3模型（13.4M） | en_PP-OCRv3_xx | 移动端&服务器端 | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_distill_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_rec_train.tar) | +| 中英文超轻量PP-OCRv2模型（13.0M） | ch_PP-OCRv2_xx | 移动端&服务器端 | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_det_distill_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_infer.tar) / [训练模型](https://paddleocr.bj.bcebos.com/PP-OCRv2/chinese/ch_PP-OCRv2_rec_train.tar) | +| 中英文超轻量PP-OCR mobile模型（9.4M） | ch_ppocr_mobile_v2.0_xx | 移动端&服务器端 | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_infer.tar) / [预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_det_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_infer.tar) / [预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_rec_pre.tar) | +| 中英文通用PP-OCR server模型（143.4M） | ch_ppocr_server_v2.0_xx | 服务器端 | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_det_infer.tar) / [预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_det_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_infer.tar) / [预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_mobile_v2.0_cls_train.tar) | [推理模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_rec_infer.tar) / [预训练模型](https://paddleocr.bj.bcebos.com/dygraph_v2.0/ch/ch_ppocr_server_v2.0_rec_pre.tar) | + +更多模型下载（包括多语言），可以参[考PP-OCR系列模型下载](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.5/doc/doc_ch/models_list.md) + +这里我们使用PP-OCRv3英文超轻量检测模型，下载并解压预训练模型： + +```python linenums="1" +# 如果更换其他模型，更新下载链接和解压指令就可以 +cd /home/aistudio/PaddleOCR +mkdir pretrain_models +cd pretrain_models +# 下载英文预训练模型 +wget https://paddleocr.bj.bcebos.com/PP-OCRv3/english/en_PP-OCRv3_det_distill_train.tar +tar xf en_PP-OCRv3_det_distill_train.tar && rm -rf en_PP-OCRv3_det_distill_train.tar +%cd .. +``` + +**模型评估** + +首先修改配置文件`configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml`中的以下字段： + +```text linenums="1" +Eval.dataset.data_dir：指向验证集图片存放目录,'/home/aistudio/dataset' +Eval.dataset.label_file_list：指向验证集标注文件,'/home/aistudio/dataset/det_gt_val.txt' +Eval.dataset.transforms.DetResizeForTest: 尺寸 + limit_side_len: 48 + limit_type: 'min' +``` + +然后在验证集上进行评估，具体代码如下： + +```python linenums="1" +cd /home/aistudio/PaddleOCR +python tools/eval.py \ + -c configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml \ + -o Global.checkpoints="./pretrain_models/en_PP-OCRv3_det_distill_train/best_accuracy" +``` + +### **4.2 预训练模型+验证集padding直接评估** + +考虑到PCB图片比较小，宽度只有25左右、高度只有140-170左右，我们在原图的基础上进行padding，再进行检测评估，padding前后效果对比如 **图4** 所示： + +
+
+将图片都padding到300*300大小，因为坐标信息发生了变化，我们同时要修改标注文件，在`/home/aistudio/dataset`目录里也提供了padding之后的图片，大家也可以尝试训练和评估：
+
+同上，我们需要修改配置文件`configs/det/ch_PP-OCRv3/ch_PP-OCRv3_det_cml.yml`中的以下字段：
+
+```text linenums="1"
+Eval.dataset.data_dir：指向验证集图片存放目录,'/home/aistudio/dataset'
+Eval.dataset.label_file_list：指向验证集标注文件,/home/aistudio/dataset/det_gt_padding_val.txt
+Eval.dataset.transforms.DetResizeForTest: 尺寸
+ limit_side_len: 1100
+ limit_type: 'min'
+```
+
+如需获取已训练模型，请加入PaddleX官方交流频道，获取20G OCR学习大礼包（内含《动手学OCR》电子书、课程回放视频、前沿论文等重磅资料）
+
+- PaddleX官方交流频道：可用于数据挖掘或对预测效率要求不高的场景。 | [模型下载](#2) | [中文](./高精度中文识别模型.md)/English |
|
+| 手写体识别 | 新增字形支持 | [模型下载](#2) | [中文](./手写文字识别.md)/English | 
|
+
+### 制造
+
+| 类别 | 亮点 | 模型下载 | 教程 | 示例图 |
+| -------------- | ------------------------------ | -------------- | ------------------------------------------------------------ | ------------------------------------------------------------ |
+| 数码管识别 | 数码管数据合成、漏识别调优 | [模型下载](#2) | [中文](./光功率计数码管字符识别/光功率计数码管字符识别.md)/English | 
|
+| 液晶屏读数识别 | 检测模型蒸馏、Serving部署 | [模型下载](#2) | [中文](./液晶屏读数识别.md)/English | 
|
+| 包装生产日期 | 点阵字符合成、过曝过暗文字识别 | [模型下载](#2) | [中文](./包装生产日期识别.md)/English | 
|
+| PCB文字识别 | 小尺寸文本检测与识别 | [模型下载](#2) | [中文](./PCB字符识别/PCB字符识别.md)/English | 
|
+| 电表识别 | 大分辨率图像检测调优 | [模型下载](#2) | | |
+| 液晶屏缺陷检测 | 非文字字符识别 | | | |
+
+### 金融
+
+| 类别 | 亮点 | 模型下载 | 教程 | 示例图 |
+| -------------- | ----------------------------- | -------------- | ----------------------------------------- | ------------------------------------------------------------ |
+| 表单VQA | 多模态通用表单结构化提取 | [模型下载](#2) | [中文](./多模态表单识别.md)/English | 
|
+| 增值税发票 | 关键信息抽取，SER、RE任务训练 | [模型下载](#2) | [中文](./发票关键信息抽取.md)/English | 
|
+| 印章检测与识别 | 端到端弯曲文本识别 | [模型下载](#2) | [中文](./印章弯曲文字识别.md)/English | 
|
+| 通用卡证识别 | 通用结构化提取 | [模型下载](#2) | [中文](./快速构建卡证类OCR.md)/English | 
|
+| 身份证识别 | 结构化提取、图像阴影 | | | |
+| 合同比对 | 密集文本检测、NLP关键信息抽取 | [模型下载](#2) | [中文](./扫描合同关键信息提取.md)/English | 
|
+
+### 交通
+
+| 类别 | 亮点 | 模型下载 | 教程 | 示例图 |
+| ----------------- | ------------------------------ | -------------- | ----------------------------------- | ------------------------------------------------------------ |
+| 车牌识别 | 多角度图像、轻量模型、端侧部署 | [模型下载](#2) | [中文](./轻量级车牌识别.md)/English | 
|
+| 驾驶证/行驶证识别 | 尽请期待 | | | |
+| 快递单识别 | 尽请期待 | | | |
+
+## 模型下载
+
+如需下载上述场景中已经训练好的垂类模型，可以加入PaddleX官方交流频道获取20G OCR学习大礼包（内含《动手学OCR》电子书、课程回放视频、前沿论文等重磅资料）
+
+- PaddleX官方交流频道：| alleadersh | 不贰过，推 | 从自己参与浙江数 | 。另一方 | ||||
| AnSha | 自己越 | 共商共建工作协商 | w.east | 抓好改革试点任务 | |||
| Edime | ImisesElec | 怀天下”。 | 22.26 | 31.61 | 4.30 | 794.94 | |
| ip | Profundi | ：2019年12月1 | Horspro | 444.48 | 2.41 | 87 | 679.98 |
| iehaiTrain | 组长蒋蕊 | Toafterdec | 203.43 | 23.54 | 4 | 4266.62 | |
| Tyint | roudlyRol | 谢您的好意，我知道 | ErChows | 48.90 | 1031 | 6 | |
| NaFlint | 一辈的 | aterreclam | 7823.86 | 9829.23 | 7.96 | 3068 | |
| 家上下游企业，5 | Tr | 景象。当地球上的我们 | Urelaw | 799.62 | 354.96 | 12.98 | 33 |
| 赛事（ | uestCh | 复制的业务模式并 | Listicjust | 9.23 | 92 | 53.22 | |
| Ca | Iskole | 扶贫"之名引导 | Papua | 7191.90 | 1.65 | 3.62 | 48 |
| 避讳 | ir | 但由于 | Fficeof | 0.22 | 6.37 | 7.17 | 3397.75 |
| ndaTurk | 百处遗址 | gMa | 1288.34 | 2053.66 | 2.29 | 885.45 | |
+
+```python linenums="1"
+import numpy as np
+import json
+import cv2
+import os
+from shapely.geometry import Polygon
+
+
+def poly2box(poly):
+ xmin = np.min(np.array(poly)[:, 0])
+ ymin = np.min(np.array(poly)[:, 1])
+ xmax = np.max(np.array(poly)[:, 0])
+ ymax = np.max(np.array(poly)[:, 1])
+ return np.array([[xmin, ymin], [xmax, ymin], [xmax, ymax], [xmin, ymax]])
+
+
+def draw_text_det_res(dt_boxes, src_im, color=(255, 255, 0)):
+ for box in dt_boxes:
+ box = np.array(box).astype(np.int32).reshape(-1, 2)
+ cv2.polylines(src_im, [box], True, color=color, thickness=2)
+ return src_im
+
+class LabelDecode(object):
+ def __init__(self, **kwargs):
+ pass
+
+ def __call__(self, data):
+ label = json.loads(data['label'])
+
+ nBox = len(label)
+ seal_boxes = self.get_seal_boxes(label)
+
+ gt_label = []
+
+ for seal_box in seal_boxes:
+ seal_anno = {'seal_box': seal_box}
+ boxes, txts, txt_tags = [], [], []
+
+ for bno in range(0, nBox):
+ box = label[bno]['points']
+ txt = label[bno]['transcription']
+ try:
+ ints = self.get_intersection(box, seal_box)
+ except Exception as E:
+ print(E)
+ continue
+
+ if abs(Polygon(box).area - self.get_intersection(box, seal_box)) < 1e-3 and \
+ abs(Polygon(box).area - self.get_union(box, seal_box)) > 1e-3:
+
+ boxes.append(box)
+ txts.append(txt)
+ if txt in ['*', '###', '待识别']:
+ txt_tags.append(True)
+ else:
+ txt_tags.append(False)
+
+ seal_anno['polys'] = boxes
+ seal_anno['texts'] = txts
+ seal_anno['ignore_tags'] = txt_tags
+
+ gt_label.append(seal_anno)
+
+ return gt_label
+
+ def get_seal_boxes(self, label):
+
+ nBox = len(label)
+ seal_box = []
+ for bno in range(0, nBox):
+ box = label[bno]['points']
+ if len(box) == 4:
+ seal_box.append(box)
+
+ if len(seal_box) == 0:
+ return None
+
+ seal_box = self.valid_seal_box(seal_box)
+ return seal_box
+
+
+ def is_seal_box(self, box, boxes):
+ is_seal = True
+ for poly in boxes:
+ if list(box.shape()) != list(box.shape.shape()):
+ if abs(Polygon(box).area - self.get_intersection(box, poly)) < 1e-3:
+ return False
+ else:
+ if np.sum(np.array(box) - np.array(poly)) < 1e-3:
+ # continue when the box is same with poly
+ continue
+ if abs(Polygon(box).area - self.get_intersection(box, poly)) < 1e-3:
+ return False
+ return is_seal
+
+
+ def valid_seal_box(self, boxes):
+ if len(boxes) == 1:
+ return boxes
+
+ new_boxes = []
+ flag = True
+ for k in range(0, len(boxes)):
+ flag = True
+ tmp_box = boxes[k]
+ for i in range(0, len(boxes)):
+ if k == i: continue
+ if abs(Polygon(tmp_box).area - self.get_intersection(tmp_box, boxes[i])) < 1e-3:
+ flag = False
+ continue
+ if flag:
+ new_boxes.append(tmp_box)
+
+ return new_boxes
+
+
+ def get_union(self, pD, pG):
+ return Polygon(pD).union(Polygon(pG)).area
+
+ def get_intersection_over_union(self, pD, pG):
+ return get_intersection(pD, pG) / get_union(pD, pG)
+
+ def get_intersection(self, pD, pG):
+ return Polygon(pD).intersection(Polygon(pG)).area
+
+ def expand_points_num(self, boxes):
+ max_points_num = 0
+ for box in boxes:
+ if len(box) > max_points_num:
+ max_points_num = len(box)
+ ex_boxes = []
+ for box in boxes:
+ ex_box = box + [box[-1]] * (max_points_num - len(box))
+ ex_boxes.append(ex_box)
+ return ex_boxes
+
+
+def gen_extract_label(data_dir, label_file, seal_gt, seal_ppocr_gt):
+ label_decode_func = LabelDecode()
+ gts = open(label_file, "r").readlines()
+
+ seal_gt_list = []
+ seal_ppocr_list = []
+
+ for idx, line in enumerate(gts):
+ img_path, label = line.strip().split("\t")
+ data = {'label': label, 'img_path':img_path}
+ res = label_decode_func(data)
+ src_img = cv2.imread(os.path.join(data_dir, img_path))
+ if res is None:
+ print("ERROR! res is None!")
+ continue
+
+ anno = []
+ for i, gt in enumerate(res):
+ # print(i, box, type(box), )
+ anno.append({'polys': gt['seal_box'], 'cls':1})
+
+ seal_gt_list.append(f"{img_path}\t{json.dumps(anno)}\n")
+ seal_ppocr_list.append(f"{img_path}\t{json.dumps(res)}\n")
+
+ if not os.path.exists(os.path.dirname(seal_gt)):
+ os.makedirs(os.path.dirname(seal_gt))
+ if not os.path.exists(os.path.dirname(seal_ppocr_gt)):
+ os.makedirs(os.path.dirname(seal_ppocr_gt))
+
+ with open(seal_gt, "w") as f:
+ f.writelines(seal_gt_list)
+ f.close()
+
+ with open(seal_ppocr_gt, 'w') as f:
+ f.writelines(seal_ppocr_list)
+ f.close()
+
+def vis_seal_ppocr(data_dir, label_file, save_dir):
+
+ datas = open(label_file, 'r').readlines()
+ for idx, line in enumerate(datas):
+ img_path, label = line.strip().split('\t')
+ img_path = os.path.join(data_dir, img_path)
+
+ label = json.loads(label)
+ src_im = cv2.imread(img_path)
+ if src_im is None:
+ continue
+
+ for anno in label:
+ seal_box = anno['seal_box']
+ txt_boxes = anno['polys']
+
+ # vis seal box
+ src_im = draw_text_det_res([seal_box], src_im, color=(255, 255, 0))
+ src_im = draw_text_det_res(txt_boxes, src_im, color=(255, 0, 0))
+
+ save_path = os.path.join(save_dir, os.path.basename(img_path))
+ if not os.path.exists(save_dir):
+ os.makedirs(save_dir)
+ # print(src_im.shape)
+ cv2.imwrite(save_path, src_im)
+
+
+def draw_html(img_dir, save_name):
+ import glob
+
+ images_dir = glob.glob(img_dir + "/*")
+ print(len(images_dir))
+
+ html_path = save_name
+ with open(html_path, 'w') as html:
+ html.write('\n\n')
+ html.write('\n')
+ html.write("")
+
+ html.write("
\n')
+ html.write('\n\n')
+ print("ok")
+
+
+def crop_seal_from_img(label_file, data_dir, save_dir, save_gt_path):
+
+ if not os.path.exists(save_dir):
+ os.makedirs(save_dir)
+
+ datas = open(label_file, 'r').readlines()
+ all_gts = []
+ count = 0
+ for idx, line in enumerate(datas):
+ img_path, label = line.strip().split('\t')
+ img_path = os.path.join(data_dir, img_path)
+
+ label = json.loads(label)
+ src_im = cv2.imread(img_path)
+ if src_im is None:
+ continue
+
+ for c, anno in enumerate(label):
+ seal_poly = anno['seal_box']
+ txt_boxes = anno['polys']
+ txts = anno['texts']
+ ignore_tags = anno['ignore_tags']
+
+ box = poly2box(seal_poly)
+ img_crop = src_im[box[0][1]:box[2][1], box[0][0]:box[2][0], :]
+
+ save_path = os.path.join(save_dir, f"{idx}_{c}.jpg")
+ cv2.imwrite(save_path, np.array(img_crop))
+
+ img_gt = []
+ for i in range(len(txts)):
+ txt_boxes_crop = np.array(txt_boxes[i])
+ txt_boxes_crop[:, 1] -= box[0, 1]
+ txt_boxes_crop[:, 0] -= box[0, 0]
+ img_gt.append({'transcription': txts[i], "points": txt_boxes_crop.tolist(), "ignore_tag": ignore_tags[i]})
+
+ if len(img_gt) >= 1:
+ count += 1
+ save_gt = f"{os.path.basename(save_path)}\t{json.dumps(img_gt)}\n"
+
+ all_gts.append(save_gt)
+
+ print(f"The num of all image: {len(all_gts)}, and the number of useful image: {count}")
+ if not os.path.exists(os.path.dirname(save_gt_path)):
+ os.makedirs(os.path.dirname(save_gt_path))
+
+ with open(save_gt_path, "w") as f:
+ f.writelines(all_gts)
+ f.close()
+ print("Done")
+
+
+if __name__ == "__main__":
+ # 数据处理
+ gen_extract_label("./seal_labeled_datas", "./seal_labeled_datas/Label.txt", "./seal_ppocr_gt/seal_det_img.txt", "./seal_ppocr_gt/seal_ppocr_img.txt")
+ vis_seal_ppocr("./seal_labeled_datas", "./seal_ppocr_gt/seal_ppocr_img.txt", "./seal_ppocr_gt/seal_ppocr_vis/")
+ draw_html("./seal_ppocr_gt/seal_ppocr_vis/", "./vis_seal_ppocr.html")
+ seal_ppocr_img_label = "./seal_ppocr_gt/seal_ppocr_img.txt"
+ crop_seal_from_img(seal_ppocr_img_label, "./seal_labeled_datas/", "./seal_img_crop", "./seal_img_crop/label.txt")
+```
+
+
+
+处理完成后，生成的文件如下：
+
+```text linenums="1"
+├── seal_img_crop/
+│ ├── 0_0.jpg
+│ ├── ...
+│ └── label.txt
+├── seal_ppocr_gt/
+│ ├── seal_det_img.txt
+│ ├── seal_ppocr_img.txt
+│ └── seal_ppocr_vis/
+│ ├── test1.png
+│ ├── ...
+└── vis_seal_ppocr.html
+
+```
+
+其中`seal_img_crop/label.txt`文件为印章识别标签文件，其内容格式为：
+
+```text linenums="1"
+0_0.jpg [{"transcription": "\u7535\u5b50\u56de\u5355", "points": [[29, 73], [96, 73], [96, 90], [29, 90]], "ignore_tag": false}, {"transcription": "\u4e91\u5357\u7701\u519c\u6751\u4fe1\u7528\u793e", "points": [[9, 58], [26, 63], [30, 49], [38, 35], [47, 29], [64, 26], [81, 32], [90, 45], [94, 63], [118, 57], [110, 35], [95, 17], [67, 0], [38, 7], [21, 23], [10, 43]], "ignore_tag": false}, {"transcription": "\u4e13\u7528\u7ae0", "points": [[29, 87], [95, 87], [95, 106], [29, 106]], "ignore_tag": false}]
+```
+
+可以直接用于PaddleOCR的PGNet算法的训练。
+
+`seal_ppocr_gt/seal_det_img.txt`为印章检测标签文件，其内容格式为：
+
+```text linenums="1"
+img/test1.png [{"polys": [[408, 232], [537, 232], [537, 352], [408, 352]], "cls": 1}]
+```
+
+为了使用PaddleDetection工具完成印章检测模型的训练，需要将`seal_det_img.txt`转换为COCO或者VOC的数据标注格式。
+
+可以直接使用下述代码将印章检测标注转换成VOC格式。
+
+| \n GT') + + for i, filename in enumerate(sorted(images_dir)): + if filename.endswith("txt"): continue + print(filename) + + base = "{}".format(filename) + if True: + html.write(" | |
| {filename}\n GT') + html.write(' | GT 310\n | ' % (base))
+ html.write("
+
+```python linenums="1"
+import numpy as np
+import json
+import cv2
+import os
+from shapely.geometry import Polygon
+
+seal_train_gt = "./seal_ppocr_gt/seal_det_img.txt"
+# 注：仅用于示例，实际使用中需要分别转换训练集和测试集的标签
+seal_valid_gt = "./seal_ppocr_gt/seal_det_img.txt"
+
+def gen_main_train_txt(mode='train'):
+ if mode == "train":
+ file_path = seal_train_gt
+ if mode in ['valid', 'test']:
+ file_path = seal_valid_gt
+
+ save_path = f"./seal_VOC/ImageSets/Main/{mode}.txt"
+ save_train_path = f"./seal_VOC/{mode}.txt"
+ if not os.path.exists(os.path.dirname(save_path)):
+ os.makedirs(os.path.dirname(save_path))
+
+ datas = open(file_path, 'r').readlines()
+ img_names = []
+ train_names = []
+ for line in datas:
+ img_name = line.strip().split('\t')[0]
+ img_name = os.path.basename(img_name)
+ (i_name, extension) = os.path.splitext(img_name)
+ t_name = 'JPEGImages/'+str(img_name)+' '+'Annotations/'+str(i_name)+'.xml\n'
+ train_names.append(t_name)
+ img_names.append(i_name + "\n")
+
+ with open(save_train_path, "w") as f:
+ f.writelines(train_names)
+ f.close()
+
+ with open(save_path, "w") as f:
+ f.writelines(img_names)
+ f.close()
+
+ print(f"{mode} save done")
+
+
+def gen_xml_label(mode='train'):
+ if mode == "train":
+ file_path = seal_train_gt
+ if mode in ['valid', 'test']:
+ file_path = seal_valid_gt
+
+ datas = open(file_path, 'r').readlines()
+ img_names = []
+ train_names = []
+ anno_path = "./seal_VOC/Annotations"
+ img_path = "./seal_VOC/JPEGImages"
+
+ if not os.path.exists(anno_path):
+ os.makedirs(anno_path)
+ if not os.path.exists(img_path):
+ os.makedirs(img_path)
+
+ for idx, line in enumerate(datas):
+ img_name, label = line.strip().split('\t')
+ img = cv2.imread(os.path.join("./seal_labeled_datas", img_name))
+ cv2.imwrite(os.path.join(img_path, os.path.basename(img_name)), img)
+ height, width, c = img.shape
+ img_name = os.path.basename(img_name)
+ (i_name, extension) = os.path.splitext(img_name)
+ label = json.loads(label)
+
+ xml_file = open(("./seal_VOC/Annotations" + '/' + i_name + '.xml'), 'w')
+ xml_file.write('\n')
+ xml_file.write(' seal_VOC \n')
+ xml_file.write(' ' + str(img_name) + ' \n')
+ xml_file.write(' ' + 'Annotations/' + str(img_name) + ' \n')
+ xml_file.write(' \n')
+ xml_file.write(' ' + str(width) + ' \n')
+ xml_file.write(' ' + str(height) + ' \n')
+ xml_file.write(' 3 \n')
+ xml_file.write(' \n')
+ xml_file.write(' 0 \n')
+
+ for anno in label:
+ poly = anno['polys']
+ if anno['cls'] == 1:
+ gt_cls = 'redseal'
+ xmin = np.min(np.array(poly)[:, 0])
+ ymin = np.min(np.array(poly)[:, 1])
+ xmax = np.max(np.array(poly)[:, 0])
+ ymax = np.max(np.array(poly)[:, 1])
+ xmin,ymin,xmax,ymax= int(xmin),int(ymin),int(xmax),int(ymax)
+ xml_file.write(' \n')
+ xml_file.write(' ')
+ xml_file.close()
+ print(f'{mode} xml save done!')
+
+
+gen_main_train_txt()
+gen_main_train_txt('valid')
+gen_xml_label('train')
+gen_xml_label('valid')
+
+```
+
+
+
+数据处理完成后，转换为VOC格式的印章检测数据存储在~/data/seal_VOC目录下，目录组织结构为：
+
+```text linenums="1"
+├── Annotations/
+├── ImageSets/
+│ └── Main/
+│ ├── train.txt
+│ └── valid.txt
+├── JPEGImages/
+├── train.txt
+└── valid.txt
+└── label_list.txt
+```
+
+Annotations下为数据的标签，JPEGImages目录下为图像文件，label_list.txt为标注检测框类别标签文件。
+
+在接下来一节中，将介绍如何使用PaddleDetection工具库完成印章检测模型的训练。
+
+## 4. 印章检测实践
+
+在实际应用中，印章多是出现在合同，发票，公告等场景中，印章文字识别的任务需要排除图像中背景文字的影响，因此需要先检测出图像中的印章区域。
+
+借助PaddleDetection目标检测库可以很容易的实现印章检测任务，使用PaddleDetection训练印章检测任务流程如下：
+
+- 选择算法
+- 修改数据集配置路径
+- 启动训练
+
+**算法选择**
+
+PaddleDetection中有许多检测算法可以选择，考虑到每条数据中印章区域较为清晰，且考虑到性能需求。在本项目中，我们采用mobilenetv3为backbone的ppyolo算法完成印章检测任务，对应的配置文件是：configs/ppyolo/ppyolo_mbv3_large.yml
+
+**修改配置文件**
+
+配置文件中的默认数据路径是COCO，
+需要修改为印章检测的数据路径，主要修改如下:
+在配置文件'configs/ppyolo/ppyolo_mbv3_large.yml'末尾增加如下内容：
+
+```yaml linenums="1"
+metric: VOC
+map_type: 11point
+num_classes: 2
+
+TrainDataset:
+ !VOCDataSet
+ dataset_dir: dataset/seal_VOC
+ anno_path: train.txt
+ label_list: label_list.txt
+ data_fields: ['image', 'gt_bbox', 'gt_class', 'difficult']
+
+EvalDataset:
+ !VOCDataSet
+ dataset_dir: dataset/seal_VOC
+ anno_path: test.txt
+ label_list: label_list.txt
+ data_fields: ['image', 'gt_bbox', 'gt_class', 'difficult']
+
+TestDataset:
+ !ImageFolder
+ anno_path: dataset/seal_VOC/label_list.txt
+```
+
+配置文件中设置的数据路径在PaddleDetection/dataset目录下，我们可以将处理后的印章检测训练数据移动到PaddleDetection/dataset目录下或者创建一个软连接。
+
+```bash linenums="1"
+!ln -s seal_VOC ./PaddleDetection/dataset/
+```
+
+另外图象中印章数量比较少，可以调整NMS后处理的检测框数量，即keep_top_k，nms_top_k 从100，1000，调整为10，100。在配置文件'configs/ppyolo/ppyolo_mbv3_large.yml'末尾增加如下内容完成后处理参数的调整
+
+```yaml linenums="1"
+BBoxPostProcess:
+ decode:
+ name: YOLOBox
+ conf_thresh: 0.005
+ downsample_ratio: 32
+ clip_bbox: true
+ scale_x_y: 1.05
+ nms:
+ name: MultiClassNMS
+ keep_top_k: 10 # 修改前100
+ nms_threshold: 0.45
+ nms_top_k: 100 # 修改前1000
+ score_threshold: 0.005
+```
+
+修改完成后，需要在PaddleDetection中增加印章数据的处理代码，即在PaddleDetection/ppdet/data/source/目录下创建seal.py文件，文件中填充如下代码：
+
+
+
+```python linenums="1"
+import os
+import numpy as np
+from ppdet.core.workspace import register, serializable
+from .dataset import DetDataset
+import cv2
+import json
+
+from ppdet.utils.logger import setup_logger
+logger = setup_logger(__name__)
+
+
+@register
+@serializable
+class SealDataSet(DetDataset):
+ """
+ Load dataset with COCO format.
+
+ Args:
+ dataset_dir (str): root directory for dataset.
+ image_dir (str): directory for images.
+ anno_path (str): coco annotation file path.
+ data_fields (list): key name of data dictionary, at least have 'image'.
+ sample_num (int): number of samples to load, -1 means all.
+ load_crowd (bool): whether to load crowded ground-truth.
+ False as default
+ allow_empty (bool): whether to load empty entry. False as default
+ empty_ratio (float): the ratio of empty record number to total
+ record's, if empty_ratio is out of [0. ,1.), do not sample the
+ records and use all the empty entries. 1. as default
+ """
+
+ def __init__(self,
+ dataset_dir=None,
+ image_dir=None,
+ anno_path=None,
+ data_fields=['image'],
+ sample_num=-1,
+ load_crowd=False,
+ allow_empty=False,
+ empty_ratio=1.):
+ super(SealDataSet, self).__init__(dataset_dir, image_dir, anno_path,
+ data_fields, sample_num)
+ self.load_image_only = False
+ self.load_semantic = False
+ self.load_crowd = load_crowd
+ self.allow_empty = allow_empty
+ self.empty_ratio = empty_ratio
+
+ def _sample_empty(self, records, num):
+ # if empty_ratio is out of [0. ,1.), do not sample the records
+ if self.empty_ratio < 0. or self.empty_ratio >= 1.:
+ return records
+ import random
+ sample_num = min(
+ int(num * self.empty_ratio / (1 - self.empty_ratio)), len(records))
+ records = random.sample(records, sample_num)
+ return records
+
+ def parse_dataset(self):
+ anno_path = os.path.join(self.dataset_dir, self.anno_path)
+ image_dir = os.path.join(self.dataset_dir, self.image_dir)
+
+ records = []
+ empty_records = []
+ ct = 0
+
+ assert anno_path.endswith('.txt'), \
+ 'invalid seal_gt file: ' + anno_path
+
+ all_datas = open(anno_path, 'r').readlines()
+
+ for idx, line in enumerate(all_datas):
+ im_path, label = line.strip().split('\t')
+ img_path = os.path.join(image_dir, im_path)
+ label = json.loads(label)
+ im_h, im_w, im_c = cv2.imread(img_path).shape
+
+ coco_rec = {
+ 'im_file': img_path,
+ 'im_id': np.array([idx]),
+ 'h': im_h,
+ 'w': im_w,
+ } if 'image' in self.data_fields else {}
+
+ if not self.load_image_only:
+ bboxes = []
+ for anno in label:
+ poly = anno['polys']
+ # poly to box
+ x1 = np.min(np.array(poly)[:, 0])
+ y1 = np.min(np.array(poly)[:, 1])
+ x2 = np.max(np.array(poly)[:, 0])
+ y2 = np.max(np.array(poly)[:, 1])
+ eps = 1e-5
+ if x2 - x1 > eps and y2 - y1 > eps:
+ clean_box = [
+ round(float(x), 3) for x in [x1, y1, x2, y2]
+ ]
+ anno = {'clean_box': clean_box, 'gt_cls':int(anno['cls'])}
+ bboxes.append(anno)
+ else:
+ logger.info("invalid box")
+
+ num_bbox = len(bboxes)
+ if num_bbox <= 0:
+ continue
+
+ gt_bbox = np.zeros((num_bbox, 4), dtype=np.float32)
+ gt_class = np.zeros((num_bbox, 1), dtype=np.int32)
+ is_crowd = np.zeros((num_bbox, 1), dtype=np.int32)
+ # gt_poly = [None] * num_bbox
+
+ for i, box in enumerate(bboxes):
+ gt_class[i][0] = box['gt_cls']
+ gt_bbox[i, :] = box['clean_box']
+ is_crowd[i][0] = 0
+
+ gt_rec = {
+ 'is_crowd': is_crowd,
+ 'gt_class': gt_class,
+ 'gt_bbox': gt_bbox,
+ # 'gt_poly': gt_poly,
+ }
+
+ for k, v in gt_rec.items():
+ if k in self.data_fields:
+ coco_rec[k] = v
+
+ records.append(coco_rec)
+ ct += 1
+ if self.sample_num > 0 and ct >= self.sample_num:
+ break
+ self.roidbs = records
+```
+
+
+
+**启动训练**
+
+启动单卡训练的命令为：
+
+```bash linenums="1"
+!python3 tools/train.py -c configs/ppyolo/ppyolo_mbv3_large.yml --eval
+
+# 分布式训练命令为：
+!python3 -m paddle.distributed.launch --gpus 0,1,2,3,4,5,6,7 tools/train.py -c configs/ppyolo/ppyolo_mbv3_large.yml --eval
+```
+
+训练完成后，日志中会打印模型的精度：
+
+```bash linenums="1"
+[07/05 11:42:09] ppdet.engine INFO: Eval iter: 0
+[07/05 11:42:14] ppdet.metrics.metrics INFO: Accumulating evaluatation results...
+[07/05 11:42:14] ppdet.metrics.metrics INFO: mAP(0.50, 11point) = 99.31%
+[07/05 11:42:14] ppdet.engine INFO: Total sample number: 112, averge FPS: 26.45840794253432
+[07/05 11:42:14] ppdet.engine INFO: Best test bbox ap is 0.996.
+```
+
+我们可以使用训练好的模型观察预测结果：
+
+```bash linenums="1"
+!python3 tools/infer.py -c configs/ppyolo/ppyolo_mbv3_large.yml -o weights=./output/ppyolo_mbv3_large/model_final.pdparams --img_dir=./test.jpg
+```
+
+预测结果如下：
+
+
+
+## 5. 印章文字识别实践
+
+在使用ppyolo检测到印章区域后，接下来借助PaddleOCR里的文字识别能力，完成印章中文字的识别。
+
+PaddleOCR中的OCR算法包含文字检测算法，文字识别算法以及OCR端对端算法。
+
+文字检测算法负责检测到图像中的文字，再由文字识别模型识别出检测到的文字，进而实现OCR的任务。文字检测+文字识别串联完成OCR任务的架构称为两阶段的OCR算法。相对应的端对端的OCR方法可以用一个算法同时完成文字检测和识别的任务。
+
+| 文字检测 | 文字识别 | 端对端算法 |
+| -------- | -------- | -------- |
+| DB\DB++\EAST\SAST\PSENet | SVTR\CRNN\NRTN\Abinet\SAR\... | PGNet |
+
+本节中将分别介绍端对端的文字检测识别算法以及两阶段的文字检测识别算法在印章检测识别任务上的实践。
+
+### 5.1 端对端印章文字识别实践
+
+本节介绍使用PaddleOCR里的PGNet算法完成印章文字识别。
+
+PGNet属于端对端的文字检测识别算法，在PaddleOCR中的配置文件为：
+[PaddleOCR/configs/e2e/e2e_r50_vd_pg.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.6/configs/e2e/e2e_r50_vd_pg.yml)
+
+使用PGNet完成文字检测识别任务的步骤为：
+
+- 修改配置文件
+- 启动训练
+
+PGNet默认配置文件的数据路径为totaltext数据集路径，本次训练中，需要修改为上一节数据处理后得到的标签文件和数据目录：
+
+训练数据配置修改后如下：
+
+```yaml linenums="1"
+Train:
+ dataset:
+ name: PGDataSet
+ data_dir: ./train_data/seal_ppocr
+ label_file_list: [./train_data/seal_ppocr/seal_ppocr_img.txt]
+ ratio_list: [1.0]
+```
+
+测试数据集配置修改后如下：
+
+```yaml linenums="1"
+Eval:
+ dataset:
+ name: PGDataSet
+ data_dir: ./train_data/seal_ppocr_test
+ label_file_list: [./train_data/seal_ppocr_test/seal_ppocr_img.txt]
+```
+
+启动训练的命令为:
+
+```bash linenums="1"
+!python3 tools/train.py -c configs/e2e/e2e_r50_vd_pg.yml
+```
+
+模型训练完成后，可以得到最终的精度为47.4%。数据量较少，以及数据质量较差会影响模型的训练精度，如果有更多的数据参与训练，精度将进一步提升。
+
+如需获取已训练模型，请点击文末的链接，加入官方交流群获取全部OCR垂类模型下载链接、《动手学OCR》电子书等全套OCR学习资料🎁
+
+### 5.2 两阶段印章文字识别实践
+
+上一节介绍了使用PGNet实现印章识别任务的训练流程。本小节将介绍使用PaddleOCR里的文字检测和文字识别算法分别完成印章文字的检测和识别。
+
+#### 5.2.1 印章文字检测
+
+PaddleOCR中包含丰富的文字检测算法，包含DB，DB++，EAST，SAST，PSENet等等。其中DB，DB++，PSENet均支持弯曲文字检测，本项目中，使用DB++作为印章弯曲文字检测算法。
+
+PaddleOCR中发布的db++文字检测算法模型是英文文本检测模型，因此需要重新训练模型。
+
+修改[DB++配置文件](DB++的默认配置文件位于[configs/det/det_r50_db++_icdar15.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.6/configs/det/det_r50_db%2B%2B_icdar15.yml)
+中的数据路径：
+
+```yaml linenums="1"
+Train:
+ dataset:
+ name: SimpleDataSet
+ data_dir: ./train_data/seal_ppocr
+ label_file_list: [./train_data/seal_ppocr/seal_ppocr_img.txt]
+ ratio_list: [1.0]
+```
+
+测试数据集配置修改后如下：
+
+```yaml linenums="1"
+Eval:
+ dataset:
+ name: SimpleDataSet
+ data_dir: ./train_data/seal_ppocr_test
+ label_file_list: [./train_data/seal_ppocr_test/seal_ppocr_img.txt]
+```
+
+启动训练：
+
+```bash linenums="1"
+!python3 tools/train.py -c configs/det/det_r50_db++_icdar15.yml -o Global.epoch_num=100
+```
+
+考虑到数据较少，通过Global.epoch_num设置仅训练100个epoch。
+模型训练完成后，在测试集上预测的可视化效果如下：
+
+
+
+如需获取已训练模型，请点击文末的链接，加入官方交流群获取全部OCR垂类模型下载链接、《动手学OCR》电子书等全套OCR学习资料🎁
+
+#### 5.2.2 印章文字识别
+
+上一节中完成了印章文字的检测模型训练，本节介绍印章文字识别模型的训练。识别模型采用SVTR算法，SVTR算法是IJCAI收录的文字识别算法，SVTR模型具备超轻量高精度的特点。
+
+在启动训练之前，需要准备印章文字识别需要的数据集，需要使用如下代码，将印章中的文字区域剪切出来构建训练集。
+
+```python linenums="1"
+import cv2
+import numpy as np
+
+def get_rotate_crop_image(img, points):
+ '''
+ img_height, img_width = img.shape[0:2]
+ left = int(np.min(points[:, 0]))
+ right = int(np.max(points[:, 0]))
+ top = int(np.min(points[:, 1]))
+ bottom = int(np.max(points[:, 1]))
+ img_crop = img[top:bottom, left:right, :].copy()
+ points[:, 0] = points[:, 0] - left
+ points[:, 1] = points[:, 1] - top
+ '''
+ assert len(points) == 4, "shape of points must be 4*2"
+ img_crop_width = int(
+ max(
+ np.linalg.norm(points[0] - points[1]),
+ np.linalg.norm(points[2] - points[3])))
+ img_crop_height = int(
+ max(
+ np.linalg.norm(points[0] - points[3]),
+ np.linalg.norm(points[1] - points[2])))
+ pts_std = np.float32([[0, 0], [img_crop_width, 0],
+ [img_crop_width, img_crop_height],
+ [0, img_crop_height]])
+ M = cv2.getPerspectiveTransform(points, pts_std)
+ dst_img = cv2.warpPerspective(
+ img,
+ M, (img_crop_width, img_crop_height),
+ borderMode=cv2.BORDER_REPLICATE,
+ flags=cv2.INTER_CUBIC)
+ dst_img_height, dst_img_width = dst_img.shape[0:2]
+ if dst_img_height * 1.0 / dst_img_width >= 1.5:
+ dst_img = np.rot90(dst_img)
+ return dst_img
+
+
+def run(data_dir, label_file, save_dir):
+ datas = open(label_file, 'r').readlines()
+ for idx, line in enumerate(datas):
+ img_path, label = line.strip().split('\t')
+ img_path = os.path.join(data_dir, img_path)
+
+ label = json.loads(label)
+ src_im = cv2.imread(img_path)
+ if src_im is None:
+ continue
+
+ for anno in label:
+ seal_box = anno['seal_box']
+ txt_boxes = anno['polys']
+ crop_im = get_rotate_crop_image(src_im, text_boxes)
+
+ save_path = os.path.join(save_dir, f'{idx}.png')
+ if not os.path.exists(save_dir):
+ os.makedirs(save_dir)
+ # print(src_im.shape)
+ cv2.imwrite(save_path, crop_im)
+
+```
+
+数据处理完成后，即可配置训练的配置文件。SVTR配置文件选择[configs/rec/PP-OCRv3/ch_PP-OCRv3_rec.yml](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.6/configs/rec/PP-OCRv3/ch_PP-OCRv3_rec.yml)
+修改SVTR配置文件中的训练数据部分如下：
+
+```yaml linenums="1"
+Train:
+ dataset:
+ name: SimpleDataSet
+ data_dir: ./train_data/seal_ppocr_crop/
+ label_file_list:
+ - ./train_data/seal_ppocr_crop/train_list.txt
+```
+
+修改预测部分配置文件：
+
+```yaml linenums="1"
+Train:
+ dataset:
+ name: SimpleDataSet
+ data_dir: ./train_data/seal_ppocr_crop/
+ label_file_list:
+ - ./train_data/seal_ppocr_crop_test/train_list.txt
+```
+
+启动训练：
+
+```bash linenums="1"
+!python3 tools/train.py -c configs/rec/PP-OCRv3/ch_PP-OCRv3_rec.yml
+
+```
+
+训练完成后可以发现测试集指标达到了61%。
+由于数据较少，训练时会发现在训练集上的acc指标远大于测试集上的acc指标，即出现过拟合现象。通过补充数据和一些数据增强可以缓解这个问题。
+
+如需获取已训练模型，请加入PaddleX官方交流频道，获取20G OCR学习大礼包（内含《动手学OCR》电子书、课程回放视频、前沿论文等重磅资料）
+
+- PaddleX官方交流频道：
+
+将下载或训练完成的模型放置在对应目录下即可完成模型推理。
+
+### 5.4 模型导出推理
+
+训练完成后，可以将训练模型转换成inference模型。inference 模型会额外保存模型的结构信息，在预测部署、加速推理上性能优越，灵活方便，适合于实际系统集成。
+
+#### 模型导出
+
+导出命令如下：
+
+```bash linenums="1"
+# 转化为推理模型
+python tools/export_model.py -c configs/rec/PP-OCRv3/ch_PP-OCRv3_rec_distillation.yml -o Global.pretrained_model="./output/ch_PP-OCR_v3_rec/best_accuracy" Global.save_inference_dir="./inference/rec_ppocrv3/"
+```
+
+#### 模型推理
+
+导出模型后，可以使用如下命令进行推理预测
+
+```bash linenums="1"
+# 推理预测
+python tools/infer/predict_rec.py --image_dir="train_data/ic15_data/test/1_crop_0.jpg" --rec_model_dir="./inference/rec_ppocrv3/Student"
+```
+
+## 6. 系统串联
+
+我们将上面训练好的检测和识别模型进行系统串联测试，命令如下：
+
+```bash linenums="1"
+#串联测试
+python3 tools/infer/predict_system.py --image_dir="./train_data/icdar2015/text_localization/test/142.jpg" --det_model_dir="./inference/det_ppocrv3/Student" --rec_model_dir="./inference/rec_ppocrv3/Student"
+```
+
+测试结果保存在`./inference_results/`目录下，可以用下面代码进行可视化
+
+```bash linenums="1"
+%cd /home/aistudio/PaddleOCR
+# 显示结果
+import matplotlib.pyplot as plt
+from PIL import Image
+img_path= "./inference_results/142.jpg"
+img = Image.open(img_path)
+plt.figure("test_img", figsize=(30,30))
+plt.imshow(img)
+plt.show()
+```
+
+
+
+### 6.1 后处理
+
+如果需要获取key-value信息，可以基于启发式的规则，将识别结果与关键字库进行匹配；如果匹配上了，则取该字段为key, 后面一个字段为value。
+
+```python linenums="1"
+def postprocess(rec_res):
+ keys = ["型号", "厂家", "版本号", "检定校准分类", "计量器具编号", "烟尘流量",
+ "累积体积", "烟气温度", "动压", "静压", "时间", "试验台编号", "预测流速",
+ "全压", "烟温", "流速", "工况流量", "标杆流量", "烟尘直读嘴", "烟尘采样嘴",
+ "大气压", "计前温度", "计前压力", "干球温度", "湿球温度", "流量", "含湿量"]
+ key_value = []
+ if len(rec_res) > 1:
+ for i in range(len(rec_res) - 1):
+ rec_str, _ = rec_res[i]
+ for key in keys:
+ if rec_str in key:
+ key_value.append([rec_str, rec_res[i + 1][0]])
+ break
+ return key_value
+key_value = postprocess(filter_rec_res)
+```
+
+## 7. PaddleServing部署
+
+首先需要安装PaddleServing部署相关的环境
+
+```bash linenums="1"
+python -m pip install paddle-serving-server-gpu
+python -m pip install paddle_serving_client
+python -m pip install paddle-serving-app
+```
+
+### 7.1 转化检测模型
+
+```bash linenums="1"
+cd deploy/pdserving/
+python -m paddle_serving_client.convert --dirname ../../inference/det_ppocrv3/Student/ \
+ --model_filename inference.pdmodel \
+ --params_filename inference.pdiparams \
+ --serving_server ./ppocr_det_v3_serving/ \
+ --serving_client ./ppocr_det_v3_client/
+```
+
+### 7.2 转化识别模型
+
+```bash linenums="1"
+python -m paddle_serving_client.convert --dirname ../../inference/rec_ppocrv3/Student \
+ --model_filename inference.pdmodel \
+ --params_filename inference.pdiparams \
+ --serving_server ./ppocr_rec_v3_serving/ \
+ --serving_client ./ppocr_rec_v3_client/
+```
+
+### 7.3 启动服务
+
+首先可以将后处理代码加入到web_service.py中，具体修改如下：
+
+```python linenums="1"
+# 代码153行后面增加下面代码
+def _postprocess(rec_res):
+ keys = ["型号", "厂家", "版本号", "检定校准分类", "计量器具编号", "烟尘流量",
+ "累积体积", "烟气温度", "动压", "静压", "时间", "试验台编号", "预测流速",
+ "全压", "烟温", "流速", "工况流量", "标杆流量", "烟尘直读嘴", "烟尘采样嘴",
+ "大气压", "计前温度", "计前压力", "干球温度", "湿球温度", "流量", "含湿量"]
+ key_value = []
+ if len(rec_res) > 1:
+ for i in range(len(rec_res) - 1):
+ rec_str, _ = rec_res[i]
+ for key in keys:
+ if rec_str in key:
+ key_value.append([rec_str, rec_res[i + 1][0]])
+ break
+ return key_value
+key_value = _postprocess(rec_list)
+res = {"result": str(key_value)}
+# res = {"result": str(result_list)}
+```
+
+启动服务端
+
+```bash linenums="1"
+python web_service.py 2>&1 >log.txt
+```
+
+### 7.4 发送请求
+
+然后再开启一个新的终端，运行下面的客户端代码
+
+```bash linenums="1"
+python pipeline_http_client.py --image_dir ../../train_data/icdar2015/text_localization/test/142.jpg
+```
+
+可以获取到最终的key-value结果：
+
+```text linenums="1"
+大气压, 100.07kPa
+干球温度, 0000℃
+计前温度, 0000℃
+湿球温度, 0000℃
+计前压力, -0000kPa
+流量, 00.0L/min
+静压, 00000kPa
+含湿量, 00.0 %
+```
diff --git "a/docs/applications/\350\275\273\351\207\217\347\272\247\350\275\246\347\211\214\350\257\206\345\210\253.md" "b/docs/applications/\350\275\273\351\207\217\347\272\247\350\275\246\347\211\214\350\257\206\345\210\253.md"
new file mode 100644
index 0000000000..de15aee6ff
--- /dev/null
+++ "b/docs/applications/\350\275\273\351\207\217\347\272\247\350\275\246\347\211\214\350\257\206\345\210\253.md"
@@ -0,0 +1,815 @@
+---
+typora-copy-images-to: images
+comments: true
+---
+
+
+# 一种基于PaddleOCR的轻量级车牌识别模型
+
+## 1. 项目介绍
+
+车牌识别(Vehicle License Plate Recognition，VLPR) 是计算机视频图像识别技术在车辆牌照识别中的一种应用。车牌识别技术要求能够将运动中的汽车牌照从复杂背景中提取并识别出来，在高速公路车辆管理，停车场管理和城市交通中得到广泛应用。
+
+本项目难点如下：
+
+1. 车牌在图像中的尺度差异大、在车辆上的悬挂位置不固定
+2. 车牌图像质量层次不齐: 角度倾斜、图片模糊、光照不足、过曝等问题严重
+3. 边缘和端测场景应用对模型大小有限制，推理速度有要求
+
+针对以上问题， 本例选用 PP-OCRv3 这一开源超轻量OCR系统进行车牌识别系统的开发。基于PP-OCRv3模型，在CCPD数据集达到99%的检测和94%的识别精度，模型大小12.8M(2.5M+10.3M)。基于量化对模型体积进行进一步压缩到5.8M(1M+4.8M), 同时推理速度提升25%。
+
+aistudio项目链接: [基于PaddleOCR的轻量级车牌识别范例](https://aistudio.baidu.com/aistudio/projectdetail/3919091?contributionType=1)
+
+## 2. 环境搭建
+
+本任务基于Aistudio完成, 具体环境如下：
+
+- 操作系统: Linux
+- PaddlePaddle: 2.3
+- paddleslim: 2.2.2
+- PaddleOCR: Release/2.5
+
+下载 PaddleOCR代码
+
+```bash linenums="1"
+git clone -b dygraph https://github.com/PaddlePaddle/PaddleOCR
+```
+
+安装依赖库
+
+```bash linenums="1"
+pip install -r PaddleOCR/requirements.txt
+```
+
+## 3. 数据集准备
+
+所使用的数据集为 CCPD2020 新能源车牌数据集，该数据集为
+
+该数据集分布如下：
+
+|数据集类型|数量|
+|---|---|
+|训练集| 5769|
+|验证集| 1001|
+|测试集| 5006|
+
+数据集图片示例如下:
+
+
+
+数据集可以从这里下载 PP-OCRv3中英文超轻量识别预训练模型| 0.04% | +|PP-OCRv3中英文超轻量检测预训练模型
PP-OCRv3中英文超轻量识别预训练模型 + 后处理去掉多识别的`·`| 78.27% | +|PP-OCRv3中英文超轻量检测预训练模型+fine-tune
PP-OCRv3中英文超轻量识别预训练模型+fine-tune| 87.14% | +|PP-OCRv3中英文超轻量检测预训练模型+fine-tune+量化
PP-OCRv3中英文超轻量识别预训练模型+fine-tune+量化| 88.00% | + +从结果中可以看到对预训练模型不做修改，只根据场景下的具体情况进行后处理的修改就能大幅提升端到端指标到78.27%，在CCPD数据集上进行 fine-tune 后指标进一步提升到87.14%, 在经过量化训练之后，由于检测模型的recall变高，指标进一步提升到88%。但是这个结果仍旧不符合检测模型+识别模型的真实性能(99%*94%=93%)，因此我们需要对 base case 进行具体分析。 + +在之前的端到端预测结果中，可以看到很多不符合车牌标注的文字被识别出来, 因此可以进行简单的过滤来提升precision + +为了快速评估，我们在 `tools/end2end/convert_ppocr_label.py` 脚本的 58 行加入如下代码，对非8个字符的结果进行过滤 + +```python linenums="1" +if len(txt) != 8: # 车牌字符串长度为8 + continue +``` + +此外，通过可视化box可以发现有很多框都是竖直翻转之后的框，并且没有完全框住车牌边界，因此需要进行框的竖直翻转以及轻微扩大，示意图如下： + + + +修改前后个方案指标对比如下： + +各个方案端到端指标如下： + +|模型|base|A:识别结果过滤|B:use_dilation|C:flip_box|best| +|---|---|---|---|---|---| +|PP-OCRv3中英文超轻量检测预训练模型
PP-OCRv3中英文超轻量识别预训练模型|0.04%|0.08%|0.02%|0.05%|0.00%(A)| +|PP-OCRv3中英文超轻量检测预训练模型
PP-OCRv3中英文超轻量识别预训练模型 + 后处理去掉多识别的`·`|78.27%|90.84%|78.61%|79.43%|91.66%(A+B+C)| +|PP-OCRv3中英文超轻量检测预训练模型+fine-tune
PP-OCRv3中英文超轻量识别预训练模型+fine-tune|87.14%|90.40%|87.66%|89.98%|92.50%(A+B+C)| +|PP-OCRv3中英文超轻量检测预训练模型+fine-tune+量化
PP-OCRv3中英文超轻量识别预训练模型+fine-tune+量化|88.00%|90.54%|88.50%|89.46%|92.02%(A+B+C)| + +从结果中可以看到对预训练模型不做修改，只根据场景下的具体情况进行后处理的修改就能大幅提升端到端指标到91.66%，在CCPD数据集上进行 fine-tune 后指标进一步提升到92.5%, 在经过量化训练之后，指标变为92.02%。 + +### 4.4 部署 + +#### 基于 Paddle Inference 的python推理 + +检测模型和识别模型分别 fine-tune 并导出为inference模型之后，可以使用如下命令基于 Paddle Inference 进行端到端推理并对结果进行可视化。 + +```bash linenums="1" +python tools/infer/predict_system.py \ + --det_model_dir=output/CCPD/det/infer/ \ + --rec_model_dir=output/CCPD/rec/infer/ \ + --image_dir="/home/aistudio/data/CCPD2020/ccpd_green/test/04131106321839081-92_258-159&509_530&611-527&611_172&599_159&509_530&525-0_0_3_32_30_31_30_30-109-106.jpg" \ + --rec_image_shape=3,48,320 +``` + +推理结果如下 + + + +#### 端侧部署 + +端侧部署我们采用基于 PaddleLite 的 cpp 推理。Paddle Lite是飞桨轻量化推理引擎，为手机、IOT端提供高效推理能力，并广泛整合跨平台硬件，为端侧部署及应用落地问题提供轻量化的部署方案。具体可参考 [PaddleOCR lite教程](../ppocr/infer_deploy/lite.md) + +### 4.5 实验总结 + +我们分别使用PP-OCRv3中英文超轻量预训练模型在车牌数据集上进行了直接评估和 fine-tune 和 fine-tune +量化3种方案的实验，并基于[PaddleOCR lite教程](../ppocr/infer_deploy/lite.md)进行了速度测试，指标对比如下： + +- 检测 + +|方案|hmeans| 模型大小 | 预测速度(lite) | +|---|---|------|------------| +|PP-OCRv3中英文超轻量检测预训练模型直接预测|76.12%|2.5M| 233ms | +|PP-OCRv3中英文超轻量检测预训练模型 fine-tune|99.00%| 2.5M | 233ms | +|PP-OCRv3中英文超轻量检测预训练模型 fine-tune + 量化|98.91%| 1.0M | 189ms | + +- 识别 + +|方案| acc | 模型大小 | 预测速度(lite) | +|---|--------|-------|------------| +|PP-OCRv3中英文超轻量识别预训练模型直接预测| 0.00% |10.3M| 4.2ms | +|PP-OCRv3中英文超轻量识别预训练模型直接预测+后处理去掉多识别的`·`| 90.97% |10.3M| 4.2ms | +|PP-OCRv3中英文超轻量识别预训练模型 fine-tune| 94.54% | 10.3M | 4.2ms | +|PP-OCRv3中英文超轻量识别预训练模型 fine-tune + 量化| 93.40% | 4.8M | 1.8ms | + +- 端到端指标如下： + +|方案|fmeasure|模型大小|预测速度(lite) | +|---|---|---|---| +|PP-OCRv3中英文超轻量检测预训练模型
PP-OCRv3中英文超轻量识别预训练模型|0.08%|12.8M|298ms| +|PP-OCRv3中英文超轻量检测预训练模型
PP-OCRv3中英文超轻量识别预训练模型 + 后处理去掉多识别的`·`|91.66%|12.8M|298ms| +|PP-OCRv3中英文超轻量检测预训练模型+fine-tune
PP-OCRv3中英文超轻量识别预训练模型+fine-tune|92.50%|12.8M|298ms| +|PP-OCRv3中英文超轻量检测预训练模型+fine-tune+量化
PP-OCRv3中英文超轻量识别预训练模型+fine-tune+量化|92.02%|5.80M|224ms| + +## **结论** + +PP-OCRv3的检测模型在未经过fine-tune的情况下，在车牌数据集上也有一定的精度，经过 fine-tune 后能够极大的提升检测效果，精度达到99%。在使用量化训练后检测模型的精度几乎无损，并且模型大小压缩60%。 + +PP-OCRv3的识别模型在未经过fine-tune的情况下，在车牌数据集上精度为0，但是经过分析可以知道，模型大部分字符都预测正确，但是会多预测一个特殊字符，去掉这个特殊字符后，精度达到90%。PP-OCRv3识别模型在经过 fine-tune 后识别精度进一步提升，达到94.4%。在使用量化训练后识别模型大小压缩53%，但是由于数据量多少，带来了1%的精度损失。 + +从端到端结果中可以看到对预训练模型不做修改，只根据场景下的具体情况进行后处理的修改就能大幅提升端到端指标到91.66%，在CCPD数据集上进行 fine-tune 后指标进一步提升到92.5%, 在经过量化训练之后，指标轻微下降到92.02%但模型大小降低54%。 diff --git "a/docs/applications/\351\253\230\347\262\276\345\272\246\344\270\255\346\226\207\350\257\206\345\210\253\346\250\241\345\236\213.md" "b/docs/applications/\351\253\230\347\262\276\345\272\246\344\270\255\346\226\207\350\257\206\345\210\253\346\250\241\345\236\213.md" new file mode 100644 index 0000000000..11d059190a --- /dev/null +++ "b/docs/applications/\351\253\230\347\262\276\345\272\246\344\270\255\346\226\207\350\257\206\345\210\253\346\250\241\345\236\213.md" @@ -0,0 +1,113 @@ +--- +typora-copy-images-to: images +comments: true +--- + +# 高精度中文场景文本识别模型SVTR + +## 1. 简介 + +PP-OCRv3是百度开源的超轻量级场景文本检测识别模型库，其中超轻量的场景中文识别模型SVTR_LCNet使用了SVTR算法结构。为了保证速度，SVTR_LCNet将SVTR模型的Local Blocks替换为LCNet，使用两层Global Blocks。在中文场景中，PP-OCRv3识别主要使用如下优化策略（[详细技术报告](../ppocr/blog/PP-OCRv3_introduction.md)）： + +- GTC：Attention指导CTC训练策略； +- TextConAug：挖掘文字上下文信息的数据增广策略； +- TextRotNet：自监督的预训练模型； +- UDML：联合互学习策略； +- UIM：无标注数据挖掘方案。 + +其中 *UIM：无标注数据挖掘方案* 使用了高精度的SVTR中文模型进行无标注文件的刷库，该模型在PP-OCRv3识别的数据集上训练，精度对比如下表。 + +|中文识别算法|模型|UIM|精度| +| --- | --- | --- |--- | +|PP-OCRv3|SVTR_LCNet| w/o |78.40%| +|PP-OCRv3|SVTR_LCNet| w |79.40%| +|SVTR|SVTR-Tiny|-|82.50%| + +aistudio项目链接: [高精度中文场景文本识别模型SVTR](https://aistudio.baidu.com/aistudio/projectdetail/4263032) + +## 2. SVTR中文模型使用 + +### 环境准备 + +本任务基于Aistudio完成, 具体环境如下： + +- 操作系统: Linux +- PaddlePaddle: 2.3 +- PaddleOCR: dygraph + +下载PaddleOCR代码 + +```bash linenums="1" +git clone -b dygraph https://github.com/PaddlePaddle/PaddleOCR +``` + +安装依赖库 + +```bash linenums="1" +pip install -r PaddleOCR/requirements.txt -i https://mirror.baidu.com/pypi/simple +``` + +### 快速使用 + +获取SVTR中文模型文件，请加入PaddleX官方交流频道，获取20G OCR学习大礼包（内含《动手学OCR》电子书、课程回放视频、前沿论文等重磅资料） + +- PaddleX官方交流频道：
+ 
+ 
+
+ 图中，橙色区域代表`header`，淡蓝色区域代表`question`, 绿色区域表`answer`，粉红色代区域表`other`。
+
+- **下载地址**：
+
+
+ 
+ 
+
+
+- **下载地址**：
+
+
+ 
+ 
+
+
+**注：** 这里对于类别为`Ignore`或者`Others`的文本，没有进行可视化。
+
+- **下载地址**：
+ - 原始数据下载地址：[链接](https://download.openmmlab.com/mmocr/data/wildreceipt.tar)
+ - 数据格式转换后适配于PaddleOCR训练的数据下载地址：[链接](https://paddleocr.bj.bcebos.com/ppstructure/dataset/wildreceipt.tar)
diff --git a/docs/datasets/layout_datasets.en.md b/docs/datasets/layout_datasets.en.md
new file mode 100644
index 0000000000..d3de87a537
--- /dev/null
+++ b/docs/datasets/layout_datasets.en.md
@@ -0,0 +1,46 @@
+---
+comments: true
+---
+
+
+## Layout Analysis Dataset
+
+Here are the common datasets of layout anlysis, which are being updated continuously. Welcome to contribute datasets.
+
+Most of the layout analysis datasets are object detection datasets. In addition to open source datasets, you can also label or synthesize datasets using tools such as [labelme](https://github.com/wkentaro/labelme) and so on.
+
+### 1. PubLayNet dataset
+
+- **Data source**:
+
+
+ 
+ 
+
+
+- **下载地址**：
+
+
+ 
+ 
+
+
+- **下载地址**：
+
+
+ 
+ 
+
+
+- **下载地址**：
+
+[google drive](https://drive.google.com/file/d/18cSWX7wXSy4G0tbKJ0d9PuIaiwRLHpjA/view) | Included in the downloaded image zip | + +#### 2.1 ICDAR 2015 + +The ICDAR 2015 dataset can be downloaded from the link in the table above for quick validation. The lmdb format dataset required by en benchmark can also be downloaded from the table above. + +Then download the PaddleOCR format annotation file from the table above. + +PaddleOCR also provides a data format conversion script, which can convert the ICDAR official website label to the data format supported by PaddleOCR. The data conversion tool is in `ppocr/utils/gen_label.py`, here is the training set as an example: + +```bash linenums="1" +# Convert the label file downloaded from the official website to rec_gt_label.txt +python gen_label.py --mode="rec" --input_path="{path/of/origin/label}" --output_label="rec_gt_label.txt" +``` + +The data format is as follows, (a) is the original picture, (b) is the Ground Truth text file corresponding to each picture: + + + +## 3. Data storage path + +The default storage path for PaddleOCR training data is `PaddleOCR/train_data`, if you already have a dataset on your disk, just create a soft link to the dataset directory: + +```bash linenums="1" +# linux and mac os +ln -sf
+
+将下载到的数据集解压到工作目录下，假设解压在 PaddleOCR/train_data/下。然后从上表中下载转换好的标注文件。
+
+PaddleOCR 也提供了数据格式转换脚本，可以将官网 label 转换支持的数据格式。 数据转换工具在 `ppocr/utils/gen_label.py`, 这里以训练集为例：
+
+```bash linenums="1"
+# 将官网下载的标签文件转换为 train_icdar2015_label.txt
+python gen_label.py --mode="det" --root_path="/path/to/icdar_c4_train_imgs/" \
+ --input_path="/path/to/ch4_training_localization_transcription_gt" \
+ --output_label="/path/to/train_icdar2015_label.txt"
+```
+
+解压数据集和下载标注文件后，PaddleOCR/train_data/ 有两个文件夹和两个文件，按照如下方式组织icdar2015数据集：
+
+```text linenums="1"
+/PaddleOCR/train_data/icdar2015/text_localization/
+ └─ icdar_c4_train_imgs/ icdar 2015 数据集的训练数据
+ └─ ch4_test_images/ icdar 2015 数据集的测试数据
+ └─ train_icdar2015_label.txt icdar 2015 数据集的训练标注
+ └─ test_icdar2015_label.txt icdar 2015 数据集的测试标注
+```
+
+## 2. 文本识别
+
+### 2.1 PaddleOCR 文字识别数据格式
+
+PaddleOCR 中的文字识别算法支持两种数据格式:
+
+- `lmdb` 用于训练以lmdb格式存储的数据集，使用 [lmdb_dataset.py](../../../ppocr/data/lmdb_dataset.py) 进行读取;
+- `通用数据` 用于训练以文本文件存储的数据集，使用 [simple_dataset.py](../../../ppocr/data/simple_dataset.py)进行读取。
+
+下面以通用数据集为例， 介绍如何准备数据集：
+
+#### 训练集
+
+建议将训练图片放入同一个文件夹，并用一个txt文件（rec_gt_train.txt）记录图片路径和标签，txt文件里的内容如下:
+
+**注意：** txt文件中默认请将图片路径和图片标签用 \t 分割，如用其他方式分割将造成训练报错。
+
+```text linenums="1"
+" 图像文件名 图像标注信息 "
+
+train_data/rec/train/word_001.jpg 简单可依赖
+train_data/rec/train/word_002.jpg 用科技让复杂的世界更简单
+...
+```
+
+最终训练集应有如下文件结构：
+
+```text linenums="1"
+|-train_data
+ |-rec
+ |- rec_gt_train.txt
+ |- train
+ |- word_001.png
+ |- word_002.jpg
+ |- word_003.jpg
+ | ...
+```
+
+除上述单张图像为一行格式之外，PaddleOCR也支持对离线增广后的数据进行训练，为了防止相同样本在同一个batch中被多次采样，我们可以将相同标签对应的图片路径写在一行中，以列表的形式给出，在训练中，PaddleOCR会随机选择列表中的一张图片进行训练。对应地，标注文件的格式如下：
+
+```text linenums="1"
+["11.jpg", "12.jpg"] 简单可依赖
+["21.jpg", "22.jpg", "23.jpg"] 用科技让复杂的世界更简单
+3.jpg ocr
+```
+
+上述示例标注文件中，"11.jpg"和"12.jpg"的标签相同，都是`简单可依赖`，在训练的时候，对于该行标注，会随机选择其中的一张图片进行训练。
+
+#### 验证集
+
+同训练集类似，验证集也需要提供一个包含所有图片的文件夹（test）和一个rec_gt_test.txt，验证集的结构如下所示：
+
+```text linenums="1"
+|-train_data
+ |-rec
+ |- rec_gt_test.txt
+ |- test
+ |- word_001.jpg
+ |- word_002.jpg
+ |- word_003.jpg
+ | ...
+```
+
+### 2.2 公开数据集
+
+| 数据集名称 | 图片下载地址 | PaddleOCR 标注下载地址 |
+|---|---|---------------------------------------------------------------------|
+| en benchmark(MJ, SJ, IIIT, SVT, IC03, IC13, IC15, SVTP, and CUTE.) | [DTRB](https://github.com/clovaai/deep-text-recognition-benchmark#download-lmdb-dataset-for-traininig-and-evaluation-from-here) | LMDB格式，可直接用[lmdb_dataset.py](../../../ppocr/data/lmdb_dataset.py)加载 |
+|ICDAR 2015| [google drive](https://drive.google.com/file/d/18cSWX7wXSy4G0tbKJ0d9PuIaiwRLHpjA/view) | 图片下载地址中已包含 | + +#### 2.1 ICDAR 2015 + +ICDAR 2015 数据集可以在上表中链接下载，用于快速验证。也可以从上表中下载 en benchmark 所需的lmdb格式数据集。 + +下载完图片后从上表中下载转换好的标注文件。 + +PaddleOCR 也提供了数据格式转换脚本，可以将ICDAR官网 label 转换为PaddleOCR支持的数据格式。 数据转换工具在 `ppocr/utils/gen_label.py`, 这里以训练集为例： + +```bash linenums="1" +# 将官网下载的标签文件转换为 rec_gt_label.txt +python gen_label.py --mode="rec" --input_path="{path/of/origin/label}" --output_label="rec_gt_label.txt" +``` + +数据样式格式如下，(a)为原始图片,(b)为每张图片对应的 Ground Truth 文本文件： + + + +## 3. 数据存放路径 + +PaddleOCR训练数据的默认存储路径是 `PaddleOCR/train_data`,如果您的磁盘上已有数据集，只需创建软链接至数据集目录： + +```bash linenums="1" +# linux and mac os +ln -sf
+ 
+ 
+
+
+- **说明**：使用该数据集时，需要遵守[CDLA-Permissive](https://cdla.io/permissive-1-0/)协议。
+
+## 2. 好未来表格识别竞赛数据集
+
+- **数据简介**：好未来表格识别竞赛数据集的训练集合中包含1.6万张图像。验证集未给出可训练的标注。
+
+
+
+
+ 
+ 
+
+
+## 3. WTW中文场景表格数据集
+
+- **数据简介**：WTW中文场景表格数据集包含表格检测和表格数据两部分数据，数据集中同时包含扫描和拍照两张场景的图像。
+
+ 
diff --git a/docs/datasets/vertical_and_multilingual_datasets.en.md b/docs/datasets/vertical_and_multilingual_datasets.en.md
new file mode 100644
index 0000000000..321c66c51a
--- /dev/null
+++ b/docs/datasets/vertical_and_multilingual_datasets.en.md
@@ -0,0 +1,72 @@
+---
+comments: true
+---
+
+
+# Vertical multi-language OCR dataset
+
+Here we have sorted out the commonly used vertical multi-language OCR dataset datasets, which are being updated continuously. We welcome you to contribute datasets ～
+
+- [Chinese urban license plate dataset](#Chinese urban license plate dataset)
+- [Bank credit card dataset](#Bank credit card dataset)
+- [Captcha dataset-Captcha](#Captcha dataset-Captcha)
+- [multi-language dataset](#multi-language dataset)
+
+## Chinese urban license plate dataset
+
+- **Data source**：[CCPD](https://github.com/detectRecog/CCPD)
+
+- **Data introduction**: It contains more than 250000 vehicle license plate images and vehicle license plate detection and recognition information labeling. It contains the following license plate image information in different scenes.
+
+ - CCPD-Base: General license plate picture
+ - CCPD-DB: The brightness of license plate area is bright, dark or uneven
+ - CCPD-FN: The license plate is farther or closer to the camera location
+ - CCPD-Rotate: License plate includes rotation (horizontal 20\~50 degrees, vertical-10\~10 degrees)
+ - CCPD-Tilt: License plate includes rotation (horizontal 15\~45 degrees, vertical 15\~45 degrees)
+ - CCPD-Blur: The license plate contains blurring due to camera lens jitter
+ - CCPD-Weather: The license plate is photographed on rainy, snowy or foggy days
+ - CCPD-Challenge: So far, some of the most challenging images in license plate detection and recognition tasks
+ - CCPD-NP: Pictures of new cars without license plates.
+
+ 
+
+- **Download address**
+ - Baidu cloud download address (extracted code is hm0U): [https://pan.baidu.com/s/1i5AOjAbtkwb17Zy-NQGqkw](https://pan.baidu.com/s/1i5AOjAbtkwb17Zy-NQGqkw)
+ - Google drive download address:[https://drive.google.com/file/d/1rdEsCUcIUaYOVRkx5IMTRNA7PcGMmSgc/view](https://drive.google.com/file/d/1rdEsCUcIUaYOVRkx5IMTRNA7PcGMmSgc/view)
+
+## Bank credit card dataset
+
+- **Data source**: [source](https://www.kesci.com/home/dataset/5954cf1372ead054a5e25870)
+
+- **Data introduction**: There are three types of training data
+ - 1.Sample card data of China Merchants Bank: including card image data and annotation data, a total of 618 pictures
+ - 2.Single character data: including pictures and annotation data, 37 pictures in total.
+ - 3.There are only other bank cards, no more detailed information, a total of 50 pictures.
+
+ - The demo image is shown as follows. The annotation information is stored in excel, and the demo image below is marked as
+ - Top 8 card number: 62257583
+ - Card type: card of our bank
+ - End of validity: 07/41
+ - Chinese phonetic alphabet of card users: MICHAEL
+
+ 
+
+- **Download address**: [cmb2017-2.zip](https://cdn.kesci.com/cmb2017-2.zip)
+
+## Captcha dataset-Captcha
+
+- **Data source**: [captcha](https://github.com/lepture/captcha)
+- **Data introduction**: This is a toolkit for data synthesis. You can output captcha images according to the input text. Use the toolkit to generate several demo images as follows.
+
+ 
+
+- **Download address**: The dataset is generated and has no download address.
+
+## multi-language dataset(Multi-lingual scene text detection and recognition)
+
+- **Data source**: [source](https://rrc.cvc.uab.es/?ch=15&com=downloads)
+- **Data introduction**: Multi language detection dataset MLT contains both language recognition and detection tasks.
+ - In the detection task, the training set contains 10000 images in 10 languages, and each language contains 1000 training images. The test set contains 10000 images.
+ - In the recognition task, the training set contains 111998 samples.
+- **Download address**: The training set is large and can be downloaded in two parts. It can only be downloaded after registering on the website:
+[source](https://rrc.cvc.uab.es/?ch=15&com=downloads)
diff --git a/docs/datasets/vertical_and_multilingual_datasets.md b/docs/datasets/vertical_and_multilingual_datasets.md
new file mode 100644
index 0000000000..f5386b7bb8
--- /dev/null
+++ b/docs/datasets/vertical_and_multilingual_datasets.md
@@ -0,0 +1,65 @@
+---
+comments: true
+---
+
+
+# 垂类多语言OCR数据集
+
+这里整理了常用垂类和多语言OCR数据集，持续更新中，欢迎各位小伙伴贡献数据集～
+
+## 中国城市车牌数据集
+
+- **数据来源**：[CCPD](https://github.com/detectRecog/CCPD)
+- **数据简介**: 包含超过25万张中国城市车牌图片及车牌检测、识别信息的标注。包含以下几种不同场景中的车牌图片信息。
+ - CCPD-Base: 通用车牌图片
+ - CCPD-DB: 车牌区域亮度较亮、较暗或者不均匀
+ - CCPD-FN: 车牌离摄像头拍摄位置相对更远或者更近
+ - CCPD-Rotate: 车牌包含旋转（水平20\~50度，竖直-10\~10度）
+ - CCPD-Tilt: 车牌包含旋转（水平15\~45度，竖直15\~45度）
+ - CCPD-Blur: 车牌包含由于摄像机镜头抖动导致的模糊情况
+ - CCPD-Weather: 车牌在雨天、雪天或者雾天拍摄得到
+ - CCPD-Challenge: 至今在车牌检测识别任务中最有挑战性的一些图片
+ - CCPD-NP: 没有安装车牌的新车图片。
+
+ 
+
+- **下载地址**
+ - 百度云下载地址(提取码是hm0U): [link](https://pan.baidu.com/s/1i5AOjAbtkwb17Zy-NQGqkw)
+ - Google drive下载地址：[link](https://drive.google.com/file/d/1rdEsCUcIUaYOVRkx5IMTRNA7PcGMmSgc/view)
+
+## 银行信用卡数据集
+
+- **数据来源**: [source](https://www.kesci.com/home/dataset/5954cf1372ead054a5e25870)
+
+- **数据简介**: 训练数据共提供了三类数据
+ - 1.招行样卡数据： 包括卡面图片数据及标注数据，总共618张图片
+ - 2.单字符数据： 包括图片及标注数据，总共37张图片。
+ - 3.仅包含其他银行卡面，不具有更细致的信息，总共50张图片。
+
+ - demo图片展示如下，标注信息存储在excel表格中，下面的demo图片标注为
+ - 前8位卡号：62257583
+ - 卡片种类：本行卡
+ - 有效期结束：07/41
+ - 卡用户拼音：MICHAEL
+
+ 
+
+- **下载地址**: [cmb2017-2.zip](https://cdn.kesci.com/cmb2017-2.zip)
+
+## 验证码数据集-Captcha
+
+- **数据来源**: [captcha](https://github.com/lepture/captcha)
+- **数据简介**: 这是一个数据合成的工具包，可以根据输入的文本，输出验证码图片，使用该工具包生成几张demo图片如下：
+
+ 
+
+- **下载地址**: 该数据集是生成得到，无下载地址。
+
+## 多语言数据集(Multi-lingual scene text detection and recognition)
+
+- **数据来源**: [source](https://rrc.cvc.uab.es/?ch=15&com=downloads)
+- **数据简介**: 多语言检测数据集MLT同时包含了语种识别和检测任务。
+ - 在检测任务中，训练集包含10000张图片，共有10种语言，每种语言包含1000张训练图片。测试集包含10000张图片。
+ - 在识别任务中，训练集包含111998个样本。
+- **下载地址**: 训练集较大，分2部分下载，需要在网站上注册之后才能下载：
+[link](https://rrc.cvc.uab.es/?ch=15&com=downloads)
diff --git a/docs/images/00006737.jpg b/docs/images/00006737.jpg
new file mode 100644
index 0000000000..d7762d2e2c
Binary files /dev/null and b/docs/images/00006737.jpg differ
diff --git a/docs/images/185310636-6ce02f7c-790d-479f-b163-ea97a5a04808-20240708082238739.jpg b/docs/images/185310636-6ce02f7c-790d-479f-b163-ea97a5a04808-20240708082238739.jpg
new file mode 100644
index 0000000000..6a5fd84c52
Binary files /dev/null and b/docs/images/185310636-6ce02f7c-790d-479f-b163-ea97a5a04808-20240708082238739.jpg differ
diff --git a/docs/images/185393805-c67ff571-cf7e-4217-a4b0-8b396c4f22bb-20240708082310650.jpg b/docs/images/185393805-c67ff571-cf7e-4217-a4b0-8b396c4f22bb-20240708082310650.jpg
new file mode 100644
index 0000000000..d406a52da8
Binary files /dev/null and b/docs/images/185393805-c67ff571-cf7e-4217-a4b0-8b396c4f22bb-20240708082310650.jpg differ
diff --git a/docs/images/185539517-ccf2372a-f026-4a7c-ad28-c741c770f60a-20240708082247529.png b/docs/images/185539517-ccf2372a-f026-4a7c-ad28-c741c770f60a-20240708082247529.png
new file mode 100644
index 0000000000..429343418e
Binary files /dev/null and b/docs/images/185539517-ccf2372a-f026-4a7c-ad28-c741c770f60a-20240708082247529.png differ
diff --git a/docs/images/185540080-0431e006-9235-4b6d-b63d-0b3c6e1de48f-20240708082316558.jpg b/docs/images/185540080-0431e006-9235-4b6d-b63d-0b3c6e1de48f-20240708082316558.jpg
new file mode 100644
index 0000000000..bdef2ed23b
Binary files /dev/null and b/docs/images/185540080-0431e006-9235-4b6d-b63d-0b3c6e1de48f-20240708082316558.jpg differ
diff --git a/docs/images/186094813-3a8e16cc-42e5-4982-b9f4-0134dfb5688d-20240708082323916.png b/docs/images/186094813-3a8e16cc-42e5-4982-b9f4-0134dfb5688d-20240708082323916.png
new file mode 100644
index 0000000000..44845fb5ce
Binary files /dev/null and b/docs/images/186094813-3a8e16cc-42e5-4982-b9f4-0134dfb5688d-20240708082323916.png differ
diff --git a/docs/images/186171245-40abc4d7-904f-4949-ade1-250f86ed3a90.jpg b/docs/images/186171245-40abc4d7-904f-4949-ade1-250f86ed3a90.jpg
new file mode 100644
index 0000000000..3e3fde4e69
Binary files /dev/null and b/docs/images/186171245-40abc4d7-904f-4949-ade1-250f86ed3a90.jpg differ
diff --git a/docs/images/197464552-69de557f-edff-4c7f-acbf-069df1ba097f-20240708082253634.png b/docs/images/197464552-69de557f-edff-4c7f-acbf-069df1ba097f-20240708082253634.png
new file mode 100644
index 0000000000..7b4c271cf2
Binary files /dev/null and b/docs/images/197464552-69de557f-edff-4c7f-acbf-069df1ba097f-20240708082253634.png differ
diff --git a/docs/images/PP-OCRv3-pic001.jpg b/docs/images/PP-OCRv3-pic001.jpg
new file mode 100644
index 0000000000..c35936cc1a
Binary files /dev/null and b/docs/images/PP-OCRv3-pic001.jpg differ
diff --git a/docs/images/PP-OCRv3-pic002.jpg b/docs/images/PP-OCRv3-pic002.jpg
new file mode 100644
index 0000000000..e5ad6a4b2a
Binary files /dev/null and b/docs/images/PP-OCRv3-pic002.jpg differ
diff --git a/docs/images/PP-OCRv3-pic003.jpg b/docs/images/PP-OCRv3-pic003.jpg
new file mode 100644
index 0000000000..dc024296bd
Binary files /dev/null and b/docs/images/PP-OCRv3-pic003.jpg differ
diff --git a/docs/images/en_1.png b/docs/images/en_1.png
new file mode 100644
index 0000000000..36245613e3
Binary files /dev/null and b/docs/images/en_1.png differ
diff --git a/docs/images/en_2.png b/docs/images/en_2.png
new file mode 100644
index 0000000000..d2df8556ad
Binary files /dev/null and b/docs/images/en_2.png differ
diff --git a/docs/images/en_3-0398013.png b/docs/images/en_3-0398013.png
new file mode 100644
index 0000000000..baf146c010
Binary files /dev/null and b/docs/images/en_3-0398013.png differ
diff --git a/docs/images/en_3.png b/docs/images/en_3.png
new file mode 100644
index 0000000000..baf146c010
Binary files /dev/null and b/docs/images/en_3.png differ
diff --git a/docs/images/japan_2.jpg b/docs/images/japan_2.jpg
new file mode 100644
index 0000000000..076ced92ad
Binary files /dev/null and b/docs/images/japan_2.jpg differ
diff --git a/docs/images/korean_1.jpg b/docs/images/korean_1.jpg
new file mode 100644
index 0000000000..f93de40e18
Binary files /dev/null and b/docs/images/korean_1.jpg differ
diff --git a/docs/images/ppocrv4.png b/docs/images/ppocrv4.png
new file mode 100644
index 0000000000..6be449cf53
Binary files /dev/null and b/docs/images/ppocrv4.png differ
diff --git a/docs/images/ppstructure-20240708082235651.gif b/docs/images/ppstructure-20240708082235651.gif
new file mode 100644
index 0000000000..bff836e3ea
Binary files /dev/null and b/docs/images/ppstructure-20240708082235651.gif differ
diff --git a/docs/images/test_add_91.jpg b/docs/images/test_add_91.jpg
new file mode 100644
index 0000000000..b5ded6e1de
Binary files /dev/null and b/docs/images/test_add_91.jpg differ
diff --git a/docs/index.en.md b/docs/index.en.md
new file mode 100644
index 0000000000..9c8baf14ad
--- /dev/null
+++ b/docs/index.en.md
@@ -0,0 +1,154 @@
+---
+comments: true
+typora-copy-images-to: images
+hide:
+ - navigation
+ - toc
+---
+
+
+
+
+ 
+
+
+## Introduction
+
+PaddleOCR aims to create multilingual, awesome, leading, and practical OCR tools that help users train better models and apply them into practice.
+
+## 🚀 Community
+
+PaddleOCR is being oversight by a [PMC](https://github.com/PaddlePaddle/PaddleOCR/issues/12122). Issues and PRs will be reviewed on a best-effort basis. For a complete overview of PaddlePaddle community, please visit [community](https://github.com/PaddlePaddle/community).
+
+⚠️ Note: The [Issues](https://github.com/PaddlePaddle/PaddleOCR/issues) module is only for reporting program 🐞 bugs, for the rest of the questions, please move to the [Discussions](https://github.com/PaddlePaddle/PaddleOCR/discussions). Please note that if the Issue mentioned is not a bug, it will be moved to the Discussions module.
+
+## 📣 Recent updates
+
+- **🔥2023.8.7 Release PaddleOCR[release/2.7](https://github.com/PaddlePaddle/PaddleOCR/tree/release/2.7)**
+
+ - Release [PP-OCRv4](./ppocr/overview.en.md), support mobile version and server version
+
+ - PP-OCRv4-mobile：When the speed is comparable, the effect of the Chinese scene is improved by 4.5% compared with PP-OCRv3, the English scene is improved by 10%, and the average recognition accuracy of the 80-language multilingual model is increased by more than 8%.
+ - PP-OCRv4-server：Release the OCR model with the highest accuracy at present, the detection model accuracy increased by 4.9% in the Chinese and English scenes, and the recognition model accuracy increased by 2%
+ refer [quickstart](./quick_start.en.md) quick use by one line command, At the same time, the whole process of model training, reasoning, and high-performance deployment can also be completed with few code in the [General OCR Industry Solution](https://aistudio.baidu.com/aistudio/modelsdetail?modelId=286) in PaddleX.
+
+ - Release[PP-ChatOCR](https://aistudio.baidu.com/aistudio/modelsdetail?modelId=332), a new scheme for extracting key information of general scenes using PP-OCR model and ERNIE LLM.
+
+- 🔨**2022.11 Add implementation of [4 cutting-edge algorithms](./algorithm/overview.en.md)**：Text Detection [DRRG](./algorithm/text_detection/algorithm_det_drrg.en.md), Text Recognition [RFL](./algorithm/text_recognition/algorithm_rec_rfl.en.md), Image Super-Resolution [Text Telescope](./algorithm/super_resolution/algorithm_sr_telescope.en.md)，Handwritten Mathematical Expression Recognition [CAN](./algorithm/formula_recognition/algorithm_rec_can.en.md)
+
+- **2022.10 release [optimized JS version PP-OCRv3 model](./ppocr/infer_deploy/paddle_js.en.md)** with 4.3M model size, 8x faster inference time, and a ready-to-use web demo
+
+ - 💥 **Live Playback: Introduction to PP-StructureV2 optimization strategy**. Scan [the QR code below](#Community) using WeChat, follow the PaddlePaddle official account and fill out the questionnaire to join the WeChat group, get the live link and 20G OCR learning materials (including PDF2Word application, 10 models in vertical scenarios, etc.)
+
+- **🔥2022.8.24 Release PaddleOCR [release/2.6](https://github.com/PaddlePaddle/PaddleOCR/tree/release/2.6)**
+
+ - Release [PP-StructureV2](./ppstructure/)，with functions and performance fully upgraded, adapted to Chinese scenes, and new support for [Layout Recovery](./ppstructure/model_train/recovery_to_doc.en.md) and **one line command to convert PDF to Word**;
+ - [Layout Analysis](./ppstructure/model_train/train_layout.en.md) optimization: model storage reduced by 95%, while speed increased by 11 times, and the average CPU time-cost is only 41ms;
+ - [Table Recognition](./ppstructure/model_train/train_table.en.md) optimization: 3 optimization strategies are designed, and the model accuracy is improved by 6% under comparable time consumption;
+ - [Key Information Extraction](./ppstructure/model_train/train_kie.en.md) optimization：a visual-independent model structure is designed, the accuracy of semantic entity recognition is increased by 2.8%, and the accuracy of relation extraction is increased by 9.1%.
+
+- **🔥2022.8 Release [OCR scene application collection](./applications/overview.md)**
+
+ - Release **9 vertical models** such as digital tube, LCD screen, license plate, handwriting recognition model, high-precision SVTR model, etc, covering the main OCR vertical applications in general, manufacturing, finance, and transportation industries.
+
+- **2022.8 Add implementation of [8 cutting-edge algorithms](./algorithm/overview.en.md)**
+
+ - Text Detection: [FCENet](./algorithm/text_detection/algorithm_det_fcenet.en.md), [DB++](./algorithm/text_detection/algorithm_det_db.en.md)
+ - Text Recognition: [ViTSTR](./algorithm/text_recognition/algorithm_rec_vitstr.en.md), [ABINet](./algorithm/text_recognition/algorithm_rec_abinet.en.md), [VisionLAN](./algorithm/text_recognition/algorithm_rec_visionlan.en.md), [SPIN](./algorithm/text_recognition/algorithm_rec_spin.en.md), [RobustScanner](./algorithm/text_recognition/algorithm_rec_robustscanner.en.md)
+ - Table Recognition: [TableMaster](./algorithm/table_recognition/algorithm_table_master.en.md)
+- **2022.5.9 Release PaddleOCR [release/2.5](https://github.com/PaddlePaddle/PaddleOCR/tree/release/2.5)**
+
+ - Release [PP-OCRv3](./ppocr/overview.en.md#pp-ocrv3): With comparable speed, the effect of Chinese scene is further improved by 5% compared with PP-OCRv2, the effect of English scene is improved by 11%, and the average recognition accuracy of 80 language multilingual models is improved by more than 5%.
+ - Release [PPOCRLabelv2](https://github.com/PFCCLab/PPOCRLabel): Add the annotation function for table recognition task, key information extraction task and irregular text image.
+ - Release interactive e-book [*"Dive into OCR"*](./blog/ocr_book.en.md), covers the cutting-edge theory and code practice of OCR full stack technology.
+- [more](./update.en.md)
+
+## 🌟 Features
+
+PaddleOCR support a variety of cutting-edge algorithms related to OCR, and developed industrial featured models/solution [PP-OCR](./ppocr/overview.md)、[PP-Structure](./ppstructure/overview.md) and [PP-ChatOCR](https://aistudio.baidu.com/aistudio/projectdetail/6488689) on this basis, and get through the whole process of data production, model training, compression, inference and deployment.
+
+
+
+> It is recommended to start with the “quick experience” in the document tutorial
+
+## 📖 Technical exchange and cooperation
+
+- PaddleX provides a one-stop full-process high-efficiency development platform for flying paddle ecological model training, pressure, and push. Its mission is to help AI technology quickly land, and its vision is to make everyone an AI Developer!
+
+ - PaddleX currently covers areas such as image classification, object detection, image segmentation, 3D, OCR, and time series prediction, and has built-in 36 basic single models, such as RP-DETR, PP-YOLOE, PP-HGNet, PP-LCNet, PP- LiteSeg, etc.; integrated 12 practical industrial solutions, such as PP-OCRv4, PP-ChatOCR, PP-ShiTu, PP-TS, vehicle-mounted road waste detection, identification of prohibited wildlife products, etc.
+ - PaddleX provides two AI development modes: "Toolbox" and "Developer". The toolbox mode can tune key hyperparameters without code, and the developer mode can perform single-model training, push and multi-model serial inference with low code, and supports both cloud and local terminals.
+ - PaddleX also supports joint innovation and development, profit sharing! At present, PaddleX is rapidly iterating, and welcomes the participation of individual developers and enterprise developers to create a prosperous AI technology ecosystem!
+
+## 🇺🇳 Guideline for New Language Requests
+
+If you want to request a new language support, a PR with 1 following files are needed：
+
+- In folder [ppocr/utils/dict](./ppocr/utils/dict),
+it is necessary to submit the dict text to this path and name it with `{language}_dict.txt` that contains a list of all characters. Please see the format example from other files in that folder.
+
+If your language has unique elements, please tell me in advance within any way, such as useful links, wikipedia and so on.
+
+More details, please refer to [Multilingual OCR Development Plan](https://github.com/PaddlePaddle/PaddleOCR/issues/1048).
+
+## Visualization
+
+## PP-OCRv3
+
+### PP-OCRv3 Chinese model
+
+
+
+
+
+
+
+
+
+
+
+### PP-OCRv3 English model
+
+
+
+
+
+
+
+### PP-OCRv3 Multilingual model
+
+
+
+
+
+#### PP-StructureV2
+
+- layout analysis + table recognition
+
+ 
+
+- SER (Semantic entity recognition)
+
+ 
+
+ 
+
+ 
+
+- RE (Relation Extraction)
+
+ 
+
+ 
+
+ 
+
+## 📄 License
+
+This project is released under Apache 2.0 license
diff --git a/docs/index.md b/docs/index.md
new file mode 100644
index 0000000000..93e112a929
--- /dev/null
+++ b/docs/index.md
@@ -0,0 +1,137 @@
+---
+comments: true
+typora-copy-images-to: images
+hide:
+ - navigation
+ - toc
+---
+
+
+
+ 
+ 
+ 
+ 
+ 
+
+
+
+## 简介
+
+PaddleOCR 旨在打造一套丰富、领先、且实用的 OCR 工具库，助力开发者训练出更好的模型，并应用落地。
+
+## 🚀 社区
+
+PaddleOCR 由 [PMC](https://github.com/PaddlePaddle/PaddleOCR/issues/12122) 监督。Issues 和 PRs 将在尽力的基础上进行审查。欲了解 PaddlePaddle 社区的完整概况，请访问 [community](https://github.com/PaddlePaddle/community)。
+
+⚠️注意：[Issues](https://github.com/PaddlePaddle/PaddleOCR/issues)模块仅用来报告程序🐞Bug，其余提问请移步[Discussions](https://github.com/PaddlePaddle/PaddleOCR/discussions)模块提问。如所提Issue不是Bug，会被移到Discussions模块，敬请谅解。
+
+## 📣 近期更新
+
+- **🔥2024.7 添加 PaddleOCR 算法模型挑战赛冠军方案**：
+
+ - 赛题一：OCR 端到端识别任务冠军方案——[场景文本识别算法-SVTRv2](./algorithm/text_recognition/algorithm_rec_svtrv2.md)；
+ - 赛题二：通用表格识别任务冠军方案——[表格识别算法-SLANet-LCNetV2](./algorithm/table_recognition/algorithm_table_slanet.md)。
+
+- **💥2024.6.27 飞桨低代码开发工具 [PaddleX 3.0](https://github.com/paddlepaddle/paddlex) 重磅更新！**
+
+ - 低代码开发范式：支持 OCR 模型全流程低代码开发，提供 Python API，支持用户自定义串联模型；
+ - 多硬件训推支持：支持英伟达 GPU、昆仑芯、昇腾和寒武纪等多种硬件进行模型训练与推理。PaddleOCR支持的模型见 [模型列表](./model/hardware/install_other_devices.md)
+
+- **📚直播和OCR实战打卡营预告**：《PP-ChatOCRv2赋能金融报告信息智能化抽取，新金融效率再升级》课程上线，破解复杂版面、表格识别、信息抽取OCR解析难题，直播时间：6月6日（周四）19：00。并于6月11日启动【政务采购合同信息抽取】实战打卡营。报名链接：
+
+
+
+
+
+
+
+## 效果展示
+
+### 超轻量PP-OCRv3效果展示
+
+#### PP-OCRv3中文模型
+
+
+
+
+
+
+
+
+
+
+
+#### PP-OCRv3英文数字模型
+
+
+
+
+
+
+
+#### PP-OCRv3多语言模型
+
+
+
+
+
+#### PP-Structure 文档分析
+
+- 版面分析+表格识别
+
+ 
+
+- SER（语义实体识别）
+
+ 
+
+ 
+
+ 
+
+- RE（关系提取）
+
+ 
+
+ 
+
+ 
+
+## 许可证书
+
+本项目的发布受Apache 2.0 license许可认证。
diff --git a/docs/javascripts/katex.min.js b/docs/javascripts/katex.min.js
new file mode 100644
index 0000000000..fee59b98a5
--- /dev/null
+++ b/docs/javascripts/katex.min.js
@@ -0,0 +1,12 @@
+document$.subscribe(({ body }) => {
+ renderMathInElement(body, {
+ delimiters: [
+ { left: "$$", right: "$$", display: true },
+ { left: "$", right: "$", display: false },
+ { left: "\\(", right: "\\)", display: false },
+ { left: "\\[", right: "\\]", display: true }
+ ],
+ })
+})
+
+!function (e, t) { "object" == typeof exports && "object" == typeof module ? module.exports = t() : "function" == typeof define && define.amd ? define([], t) : "object" == typeof exports ? exports.katex = t() : e.katex = t() }("undefined" != typeof self ? self : this, (function () { return function () { "use strict"; var e = { d: function (t, r) { for (var n in r) e.o(r, n) && !e.o(t, n) && Object.defineProperty(t, n, { enumerable: !0, get: r[n] }) }, o: function (e, t) { return Object.prototype.hasOwnProperty.call(e, t) } }, t = {}; e.d(t, { default: function () { return Yn } }); class r { constructor(e, t) { this.name = void 0, this.position = void 0, this.length = void 0, this.rawMessage = void 0; let n, o, s = "KaTeX parse error: " + e; const i = t && t.loc; if (i && i.start <= i.end) { const e = i.lexer.input; n = i.start, o = i.end, n === e.length ? s += " at end of input: " : s += " at position " + (n + 1) + ": "; const t = e.slice(n, o).replace(/[^]/g, "$&\u0332"); let r, a; r = n > 15 ? "\u2026" + e.slice(n - 15, n) : e.slice(0, n), a = o + 15 < e.length ? e.slice(o, o + 15) + "\u2026" : e.slice(o), s += r + t + a } const a = new Error(s); return a.name = "ParseError", a.__proto__ = r.prototype, a.position = n, null != n && null != o && (a.length = o - n), a.rawMessage = e, a } } r.prototype.__proto__ = Error.prototype; var n = r; const o = /([A-Z])/g, s = { "&": "&", ">": ">", "<": "<", '"': """, "'": "'" }, i = /[&><"']/g; const a = function (e) { return "ordgroup" === e.type || "color" === e.type ? 1 === e.body.length ? a(e.body[0]) : e : "font" === e.type ? a(e.body) : e }; var l = { contains: function (e, t) { return -1 !== e.indexOf(t) }, deflt: function (e, t) { return void 0 === e ? t : e }, escape: function (e) { return String(e).replace(i, (e => s[e])) }, hyphenate: function (e) { return e.replace(o, "-$1").toLowerCase() }, getBaseElem: a, isCharacterBox: function (e) { const t = a(e); return "mathord" === t.type || "textord" === t.type || "atom" === t.type }, protocolFromUrl: function (e) { const t = /^[\x00-\x20]*([^\\/#?]*?)(:|�*58|�*3a|&colon)/i.exec(e); return t ? ":" !== t[2] ? null : /^[a-zA-Z][a-zA-Z0-9+\-.]*$/.test(t[1]) ? t[1].toLowerCase() : null : "_relative" } }; const h = { displayMode: { type: "boolean", description: "Render math in display mode, which puts the math in display style (so \\int and \\sum are large, for example), and centers the math on the page on its own line.", cli: "-d, --display-mode" }, output: { type: { enum: ["htmlAndMathml", "html", "mathml"] }, description: "Determines the markup language of the output.", cli: "-F, --format 
+
+### （5）UDML：联合互学习策略
+
+UDML（Unified-Deep Mutual Learning）联合互学习是PP-OCRv2中就采用的对于文本识别非常有效的提升模型效果的策略。在PP-OCRv3中，针对两个不同的SVTR_LCNet和Attention结构，对他们之间的PP-LCNet的特征图、SVTR模块的输出和Attention模块的输出同时进行监督训练。使用该策略，识别模型的准确率进一步提升到78.4%（+1.5%）。
+
+### （6）UIM：无标注数据挖掘方案
+
+UIM（Unlabeled Images Mining）是一种非常简单的无标注数据挖掘方案。核心思想是利用高精度的文本识别大模型对无标注数据进行预测，获取伪标签，并且选择预测置信度高的样本作为训练数据，用于训练小模型。使用该策略，识别模型的准确率进一步提升到79.4%（+1%）。实际操作中，我们使用全量数据集训练高精度SVTR-Tiny模型（acc=82.5%）进行数据挖掘，点击获取[模型下载地址和使用教程](../applications/高精度中文识别模型.md)。
+
+
+
+## 4. 端到端评估
+
+经过以上优化，最终PP-OCRv3在速度可比情况下，中文场景端到端Hmean指标相比于PP-OCRv2提升5%，效果大幅提升。具体指标如下表所示：
+
+| Model | Hmean | Model Size (M) | Time Cost (CPU, ms) | Time Cost (T4 GPU, ms) |
+|-----|-----|--------|----| --- |
+| PP-OCR mobile | 50.30% | 8.1 | 356.00 | 116.00 |
+| PP-OCR server | 57.00% | 155.1 | 1056.00 | 200.00 |
+| PP-OCRv2 | 57.60% | 11.6 | 330.00 | 111.00 |
+| PP-OCRv3 | 62.90% | 15.6 | 331.00 | 86.64 |
+
+测试环境：CPU型号为Intel Gold 6148，CPU预测时开启MKLDNN加速。
+
+除了更新中文模型，本次升级也同步优化了英文数字模型，端到端效果提升11%，如下表所示：
+
+| Model | Recall | Precision | Hmean |
+|-----|-----|--------|----|
+| PP-OCR_en | 38.99% | 45.91% | 42.17% |
+| PP-OCRv3_en | 50.95% | 55.53% | 53.14% |
+
+同时，也对已支持的80余种语言识别模型进行了升级更新，在有评估集的四种语系识别准确率平均提升5%以上，如下表所示：
+
+| Model | 拉丁语系 | 阿拉伯语系 | 日语 | 韩语 |
+|-----|-----|--------|----| --- |
+| PP-OCR_mul | 69.60% | 40.50% | 38.50% | 55.40% |
+| PP-OCRv3_mul | 75.20%| 45.37% | 45.80% | 60.10% |
diff --git a/docs/ppocr/blog/PP-OCRv4_introduction.md b/docs/ppocr/blog/PP-OCRv4_introduction.md
new file mode 100644
index 0000000000..0579a607d9
--- /dev/null
+++ b/docs/ppocr/blog/PP-OCRv4_introduction.md
@@ -0,0 +1,153 @@
+---
+typora-copy-images-to: images
+comments: true
+---
+
+# PP-OCRv4
+
+## 1. 简介
+
+PP-OCRv4在PP-OCRv3的基础上进一步升级。整体的框架图保持了与PP-OCRv3相同的pipeline，针对检测模型和识别模型进行了数据、网络结构、训练策略等多个模块的优化。 PP-OCRv4系统框图如下所示：
+
+
+
+从算法改进思路上看，分别针对检测和识别模型，进行了共10个方面的改进：
+
+* 检测模块：
+ * LCNetV3：精度更高的骨干网络
+ * PFHead：并行head分支融合结构
+ * DSR: 训练中动态增加shrink ratio
+ * CML：添加Student和Teacher网络输出的KL div loss
+
+* 识别模块：
+ * SVTR_LCNetV3：精度更高的骨干网络
+ * Lite-Neck：精简的Neck结构
+ * GTC-NRTR：稳定的Attention指导分支
+ * Multi-Scale：多尺度训练策略
+ * DF: 数据挖掘方案
+ * DKD ：DKD蒸馏策略
+
+从效果上看，速度可比情况下，多种场景精度均有大幅提升：
+
+* 中文场景，相对于PP-OCRv3中文模型提升超4%；
+* 英文数字场景，相比于PP-OCRv3英文模型提升6%；
+* 多语言场景，优化80个语种识别效果，平均准确率提升超8%。
+
+## 2. 检测优化
+
+PP-OCRv4检测模型在PP-OCRv3检测模型的基础上，在网络结构，训练策略，蒸馏策略三个方面做了优化。首先，PP-OCRv4检测模型使用PP-LCNetV3替换MobileNetv3，并提出并行分支融合的PFhead结构；其次，训练时动态调整shrink ratio的比例；最后，PP-OCRv4对CML的蒸馏loss进行优化，进一步提升文字检测效果。
+
+消融实验如下：
+
+|序号|策略|模型大小|hmean|速度（cpu + mkldnn)|
+|-|-|-|-|-|
+|baseline|PP-OCRv3|3.4M|78.84%|69ms|
+|baseline student|PP-OCRv3 student|3.4M|76.22%|69ms|
+|01|+PFHead|3.6M|76.97%|96ms|
+|02|+Dynamic Shrink Ratio|3.6M|78.24%|96ms|
+|03|+PP-LCNetv3|4.8M|79.08%|94ms|
+|03|+CML|4.8M|79.87%|67ms|
+
+测试环境： Intel Gold 6148 CPU，预测引擎使用openvino。
+
+### （1）PFhead：多分支融合Head结构
+
+PFhead结构如下图所示，PFHead在经过第一个转置卷积后，分别进行上采样和转置卷积，上采样的输出通过3x3卷积得到输出结果，然后和转置卷积的分支的结果级联并经过1x1卷积层，最后1x1卷积的结果和转置卷积的结果相加得到最后输出的概率图。PP-OCRv4学生检测模型使用PFhead，hmean从76.22%增加到76.97%。
+
+ 
+
+### （2）DSR: 收缩比例动态调整策略
+
+动态shrink ratio(dynamic shrink ratio): 在训练中，shrink ratio由固定值调整为动态变化，随着训练epoch的增加，shrink ratio从0.4线性增加到0.6。该策略在PP-OCRv4学生检测模型上，hmean从76.97%提升到78.24%。
+
+### (3) PP-LCNetV3：精度更高的骨干网络
+
+PP-LCNetV3系列模型是PP-LCNet系列模型的延续，覆盖了更大的精度范围，能够适应不同下游任务的需要。PP-LCNetV3系列模型从多个方面进行了优化，提出了可学习仿射变换模块，对重参数化策略、激活函数进行了改进，同时调整了网络深度与宽度。最终，PP-LCNetV3系列模型能够在性能与效率之间达到最佳的平衡，在不同精度范围内取得极致的推理速度。使用PP-LCNetV3替换MobileNetv3 backbone，PP-OCRv4学生检测模型hmean从78.24%提升到79.08%。
+
+### （4）CML: 融合KD的互学习策略
+
+PP-OCRv4检测模型对PP-OCRv3中的CML（Collaborative Mutual Learning) 协同互学习文本检测蒸馏策略进行了优化。如下图所示，在计算Student Model和Teacher Model的distill Loss时，额外添加KL div loss，让两者输出的response maps分布接近，由此进一步提升Student网络的精度，检测Hmean从79.08%增加到79.56%，端到端指标从61.31%增加到61.87%。
+
+ 
+
+## 3. 识别优化
+
+PP-OCRv4识别模型在PP-OCRv3的基础上进一步升级。如下图所示，整体的框架图保持了与PP-OCRv3识别模型相同的pipeline，分别进行了数据、网络结构、训练策略等方面的优化。
+
+
+
+经过如图所示的策略优化，PP-OCRv4识别模型相比PP-OCRv3，在速度可比的情况下，精度进一步提升4%。 具体消融实验如下所示：
+
+| ID | 策略 | 模型大小 | 精度 | 预测耗时（CPU openvino)|
+|-----|-----|--------|----| --- |
+| 01 | PP-OCRv3 | 12M | 71.50% | 8.54ms |
+| 02 | +DF | 12M | 72.70% | 8.54ms |
+| 03 | + LiteNeck + GTC | 9.6M | 73.21% | 9.09ms |
+| 04 | + PP-LCNetV3 | 11M | 74.18% | 9.8ms |
+| 05 | + multi-scale | 11M | 74.20% | 9.8ms |
+| 06 | + TextConAug | 11M | 74.72% | 9.8ms |
+| 08 | + UDML | 11M | 75.45% | 9.8ms |
+
+注： 测试速度时，输入图片尺寸均为(3,48,320)。在实际预测时，图像为变长输入，速度会有所变化。测试环境： Intel Gold 6148 CPU，预测时使用Openvino预测引擎。
+
+### （1）DF：数据挖掘方案
+
+DF(Data Filter) 是一种简单有效的数据挖掘方案。核心思想是利用已有模型预测训练数据，通过置信度和预测结果等信息，对全量的训练数据进行筛选。具体的：首先使用少量数据快速训练得到一个低精度模型，使用该低精度模型对千万级的数据进行预测，去除置信度大于0.95的样本，该部分被认为是对提升模型精度无效的冗余样本。其次使用PP-OCRv3作为高精度模型，对剩余数据进行预测，去除置信度小于0.15的样本，该部分被认为是难以识别或质量很差的样本。
+使用该策略，千万级别训练数据被精简至百万级，模型训练时间从2周减少到5天，显著提升了训练效率，同时精度提升至72.7%(+1.2%)。
+
+
+
+### （2）PP-LCNetV3：精度更优的骨干网络
+
+PP-LCNetV3系列模型是PP-LCNet系列模型的延续，覆盖了更大的精度范围，能够适应不同下游任务的需要。PP-LCNetV3系列模型从多个方面进行了优化，提出了可学习仿射变换模块，对重参数化策略、激活函数进行了改进，同时调整了网络深度与宽度。最终，PP-LCNetV3系列模型能够在性能与效率之间达到最佳的平衡，在不同精度范围内取得极致的推理速度。
+
+### （3）Lite-Neck：精简参数的Neck结构
+
+Lite-Neck整体结构沿用PP-OCRv3版本的结构，在参数上稍作精简，识别模型整体的模型大小可从12M降低到8.5M，而精度不变；在CTCHead中，将Neck输出特征的维度从64提升到120，此时模型大小从8.5M提升到9.6M。
+
+### （4）GTC-NRTR：Attention指导CTC训练策略
+
+GTC（Guided Training of CTC），是PP-OCRv3识别模型的最有效的策略之一，融合多种文本特征的表达，有效的提升文本识别精度。在PP-OCRv4中使用训练更稳定的Transformer模型NRTR作为指导分支，相比V3版本中的SAR基于循环神经网络的结构，NRTR基于Transformer实现解码过程泛化能力更强，能有效指导CTC分支学习，解决简单场景下快速过拟合的问题。使用Lite-Neck和GTC-NRTR两个策略，识别精度提升至73.21%(+0.5%)。
+
+
+
+### （5）Multi-Scale：多尺度训练策略
+
+动态尺度训练策略，是在训练过程中随机resize输入图片的高度，以增强识别模型在端到端串联使用时的鲁棒性。在训练时，每个iter从（32，48，64）三种高度中随机选择一种高度进行resize。实验证明，使用该策略，尽管在识别测试集上准确率没有提升，但在端到端串联评估时，指标提升0.5%。
+
+
+
+### （6）DKD：蒸馏策略
+
+识别模型的蒸馏包含两个部分，NRTRhead蒸馏和CTCHead蒸馏;
+
+对于NRTR head，使用了DKD loss蒸馏，拉近学生模型和教师模型的NRTR head logits。最终NRTR head的loss是学生与教师间的DKD loss和与ground truth的cross entropy loss的加权和，用于监督学生模型的backbone训练。通过实验，我们发现加入DKD loss后，计算与ground truth的cross entropy loss时去除label smoothing可以进一步提高精度，因此我们在这里使用的是不带label smoothing的cross entropy loss。
+
+对于CTCHead，由于CTC的输出中存在Blank位，即使教师模型和学生模型的预测结果一样，二者的输出的logits分布也会存在差异，影响教师模型向学生模型的知识传递。PP-OCRv4识别模型蒸馏策略中，将CTC输出logits沿着文本长度维度计算均值，将多字符识别问题转换为多字符分类问题，用于监督CTC Head的训练。使用该策略融合NRTRhead DKD蒸馏策略，指标从74.72%提升到75.45%。
+
+## 4. 端到端评估
+
+经过以上优化，最终PP-OCRv4在速度可比情况下，中文场景端到端Hmean指标相比于PP-OCRv3提升4.5%，效果大幅提升。具体指标如下表所示：
+
+| Model | Hmean | Model Size (M) | Time Cost (CPU, ms) |
+|-----|-----|--------|----|
+| PP-OCRv3 | 57.99% | 15.6 | 78 |
+| PP-OCRv4 | 62.24% | 15.8 | 76 |
+
+测试环境：CPU型号为Intel Gold 6148，CPU预测时使用openvino。
+
+除了更新中文模型，本次升级也优化了英文数字模型，在自有评估集上文本识别准确率提升6%，如下表所示：
+
+| Model | ACC |
+|-----|-----|
+| PP-OCR_en | 54.38% |
+| PP-OCRv3_en | 64.04% |
+| PP-OCRv4_en | 70.1% |
+
+同时，对已支持的80余种语言识别模型进行了升级更新，在有评估集的四种语系识别准确率平均提升8%以上，如下表所示：
+
+| Model | 拉丁语系 | 阿拉伯语系 | 日语 | 韩语 |
+|-----|-----|--------|----| --- |
+| PP-OCR_mul | 69.60% | 40.50% | 38.50% | 55.40% |
+| PP-OCRv3_mul | 71.57%| 72.90% | 45.85% | 77.23% |
+| PP-OCRv4_mul | 80.00%| 75.48% | 56.50% | 83.25% |
diff --git a/docs/ppocr/blog/clone.en.md b/docs/ppocr/blog/clone.en.md
new file mode 100644
index 0000000000..f72ce0431c
--- /dev/null
+++ b/docs/ppocr/blog/clone.en.md
@@ -0,0 +1,31 @@
+---
+comments: true
+---
+
+# Project Clone
+
+## 1. Clone PaddleOCR
+
+```bash linenums="1"
+# Recommend
+git clone https://github.com/PaddlePaddle/PaddleOCR
+
+# If you cannot pull successfully due to network problems, you can switch to the mirror hosted on Gitee:
+
+git clone https://gitee.com/paddlepaddle/PaddleOCR
+
+# Note: The mirror on Gitee may not keep in synchronization with the latest project on GitHub. There might be a delay of 3-5 days. Please try GitHub at first.
+```
+
+## 2. Install third-party libraries
+
+```bash linenums="1"
+cd PaddleOCR
+pip3 install -r requirements.txt
+```
+
+If you getting this error `OSError: [WinError 126] The specified module could not be found` when you install shapely on windows.
+
+Please try to download Shapely whl file from [http://www.lfd.uci.edu/~gohlke/pythonlibs/#shapely](http://www.lfd.uci.edu/~gohlke/pythonlibs/#shapely).
+
+Reference: [Solve shapely installation on windows](https://stackoverflow.com/questions/44398265/install-shapely-oserror-winerror-126-the-specified-module-could-not-be-found)
diff --git a/docs/ppocr/blog/clone.md b/docs/ppocr/blog/clone.md
new file mode 100644
index 0000000000..efc4b658d7
--- /dev/null
+++ b/docs/ppocr/blog/clone.md
@@ -0,0 +1,26 @@
+---
+comments: true
+---
+
+# 项目克隆
+
+## 1. 克隆PaddleOCR repo代码
+
+```bash linenums="1"
+git clone https://github.com/PaddlePaddle/PaddleOCR
+```
+
+如果因为网络问题无法pull成功，也可选择使用码云上的托管：
+
+```bash linenums="1"
+git clone https://gitee.com/paddlepaddle/PaddleOCR
+```
+
+注：码云托管代码可能无法实时同步本github项目更新，存在3~5天延时，请优先使用推荐方式。
+
+## 2. 安装第三方库
+
+```bash linenums="1"
+cd PaddleOCR
+pip3 install -r requirements.txt
+```
diff --git a/docs/ppocr/blog/config.en.md b/docs/ppocr/blog/config.en.md
new file mode 100644
index 0000000000..21584b403a
--- /dev/null
+++ b/docs/ppocr/blog/config.en.md
@@ -0,0 +1,245 @@
+---
+comments: true
+---
+
+# Configuration
+
+## 1. Optional Parameter List
+
+The following list can be viewed through `--help`
+
+| FLAG | Supported script | Use | Defaults | Note |
+| :----------------------: | :------------: | :---------------: | :--------------: | :-----------------: |
+| -c | ALL | Specify configuration file to use | None | **Please refer to the parameter introduction for configuration file usage** |
+| -o | ALL | set configuration options | None | Configuration using -o has higher priority than the configuration file selected with -c. E.g: -o Global.use_gpu=false |
+
+## 2. Introduction to Global Parameters of Configuration File
+
+Take rec_chinese_lite_train_v2.0.yml as an example
+
+### Global
+
+| Parameter | Use | Defaults | Note |
+| :----------------------: | :---------------------: | :--------------: | :--------------------: |
+| use_gpu | Set using GPU or not | true | \ |
+| epoch_num | Maximum training epoch number | 500 | \ |
+| log_smooth_window | Log queue length, the median value in the queue each time will be printed | 20 | \ |
+| print_batch_step | Set print log interval | 10 | \ |
+| save_model_dir | Set model save path | output/{算法名称} | \ |
+| save_epoch_step | Set model save interval | 3 | \ |
+| eval_batch_step | Set the model evaluation interval | 2000 or [1000, 2000] | running evaluation every 2000 iters or evaluation is run every 2000 iterations after the 1000th iteration |
+| cal_metric_during_train | Set whether to evaluate the metric during the training process. At this time, the metric of the model under the current batch is evaluated | true | \ |
+| load_static_weights | Set whether the pre-training model is saved in static graph mode (currently only required by the detection algorithm) | true | \ |
+| pretrained_model | Set the path of the pre-trained model | ./pretrain_models/CRNN/best_accuracy | \ |
+| checkpoints | set model parameter path | None | Used to load parameters after interruption to continue training|
+| use_visualdl | Set whether to enable visualdl for visual log display | False | [Tutorial](https://www.paddlepaddle.org.cn/paddle/visualdl) |
+| use_wandb | Set whether to enable W&B for visual log display | False | [Documentation](https://docs.wandb.ai/)
+| infer_img | Set inference image path or folder path | ./infer_img | \||
+| character_dict_path | Set dictionary path | ./ppocr/utils/ppocr_keys_v1.txt | If the character_dict_path is None, model can only recognize number and lower letters |
+| max_text_length | Set the maximum length of text | 25 | \ |
+| use_space_char | Set whether to recognize spaces | True | \| |
+| label_list | Set the angle supported by the direction classifier | ['0','180'] | Only valid in angle classifier model |
+| save_res_path | Set the save address of the test model results | ./output/det_db/predicts_db.txt | Only valid in the text detection model |
+
+### Optimizer ([ppocr/optimizer](../../ppocr/optimizer))
+
+| Parameter | Use | Defaults | Note |
+| :---------------------: | :---------------------: | :--------------: | :--------------------: |
+| name | Optimizer class name | Adam | Currently supports`Momentum`,`Adam`,`RMSProp`, see [ppocr/optimizer/optimizer.py](../../ppocr/optimizer/optimizer.py) |
+| beta1 | Set the exponential decay rate for the 1st moment estimates | 0.9 | \ |
+| beta2 | Set the exponential decay rate for the 2nd moment estimates | 0.999 | \ |
+| clip_norm | The maximum norm value | - | \ |
+| **lr** | Set the learning rate decay method | - | \ |
+| name | Learning rate decay class name | Cosine | Currently supports`Linear`,`Cosine`,`Step`,`Piecewise`, see[ppocr/optimizer/learning_rate.py](../../ppocr/optimizer/learning_rate.py) |
+| learning_rate | Set the base learning rate | 0.001 | \ |
+| **regularizer** | Set network regularization method | - | \ |
+| name | Regularizer class name | L2 | Currently support`L1`,`L2`, see[ppocr/optimizer/regularizer.py](../../ppocr/optimizer/regularizer.py) |
+| factor | Regularizer coefficient | 0.00001 | \ |
+
+### Architecture ([ppocr/modeling](../../ppocr/modeling))
+
+In PaddleOCR, the network is divided into four stages: Transform, Backbone, Neck and Head
+
+| Parameter | Use | Defaults | Note |
+| :---------------------: | :---------------------: | :--------------: | :--------------------: |
+| model_type | Network Type | rec | Currently support`rec`,`det`,`cls` |
+| algorithm | Model name | CRNN | See [algorithm_overview](./algorithm_overview_en.md) for the support list |
+| **Transform** | Set the transformation method | - | Currently only recognition algorithms are supported, see [ppocr/modeling/transform](../../ppocr/modeling/transforms) for details |
+| name | Transformation class name | TPS | Currently supports `TPS` |
+| num_fiducial | Number of TPS control points | 20 | Ten on the top and bottom |
+| loc_lr | Localization network learning rate | 0.1 | \ |
+| model_name | Localization network size | small | Currently support`small`,`large` |
+| **Backbone** | Set the network backbone class name | - | see [ppocr/modeling/backbones](../../ppocr/modeling/backbones) |
+| name | backbone class name | ResNet | Currently support`MobileNetV3`,`ResNet` |
+| layers | resnet layers | 34 | Currently support18,34,50,101,152,200 |
+| model_name | MobileNetV3 network size | small | Currently support`small`,`large` |
+| **Neck** | Set network neck | - | see[ppocr/modeling/necks](../../ppocr/modeling/necks) |
+| name | neck class name | SequenceEncoder | Currently support`SequenceEncoder`,`DBFPN` |
+| encoder_type | SequenceEncoder encoder type | rnn | Currently support`reshape`,`fc`,`rnn` |
+| hidden_size | rnn number of internal units | 48 | \ |
+| out_channels | Number of DBFPN output channels | 256 | \ |
+| **Head** | Set the network head | - | see[ppocr/modeling/heads](../../ppocr/modeling/heads) |
+| name | head class name | CTCHead | Currently support`CTCHead`,`DBHead`,`ClsHead` |
+| fc_decay | CTCHead regularization coefficient | 0.0004 | \ |
+| k | DBHead binarization coefficient | 50 | \ |
+| class_dim | ClsHead output category number | 2 | \ |
+
+### Loss ([ppocr/losses](../../ppocr/losses))
+
+| Parameter | Use | Defaults | Note |
+| :---------------------: | :---------------------: | :--------------: | :--------------------: |
+| name | loss class name | CTCLoss | Currently support`CTCLoss`,`DBLoss`,`ClsLoss` |
+| balance_loss | Whether to balance the number of positive and negative samples in DBLossloss (using OHEM) | True | \ |
+| ohem_ratio | The negative and positive sample ratio of OHEM in DBLossloss | 3 | \ |
+| main_loss_type | The loss used by shrink_map in DBLossloss | DiceLoss | Currently support`DiceLoss`,`BCELoss` |
+| alpha | The coefficient of shrink_map_loss in DBLossloss | 5 | \ |
+| beta | The coefficient of threshold_map_loss in DBLossloss | 10 | \ |
+
+### PostProcess ([ppocr/postprocess](../../ppocr/postprocess))
+
+| Parameter | Use | Defaults | Note |
+| :---------------------: | :---------------------: | :--------------: | :--------------------: |
+| name | Post-processing class name | CTCLabelDecode | Currently support`CTCLoss`,`AttnLabelDecode`,`DBPostProcess`,`ClsPostProcess` |
+| thresh | The threshold for binarization of the segmentation map in DBPostProcess | 0.3 | \ |
+| box_thresh | The threshold for filtering output boxes in DBPostProcess. Boxes below this threshold will not be output | 0.7 | \ |
+| max_candidates | The maximum number of text boxes output in DBPostProcess | 1000 | |
+| unclip_ratio | The unclip ratio of the text box in DBPostProcess | 2.0 | \ |
+
+### Metric ([ppocr/metrics](../../ppocr/metrics))
+
+| Parameter | Use | Defaults | Note |
+| :---------------------: | :---------------------: | :--------------: | :--------------------: |
+| name | Metric method name | CTCLabelDecode | Currently support`DetMetric`,`RecMetric`,`ClsMetric` |
+| main_indicator | Main indicators, used to select the best model | acc | For the detection method is hmean, the recognition and classification method is acc |
+
+### Dataset ([ppocr/data](../../ppocr/data))
+
+| Parameter | Use | Defaults | Note |
+| :---------------------: | :---------------------: | :--------------: | :--------------------: |
+| **dataset** | Return one sample per iteration | - | - |
+| name | dataset class name | SimpleDataSet | Currently support`SimpleDataSet`,`LMDBDataSet` |
+| data_dir | Image folder path | ./train_data | \ |
+| label_file_list | Groundtruth file path | ["./train_data/train_list.txt"] | This parameter is not required when dataset is LMDBDataSet |
+| ratio_list | Ratio of data set | [1.0] | If there are two train_lists in label_file_list and ratio_list is [0.4,0.6], 40% will be sampled from train_list1, and 60% will be sampled from train_list2 to combine the entire dataset |
+| transforms | List of methods to transform images and labels | [DecodeImage,CTCLabelEncode,RecResizeImg,KeepKeys] | see[ppocr/data/imaug](../../ppocr/data/imaug) |
+| **loader** | dataloader related | - | |
+| shuffle | Does each epoch disrupt the order of the data set | True | \ |
+| batch_size_per_card | Single card batch size during training | 256 | \ |
+| drop_last | Whether to discard the last incomplete mini-batch because the number of samples in the data set cannot be divisible by batch_size | True | \ |
+| num_workers | The number of sub-processes used to load data, if it is 0, the sub-process is not started, and the data is loaded in the main process | 8 | \ |
+
+### Weights & Biases ([W&B](../../ppocr/utils/loggers/wandb_logger.py))
+
+| Parameter | Use | Defaults | Note |
+| :---------------------: | :---------------------: | :--------------: | :--------------------: |
+| project | Project to which the run is to be logged | uncategorized | \
+| name | Alias/Name of the run | Randomly generated by wandb | \
+| id | ID of the run | Randomly generated by wandb | \
+| entity | User or team to which the run is being logged | The logged in user | \
+| save_dir | local directory in which all the models and other data is saved | wandb | \
+| config | model configuration | None | \
+
+## 3. Multilingual Config File Generation
+
+PaddleOCR currently supports recognition for 80 languages (besides Chinese). A multi-language configuration file template is
+provided under the path `configs/rec/multi_languages`: [rec_multi_language_lite_train.yml](../../configs/rec/multi_language/rec_multi_language_lite_train.yml)。
+
+There are two ways to create the required configuration file:
+
+1. Automatically generated by script
+
+Script [generate_multi_language_configs.py](../../configs/rec/multi_language/generate_multi_language_configs.py) can help you generate configuration files for multi-language models.
+
+- Take Italian as an example, if your data is prepared in the following format:
+
+ ```text linenums="1"
+ |-train_data
+ |- it_train.txt # train_set label
+ |- it_val.txt # val_set label
+ |- data
+ |- word_001.jpg
+ |- word_002.jpg
+ |- word_003.jpg
+ | ...
+ ```
+
+ You can use the default parameters to generate a configuration file:
+
+ ```bash linenums="1"
+ # The code needs to be run in the specified directory
+ cd PaddleOCR/configs/rec/multi_language/
+ # Set the configuration file of the language to be generated through the -l or --language parameter.
+ # This command will write the default parameters into the configuration file
+ python3 generate_multi_language_configs.py -l it
+ ```
+
+- If your data is placed in another location, or you want to use your own dictionary, you can generate the configuration file by specifying the relevant parameters:
+
+ ```bash linenums="1"
+ # -l or --language field is required
+ # --train to modify the training set
+ # --val to modify the validation set
+ # --data_dir to modify the data set directory
+ # --dict to modify the dict path
+ # -o to modify the corresponding default parameters
+ cd PaddleOCR/configs/rec/multi_language/
+ python3 generate_multi_language_configs.py -l it \ # language
+ --train {path/of/train_label.txt} \ # path of train_label
+ --val {path/of/val_label.txt} \ # path of val_label
+ --data_dir {train_data/path} \ # root directory of training data
+ --dict {path/of/dict} \ # path of dict
+ -o Global.use_gpu=False # whether to use gpu
+ ...
+
+ ```
+
+Italian is made up of Latin letters, so after executing the command, you will get the rec_latin_lite_train.yml.
+
+2. Manually modify the configuration file
+
+ You can also manually modify the following fields in the template:
+
+ ```yaml linenums="1"
+ Global:
+ use_gpu: True
+ epoch_num: 500
+ ...
+ character_dict_path: {path/of/dict} # path of dict
+
+ Train:
+ dataset:
+ name: SimpleDataSet
+ data_dir: train_data/ # root directory of training data
+ label_file_list: ["./train_data/train_list.txt"] # train label path
+ ...
+
+ Eval:
+ dataset:
+ name: SimpleDataSet
+ data_dir: train_data/ # root directory of val data
+ label_file_list: ["./train_data/val_list.txt"] # val label path
+ ...
+
+ ```
+
+Currently, the multi-language algorithms supported by PaddleOCR are:
+
+| Configuration file | Algorithm name | backbone | trans | seq | pred | language |
+| :--------: | :-------: | :-------: | :-------: | :-----: | :-----: | :-----: |
+| rec_chinese_cht_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | chinese traditional |
+| rec_en_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | English(Case sensitive) |
+| rec_french_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | French |
+| rec_ger_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | German |
+| rec_japan_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | Japanese |
+| rec_korean_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | Korean |
+| rec_latin_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | Latin |
+| rec_arabic_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | arabic |
+| rec_cyrillic_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | cyrillic |
+| rec_devanagari_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | devanagari |
+
+For more supported languages, please refer to : [Multi-language model](./multi_languages.en.md)
+
+The multi-language model training method is the same as the Chinese model. The training data set is 100w synthetic data. A small amount of fonts and test data can be downloaded using the following two methods.
+
+- [Baidu Netdisk](https://pan.baidu.com/s/1bS_u207Rm7YbY33wOECKDA),Extraction code:frgi.
+- [Google drive](https://drive.google.com/file/d/18cSWX7wXSy4G0tbKJ0d9PuIaiwRLHpjA/view)
diff --git a/docs/ppocr/blog/config.md b/docs/ppocr/blog/config.md
new file mode 100644
index 0000000000..c68ebe75d2
--- /dev/null
+++ b/docs/ppocr/blog/config.md
@@ -0,0 +1,222 @@
+---
+comments: true
+---
+
+# 配置文件内容与生成
+
+## 1. 可选参数列表
+
+以下列表可以通过`--help`查看
+
+| FLAG | 支持脚本 | 用途 | 默认值 | 备注 |
+| :----------------------: | :------------: | :---------------: | :--------------: | :-----------------: |
+| -c | ALL | 指定配置文件 | None | **配置模块说明请参考 参数介绍** |
+| -o | ALL | 设置配置文件里的参数内容 | None | 使用-o配置相较于-c选择的配置文件具有更高的优先级。例如：`-o Global.use_gpu=false` |
+
+## 2. 配置文件参数介绍
+
+以 `rec_chinese_lite_train_v2.0.yml` 为例
+
+### Global
+
+| 字段 | 用途 | 默认值 | 备注 |
+| :----------------------: | :---------------------: | :--------------: | :--------------------: |
+| use_gpu | 设置代码是否在gpu运行 | true | \ |
+| epoch_num | 最大训练epoch数 | 500 | \ |
+| log_smooth_window | log队列长度，每次打印输出队列里的中间值 | 20 | \ |
+| print_batch_step | 设置打印log间隔 | 10 | \ |
+| save_model_dir | 设置模型保存路径 | output/{算法名称} | \ |
+| save_epoch_step | 设置模型保存间隔 | 3 | \ |
+| eval_batch_step | 设置模型评估间隔 | 2000 或 [1000, 2000] | 2000 表示每2000次迭代评估一次，[1000， 2000]表示从1000次迭代开始，每2000次评估一次 |
+| cal_metric_during_train | 设置是否在训练过程中评估指标，此时评估的是模型在当前batch下的指标 | true | \ |
+| load_static_weights | 设置预训练模型是否是静态图模式保存(目前仅检测算法需要) | true | \ |
+| pretrained_model | 设置加载预训练模型路径 | ./pretrain_models/CRNN/best_accuracy | \ |
+| checkpoints | 加载模型参数路径 | None | 用于中断后加载参数继续训练 |
+| use_visualdl | 设置是否启用visualdl进行可视化log展示 | False | [教程地址](https://www.paddlepaddle.org.cn/paddle/visualdl) |
+| infer_img | 设置预测图像路径或文件夹路径 | ./infer_img | \||
+| character_dict_path | 设置字典路径 | ./ppocr/utils/ppocr_keys_v1.txt | 如果为空，则默认使用小写字母+数字作为字典 |
+| max_text_length | 设置文本最大长度 | 25 | \ |
+| use_space_char | 设置是否识别空格 | True | \| |
+| label_list | 设置方向分类器支持的角度 | ['0','180'] | 仅在方向分类器中生效 |
+| save_res_path | 设置检测模型的结果保存地址 | ./output/det_db/predicts_db.txt | 仅在检测模型中生效 |
+
+### Optimizer ([ppocr/optimizer](../../ppocr/optimizer))
+
+| 字段 | 用途 | 默认值 | 备注 |
+| :---------------------: |:-------------:|:-------------:| :--------------------: |
+| name | 优化器类名 | Adam | 目前支持`Momentum`,`Adam`,`RMSProp`, 见[ppocr/optimizer/optimizer.py](../../ppocr/optimizer/optimizer.py) |
+| beta1 | 设置一阶矩估计的指数衰减率 | 0.9 | \ |
+| beta2 | 设置二阶矩估计的指数衰减率 | 0.999 | \ |
+| clip_norm | 所允许的二范数最大值 | | \ |
+| **lr** | 设置学习率decay方式 | - | \ |
+| name | 学习率decay类名 | Cosine | 目前支持`Linear`,`Cosine`,`Step`,`Piecewise`, 见[ppocr/optimizer/learning_rate.py](../../ppocr/optimizer/learning_rate.py) |
+| learning_rate | 基础学习率 | 0.001 | \ |
+| **regularizer** | 设置网络正则化方式 | - | \ |
+| name | 正则化类名 | L2 | 目前支持`L1`,`L2`, 见[ppocr/optimizer/regularizer.py](../../ppocr/optimizer/regularizer.py) |
+| factor | 正则化系数 | 0.00001 | \ |
+
+### Architecture ([ppocr/modeling](../../ppocr/modeling))
+
+在PaddleOCR中，网络被划分为Transform,Backbone,Neck和Head四个阶段
+
+| 字段 | 用途 | 默认值 | 备注 |
+| :---------------------: | :---------------------: | :--------------: | :--------------------: |
+| model_type | 网络类型 | rec | 目前支持`rec`,`det`,`cls` |
+| algorithm | 模型名称 | CRNN | 支持列表见[algorithm_overview](./algorithm_overview.md) |
+| **Transform** | 设置变换方式 | - | 目前仅rec类型的算法支持, 具体见[ppocr/modeling/transforms](../../ppocr/modeling/transforms) |
+| name | 变换方式类名 | TPS | 目前支持`TPS` |
+| num_fiducial | TPS控制点数 | 20 | 上下边各十个 |
+| loc_lr | 定位网络学习率 | 0.1 | \ |
+| model_name | 定位网络大小 | small | 目前支持`small`,`large` |
+| **Backbone** | 设置网络backbone类名 | - | 具体见[ppocr/modeling/backbones](../../ppocr/modeling/backbones) |
+| name | backbone类名 | ResNet | 目前支持`MobileNetV3`,`ResNet` |
+| layers | resnet层数 | 34 | 支持18,34,50,101,152,200 |
+| model_name | MobileNetV3 网络大小 | small | 支持`small`,`large` |
+| **Neck** | 设置网络neck | - | 具体见[ppocr/modeling/necks](../../ppocr/modeling/necks) |
+| name | neck类名 | SequenceEncoder | 目前支持`SequenceEncoder`,`DBFPN` |
+| encoder_type | SequenceEncoder编码器类型 | rnn | 支持`reshape`,`fc`,`rnn` |
+| hidden_size | rnn内部单元数 | 48 | \ |
+| out_channels | DBFPN输出通道数 | 256 | \ |
+| **Head** | 设置网络Head | - | 具体见[ppocr/modeling/heads](../../ppocr/modeling/heads) |
+| name | head类名 | CTCHead | 目前支持`CTCHead`,`DBHead`,`ClsHead` |
+| fc_decay | CTCHead正则化系数 | 0.0004 | \ |
+| k | DBHead二值化系数 | 50 | \ |
+| class_dim | ClsHead输出分类数 | 2 | \ |
+
+### Loss ([ppocr/losses](../../ppocr/losses))
+
+| 字段 | 用途 | 默认值 | 备注 |
+| :---------------------: | :---------------------: | :--------------: | :--------------------: |
+| name | 网络loss类名 | CTCLoss | 目前支持`CTCLoss`,`DBLoss`,`ClsLoss` |
+| balance_loss | DBLossloss中是否对正负样本数量进行均衡(使用OHEM) | True | \ |
+| ohem_ratio | DBLossloss中的OHEM的负正样本比例 | 3 | \ |
+| main_loss_type | DBLossloss中shrink_map所采用的loss | DiceLoss | 支持`DiceLoss`,`BCELoss` |
+| alpha | DBLossloss中shrink_map_loss的系数 | 5 | \ |
+| beta | DBLossloss中threshold_map_loss的系数 | 10 | \ |
+
+### PostProcess ([ppocr/postprocess](../../ppocr/postprocess))
+
+| 字段 | 用途 | 默认值 | 备注 |
+| :---------------------: | :---------------------: | :--------------: | :--------------------: |
+| name | 后处理类名 | CTCLabelDecode | 目前支持`CTCLoss`,`AttnLabelDecode`,`DBPostProcess`,`ClsPostProcess` |
+| thresh | DBPostProcess中分割图进行二值化的阈值 | 0.3 | \ |
+| box_thresh | DBPostProcess中对输出框进行过滤的阈值，低于此阈值的框不会输出 | 0.7 | \ |
+| max_candidates | DBPostProcess中输出的最大文本框数量 | 1000 | |
+| unclip_ratio | DBPostProcess中对文本框进行放大的比例 | 2.0 | \ |
+
+### Metric ([ppocr/metrics](../../ppocr/metrics))
+
+| 字段 | 用途 | 默认值 | 备注 |
+| :---------------------: | :---------------------: | :--------------: | :--------------------: |
+| name | 指标评估方法名称 | CTCLabelDecode | 目前支持`DetMetric`,`RecMetric`,`ClsMetric` |
+| main_indicator | 主要指标,用于选取最优模型 | acc | 对于检测方法为hmean，识别和分类方法为acc |
+
+### Dataset ([ppocr/data](../../ppocr/data))
+
+| 字段 | 用途 | 默认值 | 备注 |
+| :---------------------: | :---------------------: | :--------------: | :--------------------: |
+| **dataset** | 每次迭代返回一个样本 | - | - |
+| name | dataset类名 | SimpleDataSet | 目前支持`SimpleDataSet`和`LMDBDataSet` |
+| data_dir | 数据集图片存放路径 | ./train_data | \ |
+| label_file_list | 数据标签路径 | ["./train_data/train_list.txt"] | dataset为LMDBDataSet时不需要此参数 |
+| ratio_list | 数据集的比例 | [1.0] | 若label_file_list中有两个train_list，且ratio_list为[0.4,0.6]，则从train_list1中采样40%，从train_list2中采样60%组合整个dataset |
+| transforms | 对图片和标签进行变换的方法列表 | [DecodeImage,CTCLabelEncode,RecResizeImg,KeepKeys] | 见[ppocr/data/imaug](../../ppocr/data/imaug) |
+| **loader** | dataloader相关 | - | |
+| shuffle | 每个epoch是否将数据集顺序打乱 | True | \ |
+| batch_size_per_card | 训练时单卡batch size | 256 | \ |
+| drop_last | 是否丢弃因数据集样本数不能被 batch_size 整除而产生的最后一个不完整的mini-batch | True | \ |
+| num_workers | 用于加载数据的子进程个数，若为0即为不开启子进程，在主进程中进行数据加载 | 8 | \ |
+
+## 3. 多语言配置文件生成
+
+PaddleOCR目前已支持80种（除中文外）语种识别，`configs/rec/multi_languages` 路径下提供了一个多语言的配置文件模版: [rec_multi_language_lite_train.yml](../../configs/rec/multi_language/rec_multi_language_lite_train.yml)。
+
+您有两种方式创建所需的配置文件：
+
+1. 通过脚本自动生成
+
+[generate_multi_language_configs.py](../../configs/rec/multi_language/generate_multi_language_configs.py) 可以帮助您生成多语言模型的配置文件
+
+* 以意大利语为例，如果您的数据是按如下格式准备的：
+
+ ```text linenums="1"
+ |-train_data
+ |- it_train.txt # 训练集标签
+ |- it_val.txt # 验证集标签
+ |- data
+ |- word_001.jpg
+ |- word_002.jpg
+ |- word_003.jpg
+ | ...
+ ```
+
+ 可以使用默认参数，生成配置文件：
+
+ ```bash linenums="1"
+ # 该代码需要在指定目录运行
+ cd PaddleOCR/configs/rec/multi_language/
+ # 通过-l或者--language参数设置需要生成的语种的配置文件，该命令会将默认参数写入配置文件
+ python3 generate_multi_language_configs.py -l it
+ ```
+
+* 如果您的数据放置在其他位置，或希望使用自己的字典，可以通过指定相关参数来生成配置文件:
+
+ ```bash linenums="1"
+ # -l或者--language字段是必须的
+ # --train修改训练集，--val修改验证集，--data_dir修改数据集目录，--dict修改字典路径， -o修改对应默认参数
+ cd PaddleOCR/configs/rec/multi_language/
+ python3 generate_multi_language_configs.py -l it \ # 语种
+ --train {path/of/train_label.txt} \ # 训练标签文件的路径
+ --val {path/of/val_label.txt} \ # 验证集标签文件的路径
+ --data_dir {train_data/path} \ # 训练数据的根目录
+ --dict {path/of/dict} \ # 字典文件路径
+ -o Global.use_gpu=False # 是否使用gpu
+ ...
+
+ ```
+
+意大利文由拉丁字母组成，因此执行完命令后会得到名为 rec_latin_lite_train.yml 的配置文件。
+
+2. 手动修改配置文件
+
+ 您也可以手动修改模版中的以下几个字段得到配置文件:
+
+ ```yaml linenums="1"
+ Global:
+ use_gpu: True
+ epoch_num: 500
+ ...
+ character_dict_path: {path/of/dict} # 字典文件所在路径
+
+ Train:
+ dataset:
+ name: SimpleDataSet
+ data_dir: train_data/ # 数据存放根目录
+ label_file_list: ["./train_data/train_list.txt"] # 训练集label路径
+ ...
+
+ Eval:
+ dataset:
+ name: SimpleDataSet
+ data_dir: train_data/ # 数据存放根目录
+ label_file_list: ["./train_data/val_list.txt"] # 验证集label路径
+ ...
+
+ ```
+
+目前PaddleOCR支持的多语言算法有：
+
+| 配置文件 | 算法名称 | backbone | trans | seq | pred | language |
+| :--------: | :-------: | :-------: | :-------: | :-----: | :-----: | :-----: |
+| rec_chinese_cht_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 中文繁体 |
+| rec_en_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 英语（区分大小写） |
+| rec_french_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 法语 |
+| rec_ger_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 德语 |
+| rec_japan_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 日语 |
+| rec_korean_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 韩语 |
+| rec_latin_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 拉丁字母 |
+| rec_arabic_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 阿拉伯字母 |
+| rec_cyrillic_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 斯拉夫字母 |
+| rec_devanagari_lite_train.yml | CRNN | Mobilenet_v3 small 0.5 | None | BiLSTM | ctc | 梵文字母 |
+
+更多支持语种请参考: [多语言模型](./multi_languages.md)
diff --git a/docs/ppocr/blog/customize.en.md b/docs/ppocr/blog/customize.en.md
new file mode 100644
index 0000000000..9385af7115
--- /dev/null
+++ b/docs/ppocr/blog/customize.en.md
@@ -0,0 +1,39 @@
+---
+comments: true
+---
+
+# HOW TO MAKE YOUR OWN LIGHTWEIGHT OCR MODEL?
+
+The process of making a customized ultra-lightweight OCR models can be divided into three steps: training text detection model, training text recognition model, and concatenate the predictions from previous steps.
+
+## STEP1: TRAIN TEXT DETECTION MODEL
+
+PaddleOCR provides two text detection algorithms: EAST and DB. Both support MobileNetV3 and ResNet50_vd backbone networks, select the corresponding configuration file as needed and start training. For example, to train with MobileNetV3 as the backbone network for DB detection model :
+
+```bash linenums="1"
+python3 tools/train.py -c configs/det/det_mv3_db.yml 2>&1 | tee det_db.log
+```
+
+For more details about data preparation and training tutorials, refer to the documentation [Text detection model training/evaluation/prediction](../model_train/detection.en.md)
+
+## STEP2: TRAIN TEXT RECOGNITION MODEL
+
+PaddleOCR provides four text recognition algorithms: CRNN, Rosetta, STAR-Net, and RARE. They all support two backbone networks: MobileNetV3 and ResNet34_vd, select the corresponding configuration files as needed to start training. For example, to train a CRNN recognition model that uses MobileNetV3 as the backbone network:
+
+```bash linenums="1"
+python3 tools/train.py -c configs/rec/rec_chinese_lite_train.yml 2>&1 | tee rec_ch_lite.log
+```
+
+For more details about data preparation and training tutorials, refer to the documentation [Text recognition model training/evaluation/prediction](../model_train/recognition.en.md)
+
+## STEP3: CONCATENATE PREDICTIONS
+
+PaddleOCR provides a concatenation tool for detection and recognition models, which can connect any trained detection model and any recognition model into a two-stage text recognition system. The input image goes through four main stages: text detection, text rectification, text recognition, and score filtering to output the text position and recognition results, and at the same time, you can choose to visualize the results.
+
+When performing prediction, you need to specify the path of a single image or a image folder through the parameter `image_dir`, the parameter `det_model_dir` specifies the path of detection model, and the parameter `rec_model_dir` specifies the path of recognition model. The visualized results are saved to the `./inference_results` folder by default.
+
+```bash linenums="1"
+python3 tools/infer/predict_system.py --image_dir="./doc/imgs/11.jpg" --det_model_dir="./inference/det/" --rec_model_dir="./inference/rec/"
+```
+
+For more details about text detection and recognition concatenation, please refer to the document [Inference](../infer_deploy/python_infer.en.md)
diff --git a/docs/ppocr/blog/customize.md b/docs/ppocr/blog/customize.md
new file mode 100644
index 0000000000..cf72b2d173
--- /dev/null
+++ b/docs/ppocr/blog/customize.md
@@ -0,0 +1,39 @@
+---
+comments: true
+---
+
+# 如何生产自定义超轻量模型？
+
+生产自定义的超轻量模型可分为三步：训练文本检测模型、训练文本识别模型、模型串联预测。
+
+## step1：训练文本检测模型
+
+PaddleOCR提供了EAST、DB两种文本检测算法，均支持MobileNetV3、ResNet50_vd两种骨干网络，根据需要选择相应的配置文件，启动训练。例如，训练使用MobileNetV3作为骨干网络的DB检测模型（即超轻量模型使用的配置）：
+
+```bash linenums="1"
+python3 tools/train.py -c configs/det/det_mv3_db.yml 2>&1 | tee det_db.log
+```
+
+更详细的数据准备和训练教程参考文档教程中[文本检测模型训练/评估/预测](../model_train/detection.md)。
+
+## step2：训练文本识别模型
+
+PaddleOCR提供了CRNN、Rosetta、STAR-Net、RARE四种文本识别算法，均支持MobileNetV3、ResNet34_vd两种骨干网络，根据需要选择相应的配置文件，启动训练。例如，训练使用MobileNetV3作为骨干网络的CRNN识别模型（即超轻量模型使用的配置）：
+
+```bash linenums="1"
+python3 tools/train.py -c configs/rec/rec_chinese_lite_train.yml 2>&1 | tee rec_ch_lite.log
+```
+
+更详细的数据准备和训练教程参考文档教程中[文本识别模型训练/评估/预测](../model_train/recognition.md)。
+
+## step3：模型串联预测
+
+PaddleOCR提供了检测和识别模型的串联工具，可以将训练好的任一检测模型和任一识别模型串联成两阶段的文本识别系统。输入图像经过文本检测、检测框矫正、文本识别、得分过滤四个主要阶段输出文本位置和识别结果，同时可选择对结果进行可视化。
+
+在执行预测时，需要通过参数image_dir指定单张图像或者图像集合的路径、参数det_model_dir指定检测inference模型的路径和参数rec_model_dir指定识别inference模型的路径。可视化识别结果默认保存到 ./inference_results 文件夹里面。
+
+```bash linenums="1"
+python3 tools/infer/predict_system.py --image_dir="./doc/imgs/11.jpg" --det_model_dir="./inference/det/" --rec_model_dir="./inference/rec/"
+```
+
+更多的文本检测、识别串联推理使用方式请参考文档教程中的[基于预测引擎推理](../infer_deploy/python_infer.md)。
diff --git a/docs/ppocr/blog/distributed_training.en.md b/docs/ppocr/blog/distributed_training.en.md
new file mode 100644
index 0000000000..dd225758a4
--- /dev/null
+++ b/docs/ppocr/blog/distributed_training.en.md
@@ -0,0 +1,65 @@
+---
+comments: true
+---
+
+# Distributed training
+
+## Introduction
+
+The high performance of distributed training is one of the core advantages of PaddlePaddle. In the classification task, distributed training can achieve almost linear speedup ratio. Generally, OCR training task need massive training data. Such as recognition, PP-OCR v2.0 model is trained based on 1800W dataset, which is very time-consuming if using single machine. Therefore, the distributed training is used in PaddleOCR to speedup the training task. For more information about distributed training, please refer to [distributed training quick start tutorial](https://fleet-x.readthedocs.io/en/latest/paddle_fleet_rst/parameter_server/ps_quick_start.html).
+
+## Quick Start
+
+### Training with single machine
+
+Take recognition as an example. After the data is prepared locally, start the training task with the interface of `paddle.distributed.launch`. The start command as follows:
+
+```bash linenums="1"
+python3 -m paddle.distributed.launch \
+ --log_dir=./log/ \
+ --gpus "0,1,2,3,4,5,6,7" \
+ tools/train.py \
+ -c configs/rec/rec_mv3_none_bilstm_ctc.yml
+```
+
+### Training with multi machine
+
+Compared with single machine, training with multi machine only needs to add the parameter `--ips` to start command, which represents the IP list of machines used for distributed training, and the IP of different machines are separated by commas. The start command as follows:
+
+```bash linenums="1"
+ip_list="192.168.0.1,192.168.0.2"
+python3 -m paddle.distributed.launch \
+ --log_dir=./log/ \
+ --ips="${ip_list}" \
+ --gpus="0,1,2,3,4,5,6,7" \
+ tools/train.py \
+ -c configs/rec/rec_mv3_none_bilstm_ctc.yml
+```
+
+**Notice:**
+
+* The IP addresses of different machines need to be separated by commas, which can be queried through `ifconfig` or `ipconfig`.
+* Different machines need to be set to be secret free and can `ping` success with others directly, otherwise communication cannot establish between them.
+* The code, data and start command between different machines must be completely consistent and then all machines need to run start command. The first machine in the `ip_list` is set to `trainer0`, and so on.
+
+## Performance comparison
+
+We conducted model training on 2x8 P40 GPUs. Accuracy, training time, and multi machine acceleration ratio of different models are shown below.
+
+| Model | Configuration | Configuration | 8 GPU training time / Accuracy | 3x8 GPU training time / Accuracy | Acceleration ratio |
+|:------:|:-----:|:--------:|:--------:|:--------:|:-----:|
+| CRNN | [rec_chinese_lite_train_v2.0.yml](../../configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml) | 260k Chinese dataset | 2.50d/66.70% | 1.67d/67.00% | **1.5** |
+
+We conducted model training on 3x8 V100 GPUs. Accuracy, training time, and multi machine acceleration ratio of different models are shown below.
+
+| Model | Configuration | Configuration | 8 GPU training time / Accuracy | 3x8 GPU training time / Accuracy | Acceleration ratio |
+|:------:|:-----:|:--------:|:--------:|:--------:|:-----:|
+| SLANet | [SLANet.yml](../../configs/table/SLANet.yml) | PubTabNet | 49.80h/76.20% | 19.75h/74.77% | **2.52** |
+
+Note: when training with 3x8 GPUs, the single card batch size is unchanged compared with the 1x8 GPUs' training process, and the learning rate is multiplied by 2 (if it is multiplied by 3 by default, the accuracy is only 73.42%).
+
+We conducted model training on 4x8 V100 GPUs. Accuracy, training time, and multi machine acceleration ratio of different models are shown below.
+
+| Model | Configuration | Configuration | 8 GPU training time / Accuracy | 4x8 GPU training time / Accuracy | Acceleration ratio |
+|:------:|:-----:|:--------:|:--------:|:--------:|:-----:|
+| SVTR | [ch_PP-OCRv3_rec_distillation.yml](../../configs/rec/PP-OCRv3/ch_PP-OCRv3_rec_distillation.yml) | PP-OCRv3_rec data | 10d/- | 2.84d/74.00% | **3.5** |
diff --git a/docs/ppocr/blog/distributed_training.md b/docs/ppocr/blog/distributed_training.md
new file mode 100644
index 0000000000..182c67d17c
--- /dev/null
+++ b/docs/ppocr/blog/distributed_training.md
@@ -0,0 +1,67 @@
+---
+comments: true
+---
+
+# 分布式训练
+
+## 简介
+
+* 分布式训练的高性能，是飞桨的核心优势技术之一，在分类任务上，分布式训练可以达到几乎线性的加速比。OCR训练任务中往往包含大量训练数据，以识别为例，ppocrv2.0模型在训练时使用了1800W数据，如果使用单机训练，会非常耗时。因此，PaddleOCR中使用分布式训练接口完成训练任务，同时支持单机训练与多机训练。更多关于分布式训练的方法与文档可以参考：[分布式训练快速开始教程](https://fleet-x.readthedocs.io/en/latest/paddle_fleet_rst/parameter_server/ps_quick_start.html)。
+
+## 使用方法
+
+### 单机训练
+
+* 以识别为例，本地准备好数据之后，使用`paddle.distributed.launch`的接口启动训练任务即可。下面为运行代码示例。
+
+```bash linenums="1"
+python3 -m paddle.distributed.launch \
+ --log_dir=./log/ \
+ --gpus "0,1,2,3,4,5,6,7" \
+ tools/train.py \
+ -c configs/rec/rec_mv3_none_bilstm_ctc.yml
+```
+
+### 多机训练
+
+* 相比单机训练，多机训练时，只需要添加`--ips`的参数，该参数表示需要参与分布式训练的机器的ip列表，不同机器的ip用逗号隔开。下面为运行代码示例。
+
+```bash linenums="1"
+ip_list="192.168.0.1,192.168.0.2"
+python3 -m paddle.distributed.launch \
+ --log_dir=./log/ \
+ --ips="${ip_list}" \
+ --gpus="0,1,2,3,4,5,6,7" \
+ tools/train.py \
+ -c configs/rec/rec_mv3_none_bilstm_ctc.yml
+```
+
+**注：**
+
+* 不同机器的ip信息需要用逗号隔开，可以通过`ifconfig`或者`ipconfig`查看。
+* 不同机器之间需要做免密设置，且可以直接ping通，否则无法完成通信。
+* 不同机器之间的代码、数据与运行命令或脚本需要保持一致，且所有的机器上都需要运行设置好的训练命令或者脚本。最终`ip_list`中的第一台机器的第一块设备是trainer0，以此类推。
+
+## 性能效果测试
+
+在2机8卡P40的机器上进行模型训练，不同模型的精度、训练耗时、多机加速比情况如下所示。
+
+| 模型 | 配置 | 数据集 | 单机8卡耗时/精度 | 2机8卡耗时/精度 | 加速比 |
+|:------:|:-----:|:--------:|:--------:|:--------:|:-----:|
+| CRNN | [rec_chinese_lite_train_v2.0.yml](../../configs/rec/ch_ppocr_v2.0/rec_chinese_lite_train_v2.0.yml) | 26W中文数据集 | 2.50d/66.7% | 1.67d/67.0% | **1.5** |
+
+在3机8卡V100的机器上进行模型训练，不同模型的精度、训练耗时、多机加速比情况如下所示。
+
+| 模型 | 配置 | 数据集 | 单机8卡耗时/精度 | 3机8卡耗时/精度 | 加速比 |
+|:------:|:-----:|:--------:|:--------:|:--------:|:-----:|
+| SLANet | [SLANet.yml](../../configs/table/SLANet.yml) | PubTabNet | 49.8h/76.2% | 19.75h/74.77% | **2.52** |
+
+注意：这里3机8卡训练时，单卡batch size相比于单机8卡不变，学习率乘以2 (默认乘以3的话，精度仅有73.42%)
+
+在4机8卡V100的机器上进行模型训练，不同模型的精度、训练耗时、多机加速比情况如下所示。
+
+| 模型 | 配置 | 数据集 | 单机8卡耗时/精度 | 4机8卡耗时/精度 | 加速比 |
+|:------:|:-----:|:--------:|:--------:|:--------:|:-----:|
+| SVTR | [ch_PP-OCRv3_rec_distillation.yml](../../configs/rec/PP-OCRv3/ch_PP-OCRv3_rec_distillation.yml) | PP-OCRv3_rec data | 10d/- | 2.84d/74.0% | **3.5** |
+
+**注意**: 在训练的GPU卡数过多时，精度会稍微有所损失（1%左右），此时可以尝试通过添加warmup或者适当增加迭代轮数来弥补精度损失。
diff --git a/docs/ppocr/blog/enhanced_ctc_loss.en.md b/docs/ppocr/blog/enhanced_ctc_loss.en.md
new file mode 100644
index 0000000000..0bd4be56d5
--- /dev/null
+++ b/docs/ppocr/blog/enhanced_ctc_loss.en.md
@@ -0,0 +1,100 @@
+---
+comments: true
+typora-copy-images-to: images
+---
+
+# Enhanced CTC Loss
+
+In OCR recognition, CRNN is a text recognition algorithm widely applied in the industry. In the training phase, it uses CTCLoss to calculate the network loss. In the inference phase, it uses CTCDecode to obtain the decoding result. Although the CRNN algorithm has been proven to achieve reliable recognition results in actual business, users have endless requirements for recognition accuracy. So how to improve the accuracy of text recognition? Taking CTCLoss as the starting point, this paper explores the improved fusion scheme of CTCLoss from three different perspectives: Hard Example Mining, Multi-task Learning, and Metric Learning. Based on the exploration, we propose EnhancedCTCLoss, which includes the following 3 components: Focal-CTC Loss, A-CTC Loss, C-CTC Loss.
+
+## 1. Focal-CTC Loss
+
+Focal Loss was proposed by the paper, "[Focal Loss for Dense Object Detection](https://arxiv.org/abs/1708.02002)". When the loss was first proposed, it was mainly to solve the problem of a serious imbalance in the ratio of positive and negative samples in one-stage target detection. This loss function reduces the weight of a large number of simple negative samples in training and also can be understood as a kind of difficult sample mining.
+The form of the loss function is as follows:
+
+$$
+\begin{equation}
+L_{fl}=\left\{
+\begin{array}{cl}
+-\alpha(1 - y^{'})^{\gamma}logy^{'} ,& y = 1 \\
+-(1 - \alpha)y^{'\gamma}log(1 - y^{'}), & y = 0 \\
+\end{array} \right.
+\end{equation}
+$$
+
+Among them, y' is the output of the activation function, and the value is between 0-1. It adds a modulation factor (1-y’)^γ and a balance factor α on the basis of the original cross-entropy loss. When α = 1, y = 1, the comparison between the loss function and the cross-entropy loss is shown in the following figure:
+
+
+
+As can be seen from the above figure, when γ > 0, the adjustment coefficient (1-y’)^γ gives smaller weight to the easy-to-classify sample loss, making the network pay more attention to the difficult and misclassified samples. The adjustment factor γ is used to adjust the rate at which the weight of simple samples decreases. When γ = 0, it is the cross-entropy loss function. When γ increases, the influence of the adjustment factor will also increase. Experiments revealed that 2 is the optimal value of γ. The balance factor α is used to balance the uneven proportions of the positive and negative samples. In the text, α is taken as 0.25.
+
+For the classic CTC algorithm, suppose a certain feature sequence (f1, f2, ......ft), after CTC decoding, the probability that the result is equal to label is y', then the probability that the CTC decoding result is not equal to label is (1-y'); it is not difficult to find that the CTCLoss value and y' have the following relationship:
+
+$$
+L_{CTC} = -log(y^{'})
+$$
+
+Combining the idea of Focal Loss, assigning larger weights to difficult samples and smaller weights to simple samples can make the network focus more on the mining of difficult samples and further improve the accuracy of recognition. Therefore, we propose Focal-CTC Loss. Its definition is as follows:
+
+$$
+L_{Focal\_CTC} = \alpha * (1 - y^{'})^{\gamma} * L_{CTC}
+$$
+
+In the experiment, the value of γ is 2, α = 1, see this for specific implementation: [rec_ctc_loss.py](../../ppocr/losses/rec_ctc_loss.py)
+
+## 2. A-CTC Loss
+A-CTC Loss is short for CTC Loss + ACE Loss. Among them, ACE Loss was proposed by the paper, “[Aggregation Cross-Entropy for Sequence Recognition](https://arxiv.org/abs/1904.08364)”. Compared with CTCLoss, ACE Loss has the following two advantages:
++ ACE Loss can solve the recognition problem of 2-D text, while CTCLoss can only process 1-D text
++ ACE Loss is better than CTC loss in time complexity and space complexity
+
+The advantages and disadvantages of the OCR recognition algorithm summarized by the predecessors are shown in the following figure:
+
+
+
+Although ACELoss does handle 2D predictions, as shown in the figure above, and has advantages in memory usage and inference speed, in practice, we found that using ACELoss alone, the recognition effect is not as good as CTCLoss. Consequently, we tried to combine CTCLoss and ACELoss, and CTCLoss is the mainstay while ACELoss acts as an auxiliary supervision loss. This attempt has achieved better results. On our internal experimental data set, compared to using CTCLoss alone, the recognition accuracy can be improved by about 1%.
+A_CTC Loss is defined as follows:
+
+$$
+L_{A-CTC} = L_{CTC} + \lambda * L_{ACE}
+$$
+
+In the experiment, λ = 0.1. See the ACE loss implementation code: [ace_loss.py](../../ppocr/losses/ace_loss.py)
+
+## 3. C-CTC Loss
+C-CTC Loss is short for CTC Loss + Center Loss. Among them, Center Loss was proposed by the paper, “[A Discriminative Feature Learning Approach for Deep Face Recognition](https://link.springer.com/chapter/10.1007/978-3-319-46478-7_31)“. It was first used in face recognition tasks to increase the distance between classes and reduce the distance within classes. It is an earlier and also widely used algorithm.
+
+In the task of Chinese OCR recognition, through the analysis of bad cases, we found that a major difficulty in Chinese recognition is that there are many similar characters, which are easy to misunderstand. From this, we thought about whether we can learn from the idea of n to increase the class spacing of similar characters, to improve recognition accuracy. However, Metric Learning is mainly used in the field of image recognition, and the label of the training data is a fixed value; for OCR recognition, it is a sequence recognition task essentially, and there is no explicit alignment between features and labels. Therefore, how to combine the two is still a direction worth exploring.
+
+By trying Arcmargin, Cosmargin and other methods, we finally found that Centerloss can help further improve the accuracy of recognition. C_CTC Loss is defined as follows:
+
+$$
+L_{C-CTC} = L_{CTC} + \lambda * L_{center}
+$$
+
+In the experiment, we set λ=0.25. See the center_loss implementation code: [center_loss.py](../../ppocr/losses/center_loss.py)
+
+It is worth mentioning that in C-CTC Loss, choosing to initialize the Center randomly does not bring significant improvement. Our Center initialization method is as follows:
++ Based on the original CTCLoss, a network N is obtained by training
++ Select the training set, identify the completely correct part, and form the set G
++ Send each sample in G to the network, perform forward calculation, and extract the correspondence between the input of the last FC layer (ie feature) and the result of argmax calculation (ie index)
++ Aggregate features with the same index, calculate the average, and get the initial center of each character.
+
+Taking the configuration file `configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec.yml` as an example, the center extraction command is as follows:
+
+```bash linenums="1"
+python tools/export_center.py -c configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec.yml -o Global.pretrained_model="./output/rec_mobile_pp-OCRv2/best_accuracy"
+```
+
+After running, `train_center.pkl` will be generated in the main directory of PaddleOCR.
+
+## 4. Experiment
+
+For the above three solutions, we conducted training and evaluation based on Baidu's internal data set. The experimental conditions are shown in the following table:
+
+| algorithm | Focal_CTC | A_CTC | C-CTC |
+| :-------- | :-------- | ----: | :---: |
+| gain | +0.3% | +0.7% | +1.7% |
+
+Based on the above experimental conclusions, we adopted the C-CTC strategy in PP-OCRv2. It is worth mentioning that, because PP-OCRv2 deals with the recognition task of 6625 Chinese characters, the character set is relatively large and there are many similar characters, so the C-CTC solution brings a significant improvement on this task. But if you switch to other OCR recognition tasks, the conclusion may be different. You can try Focal-CTC, A-CTC, C-CTC, and the combined solution EnhancedCTC. We believe it will bring different degrees of improvement.
+
+The unified combined plan is shown in the following file: [rec_enhanced_ctc_loss.py](../../ppocr/losses/rec_enhanced_ctc_loss.py)
diff --git a/docs/ppocr/blog/enhanced_ctc_loss.md b/docs/ppocr/blog/enhanced_ctc_loss.md
new file mode 100644
index 0000000000..5c28c306c8
--- /dev/null
+++ b/docs/ppocr/blog/enhanced_ctc_loss.md
@@ -0,0 +1,101 @@
+---
+comments: true
+typora-copy-images-to: images
+---
+
+# Enhanced CTC Loss
+
+在OCR识别中， CRNN是一种在工业界广泛使用的文字识别算法。 在训练阶段，其采用CTCLoss来计算网络损失； 在推理阶段，其采用CTCDecode来获得解码结果。虽然CRNN算法在实际业务中被证明能够获得很好的识别效果， 然而用户对识别准确率的要求却是无止境的，如何进一步提升文字识别的准确率呢？ 本文以CTCLoss为切人点，分别从难例挖掘、 多任务学习、 Metric Learning 3个不同的角度探索了CTCLoss的改进融合方案，提出了EnhancedCTCLoss，其包括如下3个组成部分： Focal-CTC Loss，A-CTC Loss， C-CTC Loss。
+
+## 1. Focal-CTC Loss
+
+Focal Loss 出自论文《Focal Loss for Dense Object Detection》, 该loss最先提出的时候主要是为了解决one-stage目标检测中正负样本比例严重失衡的问题。该损失函数降低了大量简单负样本在训练中所占的权重，也可理解为一种困难样本挖掘。
+其损失函数形式如下：
+
+$$
+\begin{equation}
+L_{fl}=\left\{
+\begin{array}{cl}
+-\alpha(1 - y^{'})^{\gamma}logy^{'} ,& y = 1 \\
+-(1 - \alpha)y^{'\gamma}log(1 - y^{'}), & y = 0 \\
+\end{array} \right.
+\end{equation}
+$$
+
+其中, y' 是经过激活函数的输出，取值在0-1之间。其在原始的交叉熵损失的基础上加了一个调制系数（1 – y’)^ γ和平衡因子α。 当α = 1，y=1时，其损失函数与交叉熵损失的对比如下图所示:
+
+
+
+从上图可以看到, 当γ> 0时，调整系数（1-y’）^γ 赋予易分类样本损失一个更小的权重，使得网络更关注于困难的、错分的样本。 调整因子γ用于调节简单样本权重降低的速率，当γ为0时即为交叉熵损失函数，当γ增加时，调整因子的影响也会随之增大。实验发现γ为2是最优。平衡因子α用来平衡正负样本本身的比例不均，文中α取0.25。
+
+对于经典的CTC算法，假设某个特征序列（f1, f2, ......ft), 经过CTC解码之后结果等于label的概率为y’, 则CTC解码结果不为label的概率即为（1-y’)；不难发现, CTCLoss值和y’有如下关系：
+
+$$
+L_{CTC} = -log(y^{'})
+$$
+
+结合Focal Loss的思想，赋予困难样本较大的权重，简单样本较小的权重，可以使网络更加聚焦于对困难样本的挖掘，进一步提升识别的准确率，由此我们提出了Focal-CTC Loss； 其定义如下所示：
+
+$$
+L_{Focal\_CTC} = \alpha * (1 - y^{'})^{\gamma} * L_{CTC}
+$$
+
+实验中，γ取值为2, α= 1, 具体实现见: [rec_ctc_loss.py](../../ppocr/losses/rec_ctc_loss.py)
+
+## 2. A-CTC Loss
+
+A-CTC Loss是CTC Loss + ACE Loss的简称。 其中ACE Loss出自论文< Aggregation Cross-Entropy for Sequence Recognition>. ACE Loss相比于CTCLoss，主要有如下两点优势:
+
++ ACE Loss能够解决2-D文本的识别问题; CTCLoss只能够处理1-D文本
++ ACE Loss 在时间复杂度和空间复杂度上优于CTC loss
+
+前人总结的OCR识别算法的优劣如下图所示：
+
+
+
+虽然ACELoss确实如上图所说，可以处理2D预测，在内存占用及推理速度方面具备优势，但在实践过程中，我们发现单独使用ACE Loss, 识别效果并不如CTCLoss. 因此，我们尝试将CTCLoss和ACELoss进行结合，同时以CTCLoss为主，将ACELoss 定位为一个辅助监督loss。 这一尝试收到了效果，在我们内部的实验数据集上，相比单独使用CTCLoss，识别准确率可以提升1%左右。
+A_CTC Loss定义如下:
+
+$$
+L_{A-CTC} = L_{CTC} + \lambda * L_{ACE}
+$$
+
+实验中，λ = 0.1. ACE loss实现代码见: [ace_loss.py](../../ppocr/losses/ace_loss.py)
+
+## 3. C-CTC Loss
+
+C-CTC Loss是CTC Loss + Center Loss的简称。 其中Center Loss出自论文 < A Discriminative Feature Learning Approach for Deep Face Recognition>. 最早用于人脸识别任务，用于增大类间距离，减小类内距离, 是Metric Learning领域一种较早的、也比较常用的一种算法。
+在中文OCR识别任务中，通过对badcase分析， 我们发现中文识别的一大难点是相似字符多，容易误识。 由此我们想到是否可以借鉴Metric Learing的想法， 增大相似字符的类间距，从而提高识别准确率。然而，MetricLearning主要用于图像识别领域，训练数据的标签为一个固定的值；而对于OCR识别来说，其本质上是一个序列识别任务，特征和label之间并不具有显式的对齐关系，因此两者如何结合依然是一个值得探索的方向。
+通过尝试Arcmargin, Cosmargin等方法， 我们最终发现Centerloss 有助于进一步提升识别的准确率。C_CTC Loss定义如下：
+
+$$
+L_{C-CTC} = L_{CTC} + \lambda * L_{center}
+$$
+
+实验中，我们设置λ=0.25. center_loss实现代码见: [center_loss.py](../../ppocr/losses/center_loss.py)
+
+值得一提的是， 在C-CTC Loss中，选择随机初始化Center并不能够带来明显的提升. 我们的Center初始化方法如下：
+
++ 基于原始的CTCLoss， 训练得到一个网络N
++ 挑选出训练集中，识别完全正确的部分, 组成集合G
++ 将G中的每个样本送入网络，进行前向计算， 提取最后一个FC层的输入（即feature）及其经过argmax计算的结果（即index）之间的对应关系
++ 将相同index的feature进行聚合，计算平均值，得到各自字符的初始center.
+
+以配置文件`configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec.yml`为例， center提取命令如下所示:
+
+```bash linenums="1"
+python tools/export_center.py -c configs/rec/ch_PP-OCRv2/ch_PP-OCRv2_rec.yml -o Global.pretrained_model="./output/rec_mobile_pp-OCRv2/best_accuracy"
+```
+
+运行完后，会在PaddleOCR主目录下生成`train_center.pkl`.
+
+## 4. 实验
+
+对于上述的三种方案，我们基于百度内部数据集进行了训练、评测，实验情况如下表所示：
+
+|algorithm| Focal_CTC | A_CTC | C-CTC |
+|:------| :------| ------: | :------: |
+|gain| +0.3% | +0.7% | +1.7% |
+
+基于上述实验结论，我们在PP-OCRv2中，采用了C-CTC的策略。 值得一提的是，由于PP-OCRv2 处理的是6625个中文字符的识别任务，字符集比较大，形似字较多，所以在该任务上C-CTC 方案带来的提升较大。 但如果换做其他OCR识别任务，结论可能会有所不同。大家可以尝试Focal-CTC，A-CTC, C-CTC以及组合方案EnhancedCTC，相信会带来不同程度的提升效果。
+统一的融合方案见如下文件： [rec_enhanced_ctc_loss.py](../../ppocr/losses/rec_enhanced_ctc_loss.py)
diff --git a/docs/ppocr/blog/images/11_det.jpg b/docs/ppocr/blog/images/11_det.jpg
new file mode 100644
index 0000000000..fe0cd23cc2
Binary files /dev/null and b/docs/ppocr/blog/images/11_det.jpg differ
diff --git a/docs/ppocr/blog/images/11_det_rec.jpg b/docs/ppocr/blog/images/11_det_rec.jpg
new file mode 100644
index 0000000000..31c566478f
Binary files /dev/null and b/docs/ppocr/blog/images/11_det_rec.jpg differ
diff --git a/docs/ppocr/blog/images/12_det.jpg b/docs/ppocr/blog/images/12_det.jpg
new file mode 100644
index 0000000000..71627f0b8d
Binary files /dev/null and b/docs/ppocr/blog/images/12_det.jpg differ
diff --git a/docs/ppocr/blog/images/12_det_rec.jpg b/docs/ppocr/blog/images/12_det_rec.jpg
new file mode 100644
index 0000000000..9db8b57e12
Binary files /dev/null and b/docs/ppocr/blog/images/12_det_rec.jpg differ
diff --git a/docs/ppocr/blog/images/187578511-9f3c351e-b68c-4359-a6e5-475810993c61.png b/docs/ppocr/blog/images/187578511-9f3c351e-b68c-4359-a6e5-475810993c61.png
new file mode 100644
index 0000000000..86c062684a
Binary files /dev/null and b/docs/ppocr/blog/images/187578511-9f3c351e-b68c-4359-a6e5-475810993c61.png differ
diff --git a/docs/ppocr/blog/images/254-20240709081442260.jpg b/docs/ppocr/blog/images/254-20240709081442260.jpg
new file mode 100644
index 0000000000..c871fb042c
Binary files /dev/null and b/docs/ppocr/blog/images/254-20240709081442260.jpg differ
diff --git a/docs/ppocr/blog/images/5e612.png b/docs/ppocr/blog/images/5e612.png
new file mode 100644
index 0000000000..79e3d0e1e2
Binary files /dev/null and b/docs/ppocr/blog/images/5e612.png differ
diff --git a/docs/ppocr/blog/images/DF.png b/docs/ppocr/blog/images/DF.png
new file mode 100644
index 0000000000..f14953d481
Binary files /dev/null and b/docs/ppocr/blog/images/DF.png differ
diff --git a/docs/ppocr/blog/images/GTC.png b/docs/ppocr/blog/images/GTC.png
new file mode 100644
index 0000000000..30a9cdd146
Binary files /dev/null and b/docs/ppocr/blog/images/GTC.png differ
diff --git a/docs/ppocr/blog/images/LCNet_SVTR.png b/docs/ppocr/blog/images/LCNet_SVTR.png
new file mode 100644
index 0000000000..7f0d701d27
Binary files /dev/null and b/docs/ppocr/blog/images/LCNet_SVTR.png differ
diff --git a/docs/ppocr/blog/images/LKPAN.png b/docs/ppocr/blog/images/LKPAN.png
new file mode 100644
index 0000000000..6b16053623
Binary files /dev/null and b/docs/ppocr/blog/images/LKPAN.png differ
diff --git a/docs/ppocr/blog/images/PFHead.png b/docs/ppocr/blog/images/PFHead.png
new file mode 100644
index 0000000000..3728dc44e5
Binary files /dev/null and b/docs/ppocr/blog/images/PFHead.png differ
diff --git a/docs/ppocr/blog/images/RSEFPN.png b/docs/ppocr/blog/images/RSEFPN.png
new file mode 100644
index 0000000000..ddf7c52fb5
Binary files /dev/null and b/docs/ppocr/blog/images/RSEFPN.png differ
diff --git a/docs/ppocr/blog/images/SSL.png b/docs/ppocr/blog/images/SSL.png
new file mode 100644
index 0000000000..1344a2a77c
Binary files /dev/null and b/docs/ppocr/blog/images/SSL.png differ
diff --git a/docs/ppocr/blog/images/UIM.png b/docs/ppocr/blog/images/UIM.png
new file mode 100644
index 0000000000..7479bdf4a9
Binary files /dev/null and b/docs/ppocr/blog/images/UIM.png differ
diff --git a/docs/ppocr/blog/images/arabic_0.jpg b/docs/ppocr/blog/images/arabic_0.jpg
new file mode 100644
index 0000000000..9941b90642
Binary files /dev/null and b/docs/ppocr/blog/images/arabic_0.jpg differ
diff --git a/docs/ppocr/blog/images/focal_loss_image.png b/docs/ppocr/blog/images/focal_loss_image.png
new file mode 100644
index 0000000000..430550a732
Binary files /dev/null and b/docs/ppocr/blog/images/focal_loss_image.png differ
diff --git a/docs/ppocr/blog/images/french_0.jpg b/docs/ppocr/blog/images/french_0.jpg
new file mode 100644
index 0000000000..3c2abe6304
Binary files /dev/null and b/docs/ppocr/blog/images/french_0.jpg differ
diff --git a/docs/ppocr/blog/images/img_02.jpg b/docs/ppocr/blog/images/img_02.jpg
new file mode 100644
index 0000000000..3e139c76bc
Binary files /dev/null and b/docs/ppocr/blog/images/img_02.jpg differ
diff --git a/docs/ppocr/blog/images/img_12.jpg b/docs/ppocr/blog/images/img_12.jpg
new file mode 100644
index 0000000000..822d562eda
Binary files /dev/null and b/docs/ppocr/blog/images/img_12.jpg differ
diff --git a/docs/ppocr/blog/images/japan_2-20240709081138234.jpg b/docs/ppocr/blog/images/japan_2-20240709081138234.jpg
new file mode 100644
index 0000000000..7038ba2eff
Binary files /dev/null and b/docs/ppocr/blog/images/japan_2-20240709081138234.jpg differ
diff --git a/docs/ppocr/blog/images/korean.jpg b/docs/ppocr/blog/images/korean.jpg
new file mode 100644
index 0000000000..e5d863cd86
Binary files /dev/null and b/docs/ppocr/blog/images/korean.jpg differ
diff --git a/docs/ppocr/blog/images/korean_0.jpg b/docs/ppocr/blog/images/korean_0.jpg
new file mode 100644
index 0000000000..3fe6305aa0
Binary files /dev/null and b/docs/ppocr/blog/images/korean_0.jpg differ
diff --git a/docs/ppocr/blog/images/multi_scale.png b/docs/ppocr/blog/images/multi_scale.png
new file mode 100644
index 0000000000..673d306399
Binary files /dev/null and b/docs/ppocr/blog/images/multi_scale.png differ
diff --git a/docs/ppocr/blog/images/ppocrv3_det_cml.png b/docs/ppocr/blog/images/ppocrv3_det_cml.png
new file mode 100644
index 0000000000..ccb5c8b21f
Binary files /dev/null and b/docs/ppocr/blog/images/ppocrv3_det_cml.png differ
diff --git a/docs/ppocr/blog/images/ppocrv3_framework-0052468.png b/docs/ppocr/blog/images/ppocrv3_framework-0052468.png
new file mode 100644
index 0000000000..e05279f7f5
Binary files /dev/null and b/docs/ppocr/blog/images/ppocrv3_framework-0052468.png differ
diff --git a/docs/ppocr/blog/images/ppocrv3_framework.png b/docs/ppocr/blog/images/ppocrv3_framework.png
new file mode 100644
index 0000000000..e05279f7f5
Binary files /dev/null and b/docs/ppocr/blog/images/ppocrv3_framework.png differ
diff --git a/docs/ppocr/blog/images/ppocrv4_det_cml.png b/docs/ppocr/blog/images/ppocrv4_det_cml.png
new file mode 100644
index 0000000000..9132c0a67c
Binary files /dev/null and b/docs/ppocr/blog/images/ppocrv4_det_cml.png differ
diff --git a/docs/ppocr/blog/images/ppocrv4_framework.png b/docs/ppocr/blog/images/ppocrv4_framework.png
new file mode 100644
index 0000000000..fa31f4c12e
Binary files /dev/null and b/docs/ppocr/blog/images/ppocrv4_framework.png differ
diff --git a/docs/ppocr/blog/images/ppocrv4_gtc.png b/docs/ppocr/blog/images/ppocrv4_gtc.png
new file mode 100644
index 0000000000..7e6a3f5c13
Binary files /dev/null and b/docs/ppocr/blog/images/ppocrv4_gtc.png differ
diff --git a/docs/ppocr/blog/images/rec_algo_compare.png b/docs/ppocr/blog/images/rec_algo_compare.png
new file mode 100644
index 0000000000..2dde496c75
Binary files /dev/null and b/docs/ppocr/blog/images/rec_algo_compare.png differ
diff --git a/docs/ppocr/blog/images/recconaug.png b/docs/ppocr/blog/images/recconaug.png
new file mode 100644
index 0000000000..899bc430de
Binary files /dev/null and b/docs/ppocr/blog/images/recconaug.png differ
diff --git a/docs/ppocr/blog/images/svtr_g2.png b/docs/ppocr/blog/images/svtr_g2.png
new file mode 100644
index 0000000000..2573afafbb
Binary files /dev/null and b/docs/ppocr/blog/images/svtr_g2.png differ
diff --git a/docs/ppocr/blog/images/svtr_g4.png b/docs/ppocr/blog/images/svtr_g4.png
new file mode 100644
index 0000000000..f85d66d97f
Binary files /dev/null and b/docs/ppocr/blog/images/svtr_g4.png differ
diff --git a/docs/ppocr/blog/images/svtr_tiny.png b/docs/ppocr/blog/images/svtr_tiny.png
new file mode 100644
index 0000000000..01e22e74b5
Binary files /dev/null and b/docs/ppocr/blog/images/svtr_tiny.png differ
diff --git a/docs/ppocr/blog/images/teacher_dml.png b/docs/ppocr/blog/images/teacher_dml.png
new file mode 100644
index 0000000000..ea09cacda8
Binary files /dev/null and b/docs/ppocr/blog/images/teacher_dml.png differ
diff --git a/docs/ppocr/blog/images/v3_rec_pipeline.png b/docs/ppocr/blog/images/v3_rec_pipeline.png
new file mode 100644
index 0000000000..aa61cc4f16
Binary files /dev/null and b/docs/ppocr/blog/images/v3_rec_pipeline.png differ
diff --git a/docs/ppocr/blog/images/v4_rec_pipeline.png b/docs/ppocr/blog/images/v4_rec_pipeline.png
new file mode 100644
index 0000000000..b1ec7a9689
Binary files /dev/null and b/docs/ppocr/blog/images/v4_rec_pipeline.png differ
diff --git a/docs/ppocr/blog/images/word_308.png b/docs/ppocr/blog/images/word_308.png
new file mode 100644
index 0000000000..a8d094faff
Binary files /dev/null and b/docs/ppocr/blog/images/word_308.png differ
diff --git a/docs/ppocr/blog/inference_args.en.md b/docs/ppocr/blog/inference_args.en.md
new file mode 100644
index 0000000000..d03de70dd9
--- /dev/null
+++ b/docs/ppocr/blog/inference_args.en.md
@@ -0,0 +1,131 @@
+---
+comments: true
+---
+
+
+# PaddleOCR Model Inference Parameter Explanation
+
+When using PaddleOCR for model inference, you can customize the modification parameters to modify the model, data, preprocessing, postprocessing, etc. (parameter file: [utility.py](../../tools/infer/utility.py))，The detailed parameter explanation is as follows:
+
+* Global parameters
+
+| parameters | type | default | implication |
+| :--: | :--: | :--: | :--: |
+| image_dir | str | None, must be specified explicitly | Image or folder path |
+| page_num | int | 0 | Valid when the input type is pdf file, specify to predict the previous page_num pages, all pages are predicted by default |
+| vis_font_path | str | "./doc/fonts/simfang.ttf" | font path for visualization |
+| drop_score | float | 0.5 | Results with a recognition score less than this value will be discarded and will not be returned as results |
+| use_pdserving | bool | False | Whether to use Paddle Serving for prediction |
+| warmup | bool | False | Whether to enable warmup, this method can be used when statistical prediction time |
+| draw_img_save_dir | str | "./inference_results" | The saving folder of the system's tandem prediction OCR results |
+| save_crop_res | bool | False | Whether to save the recognized text image for OCR |
+| crop_res_save_dir | str | "./output" | Save the text image path recognized by OCR |
+| use_mp | bool | False | Whether to enable multi-process prediction |
+| total_process_num | int | 6 | The number of processes, which takes effect when `use_mp` is `True` |
+| process_id | int | 0 | The id number of the current process, no need to modify it yourself |
+| benchmark | bool | False | Whether to enable benchmark, and make statistics on prediction speed, memory usage, etc. |
+| save_log_path | str | "./log_output/" | Folder where log results are saved when `benchmark` is enabled |
+| show_log | bool | True | Whether to show the log information in the inference |
+| use_onnx | bool | False | Whether to enable onnx prediction |
+
+* Prediction engine related parameters
+
+| parameters | type | default | implication |
+| :--: | :--: | :--: | :--: |
+| use_gpu | bool | True | Whether to use GPU for prediction |
+| ir_optim | bool | True | Whether to analyze and optimize the calculation graph. The prediction process can be accelerated when `ir_optim` is enabled |
+| use_tensorrt | bool | False | Whether to enable tensorrt |
+| min_subgraph_size | int | 15 | The minimum subgraph size in tensorrt. When the size of the subgraph is greater than this value, it will try to use the trt engine to calculate the subgraph. |
+| precision | str | fp32 | The precision of prediction, supports `fp32`, `fp16`, `int8` |
+| enable_mkldnn | bool | True | Whether to enable mkldnn |
+| cpu_threads | int | 10 | When mkldnn is enabled, the number of threads predicted by the cpu |
+
+* Text detection model related parameters
+
+| parameters | type | default | implication |
+| :--: | :--: | :--: | :--: |
+| det_algorithm | str | "DB" | Text detection algorithm name, currently supports `DB`, `EAST`, `SAST`, `PSE`, `DB++`, `FCE` |
+| det_model_dir | str | xx | Detection inference model paths |
+| det_limit_side_len | int | 960 | image side length limit |
+| det_limit_type | str | "max" | The side length limit type, currently supports `min`and `max`. `min` means to ensure that the shortest side of the image is not less than `det_limit_side_len`, `max` means to ensure that the longest side of the image is not greater than `det_limit_side_len` |
+
+The relevant parameters of the DB algorithm are as follows
+
+| parameters | type | default | implication |
+| :--: | :--: | :--: | :--: |
+| det_db_thresh | float | 0.3 | In the probability map output by DB, only pixels with a score greater than this threshold will be considered as text pixels |
+| det_db_box_thresh | float | 0.6 | Within the detection box, when the average score of all pixels is greater than the threshold, the result will be considered as a text area |
+| det_db_unclip_ratio | float | 1.5 | The expansion factor of the `Vatti clipping` algorithm, which is used to expand the text area |
+| max_batch_size | int | 10 | max batch size |
+| use_dilation | bool | False | Whether to inflate the segmentation results to obtain better detection results |
+| det_db_score_mode | str | "fast" | DB detection result score calculation method, supports `fast` and `slow`, `fast` calculates the average score according to all pixels within the bounding rectangle of the polygon, `slow` calculates the average score according to all pixels within the original polygon, The calculation speed is relatively slower, but more accurate. |
+
+The relevant parameters of the EAST algorithm are as follows
+
+| parameters | type | default | implication |
+| :--: | :--: | :--: | :--: |
+| det_east_score_thresh | float | 0.8 | Threshold for score map in EAST postprocess |
+| det_east_cover_thresh | float | 0.1 | Average score threshold for text boxes in EAST postprocess |
+| det_east_nms_thresh | float | 0.2 | Threshold of nms in EAST postprocess |
+
+The relevant parameters of the SAST algorithm are as follows
+
+| parameters | type | default | implication |
+| :--: | :--: | :--: | :--: |
+| det_sast_score_thresh | float | 0.5 | Score thresholds in SAST postprocess |
+| det_sast_nms_thresh | float | 0.5 | Thresholding of nms in SAST postprocess |
+| det_box_type | str | 'quad' | Whether polygon detection, curved text scene (such as Total-Text) is set to 'poly' |
+
+The relevant parameters of the PSE algorithm are as follows
+
+| parameters | type | default | implication |
+| :--: | :--: | :--: | :--: |
+| det_pse_thresh | float | 0.0 | Threshold for binarizing the output image |
+| det_pse_box_thresh | float | 0.85 | Threshold for filtering boxes, below this threshold is discarded |
+| det_pse_min_area | float | 16 | The minimum area of the box, below this threshold is discarded |
+| det_box_type | str | "quad" | The type of the returned box, quad: four point coordinates, poly: all point coordinates of the curved text |
+| det_pse_scale | int | 1 | The ratio of the input image relative to the post-processed image, such as an image of `640*640`, the network output is `160*160`, and when the scale is 2, the shape of the post-processed image is `320*320`. Increasing this value can speed up the post-processing speed, but it will bring about a decrease in accuracy |
+
+* Text recognition model related parameters
+
+| parameters | type | default | implication |
+| :--: | :--: | :--: | :--: |
+| rec_algorithm | str | "CRNN" | Text recognition algorithm name, currently supports `CRNN`, `SRN`, `RARE`, `NETR`, `SAR`, `ViTSTR`, `ABINet`, `VisionLAN`, `SPIN`, `RobustScanner`, `SVTR`, `SVTR_LCNet` |
+| rec_model_dir | str | None, it is required if using the recognition model | recognition inference model paths |
+| rec_image_shape | str | "3,48,320" ] | Image size at the time of recognition |
+| rec_batch_num | int | 6 | batch size |
+| max_text_length | int | 25 | The maximum length of the recognition result, valid in `SRN` |
+| rec_char_dict_path | str | "./ppocr/utils/ppocr_keys_v1.txt" | character dictionary file |
+| use_space_char | bool | True | Whether to include spaces, if `True`, the `space` character will be added at the end of the character dictionary |
+
+* End-to-end text detection and recognition model related parameters
+
+| parameters | type | default | implication |
+| :--: | :--: | :--: | :--: |
+| e2e_algorithm | str | "PGNet" | End-to-end algorithm name, currently supports `PGNet` |
+| e2e_model_dir | str | None, it is required if using the end-to-end model | end-to-end model inference model path |
+| e2e_limit_side_len | int | 768 | End-to-end input image side length limit |
+| e2e_limit_type | str | "max" | End-to-end side length limit type, currently supports `min` and `max`. `min` means to ensure that the shortest side of the image is not less than `e2e_limit_side_len`, `max` means to ensure that the longest side of the image is not greater than `e2e_limit_side_len` |
+| e2e_pgnet_score_thresh | float | 0.5 | End-to-end score threshold, results below this threshold are discarded |
+| e2e_char_dict_path | str | "./ppocr/utils/ic15_dict.txt" | Recognition dictionary file path |
+| e2e_pgnet_valid_set | str | "totaltext" | The name of the validation set, currently supports `totaltext`, `partvgg`, the post-processing methods corresponding to different data sets are different, and it can be consistent with the training process |
+| e2e_pgnet_mode | str | "fast" | PGNet's detection result score calculation method, supports `fast` and `slow`, `fast` calculates the average score according to all pixels within the bounding rectangle of the polygon, `slow` calculates the average score according to all pixels within the original polygon, The calculation speed is relatively slower, but more accurate. |
+
+* Angle classifier model related parameters
+
+| parameters | type | default | implication |
+| :--: | :--: | :--: | :--: |
+| use_angle_cls | bool | False | whether to use an angle classifier |
+| cls_model_dir | str | None, if you need to use, you must specify the path explicitly | angle classifier inference model path |
+| cls_image_shape | str | "3,48,192" | prediction shape |
+| label_list | list | ['0', '180'] | The angle value corresponding to the class id |
+| cls_batch_num | int | 6 | batch size |
+| cls_thresh | float | 0.9 | Prediction threshold, when the model prediction result is 180 degrees, and the score is greater than the threshold, the final prediction result is considered to be 180 degrees and needs to be flipped |
+
+* OCR image preprocessing parameters
+
+| parameters | type | default | implication |
+| :--: | :--: | :--: | :--: |
+| invert | bool | False | whether to invert image before processing |
+| binarize | bool | False | whether to threshold binarize image before processing |
+| alphacolor | tuple | "255,255,255" | Replacement color for the alpha channel, if the latter is present; R,G,B integers |
diff --git a/docs/ppocr/blog/inference_args.md b/docs/ppocr/blog/inference_args.md
new file mode 100644
index 0000000000..1eec868995
--- /dev/null
+++ b/docs/ppocr/blog/inference_args.md
@@ -0,0 +1,118 @@
+# PaddleOCR模型推理参数解释
+
+在使用PaddleOCR进行模型推理时，可以自定义修改参数，来修改模型、数据、预处理、后处理等内容（参数文件：[utility.py](../../tools/infer/utility.py)），详细的参数解释如下所示。
+
+* 全局信息
+
+| 参数名称 | 类型 | 默认值 | 含义 |
+| :--: | :--: | :--: | :--: |
+| image_dir | str | 无，必须显式指定 | 图像或者文件夹路径 |
+| page_num | int | 0 | 当输入类型为pdf文件时有效，指定预测前面page_num页，默认预测所有页 |
+| vis_font_path | str | "./doc/fonts/simfang.ttf" | 用于可视化的字体路径 |
+| drop_score | float | 0.5 | 识别得分小于该值的结果会被丢弃，不会作为返回结果 |
+| use_pdserving | bool | False | 是否使用Paddle Serving进行预测 |
+| warmup | bool | False | 是否开启warmup，在统计预测耗时的时候，可以使用这种方法 |
+| draw_img_save_dir | str | "./inference_results" | 系统串联预测OCR结果的保存文件夹 |
+| save_crop_res | bool | False | 是否保存OCR的识别文本图像 |
+| crop_res_save_dir | str | "./output" | 保存OCR识别出来的文本图像路径 |
+| use_mp | bool | False | 是否开启多进程预测 |
+| total_process_num | int | 6 | 开启的进程数，`use_mp`为`True`时生效 |
+| process_id | int | 0 | 当前进程的id号，无需自己修改 |
+| benchmark | bool | False | 是否开启benchmark，对预测速度、显存占用等进行统计 |
+| save_log_path | str | "./log_output/" | 开启`benchmark`时，日志结果的保存文件夹 |
+| show_log | bool | True | 是否显示预测中的日志信息 |
+| use_onnx | bool | False | 是否开启onnx预测 |
+
+* 预测引擎相关
+
+| 参数名称 | 类型 | 默认值 | 含义 |
+| :--: | :--: | :--: | :--: |
+| use_gpu | bool | True | 是否使用GPU进行预测 |
+| ir_optim | bool | True | 是否对计算图进行分析与优化，开启后可以加速预测过程 |
+| use_tensorrt | bool | False | 是否开启tensorrt |
+| min_subgraph_size | int | 15 | tensorrt中最小子图size，当子图的size大于该值时，才会尝试对该子图使用trt engine计算 |
+| precision | str | fp32 | 预测的精度，支持`fp32`, `fp16`, `int8` 3种输入 |
+| enable_mkldnn | bool | True | 是否开启mkldnn |
+| cpu_threads | int | 10 | 开启mkldnn时，cpu预测的线程数 |
+
+* 文本检测模型相关
+
+| 参数名称 | 类型 | 默认值 | 含义 |
+| :--: | :--: | :--: | :--: |
+| det_algorithm | str | "DB" | 文本检测算法名称，目前支持`DB`, `EAST`, `SAST`, `PSE`, `DB++`, `FCE` |
+| det_model_dir | str | xx | 检测inference模型路径 |
+| det_limit_side_len | int | 960 | 检测的图像边长限制 |
+| det_limit_type | str | "max" | 检测的边长限制类型，目前支持`min`和`max`，`min`表示保证图像最短边不小于`det_limit_side_len`，`max`表示保证图像最长边不大于`det_limit_side_len` |
+
+其中，DB算法相关参数如下
+
+| 参数名称 | 类型 | 默认值 | 含义 |
+| :--: | :--: | :--: | :--: |
+| det_db_thresh | float | 0.3 | DB输出的概率图中，得分大于该阈值的像素点才会被认为是文字像素点 |
+| det_db_box_thresh | float | 0.6 | 检测结果边框内，所有像素点的平均得分大于该阈值时，该结果会被认为是文字区域 |
+| det_db_unclip_ratio | float | 1.5 | `Vatti clipping`算法的扩张系数，使用该方法对文字区域进行扩张 |
+| max_batch_size | int | 10 | 预测的batch size |
+| use_dilation | bool | False | 是否对分割结果进行膨胀以获取更优检测效果 |
+| det_db_score_mode | str | "fast" | DB的检测结果得分计算方法，支持`fast`和`slow`，`fast`是根据polygon的外接矩形边框内的所有像素计算平均得分，`slow`是根据原始polygon内的所有像素计算平均得分，计算速度相对较慢一些，但是更加准确一些。 |
+
+EAST算法相关参数如下
+
+| 参数名称 | 类型 | 默认值 | 含义 |
+| :--: | :--: | :--: | :--: |
+| det_east_score_thresh | float | 0.8 | EAST后处理中score map的阈值 |
+| det_east_cover_thresh | float | 0.1 | EAST后处理中文本框的平均得分阈值 |
+| det_east_nms_thresh | float | 0.2 | EAST后处理中nms的阈值 |
+
+SAST算法相关参数如下
+
+| 参数名称 | 类型 | 默认值 | 含义 |
+| :--: | :--: | :--: | :--: |
+| det_sast_score_thresh | float | 0.5 | SAST后处理中的得分阈值 |
+| det_sast_nms_thresh | float | 0.5 | SAST后处理中nms的阈值 |
+| det_box_type | str | quad | 是否多边形检测，弯曲文本场景（如Total-Text）设置为'poly' |
+
+PSE算法相关参数如下
+
+| 参数名称 | 类型 | 默认值 | 含义 |
+| :--: | :--: | :--: | :--: |
+| det_pse_thresh | float | 0.0 | 对输出图做二值化的阈值 |
+| det_pse_box_thresh | float | 0.85 | 对box进行过滤的阈值，低于此阈值的丢弃 |
+| det_pse_min_area | float | 16 | box的最小面积，低于此阈值的丢弃 |
+| det_box_type | str | "quad" | 返回框的类型，quad:四点坐标，poly: 弯曲文本的所有点坐标 |
+| det_pse_scale | int | 1 | 输入图像相对于进后处理的图的比例，如`640*640`的图像，网络输出为`160*160`，scale为2的情况下，进后处理的图片shape为`320*320`。这个值调大可以加快后处理速度，但是会带来精度的下降 |
+
+* 文本识别模型相关
+
+| 参数名称 | 类型 | 默认值 | 含义 |
+| :--: | :--: | :--: | :--: |
+| rec_algorithm | str | "CRNN" | 文本识别算法名称，目前支持`CRNN`, `SRN`, `RARE`, `NETR`, `SAR`, `ViTSTR`, `ABINet`, `VisionLAN`, `SPIN`, `RobustScanner`, `SVTR`, `SVTR_LCNet` |
+| rec_model_dir | str | 无，如果使用识别模型，该项是必填项 | 识别inference模型路径 |
+| rec_image_shape | str | "3,48,320" | 识别时的图像尺寸 |
+| rec_batch_num | int | 6 | 识别的batch size |
+| max_text_length | int | 25 | 识别结果最大长度，在`SRN`中有效 |
+| rec_char_dict_path | str | "./ppocr/utils/ppocr_keys_v1.txt" | 识别的字符字典文件 |
+| use_space_char | bool | True | 是否包含空格，如果为`True`，则会在最后字符字典中补充`空格`字符 |
+
+* 端到端文本检测与识别模型相关
+
+| 参数名称 | 类型 | 默认值 | 含义 |
+| :--: | :--: | :--: | :--: |
+| e2e_algorithm | str | "PGNet" | 端到端算法名称，目前支持`PGNet` |
+| e2e_model_dir | str | 无，如果使用端到端模型，该项是必填项 | 端到端模型inference模型路径 |
+| e2e_limit_side_len | int | 768 | 端到端的输入图像边长限制 |
+| e2e_limit_type | str | "max" | 端到端的边长限制类型，目前支持`min`, `max`，`min`表示保证图像最短边不小于`e2e_limit_side_len`，`max`表示保证图像最长边不大于`e2e_limit_side_len` |
+| e2e_pgnet_score_thresh | float | 0.5 | 端到端得分阈值，小于该阈值的结果会被丢弃 |
+| e2e_char_dict_path | str | "./ppocr/utils/ic15_dict.txt" | 识别的字典文件路径 |
+| e2e_pgnet_valid_set | str | "totaltext" | 验证集名称，目前支持`totaltext`, `partvgg`，不同数据集对应的后处理方式不同，与训练过程保持一致即可 |
+| e2e_pgnet_mode | str | "fast" | PGNet的检测结果得分计算方法，支持`fast`和`slow`，`fast`是根据polygon的外接矩形边框内的所有像素计算平均得分，`slow`是根据原始polygon内的所有像素计算平均得分，计算速度相对较慢一些，但是更加准确一些。 |
+
+* 方向分类器模型相关
+
+| 参数名称 | 类型 | 默认值 | 含义 |
+| :--: | :--: | :--: | :--: |
+| use_angle_cls | bool | False | 是否使用方向分类器 |
+| cls_model_dir | str | 无，如果需要使用，则必须显式指定路径 | 方向分类器inference模型路径 |
+| cls_image_shape | str | "3,48,192" | 预测尺度 |
+| label_list | list | ['0', '180'] | class id对应的角度值 |
+| cls_batch_num | int | 6 | 方向分类器预测的batch size |
+| cls_thresh | float | 0.9 | 预测阈值，模型预测结果为180度，且得分大于该阈值时，认为最终预测结果为180度，需要翻转 |
diff --git a/docs/ppocr/blog/multi_languages.en.md b/docs/ppocr/blog/multi_languages.en.md
new file mode 100644
index 0000000000..ed71a2d439
--- /dev/null
+++ b/docs/ppocr/blog/multi_languages.en.md
@@ -0,0 +1,223 @@
+---
+comments: true
+typora-copy-images-to: images
+---
+
+# Multi-language model
+
+**Recent Update**
+
+- 2022.5.8 update the `PP-OCRv3` version of the multi-language detection and recognition model, and the average recognition accuracy has increased by more than 5%.
+- 2021.4.9 supports the detection and recognition of 80 languages
+- 2021.4.9 supports **lightweight high-precision** English model detection and recognition
+
+PaddleOCR aims to create a rich, leading, and practical OCR tool library, which not only provides
+Chinese and English models in general scenarios, but also provides models specifically trained
+in English scenarios. And multilingual models covering [80 languages](#language_abbreviations).
+
+Among them, the English model supports the detection and recognition of uppercase and lowercase
+letters and common punctuation, and the recognition of space characters is optimized:
+
+
+
+The multilingual models cover Latin, Arabic, Traditional Chinese, Korean, Japanese, etc.:
+
+
+
+
+
+
+
+
+
+This document will briefly introduce how to use the multilingual model.
+
+## 1 Installation
+
+### 1.1 Paddle installation
+
+```bash linenums="1"
+# cpu
+pip install paddlepaddle
+
+# gpu
+pip install paddlepaddle-gpu
+```
+
+### 1.2 PaddleOCR package installation
+
+```bash linenums="1"
+pip install paddleocr
+```
+
+Build and install locally
+
+```bash linenums="1"
+python3 -m build
+pip3 install dist/paddleocr-x.x.x-py3-none-any.whl # x.x.x is the version number of paddleocr
+```
+
+## 2 Quick use
+
+### 2.1 Command line operation
+
+View help information
+
+```bash linenums="1"
+paddleocr -h
+```
+
+- Whole image prediction (detection + recognition)
+
+PaddleOCR currently supports 80 languages, which can be specified by the --lang parameter.
+The supported languages are listed in the [table](#language_abbreviations).
+
+``` bash
+paddleocr --image_dir doc/imgs_en/254.jpg --lang=en
+```
+
+
+
+
+
+The result is a list. Each item contains a text box, text and recognition confidence
+
+```text linenums="1"
+[('PHO CAPITAL', 0.95723116), [[66.0, 50.0], [327.0, 44.0], [327.0, 76.0], [67.0, 82.0]]]
+[('107 State Street', 0.96311164), [[72.0, 90.0], [451.0, 84.0], [452.0, 116.0], [73.0, 121.0]]]
+[('Montpelier Vermont', 0.97389287), [[69.0, 132.0], [501.0, 126.0], [501.0, 158.0], [70.0, 164.0]]]
+[('8022256183', 0.99810505), [[71.0, 175.0], [363.0, 170.0], [364.0, 202.0], [72.0, 207.0]]]
+[('REG 07-24-201706:59 PM', 0.93537045), [[73.0, 299.0], [653.0, 281.0], [654.0, 318.0], [74.0, 336.0]]]
+[('045555', 0.99346405), [[509.0, 331.0], [651.0, 325.0], [652.0, 356.0], [511.0, 362.0]]]
+[('CT1', 0.9988654), [[535.0, 367.0], [654.0, 367.0], [654.0, 406.0], [535.0, 406.0]]]
+......
+```
+
+- Recognition
+
+```bash linenums="1"
+paddleocr --image_dir doc/imgs_words_en/word_308.png --det false --lang=en
+```
+
+
+
+The result is a 2-tuple, which contains the recognition result and recognition confidence
+
+```text linenums="1"
+(0.99879867, 'LITTLE')
+```
+
+- Detection
+
+```bash linenums="1"
+paddleocr --image_dir PaddleOCR/doc/imgs/11.jpg --rec false
+```
+
+The result is a list. Each item represents the coordinates of a text box.
+
+```bash linenums="1"
+[[26.0, 457.0], [137.0, 457.0], [137.0, 477.0], [26.0, 477.0]]
+[[25.0, 425.0], [372.0, 425.0], [372.0, 448.0], [25.0, 448.0]]
+[[128.0, 397.0], [273.0, 397.0], [273.0, 414.0], [128.0, 414.0]]
+......
+```
+
+### 2.2 Run with Python script
+
+PPOCR is able to run with Python scripts for easy integration with your own code:
+
+- Whole image prediction (detection + recognition)
+
+```python linenums="1"
+from paddleocr import PaddleOCR, draw_ocr
+
+# Also switch the language by modifying the lang parameter
+ocr = PaddleOCR(lang="korean") # The model file will be downloaded automatically when executed for the first time
+img_path ='doc/imgs/korean_1.jpg'
+result = ocr.ocr(img_path)
+# Recognition and detection can be performed separately through parameter control
+# result = ocr.ocr(img_path, det=False) Only perform recognition
+# result = ocr.ocr(img_path, rec=False) Only perform detection
+# Print detection frame and recognition result
+for line in result:
+ print(line)
+
+# Visualization
+from PIL import Image
+image = Image.open(img_path).convert('RGB')
+boxes = [line[0] for line in result]
+txts = [line[1][0] for line in result]
+scores = [line[1][1] for line in result]
+im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/korean.ttf')
+im_show = Image.fromarray(im_show)
+im_show.save('result.jpg')
+```
+
+Visualization of results:
+
+
+
+PPOCR also supports direction classification. For more detailed usage, please refer to: [whl package instructions](whl_en.md).
+
+## 3 Custom training
+
+PPOCR supports using your own data for custom training or fine-tune, where the recognition model can refer to [French configuration file](../../configs/rec/multi_language/rec_french_lite_train.yml)
+Modify the training data path, dictionary and other parameters.
+
+For specific data preparation and training process, please refer to: [Text Detection](../doc_en/detection_en.md), [Text Recognition](../doc_en/recognition_en.md), more functions such as predictive deployment,
+For functions such as data annotation, you can read the complete [Document Tutorial](../../README.md).
+
+## 4 Inference and Deployment
+
+In addition to installing the whl package for quick forecasting,
+PPOCR also provides a variety of forecasting deployment methods.
+If necessary, you can read related documents:
+
+- [Python Inference](./inference_ppocr_en.md)
+- [C++ Inference](../../deploy/cpp_infer/readme.md)
+- [Serving](../../deploy/hubserving/readme_en.md)
+- [Mobile](../../deploy/lite/readme.md)
+- [Benchmark](./benchmark_en.md)
+
+## 5 Support languages and abbreviations
+
+| Language | Abbreviation | | Language | Abbreviation |
+| --- | --- | --- | --- | --- |
+|Chinese & English|ch| |Arabic|ar|
+|English|en| |Hindi|hi|
+|French|fr| |Uyghur|ug|
+|German|german| |Persian|fa|
+|Japan|japan| |Urdu|ur|
+|Korean|korean| | Serbian(latin) |rs_latin|
+|Chinese Traditional |chinese_cht| |Occitan |oc|
+| Italian |it| |Marathi|mr|
+|Spanish |es| |Nepali|ne|
+| Portuguese|pt| |Serbian(cyrillic)|rs_cyrillic|
+|Russia|ru||Bulgarian |bg|
+|Ukranian|uk| |Estonian |et|
+|Belarusian|be| |Irish |ga|
+|Telugu |te| |Croatian |hr|
+|Saudi Arabia|sa| |Hungarian |hu|
+|Tamil |ta| |Indonesian|id|
+|Afrikaans |af| |Icelandic|is|
+|Azerbaijani |az||Kurdish|ku|
+|Bosnian|bs| |Lithuanian |lt|
+|Czech|cs| |Latvian |lv|
+|Welsh |cy| |Maori|mi|
+|Danish|da| |Malay|ms|
+|Maltese |mt| |Adyghe |ady|
+|Dutch |nl| |Kabardian |kbd|
+|Norwegian |no| |Avar |ava|
+|Polish |pl| |Dargwa |dar|
+|Romanian |ro| |Ingush |inh|
+|Slovak |sk| |Lak |lbe|
+|Slovenian |sl| |Lezghian |lez|
+|Albanian |sq| |Tabassaran |tab|
+|Swedish |sv| |Bihari |bh|
+|Swahili |sw| |Maithili |mai|
+|Tagalog |tl| |Angika |ang|
+|Turkish |tr| |Bhojpuri |bho|
+|Uzbek |uz| |Magahi |mah|
+|Vietnamese |vi| |Nagpur |sck|
+|Mongolian |mn| |Newari |new|
+|Abaza |abq| |Goan Konkani|gom|
diff --git a/docs/ppocr/blog/multi_languages.md b/docs/ppocr/blog/multi_languages.md
new file mode 100644
index 0000000000..22c331121f
--- /dev/null
+++ b/docs/ppocr/blog/multi_languages.md
@@ -0,0 +1,273 @@
+---
+comments: true
+typora-copy-images-to: images
+---
+
+# 多语言模型
+
+**近期更新**
+
+- 2022.5.8 更新`PP-OCRv3`版 多语言检测和识别模型，平均识别准确率提升5%以上。
+- 2021.4.9 支持**80种**语言的检测和识别
+- 2021.4.9 支持**轻量高精度**英文模型检测识别
+
+PaddleOCR 旨在打造一套丰富、领先、且实用的OCR工具库，不仅提供了通用场景下的中英文模型，也提供了专门在英文场景下训练的模型，
+和覆盖[80个语言](#语种缩写)的小语种模型。
+
+其中英文模型支持，大小写字母和常见标点的检测识别，并优化了空格字符的识别：
+
+
+
+小语种模型覆盖了拉丁语系、阿拉伯语系、中文繁体、韩语、日语等等：
+
+
+
+
+
+
+
+
+
+本文档将简要介绍小语种模型的使用方法。
+
+## 1 安装
+
+### 1.1 paddle 安装
+
+```bash linenums="1"
+# cpu
+pip install paddlepaddle
+
+# gpu
+pip install paddlepaddle-gpu
+```
+
+### 1.2 paddleocr package 安装
+
+pip 安装
+
+```bash linenums="1"
+pip install paddleocr
+```
+
+本地构建并安装
+
+```bash linenums="1"
+python3 -m build
+pip3 install dist/paddleocr-x.x.x-py3-none-any.whl # x.x.x是paddleocr的版本号
+```
+
+## 2 快速使用
+
+### 2.1 命令行运行
+
+查看帮助信息
+
+```bash linenums="1"
+paddleocr -h
+```
+
+- 整图预测（检测+识别）
+
+Paddleocr目前支持80个语种，可以通过修改--lang参数进行切换，具体支持的[语种](#语种缩写)可查看表格。
+
+``` bash
+paddleocr --image_dir doc/imgs_en/254.jpg --lang=en
+```
+
+
+
+
+
+结果是一个list，每个item包含了文本框，文字和识别置信度
+
+```text linenums="1"
+[('PHO CAPITAL', 0.95723116), [[66.0, 50.0], [327.0, 44.0], [327.0, 76.0], [67.0, 82.0]]]
+[('107 State Street', 0.96311164), [[72.0, 90.0], [451.0, 84.0], [452.0, 116.0], [73.0, 121.0]]]
+[('Montpelier Vermont', 0.97389287), [[69.0, 132.0], [501.0, 126.0], [501.0, 158.0], [70.0, 164.0]]]
+[('8022256183', 0.99810505), [[71.0, 175.0], [363.0, 170.0], [364.0, 202.0], [72.0, 207.0]]]
+[('REG 07-24-201706:59 PM', 0.93537045), [[73.0, 299.0], [653.0, 281.0], [654.0, 318.0], [74.0, 336.0]]]
+[('045555', 0.99346405), [[509.0, 331.0], [651.0, 325.0], [652.0, 356.0], [511.0, 362.0]]]
+[('CT1', 0.9988654), [[535.0, 367.0], [654.0, 367.0], [654.0, 406.0], [535.0, 406.0]]]
+......
+```
+
+- 识别预测
+
+```bash linenums="1"
+paddleocr --image_dir doc/imgs_words_en/word_308.png --det false --lang=en
+```
+
+
+
+结果是一个tuple，返回识别结果和识别置信度
+
+```text linenums="1"
+(0.99879867, 'LITTLE')
+```
+
+- 检测预测
+
+```bash linenums="1"
+paddleocr --image_dir PaddleOCR/doc/imgs/11.jpg --rec false
+```
+
+结果是一个list，每个item只包含文本框
+
+```bash linenums="1"
+[[26.0, 457.0], [137.0, 457.0], [137.0, 477.0], [26.0, 477.0]]
+[[25.0, 425.0], [372.0, 425.0], [372.0, 448.0], [25.0, 448.0]]
+[[128.0, 397.0], [273.0, 397.0], [273.0, 414.0], [128.0, 414.0]]
+......
+```
+
+### 2.2 python 脚本运行
+
+ppocr 也支持在python脚本中运行，便于嵌入到您自己的代码中 ：
+
+- 整图预测（检测+识别）
+
+```python linenums="1"
+from paddleocr import PaddleOCR, draw_ocr
+
+# 同样也是通过修改 lang 参数切换语种
+ocr = PaddleOCR(lang="korean") # 首次执行会自动下载模型文件
+img_path = 'doc/imgs/korean_1.jpg '
+result = ocr.ocr(img_path)
+# 可通过参数控制单独执行识别、检测
+# result = ocr.ocr(img_path, det=False) 只执行识别
+# result = ocr.ocr(img_path, rec=False) 只执行检测
+# 打印检测框和识别结果
+for line in result:
+ print(line)
+
+# 可视化
+from PIL import Image
+image = Image.open(img_path).convert('RGB')
+boxes = [line[0] for line in result]
+txts = [line[1][0] for line in result]
+scores = [line[1][1] for line in result]
+im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/korean.ttf')
+im_show = Image.fromarray(im_show)
+im_show.save('result.jpg')
+```
+
+结果可视化:
+
+
+
+ppocr 还支持方向分类， 更多使用方式请参考：[whl包使用说明](https://github.com/PaddlePaddle/PaddleOCR/blob/release/2.0/doc/doc_ch/whl.md)
+
+## 3 自定义训练
+
+ppocr 支持使用自己的数据进行自定义训练或finetune, 其中识别模型可以参考[法语配置文件](../../configs/rec/multi_language/rec_french_lite_train.yml)
+修改训练数据路径、字典等参数。
+
+详细数据准备、训练过程可参考：[文本识别](../doc_ch/recognition.md)、[文本检测](../doc_ch/detection.md)。
+
+假设已经准备好了训练数据，可根据以下步骤快速启动训练：
+
+- 修改配置文件
+
+以 `rec_french_lite_train.yml` 为例：
+
+```yaml linenums="1"
+Global:
+ ...
+ # 添加自定义字典，如修改字典请将路径指向新字典
+ character_dict_path: ./ppocr/utils/dict/french_dict.txt
+ ...
+ # 识别空格
+ use_space_char: True
+
+...
+
+Train:
+ dataset:
+ # 数据集格式，支持LMDBDataSet以及SimpleDataSet
+ name: SimpleDataSet
+ # 数据集路径
+ data_dir: ./train_data/
+ # 训练集标签文件
+ label_file_list: ["./train_data/french_train.txt"]
+ ...
+
+Eval:
+ dataset:
+ # 数据集格式，支持LMDBDataSet以及SimpleDataSet
+ name: SimpleDataSet
+ # 数据集路径
+ data_dir: ./train_data
+ # 验证集标签文件
+ label_file_list: ["./train_data/french_val.txt"]
+ ...
+```
+
+- 启动训练：
+
+```bash linenums="1"
+# 下载预训练模型
+wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/multilingual/french_mobile_v2.0_rec_train.tar
+tar -xf french_mobile_v2.0_rec_train.tar
+
+#加载预训练模型 单卡训练
+python3 tools/train.py -c configs/rec/rec_french_lite_train.yml -o Global.pretrained_model=french_mobile_v2.0_rec_train/best_accuracy
+
+#加载预训练模型 多卡训练，通过--gpus参数指定卡号
+python3 -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/rec/rec_french_lite_train.yml -o Global.pretrained_model=french_mobile_v2.0_rec_train/best_accuracy
+```
+
+更多功能如预测部署、数据标注等功能可以阅读完整的[文档教程](../../README_ch.md)。
+
+## 4 预测部署
+
+除了安装whl包进行快速预测，ppocr 也提供了多种预测部署方式，如有需求可阅读相关文档：
+
+- [基于Python脚本预测引擎推理](./inference_ppocr.md)
+- [基于C++预测引擎推理](../../deploy/cpp_infer/readme_ch.md)
+- [服务化部署](../../deploy/hubserving/readme.md)
+- [端侧部署](../../deploy/lite/readme_ch.md)
+- [Benchmark](./benchmark.md)
+
+## 5 支持语种及缩写
+
+| 语种 | 描述 | 缩写 | | 语种 | 描述 | 缩写 |
+| --- | --- | --- | ---|--- | --- | --- |
+|中文|chinese and english|ch| |保加利亚文|Bulgarian |bg|
+|英文|english|en| |乌克兰文|Ukranian|uk|
+|法文|french|fr| |白俄罗斯文|Belarusian|be|
+|德文|german|german| |泰卢固文|Telugu |te|
+|日文|japan|japan| | 阿巴扎文 |Abaza | abq |
+|韩文|korean|korean| |泰米尔文|Tamil |ta|
+|中文繁体|chinese traditional |chinese_cht| |南非荷兰文 |Afrikaans |af|
+|意大利文| Italian |it| |阿塞拜疆文 |Azerbaijani |az|
+|西班牙文|Spanish |es| |波斯尼亚文|Bosnian|bs|
+|葡萄牙文| Portuguese|pt| |捷克文|Czech|cs|
+|俄罗斯文|Russia|ru| |威尔士文 |Welsh |cy|
+|阿拉伯文|Arabic|ar| |丹麦文 |Danish|da|
+|印地文|Hindi|hi| |爱沙尼亚文 |Estonian |et|
+|维吾尔|Uyghur|ug| |爱尔兰文 |Irish |ga|
+|波斯文|Persian|fa| |克罗地亚文|Croatian |hr|
+|乌尔都文|Urdu|ur| |匈牙利文|Hungarian |hu|
+|塞尔维亚文（latin)| Serbian(latin) |rs_latin| |印尼文|Indonesian|id|
+|欧西坦文|Occitan |oc| |冰岛文 |Icelandic|is|
+|马拉地文|Marathi|mr| |库尔德文 |Kurdish|ku|
+|尼泊尔文|Nepali|ne| |立陶宛文|Lithuanian |lt|
+|塞尔维亚文（cyrillic)|Serbian(cyrillic)|rs_cyrillic| |拉脱维亚文 |Latvian |lv|
+|毛利文|Maori|mi| | 达尔瓦文|Dargwa |dar|
+|马来文 |Malay|ms| | 因古什文|Ingush |inh|
+|马耳他文 |Maltese |mt| | 拉克文|Lak |lbe|
+|荷兰文 |Dutch |nl| | 莱兹甘文|Lezghian |lez|
+|挪威文 |Norwegian |no| |塔巴萨兰文 |Tabassaran |tab|
+|波兰文|Polish |pl| | 比尔哈文|Bihari |bh|
+| 罗马尼亚文|Romanian |ro| | 迈蒂利文|Maithili |mai|
+| 斯洛伐克文|Slovak |sk| | 昂加文|Angika |ang|
+| 斯洛文尼亚文|Slovenian |sl| | 孟加拉文|Bhojpuri |bho|
+| 阿尔巴尼亚文|Albanian |sq| | 摩揭陀文 |Magahi |mah|
+| 瑞典文|Swedish |sv| | 那格浦尔文|Nagpur |sck|
+| 西瓦希里文|Swahili |sw| | 尼瓦尔文|Newari |new|
+| 塔加洛文|Tagalog |tl| | 保加利亚文 |Goan Konkani|gom|
+| 土耳其文|Turkish |tr| | 沙特阿拉伯文|Saudi Arabia|sa|
+| 乌兹别克文|Uzbek |uz| | 阿瓦尔文|Avar |ava|
+| 越南文|Vietnamese |vi| | 阿瓦尔文|Avar |ava|
+| 蒙古文|Mongolian |mn| | 阿迪赫文|Adyghe |ady|
diff --git a/docs/ppocr/blog/ocr_book.en.md b/docs/ppocr/blog/ocr_book.en.md
new file mode 100644
index 0000000000..3228eb942e
--- /dev/null
+++ b/docs/ppocr/blog/ocr_book.en.md
@@ -0,0 +1,28 @@
+---
+comments: true
+typora-copy-images-to: images
+---
+
+# E-book: *Dive Into OCR*
+
+"Dive Into OCR" is a textbook that combines OCR theory and practice, written by the PaddleOCR community. The main features are as follows:
+
+- OCR full-stack technology covering text detection, recognition and document analysis
+- Closely integrate theory and practice, cross the code implementation gap, and supporting instructional videos
+- Jupyter Notebook textbook, flexibly modifying code for instant results
+
+## Structure
+
+
+
+- The first part is the preliminary knowledge of the book, including the knowledge index and resource links needed in the process of positioning and using the book content of the book
+
+- The second part is chapters 4-8 of the book, which introduce the concepts, applications, and industry practices related to the detection and identification capabilities of the OCR engine. In the "Introduction to OCR Technology", the application scenarios and challenges of OCR, the basic concepts of technology, and the pain points in industrial applications are comprehensively explained. Then, in the two chapters of "Text Detection" and "Text Recognition", the two basic tasks of OCR are introduced. In each chapter, an algorithm is accompanied by a detailed explanation of the code and practical exercises. Chapters 6 and 7 are a detailed introduction to the PP-OCR series model, PP-OCR is a set of OCR systems for industrial applications, on the basis of the basic detection and identification model, after a series of optimization strategies to achieve the general field of industrial SOTA model, while opening up a variety of predictive deployment solutions, enabling enterprises to quickly land OCR applications.
+
+- The third part is chapter 9-12 of the book, which introduces applications other than the two-stage OCR engine, including data synthesis, preprocessing algorithm, and end-to-end model, focusing on OCR's layout analysis, table recognition, visual document question and answer capabilities in the document scene, and also through the combination of algorithm and code, so that readers can deeply understand and apply.
+
+## Address
+
+- [E-book: *Dive Into OCR* (PDF)](https://paddleocr.bj.bcebos.com/ebook/Dive_into_OCR.pdf)
+- [Notebook (.ipynb)](https://github.com/PaddleOCR-Community/Dive-into-OCR)
+- [Videos (Chinese only)](https://aistudio.baidu.com/aistudio/education/group/info/25207)
diff --git a/docs/ppocr/blog/ocr_book.md b/docs/ppocr/blog/ocr_book.md
new file mode 100644
index 0000000000..e3e51a77f0
--- /dev/null
+++ b/docs/ppocr/blog/ocr_book.md
@@ -0,0 +1,29 @@
+---
+comments: true
+typora-copy-images-to: images
+---
+
+# 《动手学OCR》电子书
+
+《动手学OCR》是PaddleOCR团队携手华中科技大学博导/教授，IAPR Fellow 白翔、复旦大学青年研究员陈智能、中国移动研究院视觉领域资深专家黄文辉、中国工商银行大数据人工智能实验室研究员等产学研同仁，以及OCR开发者共同打造的结合OCR前沿理论与代码实践的教材。主要特色如下：
+
+- 覆盖从文本检测识别到文档分析的OCR全栈技术
+- 紧密结合理论实践，跨越代码实现鸿沟，并配套教学视频
+- Notebook交互式学习，灵活修改代码，即刻获得结果
+
+## 本书结构
+
+
+
+- 第一部分是本书的推荐序、序言与预备知识，包含本书的定位与使用书籍内容的过程中需要用到的知识索引、资源链接等
+- 第二部分是本书的4-8章，介绍与OCR核心的检测、识别能力相关的概念、应用与产业实践。在“OCR技术导论”中总括性的解释OCR的应用场景和挑战、技术基本概念以及在产业应用中的痛点问题。然后在
+“文本检测”与“文本识别”两章中介绍OCR的两个基本任务，并在每章中配套一个算法展开代码详解与实战练习。第6、7章是关于PP-OCR系列模型的详细介绍，PP-OCR是一套面向产业应用的OCR系统，在
+基础检测和识别模型的基础之上经过一系列优化策略达到通用领域的产业级SOTA模型，同时打通多种预测部署方案，赋能企业快速落地OCR应用。
+- 第三部分是本书的9-12章，介绍两阶段OCR引擎之外的应用，包括数据合成、预处理算法、端到端模型，重点展开了OCR在文档场景下的版面分析、表格识别、视觉文档问答的能力，同样通过算法与代码结
+合的方式使得读者能够深入理解并应用。
+
+## 资料地址
+
+- 中文版电子书下载请扫描首页二维码入群后领取
+- [notebook教程](https://github.com/PaddleOCR-Community/Dive-into-OCR)
+- [教学视频](https://aistudio.baidu.com/aistudio/education/group/info/25207)
diff --git a/docs/ppocr/blog/slice.en.md b/docs/ppocr/blog/slice.en.md
new file mode 100644
index 0000000000..0b8b95d135
--- /dev/null
+++ b/docs/ppocr/blog/slice.en.md
@@ -0,0 +1,22 @@
+---
+comments: true
+---
+
+
+# Slice Operator
+
+If you have a very large image/document that you would like to run PaddleOCR (detection and recognition) on, you can use the slice operation as follows:
+
+`ocr_inst = PaddleOCR(**ocr_settings)`
+`results = ocr_inst.ocr(img, det=True,rec=True, slice=slice, cls=False,bin=False,inv=False,alpha_color=False)`
+
+where
+`slice = {'horizontal_stride': h_stride, 'vertical_stride':v_stride, 'merge_x_thres':x_thres, 'merge_y_thres': y_thres}`
+
+Here, `h_stride`, `v_stride`, `x_thres`, and `y_thres` are user-configurable values and need to be set manually. The way the `slice` operator works is that it runs a sliding window across the large input image, creating slices of it and runs the OCR algorithms on it.
+
+The fragmented slice-level results are then merged together to output image-level detection and recognition results. The horizontal and vertical strides cannot be lower than a certain limit (as too low values would create so many slices it would be very computationally expensive to get results for each of them). However, as an example the recommended values for an image with dimensions 6616x14886 would be as follows.
+
+`slice = {'horizontal_stride': 300, 'vertical_stride':500, 'merge_x_thres':50, 'merge_y_thres': 35}`
+
+All slice-level detections with bounding boxes as close as `merge_x_thres` and `merge_y_thres` will be merged together.
diff --git a/docs/ppocr/blog/slice.md b/docs/ppocr/blog/slice.md
new file mode 100644
index 0000000000..eae91ad399
--- /dev/null
+++ b/docs/ppocr/blog/slice.md
@@ -0,0 +1,25 @@
+---
+comments: true
+---
+
+# 切片操作
+
+如果希望运行 PaddleOCR 处理一张非常大的图像或文档，对其进行检测和识别，可以使用切片操作，如下所示：
+
+```python linenums="1"
+ocr_inst = PaddleOCR(**ocr_settings)
+results = ocr_inst.ocr(img, det=True, rec=True, slice=slice, cls=False, bin=False, inv=False, alpha_color=False)
+```
+
+其中，
+`slice = {'horizontal_stride': h_stride, 'vertical_stride': v_stride, 'merge_x_thres': x_thres, 'merge_y_thres': y_thres}`
+
+这里的 `h_stride`、`v_stride`、`x_thres` 和 `y_thres` 是用户可配置的参数，需要手动设置。切片操作符的工作原理是，在大图像上运行一个滑动窗口，创建图像的切片，并在这些切片上运行 OCR 算法。
+
+然后将这些切片级别的零散结果合并，生成图像级别的检测和识别结果。水平和垂直步幅不能低于一定限度，因为过低的值会产生太多切片，导致计算结果非常耗时。例如，对于尺寸为 6616x14886 的图像，推荐使用以下参数：
+
+```python linenums="1"
+slice = {'horizontal_stride': 300, 'vertical_stride': 500, 'merge_x_thres': 50, 'merge_y_thres': 35}
+```
+
+所有边界框接近 `merge_x_thres` 和 `merge_y_thres` 的切片级检测结果将被合并在一起。
diff --git a/docs/ppocr/blog/whl.en.md b/docs/ppocr/blog/whl.en.md
new file mode 100644
index 0000000000..190a9372ca
--- /dev/null
+++ b/docs/ppocr/blog/whl.en.md
@@ -0,0 +1,488 @@
+---
+typora-copy-images-to: images
+comments: true
+---
+
+# Paddleocr Package
+
+## 1 Get started quickly
+
+### 1.1 install package
+
+install by pypi
+
+```bash linenums="1"
+pip install "paddleocr>=2.0.1" # Recommend to use version 2.0.1+
+```
+
+build own whl package and install
+
+```bash linenums="1"
+python3 -m build
+pip3 install dist/paddleocr-x.x.x-py3-none-any.whl # x.x.x is the version of paddleocr
+```
+
+## 2 Use
+
+### 2.1 Use by code
+
+The paddleocr whl package will automatically download the ppocr lightweight model as the default model, which can be customized and replaced according to the section 3 **Custom Model**.
+
+* detection angle classification and recognition
+
+```python linenums="1"
+from paddleocr import PaddleOCR,draw_ocr
+# Paddleocr supports Chinese, English, French, German, Korean and Japanese.
+# You can set the parameter `lang` as `ch`, `en`, `french`, `german`, `korean`, `japan`
+# to switch the language model in order.
+ocr = PaddleOCR(use_angle_cls=True, lang='en') # need to run only once to download and load model into memory
+img_path = 'PaddleOCR/doc/imgs_en/img_12.jpg'
+result = ocr.ocr(img_path, cls=True)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+
+# draw result
+from PIL import Image
+result = result[0]
+image = Image.open(img_path).convert('RGB')
+boxes = [line[0] for line in result]
+txts = [line[1][0] for line in result]
+scores = [line[1][1] for line in result]
+im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf')
+im_show = Image.fromarray(im_show)
+im_show.save('result.jpg')
+```
+
+Output will be a list, each item contains bounding box, text and recognition confidence
+
+```bash linenums="1"
+[[[442.0, 173.0], [1169.0, 173.0], [1169.0, 225.0], [442.0, 225.0]], ['ACKNOWLEDGEMENTS', 0.99283075]]
+[[[393.0, 340.0], [1207.0, 342.0], [1207.0, 389.0], [393.0, 387.0]], ['We would like to thank all the designers and', 0.9357758]]
+[[[399.0, 398.0], [1204.0, 398.0], [1204.0, 433.0], [399.0, 433.0]], ['contributors whohave been involved in the', 0.9592447]]
+......
+```
+
+Visualization of results
+
+
+
+* detection and recognition
+
+```python linenums="1"
+from paddleocr import PaddleOCR,draw_ocr
+ocr = PaddleOCR(lang='en') # need to run only once to download and load model into memory
+img_path = 'PaddleOCR/doc/imgs_en/img_12.jpg'
+result = ocr.ocr(img_path, cls=False)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+
+# draw result
+from PIL import Image
+result = result[0]
+image = Image.open(img_path).convert('RGB')
+boxes = [line[0] for line in result]
+txts = [line[1][0] for line in result]
+scores = [line[1][1] for line in result]
+im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf')
+im_show = Image.fromarray(im_show)
+im_show.save('result.jpg')
+```
+
+Output will be a list, each item contains bounding box, text and recognition confidence
+
+```bash linenums="1"
+[[[442.0, 173.0], [1169.0, 173.0], [1169.0, 225.0], [442.0, 225.0]], ['ACKNOWLEDGEMENTS', 0.99283075]]
+[[[393.0, 340.0], [1207.0, 342.0], [1207.0, 389.0], [393.0, 387.0]], ['We would like to thank all the designers and', 0.9357758]]
+[[[399.0, 398.0], [1204.0, 398.0], [1204.0, 433.0], [399.0, 433.0]], ['contributors whohave been involved in the', 0.9592447]]
+......
+```
+
+Visualization of results
+
+
+
+* classification and recognition
+
+```python linenums="1"
+from paddleocr import PaddleOCR
+ocr = PaddleOCR(use_angle_cls=True, lang='en') # need to run only once to load model into memory
+img_path = 'PaddleOCR/doc/imgs_words_en/word_10.png'
+result = ocr.ocr(img_path, det=False, cls=True)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+```
+
+Output will be a list, each item contains recognition text and confidence
+
+```bash linenums="1"
+['PAIN', 0.990372]
+```
+
+* only detection
+
+```python linenums="1"
+from paddleocr import PaddleOCR,draw_ocr
+ocr = PaddleOCR() # need to run only once to download and load model into memory
+img_path = 'PaddleOCR/doc/imgs_en/img_12.jpg'
+result = ocr.ocr(img_path,rec=False)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+
+# draw result
+from PIL import Image
+result = result[0]
+image = Image.open(img_path).convert('RGB')
+im_show = draw_ocr(image, result, txts=None, scores=None, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf')
+im_show = Image.fromarray(im_show)
+im_show.save('result.jpg')
+```
+
+Output will be a list, each item only contains bounding box
+
+```bash linenums="1"
+[[756.0, 812.0], [805.0, 812.0], [805.0, 830.0], [756.0, 830.0]]
+[[820.0, 803.0], [1085.0, 801.0], [1085.0, 836.0], [820.0, 838.0]]
+[[393.0, 801.0], [715.0, 805.0], [715.0, 839.0], [393.0, 836.0]]
+......
+```
+
+Visualization of results
+
+
+
+* only recognition
+
+```python linenums="1"
+from paddleocr import PaddleOCR
+ocr = PaddleOCR(lang='en') # need to run only once to load model into memory
+img_path = 'PaddleOCR/doc/imgs_words_en/word_10.png'
+result = ocr.ocr(img_path, det=False, cls=False)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+```
+
+Output will be a list, each item contains recognition text and confidence
+
+```bash linenums="1"
+['PAIN', 0.990372]
+```
+
+* only classification
+
+```python linenums="1"
+from paddleocr import PaddleOCR
+ocr = PaddleOCR(use_angle_cls=True) # need to run only once to load model into memory
+img_path = 'PaddleOCR/doc/imgs_words_en/word_10.png'
+result = ocr.ocr(img_path, det=False, rec=False, cls=True)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+```
+
+Output will be a list, each item contains classification result and confidence
+
+```bash linenums="1"
+['0', 0.99999964]
+```
+
+### 2.2 Use by command line
+
+show help information
+
+```bash linenums="1"
+paddleocr -h
+```
+
+* detection classification and recognition
+
+```bash linenums="1"
+paddleocr --image_dir PaddleOCR/doc/imgs_en/img_12.jpg --use_angle_cls true --lang en
+```
+
+Output will be a list, each item contains bounding box, text and recognition confidence
+
+```bash linenums="1"
+[[[441.0, 174.0], [1166.0, 176.0], [1165.0, 222.0], [441.0, 221.0]], ('ACKNOWLEDGEMENTS', 0.9971134662628174)]
+[[[403.0, 346.0], [1204.0, 348.0], [1204.0, 384.0], [402.0, 383.0]], ('We would like to thank all the designers and', 0.9761400818824768)]
+[[[403.0, 396.0], [1204.0, 398.0], [1204.0, 434.0], [402.0, 433.0]], ('contributors who have been involved in the', 0.9791957139968872)]
+......
+```
+
+pdf file is also supported, you can infer the first few pages by using the `page_num` parameter, the default is 0, which means infer all pages
+
+```bash linenums="1"
+paddleocr --image_dir ./xxx.pdf --use_angle_cls true --use_gpu false --page_num 2
+```
+
+* detection and recognition
+
+```bash linenums="1"
+paddleocr --image_dir PaddleOCR/doc/imgs_en/img_12.jpg --lang en
+```
+
+Output will be a list, each item contains bounding box, text and recognition confidence
+
+```bash linenums="1"
+[[[441.0, 174.0], [1166.0, 176.0], [1165.0, 222.0], [441.0, 221.0]], ('ACKNOWLEDGEMENTS', 0.9971134662628174)]
+[[[403.0, 346.0], [1204.0, 348.0], [1204.0, 384.0], [402.0, 383.0]], ('We would like to thank all the designers and', 0.9761400818824768)]
+[[[403.0, 396.0], [1204.0, 398.0], [1204.0, 434.0], [402.0, 433.0]], ('contributors who have been involved in the', 0.9791957139968872)]
+......
+```
+
+* classification and recognition
+
+```bash linenums="1"
+paddleocr --image_dir PaddleOCR/doc/imgs_words_en/word_10.png --use_angle_cls true --det false --lang en
+```
+
+Output will be a list, each item contains text and recognition confidence
+
+```bash linenums="1"
+['PAIN', 0.9934559464454651]
+```
+
+* only detection
+
+```bash linenums="1"
+paddleocr --image_dir PaddleOCR/doc/imgs_en/img_12.jpg --rec false
+```
+
+Output will be a list, each item only contains bounding box
+
+```bash linenums="1"
+[[397.0, 802.0], [1092.0, 802.0], [1092.0, 841.0], [397.0, 841.0]]
+[[397.0, 750.0], [1211.0, 750.0], [1211.0, 789.0], [397.0, 789.0]]
+[[397.0, 702.0], [1209.0, 698.0], [1209.0, 734.0], [397.0, 738.0]]
+......
+```
+
+* only recognition
+
+```bash linenums="1"
+paddleocr --image_dir PaddleOCR/doc/imgs_words_en/word_10.png --det false --lang en
+```
+
+Output will be a list, each item contains text and recognition confidence
+
+```bash linenums="1"
+['PAIN', 0.9934559464454651]
+```
+
+* only classification
+
+```bash linenums="1"
+paddleocr --image_dir PaddleOCR/doc/imgs_words_en/word_10.png --use_angle_cls true --det false --rec false
+```
+
+Output will be a list, each item contains classification result and confidence
+
+```bash linenums="1"
+['0', 0.99999964]
+```
+
+## 3 Use custom model
+
+When the built-in model cannot meet the needs, you need to use your own trained model.
+First, refer to [export](../model_train/detection.en.md#4-inference) doc to convert your det and rec model to inference model, and then use it as follows
+
+### 3.1 Use by code
+
+```python linenums="1"
+from paddleocr import PaddleOCR,draw_ocr
+# The path of detection and recognition model must contain model and params files
+ocr = PaddleOCR(det_model_dir='{your_det_model_dir}', rec_model_dir='{your_rec_model_dir}', rec_char_dict_path='{your_rec_char_dict_path}', cls_model_dir='{your_cls_model_dir}', use_angle_cls=True)
+img_path = 'PaddleOCR/doc/imgs_en/img_12.jpg'
+result = ocr.ocr(img_path, cls=True)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+
+# draw result
+from PIL import Image
+result = result[0]
+image = Image.open(img_path).convert('RGB')
+boxes = [line[0] for line in result]
+txts = [line[1][0] for line in result]
+scores = [line[1][1] for line in result]
+im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf')
+im_show = Image.fromarray(im_show)
+im_show.save('result.jpg')
+```
+
+### 3.2 Use by command line
+
+```bash linenums="1"
+paddleocr --image_dir PaddleOCR/doc/imgs/11.jpg --det_model_dir {your_det_model_dir} --rec_model_dir {your_rec_model_dir} --rec_char_dict_path {your_rec_char_dict_path} --cls_model_dir {your_cls_model_dir} --use_angle_cls true
+```
+
+## 4 Use web images or numpy array as input
+
+### 4.1 Web image
+
+* Use by code
+
+```python linenums="1"
+from paddleocr import PaddleOCR, draw_ocr
+ocr = PaddleOCR(use_angle_cls=True, lang="ch") # need to run only once to download and load model into memory
+img_path = 'http://n.sinaimg.cn/ent/transform/w630h933/20171222/o111-fypvuqf1838418.jpg'
+result = ocr.ocr(img_path, cls=True)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+
+# show result
+from PIL import Image
+result = result[0]
+image = Image.open(img_path).convert('RGB')
+boxes = [line[0] for line in result]
+txts = [line[1][0] for line in result]
+scores = [line[1][1] for line in result]
+im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf')
+im_show = Image.fromarray(im_show)
+im_show.save('result.jpg')
+```
+
+* Use by command line
+
+```bash linenums="1"
+paddleocr --image_dir http://n.sinaimg.cn/ent/transform/w630h933/20171222/o111-fypvuqf1838418.jpg --use_angle_cls=true
+```
+
+### 4.2 Numpy array
+
+Support numpy array as input only when used by code
+
+```python linenums="1"
+import cv2
+from paddleocr import PaddleOCR, draw_ocr, download_with_progressbar
+ocr = PaddleOCR(use_angle_cls=True, lang="ch") # need to run only once to download and load model into memory
+img_path = 'PaddleOCR/doc/imgs/11.jpg'
+img = cv2.imread(img_path)
+# img = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY), If your own training model supports grayscale images, you can uncomment this line
+result = ocr.ocr(img, cls=True)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+
+# show result
+from PIL import Image
+result = result[0]
+download_with_progressbar(img_path, 'tmp.jpg')
+image = Image.open('tmp.jpg').convert('RGB')
+boxes = [line[0] for line in result]
+txts = [line[1][0] for line in result]
+scores = [line[1][1] for line in result]
+im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf')
+im_show = Image.fromarray(im_show)
+im_show.save('result.jpg')
+```
+
+## 5 PDF file
+
+* Use by command line
+
+you can infer the first few pages by using the `page_num` parameter, the default is 0, which means infer all pages
+
+```bash linenums="1"
+paddleocr --image_dir ./xxx.pdf --use_angle_cls true --use_gpu false --page_num 2
+```
+
+* Use by code
+
+```python linenums="1"
+from paddleocr import PaddleOCR, draw_ocr
+
+# Paddleocr supports Chinese, English, French, German, Korean and Japanese.
+# You can set the parameter `lang` as `ch`, `en`, `fr`, `german`, `korean`, `japan`
+# to switch the language model in order.
+ocr = PaddleOCR(use_angle_cls=True, lang="ch"， page_num=2) # need to run only once to download and load model into memory
+img_path = './xxx.pdf'
+result = ocr.ocr(img_path, cls=True)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+
+# draw result
+import fitz
+from PIL import Image
+import cv2
+import numpy as np
+imgs = []
+with fitz.open(img_path) as pdf:
+ for pg in range(0, pdf.pageCount):
+ page = pdf[pg]
+ mat = fitz.Matrix(2, 2)
+ pm = page.getPixmap(matrix=mat, alpha=False)
+ # if width or height > 2000 pixels, don't enlarge the image
+ if pm.width > 2000 or pm.height > 2000:
+ pm = page.getPixmap(matrix=fitz.Matrix(1, 1), alpha=False)
+
+ img = Image.frombytes("RGB", [pm.width, pm.height], pm.samples)
+ img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)
+ imgs.append(img)
+for idx in range(len(result)):
+ res = result[idx]
+ image = imgs[idx]
+ boxes = [line[0] for line in res]
+ txts = [line[1][0] for line in res]
+ scores = [line[1][1] for line in res]
+ im_show = draw_ocr(image, boxes, txts, scores, font_path='doc/fonts/simfang.ttf')
+ im_show = Image.fromarray(im_show)
+ im_show.save('result_page_{}.jpg'.format(idx))
+```
+
+## 6 Parameter Description
+
+| Parameter | Description | Default value |
+|-------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------|
+| use_gpu | use GPU or not | TRUE |
+| gpu_mem | GPU memory size used for initialization | 8000M |
+| image_dir | The images path or folder path for predicting when used by the command line | |
+| page_num | Valid when the input type is pdf file, specify to predict the previous page_num pages, all pages are predicted by default | 0 |
+| det_algorithm | Type of detection algorithm selected | DB |
+| det_model_dir | the text detection inference model folder. There are two ways to transfer parameters, 1. None: Automatically download the built-in model to `~/.paddleocr/det`; 2. The path of the inference model converted by yourself, the model and params files must be included in the model path | None |
+| det_max_side_len | The maximum size of the long side of the image. When the long side exceeds this value, the long side will be resized to this size, and the short side will be scaled proportionally | 960 |
+| det_db_thresh | Binarization threshold value of DB output map | 0.3 |
+| det_db_box_thresh | The threshold value of the DB output box. Boxes score lower than this value will be discarded | 0.5 |
+| det_db_unclip_ratio | The expanded ratio of DB output box | 2 |
+| det_db_score_mode | The parameter that control how the score of the detection frame is calculated. There are 'fast' and 'slow' options. If the text to be detected is curved, it is recommended to use 'slow' | 'fast' |
+| det_east_score_thresh | Binarization threshold value of EAST output map | 0.8 |
+| det_east_cover_thresh | The threshold value of the EAST output box. Boxes score lower than this value will be discarded | 0.1 |
+| det_east_nms_thresh | The NMS threshold value of EAST model output box | 0.2 |
+| rec_algorithm | Type of recognition algorithm selected | CRNN |
+| rec_model_dir | the text recognition inference model folder. There are two ways to transfer parameters, 1. None: Automatically download the built-in model to `~/.paddleocr/rec`; 2. The path of the inference model converted by yourself, the model and params files must be included in the model path | None |
+| rec_image_shape | image shape of recognition algorithm | "3,32,320" |
+| rec_batch_num | When performing recognition, the batchsize of forward images | 30 |
+| max_text_length | The maximum text length that the recognition algorithm can recognize | 25 |
+| rec_char_dict_path | the alphabet path which needs to be modified to your own path when `rec_model_Name` use mode 2 | ./ppocr/utils/ppocr_keys_v1.txt |
+| use_space_char | Whether to recognize spaces | TRUE |
+| drop_score | Filter the output by score (from the recognition model), and those below this score will not be returned | 0.5 |
+| use_angle_cls | Whether to load classification model | FALSE |
+| cls_model_dir | the classification inference model folder. There are two ways to transfer parameters, 1. None: Automatically download the built-in model to `~/.paddleocr/cls`; 2. The path of the inference model converted by yourself, the model and params files must be included in the model path | None |
+| cls_image_shape | image shape of classification algorithm | "3,48,192" |
+| label_list | label list of classification algorithm | ['0','180'] |
+| cls_batch_num | When performing classification, the batchsize of forward images | 30 |
+| enable_mkldnn | Whether to enable mkldnn | FALSE |
+| use_zero_copy_run | Whether to forward by zero_copy_run | FALSE |
+| lang | The support language, now only Chinese(ch)、English(en)、French(french)、German(german)、Korean(korean)、Japanese(japan) are supported | ch |
+| det | Enable detction when `ppocr.ocr` func exec | TRUE |
+| rec | Enable recognition when `ppocr.ocr` func exec | TRUE |
+| cls | Enable classification when `ppocr.ocr` func exec((Use use_angle_cls in command line mode to control whether to start classification in the forward direction) | FALSE |
+| show_log | Whether to print log| FALSE |
+| type | Perform ocr or table structuring, the value is selected in ['ocr','structure'] | ocr |
+| ocr_version | OCR Model version number, the current model support list is as follows: PP-OCRv3 supports Chinese and English detection, recognition, multilingual recognition, direction classifier models, PP-OCRv2 support Chinese detection and recognition model, PP-OCR support Chinese detection, recognition and direction classifier, multilingual recognition model | PP-OCRv3 |
diff --git a/docs/ppocr/blog/whl.md b/docs/ppocr/blog/whl.md
new file mode 100644
index 0000000000..722c0462ed
--- /dev/null
+++ b/docs/ppocr/blog/whl.md
@@ -0,0 +1,501 @@
+---
+typora-copy-images-to: images
+comments: true
+---
+
+# paddleocr package使用说明
+
+## 1 快速上手
+
+### 1.1 安装whl包
+
+pip安装
+
+```bash linenums="1"
+pip install paddleocr
+```
+
+本地构建并安装
+
+```bash linenums="1"
+python3 -m build
+pip3 install dist/paddleocr-x.x.x-py3-none-any.whl # x.x.x是paddleocr的版本号
+```
+
+## 2 使用
+
+### 2.1 代码使用
+
+paddleocr whl包会自动下载ppocr轻量级模型作为默认模型，可以根据第3节**自定义模型**进行自定义更换。
+
+* 检测+方向分类器+识别全流程
+
+```python linenums="1"
+from paddleocr import PaddleOCR, draw_ocr
+
+# Paddleocr目前支持中英文、英文、法语、德语、韩语、日语，可以通过修改lang参数进行切换
+# 参数依次为`ch`, `en`, `french`, `german`, `korean`, `japan`。
+ocr = PaddleOCR(use_angle_cls=True, lang="ch") # need to run only once to download and load model into memory
+img_path = 'PaddleOCR/doc/imgs/11.jpg'
+result = ocr.ocr(img_path, cls=True)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+
+# 显示结果
+from PIL import Image
+result = result[0]
+image = Image.open(img_path).convert('RGB')
+boxes = [line[0] for line in result]
+txts = [line[1][0] for line in result]
+scores = [line[1][1] for line in result]
+im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf')
+im_show = Image.fromarray(im_show)
+im_show.save('result.jpg')
+```
+
+结果是一个list，每个item包含了文本框，文字和识别置信度
+
+```bash linenums="1"
+[[[24.0, 36.0], [304.0, 34.0], [304.0, 72.0], [24.0, 74.0]], ['纯臻营养护发素', 0.964739]]
+[[[24.0, 80.0], [172.0, 80.0], [172.0, 104.0], [24.0, 104.0]], ['产品信息/参数', 0.98069626]]
+[[[24.0, 109.0], [333.0, 109.0], [333.0, 136.0], [24.0, 136.0]], ['（45元/每公斤，100公斤起订）', 0.9676722]]
+......
+```
+
+结果可视化
+
+
+ 
+
+
+* 检测+识别
+
+```python linenums="1"
+from paddleocr import PaddleOCR, draw_ocr
+
+ocr = PaddleOCR() # need to run only once to download and load model into memory
+img_path = 'PaddleOCR/doc/imgs/11.jpg'
+result = ocr.ocr(img_path, cls=False)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+
+# 显示结果
+from PIL import Image
+result = result[0]
+image = Image.open(img_path).convert('RGB')
+boxes = [line[0] for line in result]
+txts = [line[1][0] for line in result]
+scores = [line[1][1] for line in result]
+im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf')
+im_show = Image.fromarray(im_show)
+im_show.save('result.jpg')
+```
+
+结果是一个list，每个item包含了文本框，文字和识别置信度
+
+```bash linenums="1"
+[[[24.0, 36.0], [304.0, 34.0], [304.0, 72.0], [24.0, 74.0]], ['纯臻营养护发素', 0.964739]]
+[[[24.0, 80.0], [172.0, 80.0], [172.0, 104.0], [24.0, 104.0]], ['产品信息/参数', 0.98069626]]
+[[[24.0, 109.0], [333.0, 109.0], [333.0, 136.0], [24.0, 136.0]], ['（45元/每公斤，100公斤起订）', 0.9676722]]
+......
+```
+
+结果可视化
+
+
+
+
+
+
+* 方向分类器+识别
+
+```python linenums="1"
+from paddleocr import PaddleOCR
+
+ocr = PaddleOCR(use_angle_cls=True) # need to run only once to download and load model into memory
+img_path = 'PaddleOCR/doc/imgs_words/ch/word_1.jpg'
+result = ocr.ocr(img_path, det=False, cls=True)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+```
+
+结果是一个list，每个item只包含识别结果和识别置信度
+
+```bash linenums="1"
+['韩国小馆', 0.9907421]
+```
+
+* 单独执行检测
+
+```python linenums="1"
+from paddleocr import PaddleOCR, draw_ocr
+
+ocr = PaddleOCR() # need to run only once to download and load model into memory
+img_path = 'PaddleOCR/doc/imgs/11.jpg'
+result = ocr.ocr(img_path, rec=False)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+
+# 显示结果
+from PIL import Image
+result = result[0]
+image = Image.open(img_path).convert('RGB')
+im_show = draw_ocr(image, result, txts=None, scores=None, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf')
+im_show = Image.fromarray(im_show)
+im_show.save('result.jpg')
+```
+
+结果是一个list，每个item只包含文本框
+
+```bash linenums="1"
+[[26.0, 457.0], [137.0, 457.0], [137.0, 477.0], [26.0, 477.0]]
+[[25.0, 425.0], [372.0, 425.0], [372.0, 448.0], [25.0, 448.0]]
+[[128.0, 397.0], [273.0, 397.0], [273.0, 414.0], [128.0, 414.0]]
+......
+```
+
+结果可视化
+
+
+
+ 
+
+
+* 单独执行识别
+
+```python linenums="1"
+from paddleocr import PaddleOCR
+
+ocr = PaddleOCR() # need to run only once to download and load model into memory
+img_path = 'PaddleOCR/doc/imgs_words/ch/word_1.jpg'
+result = ocr.ocr(img_path, det=False)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+```
+
+结果是一个list，每个item只包含识别结果和识别置信度
+
+```bash linenums="1"
+['韩国小馆', 0.9907421]
+```
+
+* 单独执行方向分类器
+
+```python linenums="1"
+from paddleocr import PaddleOCR
+
+ocr = PaddleOCR(use_angle_cls=True) # need to run only once to download and load model into memory
+img_path = 'PaddleOCR/doc/imgs_words/ch/word_1.jpg'
+result = ocr.ocr(img_path, det=False, rec=False, cls=True)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+```
+
+结果是一个list，每个item只包含分类结果和分类置信度
+
+```bash linenums="1"
+['0', 0.9999924]
+```
+
+### 2.2 通过命令行使用
+
+查看帮助信息
+
+```bash linenums="1"
+paddleocr -h
+```
+
+* 检测+方向分类器+识别全流程
+
+```bash linenums="1"
+paddleocr --image_dir PaddleOCR/doc/imgs/11.jpg --use_angle_cls true
+```
+
+结果是一个list，每个item包含了文本框，文字和识别置信度
+
+```bash linenums="1"
+[[[28.0, 37.0], [302.0, 39.0], [302.0, 72.0], [27.0, 70.0]], ('纯臻营养护发素', 0.9658738374710083)]
+......
+```
+
+此外，paddleocr也支持输入pdf文件，并且可以通过指定参数`page_num`来控制推理前面几页，默认为0，表示推理所有页。
+
+```bash linenums="1"
+paddleocr --image_dir ./xxx.pdf --use_angle_cls true --use_gpu false --page_num 2
+```
+
+* 检测+识别
+
+```bash linenums="1"
+paddleocr --image_dir PaddleOCR/doc/imgs/11.jpg
+```
+
+结果是一个list，每个item包含了文本框，文字和识别置信度
+
+```bash linenums="1"
+[[[28.0, 37.0], [302.0, 39.0], [302.0, 72.0], [27.0, 70.0]], ('纯臻营养护发素', 0.9658738374710083)]
+......
+```
+
+* 方向分类器+识别
+
+```bash linenums="1"
+paddleocr --image_dir PaddleOCR/doc/imgs_words/ch/word_1.jpg --use_angle_cls true --det false
+```
+
+结果是一个list，每个item只包含识别结果和识别置信度
+
+```bash linenums="1"
+['韩国小馆', 0.994467]
+```
+
+* 单独执行检测
+
+```bash linenums="1"
+paddleocr --image_dir PaddleOCR/doc/imgs/11.jpg --rec false
+```
+
+结果是一个list，每个item只包含文本框
+
+```bash linenums="1"
+[[27.0, 459.0], [136.0, 459.0], [136.0, 479.0], [27.0, 479.0]]
+[[28.0, 429.0], [372.0, 429.0], [372.0, 445.0], [28.0, 445.0]]
+......
+```
+
+* 单独执行识别
+
+```bash linenums="1"
+paddleocr --image_dir PaddleOCR/doc/imgs_words/ch/word_1.jpg --det false
+```
+
+结果是一个list，每个item只包含识别结果和识别置信度
+
+```bash linenums="1"
+['韩国小馆', 0.994467]
+```
+
+* 单独执行方向分类器
+
+```bash linenums="1"
+paddleocr --image_dir PaddleOCR/doc/imgs_words/ch/word_1.jpg --use_angle_cls true --det false --rec false
+```
+
+结果是一个list，每个item只包含分类结果和分类置信度
+
+```bash linenums="1"
+['0', 0.9999924]
+```
+
+## 3 自定义模型
+
+当内置模型无法满足需求时，需要使用到自己训练的模型。 首先，参照[模型导出](../model_train/detection.md#4-模型导出与预测)将检测、分类和识别模型转换为inference模型，然后按照如下方式使用
+
+### 3.1 代码使用
+
+```python linenums="1"
+from paddleocr import PaddleOCR, draw_ocr
+
+# 模型路径下必须含有model和params文件
+ocr = PaddleOCR(det_model_dir='{your_det_model_dir}', rec_model_dir='{your_rec_model_dir}',
+ rec_char_dict_path='{your_rec_char_dict_path}', cls_model_dir='{your_cls_model_dir}',
+ use_angle_cls=True)
+img_path = 'PaddleOCR/doc/imgs/11.jpg'
+result = ocr.ocr(img_path, cls=True)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+
+# 显示结果
+from PIL import Image
+result = result[0]
+image = Image.open(img_path).convert('RGB')
+boxes = [line[0] for line in result]
+txts = [line[1][0] for line in result]
+scores = [line[1][1] for line in result]
+im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf')
+im_show = Image.fromarray(im_show)
+im_show.save('result.jpg')
+```
+
+### 3.2 通过命令行使用
+
+```bash linenums="1"
+paddleocr --image_dir PaddleOCR/doc/imgs/11.jpg --det_model_dir {your_det_model_dir} --rec_model_dir {your_rec_model_dir} --rec_char_dict_path {your_rec_char_dict_path} --cls_model_dir {your_cls_model_dir} --use_angle_cls true
+```
+
+## 4 使用网络图片或者numpy数组作为输入
+
+### 4.1 网络图片
+
+* 代码使用
+
+```python linenums="1"
+from paddleocr import PaddleOCR, draw_ocr, download_with_progressbar
+
+# Paddleocr目前支持中英文、英文、法语、德语、韩语、日语，可以通过修改lang参数进行切换
+# 参数依次为`ch`, `en`, `french`, `german`, `korean`, `japan`。
+ocr = PaddleOCR(use_angle_cls=True, lang="ch") # need to run only once to download and load model into memory
+img_path = 'http://n.sinaimg.cn/ent/transform/w630h933/20171222/o111-fypvuqf1838418.jpg'
+result = ocr.ocr(img_path, cls=True)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+
+# 显示结果
+from PIL import Image
+result = result[0]
+download_with_progressbar(img_path, 'tmp.jpg')
+image = Image.open('tmp.jpg').convert('RGB')
+boxes = [line[0] for line in result]
+txts = [line[1][0] for line in result]
+scores = [line[1][1] for line in result]
+im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf')
+im_show = Image.fromarray(im_show)
+im_show.save('result.jpg')
+```
+
+* 命令行模式
+
+```bash linenums="1"
+paddleocr --image_dir http://n.sinaimg.cn/ent/transform/w630h933/20171222/o111-fypvuqf1838418.jpg --use_angle_cls=true
+```
+
+### 4.2 numpy数组
+
+仅通过代码使用时支持numpy数组作为输入
+
+```python linenums="1"
+import cv2
+from paddleocr import PaddleOCR, draw_ocr
+
+# Paddleocr目前支持中英文、英文、法语、德语、韩语、日语，可以通过修改lang参数进行切换
+# 参数依次为`ch`, `en`, `french`, `german`, `korean`, `japan`。
+ocr = PaddleOCR(use_angle_cls=True, lang="ch") # need to run only once to download and load model into memory
+img_path = 'PaddleOCR/doc/imgs/11.jpg'
+img = cv2.imread(img_path)
+# img = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY), 如果你自己训练的模型支持灰度图，可以将这句话的注释取消
+result = ocr.ocr(img, cls=True)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+
+# 显示结果
+from PIL import Image
+result = result[0]
+image = Image.open(img_path).convert('RGB')
+boxes = [line[0] for line in result]
+txts = [line[1][0] for line in result]
+scores = [line[1][1] for line in result]
+im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/fonts/simfang.ttf')
+im_show = Image.fromarray(im_show)
+im_show.save('result.jpg')
+```
+
+## 5 PDF文件作为输入
+
+* 命令行模式
+
+可以通过指定参数`page_num`来控制推理前面几页，默认为0，表示推理所有页。
+
+```bash linenums="1"
+paddleocr --image_dir ./xxx.pdf --use_angle_cls true --use_gpu false --page_num 2
+```
+
+* 代码使用
+
+```python linenums="1"
+from paddleocr import PaddleOCR, draw_ocr
+
+# Paddleocr目前支持的多语言语种可以通过修改lang参数进行切换
+# 例如`ch`, `en`, `fr`, `german`, `korean`, `japan`
+ocr = PaddleOCR(use_angle_cls=True, lang="ch"， page_num=2) # need to run only once to download and load model into memory
+img_path = './xxx.pdf'
+result = ocr.ocr(img_path, cls=True)
+for idx in range(len(result)):
+ res = result[idx]
+ for line in res:
+ print(line)
+
+# 显示结果
+import fitz
+from PIL import Image
+import cv2
+import numpy as np
+imgs = []
+with fitz.open(img_path) as pdf:
+ for pg in range(0, pdf.pageCount):
+ page = pdf[pg]
+ mat = fitz.Matrix(2, 2)
+ pm = page.getPixmap(matrix=mat, alpha=False)
+ # if width or height > 2000 pixels, don't enlarge the image
+ if pm.width > 2000 or pm.height > 2000:
+ pm = page.getPixmap(matrix=fitz.Matrix(1, 1), alpha=False)
+
+ img = Image.frombytes("RGB", [pm.width, pm.height], pm.samples)
+ img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)
+ imgs.append(img)
+for idx in range(len(result)):
+ res = result[idx]
+ image = imgs[idx]
+ boxes = [line[0] for line in res]
+ txts = [line[1][0] for line in res]
+ scores = [line[1][1] for line in res]
+ im_show = draw_ocr(image, boxes, txts, scores, font_path='doc/fonts/simfang.ttf')
+ im_show = Image.fromarray(im_show)
+ im_show.save('result_page_{}.jpg'.format(idx))
+```
+
+## 6 参数说明
+
+| 字段 | 说明| 默认值 |
+|-------------------------|----------|-------------------------|
+| use_gpu| 是否使用GPU | TRUE |
+| gpu_mem| 初始化占用的GPU内存大小 | 8000M |
+| image_dir | 通过命令行调用时执行预测的图片或文件夹路径 |
+| page_num | 当输入类型为pdf文件时有效，指定预测前面page_num页，默认预测所有页 | 0|
+| det_algorithm | 使用的检测算法类型 | DB |
+| det_model_dir | 检测模型所在文件夹。传参方式有两种，1. None: 自动下载内置模型到 `~/.paddleocr/det`；2.自己转换好的inference模型路径，模型路径下必须包含model和params文件 | None |
+| det_max_side_len | 检测算法前向时图片长边的最大尺寸，当长边超出这个值时会将长边resize到这个大小，短边等比例缩放 | 960 |
+| det_db_thresh | DB模型输出预测图的二值化阈值 | 0.3 |
+| det_db_box_thresh | DB模型输出框的阈值，低于此值的预测框会被丢弃 | 0.5 |
+| det_db_unclip_ratio | DB模型输出框扩大的比例 | 2 |
+| det_db_score_mode | 计算检测框score的方式，有'fast'和'slow'，如果要检测的文字有弯曲，建议用'slow'，'slow'模式计算的box的score偏大，box不容易被过滤掉 | 'fast' |
+| det_east_score_thresh | EAST模型输出预测图的二值化阈值 | 0.8 |
+| det_east_cover_thresh | EAST模型输出框的阈值，低于此值的预测框会被丢弃 | 0.1 |
+| det_east_nms_thresh | EAST模型输出框NMS的阈值 | 0.2 |
+| rec_algorithm | 使用的识别算法类型 | CRNN |
+| rec_model_dir | 识别模型所在文件夹。传参方式有两种，1. None: 自动下载内置模型到 `~/.paddleocr/rec`；2.自己转换好的inference模型路径，模型路径下必须包含model和params文件 | None |
+| rec_image_shape | 识别算法的输入图片尺寸 | "3,32,320" |
+| rec_batch_num | 进行识别时，同时前向的图片数 | 30 |
+| max_text_length | 识别算法能识别的最大文字长度 | 25 |
+| rec_char_dict_path | 识别模型字典路径，当rec_model_dir使用方式2传参时需要修改为自己的字典路径 | ./ppocr/utils/ppocr_keys_v1.txt |
+| use_space_char | 是否识别空格 | TRUE |
+| drop_score | 对输出按照分数(来自于识别模型)进行过滤，低于此分数的不返回 | 0.5 |
+| use_angle_cls | 是否加载分类模型 | FALSE |
+| cls_model_dir | 分类模型所在文件夹。传参方式有两种，1. None: 自动下载内置模型到 `~/.paddleocr/cls`；2.自己转换好的inference模型路径，模型路径下必须包含model和params文件 | None |
+| cls_image_shape | 分类算法的输入图片尺寸 | "3, 48, 192" |
+| label_list | 分类算法的标签列表 | ['0', '180'] |
+| cls_batch_num | 进行分类时，同时前向的图片数 |30|
+| enable_mkldnn | 是否启用mkldnn | FALSE |
+| use_zero_copy_run | 是否通过zero_copy_run的方式进行前向| FALSE |
+| lang | 模型语言类型,目前支持 目前支持中英文(ch)、英文(en)、法语(french)、德语(german)、韩语(korean)、日语(japan) | ch |
+| det | 前向时使用启动检测 | TRUE |
+| rec | 前向时是否启动识别 | TRUE |
+| cls | 前向时是否启动分类 (命令行模式下使用use_angle_cls控制前向是否启动分类)| FALSE |
+| show_log | 是否打印logger信息| FALSE |
+| type | 执行ocr或者表格结构化, 值可选['ocr','structure'] | ocr |
+| ocr_version | OCR模型版本，可选PP-OCRv3, PP-OCRv2, PP-OCR。PP-OCRv3 支持中、英文的检测、识别、多语种识别，方向分类器等模型；PP-OCRv2 目前仅支持中文的检测和识别模型；PP-OCR支持中文的检测，识别，多语种识别，方向分类器等模型 | PP-OCRv3 |
diff --git a/docs/ppocr/environment.en.md b/docs/ppocr/environment.en.md
new file mode 100644
index 0000000000..141f342015
--- /dev/null
+++ b/docs/ppocr/environment.en.md
@@ -0,0 +1,313 @@
+---
+comments: true
+---
+
+# Environment Preparation
+
+Windows and Mac users are recommended to use Anaconda to build a Python environment, and Linux users are recommended to use docker to build a Python environment.
+
+Recommended working environment:
+
+- PaddlePaddle >= 2.1.2
+- Python 3.7
+- CUDA 10.1 / CUDA 10.2
+- cuDNN 7.6
+
+> If you already have a Python environment installed, you can skip to [PaddleOCR Quick Start](./quickstart_en.md).
+
+## 1. Python Environment Setup
+
+### 1.1 Windows
+
+#### 1.1.1 Install Anaconda
+
+- Note: To use PaddlePaddle you need to install python environment first, here we choose python integrated environment Anaconda toolkit
+
+ - Anaconda is a common python package manager
+ - After installing Anaconda, you can install the python environment, as well as numpy and other required toolkit environment.
+
+- Anaconda download.
+
+ - Address:
+
+
+ - 勾选conda加入环境变量，忽略警告：
+
+ 
+
+#### 1.1.2 打开终端并创建conda环境
+
+- 打开Anaconda Prompt终端：左下角Windows Start Menu -> Anaconda3 -> Anaconda Prompt启动控制台
+
+ 
+
+- 创建新的conda环境
+
+ ```bash linenums="1"
+ # 在命令行输入以下命令，创建名为paddle_env的环境
+ # 此处为加速下载，使用清华源
+ conda create --name paddle_env python=3.8 --channel https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/ # 这是一行命令
+ ```
+
+ 该命令会创建1个名为paddle_env、python版本为3.8的可执行环境，根据网络状态，需要花费一段时间
+
+ 之后命令行中会输出提示信息，输入y并回车继续安装
+
+ 
+
+- 激活刚创建的conda环境，在命令行中输入以下命令：
+
+ ```bash linenums="1"
+ # 激活paddle_env环境
+ conda activate paddle_env
+ # 查看当前python的位置
+ where python
+ ```
+
+ 
+
+以上anaconda环境和python环境安装完毕
+
+### 1.2 Mac
+
+#### 1.2.1 安装Anaconda
+
+- 说明：使用paddlepaddle需要先安装python环境，这里我们选择python集成环境Anaconda工具包
+ - Anaconda是1个常用的python包管理程序
+ - 安装完Anaconda后，可以安装python环境，以及numpy等所需的工具包环境
+- Anaconda下载：
+ - 地址：of the
Whole System\(M\) | Model Size
of the Text
Detector\(M\) | Model Size
of the Direction
Classifier\(M\) | Model Size
of the Text
Recognizer \(M\) | F\-score | +|:-:|:-:|:-:|:-:|:-:|:-:| +| PP-OCRv2 | 11\.6 | 3\.0 | 0\.9 | 8\.6 | 0\.5224 | +| PP-OCR mobile | 8\.1 | 2\.6 | 0\.9 | 4\.6 | 0\.503 | +| PP-OCR server | 155\.1 | 47\.2 | 0\.9 | 107 | 0\.570 | + +Compares the time-consuming on CPU and T4 GPU (ms): + +| Model Name | CPU | T4 GPU | +|:-:|:-:|:-:| +| PP-OCRv2 | 330 | 111 | +| PP-OCR mobile | 356 | 116| +| PP-OCR server | 1056 | 200 | diff --git a/docs/ppocr/infer_deploy/benchmark.md b/docs/ppocr/infer_deploy/benchmark.md new file mode 100644 index 0000000000..5de86e3647 --- /dev/null +++ b/docs/ppocr/infer_deploy/benchmark.md @@ -0,0 +1,39 @@ +--- +typora-copy-images-to: images +comments: true +--- + +# Benchmark + +本文给出了中英文OCR系列模型精度指标和在各平台预测耗时的benchmark。 + +## 测试数据 + +针对OCR实际应用场景，包括合同，车牌，铭牌，火车票，化验单，表格，证书，街景文字，名片，数码显示屏等，收集的300张图像，每张图平均有17个文本框，下图给出了一些图像示例。 + + + +## 评估指标 + +说明： + +- 检测输入图像的长边尺寸是960。 +- 评估耗时阶段为图像预测耗时，不包括图像的预处理和后处理。 +- `Intel至强6148`为服务器端CPU型号，测试中使用Intel MKL-DNN 加速。 +- `骁龙855`为移动端处理平台型号。 + +预测模型大小和整体识别精度对比 + +| 模型名称 | 整体模型
大小\(M\) | 检测模型
大小\(M\) | 方向分类器
模型大小\(M\) | 识别模型
大小\(M\) | 整体识别
F\-score | +| :-----------: | :-------------------: | :-------------------: | :-------------------------: | :-------------------: | :------------------: | +| PP-OCRv2 | 11\.6 | 3\.0 | 0\.9 | 8\.6 | 0\.5224 | +| PP-OCR mobile | 8\.1 | 2\.6 | 0\.9 | 4\.6 | 0\.503 | +| PP-OCR server | 155\.1 | 47\.2 | 0\.9 | 107 | 0\.570 | + +预测模型在CPU和GPU上的速度对比，单位ms + +| 模型名称 | CPU | T4 GPU | +| :-----------: | :---: | :----: | +| PP-OCRv2 | 330 | 111 | +| PP-OCR mobile | 356 | 11 6 | +| PP-OCR server | 1056 | 200 | diff --git a/docs/ppocr/infer_deploy/cpp_infer.en.md b/docs/ppocr/infer_deploy/cpp_infer.en.md new file mode 100644 index 0000000000..f18c8c8aae --- /dev/null +++ b/docs/ppocr/infer_deploy/cpp_infer.en.md @@ -0,0 +1,433 @@ +--- +comments: true +--- + +# Server-side C++ Inference + +This chapter introduces the C++ deployment steps of the PaddleOCR model. C++ is better than Python in terms of performance. Therefore, in CPU and GPU deployment scenarios, C++ deployment is mostly used. +This section will introduce how to configure the C++ environment and deploy PaddleOCR in Linux (CPU\GPU) environment. For Windows deployment please refer to [Windows](./windows_vs2019_build.en.md) compilation guidelines. + +## 1. Prepare the Environment + +### 1.1 Environment + +- Linux, docker is recommended. +- Windows. + +### 1.2 Compile OpenCV + +- First of all, you need to download the source code compiled package in the Linux environment from the OpenCV official website. Taking OpenCV 3.4.7 as an example, the download command is as follows. + +```bash linenums="1" +cd deploy/cpp_infer +wget https://paddleocr.bj.bcebos.com/libs/opencv/opencv-3.4.7.tar.gz +tar -xf opencv-3.4.7.tar.gz +``` + +Finally, you will see the folder of `opencv-3.4.7/` in the current directory. + +- Compile OpenCV, the OpenCV source path (`root_path`) and installation path (`install_path`) should be set by yourself. Enter the OpenCV source code path and compile it in the following way. + +```bash linenums="1" +root_path=your_opencv_root_path +install_path=${root_path}/opencv3 + +rm -rf build +mkdir build +cd build + +cmake .. \ + -DCMAKE_INSTALL_PREFIX=${install_path} \ + -DCMAKE_BUILD_TYPE=Release \ + -DBUILD_SHARED_LIBS=OFF \ + -DWITH_IPP=OFF \ + -DBUILD_IPP_IW=OFF \ + -DWITH_LAPACK=OFF \ + -DWITH_EIGEN=OFF \ + -DCMAKE_INSTALL_LIBDIR=lib64 \ + -DWITH_ZLIB=ON \ + -DBUILD_ZLIB=ON \ + -DWITH_JPEG=ON \ + -DBUILD_JPEG=ON \ + -DWITH_PNG=ON \ + -DBUILD_PNG=ON \ + -DWITH_TIFF=ON \ + -DBUILD_TIFF=ON + +make -j +make install +``` + +In the above commands, `root_path` is the downloaded OpenCV source code path, and `install_path` is the installation path of OpenCV. After `make install` is completed, the OpenCV header file and library file will be generated in this folder for later OCR source code compilation. + +The final file structure under the OpenCV installation path is as follows. + +```text linenums="1" +opencv3/ +|-- bin +|-- include +|-- lib +|-- lib64 +|-- share +``` + +### 1.3 Compile or Download or the Paddle Inference Library + +- There are 2 ways to obtain the Paddle inference library, described in detail below. + +#### 1.3.1 Direct download and installation + +[Paddle inference library official website](https://www.paddlepaddle.org.cn/inference/master/guides/install/download_lib.html#linux). You can review and select the appropriate version of the inference library on the official website. + +- After downloading, use the following command to extract files. + +```bash linenums="1" +tar -xf paddle_inference.tgz +``` + +Finally you will see the folder of `paddle_inference/` in the current path. + +#### 1.3.2 Compile the inference source code + +- If you want to get the latest Paddle inference library features, you can download the latest code from Paddle GitHub repository and compile the inference library from the source code. It is recommended to download the inference library with paddle version greater than or equal to 2.0.1. + +- You can refer to [Paddle inference library](https://www.paddlepaddle.org.cn/documentation/docs/en/advanced_guide/inference_deployment/inference/build_and_install_lib_en.html) to get the Paddle source code from GitHub, and then compile To generate the latest inference library. The method of using git to access the code is as follows. + +```bash linenums="1" +git clone https://github.com/PaddlePaddle/Paddle.git +git checkout develop +``` + +- Enter the Paddle directory and run the following commands to compile the paddle inference library. + +```bash linenums="1" +rm -rf build +mkdir build +cd build + +cmake .. \ + -DWITH_CONTRIB=OFF \ + -DWITH_MKL=ON \ + -DWITH_MKLDNN=ON \ + -DWITH_TESTING=OFF \ + -DCMAKE_BUILD_TYPE=Release \ + -DWITH_INFERENCE_API_TEST=OFF \ + -DON_INFER=ON \ + -DWITH_PYTHON=ON +make -j +make inference_lib_dist +``` + +For more compilation parameter options, please refer to the [document](https://www.paddlepaddle.org.cn/documentation/docs/zh/2.0/guides/05_inference_deployment/inference/build_and_install_lib_cn.html#congyuanmabianyi). + +- After the compilation process, you can see the following files in the folder of `build/paddle_inference_install_dir/`. + +```text linenums="1" +build/paddle_inference_install_dir/ +|-- CMakeCache.txt +|-- paddle +|-- third_party +|-- version.txt +``` + +`paddle` is the Paddle library required for C++ prediction later, and `version.txt` contains the version information of the current inference library. + +## 2. Compile and Run the Demo + +### 2.1 Export the inference model + +- You can refer to [Model inference](./python_infer.en.md) and export the inference model. After the model is exported, assuming it is placed in the `inference` directory, the directory structure is as follows. + +```text linenums="1" +inference/ +|-- det_db +| |--inference.pdiparams +| |--inference.pdmodel +|-- rec_rcnn +| |--inference.pdiparams +| |--inference.pdmodel +|-- cls +| |--inference.pdiparams +| |--inference.pdmodel +|-- table +| |--inference.pdiparams +| |--inference.pdmodel +|-- layout +| |--inference.pdiparams +| |--inference.pdmodel +``` + +### 2.2 Compile PaddleOCR C++ inference demo + +- The compilation commands are as follows. The addresses of Paddle C++ inference library, opencv and other Dependencies need to be replaced with the actual addresses on your own machines. + +```bash linenums="1" +sh tools/build.sh +``` + +Specifically, you should modify the paths in `tools/build.sh`. The related content is as follows. + +```bash linenums="1" +OPENCV_DIR=your_opencv_dir +LIB_DIR=your_paddle_inference_dir +CUDA_LIB_DIR=your_cuda_lib_dir +CUDNN_LIB_DIR=your_cudnn_lib_dir +``` + +`OPENCV_DIR` is the OpenCV installation path; `LIB_DIR` is the download (`paddle_inference` folder) +or the generated Paddle inference library path (`build/paddle_inference_install_dir` folder); +`CUDA_LIB_DIR` is the CUDA library file path, in docker; it is `/usr/local/cuda/lib64`; `CUDNN_LIB_DIR` is the cuDNN library file path, in docker it is `/usr/lib/x86_64-linux-gnu/`. + +- After the compilation is completed, an executable file named `ppocr` will be generated in the `build` folder. + +### 2.3 Run the demo + +Execute the built executable file: + +```bash linenums="1" +./build/ppocr [--param1] [--param2] [...] +``` + +**Note**:ppocr uses the `PP-OCRv3` model by default, and the input shape used by the recognition model is `3, 48, 320`, if you want to use the old version model, you should add the parameter `--rec_img_h=32`. + +Specifically, + +#### 1. det+cls+rec + +```bash linenums="1" +./build/ppocr --det_model_dir=inference/det_db \ + --rec_model_dir=inference/rec_rcnn \ + --cls_model_dir=inference/cls \ + --image_dir=../../doc/imgs/12.jpg \ + --use_angle_cls=true \ + --det=true \ + --rec=true \ + --cls=true \ +``` + +#### 2. det+rec + +```bash linenums="1" +./build/ppocr --det_model_dir=inference/det_db \ + --rec_model_dir=inference/rec_rcnn \ + --image_dir=../../doc/imgs/12.jpg \ + --use_angle_cls=false \ + --det=true \ + --rec=true \ + --cls=false \ +``` + +#### 3. det + +```bash linenums="1" +./build/ppocr --det_model_dir=inference/det_db \ + --image_dir=../../doc/imgs/12.jpg \ + --det=true \ + --rec=false +``` + +#### 4. cls+rec + +```bash linenums="1" +./build/ppocr --rec_model_dir=inference/rec_rcnn \ + --cls_model_dir=inference/cls \ + --image_dir=../../doc/imgs_words/ch/word_1.jpg \ + --use_angle_cls=true \ + --det=false \ + --rec=true \ + --cls=true \ +``` + +#### 5. rec + +```bash linenums="1" +./build/ppocr --rec_model_dir=inference/rec_rcnn \ + --image_dir=../../doc/imgs_words/ch/word_1.jpg \ + --use_angle_cls=false \ + --det=false \ + --rec=true \ + --cls=false \ +``` + +#### 6. cls + +```bash linenums="1" +./build/ppocr --cls_model_dir=inference/cls \ + --cls_model_dir=inference/cls \ + --image_dir=../../doc/imgs_words/ch/word_1.jpg \ + --use_angle_cls=true \ + --det=false \ + --rec=false \ + --cls=true \ +``` + +#### 7. layout+table + +```bash linenums="1" +./build/ppocr --det_model_dir=inference/det_db \ + --rec_model_dir=inference/rec_rcnn \ + --table_model_dir=inference/table \ + --image_dir=../../ppstructure/docs/table/table.jpg \ + --layout_model_dir=inference/layout \ + --type=structure \ + --table=true \ + --layout=true +``` + +#### 8. layout + +```bash linenums="1" +./build/ppocr --layout_model_dir=inference/layout \ + --image_dir=../../ppstructure/docs/table/1.png \ + --type=structure \ + --table=false \ + --layout=true \ + --det=false \ + --rec=false +``` + +#### 9. table + +```bash linenums="1" +./build/ppocr --det_model_dir=inference/det_db \ + --rec_model_dir=inference/rec_rcnn \ + --table_model_dir=inference/table \ + --image_dir=../../ppstructure/docs/table/table.jpg \ + --type=structure \ + --table=true +``` + +More parameters are as follows, + +Common parameters + +|parameter|data type|default|meaning| +| --- | --- | --- | --- | +|use_gpu|bool|false|Whether to use GPU| +|gpu_id|int|0|GPU id when use_gpu is true| +|gpu_mem|int|4000|GPU memory requested| +|cpu_math_library_num_threads|int|10|Number of threads when using CPU inference. When machine cores is enough, the large the value, the faster the inference speed| +|enable_mkldnn|bool|true|Whether to use mkdlnn library| +|output|str|./output|Path where visualization results are saved| + +forward + +|parameter|data type|default|meaning| +| :---: | :---: | :---: | :---: | +|det|bool|true|Whether to perform text detection in the forward direction| +|rec|bool|true|Whether to perform text recognition in the forward direction| +|cls|bool|false|Whether to perform text direction classification in the forward direction| + +Detection related parameters + +|parameter|data type|default|meaning| +| --- | --- | --- | --- | +|det_model_dir|string|-|Address of detection inference model| +|max_side_len|int|960|Limit the maximum image height and width to 960| +|det_db_thresh|float|0.3|Used to filter the binarized image of DB prediction, setting 0.-0.3 has no obvious effect on the result| +|det_db_box_thresh|float|0.5|DB post-processing filter box threshold, if there is a missing box detected, it can be reduced as appropriate| +|det_db_unclip_ratio|float|1.6|Indicates the compactness of the text box, the smaller the value, the closer the text box to the text| +|det_db_score_mode|string|slow| slow: use polygon box to calculate bbox score, fast: use rectangle box to calculate. Use rectangular box to calculate faster, and polygonal box more accurate for curved text area.| +|visualize|bool|true|Whether to visualize the results，when it is set as true, the prediction results will be saved in the folder specified by the `output` field on an image with the same name as the input image.| + +Classifier related parameters + +|parameter|data type|default|meaning| +| --- | --- | --- | --- | +|use_angle_cls|bool|false|Whether to use the direction classifier| +|cls_model_dir|string|-|Address of direction classifier inference model| +|cls_thresh|float|0.9|Score threshold of the direction classifier| +|cls_batch_num|int|1|batch size of classifier| + +Recognition related parameters + +|parameter|data type|default|meaning| +| --- | --- | --- | --- | +|rec_model_dir|string|-|Address of recognition inference model| +|rec_char_dict_path|string|../../ppocr/utils/ppocr_keys_v1.txt|dictionary file| +|rec_batch_num|int|6|batch size of recognition| +|rec_img_h|int|48|image height of recognition| +|rec_img_w|int|320|image width of recognition| + +Layout related parameters + +|parameter|data type|default|meaning| +| :---: | :---: | :---: | :---: | +|layout_model_dir|string|-| Address of layout inference model| +|layout_dict_path|string|../../ppocr/utils/dict/layout_dict/layout_publaynet_dict.txt|dictionary file| +|layout_score_threshold|float|0.5|Threshold of score.| +|layout_nms_threshold|float|0.5|Threshold of nms.| + +Table recognition related parameters + +|parameter|data type|default|meaning| +| :---: | :---: | :---: | :---: | +|table_model_dir|string|-|Address of table recognition inference model| +|table_char_dict_path|string|../../ppocr/utils/dict/table_structure_dict.txt|dictionary file| +|table_max_len|int|488|The size of the long side of the input image of the table recognition model, the final input image size of the network is（table_max_len，table_max_len）| +|merge_no_span_structure|bool|true|Whether to merge
| Methods | Ext | R | P | F | FPS |
| TextSnake [18] | Syn | 85.3 | 67.9 | 75.6 | |
| CSE [17] | MiLT | 76.1 | 78.7 | 77.4 | 0.38 |
| LOMO[40] | Syn | 76.5 | 85.7 | 80.8 | 4.4 |
| ATRR[35] | Sy- | 80.2 | 80.1 | 80.1 | - |
| SegLink++ [28] | Syn | 79.8 | 82.8 | 81.3 | - |
| TextField [37] | Syn | 79.8 | 83.0 | 81.4 | 6.0 |
| MSR[38] | Syn | 79.0 | 84.1 | 81.5 | 4.3 |
| PSENet-1s [33] | MLT | 79.7 | 84.8 | 82.2 | 3.9 |
| DB [12] | Syn | 80.2 | 86.9 | 83.4 | 22.0 |
| CRAFT [2] | Syn | 81.1 | 86.0 | 83.5 | - |
| TextDragon [5] | MLT+ | 82.8 | 84.5 | 83.6 | |
| PAN [34] | Syn | 81.2 | 86.4 | 83.7 | 39.8 |
| ContourNet [36] | 84.1 | 83.7 | 83.9 | 4.5 | |
| DRRG [41] | MLT | 83.02 | 85.93 | 84.45 | - |
| TextPerception[23] | Syn | 81.9 | 87.5 | 84.6 | |
| Ours | Syn | 80.57 | 87.66 | 83.97 | 12.08 |
| Ours | 81.45 | 87.81 | 84.51 | 12.15 | |
| Ours | MLT | 83.60 | 86.45 | 85.00 | 12.21 |