How to extract and translate hardcode subtitles from videos? 2023 video hard subtitles translator

Hardcode subtitles usually refer to text that has been embedded into a video and cannot be removed by editing or playback software. If you want to extract and translate video subtitles, and finally synthesize the video, you will need to use various technologies, such as OCR, video restoration, translation, sentence simplification, video layout calculation, and audio and video processing.

With the popularity of AI artificial intelligence in 2023, is there a product that can integrate these technologies to directly extract and translate buildin subtitles from the video? Today, I will analyze the difficulties here and recommend a product that truly solves the problem of translating video hardcode subtitles.

​

The whole article is divided into several parts

1. Definition of hard subtitles in videos
2. Technical principles and difficulties of translating hard subtitles in videos
3. Translation effect of hard subtitles in videos
4. Tutorial on translating hardcode subtitles in videos
5. How to modify translated hard subtitles?

1. Definition of hard subtitles in videos

Hard subtitles are also known as embedded subtitles, internal subtitles, built-in subtitles, etc. Generally, the text of the subtitles is already embedded in the movie. This kind of subtitles are no longer text, but images. They do not have a separate subtitle file, and users cannot use editing tools or playback tools to remove this option. These subtitles cannot be changed or completely deleted.

2. Technical difficulties of translating hardcode subtitles in videos

The technical process of extracting and translating hard subtitles in videos is as follows:

1. Video analysis and subtitle extraction: Use video parsing tools or open source libraries to extract the subtitle files from the video. Hard subtitles are usually saved as pixel information in the video file and cannot be turned off or hidden. The video file is sent to an OCR recognition engine or API to convert the pixel information into text information.
2. Recognize text with OCR technology: OCR technology can recognize the text information in the subtitles and then convert it into text form. The OCR engine will establish a subtitle dataset to identify possible special characters that may appear in the subtitles, such as punctuation marks, special symbols, font colors, and capital letters, etc. One of the difficulties here is style extraction.
3. Translate subtitles: Use language models like ChatGPT for automated translation. Machine translation API can be used to pass the extracted subtitle text to the translation engine and return the translation results to the application. One of the difficulties here is translation accuracy.
4. Subtitles replacement and integration: Embed the translated subtitles into the video timeline and generate a new subtitle file (in formats such as .srt, .ass, etc.). One of the difficulties here is to accomplish video restoration and remove the original hard subtitles.
5. Matching audio files with subtitles: Use audio processing tools to synchronize the subtitles in the video with the audio file, so that they remain consistent and eliminate any time differences. One of the difficulties here is spatial and layout calculation and multi-font composition.
6. Generate the translated video file: Re-combine the synchronized audio file and video file, and output the translated video file.

The entire technical process can be automated, where OCR and ChatGPT API are utilized for natural language processing.

3. Translation effect of hardcode subtitles in videos

The following is the actual effect of the GhostCut automatic translator for hardcode subtitles .

<iframe width="100%" height="450" src="//github.com/JollyToday/GhostCut-auto_video_translation/raw/main/Video%20Translation%20and%20Dubbing-eng.mp4" scrolling="no" border="0" frameborder="no" framespacing="0" allowfullscreen="true"> 暂时不支持视频插入。 </iframe>

​

GhostCut translates the "subtitles" in the original video through OCR recognition and generates translated text. It uses AI video inpainting to remove the original subtitles in the video, and then reposts the translated subtitles to the original subtitle position, while trying to retain the size, color, and layout of the original subtitles. Compared with using multiple editing and translation software, GhostCut's video translation features are fully automated and retain the original video subtitle format with one click.

4. Tutorial on translating hard subtitles in videos

The following is how to translate video subtitles on a mobile platform:

1. Open the wechat application and search for the "GhostCut" mini program.
2. Click "Translate Video Text", upload a video from your phone gallery, select the original language type in the video, Chinese or English, and then select the target translation language type, English\Chinese or other languages.
3. Click "Submit" and wait a few minutes to complete the video translation.
4. View and download the finished product in the work list.

This AI video translation software called GhostCut, which also has a web version that can be searched on Google.

5. How to modify translated hard subtitles?

Video subtitle translation now supports adjustments to the translated text. If you find the translation inaccurate or text style is mistranslated, you can adjust it.

1. In the mini-program, click the "Adjust Video Translation" option under the work list.
2. Click to view the new and old works, and batch edit subtitles if necessary.

Enter the subtitle editing area to adjust the text. It is worth noting that the style will not change and that you should pay attention to the length of the text. If it is too long, it will be automatically wrapped.