Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(359): Support fractional fps. #364

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

fix(359): Support fractional fps. #364

wants to merge 2 commits into from

Conversation

alxbl
Copy link
Collaborator

@alxbl alxbl commented Oct 24, 2021 •
edited
Loading

This pull request fixes #359

I've done two things:

  • Default FPS is now 25. (PS: It looks like a recent refactor makes it impossible to configure the FPS for the MP4 converter)
  • Use fractional division and keep track of the drift. Whenever we drift by more than one frame, an extra frame is added.

I'm not sure if the drift was a real issue though since we only updated the previous timestamp whenever at least one frame was rendered. In other words, if the timestamp between 2 PDUs is smaller than the frame rate delta, we would just let the delta increase until we've gone past the next integer frame, then render the screen at the current PDU.

The only thing that might happen is that at low frame rates, some information is lost because the screen changes faster than our frame rate can observe. RDP doesn't really have a concept of framerate, updates are sent as they happen, so it's difficult to map it to a constant frame rate.

@obilodeau
Copy link
Collaborator

The code looks fine. I'll run it on a large conversion job and check drift values to confirm we are fixing the right behavior I've seen.

@obilodeau
Copy link
Collaborator

Based on early testing, I think this needs to be revisited. I already spoke to the submitter about this privately.

Highlight: Video duration is 10 seconds more for a 34 seconds video (if we trust the master's number). This is 33%+ longer. Does it make sense?

Below is some evidence collected on a small capture that I can share privately.

Little performance impact, most likely attributable to duration.

Original:

(venv) olivier@barachois:~/gosecure/research/projets/2021-10_pyrdp-video-precision$ time pyrdp-convert.py ../2021-08_pyrdp-1.1.0/rdp_replay_20210826_12-15-33_512_Stephen215343.pyrdp -f mp4 -o ./
[*] Converting '../2021-08_pyrdp-1.1.0/rdp_replay_20210826_12-15-33_512_Stephen215343.pyrdp' to MP4
100% (1465 of 1465) |##############################################| Elapsed Time: 0:00:42 Time:  0:00:42

[+] Succesfully wrote '/home/olivier/Documents/gosecure/research/projets/2021-10_pyrdp-video-precision/rdp_replay_20210826_12-15-33_512_Stephen215343.mp4'

real	0m46.372s
user	1m13.356s
sys	0m1.752s

with PR:

(venv) olivier@barachois:~/gosecure/research/projets/2021-10_pyrdp-video-precision$ time pyrdp-convert.py ../2021-08_pyrdp-1.1.0/rdp_replay_20210826_12-15-33_512_Stephen215343.pyrdp -f mp4 -o ./
[*] Converting '../2021-08_pyrdp-1.1.0/rdp_replay_20210826_12-15-33_512_Stephen215343.pyrdp' to MP4
100% (1465 of 1465) |##############################################| Elapsed Time: 0:00:46 Time:  0:00:46

[+] Succesfully wrote '/home/olivier/Documents/gosecure/research/projets/2021-10_pyrdp-video-precision/rdp_replay_20210826_12-15-33_512_Stephen215343.mp4'

real	0m49.827s
user	1m18.960s
sys	0m1.620s

larger file size (it's longer):

-rw-r--r-- 1 olivier olivier 5.5M Oct 27 12:00 small-fix-drift.mp4
-rw-r--r-- 1 olivier olivier 4.0M Oct 27 11:59 small-original.mp4

ffprobe on result with PR:

  Duration: 00:00:44.64, start: 0.080000, bitrate: 1021 kb/s
  Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1280x1324, 1019 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)

ffprobe on result with master:

  Duration: 00:00:34.30, start: 0.066667, bitrate: 958 kb/s
  Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1280x1324, 956 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default)

Packet(frame?) count, original:

$ ffprobe -v error -select_streams v:0 -count_packets -show_entries stream=nb_read_packets -of csv=p=0 small-original.mp4 
1029

with PR:

$ ffprobe -v error -select_streams v:0 -count_packets -show_entries stream=nb_read_packets -of csv=p=0 small-fix-drift.mp4 
1116

@alxbl
Copy link
Collaborator Author

alxbl commented Oct 27, 2021

As discussed offline, I think we need to fully revisit how to calculate FPS/insert still frames.

As a starting point, we can know the expected wallclock duration of the recording based on the first PDU time and last PDU time. From there, it should be easy to calculate the total number of frames and make sure that they are dispersed evenly through the recording.

Then PDUs should be snapped to the nearest(?) frame. I'll probably abandon this or push significant changes when I have a chance.

obilodeau
obilodeau requested changes Jan 18, 2022
View reviewed changes
Copy link
Collaborator

@obilodeau obilodeau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Marking this clearly has "needing changes" to avoid accidental merge.

@obilodeau obilodeau marked this pull request as draft April 1, 2022 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@obilodeau obilodeau obilodeau requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Increase/validate precision of converted videos
2 participants
@alxbl @obilodeau