Media Player: Get current position/timecode, precisely?

[hiili]

We are working on an interactive VR film project that involves playing back filmed material in different screen formats (360’, flat 16:9, etc.). An approach that we are considering for implementing smooth and gapless transitions between screen formats would require the following:

When a video is playing in a Media Player, is it possible to determine the exact timecode/index of the video frame that will go on screen after the current tick?

The main requirement is that we need to do modifications to the scene (switch the playback surfaces) precisely on a given frame, robustly. A C++ solution is ok (and seems inevitable).

Our current solution involves playing the film through two separate media players when in the proximity of a screen format switch, which in turn results in a video decoding performance bottleneck that we are trying to get rid of.

Platform: Windows
Video file(s): h.264 or h.265, 4K, stereo, would love to hit 60fps
Planned minimum system specs: VR Ready PC

Some possible approaches:

• UMediaPlayer::GetTime() (BP: Media Player > Get Time): The value returned by this function seems to lag behind the actual playback position by several ticks, so it does not seem usable.

• Embed frame markers in the video file, then read them from the media texture in UE: Based on some AnswerHub answers, it appears to be possible to access the actual content of the video frames via the video texture object. Consequently, reading markers from the video during playback should be possible. However, I have the suspicion that this approach might not give reliable access to the exact frame that is going to go on-screen now (for example, if the decoder runs in its own thread, if it writes directly to GPU RAM and the game thread gets a copy only later, and so on). Am I correct with these suspicions, or does this sound like a viable approach?

Predictively compensating some lag is not an option, because the prediction could be off a tick or two quite easily, which would give undesirable results in our case, especially in VR.

I would be grateful for any thoughts, or for any ideas for alternative approaches?

[gmpreussner_1 (Gerke Max Preussner)]

is it possible to determine the exact timecode/index of the video frame that will go on screen after the current tick

This is currently not possible, sorry. There a couple of spots that introduce inaccuracies, and it also depends on which player plug-in is being used. The biggest issue is that time code information is completely ignored right now. The video sink implemented in FMediaTextureResource is greedy and will consume frames as soon as they’re available. This is often wrong, because decoders may produce several frames in advance. Another problem is that the frames are going through a triple buffer in some players, i.e. in WmfMedia on Windows. When the decoder produces several frames at once, it may actually drop frames that should really be displayed.

we need to do modifications to the scene (switch the playback surfaces) precisely on a given frame

I’m currently working on time code support and proper audio/video frame-sync, which should allow you to do that. A first version of this will probably be in 4.16 as part of my big Media Framework 3.0 update. We may integrate the latest code into the Master branch two or three weeks after GDC.

[hiili]

Ok, and thank you again for the quick and detailed answer! It is really nice to have support like this (Epic’s user support was actually one of the reasons why we changed over from Unity to UE)

So we will stick with our current solution for now and keep an eye on the MF 3.0 update!