x

Search in
Sort by:

Question Status:

Search help

  • Simple searches use one or more words. Separate the words with spaces (cat dog) to search cat,dog or both. Separate the words with plus signs (cat +dog) to search for items that may contain cat but must contain dog.
  • You can further refine your search on the search results page, where you can search by keywords, author, topic. These can be combined with each other. Examples
    • cat dog --matches anything with cat,dog or both
    • cat +dog --searches for cat +dog where dog is a mandatory term
    • cat -dog -- searches for cat excluding any result containing dog
    • [cats] —will restrict your search to results with topic named "cats"
    • [cats] [dogs] —will restrict your search to results with both topics, "cats", and "dogs"

Asynchronous Media texture update

In MediaTextureResource.cpp, the rendering thread calls UpdateDeferredResource, from the FMediaTextureResource class.

The function looks like this:

uint8* TextureBuffer = (uint8*)RHILockTexture2D(Texture2D, 0, RLM_WriteOnly, Stride, false); FMemory::Memcpy(TextureBuffer, CurrentFrame->GetData(), CurrentFrame->Num()); RHIUnlockTexture2D(Texture2D, 0, false);

It's slow. I'm benchmarking 3ms for a HD texture size (1920x1080). My goal is to playback a 4k movie, at 60 fps smoothly. No hickup, no skip frame.

My first optimization was to replace the Lock-Memcpy-Unlock with RHIUpdateTexture2D. It's cutting the time by approximately a factor of 3, which is not bad. What this function ultimately calls is Direct3DDeviceIMContext->UpdateSubresource.

Now, there's this presentation form NVidia:

http://developer.download.nvidia.com/presentations/2008/GDC/GDC08-D3DDay-Performance.pdf

Slide 14, it says: Avoid UpdateSubresource() for textures! Especially bad with larger textures!

Instead, NVidia suggets to use ring buffers of staging texture, update and upload the data in the main thread, and calls map-CopyResource in the rendering thread.

I guess the right way to do it is uploading the frames as fast as they arrive in the main thread, and bind the resource in the game thread to data already on the device.

Is that something you are looking at?

Is there something in the engine that I can use to quickly try it?

Or maybe there's an alternate solution that we have'nt figure out yet?

Thanks, Bryan

Product Version: Not Selected
Tags:
more ▼

asked Dec 03 '14 at 02:05 PM in Rendering

avatar image

3dids
126 7 10 15

avatar image gmpreussner STAFF Mar 13 '15 at 06:50 PM

Hey Bryan, thanks for your feedback. This is good information. The Media Framework is still in a somewhat experimental state, and we are aware that performance optimizations are needed. It's something we are looking into now. Another aspect that slows things down is the two copies that are currently required - one from the decoder thread, and one into the render thread. We are aiming to eliminate all copies, but it requires a bit of redesign of our material system.

I don't have any alternative solutions for you right now, but I will update this thread as improvements are being checked in. If you have any other ideas, we'd love to hear about them, and we do appreciate GitHub pull requests as well :)

avatar image gmpreussner STAFF Mar 24 '15 at 07:01 PM

Bryan, just letting you know that we started working on this.

avatar image tron1977 Apr 15 '15 at 06:55 PM

I've made an Oculus presentation of a convention center and I have video playing on a bunch of monitors in the convention center.

When there is no video on the monitors the "game" runs at 58 FPS/17 ms in the Oculus it looks great and plays smoothly.

When the video files are attached it runs at 43 FPS/23 ms and it very steppy when I turn my head. It doesn't matter the size of the video texture map (wmv) I made a test file of 10 frames of black that are 9kb and it plays at the same rate as when they are 7 MB each.

Is there any solution to this issue? Is there something different I could be doing to speed up the gameplay?

It's built in 4.6.1

avatar image gmpreussner STAFF Apr 15 '15 at 07:15 PM

I think the relevant factor right now is the resolution of the video, not the file size. You could try a smaller resolution and see if that helps.

This won't get any faster until we changed the way the dynamic textures are updated. Unfortunately, everyone is still slammed with a variety of tasks, and progress is slow. I don't know if this will make it into 4.8, but we will try.

avatar image tron1977 Apr 15 '15 at 08:14 PM

Thank you. I have since come to that conclusion from reading other posts and doing some test. I'm currently doing some tests to find what the maximum resolution I can use to get optimal performance.

avatar image tron1977 Apr 15 '15 at 06:55 PM

I've made an Oculus presentation of a convention center and I have video playing on a bunch of monitors in the convention center.

When there is no video on the monitors the "game" runs at 58 FPS/17 ms in the Oculus it looks great and plays smoothly.

When the video files are attached it runs at 43 FPS/23 ms and it very steppy when I turn my head. It doesn't matter the size of the video texture map (wmv) I made a test file of 10 frames of black that are 9kb and it plays at the same rate as when they are 7 MB each.

Is there any solution to this issue? Is there something different I could be doing to speed up the gameplay?

It's built in 4.6.1

avatar image gmpreussner STAFF Sep 08 '15 at 03:15 PM

Not yet - we're working on it.

avatar image f-win Sep 14 '15 at 02:45 PM

I am also trying to improve the performance of my 4k textures. The lock/unlock calls are are particularly slow even on my fast dev machine. Could you give us some pointers what we can try to work around this until it is in the engine? Thanks a bunch!

avatar image gmpreussner STAFF Sep 16 '15 at 07:20 PM

IMediaVideoTexture has a new API for a texture copying fast-path to eliminate one extra copy: BindTexture, UnbindTexture. These are not implemented yet for any player plug-in other than PS4Media. You could maybe take a look at what we do there.

Another thing we are planning to do is to implement pixel format conversion on the GPU, which, when implemented for WmfMedia, will allow for playback of H.264 on Windows and also improve performance and quality. The basics for that are in place - again, check the PS4Media implementation for pointers.

avatar image gmpreussner STAFF Feb 10 '16 at 08:18 PM

FYI I have started working on refactoring the API, so that decoders can write directly into texture memory. I'm trying to get this in before GDC. We are also working on another mechanism that will eliminate the last remaining copy from the Render Thread to the GPU on platforms with unified memory. This might not be finished until after GDC.

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

1 answer: sort voted first

Starting with 4.13 there will be several ways to update the media texture render target:

  1. Render directly to FRHITexture (only available if player is able to write on render thread; player provides buffer)

  2. RHIUpdateTexture2D (only available if player is able to write on render thread; decoder provides buffer)

  3. Triple buffer with RHILockTexture2D/RHIUnlockTexture2D (available on any thread)

  4. Update the texture sink resource directly (available on any thread; fastest method)

more ▼

answered Jun 08 '16 at 03:58 PM

avatar image Eric.Renaud-Houd Aug 04 '16 at 08:09 PM

Hi @gmpreussner, could you quickly point us to example code for the first two techniques? EDIT: I see the VLC plugin is using the IMediaTextureSink, which seems to be the standard approach.

avatar image gmpreussner STAFF Aug 08 '16 at 09:06 AM

All methods use the IMediaTextureSink interface. Note that it exposes (currently four) sets of methods. In the 4.13 code base they're pretty well documented.

Search the code base for the respective function calls, i.e. GetTextureSinkTexture or AcquireTextureSinkBuffer to see where and how they're used.

avatar image ccandcc Sep 21 '16 at 01:46 PM

Hi @gmpreussner, could you give some example or document for the last one method?

avatar image gmpreussner STAFF Sep 21 '16 at 08:11 PM

Currently, the only player plug-in that uses this method is AvfMedia on macOS and iOS: https://github.com/EpicGames/UnrealEngine/blob/master/Engine/Plugins/Media/AvfMedia/Source/AvfMedia/Private/Player/AvfMediaTracks.cpp#L1203

You're basically giving the media texture a new FRHITexture2D instance that contains your decoded frame buffer. How to convert/wrap an existing texture target on the GPU into a FRHITexture2D compatible resource depends on the platform. In AvfMedia it seems to use the FAvfTexture2DResourceWrapper helper class.

(comments are locked)
10|2000 characters needed characters left
Viewable by all users
Your answer
toggle preview:

Up to 5 attachments (including images) can be used with a maximum of 5.2 MB each and 5.2 MB total.

Follow this question

Once you sign in you will be able to subscribe for any updates here

Answers to this question