x

Search in
Sort by:

Question Status:

Search help

  • Simple searches use one or more words. Separate the words with spaces (cat dog) to search cat,dog or both. Separate the words with plus signs (cat +dog) to search for items that may contain cat but must contain dog.
  • You can further refine your search on the search results page, where you can search by keywords, author, topic. These can be combined with each other. Examples
    • cat dog --matches anything with cat,dog or both
    • cat +dog --searches for cat +dog where dog is a mandatory term
    • cat -dog -- searches for cat excluding any result containing dog
    • [cats] —will restrict your search to results with topic named "cats"
    • [cats] [dogs] —will restrict your search to results with both topics, "cats", and "dogs"

Media Framework 360 Video?

Hello,

is it possible to create a VR friendly 360 video using UE4's Media framework? Would we go about doing that applying that to a spherical object? could we use it for a samsung gear VR experience as well?

Product Version: Not Selected
Tags:
more ▼

asked Mar 12 '15 at 06:19 PM in Using UE4

avatar image

julianque
100 9 12 19

avatar image DunkenKBliths Jul 20 '16 at 06:33 AM

Any news on this....

Just trying to create a menu structure system and a 360Video player... 4k (4096x2048 -30fps) just seems to be sooooooooo slow and jumpy...even on good hardware

i7 (4Ghz) - 980Ti (24G-Ram) / 256G SSD

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

3 answers: sort voted first

Yes, you could map a video onto a sphere. There is at least one other project I know of that is doing exactly that. You will likely have to apply some distortion to the video itself so it looks right when projected onto the sphere.

The only problem with this right now is that the video has to have a very high resolution - at least 2k x 2k, better 4k x 4k, otherwise it won't look good in the HMD. The way we currently upload video textures to the GPU is particularly slow for large videos, so you won't get a good framerate. We are in the process of fixing this and hopefully have a solution for 4.8. In the meantime, you can use a low-res video for development and testing; it won't look great, but at least you're not blocked.

more ▼

answered Mar 27 '15 at 03:46 PM

avatar image fengkan May 03 '15 at 03:16 AM

When I convert a 360 video into material and put it onto a sphere, the video seems to be reversed. How may I solve that problem?

avatar image MathiasG May 16 '15 at 01:19 PM

Fengkan, you have any blueprint or solution how you do this? Would be most kind :)

avatar image haruna Apr 08 '16 at 11:38 AM

Use this: https://docs.unrealengine.com/latest/INT/Engine/MediaFramework/index.html

Make a folder in Content called Movies and then import your movie folder there. Right click the movie to make a media texture. Then right click the texture to make a material. Put the material on a static mesh.

avatar image rharbin May 09 '15 at 10:17 PM

Is this still an improvement to look for in 4.8?

avatar image gmpreussner STAFF May 11 '15 at 04:54 PM

Nope, sorry, I was out of office for three weeks and did not have time to work on this. Pushed to 4.9 now.

avatar image devel.bmad Jul 13 '15 at 09:37 PM

Great to hear this! Could you please link the commit here for reference? Would this help also with gear VR?

avatar image gmpreussner STAFF Jul 13 '15 at 09:41 PM

We still haven't had time to work on this. I'm currently completely backed up with other high priority, and I will be on travel all of next month. Unless someone else can look into it, this won't happen before September, I'm afraid :/

avatar image polemos Jul 28 '15 at 12:47 AM

so this is not going to get into 4.9??? This is the feature I was looking forward to most!! is there a way to capture out your scene to a 360 video currently I use Allar's method https://www.youtube.com/watch?v=3m85QBjyFGE

But I can not get the frame rate of my movie textures to match the frame rate of the game even if I specifically match the frame rate of the game in the project settings to match the frame rate of the video any one know??

avatar image DunkenKBliths Jul 22 '16 at 04:59 AM

Didn't make it into 4.9 and news when it might ? This feature will stop a lot of people moving to Unity...especially as 360Video is the thing of the moment ! I really don't want to go down that path ...invested too much time and sweat ! Cheers

avatar image Wollan Aug 04 '15 at 08:25 AM

Definitely interested in this. Ultimately hoping for 60fps 4K playback on GearVR ...

avatar image mitjaprelovsek Feb 01 '17 at 04:50 PM

Hi,

I was wondering if this upgrade made it to 4.14. At the moment I am using media framework and am mapping monoscopic wmv 360 video onto an inverted sphere and the CPU really starts choking up during playback. This causes the HTC VR to drop frames and you get lag in movement of your headset and controllers. I am running i5-2500, which is not the latest, I know. It's not a problem purchasing a better CPU, but would appreciate a comment on the status of the 4k video playback and how to increase performance.

Thanks!

avatar image gmpreussner STAFF Feb 01 '17 at 06:13 PM

There were several performance updates in 4.13 and 4.14. Some more will come in 4.16. Give it a shot and see how far you get. Report back any problems or issues (in a new AnswerHub thread), thanks!

avatar image AgentMilkshake1 May 24 '17 at 10:23 AM

Hi there,

I rendered out a video using the stereoscopic 360 plugin pipeline. I am now displaying just one of the panoramic videos (4k, 60fps) within a material that is applied to the inside of a sphere, and there are regular performance drops on the GPU (I have a GTX 1080). Is there any way to support smooth video feedback on the video sphere? Cheers

avatar image quality May 24 '17 at 10:43 PM

Playing a full 4K video still has issues. We've been using Unity because of this. I suspect that it has something to do with the streaming of the texture at that resolution, because you can split a 4k video in half and put it on 2 planes stacked on top of each other and it plays back just fine.

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

I found a solution thanks to help from Rich at Hammerhead VR. Below worked for me, started with 6 cameras upped it to 18 for the best result in combination with Autopano Video:

120 fov should work.

In order to achieve the desired result we will need to create a template stitch since our standard ue4 scene isn't detailed enough for the automatic algorhythmns to work. I'm afraid I've lost the screenshots of this but I'll try to explain it and find an example image on Google.

Essentially what we want is a perfect stitch from ue4 using the exact same camera setup used in our matinee and to splice that footage into the first few frames of each matching video.

The only way I've been able to do this thus far is to create a cube with inverted normals or BSP and apply a unique, high fidelity photograph to each face. Then to capture the matinees (just a few frames) This will mean that the autopano / video stitch algorithms will be able to recreate the image. Splice those frames into the beginning of each video. And stitch from that frame, this will mean that the remainder of the footage in your video will be perfectly stitched without any extra work. This process is repeatable as well, once you have your template frames you can use them over and over again on new shots. You may like to label each face of the cube with a number in order to keep track of which is which, which way is up and down, orientation ect. I had to create a cardboard net out of a cereal box, number the sides and keep it on my desk in order to keep track of how my cube was arranged.

Please see attached for image examples of a cube room and a HDR image used as a texture in that room.

When working with 360 video in real life, people generally download a template stitch for their 360 hero rig or freedom 360 go pro rig in order to speed up processing or solve any technical challenges. Since we don't have a standard rig it won't work for us, but the benefit is that we can place an infinite amount of cameras on the exact same pixel so there should be less ghosting but of course each video will increase rendering time and workload. This is already an incredibly labour intensive process so be careful. I think this technique will work best on static images since when the camera moves there is a higher chance of seams. You should also be aware of automatic exposure control on your ue4 cameras since this will effect the continuity of your footage between the individual shots. One problem you'll run into is with 6 cameras you may not have enough shots with overlap to create a completely smooth stitch. The best way to get around this is to put a fish-eye warp on your lens. You could either create this as a post process on-camera effect or reverse engineer the oculus integration. When rendering in matinee for some reason the renderer takes a little while to spin up and produce high quality footage. It will be difficult to render each video individually this way and make them sync up, so instead we should sequence each camera in the director group and then split apart the JPEG frames into folders. Be aware, sometimes matinee loses frames for no reason so it's more tricky / slower for us to import them into premiere as a frame sequence.

more ▼

answered Jun 01 '15 at 08:22 PM

avatar image

julianque
100 9 12 19

avatar image quality Jul 13 '15 at 11:59 PM

@julianque, how are you getting around the current resolution limitation, or are you? We tried to import some 4k footage into UE4 and it was just not even possible to play.

(comments are locked)
10|2000 characters needed characters left
Viewable by all users

the whole Video play in Unreal is "hit" and miss" even in 4.11.1

Its seems fine in the dev environment... (with low res HD from a Ricoh Theta S) ...and looks ok with 4k (demo footage) but as soon as its compiled to an exe its hit and miss if it works... some setting seems to stop things working even on a PC...on a Samsung S7 Gear VR it worked once...and haven't had it working since....

Would love it all to work so we could start using this for lots of clients !!!

more ▼

answered Apr 09 '16 at 07:54 AM

avatar image

DunkenKBliths
206 3 100 15

avatar image so2gamemake Apr 11 '16 at 03:10 AM

I do not fully understand what makes a video playable or not with Media Framework just yet.

I asked about what works for GearVR and I have no answers: https://answers.unrealengine.com/questions/379821/gearvr-video-media-formats.html

I tried downloading sample videos that other users said they got to play with Media Framework but I have had no such luck. I am so confused. Please help.

avatar image gmpreussner STAFF Apr 11 '16 at 07:35 PM

The Android based player plug-in has pretty good support. I'd recommend H.264 encoded .mp4 files. You won't be able to play this same file inside the Editor though, because the Windows based player plug-in doesn't support H.264 yet (but it will work when you run on device).

The only format that works on both Android and Windows is MPEG-4 encoded .mp4 files, but the quality will be much worse.

In general, 360 video is not supported very well right now, because of performance limitations. I am currently refactoring the Media Framework to address these and other issues.

avatar image quality Jun 10 '16 at 03:13 AM

Has there been any traction on this front Gmpreussner or Chris? We've been stuck in Unity doing do all our 360 video work. I came sniffing around after reading the article about capturing 360 video from the editor by Ninja Theory, but when I fired up 4.12.1 today to test I got the same sad results with HD mp4 files being super slow to play. The movies import correctly, and at the right resolution, they are sharp in the texture viewer, but in translation through material to sphere in the editor they get all pixelated. This is literally the only thing right now keeping us from fully moving to UE4 for all our applications. Thanks for looking into things.

avatar image gmpreussner STAFF Jun 10 '16 at 03:05 PM

Still working on it. Windows has H.264 support now, but I can't integrate it into the Master branch until all player plug-ins are upgraded.

As for your pixelated videos, what resolution are they in?

avatar image quality Jun 10 '16 at 08:34 PM

good to hear it is still being worked on. The 2K is 2048x1024 and the 4K is 4096x2048 both movies play fine outside of Unreal but are really slow in Engine and their quality goes way down. It's understandable that there's a lot of computation going on to map the video to the sphere, but I don't believe it should be so slow. Unity can play 4K videos in editor without a hitch. I can even stream 2K videos directly to my mobile app. Thanks for continuing to work on it. (I've attached a comparison screenshot of a 4k video to show the quality degradation) alt text alt text

capture.png (513.1 kB)
capture2.png (1.2 MB)
avatar image gmpreussner STAFF Jun 10 '16 at 08:45 PM

Thanks, we're pretty close to getting the new Android player working, and then we'll take a look at the quality.

avatar image philboltt Jun 18 '16 at 12:16 AM

This is all good news! Thanks for all the work. What's the timeframe for getting the player plug-ins updated?

avatar image gmpreussner STAFF Jun 18 '16 at 10:39 AM

I probably need another two weeks until the first check-in into the master branch. Officially, Media 2.0 is planned to be released with 4.13.

We have the new Android player working, and it looks pretty good. 4k video playing crisp and fast on a Galaxy S7.

avatar image JHalper Jun 28 '16 at 03:32 AM

Might be a little early but any progress? Also interested to see the next implementation. I imagine it will be available on Github before official 4.13 release?

avatar image ZoltanJr Jun 28 '16 at 11:46 AM

I would love to get access to this! I have a project going out on Friday to internals and out to the public end of July. Any chance this will make it into master this week or when will preview 1 of 4.13 be coming along?

avatar image DunkenKBliths Apr 11 '16 at 10:52 PM

I have had limited success ...but only really managed to get it to work on a PC...

I too want to go to VR Gear...but it fails miserably every time when compiled for Android..Weird screen results..

To actually get it to work on a PC I have had to go back to 4.10 as 4.11 no longer seems to work...somethings broken...

avatar image Chris Babcock STAFF Apr 11 '16 at 11:24 PM

Have you tried it without GearVR to see if the movie plays on device? We have seen an odd interaction between the GearVR API rendering and the native movie player if the surface isn't in view.

avatar image DunkenKBliths Apr 13 '16 at 03:35 AM

Yes the Movies play fine on the phone and they aren't particularly high quality (working up to that !)...so not a "bandwidth" issue... all very frustrating...especially as your platform is fantastic, just the part that I need doesn't work....

avatar image Chris Babcock STAFF Apr 13 '16 at 10:06 PM

t looks to be maybe a flushing or state issue; I'll check into it more and see if I can get a proper fix but I have a workaround you can try.

Create a UMG widget and add a tiny border in the upper left corner. Set the brush color alpha to 0.01 so it isn't visible. Create and add it to the viewport in your level blueprint. This will make sure it renders something last. This cleared up the problem in our testing.

avatar image tiagovignatti Dec 13 '16 at 05:45 PM

Hey Chris, I think I'm seeing the "flushing" state issue you described here:

If I set to transparent the joystick button images on the touch interface (yes, I know it's nonsense to you it on Gear VR...) then the video is rendered all wrong, like a weird triangle on the screen. But if I leave a very tiny and non-transparent button then everything seems alright. That's how I'm working around now but I'm wondering what the right solution for this. I tested on both 4.13.2 and 4.14.1 and on both the bug persists.

Thanks.

avatar image Chris Babcock STAFF Dec 13 '16 at 05:53 PM

I have a new fix for 4.15. You can try replacing MediaPlayer14.java with the latest version from GitHub master here: link text

(comments are locked)
10|2000 characters needed characters left
Viewable by all users
Your answer
toggle preview:

Up to 5 attachments (including images) can be used with a maximum of 5.2 MB each and 5.2 MB total.

Follow this question

Once you sign in you will be able to subscribe for any updates here

Answers to this question