How can I get Programmatic Access to Camera Pixels?

anonymous_user_c7e7fb1a · April 25, 2014, 11:39am

Good day everyone. I am looking into using UE4 as the rendering and physics framework and implementing my ‘animats’ in UE4, basically with my neural network systems controlling custom actors. It looks very promising, exept for one thing; I do not know how easy it would be for me to gain programmatic (C++) access to the ‘image’ from a camera’s point of view. I want my AI to do visual processing, and it would be super if I could ask one or more ‘eye cameras’ to give me their live ‘image’ data, which I would then feed into my neural net for processing.

Thanks,

anonymous_user_f54268a3 · April 25, 2014, 1:49pm

Yes,

Look at TextureRenderTarget2D, and SceneCapture2D components.

Also, I recommend doing it not entirely in C++ but use Blueprints to ease setting up the SceneCaptture and TextureTargets, then work with the texture in C++ to process the pixels.

There are other threads regarding setting up the Blueprint, but the example to look at is the Security camera.

anonymous_user_c7e7fb1a · April 25, 2014, 4:22pm

Thank you for pointing me in the right direction.

anonymous_user_f88179b8 · March 5, 2016, 1:25pm

, did you solve this? If so, do you mind posting the pointers (or if possible code sniplets)?

Thank you

anonymous_user_c44ea37c · April 11, 2016, 5:09am

Is there any online description of your work? I’m about to attempt such a network myself. I’m amazed you did it, so the big question is how if you are willing to point me in the right direction, - please. Really cool stuff.