Extending the level build system

I’m working on a plugin idea that would really benefit from being tied into the level build system, but that doesn’t appear to be very pluggable yet. If there’s something I’m missing do let me know what delegates…etc I can use to patch into it.

My ideal system would allow me to insert a “Build Step” class into the process. Currently it seems to be a very hardcoded series of items, (see EBuildType). The Build Step class would be called for processing the custom build step and reporting progress. Generating the menu items under drop down Build button for specifically running my build extension would probably be handled by some other code for maximum customization.

For the time being I’m just going to make it it’s own menu item for testing. I was also thinking about making it a background process that runs continuously so that it’s capable of playing nicely with SIE/PIE. I imagine the navigation building system might or could also function in this way. So it might be worth having an extensible background build system as well that is easy to extend and allows the ability to run when certain conditions have been invalidated. If you already have a robust system for tracking dirty objects, that would also be valuable to expose to minimize rebuild time in some situations.

Also being able to prevent starting SIE/PIE until some of those build processes complete if it’s critical to gameplay or rendering (ex, occlusion culling) would also be important. With the idea being if someone triggers a background build by changing something that would require a navigation rebuild, and then presses play…you probably don’t want them jumping into the game with bad nav, for example.

Forgive the rambling grocery list, it started off as a shorter post, I’m still sussing out what I’ll need for my plugin to play nice with everything. :slight_smile:

Thoughts / Suggestions?

The build menu items are not automatically generated. The plan to add extensibility to those is still underway (almost done now).

As for actually having extensions to the building system, that’s not something that we had considered yet, but it certainly is a good idea! I’ll go ahead and put this idea up for discussion at the next tools meeting.

It should also be noted that the entire build system was intended to be moved to being background process that the user would never have to manually trigger, however that task is still on the queue to be worked on.

I’m curious, what exactly is the use case for this new build step?

I’m working on doing a port/rewrite of a tool called Oxel I wrote awhile back as a little research side project. It generates low poly, conservative occlusion volumes for use with Hierarchical Z-Buffer Occlusion Culling.

The goal would be to have a Rocket plugin that does the occluder generation as a tool time plugin, as well as a runtime plugin/element that modifies the rendering pipeline to do custom occlusion culling with HiZ using the generated occluders.

I definitely expect to run into all manner of roadblocks that need workarounds until there are extension points for it all :smiley:

This should be a good exercise to see how easy it would be for (tool + runtime) middleware plugins to be integrated.

This sounds like a pretty sweet project. I think you’ll be able to get a lot of it working, except perhaps the rendering bits. I don’t think we support hooking the visibility determination phase of the renderer yet. But definitely take a look and tell us what you would need exposed!

–Mike

Sounds like a great plugin!

Mike is right that there is currently no way to extend the renderer in that manner. We’re looking at ways to build in extensibility and this is one that has come up. Essentially we’d provide plugins with the ability to add “filters” during visibility determination and also a way to turn existing filters like frustum culling and latent occlusion queries on or off. That’s a quick sketch of how it may be exposed but we don’t have a timeline for implementation yet.

Keep us posted – I’d be curious what else you discover needs to be exposed!

Hey Nick!

The filter idea sound good. I’ve been jumping into the plugin the last couple of days, some thoughts/questions I’ve had:

There doesn’t appear to be a way to inject a prioritized step in the render thread that has the ability to do some custom rendering, speedtree style. Where I could just write a whole bunch of raw D3d or GLEs code and handle things myself.

If there were a wrapped cross platform rendering interface that allowed raw-ish calls (as opposed to having to write both D3d and GLES code) that would also be awesome.

I’ve found ENQUEUE_UNIQUE_RENDER_COMMAND, but I don’t think that’s what I need.

We have a thin wrapper around the underlying graphics API called RHI (Render Hardware Interface). The following headers should get you started:

/UE4/Engine/Source/Runtime/RHI/Public/RHI.h
/UE4/Engine/Source/Runtime/RHI/Public/RHIMethods.h
/UE4/Engine/Source/Runtime/RHI/Public/RHIResources.h

All RHI methods must be called from the rendering thread. You can do that from your code by using the ENQUEUE_UNIQUE_RENDER_COMMAND macros. A pattern we often use here is something like this:

class FSomeClass
{
public:
	void DoSomething(FSomeData* SomeData)
	{
		// I should always be called by the game thread!
		// Note that you always have to assert the positive.
		// In some cases Game Thread == Rendering Thread!
		check(IsInGameThread());

		// Note that SomeData must not be modified from now on
		// by the game thread. It must also remain valid and not
		// be freed until the rendering thread has finished with
		// it. One option is to copy it. Another option is to
		// transfer ownership. In this case I am transferring
		// ownership.

		// Enqueue a command for the rendering thread to do our work.
		ENQUEUE_UNIQUE_RENDER_COMMAND_TWOPARAMETER(
			FDoSomethingCommand,
			FSomeClass*,This,this,
			void*,SomeData,SomeData,
		{
			This->DoSomething_RenderThread(SomeData);
		});
	}

private:
	void DoSomething_RenderThread(FSomeData* SomeData)
	{
		// I should always be called by the render thread.
		// Note that you always have to assert the positive.
		// In some cases Game Thread == Rendering Thread!
		check(IsInRenderingThread());

		// Do whatever we need to do with this data.

		// Now we've finished with it and we own the data
		// so we are responsible for freeing it.
		delete SomeData;
	}
};

Perfect, I’m good from here. Thanks Nick!

This post is quite old now, but did the ability to extend the level build system ever get implemented? I’m unable to find any documentation about it.