Operating System In UE4

Would it be possible to build and OS inside of Unreal Engine?

Sort of, it depends what you exactly want to achieve. You could probably even embed a full virtual machine (emulator) in UE4, which then could run an off the shelve OS. It would, however, be unlikely that you’d get anything useful that way.

Instead, you may actually want an OS-like experience in your game? You can, for example, allow the player to run their own “apps”. These apps could be written in, e.g., JavaScript and executed by an embedded JavaScript engine (see Marketplace). Your game could provide custom functions to the JavaScript engine, allowing it to control your game. This would be somewhat like system APIs a regular OS offers.

Beware that when you allow the player to run arbitrary code/scripts that exposes a security threat. Even a well tested JavaScript engine, like V8, will have vulnerabilities and needs regular updates. It would be a good idea to allow the player to only run “apps” they create themselves, or better yet, only apps you provided. If players need to write their own apps, they may copy paste something insecure without understanding it. However, if all apps could be written by you, I guess you’re more after an “OS-like” user interface.

If this doesn’t answer your question at all, I guess you have to elaborate a bit on what you want to achieve with this OS.