Best practices for keeping Github integration up to date?

[Qartar]

We’re having some issues maintaining our UE4 fork due to the way that Epic manages their release branch. Epic doesn’t merge forward previous point releases until the next point release; preview releases have diverging histories which makes it difficult to prepare for the official release when it comes up. To illustrate, the simplified branch history for 4.13 and 4.14 looks something like this:

You should be able to see from the revision graph that 4.13.2-release is not merged up into any of the the 4.14 tags until 4.14.0-release. We would like to be able to do a preliminary merge when Epic releases a preview but with the current branching strategy that’s not feasible since that would require us to manually reconcile the changes that have taken place on either branch since they’ve diverged. For example, a merge from 4.14.3-release into 4.15.0-preview-1 results in around 200 conflicted files. The result is that we can’t do an effective merge until the day of the release unless we want to maintain a separate branch that merges Epic releases into Epic’s current development branch (we don’t). This puts us at a real disadvantage when trying to keep up with Epic’s release cycle.

I think it’s worth noting that Epic’s Wiki suggests that they are following the Gitflow branching model (see [“Recommended branching workflow”][2]). However, the Gitflow model recommends that changes to the release branch are merged back into the development branch but Epic does not seem to follow this practice. See [Atlassian’s description of Gitflow][3]: (emphasis mine)

Once develop has acquired enough features for a release (or a predetermined release date is approaching), you fork a release branch off of develop. Creating this branch starts the next release cycle, so no new features can be added after this point—only bug fixes, documentation generation, and other release-oriented tasks should go in this branch. Once it’s ready to ship, the release gets merged into master and tagged with a version number. In addition, it should be merged back into develop, which may have progressed since the release was initiated.

There a couple things that Epic could do to make this process easier:

1. Merge point releases back into the development branch, as recommended by Gitflow.
2. Merge preview releases into the release branch rather than tagging them to the development branch.

Is there anything aside from process changes on Epic’s side that would make this easier?

[anonymous_user_b666e59e]

Hi Qartar,

The Release branch is intended to be the branch that contains the source code for the most recent released version of the Engine. In theory, the Release branch should never receive any isolated commits that are not found in another branch, most likely the branch for that version (eg. 4.14). That means that, in theory, there should be no need for the Release branch to merge back into a development branch. In practice, there may be instances where a change does get committed into the Release branch and would require a merge back into a development branch.

I will try to explain our internal source control methods and how those correspond to our repo on GitHub. Internally, we make use of Perforce streams for source control. Each of our development teams has at least one stream that they work out of, and frequently there are multiple streams for different areas that a team covers (eg. Build, Core, Sequencer, etc.). These streams will periodically sync changes from our Main stream which, as its name implies, is our primary stream that all other streams are based on. Development work occurs in the individual development streams which are then periodically, after some basic testing to make sure nothing is horribly broken, merged back into the Main stream.

When we are preparing for a new version release (eg. 4.15), we branch off a new stream for that release from the Main stream. From this point on, no new merges from Main go into the release stream. If a major fix occurs in Main and needs to be included in the release stream, that fix is manually applied to the release stream instead of merging Main into the release stream. As fixes go into the release stream in preparation for the release of that version of the Engine, the release stream is periodically merged back into the Main stream to ensure those fixes are also included in future versions of the Engine. Once we are done working on any hotfixes for an Engine version, that release branch gets merged back into the Main stream one final time and a new release stream for the next version gets created. Ideally, previous release streams would not see any additional changes applied to them after that final merge into the Main stream.

As I am sure you have guessed by now, our GitHub repo does not currently reflect the full extent of our internal source control. Our development streams are not mirrored to GitHub right now. We want to get them mirrored over, but have not yet had an opportunity to task some of our engineers with getting that done. Here is what is currently mirrored over:

• Internal Main stream → GitHub Master
  branch
• Internal Release streams →
  Numbered GitHub branches (eg. 4.13,
  4.14, 4.15, etc.)

The Promoted branch is a branch that is updated from the Master branch when our QA team successfully completes a suite of tests on the current Master branch. They try to do this daily, but the branch will not be updated if any of the tests fail.

The Release branch is updated when the Engine in the current internal Release stream passes all of its release tests and is considered good to be released.

The Dev-VR-Editor branch is an exception to our normal process, in that this branch is an actual development branch. We try to keep it in sync with the Master branch as much as possible, but there may occasionally be some significant divergences between the two branches.

In terms of stability, the numbered release branches should initially be considered the least stable when they are first created, with stability improving over time as we get closer to releasing that version. The Master branch should also be generally considered unstable, as it receives the least amount of testing. The Promoted branch should be fairly stable, but the Release branch will be the most stable option, and is the one we recommend using for project development.

As for keeping your Engine source code up-to-date with what we are working on, there is unfortunately no one-size-fits-all solution. What meets the needs for one team may not for another team. Some teams work using our Master and Promoted branches (we do not recommend this), and update them regularly. Other teams won’t update their Engine at all once they have started a project unless they need a specific bug fix or feature. Most teams fall somewhere between those two extremes. If you are not making any changes to source code yourself, then you should be fine to resolve any conflicts in favor of the files in the later version of the Engine. If you are updating your Engine to one of our preview builds, it may be a good idea to work on a copy of your project until you are sure nothing in the preview build will have an adverse effect on your project (this is really a good idea any time you update your Engine version).

I hope this helps. If there is anything I have not adequately explained, please let me know.