Running Horde CI on a single build machine

A while back I wrote about standing up Horde as our build system. This post is about how the CI on top of it is set up, and specifically how it’s shaped by one constraint: we’ve got a single on-prem Windows build agent. No farm, no fleet, one (admittedly beefy) box that has to compile, cook, and do everything else.

That constraint matters because a good chunk of Epic’s BuildGraph is written on the assumption you’ve got a farm behind you, and on a single machine some of those defaults cost you. If you’re here for the Steam side of things instead, that’s a separate post.

Here’s what our CI does:

• every commit builds, so we catch breakage early
• UGS precompiled binaries (so the team pulls a prebuilt editor rather than compiling locally) rebuild only when code changes
• content changes get cooked, since in my experience that’s what actually catches a broken asset or a missing reference

Splitting by what changed

Horde schedules can look at what a changelist touched, via ChangeContentFlags, and only fire when it matches. That’s the core of it: a code commit and a content commit are different jobs, and I don’t want to pay for both on every push.

One thing to get right up front: the flag values are ContainsCode and ContainsContent, not Code and Content. Get the name wrong and the config won’t parse, you’ll just get The JSON value could not be converted to ChangeContentFlags, and since streams share a config file the bad name will take the other streams down with it too.

The code path is an incremental editor compile that publishes binaries, scheduled on code changes:

{
  "id": "incremental-build",
  "name": "Incremental Build (Editor binaries for UGS)",
  "initialAgentType": "IncrementalWin64",
  "showUgsBadges": true,
  "arguments": [
    "-Script=Engine/Build/Graph/Examples/BuildEditorAndTools.xml",
    "-Target=Submit PCBs",
    "-set:UProjectPath=$(ProjectPath)",
    "-set:EditorTarget=$(EditorTarget)",
    "-set:PreferredAgent=IncrementalWin64;Win64"
  ],
  "schedule": {
    "enabled": true,
    "maxActive": 1,
    "maxChanges": 1,
    "requireSubmittedChange": true,
    "filter": [ "ContainsCode" ],
    "patterns": [ { "interval": 5 } ]
  }
}

A couple of the schedule fields are worth knowing. maxChanges: 1 builds the latest new change per poll and coalesces a burst of commits into a single build, rather than queuing one build per commit. maxActive: 1 stops the template stacking multiple runs. interval: 5 polls every five minutes. If you’d rather build every individual commit so you can bisect a break, bump maxChanges, but on a single machine I’d rather coalesce and keep the queue short.

The content path is the same shape with the opposite filter (ContainsContent) and a different graph. So:

• a code-only commit publishes binaries
• a content-only commit cooks
• a commit with both fires both templates

That last case is worth being clear about. The flags aren’t mutually exclusive and there’s no clean way in Horde to say “content but not code,” so a mixed commit triggers both. That’s fine here, the second job runs on a workspace that’s already up to date, so it’s cheap, but it’s a tradeoff to be aware of if double-triggering would bother you.

Two workspaces

I split the work across two managed workspaces, and the line I draw is per-commit vs release, not code vs content:

"workspaceTypes": {
  // Fast per-commit CI: editor compile for UGS binaries, and the content cook.
  "Incremental": {
    "identifier": "WH-Main-Inc",
    "incremental": true,
    "method": "name=managedWorkspace&preferNativeClient=true"
  },
  // Clean full builds: packaged / Steam releases.
  "Full": {
    "identifier": "WH-Main",
    "method": "name=managedWorkspace&preferNativeClient=true"
  }
}

All the per-commit work lives on the incremental one. That includes the cook, so it has content. It’s tempting to think the compile and the cook will trip over each other sharing a workspace, but they don’t: a cook runs the already-built editor, it doesn’t recompile, so it never touches the compile’s Intermediate/. They even compile the same editor target, so they reuse each other’s output. The thing that does churn build state is a Shipping package or Steam compile, and that’s exactly what lives on the other workspace.

incremental: true is what keeps that workspace in place. Without it, Horde cleans the managed workspace between runs, reverting and removing files not in the have-table, which wipes untracked output like Saved/Cooked and your object files. With it, both the compile and the cook stay incremental: the first cook is a full cook, and after that only changed assets and their dependencies re-cook.

Full deliberately doesn’t have that flag. Releases sync clean, so a packaged build or a Steam upload can’t inherit a stale cooked asset from a previous run, which is the sort of thing you really don’t want discovering after you’ve shipped it. It’s a slower build, but releases are rare and being sure of what’s in them is worth more than the minutes.

Keeping the cook on one agent

Content validation is a cook, and the shape of the graph matters a lot on a single machine.

Epic’s stock graphs, BuildAndTestProject.xml and friends, split work across separate <Agent> groups so a farm can run compile, cook and stage on different machines in parallel. Each group is its own unit, so BuildGraph passes outputs between them as tagged artifacts through the server’s shared storage: an upload and a download per hop. On a farm those hops overlap with real parallel work. On a single agent there’s nothing to overlap with, so you just pay the transfer. For a cook, that means uploading the whole compiled editor to the server and pulling it straight back before the cook can start. It dominates the runtime.

So for a single agent, keep it in one <Agent>, one <Node>, everything on local disk:

<?xml version='1.0' ?>
<!--
  Content validation for a SINGLE-AGENT setup.
  One <Agent> group, one <Node>: compile the editor (and the client target),
  then cook. Everything stays on the agent's local disk - nothing is uploaded
  to / downloaded from the Horde server between steps.
  A cook error fails the node - that's the content-validation signal.
-->
<BuildGraph xmlns="http://www.epicgames.com/BuildGraph"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="http://www.epicgames.com/BuildGraph ../../../Engine/Build/Graph/Schema.xsd">

  <Option Name="ProjectFile"  DefaultValue="" Description="Path to the .uproject, relative to the branch root"/>
  <Option Name="EditorTarget" DefaultValue="" Description="Editor target name (from *.Target.cs)"/>
  <Option Name="GameTarget"   DefaultValue="" Description="Standalone game/client target name (from *.Target.cs)"/>

  <Agent Name="CI Content Validation" Type="IncrementalWin64">
    <Node Name="Compile And Cook">
      <Error Message="ProjectFile and EditorTarget are required"
             If="'$(ProjectFile)' == '' or '$(EditorTarget)' == ''"/>

      <!-- Editor is needed to cook, and compiling it builds the game's C++ modules
           (Runtime + Editor), so custom UDataAsset/asset classes are exercised by the cook. -->
      <Compile Target="$(EditorTarget)" Platform="Win64" Configuration="Development"/>

      <!-- Also verify the standalone game/client target compiles (catches WITH_EDITOR
           leakage and client-only module breaks that the editor build won't). -->
      <Compile Target="$(GameTarget)" Platform="Win64" Configuration="Development" If="'$(GameTarget)' != ''"/>

      <!-- Incremental cook of all content for Windows. Any cook error fails the node. -->
      <Cook Project="$(ProjectFile)" Platform="Windows" Arguments="-cookincremental" TagOutput="false"/>
    </Node>
  </Agent>

  <Aggregate Name="Validate Content" Requires="Compile And Cook"/>
</BuildGraph>

The Horde template that runs it is just as small, it passes the three targets in and points at the aggregate:

{
  "id": "ci-validate",
  "name": "CI Content Validation (Cook)",
  "initialAgentType": "IncrementalWin64",
  "arguments": [
    "-Script=Game/Build/CI/ContentValidation.xml",
    "-Target=Validate Content",
    "-set:ProjectFile=$(ProjectPath)",
    "-set:EditorTarget=$(EditorTarget)",
    "-set:GameTarget=$(GameTarget)"
  ],
  "schedule": {
    "enabled": true, "maxActive": 1, "maxChanges": 1,
    "requireSubmittedChange": true,
    "filter": [ "ContainsContent" ],
    "patterns": [ { "interval": 5 } ]
  }
}

Both compiles and the cook sit in the same node, so nothing crosses the network, and it does less than the stock graph since it skips staging and packaging entirely.

Two details in that graph worth calling out:

• Pass -cookincremental yourself. In Epic’s graphs, incremental cooking is gated behind an IsEpicBuildMachine check, on for them, off for you. Even with a persistent workspace you have to opt in, or you’ll do a full cook every time and get nothing from the incremental workspace.
• Node and aggregate names share one namespace. They can’t be the same string, so the node is Compile And Cook and the aggregate is Validate Content. Name them the same and the graph fails to compile with '…' is already defined.

Two jobs, one workspace

Both the binary compile and the cook run on that incremental workspace, so it’s fair to ask why they’re two templates and not one. It’s the trigger, not the workspace. Binaries only want publishing when code changes, and the cook only earns its keep when content changes, so each is keyed to its own ChangeContentFlags filter. Rolling them into one job would mean publishing binaries on content-only commits (pointless, no code changed) or cooking on code-only commits (pointless, no content changed). Two templates on the one workspace maps cleanly onto the two things I actually want to happen.

If you’re on a single build machine, the theme is probably clear by now: check what the stock graphs actually do before adopting them, and count the <Agent> boundaries, each one is a point where your data leaves the box and comes back. The Steam upload side is in the next post. As always, let me know if you hit any of the same walls.