27 June 2026

Two days chasing a 60-second timeout: a Horde UGS-PCB upload debugging story

Self-hosting Unreal's Horde with an on-prem build agent, our UGS precompiled-binary uploads kept dying at 60 seconds. A dozen dead ends later, the fix was four lines, in a process I hadn't even been looking at.

Some bugs are a quick fix in the obvious place. This was the other kind: two days, a dozen dead ends, a server resize, a reverse proxy removed, a custom-patched build agent, and the actual bug sitting in a completely different process than the one I kept patching. If you self-host Unreal’s Horde with an on-prem build agent, this one’s for you.

The symptom

Self-hosted Horde on a cheap cloud box, one beefy on-prem Windows build agent. The CI build (compile the editor, publish UGS precompiled binaries so the team syncs prebuilt editors instead of building locally) ran fine right up to the last step, uploading the ~6.85 GB PCB artifact back to the Horde server:

PUT .../storage/horde-artifacts/blobs/ugs-pcb/mh-main/27/editor/...  timed out after 60s.
PUT .../storage/horde-artifacts/blobs/ugs-pcb/mh-main/27/editor/...  retrying after 5s.

Every 10 MB blob died at 60 seconds. The artifact never completed, so the default ref never got written, so UGS couldn’t resolve it: RefNameNotFoundException: Ref 'default' not found. Two errors, one cause.

What it wasn’t

None of these were the cause. Each was plausible enough to be worth ruling out, though, and ruling them out is what eventually pointed at the real one.

  1. “It’s the reverse proxy.” Caddy sat in front of Horde for TLS + gRPC routing. I bumped its transport timeouts. One upload squeaked through, the rest still died, so it wasn’t the proxy.
  2. “The box is too small.” cx23 (2 vCPU / 4 GB) ingesting parallel blobs looked CPU-bound, so I resized to cx42 (8 vCPU / 16 GB). Still timing out, and docker stats during an upload showed ~0.6 of one core: the server was idle. The resize changed nothing.
  3. “It’s HTTP/2 flow control through Caddy.” Plausible, since h2 caps in-flight data per window, and on a latency-y link that throttles uploads regardless of bandwidth. So I removed Caddy entirely, had Kestrel terminate TLS directly (HttpsPort + a cert in Kestrel:Certificates), and cranked Http2.InitialConnectionWindowSize. Still timing out, so flow control wasn’t it either.
  4. “Limit the streams.” Set MaxStreamsPerConnection. No effect, because the agent uploads over many separate HTTP/1.1 connections, not multiplexed h2 streams. Wrong knob.
  5. “Limit the connections.” Set Kestrel MaxConcurrentConnections=10, with no change.

Meanwhile a raw scp of a 100 MB file to the same box clocked 64 Mbit/s on a single stream, the network was fine. And the uplink was 100 Mbit. A 10 MB blob should upload in ~1 second. So why were they taking 60+?

The first real insight: it’s parallelism, not bandwidth

The agent fires many blob uploads at once. Split a 100 Mbit uplink across ~40 concurrent PUTs and each one crawls at ~1.5–2.5 Mbit/s, so a 10 MB blob takes 30–60s and tips over a hard-coded 60-second timeout. Aggregate bandwidth was fine; each individual request was too slow. CPU, disk, proxy, server size, all red herrings. The constraint was concurrency vs. a fixed per-request deadline.

So: raise the timeout (let slow requests finish) or cut the parallelism (make each request fast). The timeout lived in the code, hard-coded. So I went to patch it.

The detour: a custom-patched agent that fixed nothing

HordeHttpMessageHandler.cs had it: Policy.TimeoutAsync<HttpResponseMessage>(60, ...). Change 60 to 600, rebuild the agent, deploy. That should have been the end of it, and it was the start of the harder half of the two days.

Horde centrally manages and force-upgrades agents from the server, so a hand-patched binary on the box just gets overwritten. To make a patch stick you have to replace the agent software the server distributes. That meant, in order:

  • DeployTool to push a new horde-agent build → blocked: “Cannot update the state of bundled tools.” The agent ships as a bundled tool baked into the server image.
  • So un-bundle it: mount a custom appsettings.json with horde-agent removed from BundledTools, and add it as a regular tool in globals.json.
  • Now it’s deployable but → “User does not have DownloadTool entitlement.” Bundled tools were implicitly public; regular ones aren’t. Add "public": true.
  • Agent downloads it, then infinite upgrade loop, the patched build reported Version: unknown (forgot -p:InformationalVersion), never matched the deployment version, so the server kept re-issuing the upgrade.
  • Fix the version, and the upgrade fails to launch: framework-dependent build, and the box’s system dotnet had .NET 8/9 but not 10. Discover the upgrade mechanism supports self-contained builds (runs the .exe directly, no system runtime), but transitioning a framework-dependent agent to self-contained needs a one-time manual bootstrap.
  • Self-contained agent finally runs, takes a job, and… JobDriver.exe, FILE_NOT_FOUND. The publish built HordeAgent but not the separate JobDriver program the agent shells out to. Build that too.

After all of that, the agent was healthy, self-contained, version-stamped, taking jobs, and the upload still timed out at 60 seconds.

The actual bug: I was patching the wrong process

The stack trace I should have read on hour one:

D:\HordeAgent\MH-Main-Inc\Sync\Engine\Source\Programs\AutomationTool\...\CreateArtifactTask.cs:259
D:\HordeAgent\MH-Main-Inc\Sync\Engine\Source\Programs\Shared\EpicGames.Horde\Storage\Backends\HttpStorageBackend.cs:302

The upload doesn’t run in the Horde agent at all. It runs in AutomationTool, BuildGraph’s CreateArtifact task, using the EpicGames.Horde code from the agent’s synced workspace (//UE5/main). The agent is just the host that launches the build. Every patch I’d made to the agent binary was irrelevant; the timeout that was firing lived in the engine source the agent syncs and compiles, which still had the stock 60.

And there were two timeouts in that path, not one:

  • Polly per-attempt timeout, 60s, HordeHttpMessageHandler.cs (“timed out after 60s”).
  • HttpClient.Timeout, the .NET default 100s wall-clock, the storage client is registered in HordeExtensions.cs with no timeout override, while the main client gets 210s. This is the hard ceiling that ultimately aborts the socket (SocketException 10053, “aborted by host”).

The solution

Patch both timeouts in the engine source, in //UE5/main, so every agent gets them by normal sync:

// HordeHttpMessageHandler.cs
Policy.TimeoutAsync<HttpResponseMessage>(600, OnTimeoutAsync);

// HordeExtensions.cs, give the storage client a real wall-clock budget
serviceCollection.AddHttpClient(HordeHttpClient.StorageHttpClientName)
    .ConfigureHttpClient(client => client.Timeout = TimeSpan.FromSeconds(600))
    .AddPolicyHandler(HordeHttpMessageHandler.CreateDefaultTransientErrorPolicy());

Submit to Perforce, let the agent re-sync, with one last gotcha: if you committed prebuilt Engine/Binaries/DotNET to the depot, RunUAT may load the stale EpicGames.Horde.dll instead of recompiling your patched source. Delete the cached AutomationTool binaries in the agent’s workspace so it rebuilds from source.

This is strictly better than the agent patch, too: it’s versioned in Perforce, travels to every agent via sync, survives the agent auto-upgrade, and means the entire custom-agent detour was unnecessary.

The green build

With the timeouts patched in the engine source and the agent forced to recompile from it, the next CI run went the distance: editor compiled, CreateArtifact pushed all ~685 blobs to the server without a single cancellation, the artifact completed, and the default ref finally got written. The payoff: a dev opens UnrealGameSync, points it at the Horde server, and syncs the precompiled editor instead of building it. Which was the entire point, four lines of timeout ago.

Update: one of those two fixes never fired

A lot of green builds later, the upload died again. Same SocketException 10053, “aborted by the software in your host machine.” But this time the deadline was 100 seconds, not 60, and the stack trace was completely different. No engine change, nothing on our side touched. A network blip had simply pushed one blob’s upload past a limit that, it turned out, we’d never actually moved.

Here’s the part worth owning: the tidy version above, “two timeouts, patched both, done,” was wrong. Those are two different deadlines. The 60-second deaths were the Polly per-attempt timeout, and patching that was real. The 100-second ceiling is the wall-clock HttpClient.Timeout on the storage client, and the change we made for that one, in HordeExtensions.cs, never took effect:

serviceCollection.AddHttpClient(HordeHttpClient.StorageHttpClientName)
    .ConfigureHttpClient(client => client.Timeout = TimeSpan.FromSeconds(600));

That configures the client through dependency injection, but the registration doesn’t reach the IHttpClientFactory that AutomationTool’s HttpStorageBackend mints its client from. The proof is in Epic’s own code: for the storage backend’s upload-redirect client, they don’t trust the registration either, they set the timeout inline, with a comment saying the 100s default is too low. Our self-hosted server doesn’t issue upload redirects, so every blob takes the direct path, through a storage client whose timeout was silently still the 100-second default. It went green the first time only because no single blob had ever taken longer than 100 seconds. The first slow moment that did, exposed it.

The fix that actually holds sets the timeout where the client is created, not where it’s registered, in HttpStorageBackend.cs:

HttpClient CreateClient()
{
    HttpClient httpClient = _httpClientFactory.CreateClient(HordeHttpClient.StorageHttpClientName);
    // Inline; the DI ConfigureHttpClient registration doesn't reach this factory.
    httpClient.Timeout = TimeSpan.FromSeconds(600);
    // ...
}

The better mental model, and the thing that would have saved the second round entirely: an upload like this isn’t defended by a timeout. It’s a stack of independent deadlines, a per-attempt retry timeout and a per-client wall-clock, on more than one client, and closing some of them looks exactly like closing all of them, right up until traffic finds the one you missed.

Lessons

  • Read the stack trace first. The fully-qualified path (...\AutomationTool\... vs ...\HordeAgent\...) told me on day one that the upload wasn’t the agent’s code. I didn’t look until day two.
  • Rule out the boring causes before the clever ones. I theorized HTTP/2 flow control before I’d confirmed the network (scp) or the load (docker stats). Both were trivially fine. The “clever” fixes (resize, drop Caddy) cost the most and helped the least.
  • “It’s slow” almost always means a fixed deadline vs. a variable duration, not a bandwidth wall. 40-way parallelism over a finite uplink + a 60s timeout = death, even at 100 Mbit.
  • Know where your code actually runs. In Horde, the agent orchestrates but the build (and its artifact upload) executes in AutomationTool from the synced engine. Patch the thing that runs, not the thing that launches it.
  • Centrally-managed agents fight local patches. If you must customize a Horde agent, do it through the tool-deployment path, but first ask whether the change belongs in the engine instead. Ours did.
  • A fix you watched succeed once isn’t a fix you’ve confirmed. Half of ours never fired, and a green build hid it for weeks because the traffic never happened to need it. Config applied through a registration you haven’t proven reaches the right place is a wish, not a fix.
  • There is rarely just one timeout. A big transfer is guarded by a stack of independent deadlines. Closing some of them looks exactly like closing all of them, until the one you missed gets hit.

Two days for the first fix. It took a network blip, a long time later, to show that half of it never fired. The honest version is messier than the tidy one, and a lot more useful.