Patching Unreal Toolbox's Horde proxy for large files

If you self-host Horde with a private symbol store, Unreal Toolbox is the sanctioned way for developers to reach it. It runs a small local proxy that holds an authenticated connection to Horde and exposes it on localhost, so Visual Studio or WinDbg can talk plain unauthenticated symsrv to http://localhost:13344.

It works fine for small files. It cannot serve a large one at all, and the way it fails gives you almost nothing to go on. This is what’s wrong and how to fix it. If you’re here for the symbol server itself, that’s a separate post.

The symptom

Request a multi-gigabyte PDB through the proxy and you get zero bytes for about thirty seconds, then a 500. Server-side, Horde logs an OperationCanceledException that looks like the client hung up. The same URL fetched directly from a logged-in browser downloads perfectly, so the server is fine and the file is fine.

The other symptom, if your token happens to be stale when you try, is an instant 403 complaining you don’t have ReadSymbols. Same URL, different second, different failure. That inconsistency is the tell.

Two bugs, both in HordeProxyPlugin.cs

Everything lives in Engine/Source/Programs/UnrealToolbox/Plugins/HordeProxy/HordeProxyPlugin.cs.

It buffers the entire response into memory

using HttpResponseMessage response = await httpClient.SendAsync(request, context.RequestAborted);
context.Response.StatusCode = (int)response.StatusCode;
...
using Stream stream = await response.Content.ReadAsStreamAsync(context.RequestAborted);
await stream.CopyToAsync(context.Response.Body, context.RequestAborted);

That reads like streaming, and it isn’t. HttpClient.SendAsync without HttpCompletionOption.ResponseHeadersRead defaults to ResponseContentRead, which doesn’t return until the whole body has been read into memory. The CopyToAsync underneath it only starts once the entire file has already been pulled down and buffered.

So for a 2.8 GB PDB the proxy sits there accumulating 2.8 GB of RAM while sending the debugger precisely nothing. That’s your thirty seconds of zero bytes: not a slow transfer, no transfer.

It restarts the whole listener on any state change

public HordeProxyPlugin(IHordeClientProvider hordeClientProvider)
{
    _hordeClientProvider.OnStateChanged += OnStateChanged;
    ...
}

void OnStateChanged()
{
    _restartServerEvent.Pulse();
}

and in the server loop:

Task stoppedTask = _restartServerEvent.Task.ContinueWith(_ => cancellationSource.Cancel(), ...);
await app.RunAsync(cancellationSource.Token);

A pulse cancels the token that Kestrel is running on, so the entire proxy shuts down and restarts, dropping every connection in flight.

An access token refresh is a client state change. And this is the part that turns an inefficiency into a hard ceiling: Horde’s built-in account system issues access tokens with a hardcoded thirty second lifetime, in OAuthController:

response.ExpiresIn = 30;
response.AccessToken = CreateAccessToken(globals, account, response.ExpiresIn.Value, nonce);
response.RefreshTokenExpiresIn = 7 * 24 * 60 * 60;

Thirty seconds, not configurable, with a seven day refresh token behind it. So Toolbox refreshes roughly every thirty seconds, and every refresh tears down any transfer in progress. Nothing that takes longer than half a minute can ever complete through the proxy.

Why this hasn’t bitten Epic

That controller is gated:

if (Settings.Value.AuthMethod != AuthMethod.Horde)
    => "This endpoint is only available when authentication method is set to 'Horde'."

The auth methods are Anonymous, Okta, OpenIdConnect and Horde, and the docs describe the last one as the option “if you are a smaller studio”. There’s a dedicated Okta value sitting next to the generic OIDC one, which tells you what Epic run internally.

With an external identity provider, it sets token lifetime, typically an hour or more. The refresh cadence that trips the restart simply never happens. And on a fast internal network, buffering a PDB reads as a pause rather than a failure. You need the specific combination of built-in accounts, the proxy, and a genuinely large file over the internet before any of this surfaces.

The fix

Both are one-liners.

// Stream the response instead of buffering the whole body first
using HttpResponseMessage response = await httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, context.RequestAborted);

void OnStateChanged()
{
    // Deliberately does not restart the server. Client state changes include access
    // token refreshes, and with Horde's built-in accounts those last 30 seconds, so
    // restarting here cancels every in-flight request on that cadence. Each forwarded
    // request resolves a fresh IHordeClient anyway, so a refreshed token is picked up
    // without recycling the listener. Genuine settings changes still restart it via
    // UpdateSettings().
}

Dropping the restart is safe because ForwardRequestAsync resolves IHordeClient per request, so a refreshed token gets picked up on the next request regardless. The listener never needed recycling for that.

Before and after, same file, same machine:

And a minidump symbolicates through it against a private store, which was the entire point.

Deploying it to your team

Building it is ordinary, it’s a normal .NET app:

dotnet publish Engine/Source/Programs/UnrealToolbox/UnrealToolbox.csproj -c Release -o C:\UnrealToolboxPatched

Getting it to everyone is the same dance as deploying a patched Horde agent. Toolbox self-updates from a Horde tool called unreal-toolbox, taking the newest deployment:

static ToolId ToolId { get; } = new ToolId("unreal-toolbox");
IToolDeployment deployment = tool.Deployments[^1];

That tool ships as a bundled tool, so you can’t deploy over it. Un-bundle it first: remove the unreal-toolbox entry from Horde:Plugins:Tools:BundledTools in appsettings.json, declare it in globals.json, and restart the server (appsettings.json isn’t hot-reloaded the way the config files are).

"tools": {
  "tools": [
    { "id": "unreal-toolbox", "name": "Unreal Toolbox", "namespaceId": "horde-tools", "public": true }
  ]
}

Then push a build with BuildGraph’s DeployTool task, with Duration="0" to roll out immediately rather than phasing:

<DeployTool Id="unreal-toolbox" Version="$(Version)" Directory="$(ToolboxDir)" Duration="0"/>

Run it with UE_HORDE_URL set. If the machine already has a cached Horde login it’ll authenticate off that, no token juggling required.

The version trap, which will get you

This one is worth the whole section, because it fails silently in both directions.

Toolbox decides whether to update like this:

version = version.Replace("-PF-", ".").Replace("-", ".");
return VersionNumber.TryParse(version, out versionNumber);

if (!TryParseVersion(_currentVersion, out currentVersionNumber)) return false;
if (!TryParseVersion(version, out latestVersionNumber)) return false;
return latestVersionNumber > currentVersionNumber;

Two consequences.

Your version label has to parse as a number. I first deployed mine as 5.8.0-proxyfix. Hyphens become dots, so that’s 5.8.0.proxyfix, which doesn’t parse, so TryParseVersion fails and no client ever selects it. The deploy reports success and reaches nobody. Use something like 5.8.0-54390006, numerically newer than whatever your clients are on.

The build needs a Version.json next to it. dotnet publish doesn’t produce one, and without it the About page says “No version information present”. That’s not cosmetic: an unparseable current version returns false before the code even looks at what’s available, so an unversioned build will never update to anything again. It’s stuck permanently, and nothing tells you.

// Version.json, alongside UnrealToolbox.exe
{ "Version": "5.8.0-54390006" }

If you’ve already put an unversioned build on a machine, dropping that file in and restarting is enough to unstick it.

Both of these are the same shape as the version stamping that bit me patching the Horde agent, where a build reporting unknown sat in an upgrade loop forever. Worth remembering that anything you deploy through Horde’s tool mechanism needs a real, parseable, ascending version.

Both proxy bugs are small and specific, so they’re worth reporting upstream rather than just carrying a local patch. If you’re running Toolbox’s proxy against a private store and it’s been working fine for you, I’d be interested to know what your auth setup looks like.