21 July 2026

Setting up Horde's symbol server

Horde ships a Windows symbol server that indexes PDBs out of your build artifacts, so there's no separate symbol store to feed or clean up. Here's the server config, the BuildGraph side, how to verify it end to end, and the traps worth knowing before you start.

One of the nicer things I’ve found in Horde, and one I nearly missed, is that it has a built-in Windows symbol server. It’s compatible with Visual Studio, WinDbg and Microsoft’s DIA library, so once it’s set up you can open a crash dump from a packaged build and pull down the matching PDBs.

This follows on from the other two Horde posts, on running CI on a single build machine and uploading to Steam from BuildGraph. You don’t need either to make sense of this one.

What it actually is

The bit that made me stop and re-read the docs: symbols aren’t stored separately. You upload PDBs as part of a normal Horde artifact, and internally Horde adds aliases in its storage that map symbol-server hashes to files inside that artifact.

Two things fall out of that, and both are good:

  • No extra storage cost. The symbols live in an artifact you were uploading anyway. You’re not maintaining a parallel symbol store that quietly grows forever.
  • Retention comes for free. Symbol lifetime is tied to the artifact, so whatever keepDays / keepCount rules you already have govern them too. There’s no separate cleanup job to write and forget about.

Worth setting expectations: this is a symbol server, not crash reporting. Horde has no system for collecting crash reports from players, and its “issues” feature groups CI build failures rather than runtime crashes. What you get is the ability to symbolicate a dump you already have.

Configuring the store

Configuration lives in globals.json under the symbols plugin. A store indexes an entire storage namespace:

"plugins": {
  "symbols": {
    "stores": [
      { "id": "default", "namespaceId": "horde-artifacts", "public": false }
    ]
  }
}

The config type only has four fields: Id, NamespaceId, Public and an Acl. Since a store indexes a whole namespace, one store covers every project whose artifacts land there, which is why I’ve got a single default rather than one per game. horde-artifacts is the namespace artifacts go to by default, so unless you’ve changed that, this lines up on its own.

Drop the config on the server and Horde picks it up, you should see Configuration updated (success) in the log. Check the plugin is loaded too, there’ll be a Loading /app/HordeServer.Symbols.dll line at startup.

A quick way to confirm the store registered, without any debugger involved: request a symbol that doesn’t exist and watch the status code.

curl -o /dev/null -w "%{http_code}\n" \
  https://your-horde-server/api/v1/symbols/default/nosuch.pdb/1234567890ABCDEF1234567890ABCDEF1/nosuch.pdb

A 404 means the store id isn’t registered. A 403 means it is, and you’re just not authorised yet. That difference is genuinely useful when you’re setting this up.

Retention

Because symbols ride inside an artifact, retention is just an artifact type. In the stream config:

"artifactTypes": [
  { "type": "packaged-build", "keepDays": 14 },
  { "type": "ugs-pcb",        "keepCount": 5 },
  { "type": "symbols",        "keepDays": 30 }
]

I’ve given symbols a longer window than the builds themselves. A crash report tends to arrive well after the build that caused it, and there’s no point keeping a dump you can no longer symbolicate.

Publishing symbols from a build

On the BuildGraph side there’s one attribute that matters: Symbols="true" on CreateArtifact. That’s what tells Horde to index the files for the symbol server rather than just store them.

Two things need to happen first. PDBs have to exist, and you have to tag them.

Unreal strips debug info when you pass -nodebuginfo to BuildCookRun, which most packaging scripts do by default. I made that conditional so I can build with symbols when it matters and skip the cost when it doesn’t:

<Option Name="WithSymbols" Restrict="true|false" DefaultValue="false"
        Description="Build with debug symbols (PDBs) and publish them to Horde's symbol store"/>

<!-- PDBs are stripped by default; keep them when we want symbols -->
<Property Name="DebugInfoArg" Value="-nodebuginfo"/>
<Property Name="DebugInfoArg" Value="" If="$(WithSymbols)"/>

Then $(DebugInfoArg) goes into the BuildCookRun arguments in place of the hardcoded flag, and the upload hangs off the same option:

<Do If="$(WithSymbols) and $(IsHordeEnv)">
  <Tag Files="$(StageRoot)/..." Filter="*.pdb" With="#Symbols"/>
  <CreateArtifact Name="symbols" Type="symbols" Symbols="true"
                  Description="$(GameTarget) $(Configuration) symbols CL $(Change)"
                  BaseDir="$(StageRelativeDir)" Files="#Symbols"/>
</Do>

The IsHordeEnv guard is so local runs of the same graph don’t try to upload. It’s the usual trick of checking whether UE_HORDE_JOBID is set:

<EnvVar Name="UE_HORDE_JOBID"/>
<Property Name="IsHordeEnv" Value="false"/>
<Property Name="IsHordeEnv" Value="true" If="'$(UE_HORDE_JOBID)' != ''"/>

If you’re wiring this into a Horde template, exposing the option as a checkbox saves anyone having to remember the -set: syntax:

{
  "type": "Bool",
  "label": "Build with symbols (publish PDBs to Horde symbol server)",
  "argumentIfEnabled": "-set:WithSymbols=true",
  "default": false
}

Two things not to forget

Don’t ship the PDBs. Building with symbols means they’re sitting in your staged output, and it’d be easy to package them into a store build by accident. Our Steam depot config excludes them explicitly:

"FileExclusion"  "*.pdb"

So the PDBs go up to Horde and nowhere near a player.

Make the build traceable. Symbols are useless if you can’t work out which build a dump came from. Stamp the changelist into the binaries with SetVersion before you build, and put the CL somewhere you’ll see it later. For Steam builds I put it in the build description so it shows in the Steamworks build list:

"desc"  "WH Shipping CL 51 (Horde)"

That’s the difference between “a crash in the latest build” and a specific changelist you can go and look at.

Verifying it, without guessing

This is the part I’d skip to if I were setting this up again, because I wasted a lot of time assuming things worked.

1. Find out what the debugger will actually ask for. Every PE binary embeds the PDB it wants and a signature (GUID + age). A minidump records the same thing for every loaded module. That signature is the whole ballgame: get symbols whose signature doesn’t match and nothing resolves, with no useful error. dumpbin /headers will show you a binary’s debug directory, or you can read a dump’s module list directly.

2. Prove the PDB matches before blaming the server. Point a debugger at the PDB on local disk and see if you get a callstack:

cdb -z crash.dmp -y <folder containing the pdb> -i <folder containing the exe> -c "kn; q"

If that resolves, your build produced correct symbols and anything still broken is transport. If it doesn’t, the symbols and the binary aren’t from the same build and no amount of server config will save you.

3. Watch the actual requests. cdb/WinDbg will tell you exactly what URL they’re hitting if you ask:

!sym noisy
.reload /f YourGame.exe

You’ll get lines like:

SYMSRV:  HTTPGET: /YourGame.pdb/BFE86CCC7FE04728B40A679A5870DF8A1/YourGame.pdb

which you can compare against the store, and status codes when it fails. If you want to be certain what’s being requested, point the symbol path at a throwaway local HTTP listener that logs the path and returns 404. A request arriving tells you the debugger is behaving, and hands you the exact signature it wants.

Do clear your symbol cache before reading anything into silence, though. No request doesn’t mean the debugger is broken, it usually means it already has that PDB and didn’t need to ask. That distinction is worth about an hour if you get it wrong.

The traps

Don’t test on the build machine. This one cost me an afternoon. A PE binary embeds the absolute path of its PDB from the build, something like D:\HordeAgent\...\Binaries\Win64\YourGame.pdb. On the build agent that path exists, so the debugger loads the PDB from disk and never contacts the symbol server at all. Worse, on an incremental workspace that file gets overwritten by later builds, so you end up loading a mismatched PDB and getting an unsymbolicated stack with nothing explaining why. On a normal dev machine none of this happens. If you must test on the agent, rename the local PDB first.

The local symbol cache will lie to you, in two different ways. This one caught me twice and is the single biggest time-waster here.

First, if your symbol URL returns an HTML error page with a 200, symsrv treats the download as successful and writes it into your local cache under the correct-looking path. Every later attempt loads that garbage from disk. You’ll see file system or network error reading pdb, which doesn’t hint at the real problem at all.

Second, and more confusingly, a cache entry that’s already there means the debugger has no reason to request anything. I spent a while convinced a debugger wasn’t querying the symbol server at all, watching a listener show zero requests for the game module, when in fact it had a copy sitting in %LOCALAPPDATA%\Temp\SymbolCache the whole time. Deleting it, and the requests appeared immediately.

So when you’re testing this, clear the cached entry for your module first. Otherwise you can’t tell “not requesting” from “already has it”.

Use WinDbg or Visual Studio for dumps, not Rider. Rider’s symbol server support genuinely works: set it under Settings | Build, Execution, Deployment | Debugger | Symbol Servers, tick Enable symbol servers support, set a cache directory, and point your Native Core Dump Debug configuration at both the dump and the executable from that build. Do that and it requests the right symbol and downloads it correctly.

It just didn’t then symbolicate the dump for me. I confirmed the PDB it pulled was byte-for-byte complete and valid, and handed that exact file from Rider’s own cache to cdb, which resolved the callstack immediately. So the symbols were right and something further down Rider’s native path didn’t use them. Its debugger is LLDB-based, and LLDB’s PDB reader is a good deal weaker than DbgHelp, which is my best guess for a PDB this size.

Worth knowing before you spend an afternoon on it, as I did. Keep Rider for live debugging, reach for WinDbg or Visual Studio when you’ve got a dump.

Mind the size. A Development-config PDB for a decent-sized UE project is easily multiple gigabytes, ours is 2.8 GB. That’s per build, it’s what your retention window is holding, and it’s what every developer downloads the first time they symbolicate. It’ll also find any timeout in the path between you and the server. Think about whether you need symbols for every build or only the ones you’ll realistically debug.

Public or private, and the proxy you’ll need to patch

public: true puts the store on unauthenticated endpoints, which makes debugger setup trivial: point at https://your-horde-server/api/v1/symbols/default and you’re done.

We’ve kept ours private. Our server is internet-facing and PDBs give away a lot about your code, so publishing them isn’t a trade I’m willing to make for convenience. Horde’s answer for private stores is Unreal Toolbox, which runs a local proxy holding an authenticated connection and exposes it on localhost. Install it, enable the proxy plugin, and point your debugger at:

http://localhost:13344/api/v1/symbols/default

Note that path. The proxy is a general reverse-proxy to Horde rather than a symbol-specific endpoint, so it needs the full API path, not just the host and port.

Grant ReadSymbols on the store itself. The store’s ACL doesn’t inherit the global one, so nobody has the action until it’s granted there:

"acl": {
  "entries": [
    {
      "claim": { "type": "http://epicgames.com/ue/horde/role", "value": "admin" },
      "actions": [ "ReadSymbols" ]
    }
  ]
}

A quick way to tell whether authorisation is passing: request a symbol that doesn’t exist. 403 means you’re still denied, 404 means you’re through and it simply isn’t there.

Then, if your PDBs are large, you’ll need a patched proxy. The stock Toolbox proxy buffers an entire response into memory before forwarding any of it, and restarts its listener every time the access token refreshes. With Horde’s built-in accounts those tokens last 30 seconds, so anything bigger than a small file never completes. Ours is 2.8 GB, so it never stood a chance. That’s a whole story of its own, and the fix is two lines: patching Unreal Toolbox’s Horde proxy.

With a patched proxy in place, a private store works exactly as you’d want. Same 2.8 GB PDB, pulled through localhost, no public endpoint anywhere:

http=200  ttfb=0.41s  bytes=2,844,864,512  speed=74 MB/s  total=38.6s

and a minidump symbolicates against it.

That’s the setup. If you’ve found a tidier route to authenticated symbol access, or you’re running this against an OIDC provider and it all just works, I’d be glad to hear it.