Disclaimer: the knowledge contained in this post was largely gained by spelunking through the source code of UnrealTournament, which is available on GitHub for all holders of an Epic Games account. I hope this article will save other people much of the pain of learning to use these (almost entirely undocumented) capabilities of UE4.

Intro

In this blog post I will give a high-level overview how to cook and package Unreal assets such as blueprints and meshes for loading at runtime. This is necessary if you ever want to:

  1. Publish modular Downloadable Content (DLC) for your game
  2. Allow users to create and share new content
  3. Save having to rebuild and redistribute your entire game every time you want to add new content

I will focus on the second requirement, creating and sharing new content, as it’s what first motivated me to investigate this. I may later write a follow-up article on how to write an editor plugin that largely automates this process. This post will be light on specific implementation details, but I may provide example code in a future post.

Contrary to popular opinion, it is entirely possible to build a working DLC / modding system without modifying the engine source. A small amount of C++ programming is required, but if there’s enough interest I will try to release a code plugin for use in Blueprints.

There are two main topics that are important to understand in order to achieve this: the Content Pipeline and the Asset Registry.

The Content Pipeline

The Unreal Editor is, at the time of writing, generally set up to only allow monolithic builds of your game. The encouraged workflow is to build all your content into a single application that you ship to your users. But that’s not we want to do here – we want to be able to distribute additional content at a later date without having to do a full rebuild.

Deployment

So how can we do that? Well, first we must understand what is involved in that build process.

Overview

You probably already know the three steps1 that all content must take before it can be loaded and used by a game:

  1. Cooking - the process of preparing assets for use at runtime by stripping out editor data and optimising them for the chosen platform.
  2. Packaging - the process of storing the cooked assets into a .pak file, which allows bundles of related assets to be compressed and distributed as a single file.
  3. Loading - the process of actually loading the assets during the game’s runtime, for example to instantiate them in the level.

Luckily for us, the first two steps can be achieved very easily with the editor.

Cooking & Packaging

You may already be aware that Unreal has some support for creating extra “DLC” (Downloadable Content) by hijacking the plugin system, as detailed by Tom Looman in his great tutorial on creating a modding system.

You should read it, but to summarise, the editor can be told to distinguish between a core “full game” build and a build of DLC content by simply ticking a couple of boxes in the launch profile. The produced DLC .pak file can then simply be placed in the runtime application’s directory and it will be automagically loaded on startup. Hooray, problem solved… right?

Unfortunately what is not at all obvious to the casual reader is that this alone is, to a large extent, totally useless. The reason for this is simple: if the core build is not executed with the DLC plugin present in the Plugins folder, the content within the produced .pak will not be accessible at runtime. Clearly this negates the whole point of being able to produce extra DLC content, if it must be known about at the time of the full release.

So why is this case? Why can’t we access the DLC content at runtime? To understand that, we must first understand the Asset Registry.

The Asset Registry

The Asset Registry is essentially a central index of all assets available to the game. That means blueprints, meshes, widgets – anything that you would want to load.

The advantage of having this central index is that it allows the game to quickly search for assets that satisfy some criteria. In the case of UnrealTournament, for example, it is used to populate the drop down menu of available hats. It also provides functionality for finding dependencies, which is important because a blueprint might reference a mesh, which references a material, which references a texture… and so on.

Every running instance of the game has a singleton Asset Registry sitting in memory, which is loaded from a AssetRegistry.bin file that is distributed with the content. This file is an artefact of the cook process – it lets the game know what content was cooked and where in the Unreal Filesystem (UFS) it can be found. If you’ve tried cooking DLC in the editor, you’ll know that the way it works is by taking the difference between the release AssetRegistry of the “release build” and the current one, therefore only cooking and packaging the additional content.

Remember that our aim is to allow multiple packages of content to be loaded; that means multiple cooks, which in turn means multiple Asset Registry files.

Clearly it is necessary to merge the contents of all these separate files into the runtime Asset Registry to allow the game to find all the content. It turns out that this is the only part of the process which the engine won’t do for you. The best way I’ve found to do this is simply to iterate over all the .bin files found at runtime and call IAssetRegistry::Serialize() on them2.

Here’s a diagram showing how that works:

Merging multiple asset registry files into one at runtime.

Note that in reality you’ll want all your .bin files to have different names to avoid naming clashes when you load them. The convention UnrealTournament uses is to simply add the name of the containing .pak file as a prefix. The advantage of this is that it allows you to search the content folder for all .pak files and then load the corresponding <PakName>-AssetRegistry.bin file, sidestepping the chicken-egg problem of knowing where to find them.

You’ll probably want to reserve AssetRegistry.bin for all your core release content in any case, as this is the special case that is merged in automatically.

Loading at Runtime

I mentioned before that your distributed game will automatically load any .pak files in <PathToGame>/Content/Paks. As in many other game engines, the Unreal Filesystem (UFS) is actually an abstraction over the system filesystem for the sake of portability. It’s important to know about this because otherwise it can be very confusing to see paths like /Game/MyContentFolder/Asset that don’t seem to correspond to files in the Windows (or other system) filesystem.

The interaction between .pak files and the UFS is called mounting. Much like an OS might mount a volume, when the game starts it will mount each .pak in the Content/Paks folder (and all subfolders). This maps the location of the assets within to the UFS, allowing them to be loaded.

When the .pak file is created using the UnrealPAK utility, the mount path of each file is specified. This is then used when the .pak file is loaded to determine where the file should be mapped to in the UFS. It is vitally important that the asset paths in an AssetRegistry.bin file match the mount paths used to create the .pak file.

The general convention seems to be that a file in <PathToGame>/Content/MyFolder/MyFile.uasset is mapped to /Game/MyFolder/MyFile.uasset in the UFS.

Assuming you’ve merged in the AssetRegistry.bin from your .pak file and the mount paths were correct, you should be able to load the assets within just as you normally would, e.g. using the SpawnActor blueprint node.

Conclusion

This was a whistle-stop tour of getting Unreal Engine 4 to work with modularised content. Let me know your feedback on the forum thread.

Footnotes

  1. Note that I am ignoring for now the Editor import process to turn source files (e.g. FBX models) into usable Unreal assets, as I have yet to find a way to automate this easily. 

  2. Be aware that this is just a simple merge and may add duplicate or conflicting entries to the Asset Registry.