(Interview performed in March 2015 for Seekscale company, a cloud rendering startup)
We are regularly looking for interesting technologies, and we recently came across Umbra, a middleware initially designed for occlusion optimization, which turns out to be evolving in a rendering optimization platform, with content streaming and cloud delocalization available, where geometry topology is analyzed in the cloud, assets are streamed from the cloud and rendered locally. Seekscale interviewed Jasin Bushnaief, Head of Product at Umbra.
– First of all please, let’s make sure I got everything right. Umbra is initially an occlusion engine (figures out what surfaces are visible in a given scene), available as standalone runtime, or Unity and Unreal Engine plugins. Recently, Umbra expanded into rendering optimization, content streaming, and various game logics. Is that right?
Essentially, yes. Umbra is a tool which takes an arbitrary set of 3D geometry as its input and creates a highly-optimized spatial database. Umbra’s runtime can then be used to issue various queries on this database. Occlusion culling / visibility determination (“Given this camera transformation, which of my objects are visible?”) is of course the most important runtime query we have, but there are others. Conservative ray casts, topological distances between points and connected regions within a radius can also be queried using the same data.
So in addition to rendering, many of our clients have used Umbra for audio and AI, for example. Asset streaming based on a combination of visibility, connectivity and distance is of course a very important use case for us, too. Umbra is included as a built-in component in Unity, i.e. Unity’s occlusion culling solution is in fact Umbra under the hood. There’s a plugin for Unreal Engine and of course a standalone SDK for custom engines.
– Can you elaborate on the rendering part? If I’m writing a video game, as far as real-time rendering is concerned what can I expect from Umbra middleware?
The most important feature for rendering acceleration is Umbra’s visibility query. In the simplest case, you give Umbra your camera transformation and Umbra will return you a list of static visible objects. This list is always conservative, meaning that it will return some amount of extra visible objects, but never report objects as hidden when in reality they should be visible. This means that you never get culling artifacts such as objects popping in/out when moving quickly around a corner, which is typical to many GPU-based solution. In addition to a list of static visible objects, Umbra also produces a conservative depth buffer against which you can easily test the visibility of dynamic objects.
All this happens completely on the CPU, so integration is typically very easy compared to that of some GPU solutions. The code is of course very fast as well, so typically a visibility query takes no more than a couple of milliseconds. Expanding on that, Umbra can be used to speed up shadow mapping, as Umbra can be used to cull shadow casters that would cast occluded shadows. And of course 3D content streaming.
– It all feels like you have figured out a pretty interesting data representation that turns out to be useful for a lot of other things than occlusion, correct?
Yes. Turns out, the database actually contains a spatial representation of the input geometry. We know what kinds of areas the world consists of and what the connections are between them. This makes it easy to do all kinds of reasoning about space. For instance, we did a talk in this year’s GDC about how Remedy uses Umbra for audio in Quantum Break. AI logic and asset streaming are examples of other use cases that are not strictly in the scope of rendering.
– What are the biggest technical challenges of 3D content streaming? How can you achieve sufficiently low latency?
I believe the biggest challenge is of course in retaining the maximal quality of assets while minimizing the amount of data that needs to be transferred. The latency becomes less of a problem when you don’t have to transfer assets every frame, let alone rendered frames, and of course when you are very smart about what it is actually that you transfer, how you cache and compress it and so forth. I’d say streaming 3D assets and rendering them locally is a lot less latency-sensitive than streaming rendered frames. This makes Umbra’s approach to 3D streaming pretty unique and very feasible.
– About the new Umbra cloud offering, what is done in the cloud exactly? The generation of the data representation from the scene?
Currently, the creation of Umbra’s spatial database is what happens in the cloud. Eventually, we will expose more of the asset streaming features too.
– What makes this computation viable for cloud delocalization? Is it compute intensive and not data-intensive?
The creation of the Umbra database is a perfect fit for the cloud for a lot of reasons. While there naturally is some amount of input data required, the processing is very computation intensive and pretty much embarrassingly parallel. As such, the flexibility of scaling the computation power in the cloud means that the data can be always efficiently generated regardless of the accuracy required, for both small and large worlds alike. On a longer term, we believe the assets themselves will be moving into the cloud anyway, which makes this design even more feasible.
– How can you make sure back and forth communications are not a bottleneck for the real-time performance?
Of course if we’re talking about merely creating the Umbra database in the cloud, the runtime part is still running locally, so communication is trivially simple. In the case of 3D asset streaming, it’s all about being able to smartly predict what needs streaming. Because we know the topology of the world, we can make highly sophisticated predictions of which assets are relevant over a period of time. Furthermore, by leveraging visibility information, we’re able to significantly reduce the amount of data that needs to be transferred.
– What is the business model behind the cloud feature? Is it “all you can use” for the developpers? Can a studio pay you so that their players can leverage your cloud? (if the game uses Umbra of course)
One of the key points why we decided to start looking into a cloud offering in the first place was to be able to offer something affordable like a subscription model for developers, especially for ones who don’t see an upfront license fee as feasible. While we have not yet set the pricing model in stone, the idea is to charge based on the level of usage – so you’d pay less if you use less Umbra processing. This makes the pricing scalable too, which is nice for both us and our clients, especially when considering a game where the players would leverage Umbra. And almost certainly we’ll offer a free tier for developers to try it out and get started.
– Is Umbra middleware relevant for game engines based on procedural generation? (procedural engines must manage geometry and visibility issues differently)
It definitely is now especially with Umbra Cloud. Even previously, the problem with procedural content wasn’t so much the runtime part which already works on any platform and with any content, but rather the database creation part. But now, there’s no reason why the game couldn’t have an Umbra Cloud integration even if the content was generated on the fly.
– What is your r&d roadmap for the next few years?
Without going into too much detail, one clear thing is to develop our tech into something that enables the rendering of literally any 3D content on any platform. Arbitrary 3D content automatically tailored for any specified budget, basically. Supporting other kinds of input than just triangles, such as point clouds, is also a very interesting aspect of this vision. One theme we also have is to further advance the usage of our data in other real-time contexts beyond just rendering – audio, AI and so on. Finally, making our tech easier to use and easily available to everyone something we’re very passionate about.
– According to you what are the most important levers right now to increase realism of real-time rendering?
How to render realistic graphics is well understood, but what still needs a lot of improvement and automation are tools that make it easy to create compelling content – even when the user does not have years and years of experience.
Tech from the Front Line 