In our latest video, Aaryn Flynn and the team from Improbable Edmonton take us through how SpatialOS-enabled rapid iteration has helped speed up their development.
In our latest livestream, our team from Improbable Edmonton tackled another crucial problem of multiplayer game development - testing and iterating your online game. If you missed our previous two Livestreams, we looked at an overview of the latest version of SpatialOS for Unreal 2019.1 and dived deep into computational offloading with our partners at Midwinter Entertainment.
Improbable founded our Edmonton office in September 2018. The team of 60 developers are mainly ex-Bioware staff, headed up by Aaryn Flynn, ex-General Manager of Bioware. They’re building an online RPG using Unreal Engine 4 and the SpatialOS for Unreal GDK, with the aim of pushing the technical boundaries of modern games.
“We committed to using the platform very early in the RPG’s development and we’re really happy with what we’re doing every day.” said Flynn. “Our game really benefits from fast iteration, simulated players and all the SpatialOS tools. It’s a really good environment to build an online RPG in - it’s a very risky title to make, that’s the nature of the industry these days, but it really helps to have a platform like SpatialOS.”
Our aim with the SpatialOS for Unreal 2019.1 update was to make local iteration with SpatialOS for Unreal as fast or faster than native Unreal throughout our workflow. But online iteration times for online games themselves are typically very slow - it might be a few weeks between playtests with new code and you might not be able to isolate problems or keep servers up in the early days of a project.
We aim to change that with SpatialOS. The aim is not only to make that iteration loop faster, to the level that you can test new code multiple times a day - but also to make it more effective, so you learn what works and what doesn’t, faster. This lets you address bugs and balance issues quickly.
Along with the knowledge that you’re validating every change in your final live game environment, this all means that you can have much more confidence about your design, stability and player experience when it comes to launch.
The iteration times here... I’ve never experienced anything like it.Bjorn Taylor, Senior Technical Designer, Improbable Edmonton
The first step in fast iteration is the SpatialOS launcher - a tool designed explicitly for quickly playtesting your game. Rather than running code around your office on a USB stick or, worse, sending it over insecure networks, this enables you to generate a single shareable link to play your game deployment to any player (for example, Bluedrake42 shared his deployment with 75 people). (You can limit access to users in your organisation or make it publicly accessible.) If you want to quickly test a new feature or scale up your game, this enables you to do that.
For Improbable Edmonton, their previous experience of multiplayer testing was often painful - huge projects where getting two people on the same build and connected might take days or weeks.
Yet the launcher has allowed QA to run playtests literally every day. “Something that can be challenging is co-ordinating people and making sure that everyone is able to run the latest game build.” said Deanna Dombroski, QA Gameplay Analyst at Improbable Edmonton. “So being able to send a link and they click two buttons and they’re in? It means we can focus more on the content of what we’re testing, or testing multiple times with different versions.”
“I see the development team working on multiple builds at the same time.” reports Edmonton GM, Aaryn Flynn. “Different A/B tests are happening, different experiments, and all that’s completely disambiguated for the development team - it’s just supersmooth functionality that lets them do exactly what they want to do.”
As the name implies, these are our systems that store all your logs and metrics data. They help with fast iteration by allowing developers to identify problems in online tests, both live and after the event.
The metrics dashboards show runtime, operations, performance and bandwidth analytics for your game from the very first cloud test you do. These are useful for live cloud deployments, where you might want to see player numbers, latency, CPU memory usage, the server FPS and so on, and to compare telemetry like this across different builds over time.
The Logs dashboard is where you’ll find all your messages, warnings, errors and debugging data, which is useful for local development. Both of these are also set up so you can hook in your own logging and metrics solutions, such as OpsGenie, if you prefer - allowing you to prepare for pre-production at the point of prototype.
Bjorn Taylor, Edmonton’s Senior Technical Designer, has found pairing the logs and metrics incredibly useful: “We’ve already done lots of performance optimizations, because when bad things are happening, we can see the logs and metrics, work out what was happening, and go fix in and the problem.”
“In the old days, multiplayer build tests would take days or even weeks to set up, which puts so much pressure on the development team to make it count they heavily instrument the game and spend weeks studying the logs and metrics.” said Flynn. “It’s like sending a rocket to the moon, you’re over-prepared because you only have one shot. Now, we get to go around and around, experiment and test, try ambitious things because we know we can do it quickly.”
The Inspector provides a real-time top down view of everything in your game world. It’s browser-based, not tied to your engine, and is automatically included as part of every cloud deployment. You can choose to assign colours and icons automatically to different Actor / entity types, which then appear in the overview panel. It’s a key tool in fast iteration as it gives you instant visual feedback on what’s happening in your world - and enables you to change it on the spot.
On the side panel of the Inspector are two lists, one detailing servers / workers and one listing Actors / entities. Flynn is surprisingly enthusiastic about this panel: “You can’t under-appreciate that all the data your game is currently chewing on is available to you, in the panes on the right there. Everything and anything on your playtest objects and entities is there, in real time, which just accelerates on the overall experience - it’s invaluable.”
The Inspector is a powerful tool, not only for developers and programmers, but also for game designers, level designers and QA. They can use it for monitoring to help with debugging and analysis - looking at the metrics of their server-workers, for example, or seeing which server owns which entity or entity types. “We use it as a high-level view to see what people are doing when they’re playing the game and to get a sense of the general state of the game. We can spot problems that we might have seen months too late without this tool.” said Dombroski.
For Improbable Edmonton, using the Inspector like this is already paying dividends. “I love this thing for AI. I spend as much time watching AIs in this in our playtests as I do running around the game…” said Taylor. He’s particularly found it useful to find quirks where the game isn’t broken but is just being weird - hard to find in a normal game world without massed QA. “For example, we had a random number generator issue that was causing all the AIs to randomly wander towards the right hand side of our map. Finding that in-game is very tricky, but in the Inspector it was immediately obvious.”
Daily testing is great, if you can do it. To help with that, our simulated players are fully scriptable “fake players” that support player-based actions, to whatever level of complexity, which run online or offline. “These give us the ability to test and stress other aspects of the game beyond just the compute in the backend.” says Flynn.
Importantly, as these players run as separate clients, they can check your online systems end to end, from log-in to authentication, matchmaking logic, lobby, meta-inventory access, external services like PlayFab, and finally into the game itself. Checking all these systems work before you even get into production is incredibly useful, reducing the risk of your launch.
Simulated players are also helpful when you want to test your game scenarios at scale or tune performance without necessarily resourcing hundreds or thousands of players, or tying up valuable members of staff in daily playtests. “In Edmonton, we’re about sixty staff,” says Taylor, “so it’s hard to get a tonne of players, but we can simulate a bunch more and test things like server stability so that, when we do a larger test, we can be sure that things won’t collapse.”
Indeed, the QA team are looking to simulated players to offload their workload, according to Dombroski: “QA has really monotonous manual tests that get boring very quickly, but need to be done daily and done well - like running into walls to see if you can get through them by accident. We’re freed up to do in-depth testing or planning, more interesting and meaningful tasks, thanks to our little robot friends handling it.”
Another helpful element is chaos testing. This is where you put a large number of sim players into the server and watch the Inspector to find the edge cases that humans can’t see easily, even with a large beta test. As simulated players don’t sleep, this allows the Edmonton office to maintain their work-life balance and see the results of stress or chaos tests in the morning.
The Edmonton team will reveal more about their game in the near future. In the meantime, our next livestream will be arriving soon, and the focus is on matchmaking services when using SpatialOS for Unreal. This livestream is ideal for current SpatialOS developers who want to improve team efficiency and focus more on gameplay.
Get started: easier launch with matchmaking services
September 26, 2019, 6PM BST / 10AM PST