Unity’s Scripting Duality and Object Destruction

The Unity engine provides users with tools and abstractions that ease its usage and hide its complexity. Although we often take these commodities for granted and completely forget they exist, we often face a situation in which they become apparent, usually due to an unexpected behaviour. In this article, I discuss how an example of such an abstraction tool — Unity’s scripting solution — can accidentally expose the engine’s underlying mechanisms.

The duality: managed vs. native

As we all know, Unity’s programming language of choice is C#, but we need to keep in mind that the engine code itself isn’t written in C#, but in C/C++. Consequently, every code piece that invokes engine code (e.g. transform, GetComponent, gameObject.SetActive) does not run directly on the C# side, but on the native C++ side instead. An object of a type that inherits from UnityEngine.Object (like a MonoBehaviour) has two counterparts that live in different worlds: a managed object in C# world and an object in the native engine world. These two entities are linked to each other — the managed entity holds a pointer to the native entity — but are not, in fact, the same thing. Calls to the managed entity (often referred as a “wrapper”) will be transferred to the native entity whenever necessary: when a call to engine code is invoked. On the opposite side, wrapper calls that do not invoke engine code will not send to the native entity and will execute locally.

This might seem an unnecessary deep dive into the engine, but it brings some unexpected consequences. Take the example below. A Dog is a MonoBehaviour that does two simple things: Bark and Move. The first is a pure C# method that does not invoke engine code whereas the second moves the dog’s GameObject, invoking native engine code.

public class Dog : MonoBehaviour
    public void Bark()

    public void Move()
        transform.position += new Vector3(1f,1f,1f);

Now let’s test our Dog script with the help of the DogExample script below. This script creates a dog on Awake and can perform three actions based on user input: destroy the dog, make it bark and move. Simple as that.

public class DogExample : MonoBehaviour
    private Dog _dog;

    private void Awake()
        var dogGameObject = new GameObject("Dog");
        _dog = dogGameObject.AddComponent();

    private void Update()
        if (Input.GetKeyDown(KeyCode.D))
        if (Input.GetKeyDown(KeyCode.B))
        if (Input.GetKeyDown(KeyCode.M))

Following, let’s test it, creating a new empty scene with the above script attached to a GameObject. Once we play the game, a new GameObject named “Dog” is created, with a Dog script added to it. If we hit the B key, the dog barks (in Unity’s console). If we hit the M key, it moves (we can see its position in the inspector). Then, we hit the D key to destroy that Dog instance. In the editor hierarchy, we can still see the “Dog” object, but there is no Dog script attached to it. If we try to make the dog move, a MissingReferenceException is thrown, stating that “The object of type ‘Dog’ has been destroyed but you are still trying to access it”. The exception makes sense because we’ve just destroyed the Dog instance. The same thing happens if we decide to destroy the dog.gameObject instead (but then the “Dog” game object would be removed from the hierarchy).

However, if we press the B key to make the dog bark, no errors are thrown and the “Woof!” message is displayed on the console. What just happened? The Dog script was just destroyed and an error was thrown proving it. How can the dog still bark?

It is here where the managed vs. native duality explained in the previous session becomes useful. The two entities are different: a Dog script lives in the managed C# world and its underlying components (i.e. its GameObject and Transform) live in the native, C++ engine world. They are somewhat connected, but they are not the same thing. When the wrapper dog was destroyed, its script instance was removed from the “Dog” game object and its native components were wiped out, but the managed Dog script wasn’t, and it will only be destroyed when it gets garbage-collected. As long as we keep a reference to that Dog instance, it will live. When we invoke the dog’s Bark method, we are simply invoking a regular C# method of an entity that lives in the managed world. As long as it doesn’t try to access any of its native world entities that just got destroyed, that call method should execute successfully.

This scenario changes if we try to access any of its underlying native entities, like its gameObject property, as we do in the Move method. In that case, the engine throws an error to let us know that the object of type Dog we are trying to access was destroyed. Note that the exception is a MissingReferenceException, an exception defined in the UnityEngine namespace. It is not a NullReferenceException and thus, it is not a C# built-in exception. Again, that sounds too deep of a dive into the engine details, but this subtle detail underscores the fact that the managed entity (the Dog instance) was not destroyed at all, otherwise an NullReferenceException exception would have been thrown. In fact, the wrapper object still lives. The call to Destroy will only destroy native, engine components. In other words, the lifetime of native entities will be determined by a call to Destroy (or a non-additive scene load) whereas the lifetime of managed entities will be determined by the garbage collector. Thus, the two entities have different life cycles, and sometimes we need to keep that in mind.

For a deeper look into how the Unity engine works under the hood, check this article out.

The potential problem

We just went through why user-defined MonoBehaviours still live in the managed world even after their corresponding native entity was destroyed. Also, we discussed how we can still invoke some of their methods successfully, as long as they don’t try to access their native engine components. This fact introduces two important questions: how do we easily identify that these managed entities should not be accessed anymore because their underlying entities were destroyed? Is it a good practice to still access a MonoBehaviour after is has been destroyed?

Let’s start answering the second question: no, it is not a good idea to access a MonoBehaviour after its native entities were destroyed. One might think that it’s safe to keep some of its methods free from accesses to native, engine code, but this practice hurts code maintainability badly. It introduces an unspoken (or undocumented) rule that some methods should not try to access some specific components. Consequently, some developer — unaware of this weird rule — might change the code down the development process, which could potentially introduce errors. Additionally, it goes against the idea of using a MonoBehaviour, which is to attach scripts to GameObjects so they can interact. If you want to use a objects that does not necessarily relate to a GameObjects and its life cycle, do not inherit from MonoBehaviour and use a vanilla C# class instead — or maybe you want to use a ScriptableObject.

The solution

The first question remains open: how do we easily identify that these managed entities should not be accessed anymore? The answer comes from Unity. We can easily check if the underlying entities of a MonoBehaviour were destroyed in two ways:

  • Check for equality against null:
    if (_dog == null)
  • Check it as a boolean expression:
     if (_dog == true)

    or simply

     if (_dog)
  • The UnityEngine.Object class (which MonoBehaviour inherits from) implements custom equality operators that check if the underlying entities were destroyed. This operation is more complex than simply checking if the object reference is null because it invokes native code to check if the underlying entity was destroyed. As a consequence, it is also less performant than a vanilla C# null comparison. That’s why the Rider IDE displays a warning (“Comparison to ‘null’ is expensive”) whenever this null check is performed in a performance critical context. This custom implementation was reconsidered a while ago by Unity developers, but it was kept and still exists. This tool gives us a way to safely and easily check the lifetime of a MonoBehaviour‘s underlying object, which was exactly what we were looking for. But there is one thing to keep in mind…

    One little trap

    Even though we can use Unity’s custom equality operators to check whether a MonoBehaviour has been destroyed, we need to be careful about when we perform this check. As it happens, Unity does not destroy an object exactly when Destroy in invoked. Instead, the given object is tagged for destruction, which will only happen after the end of the current Update loop, but before rendering. At that point, all objects that were tagged for destruction will be actually destroyed. As a consequence, a check for destruction invoked right after (within the same Update loop) the Destroy call will return false. For example:

    private void DestructionTest()
        Debug.Log($"Is the dog null/destroyed? {_dog == null}");

    The method above, when executed outputs “Is the dog null/destroyed? False”. If we run the same check one frame later, or even inside LateUpdate, the check returns true. That happens because the call to Destroy doesn’t actually destroy the object, it only tags it for later destruction. After the Update loop, the engine gathers all objects tagged for destruction and actually destroys them, one by one. As a consequence, we need to keep the delayed destruction in mind when checking for existing entities.

    Not all C#’s equality operators implement this custom behaviour. For a deeper look into Unity’s equality operators, check this article on the subject.


    Unity does a great job at hiding implementation details and at abstracting away the complexity of its native side by providing C# wrappers for developers. Although this abstraction layer can be often ignored, there are some nuances we should keep in mind, like the different lifetimes of managed and native entities. But once we understand what is going on behind the scenes, it becomes clear that some unexpected behaviours are just consequences of Unity’s scripting duality.

    In this article, we learned how managed entities that inherit from UnityEngine.Object have a native counterpart and how their lifetime differ. We also learned how to use Unity’s custom implementation of equality operators to safely check for object destruction. Finally, we learned how the engine’s strategy to handle destruction can interfere on the lifetime check we just mentioned.

    That’s it for today. As always, feel free to leave a comment with questions, corrections, criticism or anything else that you want to add. See you next time!


    Unity Blog: Custom == operator, should we keep it?
    Rider: Avoid null comparisons against UnityEngine.Object subclasses
    Rider: Possible unintended bypass of lifetime check of underlying Unity engine object

    C#’s Finalizer/Destructor Trap

    Let’s talk about C#’s finalizers (also called destructors in C#) and how a common mistake when using them might lead to unwanted behaviour, especially in applications made with the Unity engine.


    A finalizer is a method that is called whenever an instance of a class is being garbage-collected. It is used for cleanup, commonly to release resources. The code below contains an example class for a music player, where its destructor closes an open file. Note that finalizers always start with a tilde and can’t have any parameters.

    public class MusicPlayer
        private FileStream _file;
        public void Play(string filePath)
            _file = File.Open(filePath, FileMode.Open);
            // ...
            if (_file != null)

    Disclaimer: even though destructors and finalizers are two different things, C#’s spec treats them as the same. If you’re acquainted with both definitions, the C#’s mechanism discussed in this article is actually a finalizer. Also, if you come from a C++ background, even though C#’s syntax for finalizers resembles C++’s syntax, keep in mind that they’re not the same. C++ destructors are called explicitly (and deterministically) by the user whereas C#’s finalizers are called implicitly (and nondeterministically) by the garbage collector.

    A quick intro to Unity’s MonoBehaviour

    The Unity engine provides a handy base class (MonoBehaviour) that contains common behaviour often needed for game development. Among other things, it contain event methods that are automatically called by the engine under given scenarios. These event methods are guaranteed to be called under the right circumstances and therefore developers can rely on them. Some of these methods are called when an object is created and when it’s destroyed, analogous to constructors and finalizers (which should not exist for MonoBehaviours). For example, the Awake methods is called when an object is created and the OnDestroy method is called when it’s destroyed.

    Common practice

    It is common practice to use Awake and OnDestroy to subscribe and unsubscribe to events, respectively. The code below shows an example.

    The ExampleButton behaviour contains only an event, for example purposes:

    public class ExampleButton : MonoBehaviour
        public event Action OnClick;

    The MyBehaviour behaviour subscribes to _button‘s OnClick event when it’s created and unsubscribes to the same event when it’s destroyed.

    public class MyBehaviour : MonoBehaviour
        [SerializeField] private ExampleButton _button;
        private void Awake()
            _button.OnClick += Foo;
        private void OnDestroy()
            _button.OnClick -= Foo;
        private void Foo()
            // ...

    The code above works as expected and both initialization and cleanup execute as expected.

    The naive thought

    At some point during development, we introduce a regular, non-MonoBehaviour class called ExampleClass. Unlike MonoBehaviours, it can’t rely on event methods like Awake for initialization. As an alternative, we normally use the class’ constructor. Analogously, ExampleClass can’t rely on event methods like OnDestroy for cleanup. As an alternative, we use its finalizer.

    public class ExampleClass
        private string _name;
        private ExampleButton _button;
        public ExampleClass(string name, ExampleButton button)
            _name = name;
            _button = button;
            _button.OnClick += Bar;
            Debug.Log($"Calling {_name} destructor.");
            _button.OnClick -= Bar;
        private void Bar()
            // ...
        private static void Something()

    Let’s define ExampleBehaviour, a MonoBehaviour that contains an ExampleButton and an ExampleClass. On its Awake method, the _object field (of type ExampleClass) is initialized.

    public class ExampleBehaviour : MonoBehaviour
        [SerializeField] private ExampleButton _button;
        private ExampleClass _object;
        private void Awake()
            _object = new ExampleClass("Foo", _button);

    A new Unity scene is created and only 2 objects are added to it: one containing an ExampleButton (called MyButtonGameObject) and one containing an ExampleBehaviour (called MyBehaviourGameObject). Whenever the scene is played, MyBehaviourGameObject’s Awake method is invoked and the _object variable is assigned, as expected. Inside ExampleClass constructor, the event subscription is executed, as expected. Nothing unusual so far.

    Screenshot 2019-09-08 at 17.33.41

    Then, because of some design decision, the MyBehaviourGameObject object is destroyed along the application lifetime. We realize that we might have to do some cleanup because _object should unsubscribe from _button‘s OnClick event. But then we come to the conclusion that it actually should be alright and that no additional cleanup should be necessary. Whenever MyBehaviourGameObject gets destroyed, the garbage collector will collect its ExampleBehaviour script. Since _object only belongs to MyBehaviourGameObject, it should be collected as well, which should trigger its finalizer and unsubscribe from the events.

    The trap

    Later during development, we notice some weird behaviour whenever the user clicks on MyButtonGameObject: there’s ExampleClass behaviour still being executed. But at that point of the execution, there should be no activeExampleClass in the scene because MyBehaviourGameObject was destroyed! After checking the call stack, we find out that Bar is being called by ExampleButton‘s OnClick event. But wait a second, something is wrong. ExampleClass‘s finalizer was responsible for event unsubscription. What happened? After some more debugging, you finally find out that the object’s finalizer doesn’t ever get called. But why isn’t the finalizer being called at all?

    Maybe the garbage collector is not running, for whatever reason. Let’s try to force collection using a simple script.

    public class ManualGarbageCollector : MonoBehaviour
        private void Update()
            if (Input.GetKeyDown(KeyCode.C))

    For testing purposes, I manually delete MyBehaviourGameObject and press the C key. The finalizer is still not executing. What is going on? Is the garbage collector broken? MyBehaviourGameObject was destroyed and _object should be collected, which should trigger the event unsubscription.

    To answer those question, we need to understand how C#’s garbage collector works. First, no, it’s not broken at all. In fact, it’s doing exactly what it’s told to. It’s our minds that forgot what we were doing. An object will only get collected whenever there are no references to it. None at all. Zero. If there’s one, even if really hidden, forgotten reference to an object, it will not get collected by the garbage collector. It’s as simple as that. But it still doesn’t make sense. The only reference to _object was inside ExampleBehaviour, right?


    We forgot about the one reference we were trying to get rid of: the one inside ExampleButton‘s OnClick event. But wait a second, we didn’t store a reference to _object in that event, we just stored a reference to a method, right?

    Again, wrong. Although it looks like we’re subscribing to a method, we need to keep in mind that it’s an instance method. It belongs to an instance of a class, a.k.a. an object. Under the hood, that method’s reference consists – among other things – of a reference to the method in memory and a reference to the instance the methods should be called on. If you ever programmed in Python, think of how the first argument of an instance method is self. As a consequence, an event subscription to an instance method will keep a reference to the instance itself. Therefore, it will stop the object of being collected by the garbage collector.

    We can show show that _object‘s reference is, in fact, being kept by MyButtonGameObject by destroying the latter. Once destroyed, the garbage collector will collect ExampleButton‘s memory and later, _object.

    If Bar was a static method, this wouldn’t happen because references to static methods don’t include a reference to an object. Consequently, _object‘s reference count would drop to 0 and it would be eventually collected by the garbage collector.

    The solution

    Thankfully, there is an easy solution for that: create a cleanup method and explicitly call it whenever necessary. On this article’s example, the perfect candidate would be ExampleBehaviour‘s OnDestroy method.

    On ExampleClass:

    public void Cleanup()
        _button.OnClick -= Bar;

    On ExampleBehaviour:

    private void OnDestroy()

    This fix not only will guarantee that events get unsubscribed but it will also – ironically – remove the last references to _object, which allows its (now obsolete) finalizer to be called.

    The bonus trap

    There’s another trap regarding finalizers that is not related to the one described above. Finalizers are called by the garbage collector, which runs on a separate thread than Unity’s main thread. As a consequence, two problems might come up.

    First – as usual – Unity engine code can not be called from a separate thread. Calling something as simple as _foo.gameObject will throw a UnityEngine.UnityException with the message “get_gameObject can only be called from the main thread“.

    Second, Unity will not catch and log exceptions running on separate threads. Therefore, any exceptions thrown inside finalizers (like the one described above) or in any separate thread might fly under the radar and never get acknowledged by the developers. There are two possible fixes for this problem. One is the obvious: catch the exceptions inside the thread itself. Another one can be used for user-defined threads (and thus is not applicable to GC threads): use Task instead of Thread to start a new thread with exception handling support.


    Whenever using C# finalizers, keep in mind that they will only get called when no references to their respective objects are left, including references to instance methods. Therefore, using finalizers to unsubscribe from events and to remove delegate references might lead to unwanted behaviour. As an alternative, create a cleanup method and explicitly call it whenever necessary.

    As a good practice, try to use finalizers for what they are good for: freeing resources. For other usages, don’t rely on them and explicitly invoke cleanup methods. Some even say that pure finalizers should avoided and the disposable pattern should be used instead.

    That’s all for today. As usual, feel free to leave a comment with corrections, questions, criticism or compliments. See you next time!

    [Postmortem] Developing an educational, kiosk, VR game for Windows Mixed Reality

    In this article I analyze the development Voedingscentrum VR, an educational, kiosk-style and Virtual Reality game I helped to develop at Fantazm from October 2018 until April 2019. After a quick introduction about the project, the biggest problems that surfaced during development are explained along with the solutions we found to solve them. At the end, a quick conclusion wraps the article up.


    The project was commissioned by Voedingscentrum (The Netherlands Nutrition Center), an independent organization funded by the Dutch government. Their aim is to use relevant research regarding sustainable nutrition to promote balanced, sustainable food consumption and production.

    The concept

    The game was planned as a fun, educacional VR experience targeted at children from 9 to 12 years old. Players are placed into a cartoonish world where they need to feed Smikkel – an elephant-like creature – for 3 to 4 meals: breakfast, lunch, afternoon snack and dinner.


    A robot waiter teaches the players about the game mechanics and its main loop: for each meal the player chooses, Smikkel comes to the dinner table and you feed it by choosing, grabbing and placing food items (served by the waiter) on a plate. Once all the food items of a specific meal are chosen (between 2 and 6 items), Smikkel eats the food and its mood, energy levels and appearance change according to the food items you fed him. Following, playtime starts and the player can interact with Smikkel by petting it and throwing balls so it can fetch them. During playtime, it should be evident that the food Smikkel ate affected its behavior. Additionally, the waiter displays a report showing the selected food items and gives the user feedback on the chosen food.


    Once playtime finishes, the feeding loop starts again. Once all the meal loops are over, the waiter gives a final feedback, announces that the experience is over and instructs the player to remove their VR headset. During the entire experience, spectators can watch the gameplay through a TV.

    The educational value is clear: children learn that what they ingest might affect their mood, stamina and health. The fun aspect is present: immersing into a cartoonish VR world and interacting with fictional characters is exciting, even for adults.

    The team

    Voedingscentrum was responsible for nutrition expertise and game design insights. The development team consisted of one game designer/project manager, two game developers (I inherited the project from another developer), one 2D artist, two 3D artists, one external 3D artist/animator and an external voice actor agency.

    The timeline

    Prototyping started as early as May of 2018 and the last version of the game was deployed in April 2019. Although this period might seem long, development didn’t take place continuously because the development team didn’t work on this project exclusively. Therefore, it’s hard to estimate how many hours were put into development.

    The setup

    The VR experience was meant to be installed in Dutch museums. Besides the VR headset and controllers, walls with game art, a grass carpet and a TV (for spectators) would be installed to add a fun, immersive and inviting environment. By design, the experience should work unsupervised and require low maintenance. Windows Mixed Reality (hereafter referred as WMR) was chosen as the target platform because it offers the features we sought (VR headset with controllers and inside-out tracking) at an affordable price.

    Me playing the first setup of VoedingscentrumVR at Boerhaave museum in Leiden, the Netherlands. Source: Voedingscentrum Facebook page.

    The development environment

    The project was developed using the Unity engine (by the end of the project, 2018.3.11f) and C# as a programming language. On initial development stages, Unity Collaborate was used for version controlling, but later we switched to a Git-based solution hosted by BitBucket.

    The versions

    Three versions (1.0, 1.1 and 1.2) of the game were shipped. The first one was installed at Boerhaave Museum (Leiden, the Netherlands) in December of 2018 and it showed us right away that there were some challenges we had to overcome before the game was ready for the world. The exhibition was taken down and we came back to the drawing board. After a quick break, we started the development of version 1.1, on which we tackled most of the problems the previous version unearthed. Although most of the design problems found in version 1.0 were fixed in version 1.1, technical problems (mostly regarding Windows Mixed Reality) could not be overcome. The application was migrated from WMR to HTC VIVE + SteamVR on version 1.2.

    The Challenges

    As expected, some challenges surfaced during the development process and after we installed the experience for the first time. In this section we discuss how we overcame them – whenever possible. Keep in mind that this article was published months after development ceased. Therefore, some of these problems might not exist anymore. If that’s the case and you are aware of a possible solution, please leave a comment on the comments section so this post can be updated. Adding new solutions to this postmorten can help many developers that might run into the same problems we did.


    Arguably the biggest challenge we faced during the game development were the WMR controllers. It’s nothing about their build quality or responsiveness (which are both great, by the way), but we ran into a few different aspects that didn’t match our project. We did not find solutions for some of these challenges, and there’s a reason behind that. At the end, the accumulation of problems regarding the controllers was so overwhelming that we decided to ditch the controllers altogether. We used a Leap Motion to track hand movement so we could use the player’s hands as interaction interfaces instead. Nevertheless, the problems we ran into before ditching the controllers  are listed below.

    Although the WMR controller has many buttons (touchpad, thumbstick, menu, windows, trigger, grab), the proposed gameplay was so simple that it only required one button. This sounds like the opposite of a problem and we didn’t expect it to be one at all, but something surfaced during playtests with children. Unless they were instructed beforehand, players had a hard time figuring out how to interact with the elements in the VR world, usually because they didn’t know which button to press. Changing the button mapping didn’t seem to solve the problem because different children were attracted to different buttons. Our proposed solution was to create either a sleeve or a case around the controller to hide unused buttons and to guide the children towards the desired one. We considered modifying the controller, removing the thumbstick altogether and even placing a banner with gameplay instructions next to the headset. None of these sounded like good solutions, but we were open to trying them out in order to allow a smooth gameplay.

    The game was supposed to run unsupervised in a museum. Therefore, the entire experience should always be ready for a new game session, requiring no setup from the player whatsoever. This requirement was put into check by the controllers because when left idle, they turn off automatically and – unlike the headset – there is no way (via software or hardware) to setup the idle timer duration. Therefore, if there was enough time between game sessions to bring the controllers to an idle state, the next player would have to turn them on in order to play. We were left with no other choice but to add an instruction banner explaining how to turn the controllers on, which again, was far from ideal.

    WMR controllers are turned on by pressing their Windows key for a few seconds. Since it had already been stablished that the users might have to turn the controllers on before playing, that key must have been easily accessible. That introduced an unexpected problem that didn’t surface until we installed the first version of the game for public access: the same key serves as a “home” key for the entire WMR ecosystem. When pressed, the current application is suspended and the user is brought to the WMR home, an environment that serves as a start screen for MR applications. From there, the user can have access to other applications (e.g. web browser, other games, settings, YouTube…) without any constraints at all. Unfortunately, this feature can not be disabled via software and physically disabling the key wasn’t viable because it’s the same key that turns the controllers on. Hence, we were stuck in a pit: disabling or hiding the key incapacitates the controller, and leaving it accessible gives the user total freedom over the computer.  We could not solve this problem and it was one of the key factors that lead us to the Leap Motion switch.

    Finally, the controllers are wireless and no cords keep someone from removing them from the experience. Attaching them to a piece of hardware would add more wires to the existing headset cords and could hurt gameplay and overall enjoyment. We didn’t expect anyone to bring the controllers home as a memento and decided to risk losing them, which didn’t happen during the short period of time the installation was displayed.

    Facing all these challenges, we started to question our platform choice and to search for alternatives to WMR. But other platforms either didn’t solve our existing problems or introduced new ones. Our team had some experience with the Leap Motion and we decided it was worth a shot. We took a couple of days to implement the Leap Motion and its library into the game and by the end of it, we were quite surprised. Not only using the hands to interact with the VR elements solved all of the problems we had with the WMR controllers, but it also felt way more natural than pressing buttons. We decided to keep the WMR headset but we ditched the controllers for the Leap Motion. After some tests with children, we identified some problems regarding the Leap Motion (discussed on the next section), but they appeared to be way more manageable than the challenges we had with the WMR controllers.

    Leap Motion

    The Leap Motion was meant to replace the WMR controllers as interaction devices in the VR world. Although the switch eliminated the challenges we faced with WMR, other characteristic issues surfaced once we started to use the Leap Motion.

    Whilst migrating to the Leap Motion solution, it was clear the one of the game’s key interactions was far from ideal: throwing balls. Previously, the user could grab balls using the WMR controller’s trigger button and throw them by releasing it, using a bezier curve as an indicator of where the ball would land. Naturally, we thought that when using your hands, the user could simply grab the ball and instinctively throw it. It sounded like the obvious design choice. Although, during internal playtests, an instinctive throw was never successful and the ball would always drop during the movement. The reason became evident: a natural ball throw usually requires a backwards arm movement, bringing the player’s hands outside of the Leap Motion tracking area. Screenshot 2019-07-07 at 17.11.39In order to execute a successful throw, the user had to keep their hands in the tracking area during the entire movement, which did not look natural whatsoever. We overcame this obstacle by changing the ball throwing design and adding a ball cannon which throws balls inserted into its feeder. After some external playtests, we added some visual clues to the ball cannon (on the right) and it was clear that the ball throwing problem was solved. Children are naturally attracted to the cannon and the visual cues make its purpose obvious.

    All playtests with the Leap Motion happened during winter, which introduced an unexpected and somewhat funny new challenge. Long sleeve shirts (wore due to low temperatures) often covered part of the children’s hands, occluding them from the Leap Motion. As a consequence, tracking was regularly compromised. It’d never happen during development because the developer (a.k.a. me) comes from a warm part of Brazil and isn’t comfortable wearing long sleeve shirts. Therefore, clothing occlusion was never present during internal playtests. This is yet another – funny – example of how even the smallest cultural differences can affect game development. We didn’t focus too much on this problem because of the following reasons. First, it didn’t seem to harm gameplay substantially because the players were still able to interact with the VR elements, despite he occasional loss of tracking. Second, a small percentage of children experienced this clothing occlusion. Third, the problem was seasonal and wasn’t present for a good part of the year. Lastly, instructions to tuck away long sleeves were added to the game instructions.

    In order to integrate the Leap Motion into the VR setup, the sensor had to be fixed to the headset. During development, we examined the VR headset (Lenovo Explorer), trying to find the best place to attach the Leap Motion sensor. The gameplay focus mostly on hand movement at and above eye level. Therefore, the Leap Motion sensor should be angled on a way that is optimal for the proposed gameplay. Additionally, the sensor’s location could not block the headset sensors. Once we found the perfect spot, we attached the sensor to the Lenovo headset using regular office tape. Before shipping the game, we had to come up with a more permanent, safe solution. We ended up using glue to attach the Leap Motion to the headset. The Leap Motion never fell off the VR headset during the couple of months the installation was publicly available.

    During the installation of the experience, one problem became evident: the headset the client chose (Samsung Odyssey) was different from the one used during development and it had a different sensor placement. Finding an optimal spot to attach the Leap Motion became a challenge because there’s less free room on the Samsung headset. After a few tests, it became obvious that there was no spot that would not occlude (even if just a bit) the headset sensors without compromising the hand tracking. In the end, we glued the Leap Motion in such a way that it would slightly obstruct the headset sensor, but not enough to compromise the WMR tracking. As a lesson, make sure you use the target hardware during development to avoid problems on installation/deployment day.


    Since the game is intended to be displayed on public spaces, the environmental noise could muffle the game voices and sounds. Thus, a setup with headphones was desirable. Luckily, there was a WMR headset with built-in headphones: the Samsung Odyssey. During playtests, a new game requirement was added: the external TV used to display the gameplay to spectators should also play the game sounds. By the time the game’s first version was installed, there was a Windows Mixed Reality setting to enable or disable audio redirection to the headset, but no audio mirroring option was available. The audio could go either to the headset or to the TV, but never to both. After some research, we found some third-party Windows applications that could simulate audio cards and that could potentially solve the problem, but it introduced some delay between the headset and the TV audio, which was undesirable. By the time we started working on the game’s second version, the WMR settings were updated and a mirroring option was added, but it presented the same delay problem as the third-party mirroring solution. Currently, the delay is still present and we have not found a solution for it yet.

    Windows Mixed Reality

    Even though the WMR platform sounds really promising and the quality of the compatible hardware is quite impressive for a first generation of devices, the development platform has a lot of room for improvement.

    Sadly, the WMR settings panel is quite basic and lacks many features that are present on other VR platforms (idling timeout, stop rendering when idling, “locked” mode, etc). We needed some of these features for our game and it was sad to see that they were just not there. When searching for solutions, we found out that it was possible to develop SteamVR applications for WMR compatible headsets and controllers, so we decided to give it a try. Since we were using VRTK – which supports both WMR and SteamVR – the switch to SteamVR took less than a day and it was successful. We we able not only to use the WMR hardware with SteamVR, but we also had access to SteamVR’s control panel. Both versions of the game are actually SteamVR applications.

    Additionally, there’s not much control over WMR’s play area. First, you can’t customize the play area walls (e.g. color, shader). Second, you can’t choose which way is forward while setting up the area. Apparently, the WMR platform totally ignores which way you start setting up the area and chooses whichever play area side is the longest one. As a consequence, which way the player faces when the game starts might differ based on the WMR room setup process you perform. Being able to select which way is forward was crucial on this project because the physical, printed walls (seen at the picture in the intro) should match the virtual environment. As a workaround, scripts to rotate and save the player’s rotation using keyboard shortcuts were added to the game. After doing the WMR setup, we do our own VR play area rotation adjustments using the keyboard shortcuts. This problem could be avoided if the user had more control over the play area settings during room setup. Additionally, we often experience wrong floor height values and have to use the WMR standard “Floor Check” app to fix it. I believe that a simple and easy floor height check should be included during room setup, like the one from the Oculus Quest.

    Overall, I found the WMR platform to be too closed to developers. Sometimes it felt like I was fighting against the platform in order to accomplish my goals. Many settings that should be available (either on the WMR control panel or via an API) can’t be changed whatsoever. That being said, I did notice that the team responsible for the WMR platform is listening to developers and is slowly implementing requested features. Maybe the platform is just not mature enough and all it needs is some time to evolve.

    Kiosk mode

    The game had specific requirements that made it a kiosk application: the computer should be easy to turn on and off (ideally just one physical button), the application should open automatically when the system finishes booting up, nobody should assist or instruct players, the application should run as long as the computer was on and the game should reset every time someone takes the headset off. At the time, unlike with the HoloLens, WMR didn’t have a kiosk mode, so we had to implement the requirements by ourselves.

    We cleaned the target computer from anything that could interrupt gameplay: notifications and Windows Update were disabled, anti-virus and performance improvement programs were uninstalled, Wi-Fi was turn off and unnecessary startup items were removed. Then, we made sure the application would never be interrupted by energy saving features: we disabled Window’s energy saving options and the headset sleep timer [1, 2]. To prevent screen burn-in on the headset display, we added a “headset fade” component that fades the display to black when nobody is wearing the headset.

    The WMR Control Panel automatically shows up once a WMR application is open. This behavior may seem logical, but it might get on the way of some kiosk applications. Two applications are being loaded and are competing for windows focus (i.e. stay on top): the Control Panel and the game. Even though the VR application is set to run on full screen mode, it might lose focus for the WMR Control Panel, which is undesired, given that spectators would not be able to watch the gameplay with the Control Panel on top of the game window. Luckily, we found a workaround on Unity’s forums that worked perfectly. Once implemented, this solution enables to developer to steal windows focus anytime. The game steals focus once its setup process (waiting for the headset to return to an idle state) is over and the WMR Control Panel never occludes the game screen.

    The game should reset every time the headset is taken off so the next player gets to experience a fresh environment. We used SteamVR’s library to check for user presence via the built-in headset’s proximity sensor instead of Unity’s user presence check because the latter only changed state once the headset idled. As a consequence, if a player handed the headset to another person, the headset wouldn’t idle, the user presence variable wouldn’t change and the game wouldn’t reset. Using the proximity sensor enabled a more controlled reset process which always reset the game when players were switched.

    Once the above features were implemented, the game behaved like a kiosk application that could be turn on and off using a single button (the computer’s power button) and that would work uninterruptedly for hours on end.


    Before starting external playtests, we would introduce the game and instruct the children on how to play it. In retrospect, we shouldn’t done that. Since the game would be used as an unsupervised installation, nobody would explain to children how to play the game (in the end we added some small icons with basic guidelines on the spectator screen, but it still wasn’t as informative as the instructions we gave them). Therefore, we should’ve replicated the same kind of environment during external playtests. As a consequence, we didn’t catch usability and UX problems as soon as we should have. In fact, some of the problems were only evident after the first day the game was available to the public. On future projects, we will try to mimic the real world environment on external playtests in order to catch usability problems as early as possible.


    Attention: this subsection is targeted at programmers.

    I tried two new features on this project: async/await and C# 7. The first one was used as a replacement for coroutines because it delivers the same features but enables the usage of return values and error handling. You can read more about the motivation behind the switch and how to use async/await in Unity here. The new version of C# brings features as nested functions, pattern matching, tuples, more expression-bodied members and out variables. These features allow for more succinct but easily understandable code. You can read more about them here and here. Both async/await and C# 7 additions proved to be worth the time investment and I’m glad I took the time to learn about them.


    Looking back, I wish we’d created more tools to automate repetitive tasks we had to perform during development. An example is taking screenshots of food items within the environment for client approval. The first time I had to take screenshots, I didn’t think that we would end up having so many approval iterations, so I didn’t implement a tool to automate it and took them manually. And every time I had to take the screenshots again, I’d think “I’m sure this is the last time. It’s not worth implementing a tool just for this”. And as one can imagine, I was wrong. Many times.


    VoedingscentrumVR was a challenging project, mostly from usability and UX perspectives. We learned valuable lessons on how to design unsupervised VR applications, how to deal with WMR limitations, how to plan better playtests and how to design better interactions using the Leap Motion. It was a project that seemed an easy task at the beginning, but that showed us we had more to learn than expected. But at the end, we accomplished our task and developed a fun, educational VR game that children enjoy playing.

    This was my first project after going back to game development. It was also my first VR project since 2016 and my first WMR project ever. Besides learning more about VR development, this project taught me a lot about project and time management and client communication. Although there were times I wish this project was over already (as we do in every project), I’m now grateful I was part of it.

    As usual, if you have any questions, comments or corrections (specially corrections!), please leave a comment on the comments section. I hope to see you all on the next blog post!