Unity Serialization Part 2: Defining a Serializable type

This post is part of a series about Unity serialization. Click here for part 1: how it works and examples.

On the last article, we discussed about serialization concepts and how Unity implements it, learning which types can be serialized and which cannot. But what if we want to define our own type? How can I make it serializable so I can keep its data stored?

Understanding the problem

Let’s choose a (slight biased) model to implement as our example: a script to keep all the data to an investigation game which contains numerous cities, each one containing several places. Sounds pretty easy and straightforward, so let’s do it naively by creating MonoBehaviours: one for the database, one for the cities and one for the places. That should work:

public class MyDatabase : MonoBehaviour
{
  public List<City> cities;
}

public class City : MonoBehaviour
{
  public string name;
  public List<Place> places;
}

public class Place : MonoBehaviour 
{
  public string name;
}

There, done! But wait a second, something’s wrong: I can’t just create an instance of a MonoBehaviour, it should be attached to a GameObject and I don’t want to create a one to each instance of a city or place. Something is wrong conceptually. It happens that we can’t think of that data as behaviors, because they are not. They are simply objects, just like good old object-oriented programming. So let’s go ahead and take the MonoBehaviour inheritance from the City and Place classes.

public class City
{
  public string name;
  public List<Place> places;
}

public class Place 
{
  public string name;
}

Now let’s add the MyDatabase script to an object in the scene. Something is wrong again: I can’t see the list of cities in the inspector even though the field is public and should be serialized (therefore shown in the inspector).

Screen Shot 2015-09-20 at 7.47.14 PM

Defining a Serializable Type

That happens because we didn’t define our type as serializable, so Unity won’t serialize it. We never faced that problem before because we usually deal with classes that inherit from Unity.Object (Collider, RigidBody, Animation, MonoBehaviour…), which is a serializable type. There is an easy way to do it: add the [System.Serializable] modifier to the class:

[System.Serializable]
public class City 
{
  public string name;
  public List<Place> places;
}

[System.Serializable]
public class Place 
{
  public string name;
}

That gives us the expected result:

Screen Shot 2015-09-20 at 7.52.02 PM

By simply adding that modifier, we mark our class as serializable and solve our problem. The same process is also required when dealing with structs (serializable since Unity 4.5). In addition, Unity also serializes lists and arrays of serializable types by default.

Problems with that approach

Although this looks like a great solution, there are a few problems with it. The first (but not biggest) is that even though MyDatabase only stores data, it still is a MonoBehaviour and needs a GameObject to exist. Ideally, it should be an asset that only holds data, but we can’t simply take the MonoBehaviour inheritance off the class, otherwise we wouldn’t have a way to serialize it. What if there was a serializable type just like MonoBehaviour that doesn’t need a GameObject to live on? Keep that in mind. The other problems doesn’t involve data-storing objects only like the first one, but are also valid.

The second problem involves polymorphism and happens when a class inherits from a user-defined serialized class. Even though it’s intuitive that fields from both classes should be serialized regardless, that doesn’t happen. Let’s use the same example as Unity blog does: animals.

[System.Serializable]
public class Animal 
{
  public string name;
}

[System.Serializable]
public class Dog : Animal 
{
  public string breed;
}

public class PolymorphismExample : MonoBehaviour 
{
  public Animal[] animals;
}

Even though both Animal and Dog classes are serializable and Dog inherits from Animal, if we add a dog to our list of animals in PolymorphismExample, they will be serialized as Animals, losing the Dog type and consequently its fields. What if user-defined classes supported polymorphism? Again, keep that in mind.

The third problem is related to decoupled references, which is a fancy name to something really simple. Imagine you have the same Animal example as the problem above and you add 3 animals to your array, but all of them point to the same object. Due to how Unity’s serialization works, these references are decoupled and they are serialized as three different objects, hence changes made to any of those three objects won’t affect the other two. Unity simply forgets that those objects point to the same reference, which can be devastating to systems that keep complex relation between objects of that class.

The decoupling problem happens because these fields (primitives and user-defined) are serialized “in line” since they are actually part of the MonoBehaviour’s serialization data and not a data object itself. With objects that derive from Unity.Object though, the fields are serialized as actual references to the object, and not “in line” like custom classes. What if we could use a class that derives from Unity.Object, serializes the objects as references and maintains complex relations between our objects?

The last problem is related to recursive declarations, which can generate cycles. Consider this example:

[Serializable]
public class DepthClass  
{
  public List<DepthClass> depthObjects; 
}

And a MonoBehaviour that holds a reference to an instance of it:

public class DepthTest : MonoBehaviour 
{
  public DepthClass test;
} 

How many allocations will be done to serialize an uninitialized “DepthTest” script? The intuitive answer would be 1 – a null reference – but it happens that the Unity serializer doesn’t support null references of custom classes so it creates an empty object and serializes it instead (this is transparent to the user). And since this object is created and it has a reference to a list of objects of its own type, it creates a cycle in the serialization process that should go on forever. To prevent this cycle (for real, it’s not a joke) the Unity guys picked the – magical – limit of 7 depth levels and after reaching that level, the serializer will assume that a cycle was defined and will stop serializing fields of custom classes. What if we could use a type that supports null in the serialization pipeline?

Each problem described above has a potential solution and It turns out that all four can be fixed with the same resource: ScriptableObjects. It’s not an extremely elegant or ideal solution, but it’s the closest we get from one. Since it’s a fairly long subject, Scriptable Objects are described in depth on my next article. For now, let’s just acknowledge that those problems have a common way out and if you believe you may face one of those, take a look into it.

Modifiers and Serialization

Finally, let’s summarize the modifiers involved in serialization.

  • Use [System.Serializable] on a class or struct definition if you want it to be serialized;
  • Publics fields are serialized by default, but only if its type is serializable (constants, static and readonly fields are not serialized);
  • Use [SerializeField] if you wish to serialize a private field;
  • Use [NonSerialized] if you want to avoid serialization on a public field;
  • Use [HideInInspector] if you want to serialize but not expose the field in the inspector;

Conclusion

In this blog post we learnt how to define our own serializable types and acknowledged some problems that can emerge by doing it. On the next article, we will dive deep into a resource that can work out those problems: ScriptableObjects.

Reference

Unity Manual: Script Serialization
Unity Blog: Serialization in Unity
Unity Blog: Unity Serialization
Unity Tutorials: ScriptableObject

Unity Serialization Part 1: How it works and examples

This post is part of a series about Unity serialization.  On this series of articles we will discuss a topic that is extremely  important to Unity3D development: serialization. This subject may be a bit cloudy for beginners, but understanding it not only helps to figure out how the Unity engine works, but it can also become really handy during a game development process and assist you to build better solutions. We will concentrate our study on the following subjects:

  • What is it? (Part 1)
  • How Unity does (Part 1)
  • Examples (Part 1)
  • Defining a Serializable Type (Part 2)
  • Problems with Serialization (Part 2)
  • Scriptable Objects (Part 3)

Fell free to navigate through the sections if you are comfortable with the previous concepts.

What is it?

Among other definitions, serialization is the process of converting the state an object to a set of bytes in order to store (or transmit) the object into memory, a database or a file. In another words: it’s how you can save an object to restore its state for later use.

Let’s say you have a Vector3 and you need to store it for future use. Which fields would you save into a file to restore its state? X, Y and Z, that’s easy! Apply this rule for any object and if you serialize all the (important) data, you can reload the object exactly how it was before. We name the process of storing the state of the object as “serialization” and the reverse process of building an object out of a file “deserialization”.

As we already read, this process is really useful to store and transmit data, so think about this: you have an online store deployed on your server, probably running a database manager application aside. Eventually, we need to store the data from memory into the disk (primary storage is volatile) and we do it by serializing our objects into database tables (which basically are files). The same process happens with every application that needs to store data into a disk, and computer games (and engines) are not an exception.

How Unity Does

Unity, as a Game Engine, needs to load a lot of stuff (scripts, prefabs, scenes) from the disk into the memory within application data. Maybe more important than that, Unity needs to move data between the native C++ side of the engine and the managed C# side. Even though we may think this task is strict to loading and storing assets processes, the serialization is used in many more situations than we think (like inspector window, reloading of editor code and instantiation among other scenarios). You can learn a bit more about Unity Serialization on their own blog. I’d risk to say it’s a mandatory read for developers.

The UnityEngine.Object class (which is serializable) provides us with the serialization resource in Unity: any other class than inherits from it – that includes MonoBehaviour, ScriptableObject (more on this later on the series), Shader, Sprite and basically everything in Unity – can also be serialized. Most of its uses are invisible and don’t matter when developing a game, except for a major use: scripting. When you create scripts, you need to keep on mind which fields you want to serialize, and, to do so, your field must:

  • Be public, or have [SerializeField] attribute
  • Not be static
  • Not be const
  • Not be readonly
  • The fieldtype needs to be of a type that we can serialize.

Which fieldtypes can we serialize? According to Unity’s documentation:

  • Custom non abstract classes with [Serializable] attribute.
  • Custom structs with [Serializable] attribute. (New in Unity 4.5)
  • References to objects that derive from UntiyEngine.Object
  • Primitive data types (int,float,double,bool,string,etc)
  • Array of a fieldtype we can serialize
  • List of a fieldtype we can serialize

A really common data structure isn’t serializable: Dictionaries, even if you declare them as public, and [SerializeField] atribute. So keep that in mind when developing a game.

Ok, this is a lot of information, so let’s break it to a straightforward code example:

Examples

Let’s take the following code (MyBehaviour.cs) as an example:

using UnityEngine;

public class MyBehaviour : MonoBehaviour
{
  [SerializeFieldprivate int x = 3;
  [SerializeFieldprivate float y = 5.6f;
  public float pi = 3.1415f;
  private int mySecret = 42;
  public static int myStatic = 10;
  public const int myConst = 22;
  public readonly int myReadOnly = 99;
  public int MyProperty { get { return 100; } }
}

The only fields that are serialized are “x”, “y” and “pi” because they are not static, const, readonly, a property or private with no [SerializeField] attribute. One way to show this is by taking a look at the script’s inspector, which only shows us serialized fields (that can come handy):

Screen Shot 2015-04-08 at 10.48.08 PM

But there is a better way to show that the other fields were not serialized: by cloning the object. Remember I told you that the Instantiate method uses the Unity serialization process? So let’s test it by adding the following method to our script:

private void Start()
{
  Debug.Log("Pi:" + pi + ". MySecret:" + mySecret + ". MyStatic:" + myStatic);
}

private void Update()
{
  if (Input.GetMouseButtonDown(0))
  {
    pi = -4;
    mySecret = -11;
    myStatic = 13;
    GameObject.Instantiate(gameObject);
  }
}

This method should modify the current object and create another object in the scene when the mouse’s left button is pressed. A naive thought would be that since the second object is a clone of the first one, it should have the same value for “pi”, “mySecret” and any other field as the first object, right? Well, it does for “pi”, because it’s a public field (hence it’s serialized), but it doesn’t for “mySecret”: its value remains unchanged when the second Debug.Log() is executed:

screen-shot-2016-11-03-at-3-47-32-pm

The field “mySecret” can’t be serialized because it’s a private field with no [SerializeField] attribute (which is the only circumstance a private fields will be serialized*).

*: Private fields are serialized in some circumstances in the Unity Editor and that can lead us to some problems we will discuss later on the next article on the “Problems” session. (source)

Curiously, the value of “myStatic” changes in both objects, but does that mean static fields are serializable? The answer is no. Even though its behavior leads us to conclude that, remember that static fields belong to the class, and not to an instance of it. That said, if we, per example, create a new object and add a MyBehaviour script to it in play mode (Using “GameObject > Create Empty” in the menu), “myStatic” will have the same value as the other objects, although it isn’t a copy of any other object. However, note that the “pi” variable has its original value.

screen-shot-2016-11-03-at-3-51-03-pm

This proves what the Unity documentation tells us about how can we make a field of a serializable type be serialized. This works with most Unity objects, including MonoBehaviours, but what if I don’t want to define a MonoBehaviour and I still want to create a Serializable type? In other words, how can I define a Serializable type?

On the next article we will find out how to declare our own serializable types and how to treat problems that can come along with it.

Fell free to write me any suggestions, errors, complements or just to say hi on the comments section 🙂

Source:
Serialization in Unity by Lucas Meijer (mandatory read)
Unity Serialization Best Practices by Tim Cooper
ScriptableObject
Serialization (C# and Visual Basic)

How does Unity3D scripting work under the hood?

On the previous post we learnt how Unity exports to so many platforms. Now let’s talk about a subject that can sound unnecessary at first, but helped (at least me) a lot on the process of understanding how Unity works. We all know Unity3D is an engine, but how it really works? In this post we’ll learn about:

  • How Unity3D Engine is implemented
  • How we access the engine code
  • Wrappers
  • How to use your own Mono library inside Unity
  • Unity3D Engine vs Unity3D Editor

We’ve always been told that managed programming languages are slow compared to native code, even though some managed languages are reaching great performance. We know that Java and C# are great for web development, non-critical systems and mobile apps. But when it comes to high performance (a pit where realistic graphics always fall) we all know these guys can’t do a lot for us. Although the Unity game world is fulfilled with 2D or simple graphics games, it’s capable of delivering stunning and realistic worlds and scenes.

So we ask (at least I did) ourselves: how does Unity deliver such amazing graphics with a great performance, if all the coding is made with C#, Javascript or Boo, and runs on a virtual machine (Mono)? If these are managed languages, how does it do the job? Is it dark magic? Unfortunately not.

One more time, let’s do some researching. After consulting Google, we get to an answer. The Unity Engine itself, the core, is written using C\C++, which we know is a native language. All the graphics, sound, physics coding is mostly C++ (I can’t say how much exactly because I don’t have access to the source code). But the guys from Unity itself gave us some tips is an article about C\C++ code coverage. By now things start to make sense: Unity is written in C\C++ and that’s why it’s so fast, smooth and provides us with a good performance. It’s almost over when suddenly another question pops: So why (and how) do we program in C# if Unity runs native code under the hood?

Again, the answer lays in one single hero: Wrappers! Oh, you don’t know what a wrapper is? Let’s talk about it a bit.

So when you are coding in C\C++, it’s a common practice to create libraries with a couple of functions\classes you are using. When you do so, you choose which functions\classes you want to expose, and which you want to hide. The exposed ones will be accessible out of the library and the hidden ones are not, commonly used by other functions\classes inside it. You know, the famous exposing code problem. When you’re done, you often have a DLL file with your library inside, and you can use it as any regular library.

But if you never used wrappers, you probably used your libraries with the same language you wrote them. So here is where the wrappers do their magic. One of the wrappers uses (there are others) is giving us a layer that provides cross language interoperability. This means you can call a C\C++ code from a C#, Python or Perl programs, per example. If you never heard about wrappers and cross language use, this can be mind-blowing, but think about it: it’s all about moving data around. If you know how to handle the data between both sides, they can work in perfect harmony.

The first time I used it was in a Game Development class where our teacher gave us an assignment that consisted of compiling a C library into a DLL file, wrapping it with C# code, and import it into Unity. There are some tutorials online teaching how to do so and I think this one is really straightforward. If you are curious about, try it, it’s such a simple exercise but it can teach you a lot about how Unity works. It may sound really painful (and it can be) to wrap bigger libraries, but there are tools that help us to do such challenging job. The one we used was SWIG, it worked perfectly and it was all magic.

So what we just did? We created C code that is accessible from C# code. Do you get what we just did? Let’s call this C code “Unity Engine”, and let’s assume it has hundreds of functions to deal with graphics, sound, physics and particles. Now let’s create a library from it and then create a wrapper in C# that will access it. This way, any C# project that contains our library can access its content. This is the Unity Engine! The core is written using C\C++ and you code in C# taking advantage of the native code performance. The Unity Engine itself is a DLL in your installation folder, check it out. Also, you can add your own Mono libraries to your Unity project, the guys from Unity teach you how to play with it.

Let’s talk about one more thing: the difference between the Unity Engine and the Unity Editor. The Unity Engine is a huge library that provides us a lot of tools to run games. The Unity Editor (the gray interface, with the inspector, hierarchy, etc.) we use is another thing: it’s a user interface (UI) that helps us to build games, but it’s not necessary to run them. It’s literally just a friendly interface to help us using the Unity Engine. The editor runs some of its code from the Unity Engine, and some of its own. It is built mostly in C#, and can be expanded: you can create your own windows, inspectors, and crazy stuff like your own browser. Extending the editor IMO is one of the most powerful Unity’s features.

Summing up, the Unity Engine is a C\C++ set of libraries that help us to run games, and we access it using a wrapper, typically using C#. It’s a clever way to build games with great performance by taking advantage of the fast and easy development of C#. Finally, the Unity Editor is a user-friendly interface that helps us to build games using the Unity Engine.

Last but not least, the Unity Engine itself (native portion) runs in plenty of devices because it was compiled to all of those devices: check the Unity installation folder and you can see the dlls to every platform the engine exports to. So with this thought, we complete the circle of life circle of execution on a Unity game: the Unity Engine’s core was written in C\C++ and runs on every platform the engine targets. We can interact with that code through C# wrappers, which runs on the Mono virtual machine. Lastly, the Mono project by its nature (bringing C# to other platforms) gives us the environment to run C# applications on our devices.

That closes our discussion about how the Unity3D Engine works greatly on so many platforms with amazing performance and easy programming.

Important update (September, 2015): Unity’s IL2CPP technology is changing the way the engine’s scripting backend works, eliminating Mono’s VM and AOT compiler. Take a look at it!