Unity Serialization Part 3: Scriptable Objects

This post is part of a series about Unity serialization. Click here for part 1: how it works and examples or click here for part 2: defining a serializable type.

On the last article, we learnt how we can define our own serializable types and discovered some problems that can emerge from it. One solution (although not ideal) to our problems is ScriptableObjects. These objects have two major uses: saving and storing data in the Editor and saving and storing data as an asset. According to the documentation, they are optimized and can store huge portions of data. There are a few singularities about Scriptable Objects and we will discuss them on the following paragraphs.

Defining and Creating

Primarily let’s learn how to define a Scriptable Object using the same example we’ve been using in the last article:

public class MyDatabase : ScriptableObject
{
  public List<City> cities;
}

As we can see, the only difference is that our class now derives from ScriptableObject. There is no need to add a serializable modifier to our script, since ScriptableObject derives from Unity.Object and that should be enough to make it serializable. The next step is creating an instance of MyDatabase, which isn’t possible without some extra code. In the past, a mix of EditorUI and AssetDatabase code was required, but recently (Unity 5.1), a simple modifier can make our lives better: [CreateAssetMenu].

[CreateAssetMenu]
public class MyDatabase : ScriptableObject
{
  public List<City> cities;
}

By marking your ScriptableObject with this modifier, a menu item will be created in the Asset/Create menu. Easy like that, this should create an Asset called “New My Database” in the Assets folder and it can be referenced by any MonoBehaviour just like any other type derived from Unity.Obejct. By doing this, we solve the first problem present on the previous blog post: databases should be assets and not MonoBehaviours. Now let’s learn how ScriptableObjects act differently from custom serializable classes in Unity.

Objects are stored as references, not copies

Consider the example from the previous post and think about this: if a MonoBehaviour has two references to the same City object, how will they get serialized? How does the change made in one affect the other? The answer to these questions is slightly counterintuitive:  they are decoupled and serialized separately, hence changing one’s value won’t change the other’s. Let’s see an example using a MonoBehaviour that executes in edit mode just to make things easier:

[ExecuteInEditMode]
public class MonoBehaviourDecoupleTest : MonoBehaviour 
{
  public City city1;
  public City city2;
     
  private void OnEnable()
  {
    if (city1 == null)
    {
      City chicago = new City();
      chicago.name = "Chicago";
      city1 = chicago;
      city2 = city1;
    }       
  }
}

Based on this example, any changes made to the object “city2” are also reflected on the object “city1”, which happens… until you hit play and pause. After doing that, modifying one object doesn’t affect the other. That happens because by hitting play, we force Unity to serialize the scene data, which will serialize each object “in line” (copying all serializable variables into a new object) and not by reference. By doing that, Unity forgets that those two objects were actually the same and treats both as separate objects.

Only classes that inherit from Unity.Object are serialized as actual references and fortunately ScriptableObject can help us with that, since it is a UnityObject. Let’s use the same example, but first let’s define a new City class, this time as a ScriptableObject:

public class ScriptableCity : ScriptableObject 
{
  public string name;
}

Now let’s use it instead of a regular City like the previous example. Note that we have a peculiar way of instantiating a ScriptableObject, and the reason behind that will become clear soon:

[ExecuteInEditMode]
public class ScriptableDecoupleTest : MonoBehaviour 
{
  public ScriptableCity city1;
  public ScriptableCity city2;

  private void OnEnable()
  {   
    if (city1 == null)    
    {     
      city1 = ScriptableObject.CreateInstance<ScriptableCity>();          city1.name = "Chicago";      
      city2 = city1;
    }
    
    Debug.Log(city1.name);
    Debug.Log(city2.name);
    city1.name = "New York";
    Debug.Log(city1.name);
    Debug.Log(city2.name);
  }
}

We use Debugs to check the data since the inspector won’t show us much useful info easily. Differently from the first example, hitting play and pause won’t decouple the references and both cities will be named “New York”. Notice the difference here: the variables “city1” and “city2” were not serialized as part of the ScriptableDecoupleTest script – which happened in the first example. What Unity actually does is create a scene object (remember the odd way of instantiating it?) and keep it hidden, serialize it as part of the scene, and save its reference in “city1” and “city2”. Therefore, every time we change one variable, the other changes too, since both reference the same object, just like any reference to an UnityObject (Rigidbody, Collider, Renderer…). Thereby, ScriptableObjects solves the third problem presented on the previous blog post: reference decoupling.

We can also use ScriptableObjects as assets instead of scene objects and the effect is the same. In addition, this approach may be considered really specific, but is essential when really complicated relationships must be consistent between objects.

Polymorphism

We face a serialization problem when using polymorphism and custom serializable classes: an instance of a derived class is serialized as an instance of the base class. Let’s use the example the Unity Documentation does: Animals. See the example:

[System.Serializable]
public class Animal 
{
  public string name;

  public Animal (string name)
  {
    this.name = name;
  }
}

Now let’s define the Dog class:

public class Dog : ScriptableAnimal 
{
  public string breed;

  public Dog (string name, string breed) : base(name)
  {
    this.breed = breed;
  }
}

And a simple MonoBehaviour that runs in the editor for testing:

ExecuteInEditMode]
public class BehaviourExample : MonoBehaviour 
{
  public List<Animal> animals;
  
  private void OnEnable()
  {
    if (animals == null)
    {
      animals = new List<Animal>();
    }
    if (animals.Count == 0)
    {
      Animal elephant = new Animal("Elephant");
      Dog dog = new Dog("Dog""Bulldog");
      Animal lion = new Animal("Lion");
      
      animals.Add(elephant);
      animals.Add(dog);
      animals.Add(lion);
    }
  }

  private void OnDisable()
  {
    Debug.Log(animals[1is Dog);
  }
}

If you have a list of Animals and you add a Dog to it, it will be serialized as an Animal, not a dog. Add this script to an object in the scene and observe the console when you deactivate it, you will see it displays “True”. Nice! That means that Unity knows that that object is a Dog, right? Well, hit play, then stop and do that again. Now it doesn’t remember that anymore, but what happened to that Dog? When you hit play, you told Unity to serialize the scene data, and that’s exactly what it did. It happens that because of how Unity’s serialization process works, it forgets about the fact that the Dog is a Dog and treats it as an animal instead.

ScriptableObjects can help us with that, given that it inherits from Unity.Object and that type is always serialized as references – and never inline. Let’s change the definition of Animal a bit and make it inherits from ScriptableObject:

public class ScriptAnimal : ScriptableObject 
{
  public string name;
}

Now let’s use an equivalent example to the one above, but adapting it to the way ScriptableObjects are created and initialized:

[ExecuteInEditMode]
public class ScriptableExample : MonoBehaviour
{
  public List<ScriptAnimal> animals;

  private void OnEnable()
  {
    if (animals == null)
    {
      animals = new List<ScriptAnimal>();
    }
   
    if (animals.Count == 0)
    {
      ScriptAnimal a1 = ScriptableObject.CreateInstance<ScriptAnimal>();
      Dog dog = ScriptableObject.CreateInstance<Dog>();
      ScriptAnimal a2 = ScriptableObject.CreateInstance<ScriptAnimal>();
    
      a1.name = "Elephant";
      dog.name = "Dog";
      dog.breed = "Bulldog";
      a2.name = "Lion";
    
      animals.Add(a1);
      animals.Add(dog);
      animals.Add(a2);
    }
  }   

  private void OnDisable()
  {
    Debug.Log(animals[1is Dog);
  } 
}

This example is analogue to the previous one, so run it and observe the console. Even after hitting play, stopping the game and disabling the script, we get “True” as output, which means that the Dog is still a Dog and it was serialized as one, keeping the polymorphism intact. Therefore, ScriptableObjects solve the second problem introduced in the last blog post: polymorphism and user-defined classes. This might not be the best solution ever when dealing with serialized polymorphic code (we can’t use constructors anymore and inspecting the objects is a pain) but it may be necessary in some situations.

In addition, a fix to this polymorphism serialization problem has been extremely requested by the community in the last years and haven’t been solved it.

 Serialization depth limit (or no support for null)

Demonstrating an example of the depth serialization problem in Unity is fairly simple. Take a look:

[Serializable]
public class DepthClass  
{
  public List<DepthClass> depthObjects; 
}

This code looks harmless, but it’s not, as seen on the error thrown (on previous versions of Unity it was a warning, or simply didn’t thrown any message at all):

Serialization depth limit exceeded at ‘DepthClass’. There may be an object composition cycle in one or more of your serialized classes.

Let’s understand why. Let’s take a look at the “depthObjects” variable and try to figure out how it would be serialized in Unity if it’s an empty list. The naive thought would say that if it should be serialized as null, like any other field, but it’s not. It happens that Unity doesn’t support null serialization for custom classes (like it does for UnityObjects) and that object will be serialized as en empty instance – which is transparent to the user. On top of that, each of those empty instances has its own “depthObjects” variable, which would be serialized as another empty instance, creating a cycle that would never end, therefore crashing Unity. To avoid that, the folks at Unity Technologies set a limit (7 levels) for serialization depth, which means that after 7 levels of depth, Unity will assume that it hit a cycle and will stop the serialization at that point. Seven levels of serialization can sound like too much, but for some problems, it might be necessary.

Given that, always think twice before serializing any field that has recursive declarations. If it’s not really necessary to serialize it, don’t! But if you really want to use 7 or more levels, or you really need to serialize it, you should use ScriptableObjects. Since ScriptableObjects do support null serialization, this problem simply vanishes:

[Serializable]
public class DepthClass : ScriptableObject
{
  public List<DepthClass> depthObjects;  
}

And this simple change should get rid of the error and let you use more than 7 depth levels of references.

Data can be shared among objects

This problem (although not listed on the previous article) is exactly the same discussed in the Unity documentation about ScriptableObjects. If you store an array of integers that occupies about 4MB in a prefab and you instantiate that prefab 10 times, you would have a copy of the array in each prefab’s instance, occupying a total of 40MB. That happens because the array lives on the script itself, and copying the prefab copies the script, hence the array.

If instead of declaring the array on the prefab’s script you had a reference to a ScriptableObject which contains the array, all the instances would point to the same data. Therefore, 10 instances of that prefab would store 10 references to the same object (and not the actual data), occupying about 4MB only. Not only that substantially saves memory, but changes made to the array from any of the instances will affect the others, maintaining database consistency.

There is a catch, though. What if, instead of storing the array on a ScriptableObject, we store it on an empty prefab with a MonoBehaviour? Let me show what I mean. First let’s create a MonoBehaviour to holds the data:

public class BehaviourDatabase : MonoBehaviour 
{
  public Vector4[] vectors;
}

Let’s add this script to a scene object and save it as a prefab. Now let’s create another script that holds a reference to a BehaviourDatabase, add it to a scene object (let’s call it DatabaseModifier) and drag the prefab we just created to it’s “database” field.

public class MemoryTest : MonoBehaviour 
{
  public BehaviourDatabase database;
}

If we copy this DatabaseModifier object a dozen times and run the scene, we see that it behaviors exactly how we expected the ScriptableObject to behave: they all store a reference to the prefab on the project folder – and not the actual data. Not only that, but any change made on the array of Vector4s will also change the prefab’s data. Well, but if that also works, so why use ScriptableObjects instead of a prefab and a MonoBehaviour to store only data as an asset?* That’s what I tried to find out, but coudln’t find any solid, convincing and direct answer. Among some reasons, I found that:

  • We don’t have a GameObject and a Transform and any overhead it can create;
  • The idea behind a prefab is that it can be instantiated multiply times in runtime, which isn’t our case;

Those are not really convincing, but I personally rather have a ScriptableObject just because of the second reason. For the other uses, though, ScriptableObejcts are extremely recommended.

*Note the emphasis on _data_only_. Using a prefab and a MonoBehaviour won’t solve the other problems.

Conclusion

In this article we learnt how to solve some problems (most described on the last article) that may emerge when dealing with custom serialized types and Unity serialization. Even though some problems can be solved by using ScriptableObjects, it’s far form ideal.

Unity Serialization Part 2: Defining a Serializable type

This post is part of a series about Unity serialization. Click here for part 1: how it works and examples.

On the last article, we discussed about serialization concepts and how Unity implements it, learning which types can be serialized and which cannot. But what if we want to define our own type? How can I make it serializable so I can keep its data stored?

Understanding the problem

Let’s choose a (slight biased) model to implement as our example: a script to keep all the data to an investigation game which contains numerous cities, each one containing several places. Sounds pretty easy and straightforward, so let’s do it naively by creating MonoBehaviours: one for the database, one for the cities and one for the places. That should work:

public class MyDatabase : MonoBehaviour
{
  public List<City> cities;
}

public class City : MonoBehaviour
{
  public string name;
  public List<Place> places;
}

public class Place : MonoBehaviour 
{
  public string name;
}

There, done! But wait a second, something’s wrong: I can’t just create an instance of a MonoBehaviour, it should be attached to a GameObject and I don’t want to create a one to each instance of a city or place. Something is wrong conceptually. It happens that we can’t think of that data as behaviors, because they are not. They are simply objects, just like good old object-oriented programming. So let’s go ahead and take the MonoBehaviour inheritance from the City and Place classes.

public class City
{
  public string name;
  public List<Place> places;
}

public class Place 
{
  public string name;
}

Now let’s add the MyDatabase script to an object in the scene. Something is wrong again: I can’t see the list of cities in the inspector even though the field is public and should be serialized (therefore shown in the inspector).

Screen Shot 2015-09-20 at 7.47.14 PM

Defining a Serializable Type

That happens because we didn’t define our type as serializable, so Unity won’t serialize it. We never faced that problem before because we usually deal with classes that inherit from Unity.Object (Collider, RigidBody, Animation, MonoBehaviour…), which is a serializable type. There is an easy way to do it: add the [System.Serializable] modifier to the class:

[System.Serializable]
public class City 
{
  public string name;
  public List<Place> places;
}

[System.Serializable]
public class Place 
{
  public string name;
}

That gives us the expected result:

Screen Shot 2015-09-20 at 7.52.02 PM

By simply adding that modifier, we mark our class as serializable and solve our problem. The same process is also required when dealing with structs (serializable since Unity 4.5). In addition, Unity also serializes lists and arrays of serializable types by default.

Problems with that approach

Although this looks like a great solution, there are a few problems with it. The first (but not biggest) is that even though MyDatabase only stores data, it still is a MonoBehaviour and needs a GameObject to exist. Ideally, it should be an asset that only holds data, but we can’t simply take the MonoBehaviour inheritance off the class, otherwise we wouldn’t have a way to serialize it. What if there was a serializable type just like MonoBehaviour that doesn’t need a GameObject to live on? Keep that in mind. The other problems doesn’t involve data-storing objects only like the first one, but are also valid.

The second problem involves polymorphism and happens when a class inherits from a user-defined serialized class. Even though it’s intuitive that fields from both classes should be serialized regardless, that doesn’t happen. Let’s use the same example as Unity blog does: animals.

[System.Serializable]
public class Animal 
{
  public string name;
}

[System.Serializable]
public class Dog : Animal 
{
  public string breed;
}

public class PolymorphismExample : MonoBehaviour 
{
  public Animal[] animals;
}

Even though both Animal and Dog classes are serializable and Dog inherits from Animal, if we add a dog to our list of animals in PolymorphismExample, they will be serialized as Animals, losing the Dog type and consequently its fields. What if user-defined classes supported polymorphism? Again, keep that in mind.

The third problem is related to decoupled references, which is a fancy name to something really simple. Imagine you have the same Animal example as the problem above and you add 3 animals to your array, but all of them point to the same object. Due to how Unity’s serialization works, these references are decoupled and they are serialized as three different objects, hence changes made to any of those three objects won’t affect the other two. Unity simply forgets that those objects point to the same reference, which can be devastating to systems that keep complex relation between objects of that class.

The decoupling problem happens because these fields (primitives and user-defined) are serialized “in line” since they are actually part of the MonoBehaviour’s serialization data and not a data object itself. With objects that derive from Unity.Object though, the fields are serialized as actual references to the object, and not “in line” like custom classes. What if we could use a class that derives from Unity.Object, serializes the objects as references and maintains complex relations between our objects?

The last problem is related to recursive declarations, which can generate cycles. Consider this example:

[Serializable]
public class DepthClass  
{
  public List<DepthClass> depthObjects; 
}

And a MonoBehaviour that holds a reference to an instance of it:

public class DepthTest : MonoBehaviour 
{
  public DepthClass test;
} 

How many allocations will be done to serialize an uninitialized “DepthTest” script? The intuitive answer would be 1 – a null reference – but it happens that the Unity serializer doesn’t support null references of custom classes so it creates an empty object and serializes it instead (this is transparent to the user). And since this object is created and it has a reference to a list of objects of its own type, it creates a cycle in the serialization process that should go on forever. To prevent this cycle (for real, it’s not a joke) the Unity guys picked the – magical – limit of 7 depth levels and after reaching that level, the serializer will assume that a cycle was defined and will stop serializing fields of custom classes. What if we could use a type that supports null in the serialization pipeline?

Each problem described above has a potential solution and It turns out that all four can be fixed with the same resource: ScriptableObjects. It’s not an extremely elegant or ideal solution, but it’s the closest we get from one. Since it’s a fairly long subject, Scriptable Objects are described in depth on my next article. For now, let’s just acknowledge that those problems have a common way out and if you believe you may face one of those, take a look into it.

Modifiers and Serialization

Finally, let’s summarize the modifiers involved in serialization.

  • Use [System.Serializable] on a class or struct definition if you want it to be serialized;
  • Publics fields are serialized by default, but only if its type is serializable (constants, static and readonly fields are not serialized);
  • Use [SerializeField] if you wish to serialize a private field;
  • Use [NonSerialized] if you want to avoid serialization on a public field;
  • Use [HideInInspector] if you want to serialize but not expose the field in the inspector;

Conclusion

In this blog post we learnt how to define our own serializable types and acknowledged some problems that can emerge by doing it. On the next article, we will dive deep into a resource that can work out those problems: ScriptableObjects.

Reference

Unity Manual: Script Serialization
Unity Blog: Serialization in Unity
Unity Blog: Unity Serialization
Unity Tutorials: ScriptableObject

Unity Serialization Part 1: How it works and examples

This post is part of a series about Unity serialization.  On this series of articles we will discuss a topic that is extremely  important to Unity3D development: serialization. This subject may be a bit cloudy for beginners, but understanding it not only helps to figure out how the Unity engine works, but it can also become really handy during a game development process and assist you to build better solutions. We will concentrate our study on the following subjects:

  • What is it? (Part 1)
  • How Unity does (Part 1)
  • Examples (Part 1)
  • Defining a Serializable Type (Part 2)
  • Problems with Serialization (Part 2)
  • Scriptable Objects (Part 3)

Fell free to navigate through the sections if you are comfortable with the previous concepts.

What is it?

Among other definitions, serialization is the process of converting the state an object to a set of bytes in order to store (or transmit) the object into memory, a database or a file. In another words: it’s how you can save an object to restore its state for later use.

Let’s say you have a Vector3 and you need to store it for future use. Which fields would you save into a file to restore its state? X, Y and Z, that’s easy! Apply this rule for any object and if you serialize all the (important) data, you can reload the object exactly how it was before. We name the process of storing the state of the object as “serialization” and the reverse process of building an object out of a file “deserialization”.

As we already read, this process is really useful to store and transmit data, so think about this: you have an online store deployed on your server, probably running a database manager application aside. Eventually, we need to store the data from memory into the disk (primary storage is volatile) and we do it by serializing our objects into database tables (which basically are files). The same process happens with every application that needs to store data into a disk, and computer games (and engines) are not an exception.

How Unity Does

Unity, as a Game Engine, needs to load a lot of stuff (scripts, prefabs, scenes) from the disk into the memory within application data. Maybe more important than that, Unity needs to move data between the native C++ side of the engine and the managed C# side. Even though we may think this task is strict to loading and storing assets processes, the serialization is used in many more situations than we think (like inspector window, reloading of editor code and instantiation among other scenarios). You can learn a bit more about Unity Serialization on their own blog. I’d risk to say it’s a mandatory read for developers.

The UnityEngine.Object class (which is serializable) provides us with the serialization resource in Unity: any other class than inherits from it – that includes MonoBehaviour, ScriptableObject (more on this later on the series), Shader, Sprite and basically everything in Unity – can also be serialized. Most of its uses are invisible and don’t matter when developing a game, except for a major use: scripting. When you create scripts, you need to keep on mind which fields you want to serialize, and, to do so, your field must:

  • Be public, or have [SerializeField] attribute
  • Not be static
  • Not be const
  • Not be readonly
  • The fieldtype needs to be of a type that we can serialize.

Which fieldtypes can we serialize? According to Unity’s documentation:

  • Custom non abstract classes with [Serializable] attribute.
  • Custom structs with [Serializable] attribute. (New in Unity 4.5)
  • References to objects that derive from UntiyEngine.Object
  • Primitive data types (int,float,double,bool,string,etc)
  • Array of a fieldtype we can serialize
  • List of a fieldtype we can serialize

A really common data structure isn’t serializable: Dictionaries, even if you declare them as public, and [SerializeField] atribute. So keep that in mind when developing a game.

Ok, this is a lot of information, so let’s break it to a straightforward code example:

Examples

Let’s take the following code (MyBehaviour.cs) as an example:

using UnityEngine;

public class MyBehaviour : MonoBehaviour
{
  [SerializeFieldprivate int x = 3;
  [SerializeFieldprivate float y = 5.6f;
  public float pi = 3.1415f;
  private int mySecret = 42;
  public static int myStatic = 10;
  public const int myConst = 22;
  public readonly int myReadOnly = 99;
  public int MyProperty { get { return 100; } }
}

The only fields that are serialized are “x”, “y” and “pi” because they are not static, const, readonly, a property or private with no [SerializeField] attribute. One way to show this is by taking a look at the script’s inspector, which only shows us serialized fields (that can come handy):

Screen Shot 2015-04-08 at 10.48.08 PM

But there is a better way to show that the other fields were not serialized: by cloning the object. Remember I told you that the Instantiate method uses the Unity serialization process? So let’s test it by adding the following method to our script:

private void Start()
{
  Debug.Log("Pi:" + pi + ". MySecret:" + mySecret + ". MyStatic:" + myStatic);
}

private void Update()
{
  if (Input.GetMouseButtonDown(0))
  {
    pi = -4;
    mySecret = -11;
    myStatic = 13;
    GameObject.Instantiate(gameObject);
  }
}

This method should modify the current object and create another object in the scene when the mouse’s left button is pressed. A naive thought would be that since the second object is a clone of the first one, it should have the same value for “pi”, “mySecret” and any other field as the first object, right? Well, it does for “pi”, because it’s a public field (hence it’s serialized), but it doesn’t for “mySecret”: its value remains unchanged when the second Debug.Log() is executed:

screen-shot-2016-11-03-at-3-47-32-pm

The field “mySecret” can’t be serialized because it’s a private field with no [SerializeField] attribute (which is the only circumstance a private fields will be serialized*).

*: Private fields are serialized in some circumstances in the Unity Editor and that can lead us to some problems we will discuss later on the next article on the “Problems” session. (source)

Curiously, the value of “myStatic” changes in both objects, but does that mean static fields are serializable? The answer is no. Even though its behavior leads us to conclude that, remember that static fields belong to the class, and not to an instance of it. That said, if we, per example, create a new object and add a MyBehaviour script to it in play mode (Using “GameObject > Create Empty” in the menu), “myStatic” will have the same value as the other objects, although it isn’t a copy of any other object. However, note that the “pi” variable has its original value.

screen-shot-2016-11-03-at-3-51-03-pm

This proves what the Unity documentation tells us about how can we make a field of a serializable type be serialized. This works with most Unity objects, including MonoBehaviours, but what if I don’t want to define a MonoBehaviour and I still want to create a Serializable type? In other words, how can I define a Serializable type?

On the next article we will find out how to declare our own serializable types and how to treat problems that can come along with it.

Fell free to write me any suggestions, errors, complements or just to say hi on the comments section 🙂

Source:
Serialization in Unity by Lucas Meijer (mandatory read)
Unity Serialization Best Practices by Tim Cooper
ScriptableObject
Serialization (C# and Visual Basic)