2024-10-27 10:53:47 +03:00

16 KiB

Raw Blame History

Collections Benchmarking and Performance Tests

Overview
- Containers
- Allocators
Container Benchmarking and Performance Tests
- Example Code - List.Add
- Results - List.Add
Allocator Benchmarking and Performance Tests
- Example Code - RewindableAllocator.FixedSize
- Results - RewindableAllocator.FixedSize

Overview

com.unity.collections provides pre-defined intermediate 'glue' layers on top of the Benchmark Framework to enable relatively simple creation of performance and benchmark testing for a wide variety of code paths which may be taken when using the collections package.

Containers

Examples of provided benchmarking and performance testing include:

NativeContainer code
Burst compiled NativeContainer code with safety enabled
Burst compiled NativeContainer code with safety disabled
UnsafeContainer code
Burst compiled UnsafeContainer code with safety enabled
Burst compiled UnsafeContainer code with safety disabled

Combine those with:

Container.ParallelWriter code going wide in any of the above mentioned situations
Container.ReadOnly code going wide

and it is easy to visualize the vast number of possibilities which we want to monitor and generate concrete performance data and comparisons on.

Regarding comparisons, we also want to ensure that these burst compatible containers are competitive or better with a similar container in .NET/IL2CPP/Mono's base class library, and have a way to validate and track improvements there as well, such as those found in:

System.Collections.Generic
System.Collections.Concurrent

Allocators

Naturally, there is a similar story with the custom allocator types provided by the collections package. In this case we want to be able to compare:

A provided IAllocator implementation in a managed code path
The same in a Burst compiled code path with safety enabled
Again the same in a Burst compiled code path with safety disabled

against:

The UnityEngine built-in Allocator.Temp
The UnityEngine built-in Allocator.TempJob
The UnityEngine built-in Allocator.Persistent

Container Benchmarking and Performance Tests

Container performance testing and benchmarks are built around a small handful of types.

Type	Description
`BenchmarkContainerType`	This enum defines variations for Native and Unsafe containers with and without burst compilation - with and without safety enabled. See the inline documentation for full details.
`IBenchmarkContainer`	Tests are written as implementations of this interface. It provides means for generic int parameters, allocation and disposal of Native, Unsafe, and C# Base Class Library containers, and measurement of the same.
`BenchmarkContainerRunner`	Easy-to-use API for running measurements in a single call. See inline documentation for full details, and see below for example usage.
`IBenchmarkContainerParallel`	Similar to `IBenchmarkContainer`, but designed to support tightly designed measurement code with Unity Job system workers in mind
`BenchmarkContainerRunnerParallel`	Similar to `BenchmarkContainerRunner`, but designed to parameterize worker thread counts for performance testing and benchmarking parallel container implementations

Example Code - List.Add

Here is a real-world basic example of implementing a performance and test and benchmark comparison for lists. This measures the cost of simply adding elements to a list with the expected capacity pre-allocated.

    struct ListAdd : IBenchmarkContainer
    {
        int capacity;
        NativeList<int> nativeContainer;
        UnsafeList<int> unsafeContainer;

        void IBenchmarkContainer.SetParams(int capacity, params int[] args) => this.capacity = capacity;

        public void AllocNativeContainer(int capacity) => ListUtil.AllocInt(ref nativeContainer, capacity, false);
        public void AllocUnsafeContainer(int capacity) => ListUtil.AllocInt(ref unsafeContainer, capacity, false);
        public object AllocBclContainer(int capacity) => ListUtil.AllocBclContainer(capacity, false);

        public void MeasureNativeContainer()
        {
            for (int i = 0; i < capacity; i++)
                nativeContainer.Add(i);
        }
        public void MeasureUnsafeContainer()
        {
            for (int i = 0; i < capacity; i++)
                unsafeContainer.Add(i);
        }
        public void MeasureBclContainer(object container)
        {
            var bclContainer = (System.Collections.Generic.List<int>)container;
            for (int i = 0; i < capacity; i++)
                bclContainer.Add(i);
        }
    }

To run these measurements, the calling code is quite simple, and generates a multitude of Performance Test Framework tests which can be run from the Unity Test Runner as well as through CI regression checks, and it also supports the necessary code paths for Benchmarking to make performance comparisons on all the variations including the BCL variation. Note the BCL variation of System.Collections.Generic.List will not appear as a Performance Test Framework test - it is considered for benchmarking only.

    [Benchmark(typeof(BenchmarkContainerType))]
    class List
    {
        ... 
        [Test, Performance]
        [Category("Performance")]
        public unsafe void Add(
            [Values(10000, 100000, 1000000)] int insertions,
            [Values] BenchmarkContainerType type)
        {
            BenchmarkContainerRunner<ListAdd>.Run(insertions, type);
        }
        ...
    }

Results - List.Add

This above two code snippets generate something like the following (notice the BCL tests aren't generated):

Running the DOTS/Unity.Collections/Generate Container Benchmarks menu item will generate a markdown report, again running the same single code path per type. Here is a snippet of the full results showing only the output for List.Add:

List

Functionality NativeList (S) NativeList (S+B) NativeList (B) UnsafeList (S) UnsafeList (S+B) UnsafeList (B) List (BCL)

Add(10000) 0.178ms (0.1x) 🟠 0.057ms (0.3x)       0.018ms (0.8x)       0.041ms (0.4x)       0.006ms (2.3x) 🟢 0.014ms (1.1x)       0.015ms (1.0x)

Add(100000) 1.827ms (0.1x) 🟠 0.622ms (0.2x)       0.180ms (0.8x)       0.432ms (0.3x)       0.061ms (2.4x) 🟢 0.139ms (1.1x)       0.146ms (1.0x)

Add(1000000) 18.910ms (0.1x) 🟠 6.443ms (0.2x)       1.814ms (0.8x)       4.136ms (0.4x)       0.586ms (2.5x) 🟢 1.482ms (1.0x)       1.468ms (1.0x)

Functionality	NativeList (S)	NativeList (S+B)	NativeList (B)	UnsafeList (S)	UnsafeList (S+B)	UnsafeList (B)	List (BCL)
`Add(10000)`	0.178ms (0.1x) 🟠	0.057ms (0.3x)	0.018ms (0.8x)	0.041ms (0.4x)	0.006ms (2.3x) 🟢	0.014ms (1.1x)	0.015ms (1.0x)
`Add(100000)`	1.827ms (0.1x) 🟠	0.622ms (0.2x)	0.180ms (0.8x)	0.432ms (0.3x)	0.061ms (2.4x) 🟢	0.139ms (1.1x)	0.146ms (1.0x)
`Add(1000000)`	18.910ms (0.1x) 🟠	6.443ms (0.2x)	1.814ms (0.8x)	4.136ms (0.4x)	0.586ms (2.5x) 🟢	1.482ms (1.0x)	1.468ms (1.0x)

Allocator Benchmarking and Performance Tests

Allocator performance testing and benchmarks are built around a small handful of types.

Type	Description
`BenchmarkAllocatorType`	This enum defines variations for allocators with and without burst compilation - with and without safety enabled. See the inline documentation for full details.
`IBenchmarkAllocator`	Tests are written as implementations of this interface. It provides means for generic int parameters, creation and destruction of allocators, allocation and freeing of memory using these allocators as well as using Unity Engine's built-in allocators `Temp`, `TempJob`, and `Persistent`, and measurement of the same.
`BenchmarkAllocatorRunner`	Easy-to-use API for running measurements in a single call. See inline documentation for full details, and see below for example usage.
`BenchmarkAllocatorUtil`	Generalized API for simplifying common Setup and Teardown implementations of `IBenchmarkAllocator` derived test types

Example Code - RewindableAllocator.FixedSize

The following example will omit another utility type designed for RewindableAllocator. The type is designed to simplify setup, teardown, and Rewind functionality necessary on a per-test-run basis. See RewindableAllocatorPerformanceTests.cs for reference.

    struct Rewindable_FixedSize : IBenchmarkAllocator
    {
        RewindableAllocationInfo allocInfo;

        public void CreateAllocator(Allocator builtinOverride) => allocInfo.CreateAllocator(builtinOverride);
        public void DestroyAllocator() => allocInfo.DestroyAllocator();
        public void Setup(int workers, int size, int allocations) =>
            allocInfo.Setup(workers, size, 0, allocations);
        public void Teardown() => allocInfo.Teardown();
        public void Measure(int workerI) => allocInfo.Allocate(workerI);
    }

To run these measurements, the calling code is quite simple, and generates a multitude of Performance Test Framework tests which can be run from the Unity Test Runner as well as through CI regression checks, and it also supports the necessary code paths for Benchmarking to make performance comparisons on all the variations including the Temp, TempJob, and Persistent variations. Note these Unity Engine built-in allocator variations will not appear as a Performance Test Framework test - it is considered for benchmarking only.

    [Benchmark(typeof(BenchmarkAllocatorType))]
    [BenchmarkNameOverride("RewindableAllocator")]
    class RewindableAllocatorBenchmark
    {
        ...
        [Test, Performance]
        [Category("Performance")]
        [BenchmarkTestFootnote]
        public void FixedSize(
            [Values(1, 2, 4, 8)] int workerThreads,
            [Values(1024, 1024 * 1024)] int allocSize,
            [Values] BenchmarkAllocatorType type)
        {
            BenchmarkAllocatorRunner<Rewindable_FixedSize>.Run(type, allocSize, workerThreads);
        }
        ...
    }

Results - RewindableAllocator.FixedSize