![Metalama 2024.1: Performance Improvements Analysis [Updated for 2024.2]](/assets/images/2024/2024-04-22-performance/featured.svg)
Metalama 2024.1: Performance Improvements Analysis [Updated for 2024.2]
One of the primary objectives of Metalama 2024.1 was to improve performance. After a thorough analysis of the benchmarks, we’re excited to report a significant improvement of approximately 55% in aspect processing speed! However, we still need to make strides in reducing the fixed cost of using Metalama versus the standard Microsoft compiler.
Update: 2024-07-30: Added Metalama 2024.2 performance.
TL;DR
Using Metalama instead of the standard C# compiler will increase your dotnet build time by typically 50%. This overhead varies linearly with the number and complexity of aspects in your project. Even with one aspect on every single method, your build should not be more than twice as slow with Metalama than without it.
Our Methodology
So, how did we arrive at these results?
We forked the NopCommerce open-source project and introduced aspects to methods at random, regulated by two factors:
- the percentage of types that will have aspects (denoted $\text{Types}$ in the equation below); and
- the percentage of methods in each target type with aspects (denoted $\text{Methods}$).
Using BenchmarkDotNet, we analyzed the execution time of dotnet build /t:rebuild for 12 different combinations of the factors. We used a dedicated computer with a minimal Alpine installation to ensure no background process could interfere with the measurements.
We then computed a linear regression based on two factors, $\text{Types}$ (the percentage of types affected) and $\text{Types} \times \text{Methods}$ (the percentage of methods affected), where both $\text{Types}$ and $\text{Methods}$ are included as a percentage between 0 and 1. The output of this function, $\text{TimeRatio}$, is the ratio between the build time with Metalama and without Metalama.
$\text{TimeRatio} = a + b \times \text{Types} + c \times \text{Types} \times \text{Methods}$
Here are the results of the regression:
| Version | Constant factor ($a$) | Type factor ($b$) | Method factor ($c$) |
|---|---|---|---|
| 2024.0 | 1.41 | 0.26 | 0.72 |
| 2024.1 | 1.45 | 0.15 | 0.40 |
| 2024.2 | 1.43 | 0.12 | 0.37 |
Our linear model has an $R^2$ of 0.98, suggesting that only 2% of the variance is unexplained. Given that benchmark results variance is higher than 2%, we can confidently say this model is highly reliable.
In simple terms:
- Building your code with Metalama will always be at least 45% slower than without Metalama, and
- Metalama’s build time is linearly dependent on two factors: the percentage of affected methods in your projects and the percentage of types containing these affected methods.
The dependency on the number of types containing affected methods is because each type is typically implemented in its own file, and part of Metalama’s cost depends on the number of files that need to be rewritten.
Our Improvements
Let’s compare the improvements we made in the 2024.1 version to the 2024.0 baseline:
| Version | Constant factor | Type factor | Method factor |
|---|---|---|---|
| 2024.0 | 100% (baseline) | 100% (baseline) | 100% (baseline) |
| 2024.1 | 103% | 58% | 55% |
| 2024.2 | 101% | 46% | 51% |
As you can see, the efficiency of Metalama in processing aspects has doubled from 2024.0 to 2024.2!
Wrapping Up
Our efforts in enhancing Metalama’s performance have been successful, with aspect processing speed nearly twice as fast as in Metalama 2024.0. However, there’s still room for improvement as using Metalama will slow your build by at least 45% compared to not using it.
Rest assured, we’ll continue to prioritize performance in our future releases. Stay tuned!