C# math is fast

I was reading an article on neural networks that mentioned the usual sigmoid activation function when the inputs are real numbers in the [0, 1) interval:

1 / (1 + e−x)

The article mentions that this is probably where the program would spend at least half of its time so I thought "why not pre-compute a bunch of values and trade memory and precision for time"? It turns out, C# math is quite fast and the gain might not be worth it. (I haven't tested it yet with a NN.)

This is the code I wrote to benchmark the two options, using LinqPad 5:

void Main()
{
    const int STEPS = 1 * 1000 * 1000;
    
    Func<double, double> activation = x => 1.0 / (1.0 + Math.Exp(-x));
    var cache = Precompute(0.0, 1.0, STEPS, activation);
    
    Benchmark("Using the cache", x => cache[(int) Math.Truncate(x * STEPS)]);
    Benchmark("Calling the function each time", activation);
}

double[] Precompute(double lower, double upper, int steps, Func<double, double> func)
{
    var result = new List<double>();
    
    For(0.0, 1.0, 1.0 / steps, value => result.Add(func(value)));
    
    return result.ToArray();
}

void Benchmark(string message, Func<double, double> func)
{
    const int COUNT = 100 * 1000 * 1000;
    
    var dummy = 0.0;
    var ts = ComputeBenchmark(() => For(0.0, 1.0, 1.0 / COUNT, value => dummy += func(value)));
    
    Console.WriteLine(message + ": " + ts + " - sum = " + dummy);
}

void For(double lower, double upper, double increment, Action<double> action)
{
    for (var value = lower; value <= upper; value += increment)
        action(value);
}

TimeSpan ComputeBenchmark(Action action)
{
    var sw = new Stopwatch();
    sw.Start();
    
    action();
    
    sw.Stop();
    return sw.Elapsed;
}

These are the results; using the cache takes about 70% of the time required when calling the function every time, at the expense of several MB of memory and some loss of precision. In a large NN, where billions of evaluations are expected in training, the trade-off might be worth it. I don't yet have enough experience to decide.

Using the cache: 00:00:01.9850792 - sum = 62011439.025702
Calling the function each time: 00:00:02.7755272 - sum = 62011450.5899971

As a side-note, I have computed and printed the sum to avoid the compiler from optimizing away the function calls; I don't know if LinqPad would do that but I wanted to avoid it just in case.

Comments

Post a Comment

Popular posts from this blog

Posting dynamic Master / Detail forms with Knockout

Image
I have the following dynamic form in my GitHub Inventory project : I created the form with KnockoutJS , using some ideas from this article . The detail view looks like this: <table id="acquisition_items" class="jtable"> <thead> <tr> <th>Product Name</th> <th>Quantity</th> <th>Price</th> <th>Value</th> <th><button data-bind="click: addItem">[+]</button></th> </tr> </thead> <tbody data-bind="foreach: items"> <tr> <td> <input data-bind="value: ProductName" type="text" value="" /> </td> <td> <input data-bind="value: Quantity" type="text" value="" /> </td> <td> ...
Read more

Comparing Excel files, take two

Image
I already wrote this project once, on GitHub , but I can't say that I like the result. I wrote that code without TDD, mostly to see if I can still work in the "normal" fashion. I can, but I had a bug that took me a while to even discover and then was difficult to write tests for. I also hate the temporal dependency in that design (if you don't call the SortBy method first, the ExcludeRecords method will return incorrect results). Anyway; here's attempt number two, writing that code using TDD. Structure I start by creating my usual structure for projects of this type - a library for the logic, a console application for the main program and a test project for tests. I also use the Enable NuGet Package Restore option (right-click on the solution node) to add the three NuGet files that will allow someone to automatically download all the packages I'm referencing without wasting space in the source code repository. (Right now I've only added the Moq pa...
Read more

EF Code First: seeding with foreign keys

This is just a quick tip, more for future reference for myself than anything else. When using Entity Framework Code First, one of the things you might want to do while developing the application is to seed your database with some test data. This can be something as easy as: protected override void Seed(InventoryDB context) { context.Products.AddOrUpdate( p => p.Name, new Product { Name = "Hammer", SalePrice = 11.99m, }, new Product { Name = "Nail Pack x100", SalePrice = 0.05m, }, new Product { Name = "Saw", SalePrice = 19.99m, }, new Product { Name = "Toolkit", SalePrice = 39.99m, } ); //... } (I am using the AddOrUpdate method so the code detects whether those products already exist and doesn't add them again if they do.) The Product class is quite simple and – most importantly – doesn't depend on any other: public class Product { public int Id { g...
Read more