https://github.com/kendallb/couchbase-aspnet

The first thing we noticed when we starting using the new session state provider is that the web site would lock up for about 30 seconds when rendering certain web pages. It turns out that there were some nasty bugs in the Enyim code that would crop up if you had a lot of parallel requests to the same session state store item. This is not something most web sites would have to deal with, but in our case we have some pages that generate a flurry of AJAX requests to the server when they load that would cause the session state provider to leave a session stuck in the locked state. When this happens IIS will spin attempting to get a write lock on the session state item, until a timeout occurs (usually 30 seconds) at which time it will forcibly clear the lock and continue on.

Examining the session state code in detail and comparing it line by line to our existing MySQL session state provider (which did not have this problem), we discovered the primary cause of the issue was that the code to release the locks on a session state item was not using the Couchbase CAS operation. Hence you could end up with a situation where one thread had a lock on the session and would clear it, but it’s cleared value would get overwritten by another thread attempting to lock it and it would never get cleared. The fix for this is included in the session state provider library in the above link.

Disabling Session Exclusive Access for Performance

One annoying feature that IIS has is that it considers session state to be exclusive and will block concurrent access to the session state unless you mark the session state as read only. You can read more about it on the MSDN documentation pages for the ASP.NET session state, but the important part is as follows:

Concurrent Requests and Session State

Access to ASP.NET session state is exclusive per session, which means that if two different users make concurrent requests, access to each separate session is granted concurrently. However, if two concurrent requests are made for the same session (by using the same SessionID value), the first request gets exclusive access to the session information. The second request executes only after the first request is finished. (The second session can also get access if the exclusive lock on the information is freed because the first request exceeds the lock time-out.) If the EnableSessionState value in the @ Page directive is set to ReadOnly, a request for the read-only session information does not result in an exclusive lock on the session data. However, read-only requests for session data might still have to wait for a lock set by a read-write request for session data to clear.

Although this seems like a reasonable requirement, and in fact ensures that two threads may not stomp on the same session data, it can in fact lead to performance issues when you have pages that make a lot of concurrent AJAX requests to the server. The correct solution is to decorate your AJAX methods in ASP.NET MVC with the Read Only session state behavior using the ASP.NET MVC SessionState attribute. That way no session data is going to be written out when those AJAX methods execute so IIS will allow them to run in parallel.

However in our case we are compiling and running a ton of legacy PHP code using Phalanger, which compiles the PHP code into .NET code that executes alongside our ASP.NET MVC code. Since this kind of session locking is not implemented in PHP at all, and it is difficult (if not impossible) to change the legacy code to enable read only sessions when needed, we started looking for a way to disable this feature in our custom session handler so we could get the Phalanger compiled PHP code to perform the same as it did under real PHP.

It turns out that supporting this is quite easy by adding a new flag to the Web.config provider entry and simply avoiding any locking in the session handler if is not desired. If you are interested in the implementation, look at the code and how it uses the exclusiveAccess flag internally in the handler. If you just want to use it, you can set the value using the “exclusiveAccess” attribute of the provider entry:

<sessionState customProvider="Couchbase" mode="Custom">
  <providers>
    <add name="Couchbase" type="Couchbase.AspNet.SessionState.CouchbaseSessionStateProvider, Couchbase.AspNet" exclusiveAccess="false" />
  </providers>
</sessionState>

Although I would highly recommend you use the correct solution of marking code you need to run in parallel with the ReadOnly session state attributes, sometimes it is easier when working with legacy code to simply turn off this feature

First Version: Lists of Cached Items

The original file based solution simply managed groups of cache values as part of a cache block, and because the groups were structured as directories on disk, it was easy enough to invalidate all cache entries for a group simply by recursing over the directory and deleting all the cache files. Hence the first version of our Couchbase cache store class was implemented very similarly. When an item was added to the cache, it was also added to a list of all cached items store in a meta data object for the cache group. Then when a group of cached items needed to be invalidated, we would read the list of items from the cache and then iterate over it deleting them from the Couchbase cache.

The initial implementation of the Read and Write functions looked like this:

/// <summary>
/// Adds a value to a dictionary of type <TKey, TValue> with the given key and value using a
/// Check-And-Set operation and retries. Note that if the item exists in the list, the value
/// is updated otherwise a new entry is added.
/// </summary>
/// <typeparam name="TKey">Type of the key for the items in the list</typeparam>
/// <typeparam name="TValue">Type of the value for the items in the list</typeparam>
/// <param name="client">Couchbase client</param>
/// <param name="key">Key used to identify the hash table</param>
/// <param name="entryKey">Key for the entry to be stored in the hash table</param>
/// <param name="entryValue">Value for the entry to be stored in the hash table</param>
private void AddToHashWithCas<TKey, TValue>(
    ICouchbaseClient client,
    string key,
    TKey entryKey,
    TValue entryValue)
{
    // Now keep track of all the items in this cache group
    CasResult<bool> casResult;
    do {
        // Try to read the existing list (or create one if it does not exist)
        var result = client.GetWithCas<Dictionary<TKey, TValue>>(key);
        var items = result.Result ?? new Dictionary<TKey, TValue>();

        // See if there is an existing entry
        if (items.ContainsKey(entryKey)) {
            // Update the value if it exists already
            items[entryKey] = entryValue;
        } else {
            // Add a new entry if it does not exist
            items.Add(entryKey, entryValue);
        }

        // Attempt to save the items using a Check-And-Set operation
        casResult = client.Cas(StoreMode.Set, key, items, result.Cas);
    } while (casResult.Result == false);
}

/// <summary>
/// Writes out a serialized object to the cache
/// </summary>
/// <param name="o">Object to write to the cache</param>
/// <param name="group">Cache group that this item belongs to</param>
/// <param name="key">Key name for this cache item</param>
/// <param name="expiresAt">Optional date/time stamp that the cached item will expire at</param>
public void Write(
    object o,
    string group,
    string key,
    DateTime? expiresAt)
{
    // Get access to the cache
    var client = GT.CBCache;

    // Determine the full key name for this cache item
    var fullKey = group + '/' + key;
    var lockKey = fullKey + ".lock";

    // Store the item in the cache
    if (expiresAt == null) {
        client.Store(StoreMode.Set, fullKey, o);
    } else {
        client.Store(StoreMode.Set, fullKey, o, (DateTime)expiresAt);
    }

    // Now keep track of all the items in this cache group
    AddToHashWithCas<string, DateTime?>(client, group, fullKey, expiresAt);

    // Clear the lock
    client.Store(StoreMode.Set, lockKey, false);
}

/// <summary>
/// Reads a serialized object from the cache
/// </summary>
/// <param name="o">Place to return the object read from the cache</param>
/// <param name="group">Cache group that this item belongs to</param>
/// <param name="key">Key name for this cache item</param>
/// <returns>True if the item was read from the cache, false if not</returns>
public bool Read(
    out object o,
    string group,
    string key)
{
    // Get access to the cache
    var client = GT.CBCache;

    // Determine the keys we need. The key of the item we key off the primary group
    var fullKey = group + '/' + key;
    var lockKey = fullKey + ".lock";

    // Loop around trying to read the cache until either it is valid, or we get the write lock for the cache. This way
    // we guarantee that we will only ever have one writer actually generating the cache data at a time.
    CasResult<bool> casResult;
    CasResult<bool> locked = new CasResult<bool>();
    do {
        // Loop until the lock is clear in case another instance is writing the cache. We time out after 10 seconds
        for (int i = 0; i < TimeoutLoops; i++) {
            // Check if someone has the lock
            locked = client.GetWithCas<bool>(lockKey);
            if (!locked.Result) {
                // Not locked for writing, so we are good to go
                break;
            }

            // Sleep for 100 milliseconds to allow the lock to get cleared
            System.Threading.Thread.Sleep(TimeoutSleep);
        }

        // Read the item from the cache and return it if found
        if ((o = client.Get(fullKey)) != null) {
            return true;
        }

        // Check if we got the lock. If not then someone has the lock and we timed out, so just clear the lock
        // and move on. This should not happen, but if it does it is OK.
        if (locked.Result) {
            client.Store(StoreMode.Set, lockKey, false);
            return false;
        }

        // Attempt to set the lock using a Check-And-Set operation
        casResult = client.Cas(StoreMode.Set, lockKey, true, locked.Cas);
    } while (casResult.Result == false);

    // Return false indicating that the cache is invalid and we have the write lock
    return false;
}

/// <summary>
/// Resets all the cache entries for this specific cache group
/// </summary>
/// <param name="groupKey">Cache group to clear</param>
public void ClearGroup(
    ICouchbaseClient client,
    string groupKey)
{
    Dictionary<string, DateTime?> items;
    CasResult<bool> casResult;
    do {
        // Try to read the existing list (or create one if it does not exist)
        var result = client.GetWithCas<Dictionary<string, DateTime?>>(groupKey);
        items = result.Result ?? new Dictionary<string, DateTime?>();

        // Clear the list using a Check-And-Set operation
        casResult = client.Cas(StoreMode.Set, groupKey, new Dictionary<string, DateTime?>(), result.Cas);
    } while (casResult.Result == false);

    // Now loop through and kill off all the cache entries referenced in the list. Although
    // it is possible someone might have added a new cache entry to the list while this is
    // being processed, as long as we clear each one each cache entry will end up being
    // correctly refreshed
    foreach (var item in items) {
        client.Remove(item.Key);
    }
}

Using these functions is pretty simple. Code that wants to cache something will first call the Read() function and attempt to read an existing cache block from the cache and if that fails, it will then generate the cached block data, and call the write function:

string group = "myGroup";
string key = "myItem";
object o;
if (!Read(out o, group, key)) {
    o = "Something to cache";
    Write(o, group, key, DateTime.Now.AddMinutes(5));
}

// Do something with object o

If the Read() function finds a valid cache entry, it simply returns the result. If the Read() function fails, it will grab the lock for the cached item using a Check And Set operation in Couchbase, and then return false. The caller then generates the content to be cached, and calls the Write() function. This function then writes the cached data to the cache, adds a reference to the cached item to the list of all cache items for this group and then clears the lock. This ensures that no other code will be generating the same cached item at the same time, so if an item expires any other threads needing that same cached item will end up blocking until the locks is cleared, at which time they will pick up the newly generated cache item.

Although it may seem inefficient to make all the other threads block and wait for the first thread to generate the cached item, in practice this turns out to be more efficient than letting all the threads attempt to generate the cached item at the same time. If for instance the cached item is generated using expensive calls to a SQL database, and suddenly 100 threads all need the same cached item, the SQL server is going to be able to generate the content for 1 thread making a set of expensive SQL calls much faster than having 100 threads all make the same SQL calls in parallel.

Also it is worth pointing out that in this implementation we use a hash table to store the list of items in a cache block rather than a list, primarily because our caches end up with thousands of items per group and using a hash table makes maintaining the list much more efficient than using a link list (which would have to do a linear search to see if the item was already in the list).

Although the code above works fine, the process of invalidating cache entries is still time consuming because the ClearGroup() function has to iterate through all the items in the group and remove them from the cache store, which can take some time with thousands of items in the group. Also the cache items in the group may be already getting generated while the clear operation is in progress, since we don’t block them from being generate during clears.

On top of that we can only group cache entries together by placing them in a single group. It would be nice if there was a way to be able to make a cache entry belong to multiple groups, giving you more control over when an item is invalidated in the cache. And it would also be nice to be able to improve the performance of the clear operation. Thankfully all of this can be done using the concept of Sentinels!

Second Version: Using Sentinels to Avoid Lists

So how does a sentinel work? The basic concept is that rather than storing references to all the cache items in a list that represent a group of cache items, we instead store a list of prior sentinel values as meta data long with the cached item. In essence we are flipping things upside down, and don’t store what items are in a group, but rather what groups an item belongs to. When we need to read a cached item, we read a list of prior sentinel values stored with the item, and compare it to all the current sentinel values. If any of the values do not match, we consider the item invalidated and we ignore it even if it still exists in the cache.

What is great about using this is that the overhead of managing lists of items in a group is gone, because we no longer track what items belong to a group. Instead we have a much smaller overhead to keep track of the groups that an item might belong to. But more importantly, we can very quickly invalidate all items in a group simply by doing a fast, atomic Couchbase Increment operation on a sentinel. This will instantly invalidate any items that belong to that group the next time a read attempt is made for any items in that group.

So our final implementation of the Read and Write functions looks like this:

/// <summary>
/// Writes out a serialized object to the cache
/// </summary>
/// <param name="o">Object to write to the cache</param>
/// <param name="group">Cache group that this item belongs to</param>
/// <param name="key">Key name for this cache item</param>
/// <param name="expiresAt">Optional date/time stamp that the cached item will expire at</param>
public void Write(
    object o,
    string group,
    string key,
    DateTime? expiresAt)
{
    // Get access to the cache
    var client = GT.CBCache;

    // Determine the keys we need
    var fullKey = group + '/' + key;
    var lockKey = fullKey + ".lock";

    // Store the item in the cache
    if (expiresAt == null) {
        client.Store(StoreMode.Set, fullKey, o);
    } else {
        client.Store(StoreMode.Set, fullKey, o, (DateTime)expiresAt);
    }

    // Clear the lock to let other processes into the cached item
    client.Store(StoreMode.Set, lockKey, false);
}

/// <summary>
/// Reads a serialized object from the cache
/// </summary>
/// <param name="o">Place to return the object read from the cache</param>
/// <param name="group">Cache group that this item belongs to</param>
/// <param name="key">Key name for this cache item</param>
/// <param name="dependencies">List of dependent sentinels this item is dependent on</param>
/// <returns>True if the item was read from the cache, false if not</returns>
public bool Read(
    out object o,
    string group,
    string key,
    List<string> dependencies = null)
{
    // Get access to the cache
    var client = GT.CBCache;

    // Determine the keys we need. The key of the item we key off the primary group
    var fullKey = group + '/' + key;
    var metaKey = fullKey + ".meta";
    var lockKey = fullKey + ".lock";

    // Set the default return values
    o = null;

    // Loop around trying to read the cache until either it is valid, or we get the write lock for the cache. This way
    // we guarantee that we will only ever have one writer actually generating the cache data at a time.
    CasResult<bool> casResult;
    CasResult<bool> locked = new CasResult<bool>();
    List<string> priorSentinels, sentinels;
    do {
        // Loop until the lock is clear in case another instance is writing the cache. We time out after 10 seconds
        for (int i = 0; i < TimeoutLoops; i++) {
            // Check if someone has the lock
            locked = client.GetWithCas<bool>(lockKey);
            if (!locked.Result) {
                // Not locked for writing, so we are good to go
                break;
            }

            // First try to get the cached item. If it is still present then we can simply serve up stale
            // data while one thread is writing the new cache data is being generated
            if ((o = client.Get(fullKey)) != null) {
                return true;
            }

            // Sleep for 100 milliseconds to allow the lock to get cleared
            System.Threading.Thread.Sleep(TimeoutSleep);
        }

        // Read the items we need from the cache with a multi-get
        var keys = new List<string> { group, metaKey, fullKey };
        if (dependencies != null) {
            // Add in any dependencies that this item depends on if required
            keys.AddRange(dependencies);
        }
        var results = client.Get(keys);
        priorSentinels = results.ContainsKey(metaKey) ? (List<string>)results[metaKey] : new List<string> { "0" };
        sentinels = new List<string> { results.ContainsKey(group) ? (string)results[group] : "0" };
        if (dependencies != null) {
            // If we have dependencies, make sure we get the sentinel value or set it to the default if not present
            foreach (var dependency in dependencies) {
                sentinels.Add(results.ContainsKey(dependency) ? (string)results[dependency] : "0");
            }
        }

        // Check if any of the sentinels indicate this item is invalidated. If any sentinels don't match the values
        // last time the cache was written then this item is invalid
        if (priorSentinels.SequenceEqual(sentinels)) {
            // Check if we got the cached item
            if (results.ContainsKey(fullKey) && ((o = results[fullKey]) != null)) {
                return true;
            }
        }

        // Check if we got the lock. If not then someone has the lock and we timed out just continue on. The write
        // code will eventually clear the lock and let other threads in. But we log this so we know it happened.
        if (locked.Result) {
            // NOTE: Add code to log this error to an error log in here...
            break;
        }

        // Attempt to set the lock using a Check-And-Set operation
        casResult = client.Cas(StoreMode.Set, lockKey, true, locked.Cas);
    } while (casResult.Result == false);

    // Store the sentinels for this item before it gets written since we know the cached item will now
    // be valid after the write completes. But we cannot write the value in the write code, because we
    // have to make sure we write the sentinel values we read above in case any of them have now been changed
    // (in which case the next read for the cache would be invalid also). So it is better to write them here
    // than to expect the caller to pass them to the write function.
    client.Store(StoreMode.Set, metaKey, sentinels);

    // Return false indicating that the cache is invalid and we have the write lock
    return false;
}

/// <summary>
/// Resets all the cache entries for this specific cache group
/// </summary>
/// <param name="sentinel">Cache group to clear</param>
public void InvalidateCache(
    string sentinel)
{
    // Increment the sentinel for this group to invalidate old entries. We start with a default value of 1
    // the first time a value is written since by default when the value is not present we assume it is 0
    GT.CBCache.Increment(sentinel, 1, 1);
}

As you can see from the code above, the Write() function is now much simpler as it no longer has to update a list of items in the group when it writes an item to the cache. The read function on the other hand is more complex, as it is passed in an optional list of extra sentinel values that a cache item may depend upon. Once the read function has obtained the cache lock for the item, it then uses a fast Mult-Get operation in Couchbase to read all the values it needs, including the cached item and the value of all sentinel keys it depends upon. If the sentinels don’t exist, we give it a default value of “0″ which will happen the first time that a sentinel is used and it has never been invalidated. Once the read operation has a list of current sentinel values and prior sentinel values we can compare if the two lists are identical and if they are, return the actual cache item if it was present.

Also note that the Write() function never writes the current sentinel values to the meta data item, but rather the Read() function does that right before it exits and returns false. We know that whoever is calling the Read() function will immediately call the Write() function if Read() returns false, and we also know that the Read() function has the write lock for the cache item when it exits also.

Using the new code is pretty much the same as before, but we can also pass in an optional list of other sentinels we might want to use to invalidate that particular cache entry:

string group = "myGroup";
string key = "myItem";
object o;
if (!Read(out o, group, key, new List<string> { "secondGroup" })) {
    o = "Something to cache";
    Write(o, group, key, DateTime.Now.AddMinutes(5));
}

// Do something with object o

Finally when we decide we wish to invalidate a group of cache items using a particular sentinel, we use the InvalidateCache() function above which just increments the sentinel value by 1. Note that we always start incrementing with a value of 1 in this function since we assume the absence of a sentinel value is equivalent to a value of 0 in the read code. So the first time we write a sentinel value, it must be 1 to ensure it actually caused any items covered by the sentinel to be invalidated.

That’s it! Have fun using Couchbase and I hope this blog will help you build a better caching solution for your own software.

Faster web site == slower MySQL!

One of the things we did in our quest for performance was to re-compile all our PHP code into .NET using Phalanger. This definitely made the web site faster and lowered the CPU load on the Windows web server substantially, but we quickly discovered that having faster code on the web server had the side effect of causing more performance problems with MySQL, not less. Clearly the MySQL server was not able to keep up with the amount of traffic being sent it’s way during normal operation, which was odd because when these issues were happening there was really not a whole lot going of activity on the MySQL server.

Caching won’t solve core DB issues

The first thing we did in an attempt to solve these performance problems was to revisit our caching system, and re-implement it using Couchbase. This produced significant performance improvements in the best case when items were in the cache, but the worst case performance problems remained.

Part of our caching architecture is to ensure that only one web server thread is actively generating the cached pages at a time to ensure that the load on the MySQL server is kept as low as possible. It is much faster to have only one thread generating the SQL select traffic to the MySQL server and to have the other threads wait, rather than to have them all send the same SQL select traffic at the same time. In the best case all the threads would run at the same speed anyway, however if you have 100 threads all asking the MySQL server to do the same thing in parallel, it is going to run about 100X slower than just one thread making those calls! So when another thread tries to get a cache item that is expired and another thread needs the same cache entry, the second thread will end up blocking with small sleeps in a loop until either the cache entry is valid again, or we hit a timeout after 10 seconds. Ten seconds is a LONG time for a web page, so it seemed like a reasonable timeout that should never happen in practice.

In theory if the MySQL server was acting up and behaving really slow, we might be be getting cache timeouts. So we went ahead and added some logging to the code to keep track of any timeouts that did occur. Once we enabled the logging on the live site we could clearly see the caching code was timing out hundreds of times a day! Which means that hundreds of times a day, pages were taking longer than 10 seconds to render, which is totally unacceptable. Clearly when the web site code had to reach out to MySQL to get the data to generate the web page content, it was slow. This has to be our performance bottleneck, so we kept looking.

Slow performance when DB server seems idle?

The real question was why? Looking at the first graph below taken while the web site was responding slowly, you can see that the MySQL server is not really doing a whole lot. The CPU usage was very low sitting around 5% with spikes up to 10%. The number of MySQL connections was also low hovering around 150 connections on average, and the database activity was not really all that high at an average of maybe 180 statements per second. By all counts the MySQL server should be blazing fast, but it was not! And we knew that the web server was quite slow during the peaks in the CPU activity from 13:00 to 14:00 and again from about 15:10 to 15:40.

CPU usage using MyISAM

MySQL connections usings MyISAM

Database activity using MyISAM

Excessive table locks will KILL performance!

There had to be something else going on so we kept looking at the data with the MySQL Enterprise Monitor. One thing that caught my eye was the number of table locks going on, and here is where things start to get interesting. If you look at the graphs below you can see that there were a lot of table locks that were all grabbed immediately, but creeping in at the bottom of the graph are table locks that caused a wait condition. Then if you take a look at the table lock to wait ratio you can see big spikes in the graph that correspond to times when the web site performance was suffering! Aha! You will also notice that at the same time this happens we see spikes in CPU usage where it doubles from the low 5% to about 10% (visible in the graph above).

Table locks using MyISAM

Table lock wait ratio using MyISAM

So the core problem we had was the excessive amount of table locking going on, or more importantly, the fact that way too many threads were getting stuck waiting to get a table lock! This clearly explains why the DB server appeared slow, when in fact the CPU load indicated not much was going on. Clearly this was the cause of our MySQL server acting like it was overloaded, when nothing of significance was really going on.

MyISAM tables don’t handle update load well

So we spoke to Oracle support, and they pretty much told us the problem we had is that the MyISAM table type we are using just does not work well when you have large tables with a mix of selects and updates going on. One of the core issues with the MyISAM table type is that it does not support row level locking, but rather will lock the entire table when you need to do an update on any record in a table. Or worse, if you have deleted a record from a table which leaves a hole in the table file on disk where the deleted record was, it will also lock the table for inserts also as it will try to fill in that hole. You can work around the insert problem by setting the concurrent_insert feature to ALWAYS, which will always insert new records at the end of the MyISAM table. We had already implemented that based on Oracle’s recommendation previously, however this simply does not solve the problem of table row updates. Some of our tables are really large so the updates can take a while due to indexing, so select traffic was getting blocked by updates on tables that also get lots of select traffic. Clearly we had reached the end of the road for the venerable MyISAM table type!

InnoDB to the rescue!

One of the biggest advantages of the InnoDB table type over the MyISAM table table (other than support for transactions) is that it can properly handle row level locking. This means that if you need to update a record in the database, the only time another thread that needs to read from the same table will block is if that thread needs to read the exact same record. Otherwise there is no lock contention and all of the requests can be handled nicely in parallel.

We had been planning for some time to convert our entire database over to the InnoDB table type format but this was a significant undertaking that would require our site to be down for an extended period of time, and it would require us to re-work all our existing database backup strategies. But it was sounding like we really needed to make the change, so we set about putting together a plan to make it happen. One of the first things we discovered was how to enable a file per table for the InnoDB databases, so each table would end up in a separate file just like our MyISAM tables. This makes it much, much easier to manage backups and deal with smaller parts of the database at a time for development purposes.

The next part was to schedule the downtime and run the upgrade. It ended up taking an entire hour of downtime on the web site to complete the upgrade on all the applications that needed it. Once the upgrade was done, it was really clear the next day during normal business hours that this was the right move and we should have done this a long, LONG time ago! As you can see from the graphs below, we started the upgrade process around 18:45, and it was all finished around 19:45 and performance normalized around 21:00 or so. The total CPU usage was much lower than it was prior to the upgrade, but more importantly it was fairly constant without the odd peaks we were seeing before. The total MySQL connections dropped off to about 100 connections after the upgrade, and the total database activity saw a small drop after the upgrade.

CPU usage using InnoDB

MySQL connections usings InnoDB

Database activity using InnoDB

The big difference can be seen in the table lock to wait ratio. The total number of table locks did not really drop that much, but the table lock to wait ratio went to ZERO. More importantly, the web site performance was fantastic all day long! Now all the work we had done previously to improve the performance by compiling the web site with Phalanger and implementing better caching with Couchbase was finally paying off.

Table locks using InnoDB

Table lock wait ratio using InnoDB

Conclusion

So in conclusion, if you are still using the MyISAM table type and you still believe (as we did) that MyISAM is one of the fastest table types for most web sites where there is a mostly lots of select activity going on, think again! Clearly the InnoDB table type is significantly faster when you have large amounts of table data and even have a small amount of update traffic going on. And I would suspect that for small web sites, the performance difference between the two table tables would be negligible at best. One thing I know for sure is that I won’t build another database driven web site using MyISAM – our default table table will now always be InnoDB.

New Hardware

When we first started noticing performance issues we were running our entire web site all on one server, running under Windows Server 2008. The web site was running PHP for most of the front end, ASP.NET MVC for a lot of the back end and WCF web services for client/server warehousing applications. The initial thinking after examining the code was that the web server was just getting overloaded and we simply needed more horse power. So we ended up moving the database off the Windows machine that hosted the web site and the database, and onto a stand alone Linux machine just running MySQL. We also made sure the new machine had significantly improved hardware with more memory and more CPU. We also invested in an enterprise license for MySQL with Oracle, to get their reporting tools and technical support. After moving MySQL to the new hardware, things seemed a little better, but the problem was not solved. We still had terrible worst case performance issues where the site would stall.

Recompile PHP code with Phalanger

Around the same time we were still having performance problems with the web site we had also been experimenting with a fantastic tool called Phalanger, a PHP compiler for .NET. Since we had already been implementing all our new back end web site functionality in C# using ASP.NET MVC for the web site and WCF for our handheld warehousing computers, it made a lot of sense to work on recompiling all the PHP code into .NET so it could all run on the same platform. Not to mention I wanted to get off PHP and move towards .NET anyway for performance reasons, which I documented in one of my previous blog posts. So around March this year we finished up the project to port to Phalanger and launched the new compiled web site. I was expecting to see big performance gains since on our development and testing machines we could see Phalanger was much faster than regular PHP.

But the problem was still not solved! Although the Windows server has much lower CPU usage than it did running on real PHP, we were still having worst case performance problems where the site would slow down for no apparent reason.

Problems with Nightly Scripts

Once we launched the web site live with Phalanger, we started running into another issue that had not cropped up before. Something was going on such that the web site would basically crash and need some CPR in the middle of the night when our nightly scripts fired up (big thanks to Rackspace Rackers for restarting the web site at 3am multiple times when it died over the last 6 months!). To help us figure these issues out we got the Oracle MySQL monitoring tools installed and could see that during the night when the nightly scripts started, we would see the number of connections from the web site to MySQL start to sky rocket, and the CPU load on the MySQL server would go nuts.

One of the biggest differences between running real PHP on Windows and running PHP compiled under Phalanger, is how IIS handles scaling the load. With real PHP it runs under FastCGI and there is a limit on the number of PHP processes that will be spawned to handle the load based on how you configure FastCGI. We found our site worked well with up to about 100 PHP processes running, but things would fall apart if we let it go much past that value. However with IIS running an ASP.NET web site, it will automatically keep adding application instances to the application pool based on how slow the site is responding to incoming page requests until it is maxing out the CPU and memory on the machine. There is no way for us to control the maximum number of ‘applications’ that IIS will have running in the application pool.

The problem with that is all those new application instances then start requesting connections to MySQL and executing more and more SQL queries for the pages they are trying to serve! This then causes MySQL gets even more overloaded and starts responding even slower, which then causes IIS to spin up even more application instances. This feedback loop continues until eventually you get a site crash as the IIS applications can no longer get MySQL connections and MySQL is working so hard nothing can get done. We never had this problem with real PHP because we would only ever have a maximum of 100 PHP instances running so although the web site was really slow in the middle of the night, neither MySQL nor IIS would completely overload and cause the site to crash.

This was all pretty obvious once we had the MySQL Enterprise Monitor tools installed, but we just did not know what to do about it. With the graphs below you can see the problem clearly on April 25th, where the nightly process ran from 03:05 to 03:19. You can see the number of connections to the MySQL server start to head steadily towards the maximum, the CPU load on the server get higher and higher and the query cache start to constantly miss. It is clear from the graph that the MySQL machine is completely overloaded, as the load average goes up to around 30, when the machine only has 12 physical cores (24 with hyper threading). Pretty much any load past about 20 means the machine is running full tilt. The end result of this is aborted SQL connections and/or timeouts. We tried playing around the maximum number of connections, but all that did is cause more load on the MySQL server and eventually IIS would fall over.

CPU load on April 25th

Load Average on April 25th

Query cache misses on April 25th

MySQL connections on April 25th

So for the night of April 26th we optimized the code somewhat and nearly halved the number of iterations for the driving loop (~45K down to ~23K), and added a very small micro sleep to each iteration (about 1/40th of a second). This actually produced some completely odd results which was not what we were expecting. From the graphs you can see the nightly script in question ran from 03:07 to 03:39. Not only did it take twice as long, but the load average went all the way up to 55. Clearly adding a sleep in there simply slowed everything down, but did nothing to lower the server load and in fact made everything worse, not better.

CPU load on April 26th

Load average on April 26th

Query cache misses on April 26th

MySQL connections on April 26th

Solution: MySQL Low Priority Updates

After doing some more thinking after the failure of our changes on the night of April 26th it occurred to me that part of the problem was that the nightly scripts run full tilt and were very update heavy. We needed some way to be able to ‘slow down’ the MySQL server, like a feature similar to the UNIX nice command but for our background MySQL process. The solution to this problem came in the form of the LOW_PRIORITY keyword for the MySQL update operation, or the use of the low_priority_updates variable for all queries on a connection. The idea behind this feature is that if you issue an update operation that is considered a ‘low priority’ and mark it as such MySQL will schedule the update operation to happen ONLY when all pending select operations have completed. This essentially gives the effect of making the updates happen in the background, so that all the foreground ‘web page’ tasks get to complete and the server won’t get overload.

So for the night of April 27th, we removed the sleeps and added in the low_priority support for the critical queries. The nightly process in question ran from 03:04 to 03:16. You can see that this time the CPU utilization stayed really low the whole time, the load average barely got over 1, the cache hit ratio never dipped below 60% and the total number of connections never went above 75. The entire process actually finished faster than the original code now, with no adverse load on the MySQL server (although the iterations of the loop are effectively half what they used to be, so the low priority update versions of the nightly scripts took almost 2x as long as before).

CPU load on April 27th

Load average on April 27th

Query cache misses on April 27th

MySQL connections on April 27th

Conclusion

So in conclusion it should be pretty clear that if you are using MySQL and you need to run nightly processes that hit the database pretty hard, it is a really good idea to make sure all that code runs with low priority updates to ensure any select operations for web pages will get priority and not get blocked out. Not to mention that enabling low priority updates keeps the load on the MySQL server down which then allows it to do the same amount of work much more efficiently. Just be aware that if your MySQL server is totally overloaded already and you enable low priority updates, the update operation may get blocked indefinitely and never complete. If you have that problem you may need to consider moving select traffic off the database server onto a dedicated slave server.

File Systems are SLOW!

One of the biggest problems we had with the file system cache was simply that it was too expensive to actually get rid of items that had expired from the cache. So our simple cache system would use the file timestamp to determine if a file was old and had expired and hence needed to be refreshed. But nothing would ever come back to clean up old ‘expired’ entries that had not been looked at for a while so the file system would end up with hundreds of thousands of tiny cache files on it over time. Thankfully that problem is neatly solved with Couchbase caching as it is entirely memory based. Nothing is stored on disk, and if you use more memory than you have allocated, old stuff starts to get tossed out automatically (and efficiently).

But another major problem we had was that the file system was really, REALLY slow when it comes to deleting files and dealing with directories full of LOTS of files. And this is an area where Windows is just plain bad compared to Linux and other UNIX file systems. Having a really slow file system leads to the problem of taking a long time to delete cache files from disk, if we ever needed to purge the cache. Purging the cache which was commonly needed after a major site upgrade, and it would literally take 30 minutes or more to delete the files. We ended up changing our cache purging code to rename the old cache directory so a new one would be created to take it’s place, and then deleting the old files in the background. We even went so far as to put the cache files into a Windows FAT32 ramdisk, but it still took forever to delete the files. And once again that problem is neatly solved by Couchbase because it is all memory based. A couple of mouse clicks is all that is needed to purge the cache, or a single call to the client library can also purge it instantly.

Couchbase to the Rescue

So we recently went live with our new site caching system using Couchbase, and the results are excellent. The overhead of getting a cache entry from Couchbase over the network from an entirely separate machine is actually less than the overhead we had in managing all the little files on disk. But more importantly we now have a lot of flexibility to automatically expire items and purge the entire cache quickly if we need to.

The first part of the problem we had is that most content that is encoded in latin1 that most people probably think is ISO-8859-1 is not actually using that encoding, but is rather using the Windows-1252 character set encoding, which is a superset of ISO-8859-1. For this reason, most browsers will happily go ahead and accept pages that present themselves as ISO-8859-1 but in reality they treat the page as the Windows-1252 encoding so that all the extended characters like smart quotes (“ and ”) and the Euro symbol (€) work correctly.

Now the second part of the problem we had is that all our MySQL databases are encoded in latin1, and in fact MySQL does not have any character set called ISO-8859-1 or Windows-1252 at all. So although ISO-8859-1 is commonly known as latin1 according to Wikipedia, in the case of MySQL it is really Windows-1252, or what they call the cp1252 encoding. Specifically they state:

"latin1 is the default character set. MySQL’s latin1 is the same as the Windows cp1252 character set. This means it is the same as the official ISO 8859-1 or IANA (Internet Assigned Numbers Authority) latin1, except that IANA latin1 treats the code points between 0x80 and 0x9f as “undefined,” whereas cp1252, and therefore MySQL’s latin1, assign characters for those positions. For example, 0x80 is the Euro sign.”

So why exactly is this a problem, if the browsers will happily accept a page presenting itself as ISO-8859-1 and let characters from the Windows-1252 character set work properly? Normally it would not, because as long as we can take the string from the MySQL database and send them directly to the browser, or take string from the browser and stick them directly into the database, everything works as expected (even though the encoding is not technically legal). And in fact until recently, this never caused a problem because the pages on our web site were written in PHP and internally PHP stores the strings in the native Windows-1252 encoding. Hence reading a value from the database and sending it to the browser works just fine.

Enter Phalanger (again). Phalanger is a compiler for PHP that compiles the PHP code into native .NET code so it can run natively under ASP.NET on the web server or as native .NET executable if run from the command line. Apart from being a lot faster than running it under the normal PHP interpreter, it also allows for interoperability between PHP code and native .NET code such as C# etc. Now the problem is that when running under .NET, strings are represented in Unicode format (UTF-16 to be exact). So when we read a value from the database under Phalanger, it happily converts our strings from the database latin1/Windows-1252 encoding into a native .NET System.String type, which is encoded as a Unicode UTF-16 format string. Since the MySQL database is really using the Windows-1252 code page, the strings all get converted correctly to the appropriate characters in Unicode. Hence when it takes the Unicode strings and sends them to the browser, it has to convert them to the correct format to put into the HTML page, which we had previously set up as ISO-8859-1 since we always used that with our existing PHP pages. Now the problem is that .NET is very strict about character encoding and since we told the ASP.NET server to use the ISO-8859-1 encoding, when it came across invalid characters like smart quotes mentioned above it simply replaced them with a ? character to indicate that it could not convert them!

So after a lot of Googling, head scratching and testing, it finally became clear the ISO-8859-1 != Windows-1252. And also ISO-8859-1 != MySQL latin1! The end result is that once we changed the character encoding for our Phalanger pages to Windows-1252, everything magically worked .

The problem I see with the ‘Buy Local’ campaigns is that they generally leave out some important issues. Simply spending your money on a local business JUST because they are local, does not make any sense. Not EVERY business deserves your dollars, especially when those local businesses don’t provide good customer service or after sales support. One of the common misconceptions out there amongst small bricks and mortar shops is that buying online is an impersonal experience, and online stores can never provide the same level of customer service as a local small business. Nothing can be further from the truth! Modern online businesses have highly trained customer service representatives to take care of their customers via phone, email or live chat. More importantly, online businesses know that after sales service and customer support are critical to remain competitive, because customers simply won’t come back and shop with you again if you don’t treat them right. There are plenty of other businesses out there for them to spend their money with.

So the core point I want to make is that if a local business wants customers to spend their money with them, they need to understand that they HAVE to start providing a high level of customer service and after sales support. This should not be something new, as local businesses have always had to do this, but before the internet the local businesses had a more captive market as customers did not have much else in the way of options, especially in smaller towns such as Chico where I currently live. With the advent of the internet, the local business now has to contend with a whole new range of online businesses that are competing in their own back yard. This is similar to small businesses lining the streets in a highly competitive shopping district (such as down Bridge Road in Melbourne, Australia). With so many businesses all selling the same products in close proximity, if they DON’T learn how to treat their customers right the store next door is going to steal their sales, and they will go out of business. This is effectively what is happening now with local businesses as the online business is the new business that just moved in next door.

There is a good article from Carol Tice on Entrepreneur.com discussing what makes shoppers buy local. In this article she says:

Personal service. I have yet to talk to a small business owner who thinks they don’t have great customer service. But customers tell another story. On review sites and local forums, the fangs come out about rudeness, sloppy follow-up, or a just-don’t-care attitude.

Personal service, or just plain good customer service is critical for a business to retain customers whether they are local or online. But my experience with many local businesses in recent years (and alas some online ones), is that sometimes customer service seems to be an afterthought and something the business owner would prefer not to do. Once the sale is done, it is done. If the customer has an issue, they explain why they cannot help you and send you on your way. The end result is that single sale was saved, but the business just lost that customer and they likely won’t ever come back. Worse, if they are really upset they may tell all their friends via Facebook or Twitter about their experience, and THEY then won’t spend their money with that business.

So it is pretty clear that if any local business wants customers to spend their money with their business, they need to EARN that business. They should not ever simply EXPECT that business because they happen to be local.

An expert finds that when your new iPad tells you that it’s fully-charged, it’s not even close to being charged up. Are we looking at an Apple iPad “Batterygate?”

The core issue they identified is that “the power actually drawn by the AC Adapter and first found that the new iPad continues to charge for up to 1 hour after it claims to reach 100%. This affects the battery run time if you stop charging when it says 100%.” The article goes on to discuss the issue and wonders why the new iPad lies about the charge status, when the battery is not yet fully charged.

Dr. Raymond Soneira, president of DisplayMate, the world’s leading display and display tuning company, theorizes that “There is something wrong with the battery charge mathematical model on the iPad. It should not say 100% until it stops recharging and goes from the full recharging rate of about 10 watts to a trickle charging rate of about 1 watt. Otherwise the user will not get the maximum running time that the iPad is capable of delivering.”

After reading these blog posts and being involved in the RC hobby industry where we use Lithium Polymer batteries day in and day out, I felt I had to comment on this issue to explain a few things about Lithium battery technology. Although the new iPad has a 4.1V 11,560mAh Lithium-Ion battery and not a Lithium Polymer battery, the charging characteristics of these batteries are virtually identical. The primary difference is Li-Ion batteries have a lower overall voltage (4.1V fully charged versus 4.2V for Li-Poly) and are generally more stable for low current draw applications like phones and tablets. Li-Poly is generally used where you need high discharge rates, such as in RC helicopters, airplanes and cars.

So coming from the RC hobby industry, it is very well known that the charging cycle for these types of batteries is done in two phases. The first phase is the constant current (CC) phase, and is what is considered the ‘fast’ charging phase. During this time the battery is charged at the maximum current possible until it reaches the maximum voltage of the cells, which for Lithium-Ion batteries is 4.1V per cell. When this occurs the battery is considered at about 80% of the total charge capacity, but it has in fact already reached 100% of the maximum voltage. If you put a volt meter across the battery at this point in the charging cycle, with the charger still attached, it would read 4.1V.

After the first phase is complete, the charger changes to a constant voltage (CV) phase where the current tapers off until it effectively gets to zero, and the battery is now fully charged. During the entire time the battery is being charged in the CV phase, the battery voltage remains at 4.1V per cell but the current tapers off. In general this last 20% of battery capacity takes a LONG time to complete, and since the current tapers off it actually goes from 80% to 90% capacity in a relatively short period of time. Then it takes it takes longer and longer to get to 95%, then to 97.5% etc due to the fact that the charge current is tapering off.

Now the catch in all this is that in order to measure the effective capacity left in the battery when the battery is NOT on the charger, you need to look at the voltage of the battery and run some complex math based on the discharge curve of the battery to estimate how much capacity is really left. Now most Lithium batteries under a small load will not have much voltage drop so they will measure pretty close to the maximum voltage when they come right off the charger (and compared to RC models which can run a battery flat in 5 minutes, 10+ hour discharge times is a SMALL load!). Now when the iPad is attached to the charger, once the charger changes to the CV phase at the 80% charge point, the voltage of the batteries will ALWAYS show full 4.1V per cell because the charger is currently pegging the battery voltage at that value. So for that reason, once the battery reaches 80% of its maximum capacity, the battery meter on the iPad is going to always show 100%! However if you remove the iPad from the charger as soon as it first reads 100% on the battery meter, it would not actually have reached full capacity yet so the voltage would immediately drop.

For the most part the battery meter is really designed to let you know and approximate capacity left when the unit is NOT ON THE CHARGER! So it will always give some odd results when the battery is being charged and is not under load. So it seems to me this is all a lot of hoopla about nothing, since what is being described is pretty standard stuff for charging Lithium batteries, so I hardly think we have a ‘Batterygate’ or ‘Chargergate’ type situation on our hands. I have no doubt most (if not all) other tablets and phones that all use Li-Ion batteries do exactly the same thing.

Talk back below and let me know what you think!

Investigating the problem had me even more confused, because I was able to display the same list of packing slips in a desktop browser almost instantly from the live web site, yet it was take 4-5 seconds for the desktop app to download an XML document containing the same data feeding off a local web server on my development machine. It should not be taking that long! To get to the bottom of the problem I added some profiling information to my page to find out where it was spending all the time. What I found was rather startling! The page was able to get through the entire process of getting all the information needed from the database and put it into a PHP array in a fraction of a second, and then the other 95% of the time was spent inside the Zend Framework, turning the PHP array into an XML packet to send back to the client! Not good.

At that point I called in Steve, one of my programmers and said ‘What can we do to fix this?’. Steve of course, simply said; “Why are you using Zend Framework? Isn’t there XML-RPC server extension for PHP?”. Yes! There really is. So with a flurry of key strokes we ripped out the Zend Framework XML-RPC server code and changed it to use the extension for PHP. Magically the code that used to take so long was now humming along nice and fast! It was at that point I realized that PHP was a really, REALLY bad choice of language for complicated algorithms or any heavy lifting and business logic (Facebook figured that out also; hence the HipHop project was born). I knew PHP was a scripting language, but I thought the Zend Optimizer built into Zend Server would make it run fast enough for our use. How wrong I was! PHP is a reasonable solution for putting something together quickly and stitching together HTML for a web page, and it works OK as long as you can off-load the heavy lifting to a PHP extension written in C (like the XML-RPC extension). But for writing complex business logic and algorithms? Not a good choice.

Suddenly I was in a quandry because I needed to get moving on our new back end warehousing tools so we could start using bar code scanners, but coding the server side in PHP was not going to make any sense. I needed something better. Unlike Facebook, who probably has millions of lines of PHP code, I had the luxury of tossing out what I had started on and starting over with a new architecture. I spent some time investigating Java and as much as I like the language, the tools just did not appeal to me; after having used Zend Studio for some time which is based on Eclipse, I did not particularly want to be developing using Eclipse for my IDE. So I took a look at C# and fell in love with the Language. And alas, much to my dismay, I found Visual Studio to be an amazing piece of software and the best IDE I had used for a long time. So I was off an running with C# to develop my server code.

Pretty quickly I realized that C# has amazing support for SOAP built right in, and it was super simple to build a SOAP server using WCF and just as easy to build a client for it in C#. It was at that point that I gave up on C++ and Qt, and ended up building our client software in C# and Windows Forms (and eventually Windows Mobile for our Intermec CK71 handheld computers).

Anyway, to cut a long story short once we had decided upon C# and WCF for our SOAP server we had to host it somewhere. Rather than deal with setting up Mono on our Linux server and it’s limited WCF support, I decided to move our entire web server over to running on Windows. After all, PHP and MySQL seemed to run fine on Windows as we had been doing all our development on Windows machines for some time. So we made the switch, and started the process of writing all our new C# and SOAP code. When I had re-implemented the same program from C++/Qt with a PHP XML-RPC server into a Windows Forms application written in C# and using WCF to communicate to the server, it was significantly faster again than the C++ version using the XML-RPC extension in PHP so I knew I was on the right track to get off PHP for the server.

Now, after having run PHP on our Windows servers for probably 2 years, I can say that Windows is NOT the best platform to run PHP applications. I don’t know if the problem is IIS itself, or the Microsoft WinCache solution (forget about the commercial Zend Server on Windows; it crashed our server multiple times and Zend could not fix it). But it hangs up on a regular basis and in fact some applications (mostly forum software) were so bad we simply ended up moving them all to a cloud Linux based machine and then proxy the requests over there from our IIS server. It looks like they run on our main server, but in reality they don’t .

Our primary web site still runs mostly on PHP for the front end and we are in the process of re-writing a lot of the back end code in ASP.NET MVC and moving it off PHP. We are also in the process of getting ready to re-write parts of the front end in C# as well, since all our new code is written in C# so I then started looking around to find ways to integrate PHP and C# code on the same platform. I ended up building my own shared session architecture so that PHP pages and ASP.NET MVC pages can share session data and we can transparently move from one to the other, but there was still no way to actually call PHP code from C# or vice versa.

Enter  the Open Source project Phalanger! I forget exactly how I came across this particularly nifty piece of software (I think Steve sent me a link to it one day), but Phalanger is basically a compiler for PHP. Kind of like HipHop for PHP, but rather than compiling the PHP code into native code, it compiles it into .NET code which can then be executed natively on the server, just like C# code. By compiling the PHP code into .NET assemblies, not only is the code significantly faster than native PHP but once it is compiled into .NET code you can share the code easily with C# code. It is possible to load PHP code and call PHP functions from C# code, as well as have PHP render the entire page but be able to call across to .NET assemblies written in C#.

We have now been in the process of getting our web site to compile in Phalanger since November and are just about to launch it live. During the process we have relied heavily on a support contract with Devsense, who have been fantastic at adding missing features and helping to fix bugs in Phalanger so we can get all the features our sites needs implemented so it can run. With any luck our web site will be running entirely on .NET and compiled with Phalanger sometime next week.

So to wrap up this blog post and tie it into the title, running PHP on Windows really is silly! Native PHP runs best on Linux. But if you really want to run PHP apps on Windows (or on Linux using Mono for that matter), you really should check out Phalanger. And if you are a PHP developer and wish you could call .NET code or are a .NET developer and wish you could use some PHP libraries you found, give Phalanger a good look!

My experience on the programming side ranges from C/C++ and assembly programming and low level device driver programming to E-Commerce and web site programming in PHP, C# and ASP.NET MVC. I expect I will be writing a lot about C# and ASP.NET MVC programming, as well as database programming on these pages. My current platform of choice is C#, ASP.NET MVC and the MySQL database using the PetaPoco micro-ORM layer. Although we have a lot of PHP code on our site, and we still maintain it and add to it, I personally can’t stand programming in PHP at all and one of our goals is to get rid of PHP completely from our systems over the coming years and replace it all with C# and ASP.NET MVC code.

I am also an avid techie and I love new gadgets so you will probably see me posting about new gadgets I might come across that you might find interesting as well. As anyone who knows me will tell you, I am an avid Apple product fan and have has Apple products since back in the Apple II days. I even use primarily Macintosh computers for all my work, although at this point I am still stuck running Windows on my Macs to get real work done. I wish someone would come out with a hardware graphics accelerator for Parallels on the Mac, so I can run windows apps under Mac OS at native speeds! If I could, I would only use Mac OS as my primary OS, but the emulators are just too slow for real work for me at the moment. Anyway, the point is, don’t be shocked if you see me writing stuff about Apple products on these pages also. I have a lot of them, and I love them, so I will probably write something about them from time to time

Well, let’s get down to business and start blogging, and I hope you enjoy this site!