Evil Magento Extensions I: Email marketing

Magento Commerce is a really good source of extensions, some of them are excellent, some of them are cheap or even free, some of them are not great… and some of them are what we call “Evil Magento Extensions”!

First of all we would like to apologise if anyone gets offended, is not the purpose of this post, but to contribute to Magento ecosystem by sharing our experiences and always aiming to improve Magento community.

We always want the best Magento Extensions for our store and we want to spend as less money as possible, but depending on your specific shop needs you might want to think twice before installing one of those Evil Magento Extensions.

There are way too many extensions out there and we obviously haven’t tried them all, but from our experience usually the ones provided by email marketing platforms are the more interesting ones. It is very common to use a third party service to send the transactional emails, and they mostly tend to provide a free (and evil) Magento Extension which is pretty much “plug & play” into your store, so that you don’t have to worry about anything…. or maybe should you?

The usual case scenario is that when a user places an order, or creates a new account, an email is sent, and we’ve found out that most of this extensions relay on Magento Observers (ie. customer_register_success, sales_order_place_after), or even rewrite the Mage_Core_Model_Email_Template class in order to push the data to their servers. That’s fine until here, if it wasn’t because almost all of them do it synchronously, ie. on the same request made by your customer. This means that for instance, when your user is placing a new order, she has to wait until your server communicates with your email platform: sending the data of the order, and waiting for a response. This is a “standard” process that usually doesn’t take longer than a few milliseconds/seconds which is fine… hang on, are you sure?

The astonishing truth is that this process can (and will) hinder your customers experience in many different ways, but you probably already know about them if you are reading this post. A request from your store to an external server can take any time from few milliseconds to several seconds, minutes, or even hours! and many things can go wrong while that’s happening, often leading to an error displayed to your customer, or just to another abandoned cart, tired of waiting.

Ok, I might be exaggerating a bit, there would surely be a timeout error somewhere between this process that would prevent a request to take hours, but sadly, we have seen MANY cases where a request would hang for several minutes. Of course, it’s not the usual case, but it will happen sooner than later. It happens to Amazon, to Facebook, even to Google, it happens to the best of us, it might be due networks issues, power failures, too many concurrent users, or whatever reason.

The point is that your store, your customers, will be affected and if they have a bad experience they will not come back. Bear in mind that it’s not necessary a delay of minutes to impact your sales, just a few extra seconds can prevent your customers from having a successful check out experience.

The reason behind it is that when a new order is placed, Magento has to do quite a few database operations (decrease the stock of all products, insert all order items, save all customer data, etc. and finally create the order in sales_flat_order table) and in order to preserve data integrity, it wraps them all within a DDL transaction in the database. In other words, Magento will execute all these operations all at once, so that if an error occurs, the whole transaction will be rolled back and no changes will be done in the database.

So, what happens if there is an observer in the middle of this complex process that performs a request to the external server of your email provider? Yes, it will delay the process a few seconds. You might thing that it’s not a big deal, but did you know that while this is happening, some tables of your database are being locked by this transaction, which is effectively preventing new orders from being placed until that transaction has finished?

In other words, if two customers are placing an order at almost the same time, the second one will be blocked by the first one. Internally, the queries in the database will be queued up and executed as soon as the first transaction finished, but queued queries will wait for a limited time (typically 50 seconds) and if the lock hasn’t been released by then, an error/exception will be thrown and therefore the order will not be placed. It might seem an unlikely event, but unfortunately, It happens more often than we wish, and the more concurrent customers you have, the more probabilities for this to happen.

From our experience, we’ve seen it happening in shops with low traffic during sale periods, usually when there are around 100+ concurrent users it can happen a few times day, and with 300+ concurrent users it can happen almost every hour.

If you see something like this in your exceptions.log you might well be suffering from one of those Evil Magento Exceptions:

exception 'PDOException' with message 'SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction' in .../lib/Zend/Db/Statement/Pdo.php:228

.../app/code/core/Mage/Sales/Model/Resource/Order/Abstract.php(425): Mage_Core_Model_Resource_Db_Abstract->save(Object(Mage_Sales_Model_Order))

We’ve seen this issue with many popular Magento Extensions and well known email marketing platforms such as Dotmailer, ExactTarget or Sailthru, so next time you are in about to install a new Magento Extension, make sure it’s reviewed first by a Magento Expert to avoid awkward surprises.

If you stumble upon with this issue, do not hesitate to get in touch, and we will be able to assist you.