How we achieved light speed to send 3 Million campaigns in just 30 seconds!

While running marketing campaigns, a key component is to ensure that the campaigns (emails & SMS messages) reach the customers at the right time. When we started Xeno in 2015, it was a marketing product for small businesses who had 1000’s of customers. Ensuring campaigns reached 1000’s of customers on time wasn’t a major technological challenge.

But in mid-2018 we pivoted to work with enterprise customers & re-built our product to enable a brand with 3 million customers to create 3 million unique campaigns specific to each & every individual customer.
One to one communication means sending a personalized offer, product recommendation to the customers on their preferred channel (Facebook, Instagram, email, etc.) and at the right time.

In simple words, this would mean, a customer who orders a sandwich every Monday Morning online sitting in his office at around 10 AM should get a marketing campaign preferably on his work email with a custom offer on Monday Morning before 10 AM. 

Although as simple it may sound, the intelligence behind it, needs tons of data to process at a fraction of second. In the above example, the Xeno system had to identify breakfast customers first, the size of which almost crosses half a million for a decent sized brand.  I realized that the messages were getting delayed by almost an hour making our core intelligence DUMB.

Today I’m sharing our journey from being an embarrassingly slow sloth to lightning fast flash in just one month.

Step-1: Measure it

The first step to improving anything is to measure it. Like Peter Drucker said, “You can’t manage what you don’t measure”.

So, we decided on our key metric to be Campaign Processing Time (CPT) per 10,000 customers. CPT is the total time taken to deliver campaigns to each customer of the campaign(batch).

For eg. If a campaign was scheduled at 3 PM for 10K Customers and the campaign processor picked it up at 3 PM but let's say the last customer in the campaign got the message at 3.06 PM. So here the CPT will be  3:06 - 3:00 PM, ie. 6 minutes.

Tracking the numbers made it completely absolute, and now we were very sure that delivering millions of campaigns was taking us over an hour.
Now we could proceed to make changes to the code & see how it impacts delivery.

Step-2: Solve It

Interestingly, like anything in engineering, their is always low hanging fruit that one can fix to improve results drastically. So from simpler changes to bigger ones, here’s what worked for us.

Reducing the number of database calls

If you use a database, reducing the database calls in your code will definitely speed things up.  Databases are great at reading data. They are storage engines. However, your business logic should always live in your application. Also, Calling the database in a loop is the worst thing to do to your code.  

In the above scenario, we were making as many numbers of calls to our system as much as the customer size of the campaign making the performance of our algorithm linear, which means it will take longer the more that is in the loop's input.

While reviewing our code, we got to understand that if we had to send 1000 messages, we were making 1000 calls to the database to fetch customer personal information like name, gender or his last visited.

Once we changed it to a single query by pulling all 1000 records in a single database call, our code execution time went drastically down.

Below is sample example of DB calls:

BEFORE

String custIds = “1,2,3,4,5,6”;
Long merchantId = 5;
String message = “Pizza is never enough & especially when it's from Cheelizza! Now Get Any 4 funtime pizzas for only Rs200! Call 02267674747 now or visit!”;


List<CustomersMerchantMap> customerMerchantMapList= new ArrayList<CustomersMerchantMap>();

//DB CALL
customerMerchantMapList = customersMerchantMapDAO.findByIds(merchantId,custIds);

for (CustomersMerchantMap customerMerchantApp : customerMerchantMapList) {
//DB CALL
Customers customer = customersDAO.findById(customerMerchantApp.getCustomer_id());
sendSms(message , customer);
}

AFTER
String custIds = “1,2,3,4,5,6”;
Long merchantId = 5;
String message = “Pizza is never enough & especially when it's from Cheelizza! Now Get Any 4 funtime pizzas for only Rs200! Call 02267674747 now or visit!”;

List<Customers> customersList= new ArrayList<Customers>();
List<CustomersMerchantMap> customerMerchantMapList= new ArrayList<CustomersMerchantMap>();

//DB CALLS
customersList = customersDAO.findByIds(custIds);
customerMerchantMapList = customersMerchantMapDAO.findByIds(merchantId,custIds);

Map<Long,Customers> customerMap = new HashMap<Long,Customers>();
for(Customers customers : customersList) customerMap.put(customers.getId(),customers);

for (CustomersMerchantMap customerMerchantApp : customerMerchantMapList) {
Customers customer = customerMap.get(customerMerchantApp.getCustomer_id());
sendSms(message , customer);
}

Using Object Cloning

Today’s programming is all about costs. Saving cost is a big issue when it comes to using computer resources, so we are doing our best to find ways of improving the performance. When we talk about object creation we can find a better way to have new objects using Object cloning.

We used object-oriented programming, re-used more and more object wherever possible rather than calling DB again and again. For example, our campaign processing structure is split into 18 parts where one part is reading customer information, one of it is replacing variables, one of it is doing a sanity check, one is checking if the same SMS is being sent repeatedly and so on.

We thought of a design principle that if any information is pulled once, will never be fetched again from the database again in the entire campaign sending process. To achieve this, we stored as much information in objects and kept passing them to different parts of the code. This helped us again to reduce our processing time to almost half.

Optimising Queries

Doing the above two points did 99% of our job, but we were still figuring out if this can be taken to another level. One day, while doing some experiments with a random part of our code, we got this learning that reading a single column from a table can take 1/10th time vs reading the entire table.

For example - If we need just need the phone numbers of the customers and we are pulling the entire row instead, it will be overhead and also more time consuming specifically when you have to fetch millions of records at a time.

Step-3: Impact & Beyond

Making these 3 changes brought down the time to deliver millions of campaigns for over an hour to just seconds. 

Statistically, Xeno takes 10 microseconds to send 1 personalized campaign. 😎

We applied the same theory to our campaign processer and found the results to be inconceivable specifically for campaigns which were being sent on a large scale. Later we added this as our technology design principles, to never pull any data from the database which will not be used later in that code.

Being a SAAS product in a very competitive industry, we tend to pay more attention to build new features and ship them really fast. Almost every company does that. This might sell your product like hotcakes but ignoring basic parameters shows its impact really soon. The basic parameters like speed, stability and writing bug-free code are crucial for the success of any product.

Focussing on the basic parameters over the last couple of months, helped our business to accelerate faster. In the last quarter (Q3) we were able to reach an average of 200% uplift in marketing conversions.

If you're are responsible for marketing in Medium/ Large enterprise, and want to increase your marketing conversions, say hello@xeno.in