Introduction

Return on Ad Spend, or ROAS, is an important KPI for marketers to inform business decisions. ROAS refers to the amount of money a business earns back for each dollar spent on advertising and is a commonly used marketing term. It is designed to measure how effective a business’ advertising efforts are. A higher ROAS is the goal.

We, in CTW, choose ROAS as the key KPI for advertisement goal settings and performance tracking. So that we can understand if our budgets are spent wisely across different channels with maximised returns.

There are two major problems that we are trying to solve using this metric:

1. When are we able to fully recover the amount of advertisement cost we spend in month n?

2. What’s the payment amount that we can earn next month given budget x?

We will go through each of them, explaining why are they important for our business, as well as how we answer them.

Key concepts

Before we dive deeper into how we solve these problems, here are a few more concepts to help you comprehend everything around our ROAS system.

• Cost, aka advertisement budget.

• Payment amount, money spent by a user.

• ROAS = payment amount / cost. N.B. In this equation, we cannot use cost or payment amount from any arbitrary month and just divide them to calculate ROAS. The cost and the payment amount have to be in the same dimension. That is to say, the cost has to contribute towards the user who made the payment.

For example, to calculate 2020 January ROAS, we need to first collect the advertisement cost we spent in that month, and then
collect payment produced by the user who triggered the createrole event in that month through those advertisements we create, and lastly divide them to calculate ROAS.

• createrole, an in-game event triggered by a user when they create their in-game character, and recorded in our system, indicating a conversion of certain advertisements.

• Interval, the number of days after the user triggered the createrole event. N.B. When we collect user payment, 30 days - 360 days interval is used, with an interval of 30. (30, 60, 90…300, 330, 360). For example, when the interval is 60, we collect payment from user who spent money within 60 days of their createrole event.

Probrem 1 - When are we able to recover the cost we spend?

This is one of, if not the most commonly asked question in the world of marketing.

Right now, our ROAS system can answer this question in a month-over-month context. What that means, if we know our 30 interval ROAS of January, we are then able to extrapolate that into the future, and simulate up to 360 interval of ROAS. Let’s take a closer look at how we’ve done it.

Residual-based approach.

To achieve ROAS simulation we used a residual-based approach. First of all, let’s take a look at the sample graph.

On the x-axis, we have our 30 day based intervals, representing the number of days passed since the user’s createrole event.

On the y-axis, we have our calendar month, let’s say our game was released in January and now it’s July.

In the middle, the blue number represents the ROAS, e.g. January’s 30 interval days ROAS is 10%, which means 10% of our cost is covered by the total payment amount from those users who create their roles in January. And as for the calculation of the total payment amount, we accumulate 30 days worth of payment amount from each user, e.g. for user createroled on January 1st, we accumulate his payment up to January 30th, and for user createrole on January 31th, we accumulate his payment up to the March 2nd. This means that to fully receive January’s 30 interval days ROAS, we need to continuously collect data until 30 days have passed for all users who create their roles in January. The same concept applies to 60, 90 interval days etc.

To answer problem 1, we need to look at this graph horizontally. So that we can understand, how our cost at each month is recovered at what pace. These green arrows show how ROAS changes from interval to interval. For example, January’s 60 interval ROAS indicates a 2 times increase since the 30 intervals ROAS.

Now, imaging we are in July and we would like to know how much our cost is going to be recovered at interval 150. We can figure it out by applying the residual-based approach:

Firstly, we took the changes from every interval, and calculate the average. This number indicates the increase ratio from small interval to bigger interval. Our hypothesis here is that the increasing behaviour from interval to interval remains, regardless of which calendar month the users are coming from. As a result, we get roughly 1.7 times of increase from 30 to 60, 1.7 times increase from 60 to 90, 1.3 times of increase from 90 to 120 and lastly 1.25 times of increase from 120 to 150.

Lastly, we apply these ratios to the original data and get the simulated ROAS for every month and every interval. With this data, we will be able to see how the initial 30 intervals ROAS evolve over time.

What about the future month? You may ask. For that question, we will take a look at our second problem.

Probrem 2 - What about the future?

From the Residual-based approach, we were able to simulate the future growth of an initial ROAS. However, that’s only the so-called “horizontal future”. We still need that 30 days initial ROAS to begin with. As an example, from the vertical axis, we still don’t know how our 30 days ROAS is going to be like in June or July etc.

But why is that important? First of all, the future month budget is decided beforehand, and the marketing team is really curious about how that budget is going to pay back in the upcoming month so that they can adjust them along the way.

What we could do in this case, is to predict the “vertical future” month’s initial 30 interval ROAS based on the budget, and then simulate the rest of the interval ROAS using the above approach we just explained. So, how to predict?

ML-based approach.

We need something more sophisticated than just taking the average of the 30 interval ROAS in the past. Because ROAS is highly affected by cost and pay amount. The history ROAS will not be representative for the future if they are based on a different budget, with different channels distribution etc. Let’s see what we could do with a machine learning approach.

The machine learning approach involves two development cycles. And four stages. Let’s dive into them one at a time.

Problem Definition

Before we get excited about machine learning, let’s first define the problem that we are trying to solve. We are going to predict ROAS, however, knowing that ROAS is calculated based on cost and pay amount. We could simplify the problem from predicting ROAS to predicting the payment amount. And after we get the payment amount, we can then calculate ROAS by a simple division.

We will develop two different models, one for old games and one for new games. This is because old games have accumulated a large amount of data so that we could make the monthly prediction. As for new games, we will be using training data that is daily based. We will be making daily predictions, however, the result will be aggregated monthly, for the sake of interpretability.

Model Prototyping

At the early stage, we explore a wide variety of different models, with extensive data cleaning, feature engineering, model training, and performance evaluation, with just a minimum amount of hyper-parameter tuning, to make sure this development cycle iterates as fast as possible.

All training data are gathered from our Amazon Redshift database, for old games, data are aggregated monthly, while new games are presented daily as is. During the data cleaning stage, we developed an approach to detect and remove so-called whales (those with extremely high payment amounts) from our dataset.

Our target variable is payment amount, and the final output is ROAS.

Here is a list of different things we used in two models :

Daily model requires to be released after 30 days of data being collected. This means we have a rather limited amount of data to work with. Also to fully utilise all the data we have, we need to use interval as a feature, that way we will be able to combine users with 30 days worth of payment with users having only 1 day worth of payment and let the model decides how much weight it should distribute to each other.

We chose CatBoostRegressor because it offers great performance with a minimum amount of hyper-parameter tuning required. Only loss function are tuned to optimise the performance of each model.

We use Mean Absolute Error of ROAS as our evaluation metric, this is because we care about our model’s interpretability. At the end of the day, we need to explain how much the marketing team should trust our model so that they can adjust their expectations and their budget spending strategy accordingly.

As a result, both models achieved less than 2% of monthly ROAS MAE. To translate that into real money, consider if we predicted 10% ROAS, that means if we planned to spend 10,000,000 JPY, we are expecting 800,000 to 1,200,000 JPY payment amount for the initial 30 interval days.

Model Productisation

We deploy the model onto Amazon Elastic Container Service (Amazon ECS). And schedule the training using Apache Airflow. Model is trained and deployed 5 times a day. This is to make sure that the marketing team budget adjustment will be reflected in the model’s prediction result timely.

For the productised model, all historical data are used for training. For prediction, we receive the next month budget planning information from the marketing team at the end of each month.

One thing to keep in mind is that we are predicting a 30 interval days ROAS, meaning that sometimes we still need to predict for the past month. Why is that?

Imagine we are in mid-July. We have indeed collected all the users from June, however, not all of them have spent their 30 days after createrole, e.g. user createrole on 30th of June, will reach 30 days on the 30th of July. This means that every month’s ROAS will settle until it reaches the end of next month. As a result, we still need to predict how last month ROAS goes using our model.

Result Monitoring and Presenting

Prediction results from both ML model, as well as residual-based approach for all games and all intervals, are saved on to Amazon S3.

At the end of each prediction period, i.e. when real data is settled for that interval. We use Mean Absolute Error to monitor our model’s performance. To compute the MAE we took the average of the predictions made during that prediction period and compare it with the real data, and report the result to the marketing team so that they can decide whether any adjustments need to be made for the budget planning.

Imagine we are in early July, the final output will look like this and be presented in Tableau:

We can see that red ROAS are the ones predicted by our machine learning model, year ROAS are the ones simulated using increase ratio. And with these two forecasting methods, the marketing team will be able to understand the market ahead of time and make timely adjustments to their budget to maximize the performance of their marketing campaigns.

Limitations and Future Works

• We rely on a fixed 30 days interval for ROAS calculation.

30 days interval is easy to understand or to calculate for any long time running title. However, for any newly released games, evaluating the marketing effort after 30 days is just a bit too long.

What we could do in the future, is to make the interval configurable, ranging from 0 to 100 or larger, so that we could adjust the ROAS calculation at any time.

• We can simulate ROAS at a certain interval only if we have real history ROAS data at that interval. The residual-based approach still requires ground truth history data to carry out the simulation.

To give an example, let’s take a look at the graph that you are already familiar with:

When we only have 120 interval ROAS available, the current residual-based approach is only able to simulate up to 120 interval. Right now, we are not able to simulate any further due to the limitation of the approach.

Most of our old games are having more than 360 days of interval ROAS available so that’s not an issue, however, for newly released titles, even after three month, we still only able to simulate
up to 60 days interval ROAS, which is not sufficient for marketers to see the greater picture at the early stage of a newly released game.

This can be resolved by, first of all combining the solution from the first limitation we just explained, that is, reducing the ROAS calculation range at will, and then, applying a statistical or machine learning approach to simulate future interval ROAS given the existing ROAS.

• Our hypothesis about the residual-base approach could be wrong.

Remember what we said that the increasing behaviour from interval to interval remains, regardless of which calendar month the users are coming from. This is not true if you consider that the game is not a static product, users joined in January and users joined in December could have a different experience.

Also, people who pre-order the game and joined in the first month will have a different purchase behaviour, than the ones joined after game was released for a year. We need to consider and maybe introduce a declining factor to all of our calculations.

Conclusion

In this blog, we introduced the architecture of our ROAS Business Intelligence System. And how does it addresses two major problems raised by the business. There are still many improvements to be done in terms of both scales and functionality, and we welcome any suggestions from any newcomers.

Appendix

Version updates in chronological order

v0.1 Linear regression approach for historical month ROAS simulation up to 180 days.

v0.2 Linear regression approach for historical month ROAS simulation up to 360 days.

Limitations: low accuracy due to the non-linearity of ROAS changes over time. Unable to simulate ROAS when only 30 interval ROAS is present. Unable to predict future month ROAS.

v0.3 Machine Learning approach to predict future month ROAS for old games.

v0.4 Machine Learning approach to predict future month ROAS for new games.

v0.5 Residual approach to history month ROAS simulation

v0.6 Machine Learning approach to predict future month ROAS for all games at the category level.

v0.7 Machine Learning approach to predict future month ROAS for all games at the source level.

---

Join Us

---

• Next Post →