Monte Carlo Simulations

In the city of Monaco, France lies a district of 15,000 people famous for its nightlife, known as Monte Carlo. A bourgeoisie haven for the affluent and wealthy, here lies a famous casino where the element of chance inspired the naming of a mathematical method steeped in uncertainty.

The Monte Carlo Method was created by mathematicians John von Neumann and Stanislaw Ulam during World War II as a way to improve decision making under uncertain conditions. The premise of their idea is that solutions to mathematical problems, mainly those modeling complex systems or processes, can be derived through the random evolution of sampling. Said differently, most problems are solved with formulas & equations using precision and exact mathematics — while these complex problems are solved using averages of stochastic (random) simulations over a large number of iterations — based off the law of large numbers and the central limit theorem.

When creating mathematical models, one of the key input variables is random since that input variable (condition) is uncertain. This is akin to running surveys on a large population by choosing samples that are truly random in order to avoid introducing bias to the data, but deriving a clear picture of the population — only getting clearer with the greater the population that is surveyed. Monte Carlo simulations are also useful when you’re looking to get a range of possible outcomes, not just one specific answer.

To estimate the possible outcomes of an uncertain event, this doesn’t use standard forecasting methods, but instead builds a model of possible results by leveraging a probability distribution, such as a uniform or normal distribution, for any variable that has inherent uncertainty. Because the input is a probability distribution, we can assume that the output in this case will also be a probability distribution. The results of each iteration are then fed back into the simulation in a loop, coupled with a different set of random numbers between the minimum and maximum values of the defined range.

With an exhaustive list of possible outcomes, a person can make a better-informed decision around risks under these uncertain conditions. While most statistics problems would estimate random quantities in a deterministic manner, Monte Carlo methods are the inverse where random quantities are used to generate estimates of deterministic quantities.

Ex. Testing push notification reception with users

Let’s say we want to determine the probability of releasing a certain push notification to a cohort of users and getting positive conversion results. We can release this as an A/B test live in production and see what reception we get from users, but we’re in the habit of making cheap mistakes and starting with our best foot forward. In fact we have a rolodex of conversion results from different push notification campaigns we’ve held in the past and can use those conversions to make an educated guess and what probabilities we can associate with particular variables.

1. Define the uncertain variables. We collect all the various probabilities from prior push notification campaigns and use these as our variables. For example:
1. P1: 60% chance user meets trigger
2. P2: 80% chance user is NOT in holdout group
3. P3: 75% chance user has push enabled
4. P4: 90% chance locale is eligible
5. P5: 85% chance inventory available
2. Run a simulation (say, 10,000 times). Each simulation = 1 "virtual user". For each simulated user, we would initiate our Python code in the following manner:

import random

def simulate_one_user():
    return (
        random.random() < 0.60 and  # trigger met
        random.random() < 0.80 and  # not in holdout
        random.random() < 0.75 and  # push enabled
        random.random() < 0.90 and  # locale eligible
        random.random() < 0.85      # inventory ok
    )

Repeat for 10,000 users:

results = [simulate_one_user() for _ in range(10000)]
prob_push = sum(results) / len(results)

This gives you the estimated probability of a push being sent, based on the factors we have defined.

3. Add cohorts. To model conditional probabilities, for example the probability that a push is sent to a user who is in the new user cohort (A) — $P(\text{push}\:|\:\text{cohort = new user})$, we would do the following:
1. Tag each simulated user with attributes:
1. "cohort": "new_user" vs "cohort": "returning"
2. "locale": "US" vs "locale": "INTL"
3. "last_order_days_ago": 3 vs 10
2. Then filter your results by cohort and recompute:

# Simulated user data
simulations = []
for _ in range(10000):
    cohort = "new_user" if random.random() < 0.3 else "returning"
    push_sent = simulate_one_user()
    simulations.append((cohort, push_sent))

# Estimate: P(push | cohort = new_user)
cohort_A = [sent for c, sent in simulations if c == "new_user"]
prob_cohort_A = sum(cohort_A) / len(cohort_A)

From this Monte Carlo simulation, you get:

• An estimated baseline probability of sending the push
• Conditional probabilities by cohort
• A way to model what-if scenarios — e.g. “what if we tighten the trigger to 10 days?”

So while we can run this experiment live, these simulations give us a sandbox to validate different testing approaches so we polish our hypothesis before testing with actual, live users. In a world where Big Data continues to deepen its footprint, improved computational processes will imply greater accuracy in the output of simulations, primarily in machine learning models.