1. Intro

Statistics is an essential skill for a computer engineer, a data scientist, or a machine-learning engineer to have in their professional career. In most universities, though, we learn statistics the same way we learn differential equations or calculus: a long list of formulas with little connection to real-world use. We memorize plenty of mathematical basics we’ll never actually use, which gets in the way of seeing what statistics can really do for us in the real world.

In this article, we are about to learn two crucial statistics concepts, The Poisson Distribution and The Poisson Process, through solving a common problem in the concept of online website hosting.

2. What Poisson Distribution or Poisson Process is about?

Before we get started, let’s first ask ourselves a simple question: “Why do we need Poisson?”

The Poisson distribution helps us predict the probability of a given number of events occurring in a fixed interval of time, or, more simply, to predict the number of events in the future.

For example,

• How many visitors do you get on your website in a day?
• How many clicks do your ads get for the next month?
• How many phone calls do you get during your shift?
• How many people will die from covid-19 next year?

Every week, on average, 17 people react to my blog posts by emailing me. I’d like to predict how many people will react to my posts next week, so I can set aside enough free time to answer them properly, planning ahead matters a lot to me.

What is the probability that exactly 20 people (or 10, 30, 40, etc.) will react to my posts next week?

To answer this question, if we knew some statistics, it would come to our minds that we should be able to solve this problem using the Binomial distribution. Let’s first look at the concept of the Binomial distribution and solve the problem using it.

3. Binomial distribution

One way to solve this is to start with the number of reads. Each person who reads the blog has some probability of actually having a question or finding something interesting enough to share with me.

A binomial random variable is the number of successes \(x\) in \(n\) repeated trials. And we assume the probability of success \(p\) is constant over each trial. The Binomial distribution formula is: \begin{equation} P[X=x] = \binom{n}{x}p^x(1-p)^{n-x} \end{equation}

However, here we are given only one piece of information, \(17 \frac{emails}{week}\), which is a rate. We don’t know anything about the probability that an individual reader sends an email, \(p\), nor the number of blog visitors, \(n\). So, we use Google Analytics to retrieve this data from our blog history.

Stats from Google

By looking at the stats we can say that, in one year, a total of \(59k\) people read my blog. Out of those \(59k\) people, \(888\) of them liked my post. Therefore, the number of people who read my blog per week (\(n\)) is \(\frac{59k}{52}=1134\). The number of people who liked my posts per week (\(x\)) is \(\frac{888}{52}=17\). So, the success probability \(p\) would be \(\frac{888}{59k} = 0.015\) or \(1.5\%\).

Now Using Binomial PMF, we should be able to calculate the probability of getting 20 emails for next week as below:

We can use python or any other programming language to calculate the probability of getting emails with different values.

from scipy.stats import binom
# setting the values
# of n, p and x
n = 1134
p = 0.015
x_s = [10, 17, 20, 30, 40]

print(f"x\tBinomial P(x, n, p)")
print("----------------------------")
for x in x_s:
    print(f"{x}:\t{binom.pmf(x, n, p)}")

x	Binomial P(x, n, p)
----------------------------
10:	0.022507172903122208
17:	0.09701415708780352
20:	0.06962037106916726
30:	0.0012106250995813465
40:	6.815731666708672e-07

As you can see, by using the binomial distribution, the probability of getting 10, 17, 20, 30, and 40 emails per week is as follows:

4. Shortcomings of the Binomial Distribution

The very first problem with a binomial random variable is that “it’s assumed to be binary (0 or 1)”. In the previous example, we have \(17 \frac{emails}{week}\). This means \(\frac{17}{7}=2.4\) people will send me emails per day, and \(\frac{17}{7 \times 24} = 0.1\) people send me an email per hour. If we model the success probability by hour (0.1 email/hr) using a binomial random variable, this means most hours get zero emails, but some hours get exactly one. However, it’s also entirely possible that a given hour gets more than one email (2, 3, 5, etc.). The problem with binomial is that it can’t contain more than one event per unit of time (in this case, one hour is the unit of time). The unit of time can only have 0 or 1 event.

How about dividing that one hour into 60 minutes, making the unit of time smaller? If we do that, one hour can now contain multiple events (though a single minute will still contain exactly zero or one). But what if, during that one minute, we get multiple emails, say, someone shares my blog post on Twitter and traffic spikes at that exact minute? Then what? We can divide a minute into seconds, making a second our new unit of time, and again a minute can contain multiple events. But this binary-container problem persists no matter how small we make the unit of time. The idea is that we can make the binomial random variable handle multiple events by dividing the unit of time into ever-smaller pieces. By doing that, we let the original unit of time contain more than one event. Mathematically, this means \(n\) goes to infinity. Since we’ve assumed the rate is fixed, \(p\) must go to zero: as \(n\) grows toward infinity, the number of intervals grows along with it, and the probability of getting an email in any single interval shrinks toward zero. In other words, if \(n\) goes to infinity, \(p\) must go to zero, otherwise \(np\), which is the expected number of events, would blow up.

The second problem with the binomial random variable is that “to use the binomial distribution, the number of trials, \(n\), and the probability of success, \(p\), must be known”. If you use Binomial, you cannot calculate the success probability with just the rate (i.e. \(17\frac{emails}{week}\)). You need the additional information \(n\) and \(p\) in order to use the binomial PMF. The Poisson Distribution, on the other hand, doesn’t require you to know \(n\) or \(p\). We are assuming \(n\) is infinitely large (\(n\rightarrow\infty\)) and \(p\) is infinitesimal (\(p\rightarrow 0\)). The only parameter of the Poisson distribution is the rate \(\lambda\). (In real life only knowing the rate is much more common than knowing both \(n\) and \(p\))

5. Derive the poisson formula mathematically

Now let’s dive into the binomial distribution formula, starting from the conclusion we reached in the previous section (Section 4): to deal with the binary nature of the binomial random variable, we should increase the number of time units in our one-week interval. Mathematically, this means \(n\rightarrow\infty\):

If, \(n\rightarrow\infty\) and \(p\rightarrow 0\) then we can assume that:

Then we have:

Because:

We can say:

\begin{equation} P[X=x] = e^{-λ}\frac{λ^{x}}{x!} \end{equation}

Which is actually the Poisson random distribution formula.

6. Probability of events for a Poisson distribution

An event can occur 0, 1, 2, … times in an interval. The average number of events in an interval is designated λ. λ is the event rate, also called the rate parameter. The probability of observing k events in an interval is given by the equation:

Where:

• \(\lambda\) is the average number of events per interval
• \(e\) is the number \(2.71828…\) (Euler’s number) the base of the natural logarithm
• \(k\) takes values \(0, 1, 2, …\)
• \(k! = k(k-1)(k-2)…(2)(1)\) is the factorial of \(k\)

We calculate the probability of observing 10, 17, 20, 30 and 40 emails in the interval using the poisson distribution formula as below:

from scipy.stats import poisson

#calculate probability

# setting the values
# of lambda and x
l = 17
x_s = [10, 17, 20, 30, 40]

print(f"x\tPoisson P(x, lambda)")
print("----------------------------")
for x in x_s:
    print(f"{x}:\t{poisson.pmf(k=x, mu=l)}")

x	Poisson P(x, lambda)
----------------------------
10:	0.022999584406166312
17:	0.09628462779844556
20:	0.06915882695522822
30:	0.001278796308921649
40:	8.381188233781985e-07

As you can see, the probabilities calculated using the Poisson formula are very close to the values from the Binomial distribution formula, so we can conclude that the Poisson distribution is a really good approximation for our problem in this case. Because of its simpler formula and its single, easy-to-obtain parameter, the Poisson distribution is very useful for solving problems of this kind.

7. Some notes on Poisson random variable

Even though the Poisson distribution is often used to model rare events, the rate \(\lambda\) can be any number, it doesn’t always have to be small. The Poisson Distribution is asymmetric, always skewed toward the right, because it’s bounded by the zero-occurrence barrier (there’s no such thing as minus one email) on the left, while it’s unbounded on the right. As \(\lambda\) becomes bigger, the graph looks more like a normal distribution.

from scipy.stats import poisson
import matplotlib.pyplot as plt

x_s = [i for i in range(0, 40, 5)]
plots = []
mus = [5, 10, 15, 17, 20]

for mu in mus:
  y_s = []
  for x in x_s:
    y_s += [poisson.pmf(k=x, mu=mu)]
  plots += [(x_s, y_s)]


plt.figure(figsize=(10,10))
for i in range(len(plots)):
  plt.plot(plots[i][0], plots[i][1], label=f"lambda = {mus[i]}")

plt.legend(loc='best')
plt.show()

Poisson distribution with different lambda's for our problem.

The average rate of events per unit time in a Poisson distribution is constant. This means the number of people who visit my blog per hour might not follow a Poisson Distribution, because the hourly rate is not constant (higher during the daytime, lower during the night). Using a monthly rate for consumer or biological data would also be just an approximation, since seasonality effects are non-trivial in that domain.

In a Poisson random variable, events are independent. The arrival of my blog’s visitors might not always be independent. For example, sometimes a large number of visitors arrive together because someone popular mentioned my blog, or my blog got featured on Medium’s front page, etc. Similarly, the number of earthquakes per year in a country might not follow a Poisson Distribution either, since one large earthquake increases the probability of aftershocks.

References

1. S. M. Ross. A First Course in Probability. Pearson. (Binomial-to-Poisson limit derivation, Poisson process.)
2. SciPy. scipy.stats.poisson. docs.scipy.org/…/scipy.stats.poisson.html
3. SciPy. scipy.stats.binom. docs.scipy.org/…/scipy.stats.binom.html