Linearity of Expectation and Markov’s Inequality

Probability Background

Consider a random variable $X$. For example, $X$ could be the outcome of a fair dice roll and be equal to $1,2,3,4,5$ or $6$, each with probability $\frac{1}{6}$. Formally, we use $Pr(X=x)$ to represent the probability that the random variable $X$ is equal to the outcome $x$. The expectation of a discrete random variable is $$\mathbb{E}[X]=\sum _{x}xPr(X=x).$$ For example, the expected outcome of a fair dice roll is $\mathbb{E}[X]=1\times \frac{1}{6}+2\times \frac{1}{6}+3\times \frac{1}{6}+4\times \frac{1}{6}+5\times \frac{1}{6}+6\times \frac{1}{6}=\frac{21}{6}$. Note: If the random variable is continuous, we can similarly define its expected value using an integral.

The expected value tells us where the random variable is on average but we’re also interested in how closely the random variable concentrates around its expectation. The variance of a random variable is $$\text{Var}[X]=\mathbb{E}[(X-\mathbb{E}[X]{)}^2].$$ Notice that the variance is larger when the random variable is often far from its expectation. In the figure below, can you identify the expected value for each of the three distributions? Which distribution has the largest variance? Which has the smallest?

There are a number of useful facts about the expected value and variance. For example,

$$\mathbb{E}[\alpha X]=\alpha \mathbb{E}[X]\text{and}\text{Var}(\alpha X)={\alpha }^2\text{Var}(X)$$ where $\alpha \in \mathbb{R}$ is a real number. To see this, observe that $$\mathbb{E}[\alpha X]=\sum _x\alpha xPr(X=x)=\alpha \sum _{x}xPr(X=x)=\alpha \mathbb{E}[X]$$ and $$\text{Var}(\alpha X)=\sum _x(\alpha x-\alpha \mathbb{E}[X]{)}^2={\alpha }^2\sum _x(x-\mathbb{E}[X]{)}^2={\alpha }^2\text{Var}(X).$$

Independent Random Variables

Once we have defined random variables, we are often interested in events defined on their outcomes. Let $A$ and $B$ be two events. For example, $A$ could be the event that the dice shows $1$ or $2$ while $B$ could be the event that the dice shows an odd number. We use $Pr(A\cap B)$ to denote the probability that events $A$ and $B$ both happen. Often, we have information about one event and want to see how that changes the probability of another event. We use $Pr(A|B)$ to denote the conditional probability of event $A$ given that $B$ happened. We define

$$Pr(A|B)=\frac{Pr(A\cap B)}{Pr(B)}.$$

If information about event $B$ does not give us information about event $A$, we say that $A$ and $B$ are independent. Formally, events $A$ and $B$ are independent if $Pr(A|B)=Pr(A)$. By the definition of conditional probability, an equivalent definition of independence is $Pr(A\cap B)=Pr(A)Pr(B)$.

Let’s figure out whether the event $A$ that the dice shows 1 or 2 is independent of the event $B$ that the dice shows an odd number. Well, $Pr(A\cap B)=\frac{1}{6}$ since the only outcome that satisfy both events is when the dice shows a 1. We also know that $Pr(A)Pr(B)=\frac{2}{6}\times \frac{3}{6}=\frac{1}{6}$. So, by the second definition of independence, we can conclude that $A$ and $B$ are independent.

We’ve been talking about events defined on random variables, but we’ll also be interested in when random variables are independent. Consider random variables $X$ and $Y$. We say that $X$ and $Y$ are independent if, for all outcomes $x$ and $y$, $Pr(X=x\cap Y=y)=Pr(X=x)Pr(Y=y)$.

Linearity of Expectation

One of the most powerful theorems in all of probability is the linearity of expectation.

Theorem: Let $X$ and $Y$ be random variables. Then $$\mathbb{E}[X+Y]=\mathbb{E}[X]+\mathbb{E}[Y].$$ The result is a powerful tool that requires no assumptions on the random variables.

Proof: Observe that $$\mathbb{E}[X+Y]=\sum _{x,y}(x+y)Pr(X=x\cap Y=y)$$ Now, we’ll separate the equation into two terms and factor out the $x$ and $y$ terms, respectively. $$=\sum _{x}x\sum _{y}Pr(X=x\cap Y=y)+\sum _{y}y\sum _{x}Pr(X=x\cap Y=y)$$ Finally, using the law of total probability, we have $$=\sum _{x}xPr(X=x)+\sum _{y}yPr(Y=y)=\mathbb{E}[X]+\mathbb{E}[Y].$$

There are also several other useful facts about the expected value and variance.

Fact 1: When $X$ and $Y$ are independent, $\mathbb{E}[XY]=\mathbb{E}[X]\mathbb{E}[Y]$.

Proof: Observe that $$\mathbb{E}[XY]=\sum _{x,y}xyPr(X=x\cap Y=y)=\sum _{x,y}xyPr(X=x)Pr(Y=y)$$

$$=\sum _{x}xPr(X=x)\sum _{y}yPr(Y=y)=\mathbb{E}[X]\mathbb{E}[Y]$$ where the second equality followed by the assumption that $X$ and $Y$ are independent.

Fact 2: Consider a random variable $X$. Then $\text{Var}(X)=\mathbb{E}[X^2]-\mathbb{E}[X{]}^2$.

Proof: Observe that $$\text{Var}(X)=\mathbb{E}[(X-\mathbb{E}[X]{)}^2]$$ $$=\mathbb{E}[X^2-2X\mathbb{E}[X]+\mathbb{E}[X{]}^2]=\mathbb{E}[X^2]-\mathbb{E}[X{]}^2$$ where the first equality is by definition, the second equality is by foiling, and the third equality is by linearity of expectation and the observation that $\mathbb{E}[X]$ is a scaler.

Fact 3: When $X$ and $Y$ are independent, $\text{Var}(X+Y)=\text{Var}(X)+\text{Var}(Y)$.

Proof: Observe that $$\text{Var}(X+Y)=\mathbb{E}[(X-Y{)}^2]-\mathbb{E}[X-Y{]}^2$$ $$=(\mathbb{E}[X^2]-2\mathbb{E}[XY]+\mathbb{E}[Y^2])-(\mathbb{E}[X{]}^2-2\mathbb{E}[X]\mathbb{E}[Y]+\mathbb{E}[Y{]}^2)$$ $$=\mathbb{E}[X^2]-\mathbb{E}[X{]}^2+\mathbb{E}[Y^2]-\mathbb{E}[Y{]}^2=\text{Var}(X)+\text{Var}(Y)$$ where the first equality followed by definition, the second equality followed by foiling and linearity of expectation, and the third equality followed by Fact 1.

Markov’s Inequality

Concentration inequalities are a powerful tool in the analysis of randomized algorithms. They tell us how likely it is that a random variable differs from its expectation.

There are many concentration inequalities. Some apply in general and some apply only under special assumptions. The concentration inequalities that apply only under special assumptions tend to give stronger results. We’ll start with one of the most simple and general concentration inequalities.

Theorem: For any non-negative random variable $X$ and any positive threshold $t$, $$Pr(X\ge t)\le \frac{\mathbb{E}[X]}{t}.$$

Proof: We’ll prove the inequality directly. By the definition of expectation, we have $$\mathbb{E}[X]=\sum _{x}xPr(X=x)=\sum _{\begin{array}{c}x\\ x\ge t\end{array}}xPr(X=x)+\sum _{\begin{array}{c}x\\ x<t\end{array}}xPr(X=x)$$ $$\ge \sum _{\begin{array}{c}x\\ x\ge t\end{array}}tPr(X=x)+0=tPr(X\ge t).$$ Rearranging the above inequality gives Markov’s. Can you see where we used that all outcomes $x$ are non-negative?

Now let’s apply Markov’s inequality to our set size estimation problem. Since the number of duplicates $D$ is always positive, we satisfy the assumption of the inequality. $$Pr(D\ge 10)\le \frac{\mathbb{E}[D]}{10}=\frac{.4995}{10}=.04995$$ The probability of observing the 10 duplicates is less than $5\mathrm{\%}$! We should probably start asking the CAPTCHA company some questions.

In practice, many of the set size estimation problems are slightly different. Instead of checking a claim about the set size, we want to estimate the set size directly. Notice that we computed $\mathbb{E}[D]=\frac{m(m-1)}{2n}$. Rearranging, we see that $n=\frac{m(m-1)}{2\mathbb{E}[D]}$. Given $m$ samples, we can naturally build an estimator for the whole set size using the empirical number of duplicates we found in the sample. With a little more work, we can show the following.

Claim: If we make $m\ge c\frac{\sqrt{n}}{ϵ}$ samples for a particular constant $c$, then the estimate $\hat{n}=\frac{m(m-1)}{2D}$ satisfies $(1-ϵ)n\le \hat{n}\le (1+ϵ)n$ with probability $9/10$.

Hash Functions

Let the hash function $h$ be a random function from the set of all items $\mathcal{U}$ to the integers $\{1,\dots ,m\}$. The hash function is constructed using a seed of random numbers and then the function is fixed. Given an item $x$, the hash function always returns the same hashed output $h(x)$.

Definition: A hash function $h:\mathcal{U}\to \{1,\dots ,m\}$ is uniform random if

• $Pr(h(x)=i)=\frac{1}{m}$ for all items $x$ and $i\in \{1,\dots ,m\}$ and

• $h(x)$ and $h(y)$ are independent random variables for all $x,y\in U$.

Notice that the independence condition implies $Pr(h(x)=h(y))=\frac{1}{m}$.

In general, it is not possible to efficiently implement uniform random hash functions. But, for our application, we only need a universal hash function which can be implemented efficiently. Let $p$ be a prime number between $|\mathcal{U}|$ and $2|\mathcal{U}|$. Choose $a,b$ randomly from $0,\dots ,p$ so that $a\ne 0$, then define the hash function $$h(x)=(ax+b\mathrm{mod}p)\mathrm{mod}m.$$ Check out these lecture notes to learn why $Pr(h(x)=h(y))\le \frac{1}{m}$ for this hash function. (As we’ll soon see, this is the condition we need for the algorithm.)

Count-Min Sketch

With handy hash functions, we’re now ready to describe the algorithm. We first choose a random hash function $h$ and initialize an $m$-length array with $0$ in every cell. Items arrive in a stream and we hash each one to one of the $m$ cells in the array. We then add 1 to the number in the cell. Formally, given item $x_i$, we set $A[h(x_i)]=A[h(x_i)]+1$.

In the figure, Amazon products appear one after the other in a stream. We hash the fish tank to the second cell in the array, the basketball to the first cell, and so on. Crucially, when we see the fish tank again, we hash it to the same cell.

After processing the stream, our estimate for the frequency of item $v$ is $\hat{f}(v)=A[h(v)]$. Notice that $\hat{f}(v)\ge f(v)$ since every time we saw item $v$, we added 1 to the corresponding cell in the array. The inequality appears because we could have overcounted when other items hash to the same cell.

Formally, our estimate for the frequency is $$\hat{f}(v)=f(v)+\sum _{y\in \mathcal{U}\setminus v}f(y)\mathbb{1}[h(y)=h(v)].$$ Our estimate for the frequency contains the true frequency plus an error term for all the other items that are hashed to the same cell.

Let’s take the expectation of the error term: $$\mathbb{E}[\sum _{y\in U\setminus v}f(y)\mathbb{1}[h(y)=h(v)]]=\sum _{y\in \mathcal{U}\setminus v}f(y)\mathbb{E}[\mathbb{1}[h(y)=h(v)]]$$ $$\le \sum _{y\in \mathcal{U}\setminus v}f(y)\frac{1}{m}=\frac{1}{m}\sum _{y\in \mathcal{U}\setminus v}f(y)\le \frac{n}{m}.$$ We used linearity of expectation, the special property of indicator random variables and the probability of collisions in our hash function in the first inequality. We used that the sum of frequencies has to sum to $n$ in the last equality.

We have a non-negative random variable and a bound on its expectation, let’s apply Markov’s inequality to the error term

$$Pr(\sum _{y\in \mathcal{U}\setminus v}f(y)\mathbb{1}[h(y)=h(v)]\ge \frac{2n}{m})\le \frac{n/m}{2n/m}=\frac{1}{2}.$$ So we’ve shown that our estimate $A[h(v)]$ satisfies

$$f(v)\le A[h(v)]\le f(v)+\frac{2n}{m}$$ with probability $\frac{1}{2}$ for any $v$. If we set $m=\frac{2k}{ϵ}$, we can solve the problem with error $\frac{ϵn}{k}$. But our success probability is a little low, how can we improve it?

A common approach in many randomized algorithms is to boost the success probability by repeating the core subroutine. In our case, we’ll maintain $t$ independent hash functions and arrays.

As depicted in the figure, each item gets hashed into one cell in every array using the respective hash function. So the update on item $x_i$ for every array $j\in \{1,\dots ,t\}$ is $A_j[h_j(x_i)]=A_j[h_j(x_i)]+1$.

Then, when we’re computing our estimate for the frequnecy of an item, we look at each cell it appears in and take the minimum. Formally, $\hat{f}(v)=min{A}_j[h_j(v)]$. We take the minimum because each array only has one-sided error; its estimate for the frequency is never smaller than the true frequency. If our estimates instead had two-sided error, we would prefer to take the mean or median.

For every array $j$ and item $v$ when we set $m=\frac{2k}{ϵ}$, we know that $f(v)\le A_j[h_j(v)]\le f(v)+\frac{ϵn}{k}$ with probability $\frac{1}{2}$. Let’s compute the probability that our final estimate has this much error.

$$Pr(\hat{f}(v)>f(v)+\frac{ϵn}{k})\le {\left(\frac{1}{2}\right)}^t$$ Call the event that the frequency estimate has error more than $\frac{ϵn}{k}$ an error. The inequality holds since our final estimate fails only if every single one of the arrays also independently fails. If we set $t={\log }_{2}1/\delta$ for some small constant $\delta >0$, then the failure probability is $\delta$. The success probability, the complement of the failure probability, is $1-\delta$.

Putting it all together, we just proved that count-min sketch lets us estimate the frequency of each item in the stream up to error $\frac{ϵn}{k}$ with probability at least $1-\delta$ in $O(mt)=O(\log (1/\delta )\frac{k}{ϵ})$ space. The first term in the space complexity comes from the number of arrays $t$ and the second term comes from the space per array $m$.

However, this guarantee is only for a single item $v$. We really want a guarantee for all items.

Union Bound

Another simple and powerful tool in randomized algorithm design is the union bound. The union bound allows us to easily analyze the complicated dynamics of many random events.

Lemma: For any random events $E_1,E_2,\dots ,E_k$,

$$Pr(E_1\cup E_2\cup \dots \cup E_k)\le Pr(E_1)+Pr(E_2)+\dots +Pr(E_k).$$

Proof: We’ll give a “proof by picture”. Each circle represents the outcome space corresponding to event $E_i$. The total outcome space covered by any overlapping events is at most the outcome space covered by the events if they were to not overlap.

Let’s apply the union bound to the total failure probability of our estimate. The algorithm fails if $\hat{f}(v_i)>\frac{ϵn}{k}$ for any item $v_i$. By the union bound, $$Pr(\text{fail for }{v}_1\cup \text{fail for }{v}_2\cup \dots \cup \text{fail for }{v}_{|\mathcal{U}|})$$ $$\le Pr(\text{fail for }{v}_1)+Pr(\text{fail for }{v}_2)+\dots +Pr(\text{fail for }{v}_{|\mathcal{U}|})$$ $$=\delta +\delta +\dots +\delta =|\mathcal{U}|\delta \le n\delta .$$ Let’s set $\delta =\frac{1}{10n}$. With probability $9/10$, count-min sketch lets us estimate the frequency of all items in a stream up to error $\frac{ϵn}{k}$. Recall this is is accurate enough to solve the frequent items estimation problem if we just return all items $v$ with estimated frequency at least $\frac{n}{k}$.