Independence and sufficiency

Today

Trying again with independence/sufficiency visualization and checks

What does independence look like for two categorical variables? (mosaic plots)
What does independence look like for two continuous variables?
Same questions for conditional independence
What if we have more than two variables? (high-dimensional visualization + random projections)

Independent random variables

Recall from Lecture 2: random variables are independent if probabilities of simultaneous events can be obtained by multiplying their separate probabilities.

For two random variables, this means

\[ P(X\in A,\ Y\in B)=P(X\in A)P(Y\in B) \]

for every pair of appropriate sets $A$ and $B$.

It is not a bad approximation to think of $A$ and $B$ as being intervals for continuous random variables and individual outcomes for categorical random variables.

Independence/dependence for categorical variables

If $X$ and $Y$ take values in finite/countable sets, we can choose $A=\{x\}$ and $B=\{y\}$.

Independence means

\[ P(X=x,Y=y)=P(X=x)P(Y=y) \]

for every pair of categories.

For example, if $P(X=1)=P(Y=1)=1/2$, independence requires $P(X=1,Y=1)=1/4$.

Visualizing categorical random variables: Mosaic plot

$One two-way mosaic. Column widths show P(X=x); heights within a column show P(Y=y\mid X=x).$

Each column represents a category of $X$.

Within each column, the boxes represent categories of $Y$.

The area of the $(x,y)$ box is the fraction of observations with $X=x$ and $Y=y$.

Width, height, and area

For the box corresponding to $X=x$ and $Y=y$:

Part of the box	Probability represented
Column width	$P(X=x)$
Height within the column	$P(Y=y\mid X=x)$
Area	$P(X=x,Y=y)$

The geometry follows the probability identity

\[ P(X=x,Y=y)=P(X=x)P(Y=y\mid X=x). \]

A mosaic under independence

Independence implies

\[ P(Y=y\mid X=x)=P(Y=y) \]

whenever $P(X=x)>0$.

Independent categorical mosaic. The dashed line shows the aligned horizontal boundary.

The proportions of the $Y$-categories are the same in every column, so the horizontal boundaries line up.

The boxes can still have different areas.

A mosaic showing dependence

Everyone aboard the Titanic, by sex ($X$) and survival ($Y$).

Left: the table independence would predict from the observed margins. Right: the observed table. The dashed line is the overall proportion who died.

The distribution of $Y$ changes between the $X$-categories.

Knowing $X$ tells us something about $Y$.

A mosaic plot of simulated independent data

Mosaics from three independent samples of (X,Y). Dotted lines mark the population proportion P(Y=0)=0.6.

In a finite sample, the observed proportions will differ a little from one column to another.

We can simulate independent samples to see how much variation to expect.

Independence/dependence for continuous variables

Flipper length ($X$) and body mass ($Y$) for 342 penguins near Palmer Station, Antarctica.

Pair	How it is made
As measured	Each penguin’s own flipper length and body mass
Shuffled	Body masses randomly reassigned among the penguins

Shuffling keeps both lists of values exactly, so $X$ and $Y$ have the same distributions in both pairs.

Shuffling makes the pair independent. Their joint distributions differ.

Scatterplots of the two pairs

All 342 penguins, with body masses shuffled (left) and as measured (right).

For the independent pair, knowing approximately where $X$ lies does not change the distribution we expect for $Y$.

For the dependent pair, the distribution of $Y$ moves upward as $X$ increases.

How can we connect these pictures to the definition of independence?

Option 1: Discretize both variables and make a mosaic plot

Divide each axis into intervals $A_1,\ldots,A_4$ and $B_1,\ldots,B_4$.

Record the interval containing each observation:

\[ X_{\mathrm{bin}}=i\quad\text{if }X\in A_i, \qquad Y_{\mathrm{bin}}=j\quad\text{if }Y\in B_j. \]

Now we have two categorical variables, so we can make a mosaic.

Cuts at the sample quartiles of each variable, with bin numbers on each axis.

What independence entails for the boxes

Independence means

\[ P(X\in A_i,Y\in B_j)=P(X\in A_i)P(Y\in B_j). \]

Each interval holds about a quarter of the penguins, so each box should contain about $1/16$ of them.

Let’s check.

The independent pair, discretized

Shuffled pair: one scatterplot rectangle and its mosaic box.

Each mosaic box represents a rectangle in the original scatterplot:

\[ \{X\in A_i,\ Y\in B_j\}. \]

The dependent pair, discretized

Penguins as measured, same bins. The outlined cells are the bottom-left and top-right boxes.

The marginal distributions are exactly the same as before.

But observations concentrate in boxes where $X$ and $Y$ are both low or both high.

The conditional proportions change across the mosaic columns.

Notes

How we read the mosaic for a discretized continuous variable

Part of the mosaic box	Probability represented
Column width	$P(X\in A_i)$
Height within the column	$P(Y\in B_j\mid X\in A_i)$
Area	$P(X\in A_i,Y\in B_j)$

Under independence, every column has the same conditional proportions.
A coarse set of bins can hide dependence. The definition must hold for every pair of sets, not just the bins in one plot.
The above is not possible to check exhaustively with finite data.

Option 2: Discretize only one variable and visualize the distribution of the other

Instead of categorizing $Y$, make a histogram of its values within each $X$-interval.

Histograms of body mass within each flipper-length interval. Top: shuffled. Bottom: as measured. Each histogram sums to 1.

Each panel approximates $Y \mid X \in A_i$.

If $X$ and $Y$ are independent, the distributions shouldn’t change with $A_i$.

If $X$ and $Y$ are dependent, the distributions will change with $A_i$.

The same information shown a different way

Mirrored histograms of body mass within each flipper-length interval, with a common scale. Orange lines are the means; dotted lines are the quartile cuts.

Each strip (a mirrored histogram) summarizes the distribution of $Y$ in one vertical strip of the scatterplot.

Under independence, the histograms should agree in location, spread, and shape.

A sloping mean is one way to see dependence. Changes in spread or shape also count.

Option 3: Look at the scatterplot

All 342 penguins, with body masses shuffled (left) and as measured (right).

Look at individual strips of the $x$-axis variable. Does the distribution of the $y$-axis variable look the same in each?

Conditional independence

$X$ and $Y$ are conditionally independent given $Z$ if

\[ \begin{aligned} &P(X\in A,Y\in B\mid Z=z)\\ &\qquad=P(X\in A\mid Z=z)P(Y\in B\mid Z=z) \end{aligned} \]

for every pair of sets $A,B$ and almost every conditioning value $z$.

We write $X\perp Y\mid Z$.

After fixing $Z$, knowing $X$ does not change the distribution of $Y$.

Visualizing conditional distributions

Recall how we simulated conditional distributions:

Generate joint outcomes $(X,Y,Z)$.
Keep only those satisfying the condition.
Study distributions within that collection.

Therefore:

To inspect $(X,Y)\mid Z=z$, subset to observations with $Z=z$ and visualize $X$ and $Y$ on that subset.
For continuous variables, we usually subset to observations with $Z \approx z$.
We can use the plots from before on the subset.

Conditioning on a categorical variable, penguin example

Return to the discretized penguin measurements:

$X$: flipper length, cut into thirds within the species.
$Y$: body mass, cut into thirds within the species.
$Z$: species.

Suppose we want to know about the distribution $(X, Y) \mid Z = \text{Gentoo}$

Conditional independence predicts aligned horizontal boundaries within that species.

Conditioning: keep the Gentoo rows

Left: flipper length and body mass for all penguins, colored by species. Right: only the Gentoo rows, which is what conditioning on species = Gentoo means.

Conditioning on $Z = \text{Gentoo}$ means throwing away every other row.

The Gentoo rows, discretized

The Gentoo rows cut at Gentoo tertiles, their counts, and the Gentoo mosaic.

Does the distribution of body-mass categories change across flipper-length categories?

If it does, fixing species has not removed the relationship.

FIGURE CI01b: Use the same 342 complete-measurement penguins as the earlier slides. Within each species, cut flipper length and body mass at that species’ own tertiles, with half-open intervals. The dashed lines in the scatterplot are the Gentoo cuts: flipper 214, 220 mm; mass 4800, 5350 g. Gentoo table (rows flipper, columns mass, Low/Mid/High): [[26,8,4],[13,21,8],[1,12,30]]. Ties at the cuts make the flipper groups slightly unequal (38/42/43). Why per-species cuts: the global quartile cuts from the earlier slides left some categories nearly empty within a species (Gentoo flippers are all >=203 mm, so only the top two global flipper bins appear), and the small cells exaggerated apparent variability. Within-species tertiles give each category about a third of the species, so categories mean “a long flipper for a Gentoo”. In this setup the binned variables are functions of the original continuous measurements, so independence of the full measurements given species implies independence of these categorical versions given species. These are descriptive comparisons, not a formal conditional-independence test.

Comparing conditional distributions

Gentoo body masses within each third of Gentoo flipper lengths. Each histogram sums to 1.

Each histogram represents

\[ Y\mid X\in A_i,\ Z=\text{Gentoo}. \]

Under conditional independence, these distributions should agree across the $X$-intervals.

The common distribution may be different for a different species.

How do we condition on a continuous variable?

When the conditioning variable was categorical (species), we can select rows with exactly $Z=z$.

With a continuously distributed conditioning variable $Z$, a finite simulation bank will not contain rows with exactly a chosen value.

Instead, we might keep a range:

\[ z_0-h<Z<z_0+h. \]

That range combines different values of $Z$.

Example: Packing and posting a parcel

Consider a simplified warehouse simulator:

$Z$: the parcel’s weight.
$X$: the time needed to pack it.
$Y$: the postage charge.

Heavier parcels tend to take longer to pack and cost more to post.

At a fixed weight, the simulator generates the extra packing time and extra postage charges independently.

Thus, in this model, $X\perp Y\mid Z$.

This is a synthetic teaching model, not a claim that weight makes real packing times and postage conditionally independent. Other shared features could matter in an actual warehouse. The separate packing and shipping mechanisms make the conditional-independence assumption easy to explain. CODE CI04: Generate N=100000 rows with parcel weights Z~Gamma(shape=2, scale=1), measured in kg. Independently generate packing allowance U~Exponential(mean=.5), in minutes, and postage surcharge V~Exponential(mean=1), in dollars. Set X=2+Z+U and Y=3+2Z+V. All three input draws are independent. Means are illustrative choices, not empirical estimates or current postal prices. Equivalently, a 2kg parcel has packing time 4+U minutes and charge 7+V dollars. This replaces the earlier abstract uniform-plus-normal example with a concrete positive-valued simulator. Keep these formulas in notes or code unless students ask for the exact recipe.

Parcels in a weight range

Suppose we just have data from this model and don’t know how it was generated.

Suppose further that we want to know whether $X$ and $Y$ are conditionally independent given $Z$.

To do so, we investigate the distribution of $(X, Y) \mid Z$.

We do so by plotting observations from the simulation bank that satisfy $Z \in (z_0 - h, z_0 + h)$.

We’ll start with $z_0 = 2$ and $h = 1, .3, .05$.

Parcels from one simulation bank whose weights fall in three ranges around 2 kg.

FIGURE CI05 / CODE CI06: Filter the SAME CODE CI04 joint bank with h=1,.3,.05 around z0=2. Display scatterplots with common axes and explicit retained n. Use one reproducible plotting subset if the wide panel is too dense, but label both retained and displayed counts. Keep the direct Z=2 panel available for comparison. Do not rescale each panel separately to force similar appearances. The CODE CI04 simulator gives Cov(X,Y|Z in C)=2*Var(Z|Z in C), because selection depends only on Z and the independent U,V distributions remain unchanged. Thus any nondegenerate retained weight range has positive covariance here. As ranges shrink to 2, that covariance approaches zero. The range need not be symmetric in its actual weight distribution: Gamma weights are not uniform. Numerical integration or the simulated bank can supply the conditional variance if desired, but no calculation is needed on the slide. Generated (seed 1441): retained n = 53777, 16350, 2671; correlations .55, .12, .00; sample covariances .608, .065, .0004 against 2Var(Z|range) = .614, .060, .0017. Axes match the 2 kg figure.

Postage within packing-time strips, $h = 1$

Parcels with weights between 1 and 3 kg. In each packing-time strip, the outline is a mirrored histogram of postage; orange lines are strip medians.

Postage shifts upward from strip to strip.

Postage within packing-time strips, $h = 1$

Parcels with weights between 1 and 3 kg, colored by weight. In each packing-time strip, the outline is a mirrored histogram of postage; orange lines are strip medians.

Color by weight: the strips differ because they hold parcels of different weights.

FIGURE CI07 (all three slides): The CI05 scatterplot, with five equal-width packing-time strips spanning the 1st-99th percentiles of the retained parcels (end strips also take the tails). Each strip has a mirrored histogram of postage centred on the strip, with half-width proportional to the conditional proportion and one scale per figure; each outline covers the middle 99% of its strip’s postage, so single extreme parcels don’t draw hairline spikes. Each range gets two slides: gray points first, so students read the strips on their own, then the same figure with points colored by weight. The color scale runs over each figure’s own range (2-h to 2+h), not a fixed 1-3 kg, because a narrow range can still hide weight differences that drive a pattern. With h = 1 the colors run from purple (light, lower left) to yellow (heavy, upper right): the strips differ because they contain parcels of different weights. With h = .3 the first strip is mostly purple, matching its lower median. With h = .05 the colors are mixed across strips, so within-range weight is not lining up with packing time. Axes match CI05. h = 1: strip edges 3.13, 3.80, 4.46, 5.13, 5.79, 6.46 min; n = 14059, 17617, 14456, 5611, 2034; medians 6.35, 7.30, 8.42, 8.84, 8.84.

Postage within packing-time strips, $h = 0.3$

Parcels with weights between 1.7 and 2.3 kg. In each packing-time strip, the outline is a mirrored histogram of postage; orange lines are strip medians.

A smaller shift remains, mostly between the first strip and the rest.

Postage within packing-time strips, $h = 0.3$

Parcels with weights between 1.7 and 2.3 kg, colored by weight, with mirrored histograms of postage within packing-time strips.

The first strip holds mostly the lighter parcels in the range, which is where the remaining shift comes from.

Postage within packing-time strips, $h = 0.05$

Parcels with weights between 1.95 and 2.05 kg. In each packing-time strip, the outline is a mirrored histogram of postage; orange lines are strip medians.

The distributions agree, up to noise in the sparse right-hand strips.

Postage within packing-time strips, $h = 0.05$

Parcels with weights between 1.95 and 2.05 kg, colored by weight, with mirrored histograms of postage within packing-time strips.

Light and heavy parcels within the range are mixed across all the strips.

Interpreting a conditional-independence plot

When we filter to a range, a visible relationship can come from dependence remaining at a fixed conditioning value, or from mixing conditioning values.

Before interpreting the plot, ask:

What exactly did we condition on?
How much does the conditioning variable still vary within the retained collection?
What happens when we narrow the range, and how many observations remain?

Next big problem

We have seen how to check for independence between two random variables and how to check for conditional independence between two random variables given a third.

What if we want to check for independence/conditional independence between a random vector (e.g. a whole dataset) and a random variable?

Motivation: For ABC, wanted to know whether $S(X)$ is a sufficient statistic, (that is, whether $X$ and $\theta$ were conditionally independent given a statistic $S(X)$).

Independence with a vector

Suppose we want to know whether $\theta$ is independent of a vector

\[ \mathbf X=(X_1,\ldots,X_p). \]

We could plot $\theta$ against each coordinate.

But $\theta$ can be independent of every individual coordinate and still depend on the combination.

\[ \theta\perp X_j\text{ for every }j \quad\text{does not imply}\quad \theta\perp\mathbf X. \]

Example: Ten days of rain

Recall our weather model from before.

Let $X_t=1$ if day $t$ is rainy and $X_t=0$ if it is dry.

Day 1 is equally likely to be rainy or dry.
Each subsequent day repeats the previous day’s weather with probability $\theta$.
Otherwise, the weather switches.

Each simulation produces a vector

\[ \mathbf X=(X_1,\ldots,X_{10}). \]

Large $\theta$ gives long wet and dry spells. Small $\theta$ gives frequent switches.

Simulated weather sequences

$Twenty simulated ten-day sequences at each of three values of \theta, in simulation order.$

Each row is one simulation. Each column is one day.

$X_i$ and $\theta$ are independent

Suppose we plot $\theta$ against $X_1$, then $X_2$, and so on.

$\theta for dry and rainy outcomes, one panel per day: mirrored histograms from 100000 joint simulations, over 1500 jittered points. Orange lines are medians.$

On every day, $\theta$ has the same distribution among the dry and the rainy simulations.

But the entire vector $\mathbf X$ is not independent of $\theta$

Create a “total rainy days” statistic: $R(\mathbf X)=X_1+\cdots+X_{10}.$

$\theta and the total number of rainy days for the first 3000 joint simulations, with the total jittered vertically.$

But the entire vector $\mathbf X$ is not independent of $\theta$

$The same scatterplot, with a mirrored histogram of the total within each fifth of \theta (all 100000 simulations). Orange lines are the mean total in each strip.$

The distribution of the total changes with $\theta$.

FIGURE RAIN03a/b: Joint bank from RAIN01 (seed 1451, N = 100000). (a) theta on the horizontal axis, total rainy days R on the vertical axis, jittered by up to .3 so the integer rows are visible; first 3000 rows. (b) The same plot with five equal theta strips and, in each, a mirrored histogram of R (all rows, integer bins, one width scale), matching the earlier violin displays. Orange lines: mean R in each strip, 5.00, 5.01, 4.99, 5.00, 5.01; sd R = .62, 1.10, 1.59, 2.35, 3.82. The mean is flat, so the dependence is in the spread and shape: tight around 5 for small theta, piled at 0 and 10 for large theta. Hairline tails mark totals that occur only rarely in a strip. For the next slide, fixed-theta simulations (seed 1452, 20000 per theta): sd R = .86, 1.57, 3.68 at theta = .2, .5, .9 (theta=.5 matches Binomial(10,.5) sd 1.58); P(R in {0,10}) = 0, .002, .386; exact at theta=.9 is .9^9 = .387.

What the combination tells us

If $\theta$ were independent of $\mathbf X$, it would also be independent of every function of $\mathbf X$.

But $R(\mathbf X)$ has a distribution that changes with $\theta$.

Therefore,

\[ \theta\not\perp\mathbf X, \qquad\text{even though}\qquad \theta\perp X_t\text{ for every }t. \]

Checking individual coordinates can miss information in how they vary together.

Other combinations of the coordinates

The total is a linear combination:

\[ R(\mathbf X)=\mathbf 1^\mathsf T\mathbf X. \]

More generally, we could inspect

\[ h_{\mathbf a}(\mathbf X)=a_1X_1+\cdots+a_{10}X_{10}. \]

We may not know which weights reveal a change in distribution.

One option is to try randomly chosen weights and compare the resulting distributions across values of $\theta$.

Related: Tours

A tour smoothly changes the projection direction so that we can inspect many views of the same data.

Some views reveal structure that others conceal.

Example: Cook and Laa, Interactively Exploring High-Dimensional Data and Models in R, Chapter 1

The two clusters are clear in some projections and overlap in others.

A tour for an independence question

To examine whether $Y\perp\mathbf X$, keep $Y$ on one axis and change the projection on the other:

\[ \text{plot }Y\text{ against }T_a=a^\top\mathbf X. \]

Under independence, the distribution of $T_a$ should stay the same across values of $Y$, for every direction $a$.

We look for changes in location, spread, or shape.

Random projections and numerical checks

We can inspect these views visually, or measure dependence numerically.

Draw a direction $a$ and compute $T_a=a^\top\mathbf X$.
Compare the distributions of $T_a$ across values of $Y$.
Repeat for other directions.

For example, discretize $T_a$ and $Y$, then compare joint cell proportions with products of marginal proportions.

Results from generate_figures.R (K=50, 200 permutations): the maximum score over directions is about 1563, against a largest permuted maximum of about 41 (p=1/201, the smallest possible). The coordinate scores are 15.1 and 5.0 (9 df each). cor(Y, a’X) lies within +/-.002 for every direction. CODE H03: Draw K=50 directions by normalizing independent standard Gaussian vectors in R^p. Freeze the directions before checking the data. Compute each projection, bin Y into fixed intervals, and bin each projection using its empirical quartiles, computed once and held fixed. A possible dependence score is Pearson’s sum of (observed-expected)^2/expected over the binned table. Record one score per direction, and use their maximum as an overall score. Directions with a1a2 near zero have little signal in the 2D example. Directions with a1a2 nonzero have conditional variance 1+2a1a2*y for unit a. E[T_a|Y]=0 implies Cov(Y,T_a)=0 for EVERY direction, despite dependence. For a formal test, permute whole X rows relative to Y, use the SAME directions and binning rules, and recompute the maximum score for EACH permutation. Permuting whole rows preserves internal dependencies among X coordinates. Do not treat many unadjusted per-direction p-values as one valid test. This is a numerical use of projections, not a claim that projections are inherently nonvisual. Fixed empirical projection cutpoints are invariant to row permutations. If directions or features are learned using Y, rerun that selection in every permutation or use independent validation data.

Why inspect projections?

For numeric vectors,

\[ Y\perp\mathbf X \quad\Longleftrightarrow\quad Y\perp a^\top\mathbf X\text{ for every direction }a. \]

All one-dimensional projections determine a multivariate distribution.

A finite collection of plots or numerical checks can still miss dependence, especially with limited data.

Returning to sufficiency

For a sufficiency check, first restrict simulations to the observed summary, or a neighborhood of it.

Then inspect whether any projection of the remaining data is related to the parameter, either graphically using a tour or numerically using random projections:

\[ \theta\perp a^\top\mathbf X\mid S(\mathbf X). \]

Both inherit the earlier limitations of approximate conditioning and finite samples.

What we saw

How to visualize independence/dependence of random variables using mosaic plots or scatter plots.
How to visualize conditional independence/dependence of random variables.
Dependence of a random vector $\mathbf X = (X_1,\ldots, X_p)$ and another random quantity cannot be checked using only the individual coordinates $X_i$.
We can use can use random projections or tours for independence of random vectors, but it’s always possible to miss something.
Cook and Laa, Interactively Exploring High-Dimensional Data and Models in R, Chapter 1. Projections and tours.