Orthogonality and Correlation

class: center, middle, inverse, title-slide

# Orthogonality and Correlation
## With unequal cell sizes, why are contrast codes correlated?
### Chris Mellinger
### CU Boulder
### 2016/12/12 (updated: 2020-02-13)

---

Here's some contrast codes and data. There are 3 groups in this design, so we define two codes.

.pull-left[
<table>
 <thead>
  <tr>
   <th style="text-align:left;">   </th>
   <th style="text-align:right;"> A </th>
   <th style="text-align:right;"> B </th>
   <th style="text-align:right;"> C </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> X1 </td>
   <td style="text-align:right;"> -1 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> X2 </td>
   <td style="text-align:right;"> -1 </td>
   <td style="text-align:right;"> 2 </td>
   <td style="text-align:right;"> -1 </td>
  </tr>
</tbody>
</table>
]

---

Correlation coefficient:

`$$r_{xy} = \frac{n \sum{x_i y_i} - \sum{x_i} \sum{y_i}}{\sqrt{n \sum{x^2_i - (\sum{x_i})^2}} \sqrt{n \sum{y^2_i} - (\sum{y_i})^2}}$$`

Ignore the denominator! (our question is just about 0, which implies numerator)

Notice that the numerator is zero when:
- The dot-product is zero (sum of pair-wise multiplication)
- The sum of x and the sum of y are both zero

---

.pull-left[
So with only three people, we know we met those standards because we constructed the codes to do exactly that!
]

---

.pull-left[
It should be evident that if we repeat that same set, we still sum to zero and the dot-product is zero.
]

.pull-right[
<table>
 <thead>
  <tr>
   <th style="text-align:left;"> cond </th>
   <th style="text-align:right;"> X1 </th>
   <th style="text-align:right;"> X2 </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> A </td>
   <td style="text-align:right;"> -1 </td>
   <td style="text-align:right;"> -1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> B </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> C </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> -1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> A </td>
   <td style="text-align:right;"> -1 </td>
   <td style="text-align:right;"> -1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> B </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> C </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> -1 </td>
  </tr>
</tbody>
</table>
]

---

.pull-left[
In fact, we can repeat the set as many times as we want and still meet those criteria, implying a correlation of zero.

This is the situation for a *balanced design*: the same number of people in each group.
]

---

.pull-left[
What about this? We have people from the first two conditions, but not the third.

Obviously, we no longer sum to zero and the dot-product is no longer zero.
]

---

.pull-left[
Any full set of people we add does not fix the issue.

Thus, unbalanced designs imply that there is likely to be at least one set of people containing codes that no longer have the properties we want.
]

.pull-right[
<table>
 <thead>
  <tr>
   <th style="text-align:left;">   </th>
   <th style="text-align:left;"> cond </th>
   <th style="text-align:right;"> X1 </th>
   <th style="text-align:right;"> X2 </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> 1 </td>
   <td style="text-align:left;"> A </td>
   <td style="text-align:right;"> -1 </td>
   <td style="text-align:right;"> -1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 2 </td>
   <td style="text-align:left;"> B </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 4 </td>
   <td style="text-align:left;"> A </td>
   <td style="text-align:right;"> -1 </td>
   <td style="text-align:right;"> -1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 5 </td>
   <td style="text-align:left;"> B </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 6 </td>
   <td style="text-align:left;"> C </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> -1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 7 </td>
   <td style="text-align:left;"> A </td>
   <td style="text-align:right;"> -1 </td>
   <td style="text-align:right;"> -1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 8 </td>
   <td style="text-align:left;"> B </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 2 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 9 </td>
   <td style="text-align:left;"> C </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> -1 </td>
  </tr>
</tbody>
</table>
]

---

.pull-left[
![](OrthogonalButCorrelated_files/figure-html/unnamed-chunk-10-1.png)

]

.pull-right[
Let's look at it another way.

Each person in the full dataset is graphed (some have circles, some triangles, and some pluses so we can see all three at each point; they are also jittered a little).
]

---

.pull-left[
![](OrthogonalButCorrelated_files/figure-html/unnamed-chunk-11-1.png)

]

.pull-right[
Recall that any best fit line must pass through the mean of each variable.
]

---

.pull-left[
![](OrthogonalButCorrelated_files/figure-html/unnamed-chunk-12-1.png)

]

.pull-right[
Now, every point gets a slope vote. Here are the rules about slope votes:

- Weight: the distance from the mean on `$X$`, here Code 1; **leverage**
- Magnitude: the angle from that point to the mean on both variables.

Observations:

- Points at the mean (the three on top) don't get votes because they are equal to `$X$`'s mean, so they don't count.
- Each of the others has an exactly equal and opposite partner!

]

---

.pull-left[
![](OrthogonalButCorrelated_files/figure-html/unnamed-chunk-13-1.png)

]

.pull-right[
What if we remove 1 person from the dataset?

Observations:

- Now, all of the points get a slope vote (because the mean changed).
- Those votes are a little haphazard and in no way balanced. This leads to a relationship, or a non-horizontal best-fit line.

]

---

Technically speaking, **orthogonality** is a linear algebra term, and **correlation** is a statistical term.

**Orthogonal**: Two vectors whose dot-product is zero.

**Uncorrelated**: A change in one variable gives no information about the average change in the other.

Similar, but not identical. Dot-products (orthogonality) and product-moments (correlation) are different procedures mathematically, and are used in different situations:

- orthogonality: Mathematically meaningful sets of numbers that characterize mathematical situations.
- correlation: **Random variables**, assumed to be measured with error, that we want to examine trends for.

Caveat: it's easy to use "orthogonality" language when talking about non-correlations because of the conceptual similarity. That's ok, but know the difference for yourself:

### Orthogonal contrast codes do not imply totally uncorrelated variables!

for unbalanced studies