Traditional Procedures for Inference
When using theory as the basis for inference, there are some standard procedures that can be used as shortcuts. This page will define some of the most important formulas and definitions as a reference for the process. Recall that theory for sampling distributions only apply to some statistics. Therefore, this page will only contain information relevant to those specific statistics. (One of the advantages to simulation-based inference is that we don't need to rely on theory.) The next page will include additional forms of inference that can be performed beyond those included on this page.
Table of Important Characteristics
The following table serves as a reference with important characteristics for each parameter to help when working with confidence intervals or hypothesis testing.
Recall that it is important to confirm any conditions needed by the underlying theory so that the sampling distribution and corresponding inference and conclusions are valid. The most often used condition is included below, but additional conditions are required, and alternative conditions may be acceptable.
| Scenario | Parameter | Statistic | Sampling Distribution | Expected Value | Standard Error |
|---|---|---|---|---|---|
| For Categorical Variables | |||||
| One Proportion | $p$ | $\hat{p}$ | Normal if $np \ge 10$ and $n(1-p) \ge 10$ | $p$ | $\sqrt{\frac{p(1-p)}{n}}$ |
| Difference of Two Proportion | $p_1 - p_2$ | $\hat{p}_1 - \hat{p}_2$ | Normal if $n_1p_1 \ge 10$, $n_1(1-p_1) \ge 10$, $n_2p_2 \ge 10$, and $n_2(1-p_2) \ge 10$ | $p_1 - p_2$ | $\sqrt{\frac{p_1(1-p_1)}{n_1} + \frac{p_2(1-p_2)}{n_2}}$ |
| For Quantitative Variables | |||||
| One Mean | $\mu$ | $\bar{x}$ | Normal if $n \ge 25$ | $\mu$ | $\sqrt{\frac{\sigma}{n}}$ |
| Difference of Two Means | $\mu_1 - \mu_2$ | $\bar{x}_1 - \bar{x}_2$ | Normal if $n_1 \ge 25$ and $n_2 \ge 25$ | $\mu_1 - \mu_2$ | $\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}$ |
| Simple Linear Regression Slope | $\beta_1$ | $b_1$ or $\hat{\beta}_1$ | Normal with LINE conditions | $\beta_1$ | Formula not regularly used by hand |
Important parameter characteristics for theory-based inference.
Common Formulas and Calculations (confidence interval, test statistic, p-value)
With theory-based inference, a few common formulas are used regularly for inference procedures. Some scenarios do also include extensions or alternate versions of these formulas.
Confidence Intervals
Confidence intervals can be calculated using the following formula:
$\text{statistic} \pm \text{multiplier} \times \text{standard error of statistic}$
The statistic to be used can be found in the table above.
The exact standard error of the statistic is provided in the table above. However, we often don't know some component of the standard error. In these cases, we may need to plug in our best estimate for that value, whether it is $\hat{p}$ for $p$ or $\hat{sigma}$ for $\sigma$. Note that we typically rely on technology to provide the estimated standard error for regression rather than use and calculate the components of the formula ourselves.
Finally, we also need to use a multiplier. The multiplier is how we find the central portion of the distribution. To calculate a multiplier, we can use Python to report the percentile of a specific distribution. For our purposes, we will often use the Normal distribution. For confidence intervals for the mean or for linear regression, the exact distribution is often a t distribution; as long as the sample size is large enough, the distinction between the two multipliers is minimal. The multiplier is often represented as $z^{*}$ or $t^*$, to indicate that is a location on a distribution and not a calculation using data.
Test Statistics for Hypothesis Testing
The statistic itself can serve as a test statistic in hypothesis testing, as demonstrated in the hypothesis testing page. We can also adjust this into a standardized test statistic. Standardized test statistics allow values from different distributions to be compared directly. Additionally, standardized test statistics include a meaningful interpretation.
The test statistic is calculated as:
$\text{test statistic} = \frac{\text{statistic} - \text{null value under} H_0}{\text{estimated standard error of statistic under} H_0}$
The test statistic can then be interpreted as the number of standard errors the statistic falls from the hypothesized value for the parameter under the null hypothesis. The sign of the test statistic also indicates whether the statistic is above (positive test statistic) or below (negative test statistic) the hypothesized value.
The test statistic can also be represented with the letters $z$ for proportions and $t$ for means and linear regression. The letter of the test statistic corresponds to the underlying distribution that the test statistic follows.
Reference Distributions
As noted above, we use reference distributions (Normal or t) in the process of completing inference. We will see examples of additional reference distributions on the next page as we further explore regression.
We will use the Normal distribution for all situations, as it serves as a reasonable approximation for the t distribution, provided that the sample size is at least 25.
However, when we are working with means and regression (quantitative variables), the appropriate theoretical distribution is the t distribution. We use the t distribution because we are estimating a second parameter ($\sigma$). The t distribution relaxes the Normal distribution to account for additional uncertainty in the exact value of the standardized test statistic due to the second parameter.
If you do need a more exact p-value, you can accomplish that with the t-distribution.
Note that the t distribution also requires an additional parameter that controls the uncertainty to incorporate into the distribution. This additional parameter for the t distribution is:
- $n-1$ for one mean
- conservatively, we use the minimum of $n_1 - 1$ and $n_2 - 1$ for the difference of two means
- $n - 2$ for simple linear regression
- $n - p - 1$ for multiple linear regression, where $p$ is the number of slopes
Assessing Distributional Assumptions
In the table above, there are conditions that need to be met for the sampling distribution to be appropriate. For most of the distributions, the conditions depend, at least in part, on the sample size $n$.
These are listed because they are the most common and easiest condition to check. However, some assumptions provide alternative assumptions that can be met instead.
For example, the Central Limit Theorem states that if the population is Normally distributed, then the sampling distribution of the sample mean will also be Normally distributed. The sampling distribution of the sample mean will also be Normally distributed if the population is Normally distributed, regardless of the sample size.
How can we check if it is reasonable to assume that a distribution is Normal? The sample distribution should mirror the population distribution. We can check the shape of the sample distribution as one approach to assess if Normality is reasonable.
With some histograms, it is clear that Normality of the population distribution is reasonable. With other histograms, it is clear that Normality of the population is not reasonable. For some histograms, there may not be clear violations but it may be more challenging to determine if Normality is reasonable.
Another method of checking for Normality is to generate a QQ plot, which stands for a quantile-quantile plot. In this plot, the observations in a distribution are compared to the value that you would expect from a perfectly Normal distribution. Ideally, the data would match with the expectation, and the observations on the plot would follow a straight line. If not, the data indicates that Normality is not reasonable for the corresponding population. In this way, more subtle deviations from Normality can be identified, as it is easier to identify deviations from a straight line than from a curve. We saw the use of QQ plots in Module 12 when checking the conditions for regression.
Recall that regression also includes an assumption about Normality. The regression assumption for Normality does not include an alternative, so we do need to confirm Normality of the true errors is reasonable before relying on our inference procedures for regression.
Invalid Conditions
What happens if the necessary conditions are not met?
First, if the sample data is not well collected or representative of the population, then our results won't mean much. If we have garbage in, then we can't draw meaningful conclusions and result in garbage out.
Most remaining conditions relate to the validity of the sampling distribution for the statistic. If the conditions are not met, then the sampling distribution is not valid and any results that rely on that sampling distribution are not valid. We have two options:
- We could change to a simulation-based approach. This approach eliminates the need to specify a sampling distribution, allowing us a more flexible framework and fewer required assumptions to perform inference.
- We could perform the calculations as if the assumption(s) were met, and add the missing assumption as a limitation of the results. If this is true, the validity any p-value calculations or confidence interval calculations would not hold. While these calculations will not be exact, they can still provide an estimate as long as the underlying data is reasonable, with the extent of the violation serving as a guidepost for how bad the estimate might be. For example, if you calculate a p-value of 0.8, it would be surprising if a valid p-value would be extremely small.
Most importantly, if conditions are not valid, be sure to specify this in your discussion as a limitation to any conclusions that you can draw from the procedure.