Weighting Adjustments
for Nonresponse in Surveys
PUBHBIO 7225 Lecture 18
Topics
Activities
Readings
Assignments
Not random variables:
\(y_i\) = survey outcome of interest for unit \(i\), only observed for respondents
\(\mathbf{x}_i\) = vector of information known about unit \(i\), observed for ALL sampled units
(design information, information on the sampling frame, or information about the interview process)
Random variables:
\(Z_i\) = selection indicator for unit \(i\)
\(R_i\) = response indicator for unit \(i\)
Three types of missing data mechanisms:
MCAR: \(P(R_i=1 | y_i, \mathbf{x}_i) = P(R_i=1)\) ← missingness doesn’t depend on \(y\) or \(x\)
MAR: \(P(R_i=1 | y_i, \mathbf{x}_i) = P(R_i=1|\mathbf{x}_i)\) ← missingness doesn’t depend on \(y\)
MNAR: \(P(R_i=1 | y_i, \mathbf{x}_i)\) ← missingness depends on \(y\), possibly also \(x\)
If \(R_i \perp \!\!\! \perp Z_i\) (response is independent of selection into the sample), then: \[\begin{aligned} P(\text{unit }i & \text{ selected in sample and responds}) \\ & = P(\text{unit }i\text{ selected in sample})\times P(\text{unit }i\text{ responds}|\text{unit }i\text{ selected in sample}) \\ & = P(\text{unit }i\text{ selected in sample})\times P(\text{unit }i\text{ responds}) \\ & = \pi_i \phi_i \end{aligned}\]
We adjust the weights by estimating the \(\phi_i\) and then assigning the weights: \[w_i = \begin{cases} \frac{1}{\pi_i \hat{\phi}_i} & \text{for respondents }(R_i=1) \\ 0 & \text{for nonrespondents }(R_i=0) \end{cases}\]
Weighting methods assume that response probabilities can be estimated from variables known for all units
Info for estimating \(\phi_i\) may be limited for some surveys
Weighting Class Adjustments = create groups (“classes”) of respondents that have similar \(\mathbf{x}_i\) and estimate response propensity (\(\hat{\phi}\)) within each group (“class”)
Steps:
Group respondents and nonrespondents based on \(\mathbf{x}_i\) to create weighting adjustment classes
Estimate response probability within each class with either the unweighted or weighted response rate:
\[\hat{\phi}_c = \frac{\text{number of respondents in class } c}{\text{number of selected units in class }c} \quad \text{ or } \quad \hat{\phi}_c = \frac{\text{sum of weights for respondents in class } c}{\text{sum of weights for selected units in class }c}\]
Simple method, but very commonly used
Implicitly assumes probability of response is the same for all units within each class
All weights are up-weighted since response propensities are between 0 and 1
Alternatives to weighting classes are possible, using alternative methods to estimate \(\hat{\phi}_i\)
Ex: Logistic regression with \(R_i\) as the outcome and \(\mathbf{x}_i\) as predictors
Each respondent has predicted propensity from the model, \(\hat{\phi}_i\), and adjusted weight \(\tilde{w}_i = w_i/\hat{\phi}_i\)
Or, create classes based on grouping units with similar \(\hat{\phi}_i\)
Population of \(N=1{,}000\) graduates with PhDs in statistics/biostatistics in past year
SRS of \(n=10\) with response rate of 6/10 = 60%
Type of degree (biostat vs stat) is on the sampling frame
Type of job (academic vs not) is on the survey
Based on respondents:
Estimated proportion in Academic jobs = __________
Estimated total number in Academic jobs = _________
| ID | Biostat (\(x_i\)) | Academic (\(y_i\)) | Weight (\(w_i\)) |
|---|---|---|---|
| 1 | 0 | 1 | N/n = 1000/10 = 100 |
| 2 | 1 | 100 | |
| 3 | 0 | 100 | |
| 4 | 1 | 1 | 100 |
| 5 | 0 | 0 | 100 |
| 6 | 1 | 0 | 100 |
| 7 | 0 | 0 | 100 |
| 8 | 0 | 100 | |
| 9 | 1 | 1 | 100 |
| 10 | 0 | 100 |
Don’t like these estimates? We have two choices:
Treat the entire sample as one adjustment class, and upweight all 6 respondents the same (MCAR)
Group the sample by degree type, and apply a different adjustment factor by degree (MAR)
Option 1: Assume MCAR and use one big weighting adjustment class
Estimated probability of response = \(\hat{\phi}_i\) = 6/10 = 0.6 = same for all units
| ID | Biostat (\(x_i\)) | Academic (\(y_i\)) | Base Weight (\(w_i\)) | Response Propensity (\(\hat{\phi}_i\)) | Adj. Weight: \(\tilde{w}_i = w_i / \hat{\phi}_i\) |
|---|---|---|---|---|---|
| 1 | 0 | 1 | 100 | 6/10 = 0.6 | 100/0.6 = 166.67 |
| 2 | 1 | 100 | 0 | ||
| 3 | 0 | 100 | 0 | ||
| 4 | 1 | 1 | 100 | 0.6 | 166.67 |
| 5 | 0 | 0 | 100 | 0.6 | 166.67 |
| 6 | 1 | 0 | 100 | 0.6 | 166.67 |
| 7 | 0 | 0 | 100 | 0.6 | 166.67 |
| 8 | 0 | 100 | 0 | ||
| 9 | 1 | 1 | 100 | 0.6 | 166.67 |
| 10 | 0 | 100 | 0 | ||
| Sum: 1000 | Sum: 1000 |
Estimated total number of grads in Academic jobs: \(\hat{t} = \sum_{i \in \mathcal{S}} \tilde{w}_i y_i = 166.67(1) + 166.67(1) + 166.67(0) + \cdots = 500\)
Estimated proportion of grads in Academic jobs: \(\hat{p} = \frac{\sum_{i \in \mathcal{S}} \tilde{w}_i y_i}{\sum_{i \in \mathcal{S}} \tilde{w}_i} = \frac{500}{1000} = 0.5\)
Compared to using only respondents (“complete case”): proportion is the same (MCAR!) and total is different (better)
| Degree Type | Sample Size | Number Responded | Response Rate |
|---|---|---|---|
| Biostatistics | 4 | 3 | 3/4 = 75% |
| Statistics | 6 | 3 | 3/6 = 50% |
Response rate is different for by degree type!
(different for biostat and stat graduates)
Not MCAR!
| ID | Biostat (\(x_i\)) | Academic Job \((y_i)\) | Base Weight \((w_i)\) |
|---|---|---|---|
| 1 | 0 | 1 | 100 |
| 2 | 1 | 100 | |
| 3 | 0 | 100 | |
| 4 | 1 | 1 | 100 |
| 5 | 0 | 0 | 100 |
| 6 | 1 | 0 | 100 |
| 7 | 0 | 0 | 100 |
| 8 | 0 | 100 | |
| 9 | 1 | 1 | 100 |
| 10 | 0 | 100 |
Suppose in reality biostat grads were more likely than stat grads to be in academic jobs
Option 2: Assume MAR and create weighting adjustment classes based on degree
Estimated probability of response = \(\hat{\phi}_i\) = separately estimated for biostat, stat
Sorting the data by biostat/stat:
| ID | Biostat (\(x_i\)) | Academic (\(y_i\)) | Base Weight (\(w_i\)) | Response Propensity (\(\hat{\phi}_i\)) | Adj. Weight: \(\tilde{w}_i = w_i / \hat{\phi}_i\) |
|---|---|---|---|---|---|
| 1 | 0 | 1 | 100 | 3/6 = 0.5 | 100/0.5 = 200 |
| 3 | 0 | 100 | 0 | ||
| 5 | 0 | 0 | 100 | 0.5 | 200 |
| 7 | 0 | 0 | 100 | 0.5 | 200 |
| 8 | 0 | 100 | 0 | ||
| 10 | 0 | 100 | 0 | ||
| 2 | 1 | 100 | 0 | ||
| 4 | 1 | 1 | 100 | 3/4 = 0.75 | 100/0.75 = 133.33 |
| 6 | 1 | 0 | 100 | 0.75 | 133.33 |
| 9 | 1 | 1 | 100 | 0.75 | 133.33 |
| Sum: 1000 | Sum: 1000 |
| ID | Biostat (\(x_i\)) | Academic \((y_i)\) | Base Weight (\(w_i\)) | Response Propensity (\(\hat{\phi}_i\)) | Adj. Weight: \(\tilde{w}_i = w_i / \hat{\phi}_i\) |
|---|---|---|---|---|---|
| 1 | 0 | 1 | 100 | 3/6 = 0.5 | 100/0.5 = 200 |
| 3 | 0 | 100 | 0 | ||
| 5 | 0 | 0 | 100 | 0.5 | 200 |
| 7 | 0 | 0 | 100 | 0.5 | 200 |
| 8 | 0 | 100 | 0 | ||
| 10 | 0 | 100 | 0 | ||
| 2 | 1 | 100 | 0 | ||
| 4 | 1 | 1 | 100 | 3/4 = 0.75 | 100/0.75 = 133.33 |
| 6 | 1 | 0 | 100 | 0.75 | 133.33 |
| 9 | 1 | 1 | 100 | 0.75 | 133.33 |
| Sum: 1000 | Sum: 1000 |
Estimated total number of graduates who are in Academic jobs:
\(\hat{t} = \sum_{i \in \mathcal{S}} \tilde{w}_i y_i = 200(1) + 200(0) + 200(0) + 133.33(1) + 133.33(0) + 133.33(1) = 466.67\)
Estimated proportion of graduates who are in Academic jobs:
\(\hat{p} = \frac{\sum_{i \in \mathcal{S}} \tilde{w}_i y_i}{\sum_{i \in \mathcal{S}} \tilde{w}_i} = \frac{466.67}{1000} = 0.47\)
Compared to using only respondents (“complete case”): proportion and total are different
Can only use variables \(x_i\) that are observed for both respondents and nonrespondents
Limited to information on the sampling frame and design variables
Ex: 2021 Ohio Medicaid Assessment Survey (OMAS) (link)
In multi-stage surveys, nonresponse can happen at different stages, thus can make different adjustment classes for each level
Ex: 2017-2020 National Health and Nutrition Examination Survey (NHANES) (link to methods report)
Stages: First dwelling units are screened (to enumerate people in them), then sampled people are interviewed, then interviewed people are given a medical exam
Nonresponse can happen at any stage – refuse to participate in the screening, or the interview, or the medical exam
Screener adjustment classes: census tract/segment (only info on sampling frame!)
Interview adjustment classes: can use screener info
Medical exam adjustment classes: can use interview info
Overall goal: choose variables that are associated with \(\mathbf{R_i}\) and with \(\mathbf{y_i}\)
Units within each class are as similar as possible with respect to \(y_i\), and
Response propensities \(\hat{\phi}_i\) similar within class but (may) vary from class to class
Impact on bias and variance when constructing classes using variables that have: 1
| Association with \(R_i\) | Association with \(y_i\) | Impact on Bias | Impact on Variance |
|---|---|---|---|
| Low | Low | – | – |
| Low | High | – | ↓ |
| High | Low | – | ↑ |
| High | High | ↓ | ↓ |
Only get bias reduction if classes are constructed with variables related to \(y_i\) and \(R_i\)
“Punishment” for using variables not associated with \(y_i\) can be an increase in variance
Obvious problem: All surveys have more than one \(y\) that they care about!
Nonresponse Weight Adjustments
If >50% nonresponse in a class, weights would be multiplied by a factor greater than 2 – and this is generally avoided
Option 1: Collapse classes together until have enough respondents
Option 2: Weight trimming, by choosing a max value for the weights and truncating any adjusted weights above this value (and redistributing “excess” weight across all units)
When adjusting weights for nonresponse, we are multiplying each weight by an adjustment factor: \[\tilde{w}_i = w_i \times \frac{1}{\hat{\phi}_i} = w_i \times \text{(adjustment factor)}\]
Similar to poststratification: \[w_{hj}^{PS} = w_{hj} \times \frac{\text{Population total in post-stratum}~h}{\text{Estimated total in post-stratum}~h} = w_{hj} \times \frac{N_h}{\widehat{N}_h}\]
Weighting adjustments for nonresponse can be combined with poststratification:
First, make nonresponse adjustments to the weights
Second, use poststratification or raking to ensure that weights sum to population totals for key quantities (for which you know the population totals)
The adjustment factors just all get multiplied together: \(\tilde{w}_i = w_i \times \frac{1}{\hat{\phi}_i} \times \frac{N_h}{\widehat{N}_h}\)
where unit \(i\) belongs to poststratum \(h\)
Key difference between nonresponse weight adjustments and poststratification/raking:
For nonresponse weight adjustments, you must use variables observed for respondents and nonrespondents
For poststratification/raking, you need population totals for variables that are observed for respondents (individual nonrespondent data not needed)
Think of it this way:
Nonresponse weight adjustments are made to align the respondent sample with the originally selected sample (i.e., make the respondent sample look like the full sample)
Poststratification/Raking/Calibration are then made to align the resulting sample with the target population
Other weight adjustments are commonly done in certain scenarios as well, e.g.,:
Ineligibility adjustment – some sampled units may not be eligible (e.g., phone number that does not belong to a person in the target population), so adjust weights to reallocate the weights for these ineligible units
Dual-frame adjustment – for surveys that use two frames (e.g., cell phone frame and landline frame), adjust for the fact that some units could appear on both frames and thus have a higher chance of selection into the sample
Number of phone numbers adjustment – in phone surveys, people with more than one phone number have higher probability of selection (but don’t know who has multiple numbers at the design stage), so adjust weights accordingly (higher probability of selection \(\rightarrow\) lower weight)
PUBHBIO 7225