--- title: "Pure panels: the longitudinal weight and gross flows" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Pure panels: the longitudinal weight and gross flows} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>") library(weightflow) set.seed(20260910) # Figures are kept as standalone .svg files and embedded as data URIs. An inline #

', knitr::image_uri(file.path("figures", file)), alt) ``` A **pure panel** follows one sample over several waves without rotating it. Nobody enters after the first wave; people only leave. That makes it the right instrument for questions a series of cross-sections cannot answer -- who moved between states, how long spells last, whether the same households are poor two years running -- and it makes its weight a different object from the cross-sectional one, with a different target population, a different denominator and a different failure mode. This vignette builds a longitudinal weight over four waves and uses it for gross flows. The rotating case, where the sample is partly renewed each period and the quantity of interest is usually the net change, is `vignette("rotating-panels")`. ## The sequence, and which verb does each step ECLAC's manual prescribes a sequential procedure for the longitudinal weight (Verma, Betti and Ghellini 2006). Each step has its verb in the recipe, and the whole of it is a cross-sectional weight applied to a different universe. ```{r pl-eclac, echo = FALSE, results = "asis"} cat(wf_fig("pl-eclac.svg", "The ECLAC chapter XVI sequence and the recipe verb for each step")) ``` Step **i.a** is the one that does not appear below. It applies when the longitudinal sample is a panel *selected* among rotation groups, so that a unit enters the panel with probability less than one: `step_panel_overlap(prob = panel_pr(pd))` divides the design weight by that probability. A pure panel has no such selection, every unit is in it by construction, and `panel_pr()` says so rather than inventing a number: it requires a `rotation_group` and there is none here. The rotating case is in `vignette("rotating-panels")`. Three points in that sequence the package **cannot** check for you, and they are the three this vignette keeps coming back to: who belongs to the longitudinal population, which period the control totals describe, and what a cross-sectional estimate from this file does and does not mean. ## The longitudinal file is an intersection ```{r merge} waves <- split(panel_puro, panel_puro$wave) names(waves) <- paste0("T", 1:4) wide <- panel_merge(waves, by = c("household_id", "person_no"), require = "all") c(T1_sample = nrow(waves$T1), linked_all_four = nrow(wide)) ``` `require = "all"` keeps the units observed in **every** wave; `require = "any"` would keep the union. Each repeated variable is suffixed by wave, and a `.wf_in_` indicator records presence. Adding a wave can only remove units, never add them, which is the defining property of the design and worth checking as you go: ```{r shrink} vapply(2:4, function(k) nrow(panel_merge(waves[1:k], by = c("household_id", "person_no"), require = "all")), integer(1)) ``` ## Three ways to be missing, and only one of them is nonresponse The field disposition in these data distinguishes three things that a single "missing" flag would collapse: ```{r disposition} table(wave_4 = wide$disposition_T4) ``` * **`OS` -- out of scope.** The unit left the target population: it died, emigrated, the dwelling was demolished. It is not a nonrespondent; it is no longer a member of the universe. Its weight must be **removed**, not redistributed, or the estimated population grows every wave. * **`UNK` -- unknown eligibility.** Nobody knows whether the unit is still in scope. Treating it as eligible inflates the population; treating it as out of scope deflates it. The standard answer is to split it in the observed proportion of known-eligible units, which is what `step_unknown_eligibility()` does. * **`NR` -- nonresponse.** The unit is still in the universe and did not answer. Its weight **is** redistributed, among the units that remain. The order of the steps follows the same logic: leave the universe first, resolve what is unknown second, and only then adjust for the nonresponse of those who are certainly in. ```{r cascade} resp <- vapply(1:4, function(k) wide[[paste0("disposition_T", k)]] == "R", logical(nrow(wide))) wide$responded_always <- rowSums(resp) == 4 lw <- weighting_spec(wide, base_weights = pw_T1) |> step_drop_ineligible(disposition_T4 == "OS", reason = "left the target population") |> step_unknown_eligibility(disposition_T4 == "UNK", by = "region_T1") |> step_attrition(respondent = responded_always, method = "propensity", formula = ~ age_T1 + sex_T1 + region_T1) |> prep() lw ``` Two things to read there. The **sum of the weights** barely moves at the attrition step (293,772 to 293,769, one part in a hundred thousand) while the active sample falls from 1,980 to 1,363: that is what an adjustment by the inverse of an estimated response propensity is supposed to do, and `weighting_alerts()` says so out loud if the total moves by more than 5%. The **effective sample size** is the price: 1,820 to 1,252. The reading that matters more than either number is whether the **composition** of the panel comes back. Attrition is a problem when it is selective, so compare the weighted composition of the adjusted panel against the first wave it is supposed to represent: ```{r composition} grp <- function(a) cut(a, c(-Inf, 29, 49, Inf), labels = c("18-29", "30-49", "50+")) t1 <- waves$T1 sh <- function(w, g) round(100 * tapply(w, g, sum) / sum(w), 1) act <- lw$final_weight > 0 rbind(T1_target = sh(t1$pw, grp(t1$age)), unadjusted = sh(wide$pw_T1[wide$responded_always], grp(wide$age_T1[wide$responded_always])), adjusted = sh(lw$final_weight[act], grp(wide$age_T1[act]))) ``` In this synthetic file the three rows agree to a few tenths of a point, because attrition here is close to uniform: the continuation rate runs between 62% and 65% across the three age groups. That is why the weight total barely moved. On real microdata it is exactly this table that moves, and the gap between the first two rows is the bias the adjustment has to remove; the gap between the first and third is what is left of it. Note also how the respondent condition is defined. "Responded in **all** waves" is the definition of the longitudinal respondent set: someone who answered three of four waves contributes no complete trajectory and is a nonrespondent here, however useful they are for the cross-section. ### Attrition compounds ```{r attrition} data.frame(through = paste0("T", 1:4), complete = vapply(1:4, function(k) sum(rowSums(resp[, 1:k, drop = FALSE]) == k), 0L)) ``` Each wave keeps between 84% and 89% of the previous one, so by the fourth only two of every three initial respondents are left. This is the structural reason a longitudinal weight carries more adjustment than a cross-sectional one, and why the propensity model deserves more care: by the fourth wave it is doing the work of a much larger correction. ## Calibrate to the **first** wave A panel adds no elements over time, so it is representative only of the population of the period in which it was selected. ECLAC's household-survey manual (ch. XVI, sec. B.2.c) is explicit that the auxiliary totals must therefore represent the population of the **first** period of interest. ```{r calibrate} t1 <- waves$T1 sex_tab <- data.frame(sex_T1 = names(tapply(t1$pw, t1$sex, sum)), Freq = as.numeric(tapply(t1$pw, t1$sex, sum))) reg_tab <- data.frame(region_T1 = names(tapply(t1$pw, t1$region, sum)), Freq = as.numeric(tapply(t1$pw, t1$region, sum))) lw <- weighting_spec(wide, base_weights = pw_T1) |> step_drop_ineligible(disposition_T4 == "OS", reason = "left the target population") |> step_unknown_eligibility(disposition_T4 == "UNK", by = "region_T1") |> step_attrition(respondent = responded_always, method = "propensity", formula = ~ age_T1 + sex_T1 + region_T1) |> step_calibrate(method = "raking", totals = list(sex_tab, reg_tab), count = "Freq") |> prep() c(calibrated = sum(lw$final_weight), T1_population = sum(t1$pw)) ``` (In production those totals are demographic projections; here the design-weighted totals of the full first wave stand in for them.) This is the one convention in the whole cascade that **nothing can check for you**. A vector of control totals carries no label saying which period it belongs to, so passing the fourth wave's projections converges just as happily, closes just as cleanly, and produces weights that represent a population the sample never had a chance to cover. The recipe is auditable; the provenance of the totals is yours. ## Gross flows A net change is a difference of aggregates: it cannot tell an immobile population from one where equal numbers enter and leave employment. The longitudinal weight can. ```{r flows} STATES <- c("emp", "unemp", "inact") transition_matrix(lw, from = "lf_status_T1", to = "lf_status_T4", states = STATES, format = "row") ``` `format = "row"` gives the conditional distribution of the destination given the origin -- each row sums to one. `"joint"` gives the joint distribution and `"counts"` the weighted counts. For standard errors, resample the whole recipe and read the transitions off the replicates: ```{r boot} b <- bootstrap_weights(lw, replicates = 100, strata = "stratum_T1", psu = "psu_T1", seed = 1, progress = FALSE) boot_transition(b, "lf_status_T1", "lf_status_T4", states = STATES, format = "row") ``` Every replicate re-runs the eligibility drop, the unknown-eligibility split, the propensity model and the calibration, so the standard errors include the cost of having estimated the adjustments -- not just the sampling of the units. `boot_flows()` reports the same information as population totals, adds the **net** flow `i -> j` minus `j -> i`, and gives the margins: how many started in each state, how many ended in each, how many stayed and how many moved. ```{r grossflows} boot_flows(b, "lf_status_T1", "lf_status_T4", states = STATES) ``` A caveat about the illustration, not the method: the shipped panel datasets are synthetic, and in all of them **inactivity is an absorbing state** -- nobody enters or leaves it. So the matrices above show movement only between employment and unemployment, and the `inact` row and column are degenerate. On real microdata the transitions in and out of inactivity are usually the interesting ones. ## Cross-sectional estimates from a longitudinal file are reference only They will not reproduce the published cross-sectional figures, and they should not. The target population of a four-wave panel combination is the set of units that were in the universe at T1 **and stayed** through T4; the target population of the T4 cross-section is everyone in the universe at T4, including those who entered after T1. Two different populations, two different weights, both correct for their own question. Use the longitudinal weight for flows, transitions and durations, and the cross-sectional weight for levels. `step_cross_sectional()` marks a recipe as the second kind, which is what stops the panel-specific guards from firing on a recipe that is not trying to be longitudinal. ## Where to look next `?panel_merge` for the linkage, `?step_attrition` for the adjustment families, `?step_unknown_eligibility` and `?step_drop_ineligible` for the two things that are not nonresponse, `?transition_matrix` and `?boot_flows` for the flows, and `?step_cross_sectional` for the ECLAC conventions in full. The rotating case, net change and the chained production workflow are in `vignette("rotating-panels")`.