# Pre-Linkage Data Quality: preflight(), check_medicare(), and flock()

### Preparing your data before murmuration()

#### Dr Nicolas Smoll, SCPHU, Sunshine Coast Hospital and Health Service

#### 2026-07-10

## Why pre-linkage quality matters

The Fellegi-Sunter EM algorithm scores pairs by comparing field values. Poor data quality in any of those fields degrades the score distribution in ways that no threshold choice can fully recover:

- High missingness in a comparison variable reduces its discriminating power, flattening the valley between match and non-match clusters.
- Invalid Medicare numbers that happen to match a wrong AIR record add spurious positive weight to false pairs.
- Name typos are expected (the algorithm handles them via fuzzy string comparison) but systematic truncation or encoding differences are not.
- Duplicate IDs in the primary dataset produce multiple output rows per original record, inflating counts and biasing VE estimates.

`starling` provides three functions specifically for catching these issues before `murmuration()` is called.

## `preflight()` — structured pre-linkage audit

`preflight()` runs a battery of checks across both datasets and returns a structured report. It is the recommended first step before any linkage call.

    library(starling)
    data(cases_notifiable)
    data(vax_air)

    audit <- preflight(
      data1        = cases_notifiable,
      data2        = vax_air,
      linkage_vars = c("lettername1", "lettername2", "dob", "medicare10"),
      id_col1      = "id_var",
      id_col2      = "id_var",
      date_cols    = c("dob", "onset_date"),
      medicare_col = "medicare10",
      verbose      = TRUE
    )
    #>
    #> ================================================================
    #>   starling::preflight() - Pre-Linkage Data Quality Report
    #> ================================================================
    #>   Dataset 1: 300 records x 10 columns
    #>   Dataset 2: 400 records x 15 columns
    #>
    #> == 1. Completeness of linkage variables ==
    #>   Variable             Dataset     Present  Missing  % Missing
    #>   ------------------------------------------------------------
    #>   lettername1          data1           300        0         0%
    #>   lettername2          data1           300        0         0%
    #>   dob                  data1            NA       NA        NA%
    #>   medicare10           data1           300        0         0%
    #>   lettername1          data2           400        0         0%
    #>   lettername2          data2           400        0         0%
    #>   dob                  data2            NA       NA        NA%
    #>   medicare10           data2           400        0         0%
    #>
    #> == 2. Duplicate records (by ID column) ==
    #>   Dataset       Records   Unique IDs   Duplicates    % Duplicate
    #>   ------------------------------------------------------------
    #>   data1             300          300            0             0%
    #>   data2             400          400            0             0%
    #>
    #> == 4. Date plausibility ==
    #>   Column: dob
    #>     data1 - Missing: 0 | Future: 0 | Pre-1900: 0 | Range: 1905-04-01 to 1989-03-10
    #>     data2 - Missing: 0 | Future: 0 | Pre-1900: 0 | Range: 1905-02-09 to 1990-01-01
    #>   Column: onset_date
    #>     data1 - Missing: 0 | Future: 0 | Pre-1900: 0 | Range: 2024-01-05 to 2024-12-30
    #>     data2: column not found
    #>
    #> == 5. Medicare number validation ==
    #>   data1 - Valid: 0 (0%) | Invalid checksum: 0 (0%) | Missing: 300
    #>   data2 - Valid: 0 (0%) | Invalid checksum: 0 (0%) | Missing: 400
    #>
    #> == 6. Name field quality ==
    #>   Column               Dataset     Too short    Numeric Unusual chars
    #>   -----------------------------------------------------------------
    #>   lettername1          data1               0          0            0
    #>   lettername1          data2               0          0            0
    #>   lettername2          data1               0          0            0
    #>   lettername2          data2               0          0            0
    #>
    #> == 7. Categorical variable consistency (shared columns) ==
    #>   'gender': data1 levels = {Female, Male} | data2 levels = {Female, Male}
    #>
    #> == Summary of flags ==
    #>   2 issue(s) flagged:
    #>   [!] [HIGH MISSINGNESS] 'NA' in NA: NA missing
    #>   [!] [HIGH MISSINGNESS] 'NA' in NA: NA missing
    #> ================================================================

The `audit` list contains structured results for programmatic access:

    # Completeness table
    head(audit$completeness)
    #>      variable dataset n_records n_present n_missing pct_missing
    #> 1 lettername1   data1       300       300         0          0%
    #> 2 lettername2   data1       300       300         0          0%
    #> 3         dob   data1       300        NA        NA         NA%
    #> 4  medicare10   data1       300       300         0          0%
    #> 5 lettername1   data2       400       400         0          0%
    #> 6 lettername2   data2       400       400         0          0%

    # Duplicate ID counts
    audit$duplicates
    #>   dataset n_records n_unique_ids n_duplicate_ids pct_duplicate
    #> 1   data1       300          300               0            0%
    #> 2   data2       400          400               0            0%

    # Flags raised (empty = all clear)
    audit$flags
    #> [1] "[HIGH MISSINGNESS] 'NA' in NA: NA missing"
    #> [2] "[HIGH MISSINGNESS] 'NA' in NA: NA missing"

## `check_medicare()` — Modulus 10 checksum validation

Australian Medicare numbers contain a check digit at position 9 calculated from positions 1–8 using the weights 1, 3, 7, 9, 1, 3, 7, 9. `check_medicare()` verifies this and returns a three-state flag:

Value | Meaning
--- | ---
`1L` | Checksum passes — number is internally consistent
`0L` | Checksum fails — at least one digit is wrong
`NA` | Missing, blank, or non-numeric — not verifiable

    cases_checked <- check_medicare(
      cases_notifiable,
      medicare_col = "medicare10",
      output_col   = "medicare_valid",
      verbose      = TRUE
    )
    #>
    #> -- starling::check_medicare() ----------------------------------
    #>   Total records    : 300
    #>   Missing / blank  : 300
    #>   Present (n)      : 0
    #>   Valid checksum   : 0  (N/A%)
    #>   Invalid checksum : 0  (N/A%)
    #>   Output column    : 'medicare_valid'  (1=valid, 0=invalid, NA=missing)
    #> ----------------------------------------------------------------

    # Distribution of flags
    table(cases_checked$medicare_valid, useNA = "always")
    #>
    #> <NA>
    #>  300

### What to do with invalid Medicare numbers

Do not discard the record — it may still link correctly on name and DOB. Instead, replace the invalid number with `NA` before linkage so it does not contribute negatively to the score:

    cases_checked$medicare10 <- ifelse(
      cases_checked$medicare_valid == 1L,
      cases_checked$medicare10,
      NA_character_
    )

## `flock()` — blocking variable construction

`flock()` creates one to three blocking keys from demographic fields. Blocking restricts `murmuration()` to comparing only pairs that share a blocking key, making the search tractable for large datasets without sacrificing recall on the key variables.

    # Single-field block (gender only — broadest, lowest specificity)
    cases_blocked <- flock(
      cases_checked,
      block1_vars    = "gender",         # block1: gender only
      block2_vars    = "gender",         # block2: same here; use composite in production
      block3_vars    = "postcode",       # block3: postcode (finer)
      birth_year_col = "dob"             # derives birth_year column for composite use
    )

    vax_blocked <- flock(
      vax_air,
      block1_vars    = "gender",
      block3_vars    = "postcode",
      birth_year_col = "dob"
    )

    # Inspect blocking key distributions
    table(cases_blocked$block1)
    #>
    #> Female   Male
    #>    141    159
    head(sort(table(cases_blocked$block3), decreasing = TRUE))
    #>
    #> 4562 4555 4552 4556 4557 4560
    #>   30   29   28   28   28   25

### Choosing blocking variables for Australian datasets

Dataset size | Recommended block | Rationale
--- | --- | ---
< 10 000 records | `gender` | Broadest; halves candidate space immediately
10 000–500 000 | `gender` + `birth_year` composite | Balances sensitivity and specificity
> 500 000 | `postcode` + `birth_year` | Fine-grained; requires high postcode completeness

For maximum recall, run `murmuration()` twice — once with a broad block, once with a fine block — and union the results:

    linked_broad <- murmuration(cases_blocked, vax_blocked,
      blocking_var = "block1", ...)
    linked_fine  <- murmuration(cases_blocked, vax_blocked,
      blocking_var = "block3", ...)

    # Union, keeping the highest-scoring link per case
    linked_all <- dplyr::bind_rows(linked_broad, linked_fine) |>
      dplyr::group_by(id_var.x) |>
      dplyr::slice_max(weights, n = 1, with_ties = FALSE) |>
      dplyr::ungroup()

## Putting it together

    library(starling)
    data(cases_notifiable); data(vax_air)

    # 1. Audit
    preflight(cases_notifiable, vax_air,
              linkage_vars = c("lettername1", "lettername2", "dob", "medicare10"),
              medicare_col = "medicare10")

    # 2. Fix Medicare
    cases <- check_medicare(cases_notifiable)
    cases$medicare10 <- ifelse(cases$medicare_valid == 1L,
                                cases$medicare10, NA_character_)

    # 3. Block
    cases <- flock(cases, block1_vars = "gender", birth_year_col = "dob")
    vax   <- flock(vax_air, block1_vars = "gender", birth_year_col = "dob")

    # 4. Link
    linked <- murmuration(cases, vax,
      linkage_type    = "v2c",
      event_date      = "onset_date",
      id_var          = "id_var",
      blocking_var    = "block1",
      compare_vars    = c("lettername1", "lettername2", "dob", "medicare10"),
      threshold_value = 17)

## Session information

    sessionInfo()
    #> R version 4.5.2 (2025-10-31 ucrt)
    #> Platform: x86_64-w64-mingw32/x64
    #> Running under: Windows 11 x64 (build 26100)
    #>
    #> Matrix products: default
    #>   LAPACK version 3.12.1
    #>
    #> locale:
    #> [1] LC_COLLATE=C                       LC_CTYPE=English_Australia.utf8
    #> [3] LC_MONETARY=English_Australia.utf8 LC_NUMERIC=C
    #> [5] LC_TIME=English_Australia.utf8
    #>
    #> time zone: Australia/Brisbane
    #> tzcode source: internal
    #>
    #> attached base packages:
    #> [1] stats     graphics  grDevices utils     datasets  methods   base
    #>
    #> other attached packages:
    #> [1] starling_1.1.0
    #>
    #> loaded via a namespace (and not attached):
    #>  [1] gtable_0.3.6        jsonlite_2.0.0      dplyr_1.2.0
    #>  [4] compiler_4.5.2      Rcpp_1.1.1          tidyselect_1.2.1
    #>  [7] stringr_1.6.0       parallel_4.5.2      jquerylib_0.1.4
    #> [10] scales_1.4.0        yaml_2.3.12         fastmap_1.2.0
    #> [13] ggplot2_4.0.3       R6_2.6.1            generics_0.1.4
    #> [16] knitr_1.51          tibble_3.3.1        lubridate_1.9.5
    #> [19] bslib_0.10.0        pillar_1.11.1       RColorBrewer_1.1-3
    #> [22] rlang_1.2.0         stringi_1.8.7       cachem_1.1.0
    #> [25] reclin2_0.6.0       xfun_0.56           sass_0.4.10
    #> [28] S7_0.2.1            otel_0.2.0          timechange_0.4.0
    #> [31] cli_3.6.5           magrittr_2.0.4      stringdist_0.9.17
    #> [34] digest_0.6.39       grid_4.5.2          rstudioapi_0.18.0
    #> [37] lifecycle_1.0.5     vctrs_0.7.1         lpSolve_5.6.23
    #> [40] data.table_1.18.2.1 evaluate_1.0.5      glue_1.8.0
    #> [43] farver_2.1.2        rmarkdown_2.31      tools_4.5.2
    #> [46] pkgconfig_2.0.3     htmltools_0.5.9