High-Performance Open-Source Archive
list_statcan_pumf_catalogue() crawls the
live Statistics Canada “Public use microdata” listing and returns one
row per discovered survey edition (catalogue_id,
Title, edition, format,
url, product_url) — a discovery counterpart to
the curated list_canpumf_collection() that picks up newly
released PUMFs automatically. Editions offered in several formats
collapse to a single preferred row (CSV / flat text first), but
genuinely distinct surveys or file-types that share a reference year are
kept as separate rows (e.g. the GSS cycle and the Giving/Volunteering
survey both released in 2007, or the census
individual/family/household/hierarchical files for one year). Surveys
distributed only by Electronic File Transfer report
url = "(EFT)".list_statcan_pumf_catalogue() caches its result for the
duration of the R session and reuses it on subsequent calls with the
same arguments. Pass refresh = TRUE to re-scrape the live
catalogue and replace the cached result, e.g. to pick up a newly
released survey mid-session.cenNN /
nhsNN filename prefix (the 2011 cycle shipped as the
National Household Survey) and ind / fam /
hous / hier file type into canonical
"YYYY (individuals)"-style strings matching
list_canpumf_collection(). This is forward-compatible: the
2026 census PUMF will resolve automatically once released.list_statcan_pumf_catalogue() now returns a
SeriesTitle column (the plain-language series name matching
the acronym) alongside an edition-specific Title. For
umbrella products whose catalogue title is only the series name
(e.g. the consolidated General Social Survey, or a census year’s
individuals/hierarchical pair) the Title is synthesised as
"<series> — <edition>", where the structural
edition descriptor disambiguates colliding reference years
("General Social Survey — Cycle 16 (2002)",
"Census of Population — 2021 (individuals)"); per-edition
products keep Statistics Canada’s own title.get_pumf() now resolves download URLs from the scraped
catalogue first for the series the crawler covers (GSS, SHS, SFS, CPSS,
CIS, CHS, ITS, CCAHS), so a newly released edition is downloadable
without a package update. Series the crawler deliberately does not cover
— LFS and Census (which keep their dedicated paths) and the
Giving/Volunteering surveys (SGVP, which Statistics Canada
ships under reused zip names the umbrella crawl cannot disambiguate) —
continue to resolve through the curated
list_canpumf_collection(). URL resolution never triggers a
live crawl; it reads the cached catalogue.inst/extdata/pumf_catalogue.rds). It is the terminal,
always-available fallback for both URL resolution and
list_statcan_pumf_catalogue(): a freshly installed package
with no user cache and no network still resolves every supported
survey’s download URL, so a change to the Statistics Canada website
cannot silently break get_pumf() between releases. The
shipped snapshot is regenerated at each release.get_pumf(), get_pumf_connection() and
pumf_metadata() now fail gracefully when Statistics Canada
is unreachable: a download failure no longer raises an error but instead
emits an informative message and returns NULL.
list_available_lfs_pumf_versions() likewise returns an
empty result with a warning rather than erroring, matching the existing
behaviour of list_canpumf_collection() and
list_statcan_pumf_catalogue().close_pumf(NULL) is now a no-op, so it can be called
unconditionally on a get_pumf() result that may be
NULL.options(canpumf.cache_path = ) is not set, the
package now notes this once when attached and again on the first
download, explaining that data is written to a temporary directory (and
discarded at the end of the session) and how to configure a persistent
cache. The underlying behaviour is unchanged — without a cache path,
data is stored in tempdir() for the session.Data - Données/
folder) now extract correctly on Linux and Windows. Previously
utils::unzip() either errored with “invalid multibyte
string” (Windows) or silently dropped the affected files under a
non-UTF-8 locale (Linux), so the survey failed to import with “No
parseable metadata files found”. Extraction now uses
zip::unzip() as the primary, locale-agnostic extractor on
every platform (with the macOS
ditto/system-unzip chain retained as a
fallback for newer ZIP compression variants), giving uniform
cross-platform behaviour. zip is a new dependency.get_pumf() returns the survey’s primary
(respondent-level) module and emits a one-time message listing the
available sibling modules;
pumf_module(tbl, "<module>") opens a sibling on the
same connection so the two are joinable, and announces
the shared join key. Each module’s join key is recorded in the registry
(module_key) so it never has to be guessed (it varies:
RECID, PUMFID, MICRO_ID,
CASEID, IDNUM). Converted surveys include GSS
cycle 16 / “Aging and Social Support” 2002 (MAIN + CG4 + CG6 + CR), GSS
Time Use 1998/2010/2015/2022 (Main + Episode), the Survey of Household
Spending 2017 (Interview + Diary, each with its own bootstrap weights),
and the Giving/Volunteering/Participating cycles 1997–2010 (MAIN +
GS/VD/GIVE/VOLNTR).close_pumf() now also accepts a DuckDB connection
returned by get_pumf_connection(), closing it directly, in
addition to a lazy dplyr::tbl() returned by
get_pumf().parse_pdf_codebook() metadata parser for StatCan
bilingual PDF frequency codebooks. This recovers variable and
value labels for surveys whose only machine-readable companion is the
data file — notably CPSS cycle 1, which (unlike CPSS 2–6) ships no
variables.csv. CPSS 1 now imports with full bilingual
labels (parity with the other cycles) when pdftools is
installed. Like the existing PDF data-dictionary parser, it is a label
fallback that only fires when no command file or codebook CSV is found,
and requires pdftools (Suggests).pumf_module(), join them inside DuckDB, and use
get_pumf_connection() / close_pumf()
directly.get_pumf("LFS") (and other calls) no longer trigger
spurious RStudio “Error in dbSendQuery(…)” Connections-pane popups.
Transient internal DuckDB connections (status checks, write phases, BSW
edits) are no longer registered in the RStudio Connections pane; only
the final connection returned to the user is registered.add_bootstrap_weights() on an in-memory
data.frame/tibble that already has replicate
columns now extends the existing set (generating only the additional
replicates) instead of regenerating a full set and producing duplicate
column names. This matches the DuckDB-backed behaviour.add_bootstrap_weights() now handles rows added to a
survey table that already has bootstrap weights correctly. Previously it
generated replicates for the new rows in isolation (resampling only
among the new rows), which is statistically wrong. It now deletes and
regenerates the affected weights: every row when unstratified, or only
the strata that gained rows when strata_cols are in effect
(complete strata keep their existing weights).R CMD check) list.files()
selected the Main module’s SAS PROC FORMAT, which injected
categorical codes onto continuous clock-time, duration, decimal-hour and
birth-year variables; these are now declared force_numeric
so their values are preserved. In addition,
merge_metadata() no longer warns about label conflicts that
arise solely from lossy supplement parsers (SAS labels, PDF
dictionary/codebook) — authoritative-source conflicts still warn.
Need mirroring services?
Contact our team at info@vpspulse.com.
Mirror powered by VPSpulse
Infrastructure sponsored by VPSPulse & Secure Payments by ArionPay.