Tidyverse Skill

R data manipulation with the core tidyverse packages: dplyr (verb-based data transformation), tidyr (reshaping and tidying), readr (fast delimited file I/O), purrr (functional iteration over lists and vectors), stringr (consistent string operations), forcats (factor manipulation for categorical data), and lubridate (date-time parsing and arithmetic). Also covers arrow for parquet I/O and data.table as a high-performance alternative for large datasets. Use when the execution language is R and the task involves data wrangling, reshaping, joining, string processing, date handling, or file I/O. Python equivalent: the polars skill.

What is the Tidyverse?

The tidyverse is a collection of R packages that share a common design philosophy for data science:

• Verb-based grammar: Operations read as sentences -- filter(), select(), mutate(), summarize() describe what you want to do
• Pipe-friendly: Every function takes a data frame as its first argument and returns a data frame, enabling |> pipe chains
• Tidy data: One observation per row, one variable per column -- functions assume this structure
• Consistent API: Shared conventions across packages (tidyselect helpers, data masking, .data pronoun)
• Readable pipelines: Code reads top-to-bottom like a recipe, making it self-documenting

The tidyverse is not a single package but a curated set of packages that work together. This skill covers the data manipulation subset; for visualization see the ggplot2 skill; for statistical modeling see the r-stats skill.

Version Notes

Versions installed in the DAAF container (R 4.5.3):

Package	Version	Key Notes
dplyr	1.2.1	.by inline grouping, reframe(), pick()
tidyr	1.3.2	pivot_longer()/pivot_wider() with .value sentinel
readr	2.2.0	Second-edition parser, read_csv() returns tibble
purrr	1.2.2	list_c(), list_rbind(), list_cbind()
stringr	1.6.0	Consistent str_* functions wrapping stringi
forcats	1.0.1	fct_na_value_to_level(), fct_cross()
lubridate	1.9.5	ymd() family, interval(), date arithmetic
data.table	1.18.2.1	DT[i, j, by] syntax, fread()/fwrite()
arrow	23.0.1.2	read_parquet()/write_parquet(), Arrow-dplyr integration

dplyr 1.1+ changes to be aware of:

• .by argument in mutate(), summarize(), filter(), slice_*() for inline grouping (no group_by() needed)
• reframe() replaces summarize() when results have multiple rows per group
• pick() replaces across() inside cur_data() contexts
• recode_values() for value-matching (simpler than case_when() for direct mappings; supersedes case_match(), which is soft-deprecated as of dplyr 1.2.0)
• consecutive_id() for run-length grouping
• join_by() for inequality and overlap joins

How to Use This Skill

Reference File Structure

File	Purpose	When to Read
quickstart.md	Core verbs: filter, select, mutate, group_by + summarize, arrange, pipe	Starting an R data pipeline, basic wrangling
reshaping.md	pivot_longer, pivot_wider, separate, unite, nest/unnest	Converting between wide and long formats
joins.md	left_join, inner_join, anti_join, semi_join, join keys, validation	Combining data frames
io.md	Parquet via arrow, CSV via readr, Excel via readxl, remote fetching via httr2/glue, data import patterns	Reading or writing data files, fetching from HTTP APIs
strings-dates.md	stringr ops (str_detect, str_replace, regex), lubridate (ymd, date arithmetic)	String processing or date handling
purrr-functional.md	map/map_dfr/map2, walk, list-columns, nested data patterns	Iterating over lists or nested data
factors.md	forcats: fct_relevel, fct_reorder, fct_lump, factor ordering	Categorical variable manipulation
window-ranking.md	row_number, lag/lead, cumsum, ntile, rolling operations	Window functions and ranking within groups
data-table.md	data.table DT[i, j, by] syntax, fread/fwrite, when to prefer over dplyr	Performance-critical operations on large data
gotchas.md	NSE vs data masking, .data pronoun, across() patterns, group_by footguns	Debugging unexpected behavior

Reading Order

1. New to tidyverse? Start with quickstart.md then io.md
2. Reshaping data? Read reshaping.md
3. Combining datasets? Read joins.md
4. String or date problems? Read strings-dates.md
5. Performance issues with large data? Read data-table.md
6. Something not working? Check gotchas.md first

Related Skills

Skill	Relationship
polars	Python equivalent -- covers the same data manipulation domain for Python pipelines
data-scientist	Methodology routing -- determines which analysis to run; tidyverse implements it
ggplot2	Visualization -- takes tidy data produced by tidyverse and creates plots
r-stats	Statistical modeling -- lm/glm, sandwich robust SEs, diagnostics; tidyverse prepares data for modeling
fixest	Fixed effects regression -- high-dimensional FE, IV, DiD; tidyverse prepares data for fixest
r-python-translation	Cross-language reference -- maps tidyverse to polars for bilingual annotation

Quick Decision Trees

"I need to manipulate data"

Data manipulation task?
├─ Select/drop columns → ./references/quickstart.md (select)
├─ Filter rows → ./references/quickstart.md (filter)
├─ Create/modify columns → ./references/quickstart.md (mutate)
├─ Sort rows → ./references/quickstart.md (arrange)
├─ Aggregate by group → ./references/quickstart.md (group_by + summarize)
├─ Reshape wide to long → ./references/reshaping.md (pivot_longer)
├─ Reshape long to wide → ./references/reshaping.md (pivot_wider)
├─ Join two data frames → ./references/joins.md
├─ Window functions (lag, rank) → ./references/window-ranking.md
└─ Large data performance → ./references/data-table.md

"I need to work with specific types"

Type-specific operation?
├─ String matching/replacement → ./references/strings-dates.md (stringr)
├─ Date parsing/arithmetic → ./references/strings-dates.md (lubridate)
├─ Categorical/factor levels → ./references/factors.md (forcats)
└─ Iterate over lists → ./references/purrr-functional.md (purrr)

"I need to read or write data"

Data I/O?
├─ Read/write parquet (preferred) → ./references/io.md (arrow)
├─ Read/write CSV → ./references/io.md (readr)
├─ Read Excel → ./references/io.md (readxl)
├─ Fetch from HTTP API → ./references/io.md (httr2, glue)
└─ High-speed CSV for large files → ./references/data-table.md (fread)

File-First Execution in Research Workflows

In DAAF research pipelines, R transformations follow the file-first execution protocol -- code is written to .R script files and executed via the run_with_capture.sh wrapper, never run interactively.

The pattern:

1. Write transformation code to scripts/stage{N}_{type}/{step}_{task-name}.R
2. Execute via Bash: bash {BASE_DIR}/scripts/run_with_capture.sh {PROJECT_DIR}/scripts/{script_name}.R
3. run_with_capture.sh detects the .R extension and uses Rscript automatically
4. stdout/stderr are appended to the script file as comments
5. If a script fails, create a versioned copy (_a.R, _b.R, etc.) for fixes

Read agent_reference/SCRIPT_EXECUTION_REFERENCE.md for the complete protocol.

R script structure follows DAAF conventions:

# --- Config ---
library(dplyr)
library(arrow)

PROJECT_DIR <- "/daaf/research/YYYY-MM-DD_Project"

# --- Load ---
# INTENT: Load cleaned school-level data for analysis
df <- read_parquet(file.path(PROJECT_DIR, "data", "schools_clean.parquet"))
cat("Loaded:", nrow(df), "rows,", ncol(df), "columns\n")

# --- Transform ---
# INTENT: Calculate poverty rate by state
# ASSUMES: enrollment > 0 for all rows (validated in cleaning)
result <- df |>
  mutate(poverty_rate = frl_count / enrollment) |>
  group_by(state) |>
  summarize(
    avg_poverty = mean(poverty_rate, na.rm = TRUE),
    n_schools = n(),
    .groups = "drop"
  ) |>
  arrange(desc(avg_poverty))

# --- Validate ---
stopifnot(nrow(result) > 0)
stopifnot(all(result$avg_poverty >= 0 & result$avg_poverty <= 1))
cat("States:", nrow(result), "\n")
cat("Range:", range(result$avg_poverty), "\n")

# --- Save ---
write_parquet(result, file.path(PROJECT_DIR, "data", "state_poverty.parquet"))
cat("Saved: state_poverty.parquet\n")

Quick Reference

Essential Setup

library(dplyr)
library(tidyr)
library(readr)
library(stringr)
library(lubridate)
library(arrow)     # for parquet I/O

Core Operations

Operation	Code	Package
Filter rows	`df	> filter(x > 5)`
Select columns	`df	> select(a, b, c)`
Create column	`df	> mutate(y = x * 2)`
Sort	`df	> arrange(desc(x))`
Group + summarize	`df	> group_by(g)
Rename	`df	> rename(new = old)`
Distinct rows	`df	> distinct(a, b)`
Count	`df	> count(group_col)`
Left join	`df1	> left_join(df2, by = "key")`
Pivot longer	`df	> pivot_longer(cols, names_to, values_to)`
Pivot wider	`df	> pivot_wider(names_from, values_from)`
Read parquet	read_parquet("file.parquet")	arrow
Write parquet	write_parquet(df, "file.parquet")	arrow
Read CSV	read_csv("file.csv")	readr
String detect	str_detect(x, "pattern")	stringr
Parse date	ymd("2024-01-15")	lubridate
Native pipe	`df	> verb1()

The Pipe Operator

R 4.1+ provides the native pipe |> which passes the left-hand side as the first argument to the right-hand side. Use |> (not the magrittr %>%) for all DAAF pipelines:

# Native pipe -- preferred in DAAF
result <- df |>
  filter(year == 2020) |>
  mutate(rate = count / total) |>
  group_by(state) |>
  summarize(avg_rate = mean(rate)) |>
  arrange(desc(avg_rate))

Topic Index

Topic	Reference File
filter, select, mutate	./references/quickstart.md
arrange, group_by, summarize	./references/quickstart.md
Pipe operator	./references/quickstart.md
slice, distinct, count	./references/quickstart.md
across(), pick()	./references/quickstart.md
case_when, if_else	./references/quickstart.md
pivot_longer	./references/reshaping.md
pivot_wider	./references/reshaping.md
separate, unite	./references/reshaping.md
nest, unnest	./references/reshaping.md
complete, fill	./references/reshaping.md
left_join, inner_join	./references/joins.md
anti_join, semi_join	./references/joins.md
Join keys, by argument	./references/joins.md
Join validation	./references/joins.md
read_parquet, write_parquet	./references/io.md
read_csv, write_csv	./references/io.md
Arrow integration	./references/io.md
httr2, glue remote fetch	./references/io.md
str_detect, str_replace	./references/strings-dates.md
str_extract, regex	./references/strings-dates.md
ymd, date arithmetic	./references/strings-dates.md
Date components	./references/strings-dates.md
map, map_dfr, walk	./references/purrr-functional.md
List-columns	./references/purrr-functional.md
fct_relevel, fct_reorder	./references/factors.md
fct_lump, factor ordering	./references/factors.md
row_number, lag, lead	./references/window-ranking.md
cumsum, ntile	./references/window-ranking.md
Rolling operations	./references/window-ranking.md
DT[i, j, by] syntax	./references/data-table.md
fread, fwrite	./references/data-table.md
NSE and data masking	./references/gotchas.md
.data pronoun	./references/gotchas.md
Common pitfalls	./references/gotchas.md

Citation

When this library collection is used as a primary analytical tool, include in the report's Software & Tools references:

Wickham, H. et al. (2019). Welcome to the Tidyverse. Journal of Open Source Software, 4(43), 1686. https://doi.org/10.21105/joss.01686

Cite when: Tidyverse packages are the core data processing engine for the analysis. Do not cite when: Only used for trivial file I/O in a script primarily using another tool.