Create and Validate Data Source

Purpose

End-to-end workflow for implementing a new Earth2Studio data source wrapper that connects a remote data store (S3, GCS, Azure, HTTP, HuggingFace) to Earth2Studio's async data fetching infrastructure — from analysis through implementation, testing, validation, and PR submission.

Prerequisites

• Earth2Studio dev environment with uv (uv run python must work)
• Git configured with fork (origin) and upstream (upstream) remotes
• Access to the target remote data store (credentials if private)
• Python 3.10+

Workspace

Use the directory containing pyproject.toml. For Harbor evals, write to /workspace/output/ preserving paths. Never read evals/targets/.

Instructions

Python Environment: Always use uv run python or the local .venv. Never use the system Python directly.

Follow every step in order.

[CONFIRM] gates: Only Step 1 (Source Type) and Step 12 (Sanity-Check Plots) require explicit user approval. All other [CONFIRM] markers are advisory — present decisions inline and proceed without blocking.

Deliverables first: Write the source file and test file (Steps 6–7) before extended exploration, documentation, registration, CHANGELOG, or PR work. Skip Steps 8–14 when the user asks for implementation only.

Before you finish: Run verification commands in the repo root so results appear in the session log:

uv run pytest test/data/test_<source>.py -x
make format && make lint

Be concise: Avoid long architecture reports; summarize decisions in a few sentences and move on to file writes.

Hangs or User Feedback If agent becomes stuck or user provides a correction during this skills use, conservatively review relevant part of the skill and improve. Be concise.

One source type per invocation. Invoke again for companion types.

Reference Files

Load these on demand during the relevant steps:

File	Content	Load at
references/implementation-guide.py	Skeleton source with FILL comments	Steps 3–10
references/testing-guide.py	Test skeleton with FILL comments	Step 11
references/validation-guide.md	Plot templates, PR body template, Greptile handling	Steps 12–14 (optional, for templates)

---

Workflow Overview

Step 0: Obtain reference → Step 1: Determine type → Step 2: Dependencies
→ Step 3: Add deps → Step 4: Create lexicon → Step 5: Update vocab/schema
→ Step 6: Create skeleton → Step 7: Implement source → Step 8: Register
→ Step 9: Documentation → Step 10: CHANGELOG → Step 11: Tests
→ Step 12: Validate & plots (user confirms) → Step 13: PR + sanity comment
→ Step 14: Greptile review

---

Step 0 — Obtain Remote Data Store Reference

If $ARGUMENTS is provided, use it (URL → WebFetch; file path → read).

If empty, ask:

Please provide a URL, API documentation link, or description of the remote data store. This will be used to understand storage format, access pattern, variable inventory, temporal/spatial resolution.

---

Step 1 — Determine Source Type

Protocol	Returns	Has lead_time?	Use
DataSource	xr.DataArray	No	Gridded analysis/reanalysis
ForecastSource	xr.DataArray	Yes	Gridded forecast
DataFrameSource	pd.DataFrame	No	Sparse/station obs
ForecastFrameSource	pd.DataFrame	Yes	Sparse forecast obs

Key factors: gridded vs sparse → DataArray vs DataFrame; analysis vs forecast → Source vs ForecastSource.

[CONFIRM — Source Type]

Present recommended type with justification. Ask for confirmation.

---

Step 2 — Examine Remote Store & Propose Dependencies

Analyze: storage backend, file format, authentication, access pattern, temporal/spatial resolution, variable inventory.

Prefer fsspec:

Backend	Preferred	Avoid
AWS S3	s3fs (core dep)	boto3 directly
GCS	gcsfs (core dep)	google-cloud-storage
Azure	adlfs	azure-storage-blob
HTTP	fsspec (core dep)	requests
HuggingFace	huggingface_hub (core dep)	custom scripts

Only fall back to dedicated libraries when fsspec cannot access the store.

Check pyproject.toml — only propose packages not already present. Core deps include: s3fs, gcsfs, fsspec, zarr, netCDF4, h5py, pygrib, huggingface-hub, pandas, pyarrow.

[CONFIRM — Dependencies & Access Pattern]

Present: backend, fsspec filesystem, new packages (with license), auth method.

---

Step 3 — Add Dependencies

Load references/implementation-guide.py from here through Step 10.

If new packages needed:

1. uv add --extra data <package>
2. uv lock
3. Add optional dependency imports using OptionalDependencyFailure pattern

---

Step 4 — Create Lexicon Class

Create earth2studio/lexicon/<source_name>.py with:

• metaclass=LexiconType
• VOCAB: dict[str, str] mapping E2S names → remote keys
• get_item(cls, val) returning tuple[str, Callable]
• Use :: separator for structured keys

Map remote variables against E2STUDIO_VOCAB (282 entries in earth2studio/lexicon/base.py).

[CONFIRM — Lexicon & Variable Mapping]

Present: class name, key format, full mapping table, modifiers, reference URL.

---

Step 5 — Update E2STUDIO_VOCAB / SCHEMA (if needed)

• New vocab: surface = descriptive abbrev; pressure = {name}{level}
• New schema fields: DataFrame sources only, check E2STUDIO_SCHEMA

[CONFIRM — Vocabulary & Schema Updates]

Skip if no updates needed.

---

Step 6 — Create Skeleton Data Source File

Follow canonical method ordering:

1. Class constants
2. SCHEMA
3. __init__
4. _async_init
5. __call__
6. fetch
7. _create_tasks
8. fetch_wrapper
9. fetch_array
10. _validate_time
11. Helpers
12. cache property
13. available classmethod

Use async task dataclass pattern for parallel execution.

[CONFIRM — Skeleton]

Present: class name, file path, skeleton code, task dataclass.

---

Step 7 — Implement the Data Source & Tests

The test file is a co-equal deliverable. Create test/data/test_<filename>.py alongside the source. Add test_<source>_call_mock for async sources.

Sync sources: Use prep_data_inputs/prep_forecast_inputs, direct __call__.

Async sources: See references/implementation-guide.py for required patterns: _sync_async, managed_session, gather_with_concurrency, async_retry, pure async I/O, try/finally cleanup. Constructor params: cache=True, verbose=True, async_timeout=600, async_workers=16, retries=3. DataFrame sources add time_tolerance.

---

Step 8 — Register the Source

• earth2studio/data/__init__.py — alphabetical import
• earth2studio/lexicon/__init__.py — alphabetical import
• Verify pyproject.toml deps

---

Step 9 — Update Documentation

• Add to correct RST file (datasources_analysis.rst / _forecast.rst / _dataframe.rst)
• Class docstring: Parameters, Warning (download size), Note (reference URLs), Badges (last)
• All public methods: NumPy-style docstrings

---

Step 10 — Update CHANGELOG.md

Add entry under the current unreleased version. See references/implementation-guide.py REGISTRATION CHECKLIST for the format.

One line per source. Do NOT add separate lexicon entries.

---

Step 11 — Verify Style & Expand Tests

Run make format && make lint && make license. Load references/testing-guide.py for test skeletons. Required tests: test_<source>_fetch (slow), _cache (slow), _call_mock, _exceptions, _available. Target 90%+ coverage with --slow.

[CONFIRM — Tests]

Present test file, functions, coverage.

---

Step 12 — Validate Variables & Sanity-Check

1. Validate all lexicon vars against real data (run script, do NOT commit)
2. Remove variables with < 10% valid data
3. Create sanity-check plot (gridded or sparse template)
4. Tell user the plot path and ask for visual confirmation

[CONFIRM — Sanity-Check Plots]

User MUST visually inspect plots. Do not proceed without confirmation.

---

Step 13 — Branch, Commit & Open PR

1. Create branch feat/data-source-<name>
2. Commit (do NOT add sanity-check script/images)
3. Push to fork
4. gh pr create --repo NVIDIA/earth2studio
5. Immediately post sanity-check validation as PR comment with:
   • Variable coverage table (name, count, range, unit)
   • Data validation summary (regions, storms/stations, time range)
   • Key findings (physically reasonable values, conversions verified)
   • Full validation script in <details> block
   • Image placeholder: <!-- Drag and drop sanity-check image here -->

[CONFIRM — Ready to Submit]

Verify all steps complete before creating PR.

---

Step 14 — Automated Code Review

1. Poll for Greptile review (5 min timeout)
2. Categorize feedback (bug/style/perf/docs/suggestion/false-positive)
3. Present triage table to user
4. Implement accepted fixes
5. Respond to PR comments
6. Push

[CONFIRM — Review Triage]

User approves which comments to address.

---

Examples

User: Add a data source for the NOAA GFS analysis on S3
Agent: [loads skill, proceeds through Steps 0–14]

---

Limitations

• One source type per invocation
• Requires network access for validation (Step 12)
• SPDX license headers required in all files

---

Reminders

DO: uv run python, loguru.logger, alphabetical order in __init__.py/RST/CHANGELOG, canonical method ordering, async utilities (managed_session, gather_with_concurrency, async_retry), pure async I/O, reference URLs in docstrings, try/finally cleanup.

AVOID: asyncio.to_thread, bare tqdm.gather, xarray for loading, full file downloads.

NEVER: loop.set_default_executor(), commit secrets, commit sanity-check scripts/images.