ETL Pipeline Scaffolder
$2.99OfficialDesign and scaffold idempotent, observable extract-transform-load pipelines with safe retries, backfills, and quality gates; use when building data pipelines.
dataetldata-engineeringpipelinesidempotencyorchestrationobservability· by SkillingMain
What you get
- ✓9-step procedure
- ✓2 ready-to-run code blocks
- ✓8-point quality checklist
- ✓8 pitfalls to avoid
- ✓Installs into 6 tools
- Version
- v1 →
- Last updated
- today
- Length
- 6 min read
- Requires
- Best with a strong model (Claude Sonnet 4)
Works in: Claude Code, Codex, Cline, opencode, OpenClaw, Hermes · Handles multi-file projects
Preview
When to use
Use when designing a new data pipeline, converting one-off scripts or cron jobs into a production pipeline, adding a new source to an existing warehouse, or diagnosing a pipeline that produces duplicates, misses late-arriving data, or cannot be re-run safely. Use it before writing any ingestion code - the design decisions here are expensive to retrofit.
Inputs to gather
- Sources: type (API, database, files, event stream), access method, rate limits, and whether each exposes a reliable primary key and an updated_at-style change column.
- Whether the source records deletes at all - hard deletes are invisible to incremental extraction.
- Volume today and expected in a year
…
🔒 Buy once ($2.99) to unlock the full playbook, download it, and install it in every tool you use.