DuckDB Embedded OLAP & Parquet Lakehouse Playbook

Query remote S3/R2 Parquet datasets, JSON logs, and CSV exports in-memory at gigabytes per second with DuckDB.

---
name: duckdb-parquet-olap-pipelines
description: Query remote S3/R2 Parquet datasets, JSON logs, and CSV exports in-memory at gigabytes per second with DuckDB. Use when working with DuckDB, Parquet, SQL Analytics.
triggers:
  - duckdb parquet olap pipelines
  - duckdb
  - parquet
---

# DuckDB Embedded OLAP & Parquet Lakehouse Playbook

## 1. When to Use This Skill
Load `skill-duckdb-parquet-olap-pipelines` whenever an engineering, architecture, or operations task involves **DuckDB, Parquet, SQL Analytics**. Specifically:
- Query remote S3/R2 Parquet datasets, JSON logs, and CSV exports in-memory at gigabytes per second with DuckDB.
- Designing, reviewing, refactoring, or debugging production configurations without guesswork or deprecated flags.
- Running deterministic pre-flight validation and post-execution verification checks before reporting completion.

## 2. Prerequisites & Required Environment
- **Target Stack:** DuckDB, Parquet, SQL Analytics
- **Permissions Boundary:** Read-only inspection by default; ask for explicit human confirmation before executing destructive mutations or production deployments.
- **Reference Specification:** `https://github.com/duckdb/duckdb`

## 3. Step-by-Step Execution Playbook
1. **Context & Configuration Discovery:**
   Inspect existing project manifests (`package.json`, `tsconfig.json`, `pyproject.toml`, `Cargo.toml`, `wrangler.jsonc`, or schema declarations) before proposing changes. Never overwrite existing bindings or credentials blindly.
2. **Deterministic Implementation:**
   Apply the canonical production patterns for **DuckDB**:
   - Enforce strict schema and type validation at every external boundary.
   - Include explicit error handling, structured telemetry, and idempotency keys on all mutating calls.
   - Keep secrets in environment variables or managed secret stores—never hardcode tokens in source files.
3. **Verification & Regression Gate:**
   Run the verification commands below and inspect both exit codes and output payloads before marking the task complete.

## 4. Reference Commands & CLI Workflow
```bash
duckdb -c 'SELECT version();'
```

## 5. Edge Cases & Anti-Hallucination Guardrails
- **Zero Unverified Claims:** Never claim a build, migration, test run, or API call succeeded without inspecting the command output or HTTP status code.
- **Rollback Readiness:** Always record the previous state or migration version so changes can be reverted cleanly if downstream health checks fail.

Synthesized Engineering Blueprint & Setup Guide