Pipelines — Wrangling Data Across Formats

~10 min · interop, pipelines, yq, jq

Level 0Plaintext

0 XP0/64 lessons0/12 achievements

0/100 XP to next level100 XP to go0% complete

Real pipelines glue formats together

The shape of a typical pipeline

Many real workflows look like: fetch JSON → filter → transform → emit YAML / TOML / Markdown. The connective tissue is jq + yq + a templating step. Below: three pipelines you can adapt today.

Pipeline 1 — GitHub PR list to Markdown report

Fetch open PRs as JSON via gh, filter, format as Markdown table.

Pipeline 2 — pyproject.toml dependencies to a requirements list

Parse pyproject.toml, extract [project.dependencies], emit a flat requirements.txt.

Pipeline 3 — Kubernetes ConfigMap from a key-value YAML file

Read a flat YAML of env-var key/values, wrap in K8s ConfigMap shape.

Principle: the unix philosophy applies — small, composable, format-aware tools. cat | yq | jq | python -c '...' > out is a real workflow. Resist the reflex to write a big script when 4 pipe-separated commands do the job in one line. Save it as a shell function once and reuse forever.

Code

Pipeline 1 — PR list → Markdown·bash

# Fetch open PRs as JSON, filter, emit a Markdown table
gh pr list --json number,title,author,createdAt --limit 50 \
  | jq -r '
      ["| # | Title | Author | Created |",
       "|---|---|---|---|"] +
      (. | map("| #\(.number) | \(.title) | @\(.author.login) | \(.createdAt[:10]) |"))
      | join("\n")
    ' > pr-report.md

# Pipe directly to clipboard on macOS
# ... | pbcopy

Pipeline 2 — pyproject deps → requirements.txt·bash

# Extract deps from pyproject.toml, write requirements.txt
yq -p toml '.project.dependencies[]' pyproject.toml > requirements.txt

# Or with the [tool.poetry.dependencies] form (Poetry projects):
yq -p toml -o json '.tool.poetry.dependencies' pyproject.toml \
  | jq -r 'to_entries | map("\(.key)==\(.value)") | .[]' \
  > requirements.txt

Pipeline 3 — flat YAML → K8s ConfigMap·bash

# env.yaml — flat key/value list
cat env.yaml
# DATABASE_URL: postgresql://localhost/pippa
# LOG_LEVEL: info
# PORT: "8000"

# Emit a ConfigMap (jq-style yq filter)
yq '
  {"apiVersion": "v1",
   "kind": "ConfigMap",
   "metadata": {"name": "pippa-env"},
   "data": .}
' env.yaml > configmap.yaml

Reusable shell functions·bash

# Add to your dotfiles
yaml2json() { yq -o json "$@"; }
json2yaml() { yq -p json -o yaml "$@"; }
toml2yaml() { yq -p toml -o yaml "$@"; }
yaml2toml() { yq -o toml "$@"; }

# Usage
yaml2json config.yaml > config.json
toml2yaml pyproject.toml | less

External links

Exercise

Pick one routine you currently do by hand: a dependency audit, a status report, a config conversion. Write a 1-liner pipeline of gh + jq + yq (or similar) that produces it. Save as a shell function or Makefile target. The next time you need that report, it's ready in one command.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousRound-Trip Pitfalls — Comments, Order, Coercion Next →Picking the Right Format — A Decision Matrix

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.