references/structured-data.md

# Structured CLI data, CSV, TSV and encoding

General CSV allows commas, escaped quotes and newlines inside quoted fields. `cut -d,` and `awk -F,` do not reliably parse full CSV. Python `csv`, Miller, csvkit or DuckDB are appropriate when fields can be quoted; choose based on available tools.

Inspect encoding and BOM, delimiter, header, row counts, ragged rows, duplicate rules, nulls, date meaning and identifiers before transformations. Leading-zero identifiers must remain strings. `iconv ... -c` can discard invalid bytes, so report that loss and preserve original data. TSV with guaranteed no quoting may be processed as plain-delimited text if contract is explicit.

Write new output → validate schema, count and duplicates → publish. Provide reconciliation and reject counts where needed. For CSV containing multiline records, line count is not record count.

Primary: Python CSV module https://docs.python.org/3/library/csv.html ; DuckDB CSV docs https://duckdb.org/docs/stable/data/csv/overview.html