Update to RAG & GenAI Copilot

{
  "description": "Design and troubleshoot RAG source ingestion, parsing, chunking, embeddings, metadata, freshness, and index updates.",
  "included_files": [],
  "name": "design-ingestion-indexing",
  "skill_md_contents": "---\nname: design-ingestion-indexing\ndescription: Design and troubleshoot RAG source ingestion, parsing, chunking, embeddings, metadata, freshness, and index updates.\n---\n\n\n# Design ingestion, parsing and indexing\n**Trigger:** Missing passages, bad parsing, duplicate chunks, source updates, metadata failure, stale index or ingestion performance.\n\n1. Establish source truth and data rights; identify source connectors, file formats, OCR/parser accuracy, document IDs, source version and deletion behavior.\n2. Profile coverage: eligible source count vs parsed documents vs indexed chunks, empty/failed documents, deduplication and unmatched version IDs.\n3. Test chunk strategies on actual representative documents (tables, headers, code, PDFs, multilingual). Preserve provenance fields: document ID, source URL, section/span, version and auth tags.\n4. Make reingestion idempotent with stable IDs and versioned updates; prevent stale duplicates and orphaned chunks. Track failures and dead letters.\n5. Verify embedding/model/index compatibility, dimensionality, normalization and metadata filter behavior; compare counts and sample retrieval before and after.\n6. Produce coverage matrix, sample failures, proposed scoped changes and reindex/rollback plan. Never recommend broad production purge as first action.\n\nUse `../../references/ingestion-indexing.md` and `../../references/debugging-boundaries.md`.\n"
}