Update to CodSpeed

{
  "name": "codspeed-setup-harness",
  "description": "Set up performance benchmarks and CodSpeed harness for a project. Use this skill whenever the user wants to create benchmarks, add performance tests, set up CodSpeed, configure codspeed.yml, integrate a benchmarking framework (criterion, divan, pytest-benchmark, vitest bench, go test -bench, google benchmark), or when the user says 'add benchmarks', 'set up perf tests', 'create a benchmark', 'benchmark this', or wants to measure performance of their code for the first time. Also trigger when the optimize skill needs benchmarks that don't exist yet.",
  "included_files": [],
  "skill_md_contents": "---\nname: codspeed-setup-harness\ndescription: \"Set up performance benchmarks and CodSpeed harness for a project. Use this skill whenever the user wants to create benchmarks, add performance tests, set up CodSpeed, configure codspeed.yml, integrate a benchmarking framework (criterion, divan, pytest-benchmark, vitest bench, go test -bench, google benchmark), or when the user says 'add benchmarks', 'set up perf tests', 'create a benchmark', 'benchmark this', or wants to measure performance of their code for the first time. Also trigger when the optimize skill needs benchmarks that don't exist yet.\"\n---\n\n# Setup Harness\n\nYou are a performance engineer helping set up benchmarks and CodSpeed integration for a project. Your goal is to create useful, representative benchmarks and wire them up so CodSpeed can measure and track performance.\n\n## Step 1: Analyze the project\n\nBefore writing any benchmark code, understand what you're working with:\n\n1. **Detect the language and build system**: Look at the project structure, package files (`Cargo.toml`, `package.json`, `pyproject.toml`, `go.mod`, `CMakeLists.txt`), and source files.\n\n2. **Identify existing benchmarks**: Check for benchmark files, `codspeed.yml`, CI workflows mentioning CodSpeed or benchmarks.\n\n3. **Identify hot paths**: Look at the codebase to understand what the performance-critical code is. Public API functions, data processing pipelines, I/O-heavy operations, and algorithmic code are good candidates.\n\n4. **Check CodSpeed auth**: Ensure `codspeed auth login` has been run.\n\n## Step 2: Choose the right approach\n\nBased on the language and what the user wants to benchmark, pick the right harness:\n\n### Language-specific harnesses (recommended when available)\n\nThese integrate deeply with CodSpeed and provide per-benchmark flamegraphs, fine-grained comparison, and simulation mode support.\n\n| Language    | Framework                                        | How to set up                                                              |\n| ----------- | ------------------------------------------------ | -------------------------------------------------------------------------- |\n| **Rust**    | divan (recommended), criterion, bencher          | Add `codspeed-<framework>-compat` as dependency using `cargo add --rename` |\n| **Python**  | pytest-benchmark                                 | Install `pytest-codspeed`, use `@pytest.benchmark` or `benchmark` fixture  |\n| **Node.js** | vitest (recommended), tinybench v5, benchmark.js | Install `@codspeed/<framework>-plugin`, configure in vitest/test config    |\n| **Go**      | go test -bench                                   | No packages needed — CodSpeed instruments `go test -bench` directly        |\n| **C/C++**   | Google Benchmark                                 | Build with CMake, CodSpeed instruments via valgrind-codspeed               |\n\n### Exec harness (universal)\n\nFor any language or when you want to benchmark a whole program (not individual functions):\n\n- Use `codspeed exec -m <mode> -- <command>` for one-off benchmarks\n- Or create a `codspeed.yml` with benchmark definitions for repeatable setups\n\nThe exec harness requires no code changes — it instruments the binary externally. This is ideal for:\n\n- Languages without a dedicated CodSpeed integration\n- End-to-end benchmarks (full program execution)\n- Quick setup when you just want to track a command's performance\n\n### Choosing simulation vs walltime mode\n\n- **Simulation** (default for Rust, Python, Node.js, C/C++): Deterministic CPU simulation, <1% variance, automatic flamegraphs. Best for CPU-bound code. Does not measure system calls or I/O.\n- **Walltime** (default for Go): Measures real execution time including I/O, threading, system calls. Best for I/O-heavy or multi-threaded code. Requires consistent hardware (use CodSpeed Macro Runners in CI).\n- **Memory**: Tracks heap allocations. Best for reducing memory usage. Supported for Rust, C/C++ with libc/jemalloc/mimalloc.\n\n## Step 3: Set up the harness\n\n### Rust with divan (recommended)\n\n1. Add the dependency:\n\n```bash\ncargo add divan\ncargo add codspeed-divan-compat --rename divan --dev\n```\n\n2. Create a benchmark file in `benches/`:\n\n```rust\n// benches/my_bench.rs\nuse divan;\n\nfn main() {\n    divan::main();\n}\n\n#[divan::bench]\nfn bench_my_function() {\n    // Call the function you want to benchmark\n    // Use divan::black_box() to prevent compiler optimization\n    divan::black_box(my_crate::my_function());\n}\n```\n\n3. Add to `Cargo.toml`:\n\n```toml\n[[bench]]\nname = \"my_bench\"\nharness = false\n```\n\n4. Build and run:\n\n```bash\ncargo codspeed build -m simulation --bench my_bench\ncodspeed run -m simulation -- cargo codspeed run --bench my_bench\n```\n\n### Rust with criterion\n\n1. Add dependencies:\n\n```bash\ncargo add criterion --dev\ncargo add codspeed-criterion-compat --rename criterion --dev\n```\n\n2. Create benchmark in `benches/`:\n\n```rust\nuse criterion::{criterion_group, criterion_main, Criterion};\n\nfn bench_my_function(c: &mut Criterion) {\n    c.bench_function(\"my_function\", |b| {\n        b.iter(|| my_crate::my_function())\n    });\n}\n\ncriterion_group!(benches, bench_my_function);\ncriterion_main!(benches);\n```\n\n3. Add to `Cargo.toml` and build/run same as divan.\n\n### Python with pytest-codspeed\n\n1. Install:\n\n```bash\npip install pytest-codspeed\n# or\nuv add --dev pytest-codspeed\n```\n\n2. Create benchmark tests:\n\n```python\n# tests/test_benchmarks.py\nimport pytest\n\ndef test_my_function(benchmark):\n    result = benchmark(my_module.my_function, arg1, arg2)\n    # You can still assert on the result\n    assert result is not None\n\n# Or using the pedantic API for setup/teardown:\ndef test_with_setup(benchmark):\n    data = prepare_data()\n    benchmark.pedantic(my_module.process, args=(data,), rounds=100)\n```\n\n3. Run:\n\n```bash\ncodspeed run -m simulation -- pytest --codspeed\n```\n\n### Node.js with vitest (recommended)\n\n1. Install:\n\n```bash\nnpm install -D @codspeed/vitest-plugin\n# or\npnpm add -D @codspeed/vitest-plugin\n```\n\n2. Configure vitest (`vitest.config.ts`):\n\n```typescript\nimport { defineConfig } from \"vitest/config\";\nimport codspeed from \"@codspeed/vitest-plugin\";\n\nexport default defineConfig({\n  plugins: [codspeed()],\n});\n```\n\n3. Create benchmark file:\n\n```typescript\n// bench/my.bench.ts\nimport { bench, describe } from \"vitest\";\n\ndescribe(\"my module\", () => {\n  bench(\"my function\", () => {\n    myFunction();\n  });\n});\n```\n\n4. Run:\n\n```bash\ncodspeed run -m simulation -- npx vitest bench\n```\n\n### Go\n\nNo packages needed — CodSpeed instruments `go test -bench` directly.\n\n1. Create benchmark tests:\n\n```go\n// my_test.go\nfunc BenchmarkMyFunction(b *testing.B) {\n    for i := 0; i < b.N; i++ {\n        MyFunction()\n    }\n}\n```\n\n2. Run (walltime is the default for Go):\n\n```bash\ncodspeed run -m walltime -- go test -bench . ./...\n```\n\n### C/C++ with Google Benchmark\n\n1. Install Google Benchmark (via CMake FetchContent or system package)\n\n2. Create benchmark:\n\n```cpp\n#include <benchmark/benchmark.h>\n\nstatic void BM_MyFunction(benchmark::State& state) {\n    for (auto _ : state) {\n        MyFunction();\n    }\n}\nBENCHMARK(BM_MyFunction);\n\nBENCHMARK_MAIN();\n```\n\n3. Build and run with CodSpeed:\n\n```bash\ncmake -B build && cmake --build build\ncodspeed run -m simulation -- ./build/my_benchmark\n```\n\n### Exec harness (any language)\n\nFor benchmarking whole programs without code changes:\n\n1. Create `codspeed.yml`:\n\n```yaml\n$schema: https://raw.githubusercontent.com/CodSpeedHQ/codspeed/refs/heads/main/schemas/codspeed.schema.json\n\noptions:\n  warmup-time: \"1s\"\n  max-time: 5s\n\nbenchmarks:\n  - name: \"My program - small input\"\n    exec: ./my_binary --input small.txt\n\n  - name: \"My program - large input\"\n    exec: ./my_binary --input large.txt\n    options:\n      max-time: 30s\n```\n\n2. Run:\n\n```bash\ncodspeed run -m walltime\n```\n\nOr for a one-off:\n\n```bash\ncodspeed exec -m walltime -- ./my_binary --input data.txt\n```\n\n## Step 4: Write good benchmarks\n\nGood benchmarks are representative, isolated, and stable. Here are guidelines:\n\n- **Benchmark real workloads**: Use realistic input data and sizes. A sort benchmark on 10 elements tells you nothing about how 10 million elements will perform.\n\n- **Avoid benchmarking setup**: Use the framework's setup/teardown mechanisms to exclude initialization from measurements.\n\n- **Prevent dead code elimination**: Use `black_box()` (Rust), `benchmark::DoNotOptimize` (C++), or `Blackhole.consume` (JMH) so the compiler doesn't optimize away unused results.\n\n- **Cover the critical path**: Benchmark the functions that matter most to your users — the ones called frequently or on the hot path.\n\n- **Test multiple scenarios**: Different input sizes, different data distributions, edge cases. Performance characteristics often change with scale.\n\n- **Keep benchmarks fast**: Individual benchmarks should complete in milliseconds to low seconds. CodSpeed handles warmup and repetition — you provide the single iteration.\n\n## Step 5: Verify and run\n\nAfter setting up:\n\n1. **Run the benchmarks locally** to verify they work:\n\n```bash\n# For language-specific harnesses\ncargo codspeed build -m simulation && codspeed run -m simulation -- cargo codspeed run\n# or\ncodspeed run -m simulation -- pytest --codspeed\n# or\ncodspeed run -m simulation -- npx vitest bench\n# etc.\n\n# For exec harness\ncodspeed run -m walltime\n```\n\n2. **Check the output**: You should see a results table and a link to the CodSpeed report.\n\n3. **Verify flamegraphs**: For simulation mode, check that flamegraphs are generated by visiting the report link or using the `query_flamegraph` MCP tool.\n\n4. **Tell the user** what was set up, show the first results, and suggest next steps (e.g., adding CI integration, running the `optimize` skill).\n"
}