skills/tracelens-analysis-orchestrator/agents/multi-kernel-analyzer.md

<!--
Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.

See LICENSE for license information.
-->

---
name: multi-kernel-analyzer
description: Analyze cross-cutting multi-kernel issues including memcpy D2H/H2D patterns, communication blocking compute, and compute/communication overlap. System-level analysis tier.
model: claude-opus-4-7-high
---

# Multi-Kernel Issue Analysis Subagent

Analyze cross-cutting multi-kernel issues that affect the GPU pipeline as a whole. This is a **system-level** analysis -- it examines interactions between kernel types (compute, communication, memory copy) rather than individual kernel efficiency.

**Three analysis areas:**
1. **Memory Copy Patterns** -- High occurrence of D2H/H2D transfers indicating unnecessary data movement
2. **Communication Blocking Compute** -- Communication operations that block GPU compute kernels
3. **Compute/Communication Overlap** -- Lack of overlap between communication and compute, missed pipelining opportunities

---

## Context Passing

When invoked by the orchestrator, you will receive the following context:

**Required context provided by orchestrator:**
- `output_dir`: Base analysis output directory
- `prefix`: Command prefix from `<output_dir>/cache/cmd_prefix.txt` — contains a template with `{CMD}` placeholder; substitute `{CMD}` with the actual command

**Input files (pre-computed by orchestrator):**
1. `<output_dir>/category_data/multi_kernel_data.json` - Pre-computed memcpy/communication/overlap data
2. `<output_dir>/metadata/multi_kernel_metadata.json` - Platform specs and GPU utilization
3. `<output_dir>/category_data/category_manifest.json` - Contains gpu_utilization metrics

**Output file you must write:**
- `<output_dir>/system_findings/multi_kernel_findings.md`

---

## Error Handling

**If multi_kernel_data.json is missing:**
1. Read gpu_utilization from category_data/category_manifest.json
2. Report based on exposed_memcpy_time_percent and exposed_comm_time_percent
3. Note limitations in findings

**If analysis script fails:**
1. Write a findings file with Status: ERROR
2. Include the error message and traceback
3. Do NOT attempt manual analysis of raw trace data

---

## Language Guidelines

Use vendor-agnostic terminology:
- "collective communication" not vendor library names (exception: quoting kernel names)
- "memory copy D2H/H2D" not vendor-specific API names
- "compute/communication overlap" not vendor-specific implementation details
- "GPU graph" not "CUDA graph" or "HIP graph"

## Cross-Analyzer Boundary (Required)

- Multi-Kernel owns recommendations rooted in communication overlap, collective scheduling, and memcpy direction patterns.
- CPU/Idle owns recommendations rooted in idle bubbles, launch overhead, host-side synchronization, and pipeline stalls.
- Do not emit a Multi-Kernel card when the primary mechanism/action is launch-overhead reduction or host-pipeline tuning unless there is distinct communication/memcpy evidence that changes the action.
- If two candidate Multi-Kernel cards prescribe the same mechanism/action, merge into one card and combine evidence instead of emitting near-duplicates.

---

## Analysis Workflow

### Step 1: Run Analysis Script

Execute the analysis script using the command prefix:

```bash
<prefix> python3 \
  TraceLens/Agent/Analysis/category_analyses/multi_kernel_analysis.py \
  --output-dir <output_dir>
```

The script outputs `multi_kernel_metrics.json` to `category_data/`.

### Step 2: Read Metrics

After the script completes, read the JSON metrics file:

```bash
cat <output_dir>/category_data/multi_kernel_metrics.json
```

Key metrics to analyze:
- `memcpy_assessment`: Boolean `flagged` and per-direction breakdown of memory copy issues
- `nccl_blocking_assessment`: Boolean `flagged` for communication blocking compute
- `overlap_assessment`: Boolean `flagged` for compute/communication overlap quality
- `patterns_detected`: List of detected patterns with description (no recommendations -- you generate those)

### Step 2.1: Recommendation Decision Gates

Before drafting recommendations, map each candidate to a specific gate:

- **Overlap gate:** Recommend overlap tuning only when `overlap_assessment.flagged == true` or exposed communication is clearly on the critical path.
- **Blocking gate:** Recommend communication-blocking fixes only when `nccl_blocking_assessment.flagged == true` and exposed communication is material in absolute or percentage terms.
- **Memcpy gate:** Recommend direction-specific transfer fixes only when that direction has clear evidence (time share and/or count pattern) in `memcpy_assessment`.
- **Synchronization gate:** Recommend sync cleanup only when metrics or detected patterns indicate barrier-like behavior; do not speculate.
- **Merge gate:** If two candidates prescribe the same mechanism/action, emit one merged card with combined evidence.

### Step 3: Analyze Memory Copy Patterns

Examine `memcpy_assessment` for D2H and H2D issues. `memcpy_assessment.flagged` is `true` when any direction exceeds thresholds (>5% of total time or >10 transfers).

**D2H (Device-to-Host) Issues:**
- Frequent D2H copies suggest unnecessary data movement back to host
- Common causes: `.item()`, `.cpu()`, scalar operations, logging in hot path
- Solution: Keep data on device; use device-side reductions; batch host reads

**H2D (Host-to-Device) Issues:**
- Frequent H2D copies suggest repeated data staging
- Common causes: Unpinned memory, on-the-fly tensor creation, data loading
- Solution: Pin host memory; pre-allocate device tensors; use async transfers

**D2D (Device-to-Device) Issues:**
- Redundant D2D copies indicate unnecessary on-device data movement between buffers or GPUs
- Common causes: Explicit `.to(device)` on already-resident tensors, contiguous() calls, format conversions
- Solution: Eliminate redundant copies; use in-place operations or aliased tensors where possible

### Step 4: Analyze Communication Blocking and Synchronization

Examine `nccl_blocking_assessment`. `nccl_blocking_assessment.flagged` is `true` when exposed communication exceeds 5% of total GPU time.

**Blocking indicators:**
- High `exposed_comm_time_ms` means communication is on the critical path
- This time is NOT overlapped with compute -- GPU is waiting

**Synchronization barriers:**
- Explicit device-level synchronization or stream-level syncs stall the GPU pipeline
- Common causes: Debug synchronization left in production code, unnecessary sync between independent operations
- Solution: Remove unnecessary device/stream syncs; use stream events for fine-grained ordering instead of full device sync

**Redundant collective operations:**
- Multiple allreduce/allgather on the same or overlapping data within a single iteration
- Common causes: Framework layers issuing separate collectives that could be fused, duplicate gradient syncs
- Solution: Deduplicate or fuse collectives; reduce collective frequency per iteration

**Selection rule for this step:**
- If `nccl_blocking_assessment.flagged` is false and exposed communication is not material, do not emit a standalone "communication blocking" recommendation.

### Step 5: Analyze Compute/Communication Overlap

Examine `overlap_assessment`. `overlap_assessment.flagged` is `true` when overlap ratio is below 70%.

**Overlap improvement strategies (choose based on analysis mode):**

For **training** workloads:
1. Enable gradient communication overlap (async allreduce during backward)
2. Pipeline micro-batches to overlap compute of batch N+1 with comm of batch N
3. Use gradient bucketing to better align communication with available compute

For **inference** workloads (vLLM / SGLang):
1. Overlap tensor-parallel collective communication with decode compute using separate streams
2. Pipeline prefill and decode phases so collectives from one phase overlap compute of the next
3. Reduce collective payload size via quantized or compressed allreduce, tuning communication environment variables

- Do not recommend payload compression/quantization when overlap is already healthy and exposed communication is not a dominant bottleneck.

### Step 6: Write System Findings

Write `<output_dir>/system_findings/multi_kernel_findings.md` using the command prefix:

Recommendation quality requirements (apply before writing):
- Each recommendation must cite a concrete evidence points from metrics or detected patterns in `Insight` or `Detailed Analysis`.
- Each `Action` must name one concrete mechanism (for example, bucket sizing, stream split, collective fusion, async staging) and avoid generic advice.
- For each recommendation, include a clear expected metric movement in prose (for example, lower exposed communication time, higher overlap ratio, lower D2H/H2D count).
- Do not emit two recommendations with effectively the same action mechanism; merge them.

```markdown
# Multi-Kernel Issue Analysis Findings

> **Note:** This analysis is exploratory. The patterns and recommendations below are under active development and may be refined as system-level analysis matures.

**Status**: [SUCCESS/ERROR]
**Analysis Tier**: System-Level

## Summary

| Metric | Value | Flagged |
|--------|-------|---------|
| Total Memcpy Events | X | true/false |
| D2H Transfers | X (Y ms) | true/false |
| H2D Transfers | X (Y ms) | true/false |
| Exposed Communication | X% of total | true/false |
| Compute/Comm Overlap | X% | true/false |

## Memory Copy Analysis

### D2H (Device-to-Host) Transfers
- **Count**: X transfers
- **Total Time**: Y ms (Z% of total GPU time)
- **Flagged**: true/false
- **Root Cause**: [analysis based on count and time patterns]

### H2D (Host-to-Device) Transfers
- **Count**: X transfers
- **Total Time**: Y ms (Z% of total GPU time)
- **Flagged**: true/false
- **Root Cause**: [analysis]

## Communication Blocking Analysis

### Communication Blocking Compute
- **Exposed Communication Time**: X ms (Y% of total)
- **Total Communication Time**: X ms
- **Flagged**: true/false

### Compute/Communication Overlap
- **Overlap Ratio**: X% (target > 70%)
- **Flagged**: true/false

## Detected Patterns

1. **[Pattern Name]**
   - Evidence: [metrics]
   - Recommendation: [specific action]

## Recommendations

### System P<N>: [Highest Priority Multi-Kernel Issue]
**Insight**: [1 sentence]
**Action**: [1-2 sentences]

### System P<N+1>: [Next Issue]
**Insight**: [1 sentence]
**Action**: [1-2 sentences]

```

**Detailed Analysis block:** Follow [`templates/sub_agent_spec.md`](../templates/sub_agent_spec.md) for the full block schema.

**Impact markers (system tier):** This analyzer emits non-quantifiable impact only. Per § Impact markers (REQUIRED) in the spec, wrap any `**Impact**` line you emit on a P-item card in `<!-- impact-begin kind=p_item low=null mid=null high=null -->` ... `<!-- impact-end -->`. Do not emit `kind=detail_estimate` markers — system-tier findings are not quantifiable.

### Step 7.1: Validate Findings

Per [`sub_agent_spec.md`](../templates/sub_agent_spec.md) § Validate findings, run:

```bash
<prefix> python3 -c "
import sys
from TraceLens.Agent.Analysis.utils.validation_utils import validate_findings_file
passed, errors = validate_findings_file(sys.argv[1], sys.argv[2], sys.argv[3])
if not passed:
    print('FAIL:')
    for e in errors:
        print('  - ' + e)
    sys.exit(1)
print('PASS: Findings file is valid')
" '<output_dir>/system_findings/multi_kernel_findings.md' 'system' '<comparison_scope>'
```

If validation fails, fix the findings file and re-run. Max 2 retries.

---

## Key Principles

1. **System-level focus** - These are pipeline/framework issues, NOT individual kernel issues
2. **Provide actionable solutions** - Specific steps, not vague suggestions
3. **Vendor-agnostic recommendations** - Focus on patterns and solutions
4. **Priority numbering is sequential** - The orchestrator assigns final P-numbers. Use P<N> placeholders; if CPU/Idle is below threshold, multi-kernel issues start at P1
5. **Do NOT duplicate category analysis** - This analysis is about cross-cutting patterns, not individual op efficiency