PyTorch: ROCm Attention Push and Low-Precision Fixes
AMD accelerator work dominates with new FlexAttention kernels and faster constant handling, while 8-bit float conversions and large-scale sampling get correctness fixes. Distributed training also sees targeted performance and reliability gains.
Duration: PT2M38S
Episode overview
This episode is a short developer briefing from PyTorch.
It explains recent repository work in plain language.
- Show: PyTorch
- Published: 2026-11-10T14:00:38Z
- Audio duration: PT2M38S
Transcript excerpt
This excerpt keeps the crawler page concise. Listen to the episode or use the RSS feed for the full update.
Hello, it's Tuesday, November 10th, 2026, and this is your PyTorch developer briefing.
The lead signal today is a coordinated push for AMD performance paired with tighter low-precision numerics, both aimed at making large-model training more predictable.
First, ROCm attention is leveling up. A three-part stack, represented by PRs 204202 and 204203, extends the general Flex Attention kernels for the 950 architecture with half-precision, larger head dimensions, grouped query and cross-attention support, then adds a software-pipelined dual-wave forward and a split…
Second, small-number math is getting more correct. PR 204210 makes conversion to the 8-bit E-5-M-2 format saturate instead of producing infinity, while PR 204208 fixes rounding of tiny subnormals to the E-8-M-0 format. Paired with commits f-b-6-d-7-0-c and 5-0-0-c-6-6-0, which lift the 16-million category limit for…
Third, scale and efficiency. PR 204194 adds configurable buffer alignment for fully sharded training after finding misaligned shards cut copy bandwidth nearly in half on large jobs, and PR 204190 promotes the hybrid-sharded reduce stream to high priority. Target determination also gets smarter, with PRs 204176…
What…
Nearby episodes from PyTorch
- C++ FakeTensor Parity Push
- Stricter Export Rules and Edge-Case Hardening
- Compiler Correctness Fixes and Accelerator Generalization
- Rust Compiler Prototype and Private Device Runtime
- Inductor Backends and Memory Fixes
- Weekly Recap - Deterministic Checkpointing, Helion Backend and Accelerator Fixes
- Fake Tensor Parity and Backend Correctness
- Size-One Stride Fixes and Distributed Reliability