Upcoming - in no order

• Benchmarking setup
• Concurrency
• Matmul
  • nsight compute
• nsight sanitizer
• Tensor cores
  • mma
  • wmma
  • wgmma + TMA
• torch.compile
• MoE : FlashDMoE
• Reduction
• Prefix Scan
• Flashattention + FlashMLA
• Model Parallelism or Distributed training/inference
  • FSDP
  • Expert Parallelism
  • Context Parallelism
  • Sequence Parallelism
  • Pipeline Parallelism
  • 4D Parallelism