Changelog

v0.2.0 (07/07/2022)

New Features:

  • Support evaluation enabled and set eval_iter

  • Support customized sampler in config.py

  • Support rdma for pipeline-model-parallel

  • Support multi fused kernel

    • fused_scale_mask_softmax_dropout

    • fused_scale_tril_softmax_mask_scale

    • fused_self_attention in branch libai_bench

  • User Experience Optimization

  • Optimization for training throughput, see benchmark for more details

New Supported Models:

  • Support 3D parallel Roberta model

  • Support 2D parallel (data parallel + tensor model parallel) SimCSE model

  • Support Data parallel MAE model

  • Support Data parallel MOCOV3 model

v0.1.0 (22/03/2022)

New Features:

  • Support Data Parallelism

  • Support 1D Tensor Parallelism

  • Support Pipeline Parallelism

  • Unified distributed Layers for both single-GPU and multi-GPU training

  • LazyConfig system for more flexible syntax and no predefined structures

  • Easy-to-use trainer and engine

  • Support both CV and NLP data processing

  • Mixed Precision Training

  • Activation Checkpointing

  • Gradient Accumulation

  • Gradient Clipping

  • Zero Redundancy Optimizer (ZeRO)

Supported Models: