Changelog¶
v0.2.0 (07/07/2022)¶
New Features:
Support evaluation enabled and set
eval_iter
Support customized sampler in
config.py
Support rdma for pipeline-model-parallel
Support multi fused kernel
fused_scale_mask_softmax_dropout
fused_scale_tril_softmax_mask_scale
fused_self_attention in branch
libai_bench
User Experience Optimization
Optimization for training throughput, see benchmark for more details
New Supported Models:
v0.1.0 (22/03/2022)¶
New Features:
Support Data Parallelism
Support 1D Tensor Parallelism
Support Pipeline Parallelism
Unified distributed Layers for both single-GPU and multi-GPU training
LazyConfig
system for more flexible syntax and no predefined structuresEasy-to-use trainer and engine
Support both CV and NLP data processing
Mixed Precision Training
Activation Checkpointing
Gradient Accumulation
Gradient Clipping
Zero Redundancy Optimizer (ZeRO)
Supported Models:
Support 3D parallel BERT model
Support 3D parallel GPT-2 model
Support 3D parallel T5 model
Support 3D parallel Vision Transformer model
Support Data parallel Swin Transformer model
Support finetune task in QQP project
Support text classification task in text classification project