LiBai Model Zoo

To date, LiBai has implemented the following models:

• Vision Transformer

• Swin Transformer

• ResMLP

• BERT

• T5

• GPT-2

Parallelism Mode in LiBai

A collection of parallel training strategies is supported in LiBai:

• Data Parallel Training

• Tensor Parallel Training

• Pipeline Parallel Training

You can refer to OneFlow official tutorial to better understand the basic conception of parallelization techniques.

Supported Models in LiBai

For more details about the supported parallelism training on different models, please refer to the following table:

Model	Data Parallel	Tensor Parallel	Pipeline Parallel
Vision Transformer	✔	✔	✔
Swin Transformer	✔	-	-
ResMLP	✔	✔	✔
BERT	✔	✔	✔
T5	✔	✔	✔
GPT-2	✔	✔	✔

Additions: ✔ means you can train this model under specific parallelism techniques or combine two or three of them with ✔ for 2D or 3D paralleism training.

Baselines

Here is the collection of baselines trained with LiBai. Due to our resource constraints, we will gradually release the training results in the future.

Main Results on ImageNet with Pretrained Models

ImageNet-1K Pretrained Models

Model	Pretrain	Resolution	Acc@1	Acc@5	Download
ViT-Tiny w/o EMA	ImageNet-1K	224x224	72.7	91.0	Config | Checkpoint
ViT-Small w/o EMA	ImageNet-1K	224x224	79.3	94.5	Config | Checkpoint

Notes: w/o EMA denotes to models pretrained without Exponential Moving Average (EMA).