Self Supervised

PaddleViT DINO self-supervised learning

Use this sub-skill when a task names self_supervised_learning/dino, DINO pretraining, multi-crop augmentation, a DINO teacher/student checkpoint, or porting a DINO ViT backbone into Paddle. It is an operating guide for the PaddleViT checkout, not a promise that every released script is runnable as-is. Keep the source checkout and this skill's evidence boundary explicit.

Applicability and hard boundaries

• The supported end-to-end data path is the ImageNet2012-style path consumed by ImageNet2012Dataset: DATA.DATA_PATH/train_list.txt and DATA.DATA_PATH/val_list.txt, with image paths relative to that directory. Each line is <relative-image-path> <integer-label>; labels are ignored by DINO but are still required by the dataset reader.
• Actual DINO training needs DATA.DATASET=imagenet2012. The CIFAR branches in datasets.py return a single transformed tensor and therefore do not satisfy the multi-crop contract used by main_*_gpu.py.
• One training sample is transformed into two global views at DATA.IMAGE_SIZE and DATA.LOCAL_CROPS_NUMBER local views at DATA.SMALL_CROP_IMAGE_SIZE. Defaults are 2 global + 10 local views, 224/96 pixels, global scale [0.25, 1.0], and local scale [0.05, 0.25].
• The teacher receives only images[:2]; the student receives all crops. Crop tensors must be batched, normalized with the configured ImageNet mean/std, and grouped by spatial size because MultiCropWrapper groups consecutive equal-width crops.
• This skill does not authorize downloading ImageNet, checkpoints, or external repositories, nor starting benchmark-scale or long-running pretraining. Prefer config checks, import checks, synthetic tensors, and a bounded one-step smoke. State the required dataset, GPU count, wall-time, and output directory before proposing a real run.
• torch, timm, and the PyTorch DINO hub model are optional porting dependencies. They are not required for Paddle DINO training and are absent from the inspected environment; do not install or fetch them implicitly.

Evidence and source map

Primary evidence is the pinned checkout's self_supervised_learning/dino/{README.md,config.py,datasets.py,transformer.py,utils.py,main_dino_single_gpu.py,main_dino_multi_gpu.py,run_train_multi.sh} and self_supervised_learning/dino/port_weights/load_pytorch_weights.py. Cross-cutting launch, configuration, AMP, and porting guidance is in docs/paddlevit-{config,multi-gpu,amp,port-weights}.md. The source is an older Paddle implementation (Paddle 2.1-era assumptions); use the bundled scripts for bounded checks and treat source defects below as evidence, not as silent corrections.

Configuration procedure

1. Start from self_supervised_learning/dino/configs/vit_small_patch16_224.yaml or vit_base_patch16_224.yaml; pass it with -cfg from the DINO directory. YAML is merged into the defaults in config.py, then CLI values override selected fields. Use the same effective config for model construction and record it with the run.
2. Check the coupled dimensions before launch: IMAGE_SIZE % PATCH_SIZE == 0, SMALL_CROP_IMAGE_SIZE % PATCH_SIZE == 0, EMBED_DIM % NUM_HEADS == 0, positive OUT_DIM, two ordered crop-scale bounds in (0, 1], and LOCAL_CROPS_NUMBER >= 1. Run scripts/check_dino_config.py --config <yaml>; it is read-only.
3. Preserve the DINO defaults unless a bounded experiment justifies a change: AdamW, cosine LR/weight-decay schedules, teacher momentum warming toward 1, teacher temperature warmup, FREEZE_LAST_LAYER, and OUT_DIM=65536. Note that the README says weight-decay scheduling was not supported in the 2022 release, while the current code computes a schedule; verify the actual checkout before relying on that behavior.
4. Set DATA.BATCH_SIZE per GPU, not global batch size. Scale effective batch size and learning rate deliberately when changing GPU count; the source does not automatically apply the documented linear-LR convention.
5. Treat MODEL.PRETRAINED as a backbone/model-state input only after checking the exact state-dict shape and naming. A DINO run needs student/teacher/head compatibility, not merely a classification checkpoint.

See references/configuration.md for the field contract and scripts/check_dino_config.py for safe validation.

Teacher/student and multi-crop procedure

The intended construction is two ViTs with shared initial state: the student uses configured stochastic depth and the teacher is rebuilt with DROPPATH=0. Each is wrapped by MultiCropWrapper and a DINOHead; teacher parameters have stop_gradient=True. The student is optimized; the teacher is updated after each step by EMA using MOMENTUM_TEACHER's cosine schedule. DINOLoss sharpens and centers the teacher output, compares the two teacher global views against all nonmatching student views, and maintains a center buffer.

For a synthetic check, use two [B,3,32,32] global tensors and two or more [B,3,16,16] local tensors with a tiny ViT config. Do not infer ImageNet accuracy from this check. Run:

python scripts/dino_model_smoke.py --help
python scripts/dino_model_smoke.py --repo-root /path/to/PaddleViT --device cpu
# On a prepared CUDA host, use --device gpu:0 (no dataset or download).

A successful bounded smoke should show model construction, crop shapes, a student/teacher forward, finite DINO logits/loss, and no parameter update on the teacher. If the checkout's wrapper/API fails, retain the diagnostic and do not claim training compatibility; do not hide it by downloading PyTorch.

Launch procedure and backend gates

Single GPU

Use the single-process entrypoint only after config and smoke checks:

CUDA_VISIBLE_DEVICES=0 python main_dino_single_gpu.py \
  -cfg=./configs/vit_small_patch16_224.yaml \
  -dataset=imagenet2012 -batch_size=32 \
  -data_path=/dataset/imagenet -output=./output -amp

-batch_size is per GPU. AMP uses paddle.amp.auto_cast and GradScaler; repository docs limit FP16 AMP claims to NVIDIA Ampere, Volta, and Turing. Require a CUDA Paddle build and a successful CUDA tensor/layer smoke before using -amp. CPU is suitable for import/tiny checks only, not this training claim.

Single-node multi GPU

The source launcher uses paddle.distributed.spawn, initializes one worker per visible GPU, wraps models in paddle.DataParallel, uses DistributedBatchSampler, and all-reduces loss/center statistics. Use an explicit device list and -ngpus matching it; a distributed run requires NCCL and multiple usable CUDA devices:

CUDA_VISIBLE_DEVICES=0,1 python main_dino_multi_gpu.py \
  -ngpus=2 -cfg=./configs/vit_small_patch16_224.yaml \
  -dataset=imagenet2012 -batch_size=16 -data_path=/dataset/imagenet -amp

The repository's run_train_multi.sh is an eight-GPU, long-running ImageNet launcher and is reference-only. Do not execute it during skill use. Validate world size, rank, sampler partitioning, and rendezvous in a short controlled job before any real run. Do not call one-GPU execution a multi-GPU pass.