图例困惑

[off0set]

为啥使用LoRA训练时长比全参数训练还长啊~

[isLinXu]

因为full sft后期训练指标增长很小，为了防止过拟合，进入了早停；
而lora系列能够训练地更久，具体可见
https://wandb.ai/linxu/YOLO-Master-Exp?nw=nwuserlinxu

[Nickernel]

def train_with_lora():
"""Entry point: LoRA fine-tuning (only adapters + detection head are trainable)."""
model = YOLO(MODEL_CFG)
model.load(PRETRAIN)

# LoRA training (one-click activation)
model.train(
    **COMMON_ARGS,
    # lora_r=16,                        # fixed rank; mutually exclusive with auto ratio
    lora_auto_r_ratio=0.05,             # auto-size rank to ~5% of model params
    lora_alpha=32,                      # alpha = 2×r
    lora_dropout=0.1,
    # lora_gradient_checkpointing=True, # OFF: re-runs forward in backward -> ~1.5-2x slower.
    #                                   # Only enable when VRAM is the bottleneck.
)

# Save only LoRA adapters (small file) to a directory
model.save_lora_only("yolo11n_lora_r16")

def train_without_lora():
"""Entry point: standard full fine-tuning (all parameters trainable)."""
model = YOLO(MODEL_CFG)
model.load(PRETRAIN)

# No lora_* arguments -> LoRA stays disabled, the whole network is trained.
model.train(**COMMON_ARGS)

使用如上配置文件，对yolo11n进行训练，lora的训练时间比全量微调多一倍（100epoch，没有早停）
全量微调

lora微调

[isLinXu]

感谢提出的问题，针对每一个epoch的训练时间，目前还在优化加速以及跑批量实验中，可以等后续更稳定版本
YOLO-Master当前的PEFT实现属于实验性功能，尚未实现关键的算子融合（Operator Fusion）。
未融合的LoRA层会导致：
频繁的内存读写：每个LoRA操作都需要从显存读取数据、计算、再写回显存
内存带宽瓶颈：GPU计算单元处于等待状态，利用率下降
中间结果存储开销：需要额外存储Ax等中间张量

[isLinXu]

另外，对于少批量的数据集，建议后续可以修改为使用Few-Shot的LoRA，具体可见：https://github.com/Tencent/YOLO-Master/blob/main/ultralytics/cfg/default.yaml

# Few-Shot LoRA settings -----------------------------------------------------------------------------------------------
lora_few_shot_mode: False # (bool) Enable few-shot LoRA with enhanced regularization for small datasets
lora_few_shot_teacher: # (str, optional) Path to teacher model for knowledge distillation
lora_few_shot_dropconnect: 0.1 # (float) DropConnect rate for few-shot LoRA (better than dropout for limited data)
lora_few_shot_distill_weight: 0.5 # (float) Weight for distillation loss when teacher is provided
lora_few_shot_adaptive_rank: True # (bool) Auto-adjust rank based on data scarcity in few-shot mode
# Few-Shot LoRA: scheduling and advanced regularization
lora_few_shot_dropconnect_schedule: "cosine" # (str) DropConnect schedule: constant/linear/cosine/exponential
lora_few_shot_dropconnect_max: 0.3 # (float) Initial max DropConnect rate (for scheduled modes)
lora_few_shot_dropconnect_min: 0.0 # (float) Final min DropConnect rate (for scheduled modes)
lora_few_shot_gradient_importance_weighted: False # (bool) Use gradient-importance weighted DropConnect
lora_few_shot_hierarchical_distill: False # (bool) Enable multi-layer hierarchical distillation
lora_few_shot_distill_layers: # (list, optional) Layer indices for intermediate distillation e.g. [10, 13, 16]
lora_few_shot_variational_rank: False # (bool) Enable variational rank selection (Gumbel-Softmax)
lora_few_shot_rank_budget: 0.5 # (float) Budget ratio for rank retention in variational mode
lora_few_shot_adaptive_temperature: False # (bool) Enable task-adaptive distillation temperature
lora_few_shot_curriculum_sampling: False # (bool) Enable curriculum learning sampler
# Few-Shot LoRA: distillation scheduling and response-level transfer
lora_few_shot_distill_schedule: "cosine" # (str) Distillation weight schedule: constant/linear/cosine
lora_few_shot_distill_weight_max: 1.0 # (float) Initial max distillation weight
lora_few_shot_distill_weight_min: 0.1 # (float) Final min distillation weight
lora_few_shot_use_ema_teacher: False # (bool) Use EMA teacher for progressive self-distillation
lora_few_shot_ema_decay: 0.999 # (float) EMA teacher decay rate
lora_few_shot_response_distill: False # (bool) Enable detection head response distillation (cls + bbox)
lora_few_shot_response_distill_weight: 0.3 # (float) Weight for response distillation loss
lora_few_shot_layerwise_rank: False # (bool) Enable per-layer adaptive rank based on depth/channels
lora_few_shot_hook_cache: True # (bool) Cache hierarchical distillation hooks across batches