目录

LoRA Single Device Finetuning

This recipe supports finetuning on next-token prediction tasks using parameter efficient fine-tuning techniques (PEFT) such as Low Rank Adaptation (LoRA) and Quantized Low Rank Adaptation (QLoRA). These techniques significantly reduce memory consumption during training whilst still maintaining competitive performance.

We provide configs which you can get up and running quickly. Here is an example with llama 3.1 8B:

Note

You may need to be granted access to the Llama model you’re interested in. See here for details on accessing gated repositories.

# download the model
tune download meta-llama/Meta-Llama-3.1-8B-Instruct \
--output-dir /tmp/Meta-Llama-3.1-8B-Instruct \
--ignore-patterns "original/consolidated.00.pth"

# run the recipe
tune run lora_finetune_single_device \
--config llama3_1/8B_lora_single_device

You can customize this recipe through the torchtune CLI. For example, when fine-tuning with LoRA, you can adjust the layers which LoRA are applied to:

tune run lora_finetune_single_device \
--config llama3_1/8B_lora_single_device \
model.lora_attn_modules=[q_proj,k_proj,v_proj] \
model.apply_lora_to_mlp=True \
model.lora_rank=64 \
model.lora_alpha=128

For a deeper understanding of the different levers you can pull when using this recipe, see our documentation for the different PEFT training paradigms we support:

Many of our other memory optimization features can be used in this recipe. You can learn more about all of our memory optimization features in our memory optimization overview.

Interested in seeing this recipe in action? Check out some of our tutorials to show off how it can be used:

文档

访问 PyTorch 的全面开发人员文档

查看文档

教程

获取面向初学者和高级开发人员的深入教程

查看教程

资源

查找开发资源并解答您的问题

查看资源