Alpaca
** - Developed by Stanford, Alpaca is an instruction-following model based on LLaMA, fine-tuned for improved performance on specific tasks. Alpaca can be deployed locally, allowing researchers to work with large language models offline for greater privacy and control.
Last verified:
What is Alpaca?
Alpaca 7B is an instruction-following language model fine-tuned from Meta's LLaMA 7B model on 52K instruction-following demonstrations. The model was trained using supervised learning on data generated in the style of self-instruct using OpenAI's text-davinci-003. On preliminary evaluation of single-turn instruction following, Alpaca behaves qualitatively similarly to OpenAI's text-davinci-003 while being surprisingly small and easy/cheap to reproduce at under $600.
Key features include the release of training recipe, 52K demonstrations data, data generation process code, and training code for fine-tuning using the Hugging Face API. The model was fine-tuned in 3 hours on 8 80GB A100s costing less than $100 on most cloud compute providers. The data generation process produced 52K unique instructions and corresponding outputs at less than $500 using the OpenAI API.
Alpaca is intended only for academic research, with commercial use explicitly prohibited. It is designed for the academic community to perform controlled scientific studies on instruction-following language models, enabling reproducible science with standard datasets, models, and code for controlled comparisons and exploring extensions.
The model inherits LLaMA's non-commercial license, and the instruction data is based on text-davinci-003 whose terms prohibit developing competing models. Additionally, adequate safety measures have not been designed, so Alpaca is not ready for general deployment use.
Alpaca pricing
Pricing model: Freemium
Free for academic research only. The model is completely free to use for non-commercial academic purposes. Total reproduction cost is under $600: less than $500 for data generation using OpenAI API and less than $100 for fine-tuning on cloud compute. Commercial use is strictly prohibited due to LLaMA's non-commercial license and OpenAI's terms of use.
Alpaca pros
- Behaves qualitatively similarly to text-davinci-003 on single-turn instructions
- Surprisingly small model size at 7B parameters
- Easy and cheap to reproduce at under $600 total cost
- Training data generation cost less than $500 using OpenAI API
- Fine-tuning takes only 3 hours on 8 80GB A100s
- Fine-tuning costs less than $100 on most cloud providers
- 52K unique instruction-following demonstrations available
- Training recipe and code publicly released on GitHub
- Data generation process code openly available
- Hugging Face training framework support with FSDP and mixed precision
- Enables academic research on instruction-following models
- Facilitates reproducible science with standard datasets
- Blind pairwise comparison won 90 vs 89 against text-davinci-003
- Outputs are generally well-written
- Lightweight model serving as basis to study model deficiencies
Alpaca cons
- Commercial use is explicitly prohibited
- Only intended for academic research purposes
- Exhibits hallucination as a common failure mode
- Hallucination worse than text-davinci-003 in some cases
- Can generate toxic language
- Propagates social stereotypes
- Can spread misinformation with well-written outputs
- Not ready for general deployment due to inadequate safety measures
- Evaluation limited in scale and diversity
- Public demo is now disabled due to hosting costs and filter inadequacies
Frequently asked questions about Alpaca
What is Alpaca?
Alpaca 7B is an instruction-following language model fine-tuned from Meta's LLaMA 7B model on 52K instruction-following demonstrations generated using OpenAI's text-davinci-003 in the style of self-instruct.
How does Alpaca compare to text-davinci-003?
On preliminary evaluation of single-turn instruction following, Alpaca behaves qualitatively similarly to text-davinci-003. In blind pairwise comparison on the self-instruct evaluation set, Alpaca won 90 versus 89 comparisons against text-davinci-003.
What is the cost to reproduce Alpaca?
The total cost to reproduce Alpaca is under $600. Data generation cost less than $500 using the OpenAI API, and fine-tuning the 7B model took 3 hours on 8 80GB A100s costing less than $100 on most cloud compute providers.
Can I use Alpaca commercially?
No, commercial use is strictly prohibited. Alpaca is intended only for academic research. This is because Alpaca is based on LLaMA which has a non-commercial license, the instruction data is based on text-davinci-003 whose terms prohibit developing competing models, and adequate safety measures have not been designed.
What assets are being released?
The released assets include: an interactive demo (now disabled), 52K demonstrations used to fine-tune Alpaca, the code for generating the data, and training code for fine-tuning using the Hugging Face API. Model weights are intended to be released in the near future after obtaining guidance from Meta.
What are Alpaca's known limitations?
Alpaca exhibits common language model deficiencies including hallucination (particularly common, even compared to text-davinci-003), toxicity, and stereotypes. It can generate well-written outputs that spread misinformation. For example, it wrongly stated the capital of Tanzania is Dar es Salaam when it was replaced by Dodoma in 1974.
How was the training data generated?
The data was generated using the self-instruct method starting with 175 human-written instruction-output pairs as seed set. Text-davinci-003 was prompted to generate more instructions using the seed set as in-context examples. The process was simplified compared to self-instruct, resulting in 52K unique instructions and corresponding outputs.
Why was the public demo disabled?
The public demo was disabled because the original goal of disseminating research in an accessible way was mostly achieved, and given the hosting costs and the inadequacies of their content filters, they decided to bring down the demo.
What future research directions does Alpaca enable?
Future directions include: more rigorous evaluation starting with HELM (Holistic Evaluation of Language Models), safety research using automatic red teaming/auditing/adaptive testing, and understanding how capabilities arise from the training recipe including what properties base models need, what happens when scaling up, and alternatives to self-instruct on text-davinci-003.
Are model weights available?
Model weights are not yet released but are intended to be released in the near future. The team has reached out to Meta to obtain guidance on releasing the Alpaca model weights, both for the 7B Alpaca and for fine-tuned versions of larger LLaMA models.