LongLLaMa
LongLLaMA is a large language model that has the capability to handle long contexts. It is a model that has been fine-tuned with the Focuse...
Last verified:
What is LongLLaMa?
LongLLaMA is a large language model capable of handling long contexts of 256k tokens or even more. It is built upon the foundation of OpenLLaMA and fine-tuned using the Focused Transformer (FoT) method, which enables the model to extrapolate far beyond its training context length of 8k tokens. The repository provides a research preview including a 3B base variant licensed under Apache 2.0, inference code supporting longer contexts on Hugging Face, and code for instruction tuning and FoT continued pretraining.
Key features include the ability to handle inputs up to 256k tokens in passkey retrieval tasks, automatic splitting of long inputs into context windows loaded into a memory cache, and compatibility with standard Hugging Face API. The model uses three memory layers for context extension and supports configuration options for memory layers, memory cache data type, and memory attention grouping. LongLLaMA Code is a separate 7B variant built on Code Llama and fine-tuned with FoT for code-related tasks.
LongLLaMA is designed for researchers and developers working on tasks requiring long context processing, such as passkey retrieval, TREC question classification, WebQS question answering, and code generation. It is suitable for those who need to process documents, books, or other long-form text while maintaining performance on short-context tasks. The permissive Apache 2.0 license on the base model makes it commercially usable for research and development purposes.
The model retains performance on tasks that do not require long context while showing improvements on downstream tasks with long inputs. Evaluation results show it achieves 73.3% on TREC and 22.4% on WebQS at 8k context, with passkey retrieval accuracy reaching 94.5% at 100k tokens and 73% at 256k tokens.
LongLLaMa pricing
Pricing model: Free
Free and open source. The source code and base LongLLaMA 3B models checkpoints are licensed under Apache License Version 2.0, making them commercially usable. The instruction/chat tuned models are for research purposes only. LongLLaMA-Code 7B follows the codellama/CodeLlama-7b-hf license. No paid plans or tiers exist as this is a research project released on GitHub and Hugging Face.
LongLLaMa pros
- Handles up to 256k tokens context length or more
- Extrapolates far beyond 8k training context length
- Based on permissive Apache 2.0 license for base model
- Drop-in replacement for LLaMA in Hugging Face implementations
- Compatible with standard Hugging Face API
- Maintains performance on short-context tasks
- Provides instruction-tuned variant (LongLLaMA-Instruct-3Bv1.1)
- Includes code for FoT continued pretraining in JAX
- Includes code for instruction tuning in PyTorch
- LongLLaMA Code 7B improves reasoning and knowledge for code
- Memory cache stores key-value pairs for efficient long context
- Configurable memory layers and attention grouping parameters
- Good passkey retrieval performance at extreme context lengths
- Improvements on TREC question classification task
- Improvements on WebQS question answering task
LongLLaMa cons
- Research preview, not production-ready
- Instruction-tuned models for research purposes only
- No official releases or packages published
- Limited to 3B parameters for base LongLLaMA
- Memory attention adds complexity to implementation
- Requires trust_remote_code=True for loading
- First layers do not have access to memory cache
- LongLLaMA Code uses Code Llama license restrictions
Frequently asked questions about LongLLaMa
What is LongLLaMA?
LongLLaMA is a large language model capable of handling long contexts of 256k tokens or even more. It is built upon the foundation of OpenLLaMA and fine-tuned using the Focused Transformer (FoT) method, which allows it to extrapolate much beyond the 8k context length seen in training.
What is the Focused Transformer (FoT) method?
FoT is a simple method for endowing language models with the ability to handle contexts consisting possibly of millions of tokens while training on significantly shorter input. It permits a subset of attention layers to access a memory cache of key-value pairs to extend context length. The distinctive aspect is its contrastive learning training procedure that exposes memory attention layers to both relevant and irrelevant keys, incentivizing the model to differentiate keys connected with semantically diverse values.
What model variants are available?
The repository provides LongLLaMA-3B (base), LongLLaMA-3Bv1.1 (base with OpenLLaMA-3Bv2 source), LongLLaMA-Instruct-3Bv1.1 (instruction-tuned), and LongLLaMA-Code 7B (built on Code Llama). There is also LongLLaMA-Code 7B Instruct tuned on MathInstruct, OpenOrca and ShareGPT-Processed datasets.
What license is LongLLaMA under?
The source code and base LongLLaMA 3B models checkpoints are licensed under Apache License Version 2.0. The instruction/chat tuned models are for research purposes only. LongLLaMA-Code 7B follows the codellama/CodeLlama-7b-hf license.
How do I install LongLLaMA?
Install requirements with: pip install --upgrade pip, then pip install transformers==4.33.2 sentencepiece accelerate. Load the model using Hugging Face transformers with LlamaTokenizer and AutoModelForCausalLM from_pretrained('syzymon/long_llama_3b_v1_1') with trust_remote_code=True.
How does LongLLaMA handle long inputs?
Inputs over 2048 tokens (4096 for LongLLaMA Code) are automatically split into windows. The first windows contain 2048 tokens each, and the model processes windows one by one extending the memory cache after each. When use_cache=True, LongLLaMA uses two caches: memory cache for specified layers and local generation cache for all layers.
What is the passkey retrieval performance?
LongLLaMA 3B model achieves 94.5% accuracy at 100k tokens and 73% accuracy at 256k tokens in the passkey retrieval task, far exceeding its training context length of 8k tokens.
Can I use LongLLaMA as a drop-in replacement for LLaMA?
Yes, LongLLaMA checkpoints can be used as a drop-in replacement for LLaMA checkpoints in Hugging Face implementation of LLaMA. However, in this case they will be limited to the original context length of 2048 tokens unless using the special inference code supporting longer contexts.
What configuration parameters are available?
Available parameters include: mem_layers (specifies layers endowed with memory), mem_dtype (changes memory cache type), mem_attention_grouping (trades speed for reduced memory usage), and last_context_length (specifies tokens in last context window, default 1024).
How does LongLLaMA perform on standard benchmarks?
LongLLaMA-3B maintains an average score of 0.53 on lm-evaluation-harness, matching OpenLLaMA-3B. LongLLaMA-Instruct-3Bv1.1 achieves 0.55 average. On HumanEval, LongLLaMA-3Bv1.1 achieves 0.12 pass@1 versus 0.09 for OpenLLaMA-3Bv2, showing improvement from code fine-tuning.