ROADMAP.md

Roadmap

Functionality

• Explore tree sparsity
• Fine-tune Medusa heads together with LM head from scratch
• Distill from any model without access to the original training data
• Batched inference
• Fine-grained KV cache management

Integration

Local Deployment

• mlc-llm
• exllama
• llama.cpp

Serving

• vllm
• TGI
• lightllm

Research

• Optimize the tree-based attention to reduce additional computation
• Improve the acceptance scheme to generate more diverse sequences