KTransformers

Installation

Quick Install

pip install ktransformers

From Source

git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
pip install -e .

CUDA Setup

KTransformers requires CUDA for GPU acceleration. Make sure you have:

  1. NVIDIA GPU with compute capability 7.0+
  2. CUDA Toolkit 11.8 or higher
  3. cuDNN 8.6 or higher

Verify CUDA Installation

nvidia-smi
nvcc --version

Docker

docker pull kvcache/ktransformers:latest
docker run --gpus all -it kvcache/ktransformers

Troubleshooting

CUDA not found

Make sure CUDA is in your PATH:

export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

Out of memory

Try reducing batch size or enabling offloading in your config.