blackwell

面向大语言模型的高吞吐量与内存优化型推理服务引擎

SGLang是面向大语言模型与视觉语言模型的高速推理框架。

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.

TokenSpeed 是一款光速级的大语言模型推理引擎。