LMI Backend User Guides

LMI provides backend specific user guides that cover the following topics:

• Model Artifact Structure
• All backends support standard HuggingFace Transformers Pretrained artifacts
• The TensorRT-LLM and Transformer-NeuronX user guides provide information on compiled model artifact structures
• Supported Model Architectures
• Some Model Architectures can only be deployed using specific backends
• Quick Start Configurations
• Starter configurations in both serving.properties and environment variable formats to provide an out-of-the-box solution for that backend
• Quantization Guide
• If a backend supports quantization, we describe the different options and how to enable them
• Advanced Configurations
• Configurations that are only available with this backend

The available backends and their respective user guides are available below:

• DeepSpeed
• LMI-Dist
• vLLM
• TensorRT-LLM
• Transformers-NeuronX
• HuggingFace Accelerate