Skip to content

LMI Starting Guide

Starting with v0.27.0, the only required configuration for LMI containers is the HF_MODEL_ID environment variable. LMI will apply optimizations and configurations based on the model architecture and available hardware, removing the need manually set them.

Based on the selected container, LMI will automatically:

  • select the best backend based on the model architecture
  • enable continuous batching if supported for the model architecture to increase throughput
  • configure the engine and operation mode
  • maximize hardware use through tensor parallelism
  • calculate maximum possible tokens and size the KV-Cache
  • enable CUDA kernels and optimizations based on the available hardware and drivers

The following code example demonstrates this configuration UX using the SageMaker Python SDK.

This example will use the TheBloke/Llama2-7b-fp16 model.

# Assumes SageMaker Python SDK is installed. For example: "pip install sagemaker"
import sagemaker
from sagemaker import image_uris, Model, Predictor
from sagemaker.serializers import JSONSerializer
from sagemaker.deserializers import JSONDeserializer

# Setup role and sagemaker session
iam_role = sagemaker.get_execution_role() 
sagemaker_session = sagemaker.session.Session()
region = sagemaker_session._region_name

# Fetch the uri of the LMI DeepSpeed container that supports vLLM, LMI-Dist, HuggingFace Accelerate, DeepSpeed backends
deepspeed_image_uri = image_uris.retrieve(framework="djl-deepspeed", version="0.27.0", region=region)

# Create the SageMaker Model object. In this example we let LMI configure the deployment settings based on the model architecture  
model = Model(
    "HF_MODEL_ID": "TheBloke/Llama-2-7B-fp16",

# Deploy your model to a SageMaker Endpoint and create a Predictor to make inference requests
endpoint_name = sagemaker.utils.name_from_base("llama-7b-endpoint")
model.deploy(instance_type="ml.g5.2xlarge", initial_instance_count=1, endpoint_name=endpoint_name)
predictor = Predictor(

# Make an inference request against the llama2-7b endpoint
outputs = predictor.predict({
  "inputs": "The diamondback terrapin was the first reptile to be",
  "parameters": {
    "do_sample": True,
    "max_new_tokens": 256,

Supported Model Architectures

The following text-generation models are supported for optimized inference with the simplified configuration UX:

  • Aquila & Aquila2 (BAAI/AquilaChat2-7B, BAAI/AquilaChat2-34B, BAAI/Aquila-7B, BAAI/AquilaChat-7B, etc.)
  • Baichuan & Baichuan2 (baichuan-inc/Baichuan2-13B-Chat, baichuan-inc/Baichuan-7B, etc.)
  • BLOOM (bigscience/bloom, bigscience/bloomz, etc.)
  • ChatGLM (THUDM/chatglm2-6b, THUDM/chatglm3-6b, etc.)
  • DBRX (databricks/dbrx-base, databricks/dbrx-instruct, etc.)
  • DeciLM (Deci/DeciLM-7B, Deci/DeciLM-7B-instruct, etc.)
  • Falcon (tiiuae/falcon-7b, tiiuae/falcon-40b, tiiuae/falcon-rw-7b, etc.)
  • Gemma (google/gemma-2b, google/gemma-7b, etc.)
  • GPT-2 (gpt2, gpt2-xl, etc.)
  • GPT BigCode (bigcode/starcoder, bigcode/gpt_bigcode-santacoder, etc.)
  • GPT-J (EleutherAI/gpt-j-6b, nomic-ai/gpt4all-j, etc.)
  • GPT-NeoX (EleutherAI/gpt-neox-20b, databricks/dolly-v2-12b, stabilityai/stablelm-tuned-alpha-7b, etc.)
  • InternLM (internlm/internlm-7b, internlm/internlm-chat-7b, etc.)
  • LLaMA & LLaMA-2 (meta-llama/Llama-2-70b-hf, lmsys/vicuna-13b-v1.3, young-geng/koala, openlm-research/open_llama_13b, etc.)
  • Mistral (mistralai/Mistral-7B-v0.1, mistralai/Mistral-7B-Instruct-v0.1, etc.)
  • Mixtral (mistralai/Mixtral-8x7B-v0.1, mistralai/Mixtral-8x7B-Instruct-v0.1, etc.)
  • MPT (mosaicml/mpt-7b, mosaicml/mpt-30b, etc.)
  • OPT (facebook/opt-66b, facebook/opt-iml-max-30b, etc.)
  • Phi (microsoft/phi-1_5, microsoft/phi-2, etc.)
  • Qwen (Qwen/Qwen-7B, Qwen/Qwen-7B-Chat, etc.)
  • Qwen2 (Qwen/Qwen2-beta-7B, Qwen/Qwen2-beta-7B-Chat, etc.)
  • Yi (01-ai/Yi-6B, 01-ai/Yi-34B, etc.)
  • T5 (google/flan-t5-xxl, google/flan-t5-base, etc.)

Other text-generation models, and non-text generation models are also supported, but may not be as performant as the model architectures listed above.

Available Environment Variable Configurations

The following environment variables are exposed as part of this simplified UX:


This configuration is used to specify the location of your model artifacts. It can either be a HuggingFace Hub model-id (e.g. TheBloke/Llama-2-7B-fp16), a S3 uri (e.g. s3://my-bucket/my-model/), or a local path. If you are using SageMaker's capability to specify uncompressed model artifacts, you should set this value to /opt/ml/model. /opt/ml/model is the path in the container where model artifacts are mounted if using this mechanism.


If you are using a model from the HuggingFace Hub, this specifies the commit or branch to use when downloading the model.

This is an optional config, and does not have a default value.


If the model artifacts contain custom modeling code, you should set this to true after validating the custom code is not malicious. If you are using a HuggingFace Hub model id, you should also specify HF_REVISION to ensure you are using artifacts and code that you have validated.

This is an optional config, and defaults to False.


This value is used to specify the number of GPUs to partition the model across using tensor parallelism. The value should be less than or equal to the number of available GPUs on the instance type you are deploying with. We recommend setting this to max, which will shard the model across all available GPUs.

This is an optional config, and defaults to max.


This value represents the maximum number of requests the model will handle in a batch.

This is an optional config, and defaults to 256.


This value represents the number of active requests/client connections the model server can handle. This value should be greater than or equal to MAX_BATCH_SIZE. For requests received when a full batch is being executed, they will be queued until a free slot in the batch becomes available.

This is an optional config, and defaults to 1000.

Additional Configurations

Additional configurations are available to further tune and customize your deployment. These configurations are covered as part of the advanced deployment guide here. In most cases, the simplified configuration set described here is sufficient to achieve high performance. We recommend you first deploy using the configurations described here, and venture into the advanced configurations if you need additional customizations and performance tuning.

The configurations described in that doc follow a different naming convention when using the configuration format. The above environment variables translate to the following configurations:

  • HF_MODEL_ID: option.model_id
  • HF_REVISION: option.revision
  • HF_MODEL_TRUST_REMOTE_CODE: option.trust_remote_code
  • TENSOR_PARALLEL_DEGREE: option.tensor_parallel_degree
  • MAX_BATCH_SIZE: option.max_rolling_batch_size
  • MAX_CONCURRENT_REQUESTS: job_queue_size

API Schema

The request and response schema for interacting with the model endpoint is available here.