Skip to content

SageMaker Sample Notebooks for LLM

In this section, we provide some sample instruction to use LMI container on SageMaker.

For the list of LMI containers that is on DLC, please click here.

For the list of available BYOC containers, please clck here.

For more information on LMI documentation on SageMaker, click here.

For all the serving.properties options you could set on DJLServing, click here.

Sample notebooks

LMI_Dist Rolling Batch

HF Acc Rolling Batch

DeepSpeed (deprecated)

FasterTransformer (deprecated)

VLLM Rolling Batch

Neuron inference

TensorRT-LLM Rolling Batch

Misc

Amazon SageMaker Examples

Source Repo