deepspeed is a dense framework that optimizes inference with large-scale deep learning models, supporting techniques such as quantization, kernel fusion, pipeline parallelism, and more. you can achieve faster model performance, lower latency, and scalable deployments whether on cloud servers, an edge device, or a serverless platform—with no disadvantage in intrinsic accuracy.

what is inference optimization?

inference on models often proves to be the bottleneck in many applications, especially those using the large-scale deep learning model. the latency, hardware consumption, and deployment of ever more complex systems all surge with complexity- the higher the model complexity, the more latency and hardware consumed in deploying increasingly complex systems is higher than that. deepspeed solves such problems by offering advanced inference optimization features with promising faster model inferences, lower latency, and a greater throughput while maintaining accuracy on the model.

this chapter demonstrates how deepspeed optimizes model inference, the techniques it applies for latency reduction, and exactly how you can deploy models with optimized inference into real-world applications.

deepspeed and efficient inference of models

deepspeed is a tool targeted explicitly at models with billions of parameters so that inference runs efficiently using merely a modest amount of hardware. however, it comes pre-optimized out-of-the-box for the speed-up of inference, including quantization and kernel fusion.

example: speeding up inference

let's take an extremely simple example of how we might leverage deepspeed on hugging face model inference optimization.

import torch
from transformers import automodelforsequenceclassification, autotokenizer
import deepspeed

# load a pre-trained model and tokenizer from hugging face
model_name = "bert-base-uncased"
tokenizer = autotokenizer.from_pretrained(model_name)
model = automodelforsequenceclassification.from_pretrained(model_name)

# example input text
inputs = tokenizer("deepspeed makes model inference faster!", return_tensors="pt")

# enable deepspeed inference optimization
model = deepspeed.init_inference(
    model,
    mp_size=1,
    dtype=torch.float16,
    replace_method="auto",
)

# perform inference
with torch.no_grad():
    outputs = model(**inputs)

# output prediction
print(outputs.logits)

output

tensor([[ 1.0459, -1.0142]], device='cuda:0', dtype=torch.float16)

deepspeed optimizes memory usage and inference performance by 1.5x to 2x based on the hardware configuration.

explanation

super speed. init_inference − this initializes the model for inference with some optimizations. mp_size indicates how many gpus to use and dtype=torch.float16, it enables half-precision for faster computation.
replace_method − this is flagged to "auto" so that deepspeed will automatically apply other optimizations as well, like kernel fusion.

inference latency reduction techniques

the most critical concern for real-time applications is inference latency. the most important techniques provided by deepspeed are as follows:

1. quantization

quantization trains model weights at a lower precision than 32-bit floating point (fp32)—for instance, 16-bit floating point (fp16), or even 8-bit integers. this yields enormous amounts of savings both in computing and memory footprint with no loss of accuracy.

#quantization in deepspeed
model = deepspeed.init_inference(model, mp_size=1, dtype=torch.int8, replace_method="auto")

here, we would have used dtype=torch.int8 for 8-bit quantization, which thus saves a huge amount of model size and also time taken for the inference process.

2. kernel fusion

kernel fusion is one other technique through which more than one operation is fused in a single kernel that minimizes the number of memory accesses. this optimization decreases the overhead resulting from kernel launches and memory bandwidth usage.

3. pipeline parallelism

pipeline parallelism allows you to split huge models across multiple gpus so that data flows through the model in parallel and results are returned quickly during inference. this is helpful when having very large models because the memory of one gpu probably would not be enough.

following is an example of pipeline parallelism with deepspeed

# model partitioning for pipeline parallelism
model = deepspeed.init_inference(model, mp_size=4, dtype=torch.float16, pipeline_parallel=true)

4. tensor slicing

tensor slicing helps fit the model onto hardware with limited memory by slicing large tensors into chunks. load is distributed across the gpus, contributing to memory consumption reduction and enhancements in inference speeds.

deepspeed inference deployment strategies

the model is optimized for inference and therefore comes with several strategies where it can be efficiently deployed. here are some of the deployment strategies using deepspeed:

1. serverless inference using deepspeed

serverless architectures like aws lambda can be used for deploying inference services in scale. deepspeed can be used to optimize the model to fit in serverless function memory limits and time constraints.

following is an example of deploying deepspeed with fastapi −

from fastapi import fastapi
import torch
from transformers import automodelforsequenceclassification, autotokenizer
import deepspeed

app = fastapi()
model_name = "bert-base-uncased"
tokenizer = autotokenizer.from_pretrained(model_name)
model = automodelforsequenceclassification.from_pretrained(model_name)
model = deepspeed.init_inference(model, mp_size=1, dtype=torch.float16, replace_method="auto")
@app.post("/predict")
def predict(text: str):
    inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
        outputs = model(**inputs)
    return { " logits": outputs.logits.tolist() }

we create a restful api using fastapi. the model makes predictions over a simple endpoint.

deepspeed optimizations maximize inference throughput for super batching and multi-api request handling.

2. batching for high throughput

this can make it better in terms of throughput by ensuring that the system can serve more requests at one time through batching of multiple inputs during inference. deepspeed effectively deals with that as it splits the batches across the gpus, then there is a parallelism to hasten the process.

3. edge deployment with deepspeed

deepspeed's quantization and low-memory methods enable the deployment of large models in edge devices that have limited computational power for very low-latency inference in applications such as mobile devices and iot devices.

examples of optimized inference in real world

following are some examples of optimized inference in real world –

1. microsoft turing-nlg

microsoft has released turing-nlg using deepspeed to infer an optimized version of this largest model. techniques such as model parallelism and quantization have allowed microsoft to reduce the inference latency of this huge model by up to 4x.

2. hugging face models

ironically, most models from hugging face are already hosted in production by deepspeed. to give an example, one could reach a 2x speedup over inference on bert and gpt-2 models by optimizing them with the features of deepspeed such as quantization and kernel fusion.

3. nvidia megatron-lm

thus, the same model-diffused megatron model, megatron-lm, was optimized in both training and inference using deepspeed. it consequently leads to faster times at model serving, lower memory overhead, and enables its practicality to be deployed at large scale on cloud infrastructure.

Save or share this article