deepspeed makes a pathway for how training and fine-tuning of large scale language models like gpt and bert happen. it will enable the possibility of training in much more reduced hardware resources without losing performance.

deepspeed addresses each difficulty llm training comes with using optimizations such as zero, mixed-precision training, and gradient accumulation. processes of tuning and deployment get streamlined; therefore, llms are much better approximated and applied to real-world practicalities efficiently.

deepspeed makes large-scale ai models less intimidating and more achievable for developers and researchers alike.

training large-scale language models with deepspeed

large-scale language models, such as gpt and bert, require a tremendous amount of hardware resources, memory, and time to train. deepspeed facilitates the mitigation of these problems through several key features:

1. zero (zero redundancy optimizer)

zero optimization lies at the heart of deepspeed, achieving orders-of-magnitude reductions in memory usage for training huge models. it works by spreading all model states-gradients, optimizer states, and parameters across multiple gpus. this now becomes feasible: training on trillions of parameters-a scale that was infeasible earlier on traditional configurations of hardware.

import deepspeed
import torch
import torch.nn as nn

# define the model
class mymodel(nn.module):
    def __init__(self):
        super(mymodel, self).__init__()
        self.linear = nn.linear(512, 512)

    def forward(self, x):
        return self.linear(x)

# model, optimizer, and deepspeed configuration
model = mymodel()
optimizer = torch.optim.adamw(model.parameters(), lr=3e-4)

# deepspeed configuration dictionary
ds_config = {
    "train_batch_size": 64,
    "optimizer": {
        "type": "adamw",
        "params": {
            "lr": 3e-4
        }
    },
    "zero_optimization": {
        "stage": 1
    },
    "fp16": {
        "enabled": true
    }
}

# initialize the model with deepspeed
model_engine, optimizer, _, _ = deepspeed.initialize(
    model=model,
    optimizer=optimizer,
    config_params=ds_config
)

deepspeed will automatically partition gradients and optimizer states among the available gpus for training large models in memory-friendly ways.

2. mixed precision training

deepspeed supports mixed precision training with fp16. the memory usage does reduce, yet the train speed is accelerated without deteriorating model accuracy. this is a necessary condition for training llms under hardware constraints.

import deepspeed

# fp16 mixed precision training configuration in deepspeed
ds_config = {
    "train_batch_size": 64,
    "fp16": {
        "enabled": true
    }
}

# initialize the model with deepspeed
model_engine, optimizer, _, _ = deepspeed.initialize(
    model=model,
    optimizer=optimizer,
    config_params=ds_config
)

3. gradient accumulation and checkpointing

deepspeed also provides gradient accumulation that enables splitting the backpropagation step over several iterations to handle larger batch sizes. checkpointing of activations decreases the memory by redoing parts of the layers during the backward pass.

python
ds_config = {
    "gradient_accumulation_steps": 4,
"checkpoint_activations": true,
}

the memory will be learned much more effectively by the training. it is achieved with gradient accumulation and checkpointing. the mentioned methods can cope with larger batch sizes on significantly inferior hardware.

case studies: gpt, bert, and beyond

some of the bigger language models are now starting to experience the complete benefits of the optimizations by deepspeed.

1. training gpt-3 using deepspeed scaling

cometh the biggest model ever with 175 billion parameters: gpt-3. normally, training gpt-3 would have taken thousands of gpus and months in time; with deepspeed's zero optimizer, one can simulate thousands of gpus across a couple of hundred gpus.

example: gpt-3 mini training using deepspeed

import deepspeed
import torch
from transformers import gpt2lmheadmodel, gpt2tokenizer

# load tokenizer and model (using gpt-2 as a placeholder for a gpt-3 scaled training example)
tokenizer = gpt2tokenizer.from_pretrained("gpt2")
model = gpt2lmheadmodel.from_pretrained("gpt2")

# deepspeed configuration
ds_config = {
    "train_batch_size": 8,
    "fp16": {
        "enabled": true
    },
    "zero_optimization": {
        "stage": 1
    }
}

# initialize deepspeed
model_engine, optimizer, _, _ = deepspeed.initialize(
    model=model,
    config_params=ds_config,
    model_parameters=model.parameters()
)

# encode input text
input_text = "deepspeed helps train large models"
input_ids = tokenizer.encode(input_text, return_tensors="pt")

# move input to the correct device
input_ids = input_ids.to(model_engine.device)

# forward pass
outputs = model_engine(input_ids=input_ids)
print("model output:", outputs)

2. bert optimization using deepspeed

bert models are massively applied to nlp applications, including question answering and text classification tasks. bert is computationally expensive in the sense that it requires large amounts of datasets and computation resources to train appropriately. deepspeed facilitates faster training with efficient resource utilization of bert, especially in the case of a multi-gpu setup or fine-tuning.

example: fine-tuning bert with deepspeed

import deepspeed
import torch
from transformers import bertforsequenceclassification, berttokenizer

# load model and tokenizer
model = bertforsequenceclassification.from_pretrained('bert-base-uncased', num_labels=2)
tokenizer = berttokenizer.from_pretrained('bert-base-uncased')

# deepspeed configuration for bert fine-tuning
ds_config = {
    "train_batch_size": 32,
    "zero_optimization": {
        "stage": 2
    },
    "fp16": {
        "enabled": true
    }
}

# create an optimizer
optimizer = torch.optim.adamw(model.parameters(), lr=3e-5)

# initialize deepspeed with the optimizer
model_engine, optimizer, _, _ = deepspeed.initialize(
    model=model,
    optimizer=optimizer,  # pass the optimizer here
    config_params=ds_config,
    model_parameters=model.parameters()
)

# continue with encoding inputs and training as before...
input_ids = tokenizer.encode("this is an amazing product!", return_tensors="pt").to(model_engine.device)
labels = torch.tensor([1], dtype=torch.long).unsqueeze(0).to(model_engine.device)

# forward pass
outputs = model_engine(input_ids=input_ids, labels=labels)
loss = outputs.loss

# backward pass and optimization step
model_engine.backward(loss)
model_engine.step()

print("loss:", loss.item())

output

loss: 0.4567  # (this is a hypothetical value; actual output will vary based on input and training)

overcoming the challenges encountered during the training of llm

the training of llms is definitely not a walk in the park. some of the most common issues that crop up during the training of llms are given below:

1. high memory usage

among the techniques developed by deepspeed are zero stage optimizations, mixed precision training, and activation checkpointing-all of which will enable full models on hardware devices and have relatively lower capacities, in terms of memory.

2. overhead of communication in training of multiple gpus

communication overhead is one of the slowdown factors for the training speed of most training approaches that continue to move forward for many gpus. deepspeed has come up with advanced strategies on tensor parallelism and model parallelism with the aim of least possible communication bottlenecks.

3. long training times

deepspeed also employs pipeline parallelism in this way that training gets split across layers, which thus improves the throughput and decreases the time required for training in the general process. the next efficiency improvement technique is gradient accumulation and zero.

# configuration of memory and communication overhead management
ds_config = {
    "train_batch_size": 32,
    "pipeline": {
        "enabled": true
},
"zero_optimization":
"stage": 3
  }, 
  "fp16": {
      "enabled": true
  }

fine-tuning and deploying llms with deepspeed

fine-tuning is also necessary for a pre-trained model to suit the task-specific needs. deepspeed's efficient training strategies make fine-tuning large models on smaller datasets and hardware configurations accessible.

example: fine-tuning gpt-2 with deepspeed

from transformers import gpt2lmheadmodel, gpt2tokenizer
# load pre-trained gpt-2 model
model = gpt2lmheadmodel.from_pretrained("gpt2")
tokenizer = gpt2tokenizer.from_pretrained("gpt2")

# initialize deepspeed
ds_config = {
    "train_batch_size": 16,
    "fp16": {
        "enabled": true
    }
}
model_engine, optimizer, _, _ = deepspeed.initialize(model=model, config_params=ds_config)

# fine-tuning with custom data
input_ids = tokenizer.encode("deepspeed makes training efficient", return_tensors="pt")
loss = model_engine(input_ids=input_ids, labels=input_ids).loss
loss.backward()
model_engine.step()
from transformers import gpt2lmheadmodel, gpt2tokenizer
# load pre-trained gpt-2 model
model = gpt2lmheadmodel.from_pretrained("gpt2")
tokenizer = gpt2tokenizer.from_pretrained("gpt2")

# initialize deepspeed
ds_config = {
    "train_batch_size": 16,
    "fp16": {
        "enabled": true
    }
}
model_engine, optimizer, _, _ = deepspeed.initialize(model=model, config_params=ds_config)

# fine-tuning with custom data
input_ids = tokenizer.encode("deepspeed makes training efficient", return_tensors="pt")
loss = model_engine(input_ids=input_ids, labels=input_ids).loss
loss.backward()
model_engine.step()

deployment

the models can then be deployed after tuning with relative ease in deepspeed. it has good interoperability with the model-serving libraries, hence making it easy to carry out inference with huge models in an efficient manner.

Save or share this article