[Python] Why does applying a LoRA adapter double VRAM usage when the base model is moved to...

I'm facing a counter-intuitive VRAM issue in PyTorch when applying a LoRA adapter to a nested nn.Module and would appreciate an explanation of the underlying mechanism.

The Context:

I have a base CLIP model (~4GB in VRAM).

I'm creating a wrapper class TTD which only adds a small (~100MB) LoRA adapter to this CLIP model.

Therefore, I expect the final VRAM usage to be approximately 4.1GB (4GB + 100MB).

The Problematic Code:

Python

import torch
import torch.nn as nn
from peft import PeftModel # For demonstration

# Inner Module (Base Model)
class CLIP(nn.Module):
def __init__(self, arch='ViT-H-14-quickgelu', pretrained='dfn5b', device=torch.device('cuda:0')):
super().__init__()

self.model, self.train_transform, self.test_transform = open_clip.create_model_and_transforms(arch, pretrained=pretrained)
self.tokenizer = open_clip.get_tokenizer(arch)

try: self.image_size, _ = self.model.visual.image_size
except TypeError: self.image_size = self.model.visual.image_size

self.device = device

self.model.eval()
self.model.to(self.device) # <--- This line causes the issue (When I delete this line, the problem does not occur)
...


# Outer Wrapper Class
class TTD(nn.Module):
def __init__(self, arch, backbone, lora_path, device=torch.device('cuda:0'), pamr=False):
super().__init__()

self.model = CLIP(arch, backbone, device=device)
lora_utils.apply_lora_v3(self.model.model.visual, 64)
self.model.load_state_dict(torch.load(lora_path))

self.eval()
self.to(device)

self.arch = arch
self.device = device

self.denorm_fn = T.Denormalize()
self.image_transform = T.Compose([T.Normalize(), T.ToTensor()])

def to(self, device):
super().to(device)
self.device = device
self.model = self.model.to(device)
return self
...


#defined TTD
class TTDModel:
def __init__(self) -> None:
self.model = TTD_networks.TTD(
'ViT-H-14-quickgelu',
'dfn2b',
device=torch.device('cuda'),
tag_path=f'{APP_DIR}data/250818_OGQxMYBOX_tag_embedding.json'
)
...


The Phenomenon:

When I run the code with the self.model.to(device) line inside CLIP.__init__, the VRAM usage is not the expected 4.1GB. Instead, it balloons to ~8GB.

It seems as though the 4GB base model is being loaded onto the GPU twice. My guess is that the initial .to(device) call in CLIP places the base model on the GPU, and then the final self.to(device) call in TTD somehow creates a second, separate copy of the PEFT-wrapped model on the GPU.

My Question:

I understand that removing the initial .to(device) call in CLIP fixes the issue. However, I want to understand the underlying reason for this behavior.

Why does PyTorch's memory management cause the VRAM usage to double in this scenario? What is the internal mechanism that prevents the PEFT model from simply reusing the memory of the already-on-GPU base model, and instead allocates a completely new 4GB block?

open_clip_torch==2.32.0
torch==2.3.1
cuda\>=12.1

Continue reading...