Working on fine-tuning phi-3.5-mini, and when trying to run the trainer.train() i am getting the following error: ***** Running training ***** Num examples = 647 Num Epochs = 3 Instantaneous batch size per device = 8 Total train batch size (w. parallel, distributed & accumulation) = 32 Gradient Accumulation steps = 4 Total optimization steps = 60 Number of trainable parameters = 25,165,824 `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... /opt/conda/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py:600: UserWarning: torch.utils.checkpoint: the use_reentrant parameter should be passed explicitly. In version 2.4 we will raise an exception if use_reentrant is not passed. use_reentrant=False is recommended, but if you need to preserve the current default behavior, you can pass use_reentrant=True. Refer to docs for more details on the differences between the two variants. return fn(*args, **kwargs) You are not running the flash-attention implementation, expect numerical differences. /opt/conda/lib/python3.11/site-packages/torch/utils/checkpoint.py:295: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead. with torch.enable_grad(), device_autocast_ctx, torch.cpu.amp.autocast(**ctx.cpu_autocast_kwargs): # type: ignore[attr-defined] **Error operation not supported at line 383 in file /src/csrc/pythonInterface.cpp** mainly this part "Error operation not supported at line 383 in file /src/csrc/pythonInterface.cpp" the kernel then dies, below are packages version I am using transformers 4.44.2 torch 2.4.1 torchaudio 2.4.1 torchvision 0.19.1 accelerate 0.34.2 peft 0.12.0 and conda version is 24.3.0 trying on google colab it is working same code, but in jupyter lab it is not working Continue reading...
