RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 11.00 GiB total capacity; 8.99 GiB already allocated; 0 bytes free; 9.54 GiB reserved in total by PyTorch)

에러 발생!

RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 11.00 GiB total capacity; 8.99 GiB already allocated; 0 bytes free; 9.54 GiB reserved in total by PyTorch)  If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

https://artiiicy.tistory.com/61

[Pytorch] 특정 GPU 사용하기 / 여러개의 multi GPU parallel 하게 사용하기

알고보니 모델 구축하는 중간 과정에서

shape을 확인하기 위해

model을 불러오고 output을 뽑으면서

cuda memory를 많이 잡아먹었다

 

코드 중간중간

! nvidia-smi를 해서

 

어느 지점에서 memory를

많이 차지하는지 확인하는게 좋다

 

torch.cuda.is_available() # True
torch.cuda.get_device_name() # 'NVIDIA GeForce GTX 1080 Ti'

Output exceeds the size limit. Open the full output data in a text editor
Tue Jun 21 15:09:34 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.89       Driver Version: 465.89       CUDA Version: 11.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA Quadro K620 WDDM  | 00000000:03:00.0 Off |                  N/A |
| 35%   48C    P8     1W /  30W |     40MiB /  2048MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ... WDDM  | 00000000:04:00.0  On |                  N/A |
| 33%   51C    P8    14W / 250W |    510MiB / 11264MiB |     13%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

 

아무튼 해결완료!