Detected at node 'Pad_13' defined at (most recent call last):
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\python-3.6.8\lib\threading.py", line 885, in _bootstrap
self._bootstrap_inner()
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\python-3.6.8\lib\threading.py", line 917, in _bootstrap_inner
self.run()
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\python-3.6.8\lib\threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "<frozen mainscripts.Trainer>", line 58, in trainerThread
File "<frozen models.ModelBase>", line 207, in __init__
File "<frozen models.Model_SAEHD.Model>", line 423, in on_initialize
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\DeepFaceLab\core\leras\models\ModelBase.py", line 117, in __call__
return self.forward(*args, **kwargs)
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\DeepFaceLab\core\leras\archis\DeepFakeArchi.py", line 114, in forward
x = self.down3(x)
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\DeepFaceLab\core\leras\models\ModelBase.py", line 117, in __call__
return self.forward(*args, **kwargs)
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\DeepFaceLab\core\leras\archis\DeepFakeArchi.py", line 43, in forward
x = self.conv1(x)
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\DeepFaceLab\core\leras\layers\LayerBase.py", line 14, in __call__
return self.forward(*args, **kwargs)
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\DeepFaceLab\core\leras\layers\Conv2D.py", line 87, in forward
x = tf.pad (x, padding, mode='CONSTANT')
Node: 'Pad_13'
Detected at node 'Pad_13' defined at (most recent call last):
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\python-3.6.8\lib\threading.py", line 885, in _bootstrap
self._bootstrap_inner()
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\python-3.6.8\lib\threading.py", line 917, in _bootstrap_inner
self.run()
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\python-3.6.8\lib\threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "<frozen mainscripts.Trainer>", line 58, in trainerThread
File "<frozen models.ModelBase>", line 207, in __init__
File "<frozen models.Model_SAEHD.Model>", line 423, in on_initialize
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\DeepFaceLab\core\leras\models\ModelBase.py", line 117, in __call__
return self.forward(*args, **kwargs)
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\DeepFaceLab\core\leras\archis\DeepFakeArchi.py", line 114, in forward
x = self.down3(x)
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\DeepFaceLab\core\leras\models\ModelBase.py", line 117, in __call__
return self.forward(*args, **kwargs)
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\DeepFaceLab\core\leras\archis\DeepFakeArchi.py", line 43, in forward
x = self.conv1(x)
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\DeepFaceLab\core\leras\layers\LayerBase.py", line 14, in __call__
return self.forward(*args, **kwargs)
File "D:\yonghu\Desktop\DFL\DeepFaceLab mzhh\_internal\DeepFaceLab\core\leras\layers\Conv2D.py", line 87, in forward
x = tf.pad (x, padding, mode='CONSTANT')
Node: 'Pad_13'
2 root error(s) found.
(0) RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[4,76,76,160] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node Pad_13}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
[[concat_7/concat/_943]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
(1) RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[4,76,76,160] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node Pad_13}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.