DF-UD_320_F_高质量高参模型

Han江孤影 · 发表于 2022-9-18 20:14:58

训练的时候报错

Han江孤影 · 发表于 2022-9-18 20:16:03

Resource exhausted: failed to allocate memory
报错

Han江孤影 · 发表于 2022-9-18 20:31:24

Errors may have originated from an input operation.
Input Source operations connected to node Conv2D_82:
mul_9 (defined at D:\dfl\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\models\Model_SAEHD\Model.py:455)
D_src/in_conv/weight/read (defined at D:\dfl\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\core\leras\layers\Conv2D.py:61)

Input Source operations connected to node Conv2D_82:
mul_9 (defined at D:\dfl\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\models\Model_SAEHD\Model.py:455)
D_src/in_conv/weight/read (defined at D:\dfl\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\core\leras\layers\Conv2D.py:61)

Original stack trace for 'Conv2D_82':
  File "threading.py", line 884, in _bootstrap
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "D:\dfl\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\mainscripts\Trainer.py", line 58, in trainerThread
debug=debug)
  File "D:\dfl\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\models\ModelBase.py", line 199, in __init__
self.on_initialize()
  File "D:\dfl\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 518, in on_initialize
gpu_pred_src_src_d2          = self.D_src(gpu_pred_src_src_masked_opt)
  File "D:\dfl\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\core\leras\models\ModelBase.py", line 117, in __call__
return self.forward(*args, **kwargs)
  File "D:\dfl\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\core\leras\models\PatchDiscriminator.py", line 173, in forward
x = tf.nn.leaky_relu( self.in_conv(x), 0.2 )
  File "D:\dfl\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\core\leras\layers\LayerBase.py", line 14, in __call__
return self.forward(*args, **kwargs)
  File "D:\dfl\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\core\leras\layers\Conv2D.py", line 101, in forward
x = tf.nn.conv2d(x, weight, strides, 'VALID', dilations=dilations, data_format=nn.data_format)
  File "D:\dfl\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\util\dispatch.py", line 206, in wrapper
return target(*args, **kwargs)
  File "D:\dfl\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 2397, in conv2d
name=name)
  File "D:\dfl\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 972, in conv2d
data_format=data_format, dilations=dilations, name=name)
  File "D:\dfl\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 750, in _apply_op_helper
attrs=attr_protos, op_def=op_def)
  File "D:\dfl\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 3569, in _create_op_internal
op_def=op_def)
  File "D:\dfl\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 2045, in __init__
self._traceback = tf_stack.extract_stack_for_node(self._c_op)
这又是啥问题

Han江孤影 · 发表于 2022-9-18 20:47:52

Resource exhausted: OOM when allocating tensor with shape[4,3,319,320] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
帮我看看我哭了