BS开到2了 虚拟内存 10G-50G 开lr_dropout 崩 啥原因

deer18

星级打分

• 1
• 2
• 3
• 4
• 5

平均分:NAN  参与人数:0  我的评分:未评

Error: failed to allocate memory
         [[node Mul_237 (defined at F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\core\leras\optimizers\OptimizerBase.py:23) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.


Errors may have originated from an input operation.
Input Source operations connected to node Mul_237:
gradients/AddN_44 (defined at F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\core\leras\ops\__init__.py:55)

Original stack trace for 'Mul_237':
  File "threading.py", line 884, in _bootstrap
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\mainscripts\Trainer.py", line 58, in trainerThread
    debug=debug)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\models\ModelBase.py", line 199, in __init__
    self.on_initialize()
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 564, in on_initialize
    src_dst_loss_gv_op = self.src_dst_opt.get_update_op (nn.average_gv_list (gpu_G_loss_gvs))
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\core\leras\optimizers\RMSprop.py", line 54, in get_update_op
    g = self.tf_clip_norm(g, self.clipnorm, tf.cast(norm, g.dtype) )
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\core\leras\optimizers\OptimizerBase.py", line 23, in tf_clip_norm
    then_expression = tf.scalar_mul(c / n, g)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\util\dispatch.py", line 206, in wrapper
    return target(*args, **kwargs)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\math_ops.py", line 625, in scalar_mul
    return gen_math_ops.mul(scalar, x, name)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 6245, in mul
    "Mul", x=x, y=y, name=name)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 750, in _apply_op_helper
    attrs=attr_protos, op_def=op_def)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 3569, in _create_op_internal
    op_def=op_def)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 2045, in __init__
    self._traceback = tf_stack.extract_stack_for_node(self._c_op)

Traceback (most recent call last):
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1375, in _do_call
    return fn(*args)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1360, in _run_fn
    target_list, run_metadata)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1453, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: failed to allocate memory
         [[{{node Mul_237}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\mainscripts\Trainer.py", line 131, in trainerThread
    iter, iter_time = model.train_one_iter()
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\models\ModelBase.py", line 480, in train_one_iter
    losses = self.onTrainOneIter()
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 774, in onTrainOneIter
    src_loss, dst_loss = self.src_dst_train (warped_src, target_src, target_srcm, target_srcm_em, warped_dst, target_dst, target_dstm, target_dstm_em)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 584, in src_dst_train
    self.target_dstm_em:target_dstm_em,
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 968, in run
    run_metadata_ptr)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1191, in _run
    feed_dict_tensor, options, run_metadata)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1369, in _do_run
    run_metadata)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1394, in _do_call
    raise type(e)(node_def, op, message)  # pylint: disable=no-value-for-parameter
tensorflow.python.framework.errors_impl.ResourceExhaustedError: failed to allocate memory
         [[node Mul_237 (defined at F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\core\leras\optimizers\OptimizerBase.py:23) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.


Errors may have originated from an input operation.
Input Source operations connected to node Mul_237:
gradients/AddN_44 (defined at F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\core\leras\ops\__init__.py:55)

Original stack trace for 'Mul_237':
  File "threading.py", line 884, in _bootstrap
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\mainscripts\Trainer.py", line 58, in trainerThread
    debug=debug)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\models\ModelBase.py", line 199, in __init__
    self.on_initialize()
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 564, in on_initialize
    src_dst_loss_gv_op = self.src_dst_opt.get_update_op (nn.average_gv_list (gpu_G_loss_gvs))
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\core\leras\optimizers\RMSprop.py", line 54, in get_update_op
    g = self.tf_clip_norm(g, self.clipnorm, tf.cast(norm, g.dtype) )
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\DeepFaceLab\core\leras\optimizers\OptimizerBase.py", line 23, in tf_clip_norm
    then_expression = tf.scalar_mul(c / n, g)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\util\dispatch.py", line 206, in wrapper
    return target(*args, **kwargs)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\math_ops.py", line 625, in scalar_mul
    return gen_math_ops.mul(scalar, x, name)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 6245, in mul
    "Mul", x=x, y=y, name=name)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 750, in _apply_op_helper
    attrs=attr_protos, op_def=op_def)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 3569, in _create_op_internal
    op_def=op_def)
  File "F:\DFL\3060\DFL_maozhihanhua_RTX3000\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 2045, in __init__
    self._traceback = tf_stack.extract_stack_for_node(self._c_op)


==================== Model Summary ====================
==                                                   ==
==            Model name: new_SAEHD                  ==
==                                                   ==
==     Current iteration: 4714856                    ==
==                                                   ==
==------------------ Model Options ------------------==
==                                                   ==
==            resolution: 224                        ==
==             face_type: wf                         ==
==     models_opt_on_gpu: True                       ==
==                 archi: liae-udt                   ==
==               ae_dims: 512                        ==
==                e_dims: 64                         ==
==                d_dims: 64                         ==
==           d_mask_dims: 32                         ==
==       masked_training: True                       ==
==       eyes_mouth_prio: True                       ==
==           uniform_yaw: True                       ==
==         blur_out_mask: True                       ==
==             adabelief: True                       ==
==            lr_dropout: y                          ==
==           random_warp: False                      ==
==      random_hsv_power: 0.1                        ==
==       true_face_power: 0.0                        ==
==      face_style_power: 0.0                        ==
==        bg_style_power: 0.0                        ==
==               ct_mode: none                       ==
==              clipgrad: False                      ==
==              pretrain: False                      ==
==       autobackup_hour: 0                          ==
== write_preview_history: False                      ==
==           target_iter: 0                          ==
==       random_src_flip: False                      ==
==       random_dst_flip: True                       ==
==            batch_size: 6                          ==
==             gan_power: 0.1                        ==
==        gan_patch_size: 28                         ==
==              gan_dims: 32                         ==
==                                                   ==
==------------------- Running On --------------------==
==                                                   ==
==          Device index: 0                          ==
==                  Name: NVIDIA GeForce RTX 3080 Ti ==
==                  VRAM: 9.22GB                     ==










3060TI显卡  显存8G     坛主推荐的那个 迪丽热巴Liae万能模型  

wtxx8888

啥原因看不到最后的 00M 标识？
显卡显存不够，别看只有224的解析度，这么高参的四维，不是你这个8G版的3060T能跑的

deer18

wtxx8888 发表于 2022-9-21 01:32
啥原因看不到最后的 00M 标识？
显卡显存不够，别看只有224的解析度，这么高参的四维，不是你这个8G版的306 ...

2张3060TI 能跑吗

qiuyuande

deer18 发表于 2022-9-21 01:56
2张3060TI 能跑吗

你开了学习率  把 lrd关了 就可以跑了，学习率很占显存的。

qiuyuande

qiuyuande 发表于 2022-9-21 01:58
你开了学习率  把 lrd关了 就可以跑了

早期训练  不要开学习率  把扭曲打开  和梯度裁剪打开  把 masked training 打开就可以  bs 2 可以跑

qiuyuande

qiuyuande 发表于 2022-9-21 02:01
早期训练  不要开学习率  把扭曲打开  和梯度裁剪打开  把 masked training 打开就可以  bs 2 可以跑 ...

3060TI 不行啊  还不如 1800元买个 12G显存的2060显卡  他这丹  BS5 学习率  AB优化器 gan 0.1 同时开都没事

qiuyuande

qiuyuande 发表于 2022-9-21 02:13
3060TI 不行啊  还不如 1800元买个 12G显存的2060显卡  他这丹  BS5 学习率  AB优化器 gan 0.1 同时开都 ...

二手矿卡 3090 4300元了

qiuyuande

CPU  也要搞个好U  训练的时候压缩素材处理能力快

推荐 3990X cpu挖矿很牛。。炼丹帮助也大大提升，

deer18

wtxx8888 发表于 2022-9-21 01:32
啥原因看不到最后的 00M 标识？
显卡显存不够，别看只有224的解析度，这么高参的四维，不是你这个8G版的306 ...

主要看哪四维？是AEdims 512 就跑不了吗

qiuyuande

deer18 发表于 2022-9-21 02:26
主要看哪四维？是AEdims 512 就跑不了吗

你这卡推荐你玩正在输入的224丹 效果也很好   我就是6G显卡 开始 玩到现在 3090的

这是正在输入大佬免费公开的224丹
链接：https://pan.baidu.com/s/1y2sE5Rpf021ChIUvs8NnRw
提取码：271w