求助saehd训练开不起来，有大佬帮忙看一下吗？

zhihai8495

星级打分

• 1
• 2
• 3
• 4
• 5

平均分:NAN  参与人数:0  我的评分:未评

quick96可以打开，saehd训练读条到80%就出现下面的问题，有大佬能帮忙看一下吗？
Initializing models:  80%|##################################################4            | 4/5 [00:44<00:11, 11.15s/it]
Error: OOM when allocating tensor with shape[1024] and type float on /job:localhost/replica:0/task:0/deviceML:0 by allocator DmlAllocator
         [[node src_dst_opt/ms_decoder/upscale1/conv1/bias_0/Assign (defined at D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\framework\ops.py:1762) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.


Original stack trace for 'src_dst_opt/ms_decoder/upscale1/conv1/bias_0/Assign':
  File "threading.py", line 884, in _bootstrap
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\DeepFaceLab\mainscripts\Trainer.py", line 58, in trainerThread
    debug=debug)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\DeepFaceLab\models\ModelBase.py", line 199, in __init__
    self.on_initialize()
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\DeepFaceLab\models\Model_SAEHD\Model.py", line 341, in on_initialize
    self.src_dst_opt.initialize_variables (self.src_dst_saveable_weights, vars_on_cpu=optimizer_vars_on_cpu, lr_dropout_on_cpu=self.options['lr_dropout']=='cpu')
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py", line 37, in initialize_variables
    ms = { v.name : tf.get_variable ( f'ms_{v.name}'.replace(':','_'), v.shape, dtype=v.dtype, initializer=tf.initializers.constant(0.0), trainable=False) for v in trainable_weights }
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\DeepFaceLab\core\leras\optimizers\AdaBelief.py", line 37, in <dictcomp>
    ms = { v.name : tf.get_variable ( f'ms_{v.name}'.replace(':','_'), v.shape, dtype=v.dtype, initializer=tf.initializers.constant(0.0), trainable=False) for v in trainable_weights }
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\ops\variable_scope.py", line 1500, in get_variable
    aggregation=aggregation)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\ops\variable_scope.py", line 1243, in get_variable
    aggregation=aggregation)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\ops\variable_scope.py", line 567, in get_variable
    aggregation=aggregation)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\ops\variable_scope.py", line 519, in _true_getter
    aggregation=aggregation)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\ops\variable_scope.py", line 933, in _get_single_variable
    aggregation=aggregation)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\ops\variables.py", line 258, in __call__
    return cls._variable_v1_call(*args, **kwargs)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\ops\variables.py", line 219, in _variable_v1_call
    shape=shape)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\ops\variables.py", line 197, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\ops\variable_scope.py", line 2519, in default_variable_creator
    shape=shape)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\ops\variables.py", line 262, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\ops\variables.py", line 1688, in __init__
    shape=shape)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\ops\variables.py", line 1862, in _init_from_args
    validate_shape=validate_shape).op
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\ops\state_ops.py", line 227, in assign
    validate_shape=validate_shape)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\ops\gen_state_ops.py", line 69, in assign
    use_locking=use_locking, name=name)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\framework\op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3371, in create_op
    attrs, op_def, compute_device)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\framework\ops.py", line 3440, in _create_op_internal
    op_def=op_def)
  File "D:\AI换脸\A卡、集显适用(DirectX12)\DFL_maozhihanhua_DirectX 12\_internal\python-3.6.8\lib\site-packages\tensorflow_core\python\framework\ops.py", line 1762, in __init__
    self._traceback = tf_stack.extract_stack()

丸丸

微软的系统你为什么非要用中文呢？

DFL800

路径使用中文了

marioking

OOM
显存不足

AKERSHUS

显存2G以上就能跑224 shead模型

此昵称已存在

1.路径别带中文
2.增大虚拟内存

zhihai8495

marioking 发表于 2022-4-17 00:08
OOM
显存不足

请问怎么设置增加显存？是要更换硬件吗？还是只需要设置就好了？

zhihai8495

此昵称已存在 发表于 2022-4-17 08:38
1.路径别带中文
2.增大虚拟内存

增大虚拟内存需要怎么操作设置？麻烦科普一下，谢谢

此昵称已存在

zhihai8495 发表于 2022-4-17 09:25
增大虚拟内存需要怎么操作设置？麻烦科普一下，谢谢

右键我的电脑-属性-高级系统设置-高级-性能-设置-高级-虚拟内存更改
然后在DFL存放的硬盘设置虚拟内存

marioking

zhihai8495 发表于 2022-4-17 09:24
请问怎么设置增加显存？是要更换硬件吗？还是只需要设置就好了？

显存无法增加，你的训练参数得降低