|
发表于 2021-6-29 00:23:34
|
显示全部楼层
本帖最后由 可乐好久才 于 2021-6-29 00:25 编辑
在训练模型的时候,出现这个问题,请教大佬怎么解决,我的显卡是rtx 3050
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
启动训练程序.
[new] 没有发现模型,输入一个名字新建模型 : test
test
首次运行模型。
可用设备列表:
[CPU] : CPU
[0] : GeForce RTX 3050 Laptop GPU
[0] 选择哪一个设备? : 0
0
Initializing models: 100%|###############################################################| 5/5 [00:02<00:00, 1.72it/s]
实例加载中: 100%|###################################################################| 293/293 [00:00<00:00, 458.08it/s]
实例加载中: 100%|###################################################################| 836/836 [00:01<00:00, 489.60it/s]
================== Model Summary ===================
== ==
== Model name: test_Quick96 ==
== ==
== Current iteration: 0 ==
== ==
==---------------- Model Options -----------------==
== ==
== batch_size: 4 ==
== ==
==------------------ Running On ------------------==
== ==
== Device index: 0 ==
== Name: GeForce RTX 3050 Laptop GPU ==
== VRAM: 4.00GB ==
== ==
====================================================
Deepfake中文网:www.deepfaker.xyz
公众号:托尼是塔克
=============================================
启动中. 按 "Enter" 停止训练并保存进度。
保存时间|迭代次数|单次时间|源损失|目标损失
正在尝试运行第一个迭代. 如果出现错误, 请降低参数配置
!!!
rtx3000系列请注意. 为了正常运行程序你需开始硬件加速GPU计划
https://www.deepfaker.xyz/?p=2171
!!!
Error: OOM when allocating tensor of shape [128,1152] and type float
[[node src_dst_opt_1/ones_10 (defined at F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\core\leras\ops\__init__.py:207) ]]
Original stack trace for 'src_dst_opt_1/ones_10':
File "threading.py", line 884, in _bootstrap
File "threading.py", line 916, in _bootstrap_inner
File "threading.py", line 864, in run
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\mainscripts\Trainer.py", line 58, in trainerThread
debug=debug,
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\models\ModelBase.py", line 189, in __init__
self.on_initialize()
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\models\Model_Quick96\Model.py", line 77, in on_initialize
self.src_dst_opt.initialize_variables(self.src_dst_trainable_weights, vars_on_cpu=optimizer_vars_on_cpu )
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\core\leras\optimizers\RMSprop.py", line 41, in initialize_variables
lr_rnds = [ nn.random_binomial( v.shape, p=self.lr_dropout, dtype=v.dtype) for v in trainable_weights ]
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\core\leras\optimizers\RMSprop.py", line 41, in <listcomp>
lr_rnds = [ nn.random_binomial( v.shape, p=self.lr_dropout, dtype=v.dtype) for v in trainable_weights ]
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\core\leras\ops\__init__.py", line 207, in random_binomial
array_ops.ones(shape, dtype=dtype), array_ops.zeros(shape, dtype=dtype))
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\util\dispatch.py", line 201, in wrapper
return target(*args, **kwargs)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\array_ops.py", line 3132, in ones
output = fill(shape, constant(one, dtype=dtype), name=name)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\util\dispatch.py", line 201, in wrapper
return target(*args, **kwargs)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\array_ops.py", line 239, in fill
result = gen_array_ops.fill(dims, value, name=name)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 3357, in fill
"Fill", dims=dims, value=value, name=name)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 750, in _apply_op_helper
attrs=attr_protos, op_def=op_def)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 3536, in _create_op_internal
op_def=op_def)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 1990, in __init__
self._traceback = tf_stack.extract_stack()
Traceback (most recent call last):
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1375, in _do_call
return fn(*args)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1360, in _run_fn
target_list, run_metadata)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1453, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [128,1152] and type float
[[{{node src_dst_opt_1/ones_10}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\mainscripts\Trainer.py", line 130, in trainerThread
iter, iter_time = model.train_one_iter()
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\models\ModelBase.py", line 462, in train_one_iter
losses = self.onTrainOneIter()
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\models\Model_Quick96\Model.py", line 276, in onTrainOneIter
warped_dst, target_dst, target_dstm)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\models\Model_Quick96\Model.py", line 178, in src_dst_train
self.target_dstm:target_dstm,
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 968, in run
run_metadata_ptr)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1191, in _run
feed_dict_tensor, options, run_metadata)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1369, in _do_run
run_metadata)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\client\session.py", line 1394, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [128,1152] and type float
[[node src_dst_opt_1/ones_10 (defined at F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\core\leras\ops\__init__.py:207) ]]
Original stack trace for 'src_dst_opt_1/ones_10':
File "threading.py", line 884, in _bootstrap
File "threading.py", line 916, in _bootstrap_inner
File "threading.py", line 864, in run
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\mainscripts\Trainer.py", line 58, in trainerThread
debug=debug,
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\models\ModelBase.py", line 189, in __init__
self.on_initialize()
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\models\Model_Quick96\Model.py", line 77, in on_initialize
self.src_dst_opt.initialize_variables(self.src_dst_trainable_weights, vars_on_cpu=optimizer_vars_on_cpu )
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\core\leras\optimizers\RMSprop.py", line 41, in initialize_variables
lr_rnds = [ nn.random_binomial( v.shape, p=self.lr_dropout, dtype=v.dtype) for v in trainable_weights ]
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\core\leras\optimizers\RMSprop.py", line 41, in <listcomp>
lr_rnds = [ nn.random_binomial( v.shape, p=self.lr_dropout, dtype=v.dtype) for v in trainable_weights ]
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\DeepFaceLab\core\leras\ops\__init__.py", line 207, in random_binomial
array_ops.ones(shape, dtype=dtype), array_ops.zeros(shape, dtype=dtype))
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\util\dispatch.py", line 201, in wrapper
return target(*args, **kwargs)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\array_ops.py", line 3132, in ones
output = fill(shape, constant(one, dtype=dtype), name=name)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\util\dispatch.py", line 201, in wrapper
return target(*args, **kwargs)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\array_ops.py", line 239, in fill
result = gen_array_ops.fill(dims, value, name=name)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 3357, in fill
"Fill", dims=dims, value=value, name=name)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 750, in _apply_op_helper
attrs=attr_protos, op_def=op_def)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 3536, in _create_op_internal
op_def=op_def)
File "F:\Program Files (x86)\DeepFaceLab_NVIDIA20210104\_internal\python-3.6.8\lib\site-packages\tensorflow\python\framework\ops.py", line 1990, in __init__
self._traceback = tf_stack.extract_stack()
|
|