换了主板和CPU训练之前的丹出现错误

pp64228485 · 发表于 2022-7-22 16:25:50

星级打分

1
2
3
4
5

平均分:NAN 参与人数:0 我的评分:未评

Running trainer.

Choose one of saved models, or enter a name to create a new model.
[r] : rename
[d] : delete

[0] : new - latest
:
0
Loading new_SAEHD model...

Choose one or several GPU idxs (separated by comma).

[CPU] : CPU
  [0] : NVIDIA GeForce GTX 770

[0] Which GPU indexes to choose? :
0

Initializing models: 100%|#######################| 5/5 [00:02<00:00,  2.41it/s]
Loading samples: 100%|####################| 5788/5788 [00:32<00:00, 176.42it/s]
Loading samples: 100%|######################| 300/300 [00:02<00:00, 127.44it/s]
================== Model Summary ==================
==                                              ==
==          Model name: new_SAEHD             ==
==                                              ==
==    Current iteration: 44073                ==
==                                              ==
==---------------- Model Options ----------------==
==                                              ==
==          resolution: 128                   ==
==          face_type: f                   ==
==    models_opt_on_gpu: True                ==
==                archi: df                   ==
==             ae_dims: 256                   ==
==             e_dims: 64                   ==
==             d_dims: 64                   ==
==          d_mask_dims: 22                   ==
==    masked_training: True                ==
==          eyes_prio: False                ==
==          uniform_yaw: False                ==
==          lr_dropout: n                   ==
==          random_warp: True                ==
==          gan_power: 0.0                   ==
==    true_face_power: 0.0                   ==
==    face_style_power: 0.0                   ==
==       bg_style_power: 0.0                   ==
==             ct_mode: none                ==
==             clipgrad: False                ==
==             pretrain: False                ==
==    autobackup_hour: 0                   ==
== write_preview_history: False                ==
==          target_iter: 0                   ==
==          random_flip: True                ==
==          batch_size: 4                   ==
==                                              ==
==----------------- Running On ------------------==
==                                              ==
==       Device index: 0                   ==
==                Name: NVIDIA GeForce GTX 770 ==
==                VRAM: 4.00GB                ==
==                                              ==
===================================================
Starting. Press "Enter" to stop training and save model.
[15:32:25][#044484][1172ms][0.5405][0.6002]
2022-07-22 15:47:43.122029: E tensorflow/stream_executor/cuda/cuda_event.cc:48]
Error polling for event status: failed to query event: CUDA_ERROR_LAUNCH_TIMEOUT
: the launch timed out and was terminated
2022-07-22 15:47:43.122029: E tensorflow/stream_executor/cuda/cuda_driver.cc:101
1] could not synchronize on CUDA context: CUDA_ERROR_LAUNCH_TIMEOUT: the launch
timed out and was terminated ::
2022-07-22 15:47:43.705062: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.c
c:274] Unexpected Event status: 1
请按任意键继续. . .

刚开始可以训练一定时间以后就闪退然后就这样了