收藏本站
开启辅助访问 充值积分切换到窄版

deepfacelab中文网

 找回密码
 立即注册（仅限QQ邮箱）
搜索
deepfacelab中文网»首页 › deepfacelab专区 › 综合讨论 › 遮罩训练加载全过程耗时
返回列表 发新帖
查看: 67|回复: 0

遮罩训练加载全过程耗时

[复制链接]
day270010678

25

主题

170

帖子

1186

积分

初级丹圣

Rank: 8Rank: 8

积分
1186
 楼主| 发表于 昨天 10:25 | 显示全部楼层 |阅读模式
星级打分
  • 1
  • 2
  • 3
  • 4
  • 5
 平均分:NAN  参与人数:0  我的评分:未评
  1. [DEBUG]   nn.tf_sess is None，需要创建新 session
  2. [TIMING] ▶   创建 TensorFlow session
  3. [TIMING] ◀   创建 TensorFlow session 耗时: 526.51ms
  4. Press enter in 2 seconds to override model settings.[DEBUG]   nn.tf_sess 已存在，重用现有 session
  5. [DEBUG]   nn.tf_sess 已存在，重用现有 session
  6. [TIMING] ▶   构建 XSeg 模型
  7. [TIMING] ◀   构建 XSeg 模型 耗时: 0.00ms
  8. [TIMING] ▶   获取模型权重
  9. [TIMING] ◀   获取模型权重 耗时: 477.68ms
  10. [TIMING] ▶   初始化优化器
  11. [TIMING] ◀   初始化优化器 耗时: 858.79ms
  12. [TIMING] ▶   加载/初始化权重
  13. [DEBUG]   尝试加载权重: D:\AI\Deepfacelab30\workspace\model\XSeg_256_opt.npy
  14. [DEBUG] load_weights(D:\AI\Deepfacelab30\workspace\model\XSeg_256_opt.npy) 开始
  15. [TIMING] ▶   读取权重文件
  16. [TIMING] ◀   读取权重文件 耗时: 475.26ms
  17. [TIMING] ▶   准备权重 tuples
  18. [TIMING] ◀   准备权重 tuples 耗时: 0.00ms
  19. [TIMING] ▶   batch_set_value
  20. [DEBUG] batch_set_value() 开始，tuples 数量: 223
  21. [TIMING] ▶   创建 assign_ops
  22. [TIMING] ◀   创建 assign_ops 耗时: 149.95ms
  23. [TIMING] ▶   nn.tf_sess.run(assign_ops)
  24. [TIMING] ◀   nn.tf_sess.run(assign_ops) 耗时: 280.37ms
  25. [DEBUG] batch_set_value() 完成
  26. [TIMING] ◀   batch_set_value 耗时: 430.32ms
  27. [DEBUG] load_weights() 完成
  28. [DEBUG]   尝试加载权重: D:\AI\Deepfacelab30\workspace\model\XSeg_256.npy
  29. [DEBUG] load_weights(D:\AI\Deepfacelab30\workspace\model\XSeg_256.npy) 开始
  30. [TIMING] ▶   读取权重文件
  31. [TIMING] ◀   读取权重文件 耗时: 670.27ms
  32. [TIMING] ▶   准备权重 tuples
  33. [TIMING] ◀   准备权重 tuples 耗时: 0.00ms
  34. [TIMING] ▶   batch_set_value
  35. [DEBUG] batch_set_value() 开始，tuples 数量: 222
  36. [TIMING] ▶   创建 assign_ops
  37. [TIMING] ◀   创建 assign_ops 耗时: 143.68ms
  38. [TIMING] ▶   nn.tf_sess.run(assign_ops)
  39. [TIMING] ◀   nn.tf_sess.run(assign_ops) 耗时: 271.30ms
  40. [DEBUG] batch_set_value() 完成
  41. [TIMING] ◀   batch_set_value 耗时: 414.97ms
  42. [DEBUG] load_weights() 完成
  43. [TIMING] ◀   加载/初始化权重 耗时: 1990.82ms
  44. [DEBUG] XSegNet.__init__() 完成
  45. [DEBUG] XSegNet.__init__() 开始
  46. [DEBUG] XSegNet.__init__() 开始
  47. [DEBUG] XSegNet.__init__() 开始
  48. [DEBUG] XSegNet.__init__() 开始
  49. [DEBUG] XSegNet.__init__() 开始
  50. [DEBUG] XSegNet.__init__() 开始
  51. [DEBUG] XSegNet.__init__() 开始
  52. [DEBUG] XSegNet.__init__() 开始
  53. Loading samples: 0it [00:00, ?it/s]
  54. [DEBUG] XSegNet.__init__() 开始
  55. [DEBUG] XSegNet.__init__() 开始
  56. [DEBUG] XSegNet.__init__() 开始
  57. [DEBUG] XSegNet.__init__() 开始
  58. [DEBUG] XSegNet.__init__() 开始
  59. [DEBUG] XSegNet.__init__() 开始
  60. [DEBUG] XSegNet.__init__() 开始
  61. [DEBUG] XSegNet.__init__() 开始
  62. Loading samples: 100%|############################################################| 2069/2069 [00:10<00:00, 192.56it/s]
  63. [DEBUG] XSegNet.__init__() 开始
  64. [DEBUG] XSegNet.__init__() 开始
  65. [DEBUG] XSegNet.__init__() 开始
  66. [DEBUG] XSegNet.__init__() 开始
  67. [DEBUG] XSegNet.__init__() 开始
  68. [DEBUG] XSegNet.__init__() 开始
  69. [DEBUG] XSegNet.__init__() 开始
  70. [DEBUG] XSegNet.__init__() 开始
  71. Filtering: 100%|##################################################################| 2069/2069 [00:04<00:00, 491.57it/s]
  72. Using 11 segmented samples.
  73. [DEBUG] XSegNet.__init__() 开始
  74. [DEBUG] XSegNet.__init__() 开始
  75. [DEBUG] XSegNet.__init__() 开始
  76. [DEBUG] XSegNet.__init__() 开始
  77. [DEBUG] XSegNet.__init__() 开始
  78. [DEBUG] XSegNet.__init__() 开始
  79. [DEBUG] XSegNet.__init__() 开始
  80. [DEBUG] XSegNet.__init__() 开始
  81. ================ Model Summary =================
  82. ==                                            ==
  83. ==        Model name: XSeg                    ==
  84. ==                                            ==
  85. == Current iteration: 11546686                ==
  86. ==                                            ==
  87. ==-------------- Model Options ---------------==
  88. ==                                            ==
  89. ==         face_type: wf                      ==
  90. ==        batch_size: 4                       ==
  91. ==          pretrain: False                   ==
  92. ==                                            ==
  93. ==---------------- Running On ----------------==
  94. ==                                            ==
  95. ==      Device index: 0                       ==
  96. ==              Name: NVIDIA GeForce RTX 3050 ==
  97. ==              VRAM: 3.43GB                  ==
  98. ==                                            ==
  99. ================================================
  100. Starting. Press "Enter" to stop training and save model.

  102. ================================================================================
  103. 开始训练迭代 #11546686
  104. ================================================================================
  105. [TIMING] ▶ 训练迭代总时间
  106. [TIMING] ▶   model.train_one_iter()
  107. [DEBUG] onTrainOneIter 开始
  108. [TIMING] ▶   generate_next_samples()
  109. [TIMING] ◀   generate_next_samples() 耗时: 20.02ms
  110. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  111. [TIMING] ▶   train()
  112. [DEBUG] train() 开始
  113. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  114. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  115. [TIMING] ▶     准备 feed_dict
  116. [TIMING] ◀     准备 feed_dict 耗时: 0.00ms
  117. [DEBUG] feed_dict 准备完成
  118. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  119. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 5474.69ms
  120. [DEBUG] train() 完成
  121. [TIMING] ◀   train() 耗时: 5474.69ms
  122. [DEBUG] 损失: 0.0209
  123. [TIMING] ◀   model.train_one_iter() 耗时: 5494.71ms
  124. [TIMING] ◀ 训练迭代总时间 耗时: 5494.71ms
  125. 迭代 #11546687 完成，耗时: 5494.71ms
  126. ================================================================================

  128. [10:19:26][#11546687][5494ms][0.0209]
  129. ================================================================================
  130. 开始训练迭代 #11546687
  131. ================================================================================
  132. [TIMING] ▶ 训练迭代总时间
  133. [TIMING] ▶   model.train_one_iter()
  134. [DEBUG] onTrainOneIter 开始
  135. [TIMING] ▶   generate_next_samples()
  136. [TIMING] ◀   generate_next_samples() 耗时: 15.90ms
  137. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  138. [TIMING] ▶   train()
  139. [DEBUG] train() 开始
  140. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  141. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  142. [TIMING] ▶     准备 feed_dict
  143. [TIMING] ◀     准备 feed_dict 耗时: 0.00ms
  144. [DEBUG] feed_dict 准备完成
  145. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  146. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 2290.56ms
  147. [DEBUG] train() 完成
  148. [TIMING] ◀   train() 耗时: 2290.56ms
  149. [DEBUG] 损失: 0.0188
  150. [TIMING] ◀   model.train_one_iter() 耗时: 2322.21ms
  151. [TIMING] ◀ 训练迭代总时间 耗时: 2337.98ms
  152. 迭代 #11546688 完成，耗时: 2322.21ms
  153. ================================================================================

  155. [10:19:29][#11546688][2322ms][0.0188]
  156. ================================================================================
  157. 开始训练迭代 #11546688
  158. ================================================================================
  159. [TIMING] ▶ 训练迭代总时间
  160. [TIMING] ▶   model.train_one_iter()
  161. [DEBUG] onTrainOneIter 开始
  162. [TIMING] ▶   generate_next_samples()
  163. [TIMING] ◀   generate_next_samples() 耗时: 143.49ms
  164. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  165. [TIMING] ▶   train()
  166. [DEBUG] train() 开始
  167. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  168. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  169. [TIMING] ▶     准备 feed_dict
  170. [TIMING] ◀     准备 feed_dict 耗时: 31.82ms
  171. [DEBUG] feed_dict 准备完成
  172. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  173. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 493.01ms
  174. [DEBUG] train() 完成
  175. [TIMING] ◀   train() 耗时: 668.24ms
  176. [DEBUG] 损失: 0.0179
  177. [TIMING] ◀   model.train_one_iter() 耗时: 907.09ms
  178. [TIMING] ◀ 训练迭代总时间 耗时: 954.32ms
  179. 迭代 #11546689 完成，耗时: 891.84ms
  180. ================================================================================

  182. [10:19:30][#11546689][0891ms][0.0179]
  183. ================================================================================
  184. 开始训练迭代 #11546689
  185. ================================================================================
  186. [TIMING] ▶ 训练迭代总时间
  187. [TIMING] ▶   model.train_one_iter()
  188. [DEBUG] onTrainOneIter 开始
  189. [TIMING] ▶   generate_next_samples()
  190. [TIMING] ◀   generate_next_samples() 耗时: 173.48ms
  191. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  192. [TIMING] ▶   train()
  193. [DEBUG] train() 开始
  194. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  195. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  196. [TIMING] ▶     准备 feed_dict
  197. [TIMING] ◀     准备 feed_dict 耗时: 47.86ms
  198. [DEBUG] feed_dict 准备完成
  199. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  200. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 284.63ms
  201. [DEBUG] train() 完成
  202. [TIMING] ◀   train() 耗时: 523.61ms
  203. [DEBUG] 损失: 0.0260
  204. [TIMING] ◀   model.train_one_iter() 耗时: 840.03ms
  205. [TIMING] ◀ 训练迭代总时间 耗时: 871.75ms
  206. 迭代 #11546690 完成，耗时: 808.24ms
  207. ================================================================================

  209. [10:19:31][#11546690][0808ms][0.0260]
  210. ================================================================================
  211. 开始训练迭代 #11546690
  212. ================================================================================
  213. [TIMING] ▶ 训练迭代总时间
  214. [TIMING] ▶   model.train_one_iter()
  215. [DEBUG] onTrainOneIter 开始
  216. [TIMING] ▶   generate_next_samples()
  217. [TIMING] ◀   generate_next_samples() 耗时: 47.40ms
  218. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  219. [TIMING] ▶   train()
  220. [DEBUG] train() 开始
  221. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  222. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  223. [TIMING] ▶     准备 feed_dict
  224. [TIMING] ◀     准备 feed_dict 耗时: 15.69ms
  225. [DEBUG] feed_dict 准备完成
  226. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  227. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 286.07ms
  228. [DEBUG] train() 完成
  229. [TIMING] ◀   train() 耗时: 396.81ms
  230. [DEBUG] 损失: 0.1187
  231. [TIMING] ◀   model.train_one_iter() 耗时: 540.12ms
  232. [TIMING] ◀ 训练迭代总时间 耗时: 572.00ms
  233. 迭代 #11546691 完成，耗时: 523.99ms
  234. ================================================================================

  236. [10:19:32][#11546691][0523ms][0.1187]
  237. ================================================================================
  238. 开始训练迭代 #11546691
  239. ================================================================================
  240. [TIMING] ▶ 训练迭代总时间
  241. [TIMING] ▶   model.train_one_iter()
  242. [DEBUG] onTrainOneIter 开始
  243. [TIMING] ▶   generate_next_samples()
  244. [TIMING] ◀   generate_next_samples() 耗时: 47.78ms
  245. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  246. [TIMING] ▶   train()
  247. [DEBUG] train() 开始
  248. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  249. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  250. [TIMING] ▶     准备 feed_dict
  251. [TIMING] ◀     准备 feed_dict 耗时: 16.16ms
  252. [DEBUG] feed_dict 准备完成
  253. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  254. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 285.10ms
  255. [DEBUG] train() 完成
  256. [TIMING] ◀   train() 耗时: 459.45ms
  257. [DEBUG] 损失: 0.0194
  258. [TIMING] ◀   model.train_one_iter() 耗时: 602.22ms
  259. [TIMING] ◀ 训练迭代总时间 耗时: 617.99ms
  260. 迭代 #11546692 完成，耗时: 586.14ms
  261. ================================================================================

  263. [10:19:33][#11546692][0586ms][0.0194]
  264. ================================================================================
  265. 开始训练迭代 #11546692
  266. ================================================================================
  267. [TIMING] ▶ 训练迭代总时间
  268. [TIMING] ▶   model.train_one_iter()
  269. [DEBUG] onTrainOneIter 开始
  270. [TIMING] ▶   generate_next_samples()
  271. [TIMING] ◀   generate_next_samples() 耗时: 95.40ms
  272. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  273. [TIMING] ▶   train()
  274. [DEBUG] train() 开始
  275. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  276. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  277. [TIMING] ▶     准备 feed_dict
  278. [TIMING] ◀     准备 feed_dict 耗时: 16.07ms
  279. [DEBUG] feed_dict 准备完成
  280. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  281. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 280.33ms
  282. [DEBUG] train() 完成
  283. [TIMING] ◀   train() 耗时: 533.54ms
  284. [DEBUG] 损失: 0.0447
  285. [TIMING] ◀   model.train_one_iter() 耗时: 692.93ms
  286. [TIMING] ◀ 训练迭代总时间 耗时: 709.09ms
  287. 迭代 #11546693 完成，耗时: 677.12ms
  288. ================================================================================

  290. [10:19:33][#11546693][0677ms][0.0447]
  291. ================================================================================
  292. 开始训练迭代 #11546693
  293. ================================================================================
  294. [TIMING] ▶ 训练迭代总时间
  295. [TIMING] ▶   model.train_one_iter()
  296. [DEBUG] onTrainOneIter 开始
  297. [TIMING] ▶   generate_next_samples()
  298. [TIMING] ◀   generate_next_samples() 耗时: 9.50ms
  299. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  300. [TIMING] ▶   train()
  301. [DEBUG] train() 开始
  302. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  303. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  304. [TIMING] ▶     准备 feed_dict
  305. [TIMING] ◀     准备 feed_dict 耗时: 0.00ms
  306. [DEBUG] feed_dict 准备完成
  307. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  308. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 248.04ms
  309. [DEBUG] train() 完成
  310. [TIMING] ◀   train() 耗时: 248.04ms
  311. [DEBUG] 损失: 0.0182
  312. [TIMING] ◀   model.train_one_iter() 耗时: 273.63ms
  313. [TIMING] ◀ 训练迭代总时间 耗时: 273.63ms
  314. 迭代 #11546694 完成，耗时: 273.63ms
  315. ================================================================================

  317. [10:19:34][#11546694][0273ms][0.0182]
  318. ================================================================================
  319. 开始训练迭代 #11546694
  320. ================================================================================
  321. [TIMING] ▶ 训练迭代总时间
  322. [TIMING] ▶   model.train_one_iter()
  323. [DEBUG] onTrainOneIter 开始
  324. [TIMING] ▶   generate_next_samples()
  325. [TIMING] ◀   generate_next_samples() 耗时: 0.00ms
  326. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  327. [TIMING] ▶   train()
  328. [DEBUG] train() 开始
  329. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  330. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  331. [TIMING] ▶     准备 feed_dict
  332. [TIMING] ◀     准备 feed_dict 耗时: 0.00ms
  333. [DEBUG] feed_dict 准备完成
  334. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  335. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 253.41ms
  336. [DEBUG] train() 完成
  337. [TIMING] ◀   train() 耗时: 253.41ms
  338. [DEBUG] 损失: 0.0754
  339. [TIMING] ◀   model.train_one_iter() 耗时: 253.41ms
  340. [TIMING] ◀ 训练迭代总时间 耗时: 253.41ms
  341. 迭代 #11546695 完成，耗时: 253.41ms
  342. ================================================================================

  344. [10:19:34][#11546695][0253ms][0.0754]
  345. ================================================================================
  346. 开始训练迭代 #11546695
  347. ================================================================================
  348. [TIMING] ▶ 训练迭代总时间
  349. [TIMING] ▶   model.train_one_iter()
  350. [DEBUG] onTrainOneIter 开始
  351. [TIMING] ▶   generate_next_samples()
  352. [TIMING] ◀   generate_next_samples() 耗时: 15.69ms
  353. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  354. [TIMING] ▶   train()
  355. [DEBUG] train() 开始
  356. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  357. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  358. [TIMING] ▶     准备 feed_dict
  359. [TIMING] ◀     准备 feed_dict 耗时: 0.00ms
  360. [DEBUG] feed_dict 准备完成
  361. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  362. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 253.66ms
  363. [DEBUG] train() 完成
  364. [TIMING] ◀   train() 耗时: 253.66ms
  365. [DEBUG] 损失: 0.0190
  366. [TIMING] ◀   model.train_one_iter() 耗时: 269.35ms
  367. [TIMING] ◀ 训练迭代总时间 耗时: 269.35ms
  368. 迭代 #11546696 完成，耗时: 269.35ms
  369. ================================================================================

  371. [10:19:34][#11546696][0269ms][0.0190]
  372. ================================================================================
  373. 开始训练迭代 #11546696
  374. ================================================================================
  375. [TIMING] ▶ 训练迭代总时间
  376. [TIMING] ▶   model.train_one_iter()
  377. [DEBUG] onTrainOneIter 开始
  378. [TIMING] ▶   generate_next_samples()
  379. [TIMING] ◀   generate_next_samples() 耗时: 16.11ms
  380. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  381. [TIMING] ▶   train()
  382. [DEBUG] train() 开始
  383. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  384. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  385. [TIMING] ▶     准备 feed_dict
  386. [TIMING] ◀     准备 feed_dict 耗时: 0.00ms
  387. [DEBUG] feed_dict 准备完成
  388. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  389. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 254.67ms
  390. [DEBUG] train() 完成
  391. [TIMING] ◀   train() 耗时: 254.67ms
  392. [DEBUG] 损失: 0.0185
  393. [TIMING] ◀   model.train_one_iter() 耗时: 270.78ms
  394. [TIMING] ◀ 训练迭代总时间 耗时: 270.78ms
  395. 迭代 #11546697 完成，耗时: 270.78ms
  396. ================================================================================

  398. [10:19:34][#11546697][0270ms][0.0185]
  399. ================================================================================
  400. 开始训练迭代 #11546697
  401. ================================================================================
  402. [TIMING] ▶ 训练迭代总时间
  403. [TIMING] ▶   model.train_one_iter()
  404. [DEBUG] onTrainOneIter 开始
  405. [TIMING] ▶   generate_next_samples()
  406. [TIMING] ◀   generate_next_samples() 耗时: 17.64ms
  407. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  408. [TIMING] ▶   train()
  409. [DEBUG] train() 开始
  410. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  411. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  412. [TIMING] ▶     准备 feed_dict
  413. [TIMING] ◀     准备 feed_dict 耗时: 0.00ms
  414. [DEBUG] feed_dict 准备完成
  415. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  416. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 252.90ms
  417. [DEBUG] train() 完成
  418. [TIMING] ◀   train() 耗时: 252.90ms
  419. [DEBUG] 损失: 0.0537
  420. [TIMING] ◀   model.train_one_iter() 耗时: 270.55ms
  421. [TIMING] ◀ 训练迭代总时间 耗时: 270.55ms
  422. 迭代 #11546698 完成，耗时: 270.55ms
  423. ================================================================================

  425. [10:19:35][#11546698][0270ms][0.0537]
  426. ================================================================================
  427. 开始训练迭代 #11546698
  428. ================================================================================
  429. [TIMING] ▶ 训练迭代总时间
  430. [TIMING] ▶   model.train_one_iter()
  431. [DEBUG] onTrainOneIter 开始
  432. [TIMING] ▶   generate_next_samples()
  433. [TIMING] ◀   generate_next_samples() 耗时: 0.00ms
  434. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  435. [TIMING] ▶   train()
  436. [DEBUG] train() 开始
  437. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  438. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  439. [TIMING] ▶     准备 feed_dict
  440. [TIMING] ◀     准备 feed_dict 耗时: 0.00ms
  441. [DEBUG] feed_dict 准备完成
  442. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  443. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 254.48ms
  444. [DEBUG] train() 完成
  445. [TIMING] ◀   train() 耗时: 254.48ms
  446. [DEBUG] 损失: 0.0399
  447. [TIMING] ◀   model.train_one_iter() 耗时: 254.48ms
  448. [TIMING] ◀ 训练迭代总时间 耗时: 254.48ms
  449. 迭代 #11546699 完成，耗时: 254.48ms
  450. ================================================================================

  452. [10:19:35][#11546699][0254ms][0.0399]
  453. ================================================================================
  454. 开始训练迭代 #11546699
  455. ================================================================================
  456. [TIMING] ▶ 训练迭代总时间
  457. [TIMING] ▶   model.train_one_iter()
  458. [DEBUG] onTrainOneIter 开始
  459. [TIMING] ▶   generate_next_samples()
  460. [TIMING] ◀   generate_next_samples() 耗时: 15.79ms
  461. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  462. [TIMING] ▶   train()
  463. [DEBUG] train() 开始
  464. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  465. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  466. [TIMING] ▶     准备 feed_dict
  467. [TIMING] ◀     准备 feed_dict 耗时: 0.00ms
  468. [DEBUG] feed_dict 准备完成
  469. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  470. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 254.06ms
  471. [DEBUG] train() 完成
  472. [TIMING] ◀   train() 耗时: 254.06ms
  473. [DEBUG] 损失: 0.0206
  474. [TIMING] ◀   model.train_one_iter() 耗时: 269.85ms
  475. [TIMING] ◀ 训练迭代总时间 耗时: 269.85ms
  476. 迭代 #11546700 完成，耗时: 269.85ms
  477. ================================================================================

  479. [10:19:35][#11546700][0269ms][0.0206]
  480. ================================================================================
  481. 开始训练迭代 #11546700
  482. ================================================================================
  483. [TIMING] ▶ 训练迭代总时间
  484. [TIMING] ▶   model.train_one_iter()
  485. [DEBUG] onTrainOneIter 开始
  486. [TIMING] ▶   generate_next_samples()
  487. [TIMING] ◀   generate_next_samples() 耗时: 0.00ms
  488. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  489. [TIMING] ▶   train()
  490. [DEBUG] train() 开始
  491. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  492. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  493. [TIMING] ▶     准备 feed_dict
  494. [TIMING] ◀     准备 feed_dict 耗时: 0.00ms
  495. [DEBUG] feed_dict 准备完成
  496. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  497. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 268.98ms
  498. [DEBUG] train() 完成
  499. [TIMING] ◀   train() 耗时: 268.98ms
  500. [DEBUG] 损失: 0.0304
  501. [TIMING] ◀   model.train_one_iter() 耗时: 268.98ms
  502. [TIMING] ◀ 训练迭代总时间 耗时: 268.98ms
  503. 迭代 #11546701 完成，耗时: 268.98ms
  504. ================================================================================

  506. [10:19:35][#11546701][0268ms][0.0304]
  507. ================================================================================
  508. 开始训练迭代 #11546701
  509. ================================================================================
  510. [TIMING] ▶ 训练迭代总时间
  511. [TIMING] ▶   model.train_one_iter()
  512. [DEBUG] onTrainOneIter 开始
  513. [TIMING] ▶   generate_next_samples()
  514. [TIMING] ◀   generate_next_samples() 耗时: 0.00ms
  515. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  516. [TIMING] ▶   train()
  517. [DEBUG] train() 开始
  518. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  519. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  520. [TIMING] ▶     准备 feed_dict
  521. [TIMING] ◀     准备 feed_dict 耗时: 0.00ms
  522. [DEBUG] feed_dict 准备完成
  523. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  524. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 268.54ms
  525. [DEBUG] train() 完成
  526. [TIMING] ◀   train() 耗时: 270.05ms
  527. [DEBUG] 损失: 0.0546
  528. [TIMING] ◀   model.train_one_iter() 耗时: 270.05ms
  529. [TIMING] ◀ 训练迭代总时间 耗时: 270.05ms
  530. 迭代 #11546702 完成，耗时: 270.05ms
  531. ================================================================================

  533. [10:19:36][#11546702][0270ms][0.0546]
  534. ================================================================================
  535. 开始训练迭代 #11546702
  536. ================================================================================
  537. [TIMING] ▶ 训练迭代总时间
  538. [TIMING] ▶   model.train_one_iter()
  539. [DEBUG] onTrainOneIter 开始
  540. [TIMING] ▶   generate_next_samples()
  541. [TIMING] ◀   generate_next_samples() 耗时: 0.00ms
  542. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  543. [TIMING] ▶   train()
  544. [DEBUG] train() 开始
  545. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  546. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  547. [TIMING] ▶     准备 feed_dict
  548. [TIMING] ◀     准备 feed_dict 耗时: 0.00ms
  549. [DEBUG] feed_dict 准备完成
  550. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  551. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 253.47ms
  552. [DEBUG] train() 完成
  553. [TIMING] ◀   train() 耗时: 253.47ms
  554. [DEBUG] 损失: 0.0500
  555. [TIMING] ◀   model.train_one_iter() 耗时: 253.47ms
  556. [TIMING] ◀ 训练迭代总时间 耗时: 253.47ms
  557. 迭代 #11546703 完成，耗时: 253.47ms
  558. ================================================================================

  560. [10:19:36][#11546703][0253ms][0.0500]
  561. ================================================================================
  562. 开始训练迭代 #11546703
  563. ================================================================================
  564. [TIMING] ▶ 训练迭代总时间
  565. [TIMING] ▶   model.train_one_iter()
  566. [DEBUG] onTrainOneIter 开始
  567. [TIMING] ▶   generate_next_samples()
  568. [TIMING] ◀   generate_next_samples() 耗时: 18.34ms
  569. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  570. [TIMING] ▶   train()
  571. [DEBUG] train() 开始
  572. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  573. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  574. [TIMING] ▶     准备 feed_dict
  575. [TIMING] ◀     准备 feed_dict 耗时: 0.00ms
  576. [DEBUG] feed_dict 准备完成
  577. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  578. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 251.91ms
  579. [DEBUG] train() 完成
  580. [TIMING] ◀   train() 耗时: 251.91ms
  581. [DEBUG] 损失: 0.0154
  582. [TIMING] ◀   model.train_one_iter() 耗时: 270.25ms
  583. [TIMING] ◀ 训练迭代总时间 耗时: 270.25ms
  584. 迭代 #11546704 完成，耗时: 270.25ms
  585. ================================================================================

  587. [10:19:36][#11546704][0270ms][0.0154]
  588. ================================================================================
  589. 开始训练迭代 #11546704
  590. ================================================================================
  591. [TIMING] ▶ 训练迭代总时间
  592. [TIMING] ▶   model.train_one_iter()
  593. [DEBUG] onTrainOneIter 开始
  594. [TIMING] ▶   generate_next_samples()
  595. [TIMING] ◀   generate_next_samples() 耗时: 0.00ms
  596. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  597. [TIMING] ▶   train()
  598. [DEBUG] train() 开始
  599. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  600. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  601. [TIMING] ▶     准备 feed_dict
  602. [TIMING] ◀     准备 feed_dict 耗时: 0.00ms
  603. [DEBUG] feed_dict 准备完成
  604. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  605. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 267.56ms
  606. [DEBUG] train() 完成
  607. [TIMING] ◀   train() 耗时: 267.56ms
  608. [DEBUG] 损失: 0.0186
  609. [TIMING] ◀   model.train_one_iter() 耗时: 267.56ms
  610. [TIMING] ◀ 训练迭代总时间 耗时: 267.56ms
  611. 迭代 #11546705 完成，耗时: 267.56ms
  612. ================================================================================

  614. [10:19:37][#11546705][0267ms][0.0186]
  615. ================================================================================
  616. 开始训练迭代 #11546705
  617. ================================================================================
  618. [TIMING] ▶ 训练迭代总时间
  619. [TIMING] ▶   model.train_one_iter()
  620. [DEBUG] onTrainOneIter 开始
  621. [TIMING] ▶   generate_next_samples()
  622. [TIMING] ◀   generate_next_samples() 耗时: 7.88ms
  623. [DEBUG] 输入数据形状: (4, 3, 256, 256), (4, 1, 256, 256)
  624. [TIMING] ▶   train()
  625. [DEBUG] train() 开始
  626. [DEBUG]   input_np.shape: (4, 3, 256, 256)
  627. [DEBUG]   target_np.shape: (4, 1, 256, 256)
  628. [TIMING] ▶     准备 feed_dict
  629. [TIMING] ◀     准备 feed_dict 耗时: 0.00ms
  630. [DEBUG] feed_dict 准备完成
  631. [TIMING] ▶     nn.tf_sess.run([loss, loss_gv_op])
  632. [TIMING] ◀     nn.tf_sess.run([loss, loss_gv_op]) 耗时: 247.75ms
  633. [DEBUG] train() 完成
  634. [TIMING] ◀   train() 耗时: 247.75ms
  635. [DEBUG] 损失: 0.0294
  636. [TIMING] ◀   model.train_one_iter() 耗时: 255.63ms
  637. [TIMING] ◀ 训练迭代总时间 耗时: 255.63ms
  638. 迭代 #11546706 完成，耗时: 255.63ms
  639. ================================================================================
复制代码


回复

使用道具 举报

返回列表 发新帖

QQ|Archiver|手机版|deepfacelab中文网 |网站地图

GMT+8, 2026-2-14 06:54 , Processed in 0.099556 second(s), 29 queries .

Powered by Discuz! X3.4

Copyright © 2001-2020, Tencent Cloud.

快速回复 返回顶部 返回列表