有人在4090 linux系统上跑过dfl吗？出问题了

国际狄拉克

星级打分

• 1
• 2
• 3
• 4
• 5

平均分:NAN  参与人数:0  我的评分:未评

1. OpenBLAS blas_thread_init: pthread_create failed for thread 21 of 22: Resource temporarily unavailable
   
2. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
   
3. Traceback (most recent call last):
   
4.   File "<string>", line 1, in <module>
   
5.   File "/root/miniconda3/envs/deepfacelab/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
   
6.     exitcode = _main(fd)
   
7.   File "/root/miniconda3/envs/deepfacelab/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
   
8.     self = reduction.pickle.load(from_parent)
   
9.   File "/root/autodl-tmp/DeepFaceLab_Linux/DeepFaceLab/samplelib/__init__.py", line 1, in <module>
   
10.     from .Sample import Sample
    
11.   File "/root/autodl-tmp/DeepFaceLab_Linux/DeepFaceLab/samplelib/Sample.py", line 7, in <module>
    
12.     from core.cv2ex import *
    
13.   File "/root/autodl-tmp/DeepFaceLab_Linux/DeepFaceLab/core/cv2ex.py", line 5, in <module>
    
14.     from core import imagelib
    
15.   File "/root/autodl-tmp/DeepFaceLab_Linux/DeepFaceLab/core/imagelib/__init__.py", line 9, in <module>
    
16.     from .morph import morph_by_points
    
17.   File "/root/autodl-tmp/DeepFaceLab_Linux/DeepFaceLab/core/imagelib/morph.py", line 3, in <module>
    
18.     from scipy.spatial import Delaunay
    
19.   File "/root/miniconda3/envs/deepfacelab/lib/python3.7/site-packages/scipy/spatial/__init__.py", line 99, in <module>
    
20.     from .qhull import *
    
21.   File "<frozen importlib._bootstrap>", line 416, in parent
    
22. KeyboardInterrupt
    
23. OpenBLAS blas_thread_init: pthread_create failed for thread 2 of 22: Resource temporarily unavailable
    
24. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
25. OpenBLAS blas_thread_init: pthread_create failed for thread 3 of 22: Resource temporarily unavailable
    
26. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
27. OpenBLAS blas_thread_init: pthread_create failed for thread 4 of 22: Resource temporarily unavailable
    
28. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
29. OpenBLAS blas_thread_init: pthread_create failed for thread 5 of 22: Resource temporarily unavailable
    
30. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
31. OpenBLAS blas_thread_init: pthread_create failed for thread 6 of 22: Resource temporarily unavailable
    
32. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
33. OpenBLAS blas_thread_init: pthread_create failed for thread 7 of 22: Resource temporarily unavailable
    
34. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
35. OpenBLAS blas_thread_init: pthread_create failed for thread 8 of 22: Resource temporarily unavailable
    
36. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
37. OpenBLAS blas_thread_init: pthread_create failed for thread 9 of 22: Resource temporarily unavailable
    
38. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
39. OpenBLAS blas_thread_init: pthread_create failed for thread 10 of 22: Resource temporarily unavailable
    
40. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
41. OpenBLAS blas_thread_init: pthread_create failed for thread 11 of 22: Resource temporarily unavailable
    
42. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
43. OpenBLAS blas_thread_init: pthread_create failed for thread 12 of 22: Resource temporarily unavailable
    
44. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
45. OpenBLAS blas_thread_init: pthread_create failed for thread 13 of 22: Resource temporarily unavailable
    
46. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
47. OpenBLAS blas_thread_init: pthread_create failed for thread 14 of 22: Resource temporarily unavailable
    
48. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
49. OpenBLAS blas_thread_init: pthread_create failed for thread 15 of 22: Resource temporarily unavailable
    
50. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
51. OpenBLAS blas_thread_init: pthread_create failed for thread 16 of 22: Resource temporarily unavailable
    
52. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
53. OpenBLAS blas_thread_init: pthread_create failed for thread 17 of 22: Resource temporarily unavailable
    
54. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
55. OpenBLAS blas_thread_init: pthread_create failed for thread 18 of 22: Resource temporarily unavailable
    
56. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
57. OpenBLAS blas_thread_init: pthread_create failed for thread 19 of 22: Resource temporarily unavailable
    
58. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
59. OpenBLAS blas_thread_init: pthread_create failed for thread 20 of 22: Resource temporarily unavailable
    
60. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
61. OpenBLAS blas_thread_init: pthread_create failed for thread 21 of 22: Resource temporarily unavailable
    
62. OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
    
63. ImportError: numpy.core.multiarray failed to import
    
64. Traceback (most recent call last):
    
65.   File "<string>", line 1, in <module>
    
66.   File "/root/miniconda3/envs/deepfacelab/lib/python3.7/multiprocessing/spawn.py", line 105, in spawn_main
    
67.     exitcode = _main(fd)
    
68.   File "/root/miniconda3/envs/deepfacelab/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
    
69.     self = reduction.pickle.load(from_parent)
    
70.   File "/root/autodl-tmp/DeepFaceLab_Linux/DeepFaceLab/core/joblib/__init__.py", line 1, in <module>
    
71.     from .SubprocessorBase import Subprocessor
    
72.   File "/root/autodl-tmp/DeepFaceLab_Linux/DeepFaceLab/core/joblib/SubprocessorBase.py", line 5, in <module>
    
73.     from core.interact import interact as io
    
74.   File "/root/autodl-tmp/DeepFaceLab_Linux/DeepFaceLab/core/interact/__init__.py", line 1, in <module>
    
75.     from .interact import interact
    
76.   File "/root/autodl-tmp/DeepFaceLab_Linux/DeepFaceLab/core/interact/interact.py", line 9, in <module>
    
77.     import cv2
    
78.   File "/root/miniconda3/envs/deepfacelab/lib/python3.7/site-packages/cv2/__init__.py", line 3, in <module>
    
79.     from .cv2 import *
    
80. ImportError: numpy.core.multiarray failed to import
    
81. ERROR; return code from pthread_create() is 11
    
82.         Error detail: Resource temporarily unavailable
    
83. ERROR; return code from pthread_create() is 11
    
84.         Error detail: Resource temporarily unavailable
    
85. ERROR; return code from pthread_create() is 11
    
86.         Error detail: Resource temporarily unavailable
    
87. Process Process-129:
    
88. Traceback (most recent call last):
    
89.   File "/root/miniconda3/envs/deepfacelab/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    
90.     self.run()
    
91.   File "/root/miniconda3/envs/deepfacelab/lib/python3.7/multiprocessing/process.py", line 99, in run
    
92.     self._target(*self._args, **self._kwargs)
    
93.   File "/root/autodl-tmp/DeepFaceLab_Linux/DeepFaceLab/core/joblib/SubprocessGenerator.py", line 58, in process_func
    
94.     self.cs_queue.put (gen_data)
    
95.   File "/root/miniconda3/envs/deepfacelab/lib/python3.7/multiprocessing/queues.py", line 87, in put
    
96.     self._start_thread()
    
97.   File "/root/miniconda3/envs/deepfacelab/lib/python3.7/multiprocessing/queues.py", line 170, in _start_thread
    
98.     self._thread.start()
    
99.   File "/root/miniconda3/envs/deepfacelab/lib/python3.7/threading.py", line 852, in start
    
100.     _start_new_thread(self._bootstrap, ())
     
101. RuntimeError: can't start new thread
     
102. 
     

复制代码

运行 `bash 6_train_SAEHD.sh` 命令后就开始说没办法创建线程了

lyjly

线程过多，系统无法创建新的线程。

国际狄拉克

lyjly 发表于 2023-8-4 23:31
线程过多，系统无法创建新的线程。

可是cpu就是22线程啊，怎么会多呢？
有办法解决吗

z_charles

国际狄拉克 发表于 2023-8-4 23:44
可是cpu就是22线程啊，怎么会多呢？
有办法解决吗

AMD cpu？可能会比较麻烦，设置环境变量，根据错误去搜

国际狄拉克

z_charles 发表于 2023-8-5 00:53
AMD cpu？可能会比较麻烦，设置环境变量，根据错误去搜

这错误难道还是amd专属的？这么诡异吗

lyjly

国际狄拉克 发表于 2023-8-4 23:44
可是cpu就是22线程啊，怎么会多呢？
有办法解决吗

解释起来有点麻烦，让ChatGPT回答你
这个问题看起来是因为你的系统没有足够的资源来创建新的线程或者进程。你可能遇到的问题是超出了最大用户进程数（RLIMIT_NPROC）的限制。这可能是由于内存不足，或者系统对创建新线程的数量有限制。

可以尝试以下方法来解决此问题：

增加操作系统的用户级线程或进程限制： 在Linux系统中，你可以用ulimit命令来检查和修改用户进程的数量。例如，你可以用ulimit -u来查看当前的限制，用ulimit -u 4096来将用户进程数的最大值设定为4096。如果你是root用户，你也可以直接修改/etc/security/limits.conf文件来永久性地提高用户进程数的限制。

减少程序使用的线程数： OpenBLAS库默认会使用系统上所有可用的CPU核心。如果你的系统资源有限，可以尝试减少OpenBLAS使用的线程数。你可以通过设置环境变量OPENBLAS_NUM_THREADS来实现，例如export OPENBLAS_NUM_THREADS=4将会限制OpenBLAS只使用4个线程。

释放一些系统资源： 关闭一些不需要的程序或服务，释放更多的系统资源给你的程序使用。

升级你的硬件： 如果你经常遇到类似的问题，可能是你的硬件资源不足以满足你的程序需求。考虑升级你的内存或者CPU来获得更多的资源。

另外，你的错误信息中也提到了numpy.core.multiarray无法导入，这可能是因为你的numpy库没有正确地安装或者版本不兼容。如果你尝试过上面的方法，问题还是没有解决的话，可能需要考虑重新安装或者升级你的numpy库。

dfllearner

不能开启线程的原因之一是内存不够

Pxhpanic

是不是autodl的

国际狄拉克

Pxhpanic 发表于 2023-8-5 12:19
是不是autodl的

被你发现了

国际狄拉克

lyjly 发表于 2023-8-5 04:04
解释起来有点麻烦，让ChatGPT回答你
这个问题看起来是因为你的系统没有足够的资源来创建新的线程或者进程 ...

回头我试试

你的gpt怎么感觉比我的智能好多啊