BrokenPipeError: [Errno 32] Broken pipe

xin376151654 · 发表于 2022-1-25 21:47:36

星级打分

1
2
3
4
5

平均分:NAN 参与人数:0 我的评分:未评

开启这条后就报错[lct] Color transfer for src faceset ( none/rct/lct/mkl/idt/sot ?:help ) : rct
这个是内存的配置

大神们有没有遇到过这个问题呀？

Model first run.

Choose one or several GPU idxs (separated by comma).

[CPU] : CPU
  [0] : NVIDIA GeForce RTX 3090
  [1] : NVIDIA GeForce RTX 3090
  [2] : NVIDIA GeForce RTX 3090
  [3] : NVIDIA GeForce RTX 3090
  [4] : NVIDIA GeForce RTX 3090
  [5] : NVIDIA GeForce RTX 3090
  [6] : NVIDIA GeForce RTX 3090
  [7] : NVIDIA GeForce RTX 3090

[0,1,2,3,4,5,6,7] Which GPU indexes to choose? :
0,1,2,3,4,5,6,7
[0] Autobackup every N hour ( 0..24 ?:help ) :
0
[n] Write preview history ( y/n ?:help ) :
n
[0] Target iteration :
0
[n] Flip SRC faces randomly ( y/n ?:help ) :
n
[y] Flip DST faces randomly ( y/n ?:help ) :
y
[4] Batch_size ( ?:help ) :
4
[128] Resolution ( 64-640 ?:help ) :
128
[wf] Face type ( h/mf/f/wf/head ?:help ) :
wf
[liae-ud] AE architecture ( ?:help ) :
liae-ud
[256] AutoEncoder dimensions ( 32-1024 ?:help ) :
256
[64] Encoder dimensions ( 16-256 ?:help ) :
64
[64] Decoder dimensions ( 16-256 ?:help ) :
64
[22] Decoder mask dimensions ( 16-256 ?:help ) :
22
[y] Masked training ( y/n ?:help ) :
y
[n] Eyes and mouth priority ( y/n ?:help ) :
n
[n] Uniform yaw distribution of samples ( y/n ?:help ) :
n
[n] Blur out mask ( y/n ?:help ) :
n
[y] Place models and optimizer on GPU ( y/n ?:help ) :
y
[y] Use AdaBelief optimizer? ( y/n ?:help ) :
y
[n] Use learning rate dropout ( n/y/cpu ?:help ) :
n
[y] Enable random warp of samples ( y/n ?:help ) :
y
[0.0] Random hue/saturation/light intensity ( 0.0 .. 0.3 ?:help ) :
0.0
[0.0] GAN power ( 0.0 .. 5.0 ?:help ) :
0.0
[0.0] Face style power ( 0.0..100.0 ?:help ) :
0.0
[0.0] Background style power ( 0.0..100.0 ?:help ) :
0.0
[lct] Color transfer for src faceset ( none/rct/lct/mkl/idt/sot ?:help ) : rct
rct
[n] Enable gradient clipping ( y/n ?:help ) :
n
[n] Enable pretraining mode ( y/n ?:help ) :
n
Initializing models: 100%|###############################################################| 5/5 [00:02<00:00,  1.69it/s]
Loading samples: 100%|############################################################| 2164/2164 [00:05<00:00, 364.08it/s]
Loading samples: 100%|############################################################| 2164/2164 [00:06<00:00, 358.61it/s]
Traceback (most recent call last):
  File "multiprocessing\queues.py", line 234, in _feed
Process Process-59:
  File "multiprocessing\reduction.py", line 51, in dumps
Process Process-72:
Traceback (most recent call last):
MemoryError
Traceback (most recent call last):
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 56, in process
sample_bgr = sample.load_bgr()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\Sample.py", line 112, in load_bgr
img = cv2_imread (self.filename, loader_func=self.read_raw_file).astype(np.float32) / 255.0
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 145, in process
img = get_eyes_mouth_mask()*mask
MemoryError: Unable to allocate 3.00 MiB for an array with shape (512, 512, 3) and data type float32
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 80, in get_eyes_mouth_mask
return np.clip(mask, 0, 1)

During handling of the above exception, another exception occurred:

  File "<__array_function__ internals>", line 6, in clip
Traceback (most recent call last):
Process Process-75:
Process Process-68:
Exception in thread Thread-90:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
generator._start()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\python-3.6.8\lib\site-packages\numpy\core\fromnumeric.py", line 2097, in clip
return _wrapfunc(a, 'clip', a_min, a_max, out=out, **kwargs)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\python-3.6.8\lib\site-packages\numpy\core\fromnumeric.py", line 58, in _wrapfunc
return bound(*args, **kwds)
Traceback (most recent call last):
Traceback (most recent call last):
Exception in thread Thread-92:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
generator._start()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe
  File "multiprocessing\process.py", line 258, in _bootstrap
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\python-3.6.8\lib\site-packages\numpy\core\_methods.py", line 141, in _clip
um.clip, a, min, max, out=out, casting=casting, **kwargs)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)

  File "multiprocessing\process.py", line 93, in run
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\python-3.6.8\lib\site-packages\numpy\core\_methods.py", line 94, in _clip_dep_invoke_with_casting
return ufunc(*args, out=out, **kwargs)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 139, in process
img = get_full_face_mask()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 145, in process
img = get_eyes_mouth_mask()*mask

  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 54, in process_func
gen_data = next (self.generator_func)
MemoryError: Unable to allocate 1.00 MiB for an array with shape (512, 512, 1) and data type float32
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 69, in get_full_face_mask
full_face_mask = LandmarksProcessor.get_image_hull_mask (sample_bgr.shape, sample_landmarks, eyebrows_expand_mod=sample.eyebrows_expand_mod )
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 79, in get_eyes_mouth_mask
mask = eyes_mask + mouth_mask
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 136, in batch_func
raise Exception ("Exception occured in sample %s. Error: %s" % (sample.filename, traceback.format_exc() ) )
Process Process-71:
Process Process-36:
Process Process-76:

During handling of the above exception, another exception occurred:

  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\facelib\LandmarksProcessor.py", line 391, in get_image_hull_mask
hull_mask = np.zeros(image_shape[0:2]+(1,),dtype=np.float32)
MemoryError: Unable to allocate 1.00 MiB for an array with shape (512, 512, 1) and data type float32
Exception: Exception occured in sample C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\workspace\data_src\aligned\xianchangsucai1_100PANA__00877_0.jpg. Error: Traceback (most recent call last):
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 56, in process
sample_bgr = sample.load_bgr()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\Sample.py", line 112, in load_bgr
img = cv2_imread (self.filename, loader_func=self.read_raw_file).astype(np.float32) / 255.0
MemoryError: Unable to allocate 3.00 MiB for an array with shape (512, 512, 3) and data type float32

Exception in thread Thread-73:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
generator._start()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
Process Process-37:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
MemoryError: Unable to allocate 1.00 MiB for an array with shape (512, 512, 1) and data type float32

During handling of the above exception, another exception occurred:

  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
Traceback (most recent call last):
  File "multiprocessing\process.py", line 258, in _bootstrap
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 143, in process
mask = get_full_face_mask().copy()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 192, in process
ct_sample_bgr = ct_sample.load_bgr()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
  File "multiprocessing\process.py", line 93, in run
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 192, in process
ct_sample_bgr = ct_sample.load_bgr()
Traceback (most recent call last):
  File "multiprocessing\process.py", line 258, in _bootstrap
Exception in thread Thread-104:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
generator._start()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe
MemoryError: Unable to allocate 1.00 MiB for an array with shape (512, 512, 1) and data type float32
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\Sample.py", line 112, in load_bgr
img = cv2_imread (self.filename, loader_func=self.read_raw_file).astype(np.float32) / 255.0
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 139, in process
img = get_full_face_mask()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 54, in process_func
gen_data = next (self.generator_func)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\Sample.py", line 112, in load_bgr
img = cv2_imread (self.filename, loader_func=self.read_raw_file).astype(np.float32) / 255.0
  File "multiprocessing\process.py", line 258, in _bootstrap
  File "multiprocessing\process.py", line 93, in run

During handling of the above exception, another exception occurred:

MemoryError: Unable to allocate 3.00 MiB for an array with shape (512, 512, 3) and data type float32
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 69, in get_full_face_mask
full_face_mask = LandmarksProcessor.get_image_hull_mask (sample_bgr.shape, sample_landmarks, eyebrows_expand_mod=sample.eyebrows_expand_mod )
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 136, in batch_func
raise Exception ("Exception occured in sample %s. Error: %s" % (sample.filename, traceback.format_exc() ) )
MemoryError: Unable to allocate 3.00 MiB for an array with shape (512, 512, 3) and data type float32

During handling of the above exception, another exception occurred:

  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 54, in process_func
gen_data = next (self.generator_func)
Exception in thread Thread-97:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
generator._start()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):

During handling of the above exception, another exception occurred:

  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\facelib\LandmarksProcessor.py", line 391, in get_image_hull_mask
hull_mask = np.zeros(image_shape[0:2]+(1,),dtype=np.float32)
Exception: Exception occured in sample C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\workspace\data_src\aligned\xianchangsucai1_102PANA__00215_0.jpg. Error: Traceback (most recent call last):
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 145, in process
img = get_eyes_mouth_mask()*mask
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 80, in get_eyes_mouth_mask
return np.clip(mask, 0, 1)
  File "<__array_function__ internals>", line 6, in clip
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\python-3.6.8\lib\site-packages\numpy\core\fromnumeric.py", line 2097, in clip
return _wrapfunc(a, 'clip', a_min, a_max, out=out, **kwargs)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\python-3.6.8\lib\site-packages\numpy\core\fromnumeric.py", line 58, in _wrapfunc
return bound(*args, **kwds)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\python-3.6.8\lib\site-packages\numpy\core\_methods.py", line 141, in _clip
um.clip, a, min, max, out=out, casting=casting, **kwargs)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\python-3.6.8\lib\site-packages\numpy\core\_methods.py", line 94, in _clip_dep_invoke_with_casting
return ufunc(*args, out=out, **kwargs)
MemoryError: Unable to allocate 1.00 MiB for an array with shape (512, 512, 1) and data type float32

  File "multiprocessing\process.py", line 93, in run
Traceback (most recent call last):
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 136, in batch_func
raise Exception ("Exception occured in sample %s. Error: %s" % (sample.filename, traceback.format_exc() ) )
Exception in thread Thread-95:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
generator._start()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe
  File "multiprocessing\process.py", line 258, in _bootstrap
Traceback (most recent call last):
MemoryError: Unable to allocate 1.00 MiB for an array with shape (512, 512, 1) and data type float32
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 54, in process_func
gen_data = next (self.generator_func)
  File "multiprocessing\process.py", line 258, in _bootstrap
Exception: Exception occured in sample C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\workspace\data_src\aligned\xianchangsucai1_100PANA__00455_0.jpg. Error: Traceback (most recent call last):
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 139, in process
img = get_full_face_mask()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 69, in get_full_face_mask
full_face_mask = LandmarksProcessor.get_image_hull_mask (sample_bgr.shape, sample_landmarks, eyebrows_expand_mod=sample.eyebrows_expand_mod )
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\facelib\LandmarksProcessor.py", line 391, in get_image_hull_mask
hull_mask = np.zeros(image_shape[0:2]+(1,),dtype=np.float32)
MemoryError: Unable to allocate 1.00 MiB for an array with shape (512, 512, 1) and data type float32

  File "multiprocessing\process.py", line 93, in run
  File "multiprocessing\process.py", line 258, in _bootstrap

During handling of the above exception, another exception occurred:

  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 136, in batch_func
raise Exception ("Exception occured in sample %s. Error: %s" % (sample.filename, traceback.format_exc() ) )
  File "multiprocessing\process.py", line 93, in run

  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 54, in process_func
gen_data = next (self.generator_func)
  File "multiprocessing\process.py", line 93, in run
Traceback (most recent call last):
Exception: Exception occured in sample C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\workspace\data_src\aligned\xianchangsucai1_102PANA__00143_0.jpg. Error: Traceback (most recent call last):
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 145, in process
img = get_eyes_mouth_mask()*mask
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 79, in get_eyes_mouth_mask
mask = eyes_mask + mouth_mask
MemoryError: Unable to allocate 1.00 MiB for an array with shape (512, 512, 1) and data type float32

  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 54, in process_func
gen_data = next (self.generator_func)
Exception in thread Thread-80:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
generator._start()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe

  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 54, in process_func
gen_data = next (self.generator_func)
  File "multiprocessing\process.py", line 258, in _bootstrap
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 136, in batch_func
raise Exception ("Exception occured in sample %s. Error: %s" % (sample.filename, traceback.format_exc() ) )
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 136, in batch_func
raise Exception ("Exception occured in sample %s. Error: %s" % (sample.filename, traceback.format_exc() ) )
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 136, in batch_func
raise Exception ("Exception occured in sample %s. Error: %s" % (sample.filename, traceback.format_exc() ) )
  File "multiprocessing\process.py", line 93, in run
Exception: Exception occured in sample C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\workspace\data_src\aligned\xianchangsucai1_102PANA__00186_0.jpg. Error: Traceback (most recent call last):
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 192, in process
ct_sample_bgr = ct_sample.load_bgr()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\Sample.py", line 112, in load_bgr
img = cv2_imread (self.filename, loader_func=self.read_raw_file).astype(np.float32) / 255.0
MemoryError: Unable to allocate 3.00 MiB for an array with shape (512, 512, 3) and data type float32

Exception: Exception occured in sample C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\workspace\data_src\aligned\xinshangsucai1_101PANA__00019_0.jpg. Error: Traceback (most recent call last):
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 143, in process
mask = get_full_face_mask().copy()
MemoryError: Unable to allocate 1.00 MiB for an array with shape (512, 512, 1) and data type float32

Exception: Exception occured in sample C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\workspace\data_src\aligned\xianchangsucai1_102PANA__00101_0.jpg. Error: Traceback (most recent call last):
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 192, in process
ct_sample_bgr = ct_sample.load_bgr()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\Sample.py", line 112, in load_bgr
img = cv2_imread (self.filename, loader_func=self.read_raw_file).astype(np.float32) / 255.0
MemoryError: Unable to allocate 3.00 MiB for an array with shape (512, 512, 3) and data type float32

Exception in thread Thread-96:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
generator._start()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 54, in process_func
gen_data = next (self.generator_func)

  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 136, in batch_func
raise Exception ("Exception occured in sample %s. Error: %s" % (sample.filename, traceback.format_exc() ) )
Exception in thread Thread-91:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
generator._start()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe
Exception: Exception occured in sample C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\workspace\data_src\aligned\xinshangsucai1_101PANA__00198_0.jpg. Error: Traceback (most recent call last):
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleGeneratorFace.py", line 134, in batch_func
x, = SampleProcessor.process ([sample], self.sample_process_options, self.output_sample_types, self.debug, ct_sample=ct_sample)
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 139, in process
img = get_full_face_mask()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\samplelib\SampleProcessor.py", line 69, in get_full_face_mask
full_face_mask = LandmarksProcessor.get_image_hull_mask (sample_bgr.shape, sample_landmarks, eyebrows_expand_mod=sample.eyebrows_expand_mod )
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\facelib\LandmarksProcessor.py", line 391, in get_image_hull_mask
hull_mask = np.zeros(image_shape[0:2]+(1,),dtype=np.float32)
MemoryError: Unable to allocate 1.00 MiB for an array with shape (512, 512, 1) and data type float32

Exception in thread Thread-82:
Traceback (most recent call last):
  File "threading.py", line 916, in _bootstrap_inner
  File "threading.py", line 864, in run
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 11, in launch_thread
generator._start()
  File "C:\zhang\DeepFaceLab_NVIDIA_RTX3000_new\_internal\DeepFaceLab\core\joblib\SubprocessGenerator.py", line 43, in _start
p.start()
  File "multiprocessing\process.py", line 105, in start
  File "multiprocessing\context.py", line 223, in _Popen
  File "multiprocessing\context.py", line 322, in _Popen
  File "multiprocessing\popen_spawn_win32.py", line 65, in __init__
  File "multiprocessing\reduction.py", line 60, in dump
BrokenPipeError: [Errno 32] Broken pipe