numpy.core._exceptions._ArrayMemoryError

[SimonDK]

Hi,
I am trying to extract faces from a folder of images, they are 4160x6240 pixel JPGs, so rather large (around 5 megabytes a piece). I have tried reencoding them to 80% jpg, and now they are around 1.10 megabytes to 1.5 megabytes.,

I am getting this:

Code: Select all

Loading...
Setting Faceswap backend to NVIDIA
12/24/2022 07:27:52 INFO     Log level set to: INFO
12/24/2022 07:27:54 INFO     Loading Detect from S3Fd plugin...
12/24/2022 07:27:54 INFO     Loading Align from Fan plugin...
12/24/2022 07:27:54 INFO     Loading Mask from Components plugin...
12/24/2022 07:27:54 INFO     Loading Mask from Extended plugin...
12/24/2022 07:27:54 INFO     Loading Mask from Unet_Dfl plugin...
12/24/2022 07:27:54 INFO     Loading Mask from Vgg_Obstructed plugin...
12/24/2022 07:27:54 INFO     Loading Mask from Custom plugin...
12/24/2022 07:27:54 INFO     Loading Mask from Bisenet_Fp plugin...
12/24/2022 07:27:54 INFO     Loading Mask from Vgg_Clear plugin...
12/24/2022 07:27:54 INFO     Starting, this may take a while...
12/24/2022 07:27:54 INFO     Output Directory: D:\FS\training\Michelle\Session1
12/24/2022 07:27:54 INFO     Initializing S3FD (Detect)...
12/24/2022 07:27:55 INFO     Initialized S3FD (Detect) with batchsize of 1
12/24/2022 07:27:55 INFO     Initializing FAN (Align)...
12/24/2022 07:28:03 INFO     Initialized FAN (Align) with batchsize of 1
12/24/2022 07:28:03 INFO     Initializing Components (Mask)...
12/24/2022 07:28:03 INFO     Initialized Components (Mask) with batchsize of 1
12/24/2022 07:28:03 INFO     Initializing Extended (Mask)...
12/24/2022 07:28:03 INFO     Initialized Extended (Mask) with batchsize of 1
12/24/2022 07:28:03 INFO     Initializing U-Net (Mask)...
12/24/2022 07:28:04 INFO     Initialized U-Net (Mask) with batchsize of 1
12/24/2022 07:28:04 INFO     Initializing VGG Obstructed (Mask)...
12/24/2022 07:28:06 INFO     Initialized VGG Obstructed (Mask) with batchsize of 1
12/24/2022 07:28:06 INFO     Initializing Custom (Mask)...
12/24/2022 07:28:06 INFO     Initialized Custom (Mask) with batchsize of 1
12/24/2022 07:28:06 INFO     Initializing BiSeNet - Face Parsing (Mask)...
12/24/2022 07:28:08 INFO     Initialized BiSeNet - Face Parsing (Mask) with batchsize of 1
12/24/2022 07:28:08 INFO     Initializing VGG Clear (Mask)...
12/24/2022 07:28:12 ERROR    Got Exception on main handler:
Traceback (most recent call last):
  File "C:\Users\Simon\faceswap\lib\cli\launcher.py", line 217, in execute_script
    process.process()
  File "C:\Users\Simon\faceswap\scripts\extract.py", line 165, in process
    extract.process()
  File "C:\Users\Simon\faceswap\scripts\extract.py", line 689, in process
    self._run_extraction()
  File "C:\Users\Simon\faceswap\scripts\extract.py", line 709, in _run_extraction
    self._extractor.launch()
  File "C:\Users\Simon\faceswap\plugins\extract\pipeline.py", line 272, in launch
    self._launch_plugin(phase)
  File "C:\Users\Simon\faceswap\plugins\extract\pipeline.py", line 701, in _launch_plugin
    plugin.initialize(**kwargs)
  File "C:\Users\Simon\faceswap\plugins\extract\_base.py", line 482, in initialize
    self.init_model()
  File "C:\Users\Simon\faceswap\plugins\extract\mask\vgg_clear.py", line 43, in init_model
    self.model = VGGClear(self.model_path,
  File "C:\Users\Simon\faceswap\plugins\extract\mask\vgg_clear.py", line 107, in __init__
    self.load_model_weights()
  File "C:\Users\Simon\faceswap\lib\model\session.py", line 235, in load_model_weights
    self._model.load_weights(self._model_path)
  File "C:\Users\Simon\MiniConda3\envs\faceswap\lib\site-packages\keras\utils\traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "C:\Users\Simon\MiniConda3\envs\faceswap\lib\site-packages\h5py\_hl\dataset.py", line 1040, in __array__
    arr = numpy.zeros(self.shape, dtype=self.dtype if dtype is None else dtype)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 392. MiB for an array with shape (7, 7, 512, 4096) and data type float32

I am trying to avoid resizing the images to something smaller as the actual facepart of them is not that much, also trying to avoid having to manually cut the face from each image as this folder has 760 and there are more folders
I am running a RTX 4090 with 24GB of RAM. System was a fresh reboot running on 32GB of RAM with at least 26GB free at program start.
I tried setting all the batchsizes that had anything to do with extract to 1 to minimize the load.

Any help is greatly appreciated!

Merry x-mas (of cause not expecting any answers before after the holidays, but I wanted to get it posted)!
Simon

[torzdf]

You shouldn't be running out of RAM on that setup (to be clear, it is system RAM that is the issue here, not VRAM).

Can you open up task manager, switch to "Performance" and monitor the Memory graph when running. A screengrab may also be useful.

[SimonDK]

Hi,
Thanks for getting back to me.
Here is before pushing the start button:

pre.JPG (91.25 KiB) Viewed 2184 times

Here is after crashing (instantly):

post.JPG (85.38 KiB) Viewed 2184 times

And here is pythons usage:

python.JPG (23.55 KiB) Viewed 2184 times

I worked around the issue by writing a program to cut the image files in to half size pictures 5 different ways to make sure I get a full face from each. faceswap processed those without problems (running with batch set to lowest on each setting where I could change it).
So the issue is no longer critical to me, but I find it a bit strange if it is system memory that I am running out of!

Best regards,
Simon

[SimonDK]

Hi again,
I had accidently paused the updates on the other view, but here is a fresh one, as you can see there are no spikes that come anywhere near my memory limit.

faceswap mem.JPG (82.54 KiB) Viewed 2179 times

Best regards,
Simon

[torzdf]

Ok, not sure what is going on then. There is no leak, so my next suggestion would be that you have some bad RAM somewhere.

The object it is erroring on is only c.392MB and there is clearly plenty of headroom available

[SimonDK]

Thanks for getting back to me, I'll schedule a memtest in the near future just to be sure. But it is strange that it only happens while extracting faces from large images.
Regards,
Simon

[SimonDK]

Hi,

I ran multiple RAM tests (memtest86 and the build in windows one), no problems.

But I finally got this figured out, it is a pagefile.sys thing. My old pagefile was set to 1024MB-4096MB, I had configured it kind of small on purpose since I have 32GB RAM and not very often expected to hit that.
Now what happens when I run faceswap seems to be the python process very actively using the pagefile instead of the free system RAM, I guess it is Win10 handling it, I am not sure why it would do this instead of using free RAM first.
I have since set the MAX size of my pagefile to 32GB and can now run the large images folder. The python process seems to be using 12-15GB of RAM and the pagefile is hovering around 30GB (!) - I am running chrome with around 20tabs open in the background this time (didn't when the problem originally occured), just to see if I can get the error (I can't).
Pagefile shrunk back to 15GB right after the process finished.
I think I will be setting the pagefile to 64GB MAX instead, just to be on the safe side.

Just wanted to leave this info here for anyone running in to similar problems

Regards,
Simon

[torzdf]

Thanks for reporting back. This very useful information!