Conversion error - Out of system RAM

Getting errors or found a bug when converting faces from a trained model? Post about them here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Convert process. If you want to get tips, or better understand the Convert process, then you should look in the Convert Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Lepiloff
Posts: 10
Joined: Wed Jan 08, 2020 2:32 pm
Answers: 1
Has thanked: 6 times
Been thanked: 1 time

Conversion error - Out of system RAM

Post by Lepiloff »

After I running convertation I get an error

Code: Select all

/bin/sh: 1: ldconfig: not found
Traceback (most recent call last):
File "/home/evgendelaetml/miniconda3/envs/fs/lib/python3.6/site-packages/imageio_ffmpeg/_io.py", line 174, in read_frames
"End of file reached before full frame could be read."
RuntimeError: End of file reached before full frame could be read.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/evgendelaetml/faceswap/lib/cli.py", line 128, in execute_script
process.process()
File "/home/evgendelaetml/faceswap/scripts/convert.py", line 157, in process
self._convert_images()
File "/home/evgendelaetml/faceswap/scripts/convert.py", line 184, in _convert_images
self._check_thread_error()
File "/home/evgendelaetml/faceswap/scripts/convert.py", line 204, in _check_thread_error
thread.check_and_raise_error()
File "/home/evgendelaetml/faceswap/lib/multithreading.py", line 84, in check_and_raise_error
raise error[1].with_traceback(error[2])
File "/home/evgendelaetml/faceswap/lib/multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "/home/evgendelaetml/faceswap/scripts/convert.py", line 452, in _load
for filename, image in self._images.load():
File "/home/evgendelaetml/faceswap/scripts/fsmedia.py", line 247, in load
for filename, image in iterator():
File "/home/evgendelaetml/faceswap/scripts/fsmedia.py", line 280, in _load_video_frames
for i, frame in enumerate(reader):
File "/home/evgendelaetml/miniconda3/envs/fs/lib/python3.6/site-packages/imageio/core/format.py", line 399, in iter_data
im, meta = self._get_data(i)
File "/home/evgendelaetml/miniconda3/envs/fs/lib/python3.6/site-packages/imageio/plugins/ffmpeg.py", line 384, in _get_data
result, is_new = self._read_frame()
File "/home/evgendelaetml/miniconda3/envs/fs/lib/python3.6/site-packages/imageio/plugins/ffmpeg.py", line 486, in _read_frame
s = self._read_gen.__next__()
File "/home/evgendelaetml/miniconda3/envs/fs/lib/python3.6/site-packages/imageio_ffmpeg/_io.py", line 182, in read_frames
raise RuntimeError(fmt.format(framenr, err1, err2))
RuntimeError: Could not read frame 449:
End of file reached before full frame could be read.
=== stderr ===
ffmpeg version 4.2 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 7.3.0 (crosstool-NG 1.23.0.449-a04d0)
configuration: --prefix=/home/evgendelaetml/miniconda3/envs/fs --cc=/home/conda/feedstock_root/build_artifacts/ffmpeg_1566210161358/_build_env/bin/x86_64-conda_cos6-linux-gnu-cc --disable-doc --disable-openssl --enable-avresample --enable-gnutls --enable-gpl --enable-hardcoded-tables --enable-libfreetype --enable-libopenh264 --enable-libx264 --enable-pic --enable-pthreads --enable-shared --enable-static --enable-version3 --enable-zlib --enable-libmp3lame
libavutil      56. 31.100 / 56. 31.100
libavcodec     58. 54.100 / 58. 54.100
libavformat    58. 29.100 / 58. 29.100
libavdevice    58.  8.100 / 58.  8.100
libavfilter     7. 57.100 /  7. 57.100
libavresample   4.  0.  0 /  4.  0.  0
libswscale      5.  5.100 /  5.  5.100
libswresample   3.  5.100 /  3.  5.100
libpostproc    55.  5.100 / 55.  5.100
Input #0, matroska,webm, from '/home/evgendelaetml/t24k.webm':
Metadata:
encoder         : google/video-file
Duration: 00:05:16.97, start: 0.000000, bitrate: 6413 kb/s
Stream #0:0(eng): Video: vp9 (Profile 0), yuv420p(tv, bt709), 3840x2160, SAR 1:1 DAR 16:9, 30 fps, 30 tbr, 1k tbn, 1k tbc (default)
Stream mapping:
Stream #0:0 -> #0:0 (vp9 (native) -> rawvideo (native))
Press [q] to stop, [?] for help
Output #0, image2pipe, to 'pipe:':
Metadata:
encoder         : Lavf58.29.100
Stream #0:0(eng): Video: rawvideo (RGB[24] / 0x18424752), rgb24, 3840x2160 [SAR 1:1 DAR 16:9], q=2-31, 5971968 kb/s, 30 fps, 30 tbn, 30 tbc (default)
Metadata:
... showing only last few lines ...
frame=  340 fps=4.5 q=-0.0 size= 8262000kB time=00:00:11.33 bitrate=5971968.2kbits/s speed=0.151x
frame=  343 fps=4.5 q=-0.0 size= 8334900kB time=00:00:11.43 bitrate=5971968.2kbits/s speed=0.151x
frame=  346 fps=4.5 q=-0.0 size= 8407800kB time=00:00:11.53 bitrate=5971968.2kbits/s speed=0.151x
frame=  349 fps=4.5 q=-0.0 size= 8480700kB time=00:00:11.63 bitrate=5971968.2kbits/s speed=0.151x
frame=  352 fps=4.5 q=-0.0 size= 8553600kB time=00:00:11.73 bitrate=5971968.2kbits/s speed=0.151x
frame=  356 fps=4.5 q=-0.0 size= 8650800kB time=00:00:11.86 bitrate=5971967.8kbits/s speed=0.151x
frame=  358 fps=4.5 q=-0.0 size= 8699400kB time=00:00:11.93 bitrate=5971968.2kbits/s speed=0.151x
frame=  361 fps=4.5 q=-0.0 size= 8772300kB time=00:00:12.03 bitrate=5971968.2kbits/s speed=0.151x
frame=  364 fps=4.5 q=-0.0 size= 8845200kB time=00:00:12.13 bitrate=5971968.2kbits/s speed=0.151x
frame=  368 fps=4.5 q=-0.0 size= 8942400kB time=00:00:12.26 bitrate=5971967.8kbits/s speed=0.151x
frame=  371 fps=4.5 q=-0.0 size= 9015300kB time=00:00:12.36 bitrate=5971967.8kbits/s speed=0.152x
frame=  374 fps=4.5 q=-0.0 size= 9088200kB time=00:00:12.46 bitrate=5971967.8kbits/s speed=0.151x
frame=  378 fps=4.6 q=-0.0 size= 9185400kB time=00:00:12.60 bitrate=5971968.0kbits/s speed=0.152x
frame=  379 fps=4.5 q=-0.0 size= 9209700kB time=00:00:12.63 bitrate=5971968.2kbits/s speed=0.151x
frame=  380 fps=4.5 q=-0.0 size= 9234000kB time=00:00:12.66 bitrate=5971967.8kbits/s speed=0.15x
frame=  383 fps=4.5 q=-0.0 size= 9306900kB time=00:00:12.76 bitrate=5971967.8kbits/s speed=0.15x
frame=  385 fps=4.5 q=-0.0 size= 9355500kB time=00:00:12.83 bitrate=5971968.2kbits/s speed=0.149x
frame=  389 fps=4.5 q=-0.0 size= 9452700kB time=00:00:12.96 bitrate=5971967.8kbits/s speed=0.15x
frame=  390 fps=4.4 q=-0.0 size= 9477000kB time=00:00:13.00 bitrate=5971968.0kbits/s speed=0.148x
frame=  393 fps=4.4 q=-0.0 size= 9549900kB time=00:00:13.10 bitrate=5971968.0kbits/s speed=0.148x
frame=  395 fps=4.4 q=-0.0 size= 9598500kB time=00:00:13.16 bitrate=5971967.8kbits/s speed=0.148x
frame=  396 fps=4.4 q=-0.0 size= 9622800kB time=00:00:13.20 bitrate=5971968.0kbits/s speed=0.147x
frame=  400 fps=4.4 q=-0.0 size= 9720000kB time=00:00:13.33 bitrate=5971968.1kbits/s speed=0.148x
frame=  401 fps=4.4 q=-0.0 size= 9744300kB time=00:00:13.36 bitrate=5971967.9kbits/s speed=0.147x
frame=  404 fps=4.4 q=-0.0 size= 9817200kB time=00:00:13.46 bitrate=5971967.9kbits/s speed=0.147x
frame=  407 fps=4.4 q=-0.0 size= 9890100kB time=00:00:13.56 bitrate=5971967.9kbits/s speed=0.147x
frame=  409 fps=4.4 q=-0.0 size= 9938700kB time=00:00:13.63 bitrate=5971968.1kbits/s speed=0.147x
frame=  412 fps=4.4 q=-0.0 size=10011600kB time=00:00:13.73 bitrate=5971968.1kbits/s speed=0.146x
frame=  413 fps=4.3 q=-0.0 size=10035900kB time=00:00:13.76 bitrate=5971967.9kbits/s speed=0.144x
frame=  416 fps=4.2 q=-0.0 size=10108800kB time=00:00:13.86 bitrate=5971967.9kbits/s speed=0.141x
frame=  417 fps=4.2 q=-0.0 size=10133100kB time=00:00:13.90 bitrate=5971968.0kbits/s speed=0.141x
frame=  419 fps=4.2 q=-0.0 size=10181700kB time=00:00:13.96 bitrate=5971967.9kbits/s speed=0.138x
frame=  420 fps=4.1 q=-0.0 size=10206000kB time=00:00:14.00 bitrate=5971968.0kbits/s speed=0.136x
frame=  423 fps=4.1 q=-0.0 size=10278900kB time=00:00:14.10 bitrate=5971968.0kbits/s speed=0.135x
frame=  424 fps=4.1 q=-0.0 size=10303200kB time=00:00:14.13 bitrate=5971968.1kbits/s speed=0.135x
frame=  426 fps=4.0 q=-0.0 size=10351800kB time=00:00:14.20 bitrate=5971968.0kbits/s speed=0.132x
frame=  427 fps=3.9 q=-0.0 size=10376100kB time=00:00:14.23 bitrate=5971968.1kbits/s speed=0.131x
frame=  429 fps=3.9 q=-0.0 size=10424700kB time=00:00:14.30 bitrate=5971968.0kbits/s speed=0.131x
frame=  431 fps=3.9 q=-0.0 size=10473300kB time=00:00:14.36 bitrate=5971967.9kbits/s speed=0.131x
frame=  432 fps=3.8 q=-0.0 size=10497600kB time=00:00:14.40 bitrate=5971968.0kbits/s speed=0.127x
frame=  433 fps=3.8 q=-0.0 size=10521900kB time=00:00:14.43 bitrate=5971968.1kbits/s speed=0.127x
frame=  436 fps=3.7 q=-0.0 size=10594800kB time=00:00:14.53 bitrate=5971968.1kbits/s speed=0.124x
frame=  438 fps=3.7 q=-0.0 size=10643400kB time=00:00:14.60 bitrate=5971968.0kbits/s speed=0.124x
frame=  440 fps=3.7 q=-0.0 size=10692000kB time=00:00:14.66 bitrate=5971967.9kbits/s speed=0.123x
frame=  441 fps=3.6 q=-0.0 size=10716300kB time=00:00:14.70 bitrate=5971968.0kbits/s speed=0.119x
frame=  442 fps=3.5 q=-0.0 size=10740600kB time=00:00:14.73 bitrate=5971968.1kbits/s speed=0.118x
frame=  442 fps=3.5 q=-0.0 size=10740600kB time=00:00:14.73 bitrate=5971968.1kbits/s speed=0.116x
frame=  445 fps=3.4 q=-0.0 size=10813500kB time=00:00:14.83 bitrate=5971968.1kbits/s speed=0.112x
frame=  446 fps=3.4 q=-0.0 size=10837800kB time=00:00:14.86 bitrate=5971967.9kbits/s speed=0.112x
frame=  448 fps=3.3 q=-0.0 size=10886400kB time=00:00:14.93 bitrate=5971968.1kbits/s speed=0.11x
01/14/2020 08:30:08 CRITICAL An unexpected crash has occurred. Crash report written to '/home/evgendelaetml/faceswap/crash_report.2020.01.14.083008216270.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting

terminate called without an active exception
Process exited.

Would you help me to solve this error?

by Lepiloff » Wed Jan 15, 2020 8:15 am

I found that cause of error is allocate memory. The conversion process requires up to 14 G of RAM memory.

Go to full post
User avatar
Lepiloff
Posts: 10
Joined: Wed Jan 08, 2020 2:32 pm
Answers: 1
Has thanked: 6 times
Been thanked: 1 time

Re: Convertaion error

Post by Lepiloff »

I try to retrain a new model, and after start converting got the same error.

User avatar
Lepiloff
Posts: 10
Joined: Wed Jan 08, 2020 2:32 pm
Answers: 1
Has thanked: 6 times
Been thanked: 1 time

Re: Convertaion error

Post by Lepiloff »

I found that cause of error is allocate memory. The conversion process requires up to 14 G of RAM memory.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Convertaion error

Post by torzdf »

14GB of system memory? That doesn't sound right to me.

My word is final

User avatar
Lepiloff
Posts: 10
Joined: Wed Jan 08, 2020 2:32 pm
Answers: 1
Has thanked: 6 times
Been thanked: 1 time

Re: Convertaion error

Post by Lepiloff »

Exacly, it sound strange but I measured the memory consumption with a command

Code: Select all

 free -m

and during conversion process memory usage increased from 600M to 13.5G.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: Convertaion error

Post by torzdf »

Ahh, that may be reserved RAM, rather than explicitly in use. Are you looking at the free column or the available column?

The total ram less available will give you a better indication of the amount of RAM actually being used.

My word is final

User avatar
Lepiloff
Posts: 10
Joined: Wed Jan 08, 2020 2:32 pm
Answers: 1
Has thanked: 6 times
Been thanked: 1 time

Re: Convertaion error

Post by Lepiloff »

Its look like

Code: Select all

evgendelaetml@instance-2:~/faceswap$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15043        5624        5192        2165        4226        6977
Swap:             0           0           0
evgendelaetml@instance-2:~/faceswap$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15043        6467        4336        2165        4239        6133
Swap:             0           0           0
evgendelaetml@instance-2:~/faceswap$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15043        7016        3780        2165        4247        5585
Swap:             0           0           0
evgendelaetml@instance-2:~/faceswap$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15043        8032        2750        2165        4260        4568
Swap:             0           0           0
evgendelaetml@instance-2:~/faceswap$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15043        8738        2031        2165        4272        3861
Swap:             0           0           0
evgendelaetml@instance-2:~/faceswap$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15043        9834         919        2165        4289        2765
Swap:             0           0           0
evgendelaetml@instance-2:~/faceswap$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15043       10373         368        2165        4301        2226
Swap:             0           0           0
evgendelaetml@instance-2:~/faceswap$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15043       10513         220        2165        4309        2086
Swap:             0           0           0
evgendelaetml@instance-2:~/faceswap$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15043       10444         643        2165        3955        2167
Swap:             0           0           0
evgendelaetml@instance-2:~/faceswap$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15043       11840         485        2165        2717         775
Swap:             0           0           0
evgendelaetml@instance-2:~/faceswap$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15043       12148         587        2165        2307         497
Swap:             0           0           0
evgendelaetml@instance-2:~/faceswap$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15043       12223         580        2165        2239         456
Swap:             0           0           0
evgendelaetml@instance-2:~/faceswap$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15043         679       14208          61         155       14093
Swap:             0           0           0

All converts setting is default
Last string of log file:

Code: Select all

Exception occured trying to retrieve sysinfo: [Errno 12] Cannot allocate memory
Locked