The recovered images are quite blurred

Want to understand the training process better? Got tips for which model to use and when? This is the place for you


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for discussing tips and understanding the process involved with Training a Faceswap model.

If you have found a bug are having issues with the Training process not working, then you should post in the Training Support forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Kevin_Z
Posts: 11
Joined: Tue Apr 28, 2020 2:36 am
Has thanked: 1 time
Been thanked: 1 time

The recovered images are quite blurred

Post by Kevin_Z »

I have trained the model for 3000+ iterations. The loss of both are about 0.05. And the images are quite blurred. I want to know whats wrong. THX

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: The recovered images are quite blurred

Post by torzdf »

3000 iteration is a very small amount

app.php/faqpage#f3r1

My word is final

User avatar
Kevin_Z
Posts: 11
Joined: Tue Apr 28, 2020 2:36 am
Has thanked: 1 time
Been thanked: 1 time

Re: The recovered images are quite blurred

Post by Kevin_Z »

so what is a reasonable iterations? And any other suggestion? THX!

User avatar
Kevin_Z
Posts: 11
Joined: Tue Apr 28, 2020 2:36 am
Has thanked: 1 time
Been thanked: 1 time

Re: The recovered images are quite blurred

Post by Kevin_Z »

What is the suitable model with 2080ti? And the extraction resoluion? Numbers of images ? THX!

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: The recovered images are quite blurred

Post by torzdf »

This is all covered in the training guide and faq, so you're best off starting there:
viewtopic.php?f=6&t=146

The number of iterations will depend on the model. For 2080Ti use Villain or Dlight

How long did it take you to get to 3k iterations?

My word is final

User avatar
Kevin_Z
Posts: 11
Joined: Tue Apr 28, 2020 2:36 am
Has thanked: 1 time
Been thanked: 1 time

Re: The recovered images are quite blurred

Post by Kevin_Z »

Right now I'm running on 1650 one, and it took 12 hours more or less, using Dfaker model and extraction resolution is 128x128. Data size is about 5k for each. Now I'm going to try on 2080ti one, so I ask some suggestion to guarantee better performance.

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: The recovered images are quite blurred

Post by torzdf »

12 hours to reach 3000 iterations is waaaaay too slow and sounds like you are using CPU.

Please can you go Help > Output System Information and paste the output here,

Also, I suggest just extracting at default (256px)

My word is final

User avatar
Kevin_Z
Posts: 11
Joined: Tue Apr 28, 2020 2:36 am
Has thanked: 1 time
Been thanked: 1 time

Re: The recovered images are quite blurred

Post by Kevin_Z »

It's the time that I run it on 1650, and 4GB ram is all full, I checked.

Code: Select all

encoding:            cp936
git_branch:          master
git_commits:         d5f42b6 Bugfix: lib.gui.project - Reset invalid
Thank you for your reply, I was in the correct folder, although it refused to work. I did the old restart PC "fix" and it worked. Very odd.
 choices to default if an invalid choice is discovered when loading a .fsw file. cbba53e Core Updates (#1015). 2b66013 Bugfix: logger - Still output crash report if system information fails to load. 3f0a016 bugfix: Extract - raise error if Keras fails to import. ff8d851 Cli Restructure + Multi-Mask Select on Extract (#1012)
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce GTX 1650
gpu_devices_active:  GPU_0
gpu_driver:          441.08
gpu_vram:            GPU_0: 4096MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.17763-SP0
os_release:          10
py_command:          faceswap.py gui
py_conda_version:    conda 4.8.2
py_implementation:   CPython
py_version:          3.7.7
py_virtual_env:      True
sys_cores:           6
sys_processor:       Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
sys_ram:             Total: 16314MB, Available: 6572MB, Used: 9741MB, Free: 6572MB

=============== Pip Packages ===============
absl-py==0.9.0
asn1crypto==1.3.0
astor==0.8.0
blinker==1.4
cachetools==3.1.1
certifi==2020.4.5.1
cffi==1.14.0
chardet==3.0.4
click==7.1.1
cloudpickle==1.3.0
cryptography==2.8
cycler==0.10.0
cytoolz==0.10.1
dask==2.14.0
decorator==4.4.2
fastcluster==1.1.26
ffmpy==0.2.2
gast==0.2.2
google-auth==1.13.1
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.27.2
h5py==2.9.0
idna==2.9
imageio==2.6.1
imageio-ffmpeg==0.4.1
joblib==0.14.1
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
Markdown==3.1.1
matplotlib==3.1.3
mkl-fft==1.0.15
mkl-random==1.1.0
mkl-service==2.3.0
networkx==2.4
numpy==1.17.4
nvidia-ml-py3==7.352.1
oauthlib==3.1.0
olefile==0.46
opencv-python==4.1.2.30
opt-einsum==3.1.0
pathlib==1.0.1
Pillow==6.2.1
protobuf==3.11.4
psutil==5.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.7
pycparser==2.20
PyJWT==1.7.1
pyOpenSSL==19.1.0
pyparsing==2.4.6
pyreadline==2.1
PySocks==1.7.1
python-dateutil==2.8.1
pytz==2019.3
PyWavelets==1.1.1
pywin32==227
PyYAML==5.3.1
requests==2.23.0
requests-oauthlib==1.3.0
rsa==4.0
scikit-image==0.16.2
scikit-learn==0.22.1
scipy==1.4.1
six==1.14.0
tensorboard==2.1.0
tensorflow==1.15.0
tensorflow-estimator==1.15.1
termcolor==1.1.0
toolz==0.10.0
toposort==1.5
tornado==6.0.4
tqdm==4.45.0
urllib3==1.25.8
Werkzeug==0.16.1
win-inet-pton==1.1.0
wincertstore==0.2
wrapt==1.12.1

============== Conda Packages ==============
# packages in environment at C:\Users\EDZ\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
_tflow_select             2.1.0                       gpu  
absl-py 0.9.0 py37_0
asn1crypto 1.3.0 py37_0
astor 0.8.0 py37_0
blas 1.0 mkl
blinker 1.4 py37_0
ca-certificates 2020.1.1 0
cachetools 3.1.1 py_0
certifi 2020.4.5.1 py37_0
cffi 1.14.0 py37h7a1dbc1_0
chardet 3.0.4 py37_1003
click 7.1.1 py_0
cloudpickle 1.3.0 py_0
cryptography 2.8 py37h7a1dbc1_0
cudatoolkit 10.0.130 0
cudnn 7.6.5 cuda10.0_0
cycler 0.10.0 py37_0
cytoolz 0.10.1 py37he774522_0
dask-core 2.14.0 py_0
decorator 4.4.2 py_0
fastcluster 1.1.26 py37he350917_0 conda-forge ffmpeg 4.2 h6538335_0 conda-forge ffmpy 0.2.2 pypi_0 pypi freetype 2.9.1 ha9979f8_1
gast 0.2.2 py37_0
git 2.23.0 h6bb4b03_0
google-auth 1.13.1 py_0
google-auth-oauthlib 0.4.1 py_2
google-pasta 0.2.0 py_0
grpcio 1.27.2 py37h351948d_0
h5py 2.9.0 py37h5e291fa_0
hdf5 1.10.4 h7ebc959_0
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha66f8fd_1
idna 2.9 py_1
imageio 2.6.1 py37_0
imageio-ffmpeg 0.4.1 py_0 conda-forge intel-openmp 2020.0 166
joblib 0.14.1 py_0
jpeg 9b hb83a4c4_2
keras 2.2.4 0
keras-applications 1.0.8 py_0
keras-base 2.2.4 py37_0
keras-preprocessing 1.1.0 py_1
kiwisolver 1.1.0 py37ha925a31_0
libpng 1.6.37 h2a8f88b_0
libprotobuf 3.11.4 h7bd577a_0
libtiff 4.1.0 h56a325e_0
markdown 3.1.1 py37_0
matplotlib 3.1.1 py37hc8f65d3_0
matplotlib-base 3.1.3 py37h64f37c6_0
mkl 2020.0 166
mkl-service 2.3.0 py37hb782905_0
mkl_fft 1.0.15 py37h14836fe_0
mkl_random 1.1.0 py37h675688f_0
networkx 2.4 py_0
numpy 1.17.4 py37h4320e6b_0
numpy-base 1.17.4 py37hc3f5095_0
nvidia-ml-py3 7.352.1 pypi_0 pypi oauthlib 3.1.0 py_0
olefile 0.46 py37_0
opencv-python 4.1.2.30 pypi_0 pypi openssl 1.1.1g he774522_0
opt_einsum 3.1.0 py_0
pathlib 1.0.1 py37_1
pillow 6.2.1 py37hdc69c19_0
pip 20.0.2 py37_1
protobuf 3.11.4 py37h33f27b4_0
psutil 5.7.0 py37he774522_0
pyasn1 0.4.8 py_0
pyasn1-modules 0.2.7 py_0
pycparser 2.20 py_0
pyjwt 1.7.1 py37_0
pyopenssl 19.1.0 py37_0
pyparsing 2.4.6 py_0
pyqt 5.9.2 py37h6538335_2
pyreadline 2.1 py37_1
pysocks 1.7.1 py37_0
python 3.7.7 h60c2a47_2
python-dateutil 2.8.1 py_0
python_abi 3.7 1_cp37m conda-forge pytz 2019.3 py_0
pywavelets 1.1.1 py37he774522_0
pywin32 227 py37he774522_1
pyyaml 5.3.1 py37he774522_0
qt 5.9.7 vc14h73c81de_0
requests 2.23.0 py37_0
requests-oauthlib 1.3.0 py_0
rsa 4.0 py_0
scikit-image 0.16.2 py37h47e9c7a_0
scikit-learn 0.22.1 py37h6288b17_0
scipy 1.4.1 py37h9439919_0
setuptools 46.1.3 py37_0
sip 4.19.8 py37h6538335_0
six 1.14.0 py37_0
sqlite 3.31.1 h2a8f88b_1
tensorboard 2.1.0 py3_0
tensorflow 1.15.0 gpu_py37hc3743a6_0
tensorflow-base 1.15.0 gpu_py37h1afeea4_0
tensorflow-estimator 1.15.1 pyh2649769_0
tensorflow-gpu 1.15.0 h0d30ee6_0
termcolor 1.1.0 py37_1
tk 8.6.8 hfa6e2cd_0
toolz 0.10.0 py_0
toposort 1.5 py_3 conda-forge tornado 6.0.4 py37he774522_1
tqdm 4.45.0 py_0
urllib3 1.25.8 py37_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.16.27012 hf0eaf9b_1
werkzeug 0.16.1 py_0
wheel 0.34.2 py37_0
win_inet_pton 1.1.0 py37_0
wincertstore 0.2 py37_0
wrapt 1.12.1 py37he774522_1
xz 5.2.5 h62dcd97_0
yaml 0.1.7 hc54c509_2
zlib 1.2.11 h62dcd97_4
zstd 1.3.7 h508b16e_0 ================= Configs ================== --------- .faceswap --------- backend: nvidia --------- convert.ini --------- [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [color.match_hist] threshold: 99.0 [mask.box_blend] type: gaussian distance: 11.0 radius: 5.0 passes: 1 [mask.mask_blend] type: normalized kernel_size: 3 passes: 4 threshold: 4 erosion: 0.0 [scaling.sharpen] method: unsharp_mask amount: 150 radius: 0.3 threshold: 5.0 [writer.ffmpeg] container: mp4 codec: libx264 crf: 23 preset: medium tune: none profile: auto level: auto [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.opencv] format: png draw_transparent: False jpg_quality: 75 png_compress_level: 3 [writer.pillow] format: png draw_transparent: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate --------- extract.ini --------- [global] allow_growth: True [align.fan] batch-size: 12 [detect.cv2_dnn] confidence: 50 [detect.mtcnn] minsize: 20 threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 scalefactor: 0.709 batch-size: 8 [detect.s3fd] confidence: 70 batch-size: 4 [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 2 --------- gui.ini --------- [global] fullscreen: False tab: extract options_panel_width: 30 console_panel_height: 20 icon_size: 14 font: default font_size: 9 autosave_last_session: prompt timeout: 120 auto_load_model_stats: True --------- train.ini --------- [global] coverage: 62.5 mask_type: none mask_blur_kernel: 0 mask_threshold: 4 learn_mask: False icnr_init: False conv_aware_init: False subpixel_upscaling: False reflect_padding: False penalized_mask_loss: False loss_function: mae learning_rate: 5e-05 [model.dfl_h128] lowmem: False [model.dfl_sae] input_size: 128 clipnorm: True architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False [model.dlight] features: best details: good output_size: 256 [model.original] lowmem: False [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.unbalanced] input_size: 128 lowmem: False clipnorm: True nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.villain] lowmem: False [trainer.original] preview_images: 14 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4
User avatar
Kevin_Z
Posts: 11
Joined: Tue Apr 28, 2020 2:36 am
Has thanked: 1 time
Been thanked: 1 time

Re: The recovered images are quite blurred

Post by Kevin_Z »

BTW, I have 4k training images for each one, is that enough?

User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 623 times

Re: The recovered images are quite blurred

Post by torzdf »

Ok, that looks fine. 4k should be fine

My word is final

User avatar
Kevin_Z
Posts: 11
Joined: Tue Apr 28, 2020 2:36 am
Has thanked: 1 time
Been thanked: 1 time

Re: The recovered images are quite blurred

Post by Kevin_Z »

I have trained for 20K iterations, and recovered images are quite better, the loss for each is about 0.039. I want to it would got better if I train longer, and when to stop ? Depends on loss?

User avatar
djandg
Posts: 43
Joined: Mon Dec 09, 2019 7:00 pm
Has thanked: 4 times
Been thanked: 2 times

Re: The recovered images are quite blurred

Post by djandg »

I typically run Realface or Villain and 300K iterations is generally minimum I do, even for a simple swap.
I've run one particularly difficult swap, due to lots of face movements, lighting changes and size changes, up to 1million !
If the face to swap is static. like a newsreader, you can get away with lower iterations. Loss on training is realatively meaningleess as it depends on the swap you want to do, a straight on face or action movie.

2 x 1070ti's, keeps tthe shack toasty when it's cold :)

An example, Facwswap Russell Crowe to Rylan Clark-Neal (A UK TV celeb) in the Gladiator. Two scenes with close up face and large change in lighting one side of face to other return only small improvements with more training. 450k, 600k and 1 million. Further away and 600k is acceptable. Click thumbnail for larger size.

Image

Last edited by djandg on Fri May 01, 2020 2:33 pm, edited 1 time in total.
User avatar
Grassone
Posts: 45
Joined: Sun Apr 19, 2020 7:32 pm
Has thanked: 6 times
Been thanked: 2 times

Re: The recovered images are quite blurred

Post by Grassone »

My recent experience:

Dlight: batches of 4 (sorry... little Gpu-RAM on my GTX1050) Almost 520K iterations and going. Almost 2K faces per set.

The eyes are just beginning to look realistic.... and looking in the right directions. Only very little photos still have a "blank" cornea or -quite odd- an inverted iris (Dark cornea and white Iris... just like the kids in the Village of the Damned)
Also the teeth are getting some nice details.

The input is the usual 256x256... the output 128x128.

I believe that patience is all, in this game (which is not bad... I work 9-11 hrs/day... and I launch the training on my pee-see in the morning before getting to work.... so... I got nothing to do but see the results in the evening.

P.S.
Any update on the new manual alignment tool? How is it going ?

User avatar
abigflea
Posts: 182
Joined: Sat Feb 22, 2020 10:59 pm
Answers: 2
Has thanked: 20 times
Been thanked: 62 times

Re: The recovered images are quite blurred

Post by abigflea »

Grassone wrote: Fri May 01, 2020 12:33 pm

P.S.
Any update on the new manual alignment tool? How is it going ?

Become a patreon and find out. Buy these fellas some coffee.

:o I dunno what I'm doing :shock:
2X RTX 3090 : RTX 3080 : RTX: 2060 : 2x RTX 2080 Super : Ghetto 1060

User avatar
abigflea
Posts: 182
Joined: Sat Feb 22, 2020 10:59 pm
Answers: 2
Has thanked: 20 times
Been thanked: 62 times

Re: The recovered images are quite blurred

Post by abigflea »

Kevin_Z wrote: Thu Apr 30, 2020 2:15 am

I have trained for 20K iterations, and recovered images are quite better, the loss for each is about 0.039. I want to it would got better if I train longer, and when to stop ? Depends on loss?

Although it may look good before a million :-)
I've done it couple around 80k, with excellent facial data, to a 30sec clip and it looked pretty dang good.

:o I dunno what I'm doing :shock:
2X RTX 3090 : RTX 3080 : RTX: 2060 : 2x RTX 2080 Super : Ghetto 1060

User avatar
bryanlyon
Site Admin
Posts: 793
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 218 times
Contact:

Re: The recovered images are quite blurred

Post by bryanlyon »

Grassone wrote: Fri May 01, 2020 12:33 pm

P.S.
Any update on the new manual alignment tool? How is it going ?

Like ABigFlea said, this has been released to the Patrons in a beta test format. Torzdf will relelase it to everyone once he's happy with how it's working for them.

Locked