Page 1 of 1

Error code viewing graph whilst training.

Posted: Sun Jul 17, 2022 4:34 pm
by sergor

Hello fellow swappers,

System runs perfect, I might have received this error as the program was overworked while switching through graphs and analysis.

Has anyone else seen this message:

2022-07-17 18:29:36.846890: F tensorflow/python/lib/core/py_exception_registry.cc:69] Check failed: code != TF_OK (0 vs. 0)

Thanks!

Sergio


Re: Error code viewing graph whilst training.

Posted: Sun Jul 17, 2022 5:42 pm
by torzdf

Unfortunately that's just a message saying that Tensorflow is not OK. There's not really much I can do with it :(


Re: Error code viewing graph whilst training.

Posted: Sun Jul 17, 2022 5:49 pm
by sergor

Thanks Matt, I tried to recreate the error - while it was training I switched from graph to preview continuously - it cancels out if you move the bar from 0 iterations to actual. So the answer is leave it alone while training.

As we speak it´s still training - no errors, no damage to the model. All good!


Training crash?

Posted: Mon Jul 18, 2022 3:46 pm
by SirDeepsALot

Any idea what happened here:

I was training a model, click from analysis to graph I think and everything just closed.

System info if you need it:

Code: Select all

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         66c2b7b Bugfix: Stats elapsed time. 4d42591 Mixed Precision bugfix for AMD. 0d2eafa Enable change of precision for existing models. da942e3 Nan bugfix in Stats. 5d700e8 stats bugfixes
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: NVIDIA GeForce GTX 1080
gpu_devices_active:  GPU_0
gpu_driver:          516.01
gpu_vram:            GPU_0: 8192MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.19044-SP0
os_release:          10
py_command:          C:\Users\dkroker\faceswap/faceswap.py gui
py_conda_version:    conda 4.13.0
py_implementation:   CPython
py_version:          3.9.12
py_virtual_env:      True
sys_cores:           12
sys_processor:       Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
sys_ram:             Total: 32710MB, Available: 19128MB, Used: 13582MB, Free: 19128MB

=============== Pip Packages ===============
absl-py==1.1.0
astunparse==1.6.3
cachetools==5.2.0
certifi==2022.5.18.1
charset-normalizer==2.0.12
colorama @ file:///tmp/build/80754af9/colorama_1607707115595/work
cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
fastcluster @ file:///D:/bld/fastcluster_1649783471014/work
ffmpy==0.2.3
flatbuffers==2.0
fonttools==4.25.0
gast==0.5.3
google-auth==2.8.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.46.3
h5py==3.7.0
idna==3.3
imageio @ file:///tmp/build/80754af9/imageio_1617700267927/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1649960641006/work
importlib-metadata==4.11.4
joblib @ file:///tmp/build/80754af9/joblib_1635411271373/work
keras==2.8.0
Keras-Preprocessing==1.1.2
kiwisolver @ file:///C:/ci/kiwisolver_1653292407425/work
libclang==14.0.1
Markdown==3.3.7
matplotlib @ file:///C:/ci/matplotlib-suite_1647423638658/work
mkl-fft==1.3.1
mkl-random @ file:///C:/ci/mkl_random_1626186184308/work
mkl-service==2.4.0
munkres==1.1.4
numpy @ file:///C:/ci/numpy_and_numpy_base_1652784039997/work
nvidia-ml-py==11.510.69
oauthlib==3.2.0
opencv-python==4.6.0.66
opt-einsum==3.3.0
packaging @ file:///tmp/build/80754af9/packaging_1637314298585/work
Pillow==9.0.1
protobuf==3.19.4
psutil @ file:///C:/ci/psutil_1612298199233/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing @ file:///tmp/build/80754af9/pyparsing_1635766073266/work
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
pywin32==302
requests==2.28.0
requests-oauthlib==1.3.1
rsa==4.8
scikit-learn @ file:///C:/ci/scikit-learn_1642617276183/work
scipy @ file:///C:/ci/scipy_1641555170412/work
sip==4.19.13
six @ file:///tmp/build/80754af9/six_1644875935023/work
tensorboard==2.8.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow-estimator==2.8.0
tensorflow-gpu==2.8.2
tensorflow-io-gcs-filesystem==0.26.0
termcolor==1.1.0
threadpoolctl @ file:///Users/ktietz/demo/mc3/conda-bld/threadpoolctl_1629802263681/work
tornado @ file:///C:/ci/tornado_1606924294691/work
tqdm @ file:///C:/ci/tqdm_1650636210717/work
typing_extensions @ file:///opt/conda/conda-bld/typing_extensions_1647553014482/work
urllib3==1.26.9
Werkzeug==2.1.2
wincertstore==0.2
wrapt==1.14.1
zipp==3.8.0

============== Conda Packages ==============
# packages in environment at C:\Users\dkroker\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
absl-py                   1.1.0                    pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
blas                      1.0                         mkl  
brotli                    1.0.9                ha925a31_2  
ca-certificates           2022.5.18.1          h5b45459_0    conda-forge
cachetools                5.2.0                    pypi_0    pypi
certifi                   2022.5.18.1      py39hcbf5309_0    conda-forge
charset-normalizer        2.0.12                   pypi_0    pypi
colorama                  0.4.4              pyhd3eb1b0_0  
cudatoolkit               11.2.2              h933977f_10    conda-forge
cudnn                     8.1.0.77             h3e0f4f4_0    conda-forge
cycler                    0.11.0             pyhd3eb1b0_0  
fastcluster               1.2.6            py39h2e25243_1    conda-forge
ffmpeg                    4.3.1                ha925a31_0    conda-forge
ffmpy                     0.2.3                    pypi_0    pypi
flatbuffers               2.0                      pypi_0    pypi
fonttools                 4.25.0             pyhd3eb1b0_0  
freetype                  2.10.4               hd328e21_0  
gast                      0.5.3                    pypi_0    pypi
git                       2.34.1               haa95532_0  
google-auth               2.8.0                    pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.46.3                   pypi_0    pypi
h5py                      3.7.0                    pypi_0    pypi
icc_rt                    2019.0.0             h0cc432a_1  
icu                       58.2                 ha925a31_3  
idna                      3.3                      pypi_0    pypi
imageio                   2.9.0              pyhd3eb1b0_0  
imageio-ffmpeg            0.4.7              pyhd8ed1ab_0    conda-forge
importlib-metadata        4.11.4                   pypi_0    pypi
intel-openmp              2021.4.0          haa95532_3556  
joblib                    1.1.0              pyhd3eb1b0_0  
jpeg                      9e                   h2bbff1b_0  
keras                     2.8.0                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.4.2            py39hd77b12b_0  
libclang                  14.0.1                   pypi_0    pypi
libpng                    1.6.37               h2a8f88b_0  
libtiff                   4.2.0                he0120a3_1  
libwebp                   1.2.2                h2bbff1b_0  
lz4-c                     1.9.3                h2bbff1b_1  
markdown                  3.3.7                    pypi_0    pypi
matplotlib                3.5.1            py39haa95532_1  
matplotlib-base           3.5.1            py39hd77b12b_1  
mkl                       2021.4.0           haa95532_640  
mkl-service               2.4.0            py39h2bbff1b_0  
mkl_fft                   1.3.1            py39h277e83a_0  
mkl_random                1.2.2            py39hf11a4ad_0  
munkres                   1.1.4                      py_0  
numpy                     1.22.3           py39h7a0a035_0  
numpy-base                1.22.3           py39hca35cd5_0  
nvidia-ml-py              11.510.69                pypi_0    pypi
oauthlib                  3.2.0                    pypi_0    pypi
opencv-python             4.6.0.66                 pypi_0    pypi
openssl                   1.1.1o               h8ffe710_0    conda-forge
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 21.3               pyhd3eb1b0_0  
pillow                    9.0.1            py39hdc2b20a_0  
pip                       21.2.4           py39haa95532_0  
protobuf                  3.19.4                   pypi_0    pypi
psutil                    5.8.0            py39h2bbff1b_1  
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pyparsing                 3.0.4              pyhd3eb1b0_0  
pyqt                      5.9.2            py39hd77b12b_6  
python                    3.9.12               h6244533_0  
python-dateutil           2.8.2              pyhd3eb1b0_0  
python_abi                3.9                      2_cp39    conda-forge
pywin32                   302              py39h2bbff1b_2  
qt                        5.9.7            vc14h73c81de_0  
requests                  2.28.0                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rsa                       4.8                      pypi_0    pypi
scikit-learn              1.0.2            py39hf11a4ad_1  
scipy                     1.7.3            py39h0a974cb_0  
setuptools                61.2.0           py39haa95532_0  
sip                       4.19.13          py39hd77b12b_0  
six                       1.16.0             pyhd3eb1b0_1  
sqlite                    3.38.3               h2bbff1b_0  
tensorboard               2.8.0                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow-estimator      2.8.0                    pypi_0    pypi
tensorflow-gpu            2.8.2                    pypi_0    pypi
tensorflow-io-gcs-filesystem 0.26.0                   pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
threadpoolctl             2.2.0              pyh0d69192_0  
tk                        8.6.12               h2bbff1b_0  
tornado                   6.1              py39h2bbff1b_0  
tqdm                      4.64.0           py39haa95532_0  
typing-extensions         4.1.1                hd3eb1b0_0  
typing_extensions         4.1.1              pyh06a4308_0  
tzdata                    2022a                hda174b7_0  
urllib3                   1.26.9                   pypi_0    pypi
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
werkzeug                  2.1.2                    pypi_0    pypi
wheel                     0.37.1             pyhd3eb1b0_0  
wincertstore              0.2              py39haa95532_2  
wrapt                     1.14.1                   pypi_0    pypi
xz                        5.2.5                h8cc25b3_1  
zipp                      3.8.0                    pypi_0    pypi
zlib                      1.2.12               h8cc25b3_2  
zstd                      1.5.2                h19a0ad4_0  

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0
erosion_top:              0.0
erosion_bottom:           0.0
erosion_left:             0.0
erosion_right:            0.0

[scaling.sharpen]
method:                   none
amount:                   150
radius:                   0.3
threshold:                5.0

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
jpg_quality:              75
png_compress_level:       3

[writer.pillow]
format:                   png
draw_transparent:         False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

--------- extract.ini ---------

[global]
allow_growth:             False

[align.fan]
batch-size:               12

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
scalefactor:              0.709
batch-size:               8
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7

[detect.s3fd]
confidence:               70
batch-size:               4

[mask.bisenet_fp]
batch-size:               8
weights:                  faceswap
include_ears:             False
include_hair:             False
include_glasses:          True

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

--------- gui.ini ---------

[global]
fullscreen:               False
tab:                      extract
options_panel_width:      30
console_panel_height:     20
icon_size:                14
font:                     default
font_size:                9
autosave_last_session:    prompt
timeout:                  120
auto_load_model_stats:    True

--------- train.ini ---------

[global]
centering:                head
coverage:                 87.5
icnr_init:                False
conv_aware_init:          False
optimizer:                adam
learning_rate:            5e-05
epsilon_exponent:         -7
reflect_padding:          False
allow_growth:             False
mixed_precision:          False
nan_protection:           True
convert_batchsize:        16

[global.loss]
loss_function:            ssim
loss_function_2:          mse
loss_weight_2:            100
loss_function_3:          none
loss_weight_3:            0
loss_function_4:          none
loss_weight_4:            0
mask_loss_function:       mse
eye_multiplier:           3
mouth_multiplier:         2
penalized_mask_loss:      True
mask_type:                bisenet-fp_face
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False

[model.dfaker]
output_size:              128

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
clipnorm:                 True
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.original]
lowmem:                   False

[model.phaze_a]
output_size:              128
shared_fc:                None
enable_gblock:            True
split_fc:                 True
split_gblock:             False
split_decoders:           False
enc_architecture:         fs_original
enc_scaling:              7
enc_load_weights:         True
bottleneck_type:          dense
bottleneck_norm:          None
bottleneck_size:          1024
bottleneck_in_encoder:    True
fc_depth:                 1
fc_min_filters:           1024
fc_max_filters:           1024
fc_dimensions:            4
fc_filter_slope:          -0.5
fc_dropout:               0.0
fc_upsampler:             upsample2d
fc_upsamples:             1
fc_upsample_filters:      512
fc_gblock_depth:          3
fc_gblock_min_nodes:      512
fc_gblock_max_nodes:      512
fc_gblock_filter_slope:   -0.5
fc_gblock_dropout:        0.0
dec_upscale_method:       subpixel
dec_upscales_in_fc:       0
dec_norm:                 None
dec_min_filters:          64
dec_max_filters:          512
dec_slope_mode:           full
dec_filter_slope:         -0.45
dec_res_blocks:           1
dec_output_kernel:        5
dec_gaussian:             True
dec_skip_last_residual:   True
freeze_layers:            keras_encoder
load_layers:              encoder
fs_original_depth:        4
fs_original_min_filters:  128
fs_original_max_filters:  1024
fs_original_use_alt:      False
mobilenet_width:          1.0
mobilenet_depth:          1
mobilenet_dropout:        0.001
mobilenet_minimalistic:   False

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.unbalanced]
input_size:               128
lowmem:                   False
clipnorm:                 True
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4