If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here
Forum rules
Read the FAQs and search the forum before posting a new topic.
This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.
Please mark any answers that fixed your problems so others can find the solutions.
sergor
Posts: 18 Joined: Wed Jan 15, 2020 7:59 pm
Has thanked: 9 times
Been thanked: 1 time
Post
by sergor » Sun Jul 17, 2022 4:34 pm
Hello fellow swappers,
System runs perfect, I might have received this error as the program was overworked while switching through graphs and analysis.
Has anyone else seen this message:
2022-07-17 18:29:36.846890: F tensorflow/python/lib/core/py_exception_registry.cc:69] Check failed: code != TF_OK (0 vs. 0)
Thanks!
Sergio
torzdf
Posts: 2684 Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 134 times
Been thanked: 626 times
Post
by torzdf » Sun Jul 17, 2022 5:42 pm
Unfortunately that's just a message saying that Tensorflow is not OK. There's not really much I can do with it
sergor
Posts: 18 Joined: Wed Jan 15, 2020 7:59 pm
Has thanked: 9 times
Been thanked: 1 time
Post
by sergor » Sun Jul 17, 2022 5:49 pm
Thanks Matt, I tried to recreate the error - while it was training I switched from graph to preview continuously - it cancels out if you move the bar from 0 iterations to actual. So the answer is leave it alone while training.
As we speak it´s still training - no errors, no damage to the model. All good!
SirDeepsALot
Posts: 35 Joined: Sat May 28, 2022 7:10 am
Has thanked: 7 times
Been thanked: 2 times
Post
by SirDeepsALot » Mon Jul 18, 2022 3:46 pm
Any idea what happened here:
I was training a model, click from analysis to graph I think and everything just closed.
System info if you need it:
Code: Select all
============ System Information ============
encoding: cp1252
git_branch: master
git_commits: 66c2b7b Bugfix: Stats elapsed time. 4d42591 Mixed Precision bugfix for AMD. 0d2eafa Enable change of precision for existing models. da942e3 Nan bugfix in Stats. 5d700e8 stats bugfixes
gpu_cuda: No global version found. Check Conda packages for Conda Cuda
gpu_cudnn: No global version found. Check Conda packages for Conda cuDNN
gpu_devices: GPU_0: NVIDIA GeForce GTX 1080
gpu_devices_active: GPU_0
gpu_driver: 516.01
gpu_vram: GPU_0: 8192MB
os_machine: AMD64
os_platform: Windows-10-10.0.19044-SP0
os_release: 10
py_command: C:\Users\dkroker\faceswap/faceswap.py gui
py_conda_version: conda 4.13.0
py_implementation: CPython
py_version: 3.9.12
py_virtual_env: True
sys_cores: 12
sys_processor: Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
sys_ram: Total: 32710MB, Available: 19128MB, Used: 13582MB, Free: 19128MB
=============== Pip Packages ===============
absl-py==1.1.0
astunparse==1.6.3
cachetools==5.2.0
certifi==2022.5.18.1
charset-normalizer==2.0.12
colorama @ file:///tmp/build/80754af9/colorama_1607707115595/work
cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work
fastcluster @ file:///D:/bld/fastcluster_1649783471014/work
ffmpy==0.2.3
flatbuffers==2.0
fonttools==4.25.0
gast==0.5.3
google-auth==2.8.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.46.3
h5py==3.7.0
idna==3.3
imageio @ file:///tmp/build/80754af9/imageio_1617700267927/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1649960641006/work
importlib-metadata==4.11.4
joblib @ file:///tmp/build/80754af9/joblib_1635411271373/work
keras==2.8.0
Keras-Preprocessing==1.1.2
kiwisolver @ file:///C:/ci/kiwisolver_1653292407425/work
libclang==14.0.1
Markdown==3.3.7
matplotlib @ file:///C:/ci/matplotlib-suite_1647423638658/work
mkl-fft==1.3.1
mkl-random @ file:///C:/ci/mkl_random_1626186184308/work
mkl-service==2.4.0
munkres==1.1.4
numpy @ file:///C:/ci/numpy_and_numpy_base_1652784039997/work
nvidia-ml-py==11.510.69
oauthlib==3.2.0
opencv-python==4.6.0.66
opt-einsum==3.3.0
packaging @ file:///tmp/build/80754af9/packaging_1637314298585/work
Pillow==9.0.1
protobuf==3.19.4
psutil @ file:///C:/ci/psutil_1612298199233/work
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing @ file:///tmp/build/80754af9/pyparsing_1635766073266/work
python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work
pywin32==302
requests==2.28.0
requests-oauthlib==1.3.1
rsa==4.8
scikit-learn @ file:///C:/ci/scikit-learn_1642617276183/work
scipy @ file:///C:/ci/scipy_1641555170412/work
sip==4.19.13
six @ file:///tmp/build/80754af9/six_1644875935023/work
tensorboard==2.8.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow-estimator==2.8.0
tensorflow-gpu==2.8.2
tensorflow-io-gcs-filesystem==0.26.0
termcolor==1.1.0
threadpoolctl @ file:///Users/ktietz/demo/mc3/conda-bld/threadpoolctl_1629802263681/work
tornado @ file:///C:/ci/tornado_1606924294691/work
tqdm @ file:///C:/ci/tqdm_1650636210717/work
typing_extensions @ file:///opt/conda/conda-bld/typing_extensions_1647553014482/work
urllib3==1.26.9
Werkzeug==2.1.2
wincertstore==0.2
wrapt==1.14.1
zipp==3.8.0
============== Conda Packages ==============
# packages in environment at C:\Users\dkroker\MiniConda3\envs\faceswap:
#
# Name Version Build Channel
absl-py 1.1.0 pypi_0 pypi
astunparse 1.6.3 pypi_0 pypi
blas 1.0 mkl
brotli 1.0.9 ha925a31_2
ca-certificates 2022.5.18.1 h5b45459_0 conda-forge
cachetools 5.2.0 pypi_0 pypi
certifi 2022.5.18.1 py39hcbf5309_0 conda-forge
charset-normalizer 2.0.12 pypi_0 pypi
colorama 0.4.4 pyhd3eb1b0_0
cudatoolkit 11.2.2 h933977f_10 conda-forge
cudnn 8.1.0.77 h3e0f4f4_0 conda-forge
cycler 0.11.0 pyhd3eb1b0_0
fastcluster 1.2.6 py39h2e25243_1 conda-forge
ffmpeg 4.3.1 ha925a31_0 conda-forge
ffmpy 0.2.3 pypi_0 pypi
flatbuffers 2.0 pypi_0 pypi
fonttools 4.25.0 pyhd3eb1b0_0
freetype 2.10.4 hd328e21_0
gast 0.5.3 pypi_0 pypi
git 2.34.1 haa95532_0
google-auth 2.8.0 pypi_0 pypi
google-auth-oauthlib 0.4.6 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
grpcio 1.46.3 pypi_0 pypi
h5py 3.7.0 pypi_0 pypi
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 3.3 pypi_0 pypi
imageio 2.9.0 pyhd3eb1b0_0
imageio-ffmpeg 0.4.7 pyhd8ed1ab_0 conda-forge
importlib-metadata 4.11.4 pypi_0 pypi
intel-openmp 2021.4.0 haa95532_3556
joblib 1.1.0 pyhd3eb1b0_0
jpeg 9e h2bbff1b_0
keras 2.8.0 pypi_0 pypi
keras-preprocessing 1.1.2 pypi_0 pypi
kiwisolver 1.4.2 py39hd77b12b_0
libclang 14.0.1 pypi_0 pypi
libpng 1.6.37 h2a8f88b_0
libtiff 4.2.0 he0120a3_1
libwebp 1.2.2 h2bbff1b_0
lz4-c 1.9.3 h2bbff1b_1
markdown 3.3.7 pypi_0 pypi
matplotlib 3.5.1 py39haa95532_1
matplotlib-base 3.5.1 py39hd77b12b_1
mkl 2021.4.0 haa95532_640
mkl-service 2.4.0 py39h2bbff1b_0
mkl_fft 1.3.1 py39h277e83a_0
mkl_random 1.2.2 py39hf11a4ad_0
munkres 1.1.4 py_0
numpy 1.22.3 py39h7a0a035_0
numpy-base 1.22.3 py39hca35cd5_0
nvidia-ml-py 11.510.69 pypi_0 pypi
oauthlib 3.2.0 pypi_0 pypi
opencv-python 4.6.0.66 pypi_0 pypi
openssl 1.1.1o h8ffe710_0 conda-forge
opt-einsum 3.3.0 pypi_0 pypi
packaging 21.3 pyhd3eb1b0_0
pillow 9.0.1 py39hdc2b20a_0
pip 21.2.4 py39haa95532_0
protobuf 3.19.4 pypi_0 pypi
psutil 5.8.0 py39h2bbff1b_1
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pyparsing 3.0.4 pyhd3eb1b0_0
pyqt 5.9.2 py39hd77b12b_6
python 3.9.12 h6244533_0
python-dateutil 2.8.2 pyhd3eb1b0_0
python_abi 3.9 2_cp39 conda-forge
pywin32 302 py39h2bbff1b_2
qt 5.9.7 vc14h73c81de_0
requests 2.28.0 pypi_0 pypi
requests-oauthlib 1.3.1 pypi_0 pypi
rsa 4.8 pypi_0 pypi
scikit-learn 1.0.2 py39hf11a4ad_1
scipy 1.7.3 py39h0a974cb_0
setuptools 61.2.0 py39haa95532_0
sip 4.19.13 py39hd77b12b_0
six 1.16.0 pyhd3eb1b0_1
sqlite 3.38.3 h2bbff1b_0
tensorboard 2.8.0 pypi_0 pypi
tensorboard-data-server 0.6.1 pypi_0 pypi
tensorboard-plugin-wit 1.8.1 pypi_0 pypi
tensorflow-estimator 2.8.0 pypi_0 pypi
tensorflow-gpu 2.8.2 pypi_0 pypi
tensorflow-io-gcs-filesystem 0.26.0 pypi_0 pypi
termcolor 1.1.0 pypi_0 pypi
threadpoolctl 2.2.0 pyh0d69192_0
tk 8.6.12 h2bbff1b_0
tornado 6.1 py39h2bbff1b_0
tqdm 4.64.0 py39haa95532_0
typing-extensions 4.1.1 hd3eb1b0_0
typing_extensions 4.1.1 pyh06a4308_0
tzdata 2022a hda174b7_0
urllib3 1.26.9 pypi_0 pypi
vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
werkzeug 2.1.2 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0
wincertstore 0.2 py39haa95532_2
wrapt 1.14.1 pypi_0 pypi
xz 5.2.5 h8cc25b3_1
zipp 3.8.0 pypi_0 pypi
zlib 1.2.12 h8cc25b3_2
zstd 1.5.2 h19a0ad4_0
================= Configs ==================
--------- .faceswap ---------
backend: nvidia
--------- convert.ini ---------
[color.color_transfer]
clip: True
preserve_paper: True
[color.manual_balance]
colorspace: HSV
balance_1: 0.0
balance_2: 0.0
balance_3: 0.0
contrast: 0.0
brightness: 0.0
[color.match_hist]
threshold: 99.0
[mask.mask_blend]
type: normalized
kernel_size: 3
passes: 4
threshold: 4
erosion: 0.0
erosion_top: 0.0
erosion_bottom: 0.0
erosion_left: 0.0
erosion_right: 0.0
[scaling.sharpen]
method: none
amount: 150
radius: 0.3
threshold: 5.0
[writer.ffmpeg]
container: mp4
codec: libx264
crf: 23
preset: medium
tune: none
profile: auto
level: auto
skip_mux: False
[writer.gif]
fps: 25
loop: 0
palettesize: 256
subrectangles: False
[writer.opencv]
format: png
draw_transparent: False
jpg_quality: 75
png_compress_level: 3
[writer.pillow]
format: png
draw_transparent: False
optimize: False
gif_interlace: True
jpg_quality: 75
png_compress_level: 3
tif_compression: tiff_deflate
--------- extract.ini ---------
[global]
allow_growth: False
[align.fan]
batch-size: 12
[detect.cv2_dnn]
confidence: 50
[detect.mtcnn]
minsize: 20
scalefactor: 0.709
batch-size: 8
threshold_1: 0.6
threshold_2: 0.7
threshold_3: 0.7
[detect.s3fd]
confidence: 70
batch-size: 4
[mask.bisenet_fp]
batch-size: 8
weights: faceswap
include_ears: False
include_hair: False
include_glasses: True
[mask.unet_dfl]
batch-size: 8
[mask.vgg_clear]
batch-size: 6
[mask.vgg_obstructed]
batch-size: 2
--------- gui.ini ---------
[global]
fullscreen: False
tab: extract
options_panel_width: 30
console_panel_height: 20
icon_size: 14
font: default
font_size: 9
autosave_last_session: prompt
timeout: 120
auto_load_model_stats: True
--------- train.ini ---------
[global]
centering: head
coverage: 87.5
icnr_init: False
conv_aware_init: False
optimizer: adam
learning_rate: 5e-05
epsilon_exponent: -7
reflect_padding: False
allow_growth: False
mixed_precision: False
nan_protection: True
convert_batchsize: 16
[global.loss]
loss_function: ssim
loss_function_2: mse
loss_weight_2: 100
loss_function_3: none
loss_weight_3: 0
loss_function_4: none
loss_weight_4: 0
mask_loss_function: mse
eye_multiplier: 3
mouth_multiplier: 2
penalized_mask_loss: True
mask_type: bisenet-fp_face
mask_blur_kernel: 3
mask_threshold: 4
learn_mask: False
[model.dfaker]
output_size: 128
[model.dfl_h128]
lowmem: False
[model.dfl_sae]
input_size: 128
clipnorm: True
architecture: df
autoencoder_dims: 0
encoder_dims: 42
decoder_dims: 21
multiscale_decoder: False
[model.dlight]
features: best
details: good
output_size: 256
[model.original]
lowmem: False
[model.phaze_a]
output_size: 128
shared_fc: None
enable_gblock: True
split_fc: True
split_gblock: False
split_decoders: False
enc_architecture: fs_original
enc_scaling: 7
enc_load_weights: True
bottleneck_type: dense
bottleneck_norm: None
bottleneck_size: 1024
bottleneck_in_encoder: True
fc_depth: 1
fc_min_filters: 1024
fc_max_filters: 1024
fc_dimensions: 4
fc_filter_slope: -0.5
fc_dropout: 0.0
fc_upsampler: upsample2d
fc_upsamples: 1
fc_upsample_filters: 512
fc_gblock_depth: 3
fc_gblock_min_nodes: 512
fc_gblock_max_nodes: 512
fc_gblock_filter_slope: -0.5
fc_gblock_dropout: 0.0
dec_upscale_method: subpixel
dec_upscales_in_fc: 0
dec_norm: None
dec_min_filters: 64
dec_max_filters: 512
dec_slope_mode: full
dec_filter_slope: -0.45
dec_res_blocks: 1
dec_output_kernel: 5
dec_gaussian: True
dec_skip_last_residual: True
freeze_layers: keras_encoder
load_layers: encoder
fs_original_depth: 4
fs_original_min_filters: 128
fs_original_max_filters: 1024
fs_original_use_alt: False
mobilenet_width: 1.0
mobilenet_depth: 1
mobilenet_dropout: 0.001
mobilenet_minimalistic: False
[model.realface]
input_size: 64
output_size: 128
dense_nodes: 1536
complexity_encoder: 128
complexity_decoder: 512
[model.unbalanced]
input_size: 128
lowmem: False
clipnorm: True
nodes: 1024
complexity_encoder: 128
complexity_decoder_a: 384
complexity_decoder_b: 512
[model.villain]
lowmem: False
[trainer.original]
preview_images: 14
zoom_amount: 5
rotation_range: 10
shift_range: 5
flip_chance: 50
color_lightness: 30
color_ab: 8
color_clahe_chance: 50
color_clahe_max_size: 4