Training crashes before start: ‘Enabled TensorBoard Logging'

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
Mr.Man
Posts: 9
Joined: Thu Jun 25, 2020 9:28 pm
Answers: 0

Training crashes before start: ‘Enabled TensorBoard Logging'

Post by Mr.Man »

Hope you can help with this one – I have extracted my models A and B, but beginning training the process hangs at ‘Enabled TensorBoard Logging’. No error message or crashlog, it just freezes.

Code: Select all

Loading...
Setting Faceswap backend to NVIDIA
08/17/2020 00:44:08 INFO     Log level set to: INFO
08/17/2020 00:44:10 INFO     Model A Directory: J:\SYNC\Sync\ENCRYPTSYNC\faceswap\src\MODELA\TRAINING
08/17/2020 00:44:10 INFO     Model B Directory: J:\SYNC\Sync\ENCRYPTSYNC\faceswap\src\MODELB\2\Training2
08/17/2020 00:44:10 INFO     Training data directory: J:\SYNC\Sync\ENCRYPTSYNC\faceswap\src\MODELB\2\CONVERT1
08/17/2020 00:44:10 INFO     ===================================================
08/17/2020 00:44:10 INFO       Starting
08/17/2020 00:44:10 INFO       Press 'Stop' to save and quit
08/17/2020 00:44:10 INFO     ===================================================
08/17/2020 00:44:11 INFO     Loading data, this may take a while...
08/17/2020 00:44:11 INFO     Loading Model from Villain plugin...
08/17/2020 00:44:12 INFO     No existing state file found. Generating.
08/17/2020 00:44:18 INFO     Loading Trainer from Original plugin...

Reading training images (A):   0%|          | 0/13750 [00:00<?, ?it/s]
Reading training images (A):   0%|          | 1/13750 [00:00<1:08:55,  3.32it/s]
Reading training images (A):   1%|1         | 174/13750 [00:00<47:40,  4.75it/s]
Reading training images (A):   2%|1         | 274/13750 [00:00<33:12,  6.76it/s]
Reading training images (A):   3%|2         | 367/13750 [00:00<23:09,  9.63it/s]
Reading training images (A):   3%|3         | 447/13750 [00:00<16:11, 13.69it/s]
Reading training images (A):   4%|3         | 537/13750 [00:00<11:20, 19.42it/s]
Reading training images (A):   5%|4         | 634/13750 [00:00<07:56, 27.50it/s]
Reading training images (A):   5%|5         | 717/13750 [00:01<05:37, 38.67it/s]
Reading training images (A):   6%|5         | 798/13750 [00:01<04:00, 53.88it/s]
Reading training images (A):   6%|6         | 881/13750 [00:01<02:51, 74.86it/s]
Reading training images (A):   7%|7         | 969/13750 [00:01<02:03, 103.17it/s]
Reading training images (A):   8%|7         | 1066/13750 [00:01<01:29, 140.94it/s]
Reading training images (A):   8%|8         | 1152/13750 [00:01<01:08, 184.82it/s]
Reading training images (A):   9%|8         | 1233/13750 [00:01<00:53, 236.02it/s]
Reading training images (A):  10%|9         | 1323/13750 [00:01<00:41, 303.03it/s]
Reading training images (A):  10%|#         | 1427/13750 [00:01<00:32, 384.72it/s]
Reading training images (A):  11%|#1        | 1515/13750 [00:02<00:26, 455.48it/s]
Reading training images (A):  12%|#1        | 1633/13750 [00:02<00:21, 558.11it/s]
Reading training images (A):  13%|#2        | 1728/13750 [00:02<00:19, 601.44it/s]
Reading training images (A):  13%|#3        | 1817/13750 [00:02<00:18, 645.51it/s]
Reading training images (A):  14%|#4        | 1947/13750 [00:02<00:15, 759.99it/s]
Reading training images (A):  15%|#4        | 2059/13750 [00:02<00:13, 840.59it/s]
Reading training images (A):  16%|#5        | 2162/13750 [00:02<00:13, 830.82it/s]
Reading training images (A):  17%|#6        | 2286/13750 [00:02<00:12, 921.55it/s]
Reading training images (A):  17%|#7        | 2391/13750 [00:02<00:12, 913.44it/s]
Reading training images (A):  18%|#8        | 2493/13750 [00:03<00:11, 942.30it/s]
Reading training images (A):  19%|#9        | 2624/13750 [00:03<00:10, 1028.29it/s]
Reading training images (A):  20%|#9        | 2734/13750 [00:03<00:12, 894.24it/s]
Reading training images (A):  21%|##        | 2842/13750 [00:03<00:11, 942.24it/s]
Reading training images (A):  21%|##1       | 2943/13750 [00:03<00:11, 911.31it/s]
Reading training images (A):  22%|##2       | 3058/13750 [00:03<00:11, 966.26it/s]
Reading training images (A):  23%|##2       | 3160/13750 [00:03<00:11, 929.91it/s]
Reading training images (A):  24%|##3       | 3257/13750 [00:03<00:11, 916.78it/s]
Reading training images (A):  24%|##4       | 3354/13750 [00:03<00:11, 931.41it/s]
Reading training images (A):  25%|##5       | 3485/13750 [00:04<00:10, 1016.81it/s]
Reading training images (A):  26%|##6       | 3591/13750 [00:04<00:10, 969.19it/s]
Reading training images (A):  27%|##6       | 3691/13750 [00:04<00:10, 977.46it/s]
Reading training images (A):  28%|##7       | 3793/13750 [00:04<00:10, 989.08it/s]
Reading training images (A):  28%|##8       | 3894/13750 [00:04<00:09, 988.62it/s]
Reading training images (A):  29%|##9       | 3994/13750 [00:04<00:11, 860.39it/s]
Reading training images (A):  30%|##9       | 4084/13750 [00:04<00:11, 830.14it/s]
Reading training images (A):  31%|###       | 4216/13750 [00:04<00:10, 933.61it/s]
Reading training images (A):  31%|###1      | 4317/13750 [00:04<00:10, 935.93it/s]
Reading training images (A):  32%|###2      | 4415/13750 [00:05<00:09, 937.09it/s]
Reading training images (A):  33%|###2      | 4525/13750 [00:05<00:09, 959.40it/s]
Reading training images (A):  34%|###3      | 4645/13750 [00:05<00:08, 1017.50it/s]
Reading training images (A):  35%|###4      | 4750/13750 [00:05<00:08, 1020.24it/s]
Reading training images (A):  35%|###5      | 4877/13750 [00:05<00:08, 1080.69it/s]
Reading training images (A):  36%|###6      | 4988/13750 [00:05<00:08, 1075.77it/s]
Reading training images (A):  37%|###7      | 5098/13750 [00:05<00:08, 962.46it/s]
Reading training images (A):  38%|###8      | 5225/13750 [00:05<00:08, 1034.64it/s]
Reading training images (A):  39%|###8      | 5345/13750 [00:05<00:07, 1069.81it/s]
Reading training images (A):  40%|###9      | 5455/13750 [00:06<00:07, 1049.99it/s]
Reading training images (A):  40%|####      | 5566/13750 [00:06<00:07, 1066.48it/s]
Reading training images (A):  41%|####1     | 5675/13750 [00:06<00:07, 1047.77it/s]
Reading training images (A):  42%|####2     | 5801/13750 [00:06<00:07, 1102.76it/s]
Reading training images (A):  43%|####3     | 5913/13750 [00:06<00:07, 1103.72it/s]
Reading training images (A):  44%|####3     | 6025/13750 [00:06<00:07, 1081.93it/s]
Reading training images (A):  45%|####4     | 6155/13750 [00:06<00:06, 1138.46it/s]
Reading training images (A):  46%|####5     | 6276/13750 [00:06<00:06, 1151.50it/s]
Reading training images (A):  47%|####6     | 6401/13750 [00:06<00:06, 1178.49it/s]
Reading training images (A):  47%|####7     | 6520/13750 [00:06<00:06, 1060.66it/s]
Reading training images (A):  48%|####8     | 6629/13750 [00:07<00:07, 1003.37it/s]
Reading training images (A):  49%|####8     | 6732/13750 [00:07<00:07, 962.15it/s]
Reading training images (A):  50%|####9     | 6847/13750 [00:07<00:06, 1008.36it/s]
Reading training images (A):  51%|#####     | 6982/13750 [00:07<00:06, 1087.85it/s]
Reading training images (A):  52%|#####1    | 7095/13750 [00:07<00:06, 1092.91it/s]
Reading training images (A):  53%|#####2    | 7229/13750 [00:07<00:05, 1153.12it/s]
Reading training images (A):  53%|#####3    | 7347/13750 [00:07<00:05, 1068.50it/s]
Reading training images (A):  54%|#####4    | 7466/13750 [00:07<00:05, 1098.39it/s]
Reading training images (A):  55%|#####5    | 7579/13750 [00:07<00:05, 1045.23it/s]
Reading training images (A):  56%|#####5    | 7686/13750 [00:08<00:07, 844.48it/s]
Reading training images (A):  57%|#####6    | 7810/13750 [00:08<00:06, 933.28it/s]
Reading training images (A):  58%|#####7    | 7922/13750 [00:08<00:05, 981.72it/s]
Reading training images (A):  59%|#####8    | 8048/13750 [00:08<00:05, 1048.05it/s]
Reading training images (A):  59%|#####9    | 8176/13750 [00:08<00:05, 1096.13it/s]
Reading training images (A):  60%|######    | 8308/13750 [00:08<00:04, 1154.08it/s]
Reading training images (A):  61%|######1   | 8428/13750 [00:08<00:04, 1139.91it/s]
Reading training images (A):  62%|######2   | 8553/13750 [00:08<00:04, 1169.97it/s]
Reading training images (A):  63%|######3   | 8673/13750 [00:08<00:04, 1144.11it/s]
Reading training images (A):  64%|######3   | 8790/13750 [00:09<00:05, 955.73it/s]
Reading training images (A):  65%|######4   | 8892/13750 [00:09<00:05, 893.86it/s]
Reading training images (A):  65%|######5   | 8987/13750 [00:09<00:05, 909.30it/s]
Reading training images (A):  66%|######6   | 9119/13750 [00:09<00:04, 1002.30it/s]
Reading training images (A):  67%|######7   | 9225/13750 [00:09<00:04, 952.13it/s]
Reading training images (A):  68%|######7   | 9338/13750 [00:09<00:04, 998.62it/s]
Reading training images (A):  69%|######8   | 9447/13750 [00:09<00:04, 1023.61it/s]
Reading training images (A):  70%|######9   | 9574/13750 [00:09<00:03, 1086.13it/s]
Reading training images (A):  70%|#######   | 9686/13750 [00:10<00:04, 933.25it/s]
Reading training images (A):  71%|#######1  | 9790/13750 [00:10<00:04, 962.19it/s]
Reading training images (A):  72%|#######2  | 9911/13750 [00:10<00:03, 1024.48it/s]
Reading training images (A):  73%|#######2  | 10020/13750 [00:10<00:03, 1042.50it/s]
Reading training images (A):  74%|#######3  | 10134/13750 [00:10<00:03, 1069.15it/s]
Reading training images (A):  75%|#######4  | 10244/13750 [00:10<00:03, 989.89it/s]
Reading training images (A):  75%|#######5  | 10346/13750 [00:10<00:03, 883.54it/s]
Reading training images (A):  76%|#######5  | 10439/13750 [00:11<00:06, 544.37it/s]
Reading training images (A):  76%|#######6  | 10513/13750 [00:11<00:05, 583.87it/s]
Reading training images (A):  78%|#######7  | 10692/13750 [00:11<00:04, 731.56it/s]
Reading training images (A):  79%|#######8  | 10797/13750 [00:11<00:05, 536.20it/s]
Reading training images (A):  79%|#######9  | 10880/13750 [00:11<00:05, 495.19it/s]
Reading training images (A):  80%|########  | 11030/13750 [00:11<00:04, 619.53it/s]
Reading training images (A):  81%|########  | 11125/13750 [00:11<00:03, 691.31it/s]
Reading training images (A):  82%|########1 | 11256/13750 [00:12<00:03, 805.03it/s]
Reading training images (A):  83%|########3 | 11446/13750 [00:12<00:02, 972.91it/s]
Reading training images (A):  84%|########4 | 11577/13750 [00:12<00:02, 975.72it/s]
Reading training images (A):  85%|########5 | 11719/13750 [00:12<00:01, 1076.14it/s]
Reading training images (A):  86%|########6 | 11847/13750 [00:12<00:01, 1117.47it/s]
Reading training images (A):  87%|########7 | 11982/13750 [00:12<00:01, 1177.54it/s]
Reading training images (A):  88%|########8 | 12120/13750 [00:12<00:01, 1230.87it/s]
Reading training images (A):  89%|########9 | 12286/13750 [00:12<00:01, 1333.50it/s]
Reading training images (A):  91%|######### | 12482/13750 [00:12<00:00, 1474.05it/s]
Reading training images (A):  92%|#########1| 12640/13750 [00:13<00:00, 1298.75it/s]
Reading training images (A):  93%|#########2| 12781/13750 [00:13<00:00, 1206.11it/s]
Reading training images (A):  94%|#########4| 12960/13750 [00:13<00:00, 1336.14it/s]
Reading training images (A):  96%|#########5| 13138/13750 [00:13<00:00, 1429.30it/s]
Reading training images (A):  97%|#########6| 13301/13750 [00:13<00:00, 1483.04it/s]
Reading training images (A):  98%|#########7| 13459/13750 [00:13<00:00, 1509.70it/s]
Reading training images (A):  99%|#########9| 13616/13750 [00:13<00:00, 1298.34it/s]


Reading training images (B):   0%|          | 0/1370 [00:00<?, ?it/s]
Reading training images (B):   6%|6         | 89/1370 [00:00<00:01, 821.89it/s]
Reading training images (B):  16%|#5        | 214/1370 [00:00<00:01, 915.47it/s]
Reading training images (B):  26%|##6       | 360/1370 [00:00<00:00, 1030.23it/s]
Reading training images (B):  38%|###7      | 514/1370 [00:00<00:00, 1140.59it/s]
Reading training images (B):  45%|####5     | 617/1370 [00:00<00:00, 1096.98it/s]
Reading training images (B):  53%|#####2    | 720/1370 [00:00<00:00, 999.75it/s]
Reading training images (B):  60%|#####9    | 817/1370 [00:00<00:00, 946.26it/s]
Reading training images (B):  66%|######6   | 910/1370 [00:00<00:00, 910.11it/s]
Reading training images (B):  73%|#######3  | 1001/1370 [00:00<00:00, 857.78it/s]
Reading training images (B):  80%|########  | 1102/1370 [00:01<00:00, 883.59it/s]
Reading training images (B):  87%|########6 | 1191/1370 [00:01<00:00, 809.73it/s]
Reading training images (B):  93%|#########2| 1274/1370 [00:01<00:00, 708.35it/s]
08/17/2020 00:44:34 INFO     Reading alignments from: 'J:\SYNC\Sync\ENCRYPTSYNC\faceswap\src\MODELA\TRAINING\alignments.fsa'
08/17/2020 00:44:37 INFO     Reading alignments from: 'J:\SYNC\Sync\ENCRYPTSYNC\faceswap\src\MODELB\2\Alignments\Training\alignments.fsa'
08/17/2020 00:44:38 INFO     Enabled TensorBoard Logging

Any idea on what could be causing this? Here is my output system information:

Code: Select all

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         6b2aac6 Enable MTCNN for CPU extraction
gpu_cuda:            10.1
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce GTX 1070 Ti
gpu_devices_active:  GPU_0
gpu_driver:          425.25
gpu_vram:            GPU_0: 8192MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.16299-SP0
os_release:          10
py_command:          C:\Users\ME\faceswap/faceswap.py gui
py_conda_version:    conda 4.8.4
py_implementation:   CPython
py_version:          3.8.5
py_virtual_env:      True
sys_cores:           6
sys_processor:       Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
sys_ram:             Total: 16344MB, Available: 11160MB, Used: 5184MB, Free: 11160MB

=============== Pip Packages ===============
absl-py==0.9.0
astunparse==1.6.3
cachetools==4.1.1
certifi==2020.6.20
chardet==3.0.4
cycler==0.10.0
fastcluster==1.1.26
ffmpy==0.2.3
gast==0.3.3
google-auth==1.20.1
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.31.0
h5py==2.10.0
idna==2.10
imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1589202782679/work
joblib @ file:///tmp/build/80754af9/joblib_1594236160679/work
Keras-Preprocessing==1.1.2
kiwisolver==1.2.0
Markdown==3.2.2
matplotlib @ file:///C:/ci/matplotlib-base_1592837548929/work
mkl-fft==1.1.0
mkl-random==1.1.1
mkl-service==2.3.0
numpy @ file:///C:/ci/numpy_and_numpy_base_1596215850360/work
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib==3.1.0
olefile==0.46
opencv-python==4.4.0.40
opt-einsum==3.3.0
pathlib==1.0.1
Pillow @ file:///C:/ci/pillow_1594298230227/work
protobuf==3.13.0
psutil==5.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==2.4.7
python-dateutil==2.8.1
pywin32==227
requests==2.24.0
requests-oauthlib==1.3.0
rsa==4.6
scikit-learn @ file:///C:/ci/scikit-learn_1592853510272/work
scipy==1.4.1
sip==4.19.13
six==1.15.0
tensorboard==2.2.2
tensorboard-plugin-wit==1.7.0
tensorflow==2.2.0
tensorflow-estimator==2.2.0
tensorflow-gpu==2.2.0
tensorflow-gpu-estimator==2.2.0
termcolor==1.1.0
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
tornado==6.0.4
tqdm @ file:///tmp/build/80754af9/tqdm_1596810128862/work
urllib3==1.25.10
Werkzeug==1.0.1
wincertstore==0.2
wrapt==1.12.1

============== Conda Packages ==============
# packages in environment at C:\Users\ME\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
absl-py                   0.9.0                    pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
blas                      1.0                         mkl  

ca-certificates           2020.6.24                     0  

cachetools                4.1.1                    pypi_0    pypi
certifi                   2020.6.20                py38_0  

chardet                   3.0.4                    pypi_0    pypi
cudatoolkit               10.1.243             h74a9793_0  

cudnn                     7.6.5                cuda10.1_0  

cycler                    0.10.0                   py38_0  

fastcluster               1.1.26           py38hbe40bda_1    conda-forge
ffmpeg                    4.3.1                ha925a31_0    conda-forge
ffmpy                     0.2.3                    pypi_0    pypi
freetype                  2.10.2               hd328e21_0  

gast                      0.3.3                    pypi_0    pypi
git                       2.23.0               h6bb4b03_0  

google-auth               1.20.1                   pypi_0    pypi
google-auth-oauthlib      0.4.1                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.31.0                   pypi_0    pypi
h5py                      2.10.0                   pypi_0    pypi
icc_rt                    2019.0.0             h0cc432a_1  

icu                       58.2                 ha925a31_3  

idna                      2.10                     pypi_0    pypi
imageio                   2.9.0                      py_0  

imageio-ffmpeg            0.4.2                      py_0    conda-forge
intel-openmp              2020.1                      216  

joblib                    0.16.0                     py_0  

jpeg                      9b                   hb83a4c4_2  

keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.2.0            py38h74a9793_0  

libpng                    1.6.37               h2a8f88b_0  

libtiff                   4.1.0                h56a325e_1  

lz4-c                     1.9.2                h62dcd97_1  

markdown                  3.2.2                    pypi_0    pypi
matplotlib                3.2.2                         0  

matplotlib-base           3.2.2            py38h64f37c6_0  

mkl                       2020.1                      216  

mkl-service               2.3.0            py38hb782905_0  

mkl_fft                   1.1.0            py38h45dec08_0  

mkl_random                1.1.1            py38h47e9c7a_0  

numpy                     1.19.1           py38h5510c5b_0  

numpy-base                1.19.1           py38ha3acd2a_0  

nvidia-ml-py3             7.352.1                  pypi_0    pypi
oauthlib                  3.1.0                    pypi_0    pypi
olefile                   0.46                       py_0  

opencv-python             4.4.0.40                 pypi_0    pypi
openssl                   1.1.1g               he774522_1  

opt-einsum                3.3.0                    pypi_0    pypi
pathlib                   1.0.1                      py_1  

pillow                    7.2.0            py38hcc1f983_0  

pip                       20.2.2                   py38_0  

protobuf                  3.13.0                   pypi_0    pypi
psutil                    5.7.0            py38he774522_0  

pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pyparsing                 2.4.7                      py_0  

pyqt                      5.9.2            py38ha925a31_4  

python                    3.8.5                he1778fa_0  

python-dateutil           2.8.1                      py_0  

python_abi                3.8                      1_cp38    conda-forge
pywin32                   227              py38he774522_1  

qt                        5.9.7            vc14h73c81de_0  

requests                  2.24.0                   pypi_0    pypi
requests-oauthlib         1.3.0                    pypi_0    pypi
rsa                       4.6                      pypi_0    pypi
scikit-learn              0.23.1           py38h25d0782_0  

scipy                     1.4.1                    pypi_0    pypi
setuptools                49.6.0                   py38_0  

sip                       4.19.13          py38ha925a31_0  

six                       1.15.0                     py_0  

sqlite                    3.32.3               h2a8f88b_0  

tensorboard               2.2.2                    pypi_0    pypi
tensorboard-plugin-wit    1.7.0                    pypi_0    pypi
tensorflow                2.2.0                    pypi_0    pypi
tensorflow-estimator      2.2.0                    pypi_0    pypi
tensorflow-gpu            2.2.0                    pypi_0    pypi
tensorflow-gpu-estimator  2.2.0                    pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
threadpoolctl             2.1.0              pyh5ca1d4c_0  

tk                        8.6.10               he774522_0  

tornado                   6.0.4            py38he774522_1  

tqdm                      4.48.2                     py_0  

urllib3                   1.25.10                  pypi_0    pypi
vc                        14.1                 h0510ff6_4  

vs2015_runtime            14.16.27012          hf0eaf9b_3  

werkzeug                  1.0.1                    pypi_0    pypi
wheel                     0.34.2                   py38_0  

wincertstore              0.2                      py38_0  

wrapt                     1.12.1                   pypi_0    pypi
xz                        5.2.5                h62dcd97_0  

zlib                      1.2.11               h62dcd97_4  

zstd                      1.4.5                h04227a9_0  

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[mask.box_blend]
type:                     gaussian
distance:                 11.0
radius:                   5.0
passes:                   1

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0

[scaling.sharpen]
method:                   unsharp_mask
amount:                   150
radius:                   0.3
threshold:                5.0

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
jpg_quality:              75
png_compress_level:       3

[writer.pillow]
format:                   png
draw_transparent:         False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

--------- extract.ini ---------

[global]
allow_growth:             False

[align.fan]
batch-size:               12

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7
scalefactor:              0.709
batch-size:               8

[detect.s3fd]
confidence:               70
batch-size:               4

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

--------- gui.ini ---------

[global]
fullscreen:               False
tab:                      extract
options_panel_width:      30
console_panel_height:     20
icon_size:                14
font:                     default
font_size:                9
autosave_last_session:    prompt
timeout:                  120
auto_load_model_stats:    True

--------- train.ini ---------

[global]
coverage:                 68.75
mask_type:                extended
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False
icnr_init:                False
conv_aware_init:          False
reflect_padding:          False
allow_growth:             False
penalized_mask_loss:      True
loss_function:            mae
learning_rate:            5e-05

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
clipnorm:                 True
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.original]
lowmem:                   False

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.unbalanced]
input_size:               128
lowmem:                   False
clipnorm:                 True
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4

Any help would be greatly appreciated.

Thank you

Top
User avatar
torzdf
Posts: 2796
Joined: Fri Jul 12, 2019 12:53 am
Answers: 160
Has thanked: 142 times
Been thanked: 650 times

Re: Training crashes before start: ‘Enabled TensorBoard Logging'

Post by torzdf »

I would suspect that this is a Cuda initialization issue.

First and foremost, update your Nvidia drivers to the newest version (use the game ready drivers, rather than the studio drivers).

Secondly, remove the system installed version of Cuda you have on your system (will be listed under Nvidia Cuda in Add/Remove programs... there is likely going to be a fair amount of apps to remove).

Once done, reboot, try again, report back.

My word is final

Top
User avatar
Mr.Man
Posts: 9
Joined: Thu Jun 25, 2020 9:28 pm
Answers: 0

Re: Training crashes before start: ‘Enabled TensorBoard Logging'

Post by Mr.Man »

Thank you for such a quick response! I've followed your instructions: updated Nvidia drivers, using the game not studio version, and have removed the system installed CUDA versions.

During training, it pushed past the 'enabled tensorboard logging' but then 'Caught exception in thread: '_training_0' and crashed. There is a crash report though which I'm sure will be useful. Here is the process:

Code: Select all

Loading...
Setting Faceswap backend to NVIDIA
08/17/2020 17:06:33 INFO     Log level set to: INFO
08/17/2020 17:06:40 INFO     Model A Directory: J:\SYNC\Sync\ENCRYPTSYNC\faceswap\src\MODELA\TRAINING
08/17/2020 17:06:40 INFO     Model B Directory: J:\SYNC\Sync\ENCRYPTSYNC\faceswap\src\MODELB\2\Training2
08/17/2020 17:06:40 INFO     Training data directory: J:\SYNC\Sync\ENCRYPTSYNC\faceswap\src\MODELB\2\CONVERT
08/17/2020 17:06:40 INFO     ===================================================
08/17/2020 17:06:40 INFO       Starting
08/17/2020 17:06:40 INFO       Press 'Stop' to save and quit
08/17/2020 17:06:40 INFO     ===================================================
08/17/2020 17:06:41 INFO     Loading data, this may take a while...
08/17/2020 17:06:41 INFO     Loading Model from Villain plugin...
08/17/2020 17:06:41 INFO     No existing state file found. Generating.
08/17/2020 17:06:47 INFO     Loading Trainer from Original plugin...

Reading training images (A):   0%|          | 0/13750 [00:00<?, ?it/s]
Reading training images (A):   0%|          | 1/13750 [00:00<1:07:18,  3.40it/s]
Reading training images (A):   1%|          | 75/13750 [00:00<46:57,  4.85it/s]
Reading training images (A):   1%|1         | 144/13750 [00:00<32:48,  6.91it/s]
Reading training images (A):   1%|1         | 182/13750 [00:00<23:04,  9.80it/s]
Reading training images (A):   2%|1         | 219/13750 [00:00<16:23, 13.76it/s]
Reading training images (A):   2%|1         | 253/13750 [00:00<11:39, 19.28it/s]
Reading training images (A):   2%|2         | 286/13750 [00:00<08:25, 26.62it/s]
Reading training images (A):   2%|2         | 330/13750 [00:01<06:04, 36.85it/s]
Reading training images (A):   3%|2         | 362/13750 [00:01<04:27, 50.09it/s]
Reading training images (A):   3%|2         | 395/13750 [00:01<03:20, 66.74it/s]
Reading training images (A):   3%|3         | 429/13750 [00:01<02:33, 86.91it/s]
Reading training images (A):   3%|3         | 463/13750 [00:01<01:59, 111.23it/s]
Reading training images (A):   4%|3         | 512/13750 [00:01<01:31, 144.20it/s]
Reading training images (A):   4%|4         | 557/13750 [00:01<01:13, 178.68it/s]
Reading training images (A):   4%|4         | 595/13750 [00:01<01:01, 212.19it/s]
Reading training images (A):   5%|4         | 633/13750 [00:02<00:58, 222.50it/s]
Reading training images (A):   5%|4         | 667/13750 [00:02<00:58, 223.54it/s]
Reading training images (A):   5%|5         | 698/13750 [00:02<00:54, 238.73it/s]
Reading training images (A):   5%|5         | 753/13750 [00:02<00:45, 287.43it/s]
Reading training images (A):   6%|5         | 800/13750 [00:02<00:40, 322.43it/s]
Reading training images (A):   6%|6         | 844/13750 [00:02<00:38, 334.49it/s]
Reading training images (A):   6%|6         | 883/13750 [00:02<00:43, 297.71it/s]
Reading training images (A):   7%|6         | 922/13750 [00:02<00:42, 303.81it/s]
Reading training images (A):   7%|6         | 962/13750 [00:02<00:39, 327.24it/s]
Reading training images (A):   7%|7         | 998/13750 [00:03<00:46, 276.67it/s]
Reading training images (A):   8%|7         | 1033/13750 [00:03<00:45, 281.39it/s]
Reading training images (A):   8%|7         | 1084/13750 [00:03<00:40, 310.98it/s]
Reading training images (A):   8%|8         | 1118/13750 [00:03<00:41, 301.38it/s]
Reading training images (A):   8%|8         | 1151/13750 [00:03<00:43, 288.62it/s]
Reading training images (A):   9%|8         | 1188/13750 [00:03<00:41, 299.66it/s]
Reading training images (A):   9%|8         | 1229/13750 [00:03<00:38, 321.36it/s]
Reading training images (A):  10%|9         | 1314/13750 [00:03<00:31, 394.30it/s]
Reading training images (A):  10%|#         | 1381/13750 [00:04<00:27, 448.68it/s]
Reading training images (A):  10%|#         | 1435/13750 [00:04<00:26, 461.40it/s]
Reading training images (A):  11%|#         | 1488/13750 [00:04<00:28, 423.98it/s]
Reading training images (A):  11%|#1        | 1536/13750 [00:04<00:29, 417.37it/s]
Reading training images (A):  12%|#1        | 1582/13750 [00:04<00:30, 399.20it/s]
Reading training images (A):  12%|#1        | 1635/13750 [00:04<00:28, 423.60it/s]
Reading training images (A):  12%|#2        | 1680/13750 [00:04<00:28, 419.17it/s]
Reading training images (A):  13%|#2        | 1724/13750 [00:04<00:31, 378.76it/s]
Reading training images (A):  13%|#2        | 1764/13750 [00:05<00:32, 369.18it/s]
Reading training images (A):  13%|#3        | 1803/13750 [00:05<00:35, 335.44it/s]
Reading training images (A):  13%|#3        | 1848/13750 [00:05<00:33, 352.68it/s]
Reading training images (A):  14%|#3        | 1885/13750 [00:05<00:34, 343.06it/s]
Reading training images (A):  14%|#3        | 1921/13750 [00:05<00:35, 329.75it/s]
Reading training images (A):  14%|#4        | 1955/13750 [00:05<00:36, 325.00it/s]
Reading training images (A):  15%|#4        | 2002/13750 [00:05<00:32, 357.90it/s]
Reading training images (A):  15%|#5        | 2077/13750 [00:05<00:27, 422.83it/s]
Reading training images (A):  15%|#5        | 2126/13750 [00:06<00:31, 364.92it/s]
Reading training images (A):  16%|#5        | 2173/13750 [00:06<00:29, 390.38it/s]
Reading training images (A):  16%|#6        | 2228/13750 [00:06<00:26, 427.24it/s]
Reading training images (A):  17%|#6        | 2275/13750 [00:06<00:40, 284.36it/s]
Reading training images (A):  17%|#7        | 2342/13750 [00:06<00:34, 335.14it/s]
Reading training images (A):  17%|#7        | 2393/13750 [00:06<00:30, 370.89it/s]
Reading training images (A):  18%|#7        | 2469/13750 [00:06<00:25, 437.98it/s]
Reading training images (A):  18%|#8        | 2524/13750 [00:06<00:26, 423.86it/s]
Reading training images (A):  19%|#8        | 2574/13750 [00:07<00:28, 391.30it/s]
Reading training images (A):  19%|#9        | 2619/13750 [00:07<00:28, 387.45it/s]
Reading training images (A):  19%|#9        | 2662/13750 [00:07<00:30, 368.89it/s]
Reading training images (A):  20%|#9        | 2703/13750 [00:07<00:29, 377.95it/s]
Reading training images (A):  20%|##        | 2762/13750 [00:07<00:26, 416.16it/s]
Reading training images (A):  20%|##        | 2807/13750 [00:07<00:28, 390.73it/s]
Reading training images (A):  21%|##        | 2849/13750 [00:07<00:27, 390.16it/s]
Reading training images (A):  21%|##1       | 2890/13750 [00:07<00:30, 359.58it/s]
Reading training images (A):  21%|##1       | 2928/13750 [00:08<00:31, 345.32it/s]
Reading training images (A):  22%|##1       | 2964/13750 [00:08<00:34, 315.36it/s]
Reading training images (A):  22%|##1       | 2997/13750 [00:08<00:35, 306.84it/s]
Reading training images (A):  22%|##2       | 3051/13750 [00:08<00:31, 345.02it/s]
Reading training images (A):  23%|##2       | 3100/13750 [00:08<00:28, 376.53it/s]
Reading training images (A):  23%|##3       | 3165/13750 [00:08<00:24, 423.90it/s]
Reading training images (A):  23%|##3       | 3212/13750 [00:08<00:26, 395.87it/s]
Reading training images (A):  24%|##3       | 3255/13750 [00:08<00:27, 381.36it/s]
Reading training images (A):  24%|##3       | 3296/13750 [00:09<00:27, 375.08it/s]
Reading training images (A):  24%|##4       | 3352/13750 [00:09<00:25, 414.03it/s]
Reading training images (A):  25%|##4       | 3396/13750 [00:09<00:27, 378.06it/s]
Reading training images (A):  25%|##4       | 3437/13750 [00:09<00:29, 344.91it/s]
Reading training images (A):  25%|##5       | 3474/13750 [00:09<00:29, 351.40it/s]
Reading training images (A):  26%|##5       | 3515/13750 [00:09<00:28, 364.51it/s]
Reading training images (A):  26%|##5       | 3562/13750 [00:09<00:26, 379.19it/s]
Reading training images (A):  26%|##6       | 3628/13750 [00:09<00:24, 417.59it/s]
Reading training images (A):  27%|##6       | 3681/13750 [00:09<00:24, 413.61it/s]
Reading training images (A):  27%|##7       | 3724/13750 [00:10<00:26, 380.46it/s]
Reading training images (A):  27%|##7       | 3764/13750 [00:10<00:30, 325.66it/s]
Reading training images (A):  28%|##7       | 3801/13750 [00:10<00:29, 337.74it/s]
Reading training images (A):  28%|##7       | 3837/13750 [00:10<00:32, 309.20it/s]
Reading training images (A):  28%|##8       | 3877/13750 [00:10<00:30, 320.38it/s]
Reading training images (A):  29%|##8       | 3920/13750 [00:10<00:28, 344.26it/s]
Reading training images (A):  29%|##8       | 3956/13750 [00:10<00:30, 321.37it/s]
Reading training images (A):  29%|##9       | 3990/13750 [00:11<00:32, 301.01it/s]
Reading training images (A):  29%|##9       | 4022/13750 [00:11<00:33, 293.32it/s]
Reading training images (A):  29%|##9       | 4053/13750 [00:11<00:34, 281.50it/s]
Reading training images (A):  30%|##9       | 4085/13750 [00:11<00:33, 284.92it/s]
Reading training images (A):  30%|###       | 4126/13750 [00:11<00:32, 297.13it/s]
Reading training images (A):  30%|###       | 4157/13750 [00:11<00:33, 290.26it/s]
Reading training images (A):  31%|###       | 4198/13750 [00:11<00:31, 304.97it/s]
Reading training images (A):  31%|###       | 4237/13750 [00:11<00:30, 313.68it/s]
Reading training images (A):  31%|###1      | 4269/13750 [00:11<00:30, 309.79it/s]
Reading training images (A):  31%|###1      | 4309/13750 [00:12<00:29, 322.35it/s]
Reading training images (A):  32%|###1      | 4357/13750 [00:12<00:26, 354.08it/s]
Reading training images (A):  32%|###1      | 4394/13750 [00:12<00:26, 353.19it/s]
Reading training images (A):  32%|###2      | 4431/13750 [00:12<00:31, 295.56it/s]
Reading training images (A):  32%|###2      | 4463/13750 [00:12<00:32, 285.45it/s]
Reading training images (A):  33%|###2      | 4500/13750 [00:12<00:32, 283.05it/s]
Reading training images (A):  33%|###2      | 4532/13750 [00:12<00:32, 280.77it/s]
Reading training images (A):  33%|###3      | 4574/13750 [00:12<00:29, 309.14it/s]
Reading training images (A):  34%|###3      | 4607/13750 [00:13<00:30, 301.53it/s]
Reading training images (A):  34%|###3      | 4639/13750 [00:13<00:32, 279.36it/s]
Reading training images (A):  34%|###3      | 4668/13750 [00:13<00:32, 278.87it/s]
Reading training images (A):  34%|###4      | 4700/13750 [00:13<00:32, 275.48it/s]
Reading training images (A):  34%|###4      | 4736/13750 [00:13<00:31, 283.18it/s]
Reading training images (A):  35%|###4      | 4787/13750 [00:13<00:29, 303.45it/s]
Reading training images (A):  35%|###5      | 4831/13750 [00:13<00:26, 334.08it/s]
Reading training images (A):  35%|###5      | 4866/13750 [00:13<00:29, 304.81it/s]
Reading training images (A):  36%|###5      | 4898/13750 [00:14<00:33, 264.96it/s]
Reading training images (A):  36%|###5      | 4933/13750 [00:14<00:31, 280.69it/s]
Reading training images (A):  36%|###6      | 4963/13750 [00:14<00:31, 276.11it/s]
Reading training images (A):  36%|###6      | 4992/13750 [00:14<00:31, 277.11it/s]
Reading training images (A):  37%|###6      | 5021/13750 [00:14<00:32, 271.70it/s]
Reading training images (A):  37%|###6      | 5050/13750 [00:14<00:31, 275.49it/s]
Reading training images (A):  37%|###6      | 5078/13750 [00:14<00:31, 273.47it/s]
Reading training images (A):  37%|###7      | 5110/13750 [00:14<00:31, 273.72it/s]
Reading training images (A):  37%|###7      | 5145/13750 [00:14<00:29, 292.58it/s]
Reading training images (A):  38%|###7      | 5186/13750 [00:14<00:27, 316.60it/s]
Reading training images (A):  38%|###7      | 5222/13750 [00:15<00:27, 312.84it/s]
Reading training images (A):  38%|###8      | 5258/13750 [00:15<00:26, 325.28it/s]
Reading training images (A):  38%|###8      | 5292/13750 [00:15<00:27, 311.71it/s]
Reading training images (A):  39%|###8      | 5325/13750 [00:15<00:27, 308.48it/s]
Reading training images (A):  39%|###8      | 5357/13750 [00:15<00:26, 311.36it/s]
Reading training images (A):  39%|###9      | 5391/13750 [00:15<00:27, 304.84it/s]
Reading training images (A):  39%|###9      | 5425/13750 [00:15<00:26, 312.95it/s]
Reading training images (A):  40%|###9      | 5457/13750 [00:15<00:28, 295.84it/s]
Reading training images (A):  40%|###9      | 5487/13750 [00:15<00:28, 285.09it/s]
Reading training images (A):  40%|####      | 5522/13750 [00:16<00:27, 298.13it/s]
Reading training images (A):  40%|####      | 5560/13750 [00:16<00:26, 303.54it/s]
Reading training images (A):  41%|####      | 5597/13750 [00:16<00:25, 317.57it/s]
Reading training images (A):  41%|####      | 5634/13750 [00:16<00:24, 328.56it/s]
Reading training images (A):  41%|####1     | 5668/13750 [00:16<00:26, 305.86it/s]
Reading training images (A):  42%|####1     | 5711/13750 [00:16<00:24, 326.32it/s]
Reading training images (A):  42%|####1     | 5757/13750 [00:16<00:22, 351.78it/s]
Reading training images (A):  42%|####2     | 5814/13750 [00:16<00:20, 396.62it/s]
Reading training images (A):  43%|####2     | 5857/13750 [00:17<00:20, 376.85it/s]
Reading training images (A):  43%|####2     | 5897/13750 [00:17<00:21, 360.32it/s]
Reading training images (A):  43%|####3     | 5935/13750 [00:17<00:21, 358.99it/s]
Reading training images (A):  43%|####3     | 5973/13750 [00:17<00:23, 328.23it/s]
Reading training images (A):  44%|####3     | 6020/13750 [00:17<00:22, 346.25it/s]
Reading training images (A):  44%|####4     | 6057/13750 [00:17<00:22, 338.09it/s]
Reading training images (A):  44%|####4     | 6092/13750 [00:17<00:22, 335.04it/s]
Reading training images (A):  45%|####4     | 6136/13750 [00:17<00:21, 359.11it/s]
Reading training images (A):  45%|####4     | 6185/13750 [00:17<00:19, 390.14it/s]
Reading training images (A):  45%|####5     | 6226/13750 [00:18<00:20, 361.01it/s]
Reading training images (A):  46%|####5     | 6269/13750 [00:18<00:20, 363.58it/s]
Reading training images (A):  46%|####5     | 6307/13750 [00:18<00:20, 363.83it/s]
Reading training images (A):  46%|####6     | 6361/13750 [00:18<00:18, 401.25it/s]
Reading training images (A):  47%|####6     | 6408/13750 [00:18<00:19, 385.24it/s]
Reading training images (A):  47%|####6     | 6459/13750 [00:18<00:18, 404.58it/s]
Reading training images (A):  47%|####7     | 6511/13750 [00:18<00:16, 433.15it/s]
Reading training images (A):  48%|####7     | 6556/13750 [00:18<00:18, 399.19it/s]
Reading training images (A):  48%|####7     | 6598/13750 [00:18<00:19, 368.56it/s]
Reading training images (A):  48%|####8     | 6637/13750 [00:19<00:19, 371.24it/s]
Reading training images (A):  49%|####8     | 6679/13750 [00:19<00:19, 371.07it/s]
Reading training images (A):  49%|####8     | 6717/13750 [00:19<00:19, 359.59it/s]
Reading training images (A):  49%|####9     | 6754/13750 [00:19<00:19, 361.30it/s]
Reading training images (A):  49%|####9     | 6791/13750 [00:19<00:21, 328.62it/s]
Reading training images (A):  50%|####9     | 6825/13750 [00:19<00:21, 320.41it/s]
Reading training images (A):  50%|####9     | 6869/13750 [00:19<00:20, 339.73it/s]
Reading training images (A):  50%|#####     | 6904/13750 [00:19<00:21, 325.24it/s]
Reading training images (A):  50%|#####     | 6938/13750 [00:20<00:21, 321.78it/s]
Reading training images (A):  51%|#####     | 6971/13750 [00:20<00:22, 302.51it/s]
Reading training images (A):  51%|#####     | 7002/13750 [00:20<00:22, 302.69it/s]
Reading training images (A):  51%|#####1    | 7033/13750 [00:20<00:22, 301.93it/s]
Reading training images (A):  51%|#####1    | 7070/13750 [00:20<00:21, 317.70it/s]
Reading training images (A):  52%|#####1    | 7103/13750 [00:20<00:22, 298.34it/s]
Reading training images (A):  52%|#####1    | 7134/13750 [00:20<00:23, 277.93it/s]
Reading training images (A):  52%|#####2    | 7163/13750 [00:20<00:24, 274.03it/s]
Reading training images (A):  52%|#####2    | 7191/13750 [00:20<00:24, 269.98it/s]
Reading training images (A):  53%|#####2    | 7219/13750 [00:21<00:24, 265.69it/s]
Reading training images (A):  53%|#####2    | 7249/13750 [00:21<00:24, 269.00it/s]
Reading training images (A):  53%|#####3    | 7290/13750 [00:21<00:21, 298.46it/s]
Reading training images (A):  53%|#####3    | 7321/13750 [00:21<00:21, 293.85it/s]
Reading training images (A):  54%|#####3    | 7364/13750 [00:21<00:19, 322.30it/s]
Reading training images (A):  54%|#####3    | 7412/13750 [00:21<00:18, 347.14it/s]
Reading training images (A):  54%|#####4    | 7449/13750 [00:21<00:19, 326.14it/s]
Reading training images (A):  55%|#####4    | 7498/13750 [00:21<00:17, 355.96it/s]
Reading training images (A):  55%|#####4    | 7547/13750 [00:21<00:16, 387.54it/s]
Reading training images (A):  55%|#####5    | 7604/13750 [00:21<00:14, 424.61it/s]
Reading training images (A):  56%|#####5    | 7649/13750 [00:22<00:14, 430.26it/s]
Reading training images (A):  56%|#####5    | 7694/13750 [00:22<00:15, 393.12it/s]
Reading training images (A):  56%|#####6    | 7736/13750 [00:22<00:16, 367.59it/s]
Reading training images (A):  57%|#####6    | 7775/13750 [00:22<00:17, 342.44it/s]
Reading training images (A):  57%|#####6    | 7811/13750 [00:22<00:18, 318.19it/s]
Reading training images (A):  57%|#####7    | 7849/13750 [00:22<00:17, 333.62it/s]
Reading training images (A):  57%|#####7    | 7896/13750 [00:22<00:16, 356.24it/s]
Reading training images (A):  58%|#####7    | 7933/13750 [00:22<00:18, 319.71it/s]
Reading training images (A):  58%|#####7    | 7967/13750 [00:23<00:17, 325.29it/s]
Reading training images (A):  58%|#####8    | 8001/13750 [00:23<00:30, 190.94it/s]
Reading training images (A):  59%|#####8    | 8044/13750 [00:23<00:25, 225.84it/s]
Reading training images (A):  59%|#####8    | 8094/13750 [00:23<00:21, 269.18it/s]
Reading training images (A):  59%|#####9    | 8133/13750 [00:23<00:19, 290.80it/s]
Reading training images (A):  59%|#####9    | 8179/13750 [00:23<00:18, 309.20it/s]
Reading training images (A):  60%|#####9    | 8217/13750 [00:23<00:17, 319.71it/s]
Reading training images (A):  60%|######    | 8253/13750 [00:24<00:17, 322.29it/s]
Reading training images (A):  60%|######    | 8291/13750 [00:24<00:16, 332.89it/s]
Reading training images (A):  61%|######    | 8327/13750 [00:24<00:15, 339.25it/s]
Reading training images (A):  61%|######    | 8363/13750 [00:24<00:16, 331.19it/s]
Reading training images (A):  61%|######1   | 8398/13750 [00:24<00:16, 328.83it/s]
Reading training images (A):  61%|######1   | 8435/13750 [00:24<00:15, 339.37it/s]
Reading training images (A):  62%|######1   | 8474/13750 [00:24<00:15, 344.35it/s]
Reading training images (A):  62%|######1   | 8510/13750 [00:24<00:16, 322.01it/s]
Reading training images (A):  62%|######2   | 8543/13750 [00:25<00:18, 288.59it/s]
Reading training images (A):  62%|######2   | 8577/13750 [00:25<00:18, 278.18it/s]
Reading training images (A):  63%|######2   | 8606/13750 [00:25<00:18, 271.63it/s]
Reading training images (A):  63%|######2   | 8634/13750 [00:25<00:19, 256.89it/s]
Reading training images (A):  63%|######3   | 8679/13750 [00:25<00:17, 294.01it/s]
Reading training images (A):  63%|######3   | 8711/13750 [00:25<00:18, 269.99it/s]
Reading training images (A):  64%|######3   | 8740/13750 [00:25<00:19, 255.12it/s]
Reading training images (A):  64%|######3   | 8768/13750 [00:25<00:20, 244.01it/s]
Reading training images (A):  64%|######3   | 8794/13750 [00:26<00:22, 215.72it/s]
Reading training images (A):  64%|######4   | 8823/13750 [00:26<00:21, 231.64it/s]
Reading training images (A):  65%|######4   | 8869/13750 [00:26<00:18, 271.02it/s]
Reading training images (A):  65%|######4   | 8900/13750 [00:26<00:17, 275.40it/s]
Reading training images (A):  65%|######5   | 8945/13750 [00:26<00:16, 298.74it/s]
Reading training images (A):  65%|######5   | 8978/13750 [00:26<00:16, 281.86it/s]
Reading training images (A):  66%|######5   | 9009/13750 [00:26<00:22, 211.31it/s]
Reading training images (A):  66%|######5   | 9054/13750 [00:26<00:18, 250.30it/s]
Reading training images (A):  66%|######6   | 9105/13750 [00:27<00:15, 293.55it/s]
Reading training images (A):  67%|######6   | 9156/13750 [00:27<00:13, 336.18it/s]
Reading training images (A):  67%|######6   | 9202/13750 [00:27<00:12, 357.11it/s]
Reading training images (A):  67%|######7   | 9243/13750 [00:27<00:15, 296.13it/s]
Reading training images (A):  67%|######7   | 9279/13750 [00:27<00:15, 283.20it/s]
Reading training images (A):  68%|######8   | 9378/13750 [00:27<00:12, 360.23it/s]
Reading training images (A):  69%|######8   | 9431/13750 [00:27<00:11, 367.52it/s]
Reading training images (A):  69%|######8   | 9480/13750 [00:27<00:12, 349.67it/s]
Reading training images (A):  69%|######9   | 9524/13750 [00:28<00:12, 335.17it/s]
Reading training images (A):  70%|######9   | 9564/13750 [00:28<00:13, 308.61it/s]
Reading training images (A):  70%|######9   | 9600/13750 [00:28<00:14, 293.45it/s]
Reading training images (A):  70%|#######   | 9633/13750 [00:28<00:14, 284.94it/s]
Reading training images (A):  70%|#######   | 9664/13750 [00:28<00:16, 240.73it/s]
Reading training images (A):  71%|#######   | 9696/13750 [00:28<00:15, 257.31it/s]
Reading training images (A):  71%|#######   | 9748/13750 [00:28<00:13, 300.82it/s]
Reading training images (A):  71%|#######1  | 9792/13750 [00:29<00:12, 320.74it/s]
Reading training images (A):  72%|#######1  | 9850/13750 [00:29<00:10, 363.46it/s]
Reading training images (A):  72%|#######1  | 9891/13750 [00:29<00:11, 333.18it/s]
Reading training images (A):  72%|#######2  | 9930/13750 [00:29<00:11, 339.39it/s]
Reading training images (A):  72%|#######2  | 9967/13750 [00:29<00:10, 346.07it/s]
Reading training images (A):  73%|#######2  | 10014/13750 [00:29<00:09, 374.65it/s]
Reading training images (A):  73%|#######3  | 10054/13750 [00:29<00:09, 377.08it/s]
Reading training images (A):  73%|#######3  | 10094/13750 [00:29<00:09, 368.83it/s]
Reading training images (A):  74%|#######3  | 10132/13750 [00:29<00:09, 371.67it/s]
Reading training images (A):  74%|#######3  | 10171/13750 [00:30<00:09, 367.34it/s]
Reading training images (A):  74%|#######4  | 10209/13750 [00:30<00:10, 335.40it/s]
Reading training images (A):  75%|#######4  | 10244/13750 [00:30<00:11, 302.78it/s]
Reading training images (A):  75%|#######4  | 10276/13750 [00:30<00:11, 298.17it/s]
Reading training images (A):  75%|#######5  | 10343/13750 [00:30<00:09, 357.18it/s]
Reading training images (A):  76%|#######5  | 10385/13750 [00:30<00:09, 351.02it/s]
Reading training images (A):  76%|#######5  | 10425/13750 [00:30<00:11, 302.20it/s]
Reading training images (A):  76%|#######6  | 10460/13750 [00:30<00:12, 273.44it/s]
Reading training images (A):  76%|#######6  | 10500/13750 [00:31<00:11, 293.14it/s]
Reading training images (A):  77%|#######6  | 10533/13750 [00:31<00:13, 247.14it/s]
Reading training images (A):  77%|#######6  | 10569/13750 [00:31<00:11, 270.95it/s]
Reading training images (A):  77%|#######7  | 10615/13750 [00:31<00:10, 304.63it/s]
Reading training images (A):  77%|#######7  | 10649/13750 [00:31<00:10, 306.02it/s]
Reading training images (A):  78%|#######7  | 10699/13750 [00:31<00:09, 309.23it/s]
Reading training images (A):  78%|#######8  | 10732/13750 [00:31<00:11, 252.41it/s]
Reading training images (A):  78%|#######8  | 10761/13750 [00:32<00:12, 234.99it/s]
Reading training images (A):  79%|#######8  | 10813/13750 [00:32<00:10, 274.32it/s]
Reading training images (A):  79%|#######8  | 10845/13750 [00:32<00:11, 248.48it/s]
Reading training images (A):  79%|#######9  | 10885/13750 [00:32<00:10, 272.80it/s]
Reading training images (A):  79%|#######9  | 10920/13750 [00:32<00:10, 276.31it/s]
Reading training images (A):  80%|#######9  | 10985/13750 [00:32<00:08, 324.17it/s]
Reading training images (A):  80%|########  | 11023/13750 [00:32<00:08, 329.02it/s]
Reading training images (A):  80%|########  | 11060/13750 [00:32<00:08, 334.53it/s]
Reading training images (A):  81%|########  | 11096/13750 [00:33<00:08, 317.87it/s]
Reading training images (A):  81%|########  | 11132/13750 [00:33<00:08, 316.29it/s]
Reading training images (A):  81%|########1 | 11169/13750 [00:33<00:07, 330.48it/s]
Reading training images (A):  81%|########1 | 11204/13750 [00:33<00:08, 310.01it/s]
Reading training images (A):  82%|########1 | 11245/13750 [00:33<00:07, 319.82it/s]
Reading training images (A):  82%|########2 | 11292/13750 [00:33<00:07, 331.81it/s]
Reading training images (A):  82%|########2 | 11341/13750 [00:33<00:06, 359.43it/s]
Reading training images (A):  83%|########2 | 11404/13750 [00:33<00:05, 410.37it/s]
Reading training images (A):  83%|########3 | 11449/13750 [00:33<00:05, 420.41it/s]
Reading training images (A):  84%|########3 | 11494/13750 [00:34<00:05, 412.00it/s]
Reading training images (A):  84%|########3 | 11537/13750 [00:34<00:05, 408.93it/s]
Reading training images (A):  84%|########4 | 11580/13750 [00:34<00:05, 362.45it/s]
Reading training images (A):  85%|########4 | 11620/13750 [00:34<00:05, 371.33it/s]
Reading training images (A):  85%|########4 | 11663/13750 [00:34<00:05, 372.25it/s]
Reading training images (A):  85%|########5 | 11702/13750 [00:34<00:05, 373.00it/s]
Reading training images (A):  85%|########5 | 11745/13750 [00:34<00:05, 387.54it/s]
Reading training images (A):  86%|########5 | 11785/13750 [00:34<00:05, 358.11it/s]
Reading training images (A):  86%|########6 | 11830/13750 [00:34<00:05, 381.47it/s]
Reading training images (A):  86%|########6 | 11870/13750 [00:35<00:05, 370.09it/s]
Reading training images (A):  87%|########6 | 11908/13750 [00:35<00:05, 323.22it/s]
Reading training images (A):  87%|########6 | 11953/13750 [00:35<00:05, 352.40it/s]
Reading training images (A):  87%|########7 | 11991/13750 [00:35<00:05, 319.50it/s]
Reading training images (A):  87%|########7 | 12025/13750 [00:35<00:06, 282.14it/s]
Reading training images (A):  88%|########7 | 12062/13750 [00:35<00:05, 295.45it/s]
Reading training images (A):  88%|########7 | 12094/13750 [00:35<00:05, 301.61it/s]
Reading training images (A):  88%|########8 | 12136/13750 [00:35<00:04, 329.18it/s]
Reading training images (A):  89%|########8 | 12171/13750 [00:36<00:04, 324.45it/s]
Reading training images (A):  89%|########8 | 12205/13750 [00:36<00:04, 328.34it/s]
Reading training images (A):  89%|########9 | 12246/13750 [00:36<00:04, 344.49it/s]
Reading training images (A):  89%|########9 | 12284/13750 [00:36<00:04, 343.15it/s]
Reading training images (A):  90%|########9 | 12319/13750 [00:36<00:04, 304.67it/s]
Reading training images (A):  90%|########9 | 12351/13750 [00:36<00:04, 301.61it/s]
Reading training images (A):  90%|######### | 12382/13750 [00:36<00:04, 292.27it/s]
Reading training images (A):  90%|######### | 12422/13750 [00:36<00:04, 302.99it/s]
Reading training images (A):  91%|######### | 12488/13750 [00:36<00:03, 355.49it/s]
Reading training images (A):  91%|#########1| 12531/13750 [00:37<00:03, 372.37it/s]
Reading training images (A):  91%|#########1| 12572/13750 [00:37<00:03, 348.60it/s]
Reading training images (A):  92%|#########1| 12610/13750 [00:37<00:03, 335.14it/s]
Reading training images (A):  92%|#########1| 12646/13750 [00:37<00:03, 325.86it/s]
Reading training images (A):  92%|#########2| 12686/13750 [00:37<00:03, 330.34it/s]
Reading training images (A):  93%|#########2| 12733/13750 [00:37<00:02, 354.93it/s]
Reading training images (A):  93%|#########3| 12797/13750 [00:37<00:02, 406.22it/s]
Reading training images (A):  94%|#########3| 12862/13750 [00:37<00:01, 457.42it/s]
Reading training images (A):  94%|#########3| 12915/13750 [00:38<00:01, 475.31it/s]
Reading training images (A):  94%|#########4| 12974/13750 [00:38<00:01, 504.20it/s]
Reading training images (A):  95%|#########4| 13028/13750 [00:38<00:01, 423.20it/s]
Reading training images (A):  95%|#########5| 13077/13750 [00:38<00:01, 425.03it/s]
Reading training images (A):  95%|#########5| 13123/13750 [00:38<00:01, 430.78it/s]
Reading training images (A):  96%|#########5| 13169/13750 [00:38<00:01, 375.16it/s]
Reading training images (A):  96%|#########6| 13210/13750 [00:38<00:01, 377.13it/s]
Reading training images (A):  96%|#########6| 13250/13750 [00:38<00:01, 327.81it/s]
Reading training images (A):  97%|#########6| 13286/13750 [00:39<00:01, 312.03it/s]
Reading training images (A):  97%|#########6| 13320/13750 [00:39<00:01, 302.26it/s]
Reading training images (A):  97%|#########7| 13357/13750 [00:39<00:01, 300.81it/s]
Reading training images (A):  97%|#########7| 13390/13750 [00:39<00:01, 294.75it/s]
Reading training images (A):  98%|#########7| 13430/13750 [00:39<00:01, 301.41it/s]
Reading training images (A):  98%|#########7| 13461/13750 [00:39<00:00, 299.70it/s]
Reading training images (A):  98%|#########8| 13504/13750 [00:39<00:00, 329.35it/s]
Reading training images (A):  98%|#########8| 13540/13750 [00:39<00:00, 336.54it/s]
Reading training images (A):  99%|#########8| 13575/13750 [00:39<00:00, 309.18it/s]
Reading training images (A):  99%|#########9| 13621/13750 [00:40<00:00, 328.70it/s]
Reading training images (A):  99%|#########9| 13664/13750 [00:40<00:00, 352.38it/s]
Reading training images (A): 100%|#########9| 13728/13750 [00:40<00:00, 392.53it/s]


Reading training images (B):   0%|          | 0/1370 [00:00<?, ?it/s]
Reading training images (B):   2%|2         | 33/1370 [00:00<00:04, 291.15it/s]
Reading training images (B):   6%|5         | 81/1370 [00:00<00:03, 328.37it/s]
Reading training images (B):   9%|8         | 117/1370 [00:00<00:03, 316.02it/s]
Reading training images (B):  12%|#1        | 161/1370 [00:00<00:03, 343.82it/s]
Reading training images (B):  15%|#4        | 201/1370 [00:00<00:03, 348.02it/s]
Reading training images (B):  18%|#8        | 249/1370 [00:00<00:03, 373.34it/s]
Reading training images (B):  21%|##        | 285/1370 [00:00<00:03, 342.03it/s]
Reading training images (B):  24%|##4       | 332/1370 [00:00<00:02, 358.51it/s]
Reading training images (B):  29%|##9       | 400/1370 [00:01<00:02, 417.62it/s]
Reading training images (B):  33%|###2      | 446/1370 [00:01<00:02, 383.76it/s]
Reading training images (B):  37%|###7      | 509/1370 [00:01<00:01, 434.39it/s]
Reading training images (B):  43%|####2     | 584/1370 [00:01<00:01, 468.86it/s]
Reading training images (B):  46%|####6     | 635/1370 [00:01<00:01, 431.00it/s]
Reading training images (B):  51%|#####     | 694/1370 [00:01<00:01, 467.53it/s]
Reading training images (B):  54%|#####4    | 744/1370 [00:01<00:01, 474.16it/s]
Reading training images (B):  59%|#####9    | 812/1370 [00:01<00:01, 508.30it/s]
Reading training images (B):  63%|######3   | 866/1370 [00:01<00:01, 495.41it/s]
Reading training images (B):  69%|######9   | 950/1370 [00:02<00:00, 562.36it/s]
Reading training images (B):  75%|#######5  | 1034/1370 [00:02<00:00, 622.50it/s]
Reading training images (B):  80%|########  | 1102/1370 [00:02<00:00, 599.31it/s]
Reading training images (B):  85%|########5 | 1166/1370 [00:02<00:00, 560.63it/s]
Reading training images (B):  89%|########9 | 1226/1370 [00:02<00:00, 472.36it/s]
Reading training images (B):  93%|#########3| 1278/1370 [00:02<00:00, 407.15it/s]
Reading training images (B):  97%|#########6| 1324/1370 [00:02<00:00, 368.34it/s]
Reading training images (B): 100%|#########9| 1365/1370 [00:03<00:00, 359.00it/s]
08/17/2020 17:07:31 INFO     Reading alignments from: 'J:\SYNC\Sync\ENCRYPTSYNC\faceswap\src\MODELA\TRAINING\alignments.fsa'
08/17/2020 17:07:34 INFO     Reading alignments from: 'J:\SYNC\Sync\ENCRYPTSYNC\faceswap\src\MODELB\2\Alignments\Training\alignments.fsa'
08/17/2020 17:07:36 INFO     Enabled TensorBoard Logging

08/17/2020 17:07:51 ERROR    Caught exception in thread: '_training_0'
08/17/2020 17:07:55 ERROR    Got Exception on main handler:
Traceback (most recent call last):
File "C:\Users\ME\faceswap\lib\cli\launcher.py", line 156, in execute_script
process.process()
File "C:\Users\ME\faceswap\scripts\train.py", line 135, in process
self._end_thread(thread, err)
File "C:\Users\ME\faceswap\scripts\train.py", line 175, in _end_thread
thread.join()
File "C:\Users\ME\faceswap\lib\multithreading.py", line 121, in join
raise thread.err[1].with_traceback(thread.err[2])
File "C:\Users\ME\faceswap\lib\multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\ME\faceswap\scripts\train.py", line 197, in _training
raise err
File "C:\Users\ME\faceswap\scripts\train.py", line 187, in _training
self._run_training_cycle(model, trainer)
File "C:\Users\ME\faceswap\scripts\train.py", line 268, in _run_training_cycle
trainer.train_one_step(viewer, timelapse)
File "C:\Users\ME\faceswap\plugins\train\trainer\_base.py", line 227, in train_one_step
self._timelapse.output_timelapse(timelapse_kwargs)
File "C:\Users\ME\faceswap\plugins\train\trainer\_base.py", line 988, in output_timelapse
self._samples.images = self._feeder.compile_timelapse_sample()
File "C:\Users\ME\faceswap\plugins\train\trainer\_base.py", line 530, in compile_timelapse_sample
batch = next(self._display_feeds["timelapse"][side])
File "C:\Users\ME\faceswap\lib\multithreading.py", line 156, in iterator
self.check_and_raise_error()
File "C:\Users\ME\faceswap\lib\multithreading.py", line 84, in check_and_raise_error
raise error[1].with_traceback(error[2])
File "C:\Users\ME\faceswap\lib\multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\ME\faceswap\lib\multithreading.py", line 145, in _run
for item in self.generator(*self._gen_args, **self._gen_kwargs):
File "C:\Users\ME\faceswap\lib\training_data.py", line 186, in _minibatch
yield self._process_batch(img_paths, side)
File "C:\Users\ME\faceswap\lib\training_data.py", line 195, in _process_batch
batch = self._apply_mask(filenames, batch, side)
File "C:\Users\ME\faceswap\lib\training_data.py", line 249, in _apply_mask
masks = np.array([self._masks[side][filename].mask
File "C:\Users\ME\faceswap\lib\training_data.py", line 249, in <listcomp>
masks = np.array([self._masks[side][filename].mask
KeyError: 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\00312.png'
08/17/2020 17:07:55 CRITICAL An unexpected crash has occurred. Crash report written to 'C:\Users\ME\faceswap\crash_report.2020.08.17.170751279960.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting

Process exited.

The character limit was reached in this post so I will share the crash report in a second post.

Top
User avatar
Mr.Man
Posts: 9
Joined: Thu Jun 25, 2020 9:28 pm
Answers: 0

Re: Training crashes before start: ‘Enabled TensorBoard Logging'

Post by Mr.Man »

And here is the crash report:

Code: Select all

08/17/2020 17:07:50 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
08/17/2020 17:07:50 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'a' shape: (14, 128, 128, 3))
08/17/2020 17:07:50 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
08/17/2020 17:07:50 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'a' shape: (14, 128, 128, 3))
08/17/2020 17:07:50 MainProcess     _training_0     _base           _get_headers              DEBUG    side: 'a', width: 128
08/17/2020 17:07:50 MainProcess     _training_0     _base           _get_headers              DEBUG    height: 32, total_width: 384
08/17/2020 17:07:50 MainProcess     _training_0     _base           _get_headers              DEBUG    texts: ['Original (A)', 'Original > Original', 'Original > Swap'], text_sizes: [(72, 9), (116, 9), (102, 9)], text_x: [28, 134, 269], text_y: 20
08/17/2020 17:07:50 MainProcess     _training_0     _base           _get_headers              DEBUG    header_box.shape: (32, 384, 3)
08/17/2020 17:07:50 MainProcess     _training_0     _base           _to_full_frame            DEBUG    side: 'b', number of sample arrays: 3, prediction.shapes: [(14, 128, 128, 3), (14, 128, 128, 3)])
08/17/2020 17:07:50 MainProcess     _training_0     _base           _frame_overlay            DEBUG    full_size: 256, target_size: 176, color: (0, 0, 255)
08/17/2020 17:07:50 MainProcess     _training_0     _base           _frame_overlay            DEBUG    Overlayed background. Shape: (14, 256, 256, 3)
08/17/2020 17:07:50 MainProcess     _training_0     _base           _compile_masked           DEBUG    masked shapes: [(14, 128, 128, 3), (14, 128, 128, 3), (14, 128, 128, 3)]
08/17/2020 17:07:50 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 128, 128, 3), target_size: 176, scale: 1.375)
08/17/2020 17:07:50 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 176, 176, 3))
08/17/2020 17:07:50 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 128, 128, 3), target_size: 176, scale: 1.375)
08/17/2020 17:07:50 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 176, 176, 3))
08/17/2020 17:07:50 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 128, 128, 3), target_size: 176, scale: 1.375)
08/17/2020 17:07:50 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 176, 176, 3))
08/17/2020 17:07:50 MainProcess     _training_0     _base           _overlay_foreground       DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
08/17/2020 17:07:50 MainProcess     _training_0     _base           _overlay_foreground       DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
08/17/2020 17:07:50 MainProcess     _training_0     _base           _overlay_foreground       DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
08/17/2020 17:07:50 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
08/17/2020 17:07:50 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 128, 128, 3))
08/17/2020 17:07:50 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
08/17/2020 17:07:50 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 128, 128, 3))
08/17/2020 17:07:50 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
08/17/2020 17:07:50 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 128, 128, 3))
08/17/2020 17:07:50 MainProcess     _training_0     _base           _get_headers              DEBUG    side: 'b', width: 128
08/17/2020 17:07:50 MainProcess     _training_0     _base           _get_headers              DEBUG    height: 32, total_width: 384
08/17/2020 17:07:50 MainProcess     _training_0     _base           _get_headers              DEBUG    texts: ['Swap (B)', 'Swap > Swap', 'Swap > Original'], text_sizes: [(59, 9), (87, 9), (102, 9)], text_x: [34, 148, 269], text_y: 20
08/17/2020 17:07:50 MainProcess     _training_0     _base           _get_headers              DEBUG    header_box.shape: (32, 384, 3)
08/17/2020 17:07:50 MainProcess     _training_0     _base           _duplicate_headers        DEBUG    side: a header.shape: (32, 384, 3)
08/17/2020 17:07:50 MainProcess     _training_0     _base           _duplicate_headers        DEBUG    side: b header.shape: (32, 384, 3)
08/17/2020 17:07:50 MainProcess     _training_0     _base           _stack_images             DEBUG    Stack images
08/17/2020 17:07:50 MainProcess     _training_0     _base           get_transpose_axes        DEBUG    Even number of images to stack
08/17/2020 17:07:50 MainProcess     _training_0     _base           _stack_images             DEBUG    Stacked images
08/17/2020 17:07:50 MainProcess     _training_0     _base           show_sample               DEBUG    Compiled sample
08/17/2020 17:07:51 MainProcess     _training_0     train           _show                     DEBUG    Updating preview: (name: Training - 'S': Save Now. 'R': Refresh Preview. 'ENTER': Save and Quit)
08/17/2020 17:07:51 MainProcess     _training_0     train           _show                     DEBUG    Generating preview for GUI
08/17/2020 17:07:51 MainProcess     _training_0     train           _show                     DEBUG    Generated preview for GUI: '.gui_training_preview.jpg'
08/17/2020 17:07:51 MainProcess     _training_0     train           _show                     DEBUG    Updated preview: (name: Training - 'S': Save Now. 'R': Refresh Preview. 'ENTER': Save and Quit)
08/17/2020 17:07:51 MainProcess     _training_0     _base           output_timelapse          DEBUG    Ouputting time-lapse
08/17/2020 17:07:51 MainProcess     _training_0     _base           _setup                    DEBUG    Setting up time-lapse
08/17/2020 17:07:51 MainProcess     _training_0     _base           _setup                    DEBUG    Time-lapse output set to 'J:\SYNC\Sync\ENCRYPTSYNC\faceswap\src\MODELB\2\Timelapse'
08/17/2020 17:07:51 MainProcess     _training_0     utils           get_image_paths           DEBUG    Scanned Folder contains 14 files
08/17/2020 17:07:51 MainProcess     _training_0     utils           get_image_paths           DEBUG    Returning 14 images
08/17/2020 17:07:51 MainProcess     _training_0     utils           get_image_paths           DEBUG    Scanned Folder contains 14 files
08/17/2020 17:07:51 MainProcess     _training_0     utils           get_image_paths           DEBUG    Returning 14 images
08/17/2020 17:07:51 MainProcess     _training_0     _base           set_timelapse_feed        DEBUG    Setting time-lapse feed: (input_images: '{'a': ['J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\00312.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\00708.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\01187.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\02059.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\02241.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\04517.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\05259.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\05779.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\07367.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\08420.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\08785.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\08834.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\10258.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\13233.png'], 'b': ['J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELB\\2\\B FACES\\00067.jpg', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELB\\2\\B FACES\\00495.jpg', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELB\\2\\B FACES\\01251.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELB\\2\\B FACES\\01407.jpg', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELB\\2\\B FACES\\01432.jpg', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELB\\2\\B FACES\\02060.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELB\\2\\B FACES\\02352.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELB\\2\\B FACES\\02388.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELB\\2\\B FACES\\02462.jpg', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELB\\2\\B FACES\\02805.jpg', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELB\\2\\B FACES\\02866.png', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELB\\2\\B FACES\\02952.jpg', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELB\\2\\B FACES\\02992.jpg', 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELB\\2\\B FACES\\03040.jpg']}', batch_size: 14)
08/17/2020 17:07:51 MainProcess     _training_0     _base           _load_generator           DEBUG    Loading generator
08/17/2020 17:07:51 MainProcess     _training_0     _base           _load_generator           DEBUG    input_size: 128, output_shapes: [(128, 128, 3)]
08/17/2020 17:07:51 MainProcess     _training_0     _base           coverage_ratio            DEBUG    Requested coverage_ratio: 0.6875
08/17/2020 17:07:51 MainProcess     _training_0     _base           coverage_ratio            DEBUG    Final coverage_ratio: 0.6875
08/17/2020 17:07:51 MainProcess     _training_0     training_data   __init__                  DEBUG    Initializing TrainingDataGenerator: (model_input_size: 128, model_output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, augment_color: True, no_flip: False, warp_to_landmarks: False, alignments: ['masks'], config: {'coverage': 68.75, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'icnr_init': False, 'conv_aware_init': False, 'reflect_padding': False, 'allow_growth': False, 'penalized_mask_loss': True, 'loss_function': 'mae', 'learning_rate': 5e-05, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
08/17/2020 17:07:51 MainProcess     _training_0     training_data   __init__                  DEBUG    Initialized TrainingDataGenerator
08/17/2020 17:07:51 MainProcess     _training_0     training_data   minibatch_ab              DEBUG    Queue batches: (image_count: 14, batchsize: 14, side: 'a', do_shuffle: False, is_preview, False, is_timelapse: True)
08/17/2020 17:07:51 MainProcess     _training_0     training_data   __init__                  DEBUG    Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 128, output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, config: {'coverage': 68.75, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'icnr_init': False, 'conv_aware_init': False, 'reflect_padding': False, 'allow_growth': False, 'penalized_mask_loss': True, 'loss_function': 'mae', 'learning_rate': 5e-05, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
08/17/2020 17:07:51 MainProcess     _training_0     training_data   __init__                  DEBUG    Output sizes: [128]
08/17/2020 17:07:51 MainProcess     _training_0     training_data   __init__                  DEBUG    Initialized ImageAugmentation
08/17/2020 17:07:51 MainProcess     _training_0     multithreading  __init__                  DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
08/17/2020 17:07:51 MainProcess     _training_0     multithreading  __init__                  DEBUG    Initialized BackgroundGenerator: '_run'
08/17/2020 17:07:51 MainProcess     _training_0     multithreading  start                     DEBUG    Starting thread(s): '_run'
08/17/2020 17:07:51 MainProcess     _training_0     multithreading  start                     DEBUG    Starting thread 1 of 2: '_run_0'
08/17/2020 17:07:51 MainProcess     _run_0          training_data   _minibatch                DEBUG    Loading minibatch generator: (image_count: 14, side: 'a', do_shuffle: False)
08/17/2020 17:07:51 MainProcess     _training_0     multithreading  start                     DEBUG    Starting thread 2 of 2: '_run_1'
08/17/2020 17:07:51 MainProcess     _run_1          training_data   _minibatch                DEBUG    Loading minibatch generator: (image_count: 14, side: 'a', do_shuffle: False)
08/17/2020 17:07:51 MainProcess     _training_0     multithreading  start                     DEBUG    Started all threads '_run': 2
08/17/2020 17:07:51 MainProcess     _training_0     _base           _load_generator           DEBUG    Loading generator
08/17/2020 17:07:51 MainProcess     _training_0     _base           _load_generator           DEBUG    input_size: 128, output_shapes: [(128, 128, 3)]
08/17/2020 17:07:51 MainProcess     _training_0     _base           coverage_ratio            DEBUG    Requested coverage_ratio: 0.6875
08/17/2020 17:07:51 MainProcess     _training_0     _base           coverage_ratio            DEBUG    Final coverage_ratio: 0.6875
08/17/2020 17:07:51 MainProcess     _training_0     training_data   __init__                  DEBUG    Initializing TrainingDataGenerator: (model_input_size: 128, model_output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, augment_color: True, no_flip: False, warp_to_landmarks: False, alignments: ['masks'], config: {'coverage': 68.75, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'icnr_init': False, 'conv_aware_init': False, 'reflect_padding': False, 'allow_growth': False, 'penalized_mask_loss': True, 'loss_function': 'mae', 'learning_rate': 5e-05, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
08/17/2020 17:07:51 MainProcess     _training_0     training_data   __init__                  DEBUG    Initialized TrainingDataGenerator
08/17/2020 17:07:51 MainProcess     _training_0     training_data   minibatch_ab              DEBUG    Queue batches: (image_count: 14, batchsize: 14, side: 'b', do_shuffle: False, is_preview, False, is_timelapse: True)
08/17/2020 17:07:51 MainProcess     _training_0     training_data   __init__                  DEBUG    Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 128, output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, config: {'coverage': 68.75, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'icnr_init': False, 'conv_aware_init': False, 'reflect_padding': False, 'allow_growth': False, 'penalized_mask_loss': True, 'loss_function': 'mae', 'learning_rate': 5e-05, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
08/17/2020 17:07:51 MainProcess     _training_0     training_data   __init__                  DEBUG    Output sizes: [128]
08/17/2020 17:07:51 MainProcess     _training_0     training_data   __init__                  DEBUG    Initialized ImageAugmentation
08/17/2020 17:07:51 MainProcess     _training_0     multithreading  __init__                  DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
08/17/2020 17:07:51 MainProcess     _training_0     multithreading  __init__                  DEBUG    Initialized BackgroundGenerator: '_run'
08/17/2020 17:07:51 MainProcess     _training_0     multithreading  start                     DEBUG    Starting thread(s): '_run'
08/17/2020 17:07:51 MainProcess     _training_0     multithreading  start                     DEBUG    Starting thread 1 of 2: '_run_0'
08/17/2020 17:07:51 MainProcess     _run_0          training_data   _minibatch                DEBUG    Loading minibatch generator: (image_count: 14, side: 'b', do_shuffle: False)
08/17/2020 17:07:51 MainProcess     _training_0     multithreading  start                     DEBUG    Starting thread 2 of 2: '_run_1'
08/17/2020 17:07:51 MainProcess     _run_1          training_data   _minibatch                DEBUG    Loading minibatch generator: (image_count: 14, side: 'b', do_shuffle: False)
08/17/2020 17:07:51 MainProcess     _training_0     multithreading  start                     DEBUG    Started all threads '_run': 2
08/17/2020 17:07:51 MainProcess     _training_0     _base           set_timelapse_feed        DEBUG    Set time-lapse feed: {'a': <generator object BackgroundGenerator.iterator at 0x000001B9D7C030B0>, 'b': <generator object BackgroundGenerator.iterator at 0x000001B9D7C03350>}
08/17/2020 17:07:51 MainProcess     _training_0     _base           _setup                    DEBUG    Set up time-lapse
08/17/2020 17:07:51 MainProcess     _training_0     _base           output_timelapse          DEBUG    Getting time-lapse samples
08/17/2020 17:07:51 MainProcess     _run_1          multithreading  run                       DEBUG    Error in thread (_run_1): 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELB\\2\\B FACES\\00067.jpg'
08/17/2020 17:07:51 MainProcess     _run_1          multithreading  run                       DEBUG    Error in thread (_run_1): 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\00312.png'
08/17/2020 17:07:51 MainProcess     _training_0     multithreading  check_and_raise_error     DEBUG    Thread error caught: [(<class 'KeyError'>, KeyError('J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\00312.png'), <traceback object at 0x000001B9E4307640>)]
08/17/2020 17:07:51 MainProcess     _training_0     multithreading  run                       DEBUG    Error in thread (_training_0): 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\00312.png'
08/17/2020 17:07:51 MainProcess     _run_0          multithreading  run                       DEBUG    Error in thread (_run_0): 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\00312.png'
08/17/2020 17:07:51 MainProcess     MainThread      train           _monitor                  DEBUG    Thread error detected
08/17/2020 17:07:51 MainProcess     MainThread      train           _monitor                  DEBUG    Closed Monitor
08/17/2020 17:07:51 MainProcess     MainThread      train           _end_thread               DEBUG    Ending Training thread
08/17/2020 17:07:51 MainProcess     MainThread      train           _end_thread               CRITICAL Error caught! Exiting...
08/17/2020 17:07:51 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Threads: '_training'
08/17/2020 17:07:51 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Thread: '_training_0'
08/17/2020 17:07:51 MainProcess     MainThread      multithreading  join                      ERROR    Caught exception in thread: '_training_0'
Traceback (most recent call last):
  File "C:\Users\ME\faceswap\lib\cli\launcher.py", line 156, in execute_script
    process.process()
  File "C:\Users\ME\faceswap\scripts\train.py", line 135, in process
    self._end_thread(thread, err)
  File "C:\Users\ME\faceswap\scripts\train.py", line 175, in _end_thread
    thread.join()
  File "C:\Users\ME\faceswap\lib\multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "C:\Users\ME\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\ME\faceswap\scripts\train.py", line 197, in _training
    raise err
  File "C:\Users\ME\faceswap\scripts\train.py", line 187, in _training
    self._run_training_cycle(model, trainer)
  File "C:\Users\ME\faceswap\scripts\train.py", line 268, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "C:\Users\ME\faceswap\plugins\train\trainer\_base.py", line 227, in train_one_step
    self._timelapse.output_timelapse(timelapse_kwargs)
  File "C:\Users\ME\faceswap\plugins\train\trainer\_base.py", line 988, in output_timelapse
    self._samples.images = self._feeder.compile_timelapse_sample()
  File "C:\Users\ME\faceswap\plugins\train\trainer\_base.py", line 530, in compile_timelapse_sample
    batch = next(self._display_feeds["timelapse"][side])
  File "C:\Users\ME\faceswap\lib\multithreading.py", line 156, in iterator
    self.check_and_raise_error()
  File "C:\Users\ME\faceswap\lib\multithreading.py", line 84, in check_and_raise_error
    raise error[1].with_traceback(error[2])
  File "C:\Users\ME\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\ME\faceswap\lib\multithreading.py", line 145, in _run
    for item in self.generator(*self._gen_args, **self._gen_kwargs):
  File "C:\Users\ME\faceswap\lib\training_data.py", line 186, in _minibatch
    yield self._process_batch(img_paths, side)
  File "C:\Users\ME\faceswap\lib\training_data.py", line 195, in _process_batch
    batch = self._apply_mask(filenames, batch, side)
  File "C:\Users\ME\faceswap\lib\training_data.py", line 249, in _apply_mask
    masks = np.array([self._masks[side][filename].mask
  File "C:\Users\ME\faceswap\lib\training_data.py", line 249, in <listcomp>
    masks = np.array([self._masks[side][filename].mask
KeyError: 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\00312.png'

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         6b2aac6 Enable MTCNN for CPU extraction
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce GTX 1070 Ti
gpu_devices_active:  GPU_0
gpu_driver:          452.06
gpu_vram:            GPU_0: 8192MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.16299-SP0
os_release:          10
py_command:          C:\Users\ME\faceswap\faceswap.py train -A J:/SYNC/Sync/ENCRYPTSYNC/faceswap/src/MODELA/TRAINING -ala J:/SYNC/Sync/ENCRYPTSYNC/faceswap/src/MODELA/TRAINING/alignments.fsa -B J:/SYNC/Sync/ENCRYPTSYNC/faceswap/src/MODELB/2/Training2 -alb J:/SYNC/Sync/ENCRYPTSYNC/faceswap/src/MODELB/2/Alignments/Training/alignments.fsa -m J:/SYNC/Sync/ENCRYPTSYNC/faceswap/src/MODELB/2/CONVERT -t villain -bs 8 -it 1000000 -s 250 -ss 25000 -tia J:/SYNC/Sync/ENCRYPTSYNC/faceswap/src/MODELA/A FACES -tib J:/SYNC/Sync/ENCRYPTSYNC/faceswap/src/MODELB/2/B FACES -to J:/SYNC/Sync/ENCRYPTSYNC/faceswap/src/MODELB/2/Timelapse -ps 50 -L INFO -gui
py_conda_version:    conda 4.8.4
py_implementation:   CPython
py_version:          3.8.5
py_virtual_env:      True
sys_cores:           6
sys_processor:       Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
sys_ram:             Total: 16344MB, Available: 10806MB, Used: 5538MB, Free: 10806MB

=============== Pip Packages ===============
absl-py==0.9.0
astunparse==1.6.3
cachetools==4.1.1
certifi==2020.6.20
chardet==3.0.4
cycler==0.10.0
fastcluster==1.1.26
ffmpy==0.2.3
gast==0.3.3
google-auth==1.20.1
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.31.0
h5py==2.10.0
idna==2.10
imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1589202782679/work
joblib @ file:///tmp/build/80754af9/joblib_1594236160679/work
Keras-Preprocessing==1.1.2
kiwisolver==1.2.0
Markdown==3.2.2
matplotlib @ file:///C:/ci/matplotlib-base_1592837548929/work
mkl-fft==1.1.0
mkl-random==1.1.1
mkl-service==2.3.0
numpy @ file:///C:/ci/numpy_and_numpy_base_1596215850360/work
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib==3.1.0
olefile==0.46
opencv-python==4.4.0.40
opt-einsum==3.3.0
pathlib==1.0.1
Pillow @ file:///C:/ci/pillow_1594298230227/work
protobuf==3.13.0
psutil==5.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==2.4.7
python-dateutil==2.8.1
pywin32==227
requests==2.24.0
requests-oauthlib==1.3.0
rsa==4.6
scikit-learn @ file:///C:/ci/scikit-learn_1592853510272/work
scipy==1.4.1
sip==4.19.13
six==1.15.0
tensorboard==2.2.2
tensorboard-plugin-wit==1.7.0
tensorflow==2.2.0
tensorflow-estimator==2.2.0
tensorflow-gpu==2.2.0
tensorflow-gpu-estimator==2.2.0
termcolor==1.1.0
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
tornado==6.0.4
tqdm @ file:///tmp/build/80754af9/tqdm_1596810128862/work
urllib3==1.25.10
Werkzeug==1.0.1
wincertstore==0.2
wrapt==1.12.1

============== Conda Packages ==============
# packages in environment at C:\Users\ME\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
absl-py                   0.9.0                    pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
blas                      1.0                         mkl  

ca-certificates           2020.6.24                     0  

cachetools                4.1.1                    pypi_0    pypi
certifi                   2020.6.20                py38_0  

chardet                   3.0.4                    pypi_0    pypi
cudatoolkit               10.1.243             h74a9793_0  

cudnn                     7.6.5                cuda10.1_0  

cycler                    0.10.0                   py38_0  

fastcluster               1.1.26           py38hbe40bda_1    conda-forge
ffmpeg                    4.3.1                ha925a31_0    conda-forge
ffmpy                     0.2.3                    pypi_0    pypi
freetype                  2.10.2               hd328e21_0  

gast                      0.3.3                    pypi_0    pypi
git                       2.23.0               h6bb4b03_0  

google-auth               1.20.1                   pypi_0    pypi
google-auth-oauthlib      0.4.1                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.31.0                   pypi_0    pypi
h5py                      2.10.0                   pypi_0    pypi
icc_rt                    2019.0.0             h0cc432a_1  

icu                       58.2                 ha925a31_3  

idna                      2.10                     pypi_0    pypi
imageio                   2.9.0                      py_0  

imageio-ffmpeg            0.4.2                      py_0    conda-forge
intel-openmp              2020.1                      216  

joblib                    0.16.0                     py_0  

jpeg                      9b                   hb83a4c4_2  

keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.2.0            py38h74a9793_0  

libpng                    1.6.37               h2a8f88b_0  

libtiff                   4.1.0                h56a325e_1  

lz4-c                     1.9.2                h62dcd97_1  

markdown                  3.2.2                    pypi_0    pypi
matplotlib                3.2.2                         0  

matplotlib-base           3.2.2            py38h64f37c6_0  

mkl                       2020.1                      216  

mkl-service               2.3.0            py38hb782905_0  

mkl_fft                   1.1.0            py38h45dec08_0  

mkl_random                1.1.1            py38h47e9c7a_0  

numpy                     1.19.1           py38h5510c5b_0  

numpy-base                1.19.1           py38ha3acd2a_0  

nvidia-ml-py3             7.352.1                  pypi_0    pypi
oauthlib                  3.1.0                    pypi_0    pypi
olefile                   0.46                       py_0  

opencv-python             4.4.0.40                 pypi_0    pypi
openssl                   1.1.1g               he774522_1  

opt-einsum                3.3.0                    pypi_0    pypi
pathlib                   1.0.1                      py_1  

pillow                    7.2.0            py38hcc1f983_0  

pip                       20.2.2                   py38_0  

protobuf                  3.13.0                   pypi_0    pypi
psutil                    5.7.0            py38he774522_0  

pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pyparsing                 2.4.7                      py_0  

pyqt                      5.9.2            py38ha925a31_4  

python                    3.8.5                he1778fa_0  

python-dateutil           2.8.1                      py_0  

python_abi                3.8                      1_cp38    conda-forge
pywin32                   227              py38he774522_1  

qt                        5.9.7            vc14h73c81de_0  

requests                  2.24.0                   pypi_0    pypi
requests-oauthlib         1.3.0                    pypi_0    pypi
rsa                       4.6                      pypi_0    pypi
scikit-learn              0.23.1           py38h25d0782_0  

scipy                     1.4.1                    pypi_0    pypi
setuptools                49.6.0                   py38_0  

sip                       4.19.13          py38ha925a31_0  

six                       1.15.0                     py_0  

sqlite                    3.32.3               h2a8f88b_0  

tensorboard               2.2.2                    pypi_0    pypi
tensorboard-plugin-wit    1.7.0                    pypi_0    pypi
tensorflow                2.2.0                    pypi_0    pypi
tensorflow-estimator      2.2.0                    pypi_0    pypi
tensorflow-gpu            2.2.0                    pypi_0    pypi
tensorflow-gpu-estimator  2.2.0                    pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
threadpoolctl             2.1.0              pyh5ca1d4c_0  

tk                        8.6.10               he774522_0  

tornado                   6.0.4            py38he774522_1  

tqdm                      4.48.2                     py_0  

urllib3                   1.25.10                  pypi_0    pypi
vc                        14.1                 h0510ff6_4  

vs2015_runtime            14.16.27012          hf0eaf9b_3  

werkzeug                  1.0.1                    pypi_0    pypi
wheel                     0.34.2                   py38_0  

wincertstore              0.2                      py38_0  

wrapt                     1.12.1                   pypi_0    pypi
xz                        5.2.5                h62dcd97_0  

zlib                      1.2.11               h62dcd97_4  

zstd                      1.4.5                h04227a9_0  

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[mask.box_blend]
type:                     gaussian
distance:                 11.0
radius:                   5.0
passes:                   1

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0

[scaling.sharpen]
method:                   unsharp_mask
amount:                   150
radius:                   0.3
threshold:                5.0

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
jpg_quality:              75
png_compress_level:       3

[writer.pillow]
format:                   png
draw_transparent:         False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

--------- extract.ini ---------

[global]
allow_growth:             False

[align.fan]
batch-size:               12

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7
scalefactor:              0.709
batch-size:               8

[detect.s3fd]
confidence:               70
batch-size:               4

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

--------- gui.ini ---------

[global]
fullscreen:               False
tab:                      extract
options_panel_width:      30
console_panel_height:     20
icon_size:                14
font:                     default
font_size:                9
autosave_last_session:    prompt
timeout:                  120
auto_load_model_stats:    True

--------- train.ini ---------

[global]
coverage:                 68.75
mask_type:                extended
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False
icnr_init:                False
conv_aware_init:          False
reflect_padding:          False
allow_growth:             False
penalized_mask_loss:      True
loss_function:            mae
learning_rate:            5e-05

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
clipnorm:                 True
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.original]
lowmem:                   False

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.unbalanced]
input_size:               128
lowmem:                   False
clipnorm:                 True
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4

To confirm, I have checked and faceswap is up to date. Thanks for any help you can offer!

Top
User avatar
bryanlyon
Site Admin
Posts: 805
Joined: Fri Jul 12, 2019 12:49 am
Answers: 44
Location: San Francisco
Has thanked: 4 times
Been thanked: 224 times
Contact:
Contact bryanlyon

Re: Training crashes before start: ‘Enabled TensorBoard Logging'

Post by bryanlyon »

Code: Select all

KeyError: 'J:\\SYNC\\Sync\\ENCRYPTSYNC\\faceswap\\src\\MODELA\\A FACES\\00312.png'

This error says that this image isn't in the alignments that you provided. This may be because of cloud syncing. Cloud sync providers often compress images which will cause faceswap to fail due to the images not being correct anymore. It could also be a read error or something similar. Check first that all the folders are correct and only the correct images are included. If you need to re-create the images due to cloud compression you can use the alignment tool to re-extract the faces from the video much faster using the fsa alignments file.

Top
User avatar
Mr.Man
Posts: 9
Joined: Thu Jun 25, 2020 9:28 pm
Answers: 0

Re: Training crashes before start: ‘Enabled TensorBoard Logging'

Post by Mr.Man »

Thanks again - I'll give this a go.

Top
User avatar
Mr.Man
Posts: 9
Joined: Thu Jun 25, 2020 9:28 pm
Answers: 0

Re: Training crashes before start: ‘Enabled TensorBoard Logging'

Post by Mr.Man »

Unfortunately no such luck this time. I re-extracted MODELA and basically did the whole process again as have learnt a lot since first extraction so saw it as opportunity to improve anyway. I moved everything onto local storage.

It crashed on ‘ J:\\FS\\SRC\\MODELA\\A FACES\\00708.png’ which is NOT in the MODELA folder – I have cleaned the alignments files and merged them but I’m guessing there is another step I need to do?

I received the following reports:

Code: Select all

Loading...
Setting Faceswap backend to NVIDIA
08/18/2020 13:48:54 INFO     Log level set to: INFO
08/18/2020 13:48:56 INFO     Model A Directory: J:\FS\SRC\MODELA\TRAINING
08/18/2020 13:48:56 INFO     Model B Directory: J:\FS\SRC\MODELB\2\Training2
08/18/2020 13:48:56 INFO     Training data directory: J:\FS\SRC\MODELB\TRAININGMODEL
08/18/2020 13:48:56 INFO     ===================================================
08/18/2020 13:48:56 INFO       Starting
08/18/2020 13:48:56 INFO       Press 'Stop' to save and quit
08/18/2020 13:48:56 INFO     ===================================================
08/18/2020 13:48:57 INFO     Loading data, this may take a while...
08/18/2020 13:48:57 INFO     Loading Model from Villain plugin...
08/18/2020 13:48:57 INFO     No existing state file found. Generating.
08/18/2020 13:49:03 INFO     Loading Trainer from Original plugin...

Reading training images (A):   0%|          | 0/8777 [00:00<?, ?it/s]
Reading training images (A):   0%|          | 1/8777 [00:00<26:24,  5.54it/s]
Reading training images (A):   4%|4         | 352/8777 [00:00<17:45,  7.91it/s]
Reading training images (A):   7%|7         | 635/8777 [00:00<12:01, 11.28it/s]
Reading training images (A):  10%|#         | 891/8777 [00:00<08:10, 16.09it/s]
Reading training images (A):  13%|#3        | 1160/8777 [00:00<05:32, 22.93it/s]
Reading training images (A):  16%|#6        | 1415/8777 [00:00<03:45, 32.63it/s]
Reading training images (A):  19%|#9        | 1701/8777 [00:00<02:32, 46.38it/s]
Reading training images (A):  23%|##2       | 1984/8777 [00:00<01:43, 65.79it/s]
Reading training images (A):  26%|##5       | 2268/8777 [00:00<01:09, 93.07it/s]
Reading training images (A):  29%|##8       | 2538/8777 [00:01<00:47, 131.01it/s]
Reading training images (A):  32%|###2      | 2819/8777 [00:01<00:32, 183.37it/s]
Reading training images (A):  36%|###5      | 3121/8777 [00:01<00:22, 255.30it/s]
Reading training images (A):  39%|###8      | 3399/8777 [00:01<00:15, 350.34it/s]
Reading training images (A):  42%|####1     | 3686/8777 [00:01<00:10, 475.54it/s]
Reading training images (A):  45%|####5     | 3975/8777 [00:01<00:07, 634.48it/s]
Reading training images (A):  48%|####8     | 4256/8777 [00:01<00:05, 823.32it/s]
Reading training images (A):  52%|#####1    | 4534/8777 [00:01<00:04, 1042.21it/s]
Reading training images (A):  55%|#####4    | 4818/8777 [00:01<00:03, 1280.83it/s]
Reading training images (A):  58%|#####8    | 5100/8777 [00:02<00:02, 1530.95it/s]
Reading training images (A):  61%|######1   | 5378/8777 [00:02<00:01, 1738.61it/s]
Reading training images (A):  65%|######4   | 5664/8777 [00:02<00:01, 1969.30it/s]
Reading training images (A):  68%|######7   | 5948/8777 [00:02<00:01, 2167.45it/s]
Reading training images (A):  71%|#######1  | 6247/8777 [00:02<00:01, 2355.33it/s]
Reading training images (A):  74%|#######4  | 6531/8777 [00:02<00:00, 2474.19it/s]
Reading training images (A):  78%|#######7  | 6823/8777 [00:02<00:00, 2591.11it/s]
Reading training images (A):  81%|########  | 7108/8777 [00:02<00:00, 2646.84it/s]
Reading training images (A):  84%|########4 | 7396/8777 [00:02<00:00, 2710.68it/s]
Reading training images (A):  88%|########7 | 7681/8777 [00:02<00:00, 2748.90it/s]
Reading training images (A):  91%|######### | 7966/8777 [00:03<00:00, 2776.30it/s]
Reading training images (A):  94%|#########4| 8251/8777 [00:03<00:00, 2787.58it/s]
Reading training images (A):  97%|#########7| 8535/8777 [00:03<00:00, 2792.61it/s]


Reading training images (B):   0%|          | 0/1370 [00:00<?, ?it/s]
Reading training images (B):  22%|##1       | 299/1370 [00:00<00:00, 2982.05it/s]
Reading training images (B):  52%|#####2    | 714/1370 [00:00<00:00, 3254.99it/s]
Reading training images (B):  76%|#######5  | 1036/1370 [00:00<00:00, 3241.80it/s]
Reading training images (B):  98%|#########8| 1349/1370 [00:00<00:00, 3204.80it/s]
08/18/2020 13:49:07 INFO     Reading alignments from: 'J:\FS\SRC\MODELA\TRAINING\alignments.fsa'
08/18/2020 13:49:08 INFO     Reading alignments from: 'J:\FS\SRC\MODELB\2\Training2\alignments.fsa'

08/18/2020 13:49:24 ERROR    Caught exception in thread: '_training_0'
08/18/2020 13:49:29 ERROR    Got Exception on main handler:
Traceback (most recent call last):
File "C:\Users\ME\faceswap\lib\cli\launcher.py", line 156, in execute_script
process.process()
File "C:\Users\ME\faceswap\scripts\train.py", line 135, in process
self._end_thread(thread, err)
File "C:\Users\ME\faceswap\scripts\train.py", line 175, in _end_thread
thread.join()
File "C:\Users\ME\faceswap\lib\multithreading.py", line 121, in join
raise thread.err[1].with_traceback(thread.err[2])
File "C:\Users\ME\faceswap\lib\multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\ME\faceswap\scripts\train.py", line 197, in _training
raise err
File "C:\Users\ME\faceswap\scripts\train.py", line 187, in _training
self._run_training_cycle(model, trainer)
File "C:\Users\ME\faceswap\scripts\train.py", line 268, in _run_training_cycle
trainer.train_one_step(viewer, timelapse)
File "C:\Users\ME\faceswap\plugins\train\trainer\_base.py", line 227, in train_one_step
self._timelapse.output_timelapse(timelapse_kwargs)
File "C:\Users\ME\faceswap\plugins\train\trainer\_base.py", line 988, in output_timelapse
self._samples.images = self._feeder.compile_timelapse_sample()
File "C:\Users\ME\faceswap\plugins\train\trainer\_base.py", line 530, in compile_timelapse_sample
batch = next(self._display_feeds["timelapse"][side])
File "C:\Users\ME\faceswap\lib\multithreading.py", line 156, in iterator
self.check_and_raise_error()
File "C:\Users\ME\faceswap\lib\multithreading.py", line 84, in check_and_raise_error
raise error[1].with_traceback(error[2])
File "C:\Users\ME\faceswap\lib\multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\ME\faceswap\lib\multithreading.py", line 145, in _run
for item in self.generator(*self._gen_args, **self._gen_kwargs):
File "C:\Users\ME\faceswap\lib\training_data.py", line 186, in _minibatch
yield self._process_batch(img_paths, side)
File "C:\Users\ME\faceswap\lib\training_data.py", line 195, in _process_batch
batch = self._apply_mask(filenames, batch, side)
File "C:\Users\ME\faceswap\lib\training_data.py", line 249, in _apply_mask
masks = np.array([self._masks[side][filename].mask
File "C:\Users\ME\faceswap\lib\training_data.py", line 249, in <listcomp>
masks = np.array([self._masks[side][filename].mask
KeyError: 'J:\\FS\\SRC\\MODELA\\A FACES\\00708.png'
08/18/2020 13:49:29 CRITICAL An unexpected crash has occurred. Crash report written to 'C:\Users\ME\faceswap\crash_report.2020.08.18.134924772740.log'. You MUST provide this file if seeking assistance. Please verify you are running the latest version of faceswap before reporting

Process exited.

And the crash log:

Code: Select all

08/18/2020 13:49:23 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
08/18/2020 13:49:23 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'a' shape: (14, 128, 128, 3))
08/18/2020 13:49:23 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
08/18/2020 13:49:23 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'a' shape: (14, 128, 128, 3))
08/18/2020 13:49:23 MainProcess     _training_0     _base           _get_headers              DEBUG    side: 'a', width: 128
08/18/2020 13:49:23 MainProcess     _training_0     _base           _get_headers              DEBUG    height: 32, total_width: 384
08/18/2020 13:49:23 MainProcess     _training_0     _base           _get_headers              DEBUG    texts: ['Original (A)', 'Original > Original', 'Original > Swap'], text_sizes: [(72, 9), (116, 9), (102, 9)], text_x: [28, 134, 269], text_y: 20
08/18/2020 13:49:23 MainProcess     _training_0     _base           _get_headers              DEBUG    header_box.shape: (32, 384, 3)
08/18/2020 13:49:23 MainProcess     _training_0     _base           _to_full_frame            DEBUG    side: 'b', number of sample arrays: 3, prediction.shapes: [(14, 128, 128, 3), (14, 128, 128, 3)])
08/18/2020 13:49:23 MainProcess     _training_0     _base           _frame_overlay            DEBUG    full_size: 256, target_size: 176, color: (0, 0, 255)
08/18/2020 13:49:23 MainProcess     _training_0     _base           _frame_overlay            DEBUG    Overlayed background. Shape: (14, 256, 256, 3)
08/18/2020 13:49:23 MainProcess     _training_0     _base           _compile_masked           DEBUG    masked shapes: [(14, 128, 128, 3), (14, 128, 128, 3), (14, 128, 128, 3)]
08/18/2020 13:49:23 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 128, 128, 3), target_size: 176, scale: 1.375)
08/18/2020 13:49:23 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 176, 176, 3))
08/18/2020 13:49:23 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 128, 128, 3), target_size: 176, scale: 1.375)
08/18/2020 13:49:23 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 176, 176, 3))
08/18/2020 13:49:23 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 128, 128, 3), target_size: 176, scale: 1.375)
08/18/2020 13:49:23 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 176, 176, 3))
08/18/2020 13:49:23 MainProcess     _training_0     _base           _overlay_foreground       DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
08/18/2020 13:49:23 MainProcess     _training_0     _base           _overlay_foreground       DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
08/18/2020 13:49:23 MainProcess     _training_0     _base           _overlay_foreground       DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
08/18/2020 13:49:23 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
08/18/2020 13:49:23 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 128, 128, 3))
08/18/2020 13:49:23 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
08/18/2020 13:49:23 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 128, 128, 3))
08/18/2020 13:49:23 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
08/18/2020 13:49:23 MainProcess     _training_0     _base           _resize_sample            DEBUG    Resized sample: (side: 'b' shape: (14, 128, 128, 3))
08/18/2020 13:49:23 MainProcess     _training_0     _base           _get_headers              DEBUG    side: 'b', width: 128
08/18/2020 13:49:23 MainProcess     _training_0     _base           _get_headers              DEBUG    height: 32, total_width: 384
08/18/2020 13:49:23 MainProcess     _training_0     _base           _get_headers              DEBUG    texts: ['Swap (B)', 'Swap > Swap', 'Swap > Original'], text_sizes: [(59, 9), (87, 9), (102, 9)], text_x: [34, 148, 269], text_y: 20
08/18/2020 13:49:23 MainProcess     _training_0     _base           _get_headers              DEBUG    header_box.shape: (32, 384, 3)
08/18/2020 13:49:23 MainProcess     _training_0     _base           _duplicate_headers        DEBUG    side: a header.shape: (32, 384, 3)
08/18/2020 13:49:23 MainProcess     _training_0     _base           _duplicate_headers        DEBUG    side: b header.shape: (32, 384, 3)
08/18/2020 13:49:23 MainProcess     _training_0     _base           _stack_images             DEBUG    Stack images
08/18/2020 13:49:23 MainProcess     _training_0     _base           get_transpose_axes        DEBUG    Even number of images to stack
08/18/2020 13:49:23 MainProcess     _training_0     _base           _stack_images             DEBUG    Stacked images
08/18/2020 13:49:23 MainProcess     _training_0     _base           show_sample               DEBUG    Compiled sample
08/18/2020 13:49:24 MainProcess     _training_0     train           _show                     DEBUG    Updating preview: (name: Training - 'S': Save Now. 'R': Refresh Preview. 'ENTER': Save and Quit)
08/18/2020 13:49:24 MainProcess     _training_0     train           _show                     DEBUG    Generating preview for GUI
08/18/2020 13:49:24 MainProcess     _training_0     train           _show                     DEBUG    Generated preview for GUI: '.gui_training_preview.jpg'
08/18/2020 13:49:24 MainProcess     _training_0     train           _show                     DEBUG    Updated preview: (name: Training - 'S': Save Now. 'R': Refresh Preview. 'ENTER': Save and Quit)
08/18/2020 13:49:24 MainProcess     _training_0     _base           output_timelapse          DEBUG    Ouputting time-lapse
08/18/2020 13:49:24 MainProcess     _training_0     _base           _setup                    DEBUG    Setting up time-lapse
08/18/2020 13:49:24 MainProcess     _training_0     _base           _setup                    DEBUG    Time-lapse output set to 'J:\FS\SRC\MODELB\TIMELAPSE'
08/18/2020 13:49:24 MainProcess     _training_0     utils           get_image_paths           DEBUG    Scanned Folder contains 14 files
08/18/2020 13:49:24 MainProcess     _training_0     utils           get_image_paths           DEBUG    Returning 14 images
08/18/2020 13:49:24 MainProcess     _training_0     utils           get_image_paths           DEBUG    Scanned Folder contains 14 files
08/18/2020 13:49:24 MainProcess     _training_0     utils           get_image_paths           DEBUG    Returning 14 images
08/18/2020 13:49:24 MainProcess     _training_0     _base           set_timelapse_feed        DEBUG    Setting time-lapse feed: (input_images: '{'a': ['J:\\FS\\SRC\\MODELA\\A FACES\\00708.png', 'J:\\FS\\SRC\\MODELA\\A FACES\\01187.png', 'J:\\FS\\SRC\\MODELA\\A FACES\\01762.png', 'J:\\FS\\SRC\\MODELA\\A FACES\\02059.png', 'J:\\FS\\SRC\\MODELA\\A FACES\\02241.png', 'J:\\FS\\SRC\\MODELA\\A FACES\\04517.png', 'J:\\FS\\SRC\\MODELA\\A FACES\\05028.png', 'J:\\FS\\SRC\\MODELA\\A FACES\\05779.png', 'J:\\FS\\SRC\\MODELA\\A FACES\\08837.png', 'J:\\FS\\SRC\\MODELA\\A FACES\\10258.png', 'J:\\FS\\SRC\\MODELA\\A FACES\\10809.png', 'J:\\FS\\SRC\\MODELA\\A FACES\\11892.png', 'J:\\FS\\SRC\\MODELA\\A FACES\\12348.png', 'J:\\FS\\SRC\\MODELA\\A FACES\\13144.png'], 'b': ['J:\\FS\\SRC\\MODELB\\B FACES\\00067.jpg', 'J:\\FS\\SRC\\MODELB\\B FACES\\01251.png', 'J:\\FS\\SRC\\MODELB\\B FACES\\01407.jpg', 'J:\\FS\\SRC\\MODELB\\B FACES\\01455_0.jpg', 'J:\\FS\\SRC\\MODELB\\B FACES\\01668_0.jpg', 'J:\\FS\\SRC\\MODELB\\B FACES\\02229_0.jpg', 'J:\\FS\\SRC\\MODELB\\B FACES\\02388.png', 'J:\\FS\\SRC\\MODELB\\B FACES\\02497_0.jpg', 'J:\\FS\\SRC\\MODELB\\B FACES\\02559_0.jpg', 'J:\\FS\\SRC\\MODELB\\B FACES\\02805.jpg', 'J:\\FS\\SRC\\MODELB\\B FACES\\02866.png', 'J:\\FS\\SRC\\MODELB\\B FACES\\03039_0.jpg', 'J:\\FS\\SRC\\MODELB\\B FACES\\UIFI_0_0.PNG', 'J:\\FS\\SRC\\MODELB\\B FACES\\YDRTDFHG_0_0.PNG']}', batch_size: 14)
08/18/2020 13:49:24 MainProcess     _training_0     _base           _load_generator           DEBUG    Loading generator
08/18/2020 13:49:24 MainProcess     _training_0     _base           _load_generator           DEBUG    input_size: 128, output_shapes: [(128, 128, 3)]
08/18/2020 13:49:24 MainProcess     _training_0     _base           coverage_ratio            DEBUG    Requested coverage_ratio: 0.6875
08/18/2020 13:49:24 MainProcess     _training_0     _base           coverage_ratio            DEBUG    Final coverage_ratio: 0.6875
08/18/2020 13:49:24 MainProcess     _training_0     training_data   __init__                  DEBUG    Initializing TrainingDataGenerator: (model_input_size: 128, model_output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, augment_color: True, no_flip: False, warp_to_landmarks: False, alignments: ['masks'], config: {'coverage': 68.75, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'penalized_mask_loss': True, 'loss_function': 'mae', 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
08/18/2020 13:49:24 MainProcess     _training_0     training_data   __init__                  DEBUG    Initialized TrainingDataGenerator
08/18/2020 13:49:24 MainProcess     _training_0     training_data   minibatch_ab              DEBUG    Queue batches: (image_count: 14, batchsize: 14, side: 'a', do_shuffle: False, is_preview, False, is_timelapse: True)
08/18/2020 13:49:24 MainProcess     _training_0     training_data   __init__                  DEBUG    Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 128, output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, config: {'coverage': 68.75, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'penalized_mask_loss': True, 'loss_function': 'mae', 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
08/18/2020 13:49:24 MainProcess     _training_0     training_data   __init__                  DEBUG    Output sizes: [128]
08/18/2020 13:49:24 MainProcess     _training_0     training_data   __init__                  DEBUG    Initialized ImageAugmentation
08/18/2020 13:49:24 MainProcess     _training_0     multithreading  __init__                  DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
08/18/2020 13:49:24 MainProcess     _training_0     multithreading  __init__                  DEBUG    Initialized BackgroundGenerator: '_run'
08/18/2020 13:49:24 MainProcess     _training_0     multithreading  start                     DEBUG    Starting thread(s): '_run'
08/18/2020 13:49:24 MainProcess     _training_0     multithreading  start                     DEBUG    Starting thread 1 of 2: '_run_0'
08/18/2020 13:49:24 MainProcess     _run_0          training_data   _minibatch                DEBUG    Loading minibatch generator: (image_count: 14, side: 'a', do_shuffle: False)
08/18/2020 13:49:24 MainProcess     _training_0     multithreading  start                     DEBUG    Starting thread 2 of 2: '_run_1'
08/18/2020 13:49:24 MainProcess     _run_1          training_data   _minibatch                DEBUG    Loading minibatch generator: (image_count: 14, side: 'a', do_shuffle: False)
08/18/2020 13:49:24 MainProcess     _training_0     multithreading  start                     DEBUG    Started all threads '_run': 2
08/18/2020 13:49:24 MainProcess     _training_0     _base           _load_generator           DEBUG    Loading generator
08/18/2020 13:49:24 MainProcess     _training_0     _base           _load_generator           DEBUG    input_size: 128, output_shapes: [(128, 128, 3)]
08/18/2020 13:49:24 MainProcess     _training_0     _base           coverage_ratio            DEBUG    Requested coverage_ratio: 0.6875
08/18/2020 13:49:24 MainProcess     _training_0     _base           coverage_ratio            DEBUG    Final coverage_ratio: 0.6875
08/18/2020 13:49:24 MainProcess     _training_0     training_data   __init__                  DEBUG    Initializing TrainingDataGenerator: (model_input_size: 128, model_output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, augment_color: True, no_flip: False, warp_to_landmarks: False, alignments: ['masks'], config: {'coverage': 68.75, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'penalized_mask_loss': True, 'loss_function': 'mae', 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
08/18/2020 13:49:24 MainProcess     _training_0     training_data   __init__                  DEBUG    Initialized TrainingDataGenerator
08/18/2020 13:49:24 MainProcess     _training_0     training_data   minibatch_ab              DEBUG    Queue batches: (image_count: 14, batchsize: 14, side: 'b', do_shuffle: False, is_preview, False, is_timelapse: True)
08/18/2020 13:49:24 MainProcess     _training_0     training_data   __init__                  DEBUG    Initializing ImageAugmentation: (batchsize: 14, is_display: True, input_size: 128, output_shapes: [(128, 128, 3)], coverage_ratio: 0.6875, config: {'coverage': 68.75, 'mask_type': 'extended', 'mask_blur_kernel': 3, 'mask_threshold': 4, 'learn_mask': False, 'penalized_mask_loss': True, 'loss_function': 'mae', 'icnr_init': False, 'conv_aware_init': False, 'optimizer': 'adam', 'learning_rate': 5e-05, 'reflect_padding': False, 'allow_growth': False, 'mixed_precision': False, 'preview_images': 14, 'zoom_amount': 5, 'rotation_range': 10, 'shift_range': 5, 'flip_chance': 50, 'color_lightness': 30, 'color_ab': 8, 'color_clahe_chance': 50, 'color_clahe_max_size': 4})
08/18/2020 13:49:24 MainProcess     _training_0     training_data   __init__                  DEBUG    Output sizes: [128]
08/18/2020 13:49:24 MainProcess     _training_0     training_data   __init__                  DEBUG    Initialized ImageAugmentation
08/18/2020 13:49:24 MainProcess     _training_0     multithreading  __init__                  DEBUG    Initializing BackgroundGenerator: (target: '_run', thread_count: 2)
08/18/2020 13:49:24 MainProcess     _training_0     multithreading  __init__                  DEBUG    Initialized BackgroundGenerator: '_run'
08/18/2020 13:49:24 MainProcess     _training_0     multithreading  start                     DEBUG    Starting thread(s): '_run'
08/18/2020 13:49:24 MainProcess     _training_0     multithreading  start                     DEBUG    Starting thread 1 of 2: '_run_0'
08/18/2020 13:49:24 MainProcess     _run_0          training_data   _minibatch                DEBUG    Loading minibatch generator: (image_count: 14, side: 'b', do_shuffle: False)
08/18/2020 13:49:24 MainProcess     _training_0     multithreading  start                     DEBUG    Starting thread 2 of 2: '_run_1'
08/18/2020 13:49:24 MainProcess     _run_1          training_data   _minibatch                DEBUG    Loading minibatch generator: (image_count: 14, side: 'b', do_shuffle: False)
08/18/2020 13:49:24 MainProcess     _training_0     multithreading  start                     DEBUG    Started all threads '_run': 2
08/18/2020 13:49:24 MainProcess     _training_0     _base           set_timelapse_feed        DEBUG    Set time-lapse feed: {'a': <generator object BackgroundGenerator.iterator at 0x00000250D8E8F200>, 'b': <generator object BackgroundGenerator.iterator at 0x00000250D8E8F970>}
08/18/2020 13:49:24 MainProcess     _training_0     _base           _setup                    DEBUG    Set up time-lapse
08/18/2020 13:49:24 MainProcess     _training_0     _base           output_timelapse          DEBUG    Getting time-lapse samples
08/18/2020 13:49:24 MainProcess     _run_0          multithreading  run                       DEBUG    Error in thread (_run_0): 'J:\\FS\\SRC\\MODELA\\A FACES\\00708.png'
08/18/2020 13:49:24 MainProcess     _training_0     multithreading  check_and_raise_error     DEBUG    Thread error caught: [(<class 'KeyError'>, KeyError('J:\\FS\\SRC\\MODELA\\A FACES\\00708.png'), <traceback object at 0x00000250D8E81900>)]
08/18/2020 13:49:24 MainProcess     _training_0     multithreading  run                       DEBUG    Error in thread (_training_0): 'J:\\FS\\SRC\\MODELA\\A FACES\\00708.png'
08/18/2020 13:49:24 MainProcess     _run_1          multithreading  run                       DEBUG    Error in thread (_run_1): 'J:\\FS\\SRC\\MODELA\\A FACES\\00708.png'
08/18/2020 13:49:24 MainProcess     _run_1          multithreading  run                       DEBUG    Error in thread (_run_1): 'J:\\FS\\SRC\\MODELB\\B FACES\\00067.jpg'
08/18/2020 13:49:24 MainProcess     MainThread      train           _monitor                  DEBUG    Thread error detected
08/18/2020 13:49:24 MainProcess     MainThread      train           _monitor                  DEBUG    Closed Monitor
08/18/2020 13:49:24 MainProcess     MainThread      train           _end_thread               DEBUG    Ending Training thread
08/18/2020 13:49:24 MainProcess     MainThread      train           _end_thread               CRITICAL Error caught! Exiting...
08/18/2020 13:49:24 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Threads: '_training'
08/18/2020 13:49:24 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Thread: '_training_0'
08/18/2020 13:49:24 MainProcess     MainThread      multithreading  join                      ERROR    Caught exception in thread: '_training_0'
Traceback (most recent call last):
  File "C:\Users\ME\faceswap\lib\cli\launcher.py", line 156, in execute_script
    process.process()
  File "C:\Users\ME\faceswap\scripts\train.py", line 135, in process
    self._end_thread(thread, err)
  File "C:\Users\ME\faceswap\scripts\train.py", line 175, in _end_thread
    thread.join()
  File "C:\Users\ME\faceswap\lib\multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "C:\Users\ME\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\ME\faceswap\scripts\train.py", line 197, in _training
    raise err
  File "C:\Users\ME\faceswap\scripts\train.py", line 187, in _training
    self._run_training_cycle(model, trainer)
  File "C:\Users\ME\faceswap\scripts\train.py", line 268, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "C:\Users\ME\faceswap\plugins\train\trainer\_base.py", line 227, in train_one_step
    self._timelapse.output_timelapse(timelapse_kwargs)
  File "C:\Users\ME\faceswap\plugins\train\trainer\_base.py", line 988, in output_timelapse
    self._samples.images = self._feeder.compile_timelapse_sample()
  File "C:\Users\ME\faceswap\plugins\train\trainer\_base.py", line 530, in compile_timelapse_sample
    batch = next(self._display_feeds["timelapse"][side])
  File "C:\Users\ME\faceswap\lib\multithreading.py", line 156, in iterator
    self.check_and_raise_error()
  File "C:\Users\ME\faceswap\lib\multithreading.py", line 84, in check_and_raise_error
    raise error[1].with_traceback(error[2])
  File "C:\Users\ME\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\ME\faceswap\lib\multithreading.py", line 145, in _run
    for item in self.generator(*self._gen_args, **self._gen_kwargs):
  File "C:\Users\ME\faceswap\lib\training_data.py", line 186, in _minibatch
    yield self._process_batch(img_paths, side)
  File "C:\Users\ME\faceswap\lib\training_data.py", line 195, in _process_batch
    batch = self._apply_mask(filenames, batch, side)
  File "C:\Users\ME\faceswap\lib\training_data.py", line 249, in _apply_mask
    masks = np.array([self._masks[side][filename].mask
  File "C:\Users\ME\faceswap\lib\training_data.py", line 249, in <listcomp>
    masks = np.array([self._masks[side][filename].mask
KeyError: 'J:\\FS\\SRC\\MODELA\\A FACES\\00708.png'

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         725d864 Training - Move Mixed Precision to model config. fa4783f training config - Add selectable optimizers. 6b2aac6 Enable MTCNN for CPU extraction
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce GTX 1070 Ti
gpu_devices_active:  GPU_0
gpu_driver:          452.06
gpu_vram:            GPU_0: 8192MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.16299-SP0
os_release:          10
py_command:          C:\Users\ME\faceswap\faceswap.py train -A J:/FS/SRC/MODELA/TRAINING -ala J:/FS/SRC/MODELA/TRAINING/alignments.fsa -B J:/FS/SRC/MODELB/2/Training2 -alb J:/FS/SRC/MODELB/2/Training2/alignments.fsa -m J:/FS/SRC/MODELB/TRAININGMODEL -t villain -bs 8 -it 1000000 -s 250 -ss 25000 -tia J:/FS/SRC/MODELA/A FACES -tib J:/FS/SRC/MODELB/B FACES -to J:/FS/SRC/MODELB/TIMELAPSE -ps 50 -L INFO -gui
py_conda_version:    conda 4.8.4
py_implementation:   CPython
py_version:          3.8.5
py_virtual_env:      True
sys_cores:           6
sys_processor:       Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
sys_ram:             Total: 16344MB, Available: 9363MB, Used: 6981MB, Free: 9363MB

=============== Pip Packages ===============
absl-py==0.9.0
astunparse==1.6.3
cachetools==4.1.1
certifi==2020.6.20
chardet==3.0.4
cycler==0.10.0
fastcluster==1.1.26
ffmpy==0.2.3
gast==0.3.3
google-auth==1.20.1
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.31.0
h5py==2.10.0
idna==2.10
imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work
imageio-ffmpeg @ file:///home/conda/feedstock_root/build_artifacts/imageio-ffmpeg_1589202782679/work
joblib @ file:///tmp/build/80754af9/joblib_1594236160679/work
Keras-Preprocessing==1.1.2
kiwisolver==1.2.0
Markdown==3.2.2
matplotlib @ file:///C:/ci/matplotlib-base_1592837548929/work
mkl-fft==1.1.0
mkl-random==1.1.1
mkl-service==2.3.0
numpy @ file:///C:/ci/numpy_and_numpy_base_1596215850360/work
nvidia-ml-py3 @ git+https://github.com/deepfakes/nvidia-ml-py3.git@6fc29ac84b32bad877f078cb4a777c1548a00bf6
oauthlib==3.1.0
olefile==0.46
opencv-python==4.4.0.40
opt-einsum==3.3.0
pathlib==1.0.1
Pillow @ file:///C:/ci/pillow_1594298230227/work
protobuf==3.13.0
psutil==5.7.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==2.4.7
python-dateutil==2.8.1
pywin32==227
requests==2.24.0
requests-oauthlib==1.3.0
rsa==4.6
scikit-learn @ file:///C:/ci/scikit-learn_1592853510272/work
scipy==1.4.1
sip==4.19.13
six==1.15.0
tensorboard==2.2.2
tensorboard-plugin-wit==1.7.0
tensorflow==2.2.0
tensorflow-estimator==2.2.0
tensorflow-gpu==2.2.0
tensorflow-gpu-estimator==2.2.0
termcolor==1.1.0
threadpoolctl @ file:///tmp/tmp9twdgx9k/threadpoolctl-2.1.0-py3-none-any.whl
tornado==6.0.4
tqdm @ file:///tmp/build/80754af9/tqdm_1596810128862/work
urllib3==1.25.10
Werkzeug==1.0.1
wincertstore==0.2
wrapt==1.12.1

============== Conda Packages ==============
# packages in environment at C:\Users\ME\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
absl-py                   0.9.0                    pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
blas                      1.0                         mkl  

ca-certificates           2020.6.24                     0  

cachetools                4.1.1                    pypi_0    pypi
certifi                   2020.6.20                py38_0  

chardet                   3.0.4                    pypi_0    pypi
cudatoolkit               10.1.243             h74a9793_0  

cudnn                     7.6.5                cuda10.1_0  

cycler                    0.10.0                   py38_0  

fastcluster               1.1.26           py38hbe40bda_1    conda-forge
ffmpeg                    4.3.1                ha925a31_0    conda-forge
ffmpy                     0.2.3                    pypi_0    pypi
freetype                  2.10.2               hd328e21_0  

gast                      0.3.3                    pypi_0    pypi
git                       2.23.0               h6bb4b03_0  

google-auth               1.20.1                   pypi_0    pypi
google-auth-oauthlib      0.4.1                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.31.0                   pypi_0    pypi
h5py                      2.10.0                   pypi_0    pypi
icc_rt                    2019.0.0             h0cc432a_1  

icu                       58.2                 ha925a31_3  

idna                      2.10                     pypi_0    pypi
imageio                   2.9.0                      py_0  

imageio-ffmpeg            0.4.2                      py_0    conda-forge
intel-openmp              2020.1                      216  

joblib                    0.16.0                     py_0  

jpeg                      9b                   hb83a4c4_2  

keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.2.0            py38h74a9793_0  

libpng                    1.6.37               h2a8f88b_0  

libtiff                   4.1.0                h56a325e_1  

lz4-c                     1.9.2                h62dcd97_1  

markdown                  3.2.2                    pypi_0    pypi
matplotlib                3.2.2                         0  

matplotlib-base           3.2.2            py38h64f37c6_0  

mkl                       2020.1                      216  

mkl-service               2.3.0            py38hb782905_0  

mkl_fft                   1.1.0            py38h45dec08_0  

mkl_random                1.1.1            py38h47e9c7a_0  

numpy                     1.19.1           py38h5510c5b_0  

numpy-base                1.19.1           py38ha3acd2a_0  

nvidia-ml-py3             7.352.1                  pypi_0    pypi
oauthlib                  3.1.0                    pypi_0    pypi
olefile                   0.46                       py_0  

opencv-python             4.4.0.40                 pypi_0    pypi
openssl                   1.1.1g               he774522_1  

opt-einsum                3.3.0                    pypi_0    pypi
pathlib                   1.0.1                      py_1  

pillow                    7.2.0            py38hcc1f983_0  

pip                       20.2.2                   py38_0  

protobuf                  3.13.0                   pypi_0    pypi
psutil                    5.7.0            py38he774522_0  

pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pyparsing                 2.4.7                      py_0  

pyqt                      5.9.2            py38ha925a31_4  

python                    3.8.5                he1778fa_0  

python-dateutil           2.8.1                      py_0  

python_abi                3.8                      1_cp38    conda-forge
pywin32                   227              py38he774522_1  

qt                        5.9.7            vc14h73c81de_0  

requests                  2.24.0                   pypi_0    pypi
requests-oauthlib         1.3.0                    pypi_0    pypi
rsa                       4.6                      pypi_0    pypi
scikit-learn              0.23.1           py38h25d0782_0  

scipy                     1.4.1                    pypi_0    pypi
setuptools                49.6.0                   py38_0  

sip                       4.19.13          py38ha925a31_0  

six                       1.15.0                     py_0  

sqlite                    3.32.3               h2a8f88b_0  

tensorboard               2.2.2                    pypi_0    pypi
tensorboard-plugin-wit    1.7.0                    pypi_0    pypi
tensorflow                2.2.0                    pypi_0    pypi
tensorflow-estimator      2.2.0                    pypi_0    pypi
tensorflow-gpu            2.2.0                    pypi_0    pypi
tensorflow-gpu-estimator  2.2.0                    pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
threadpoolctl             2.1.0              pyh5ca1d4c_0  

tk                        8.6.10               he774522_0  

tornado                   6.0.4            py38he774522_1  

tqdm                      4.48.2                     py_0  

urllib3                   1.25.10                  pypi_0    pypi
vc                        14.1                 h0510ff6_4  

vs2015_runtime            14.16.27012          hf0eaf9b_3  

werkzeug                  1.0.1                    pypi_0    pypi
wheel                     0.34.2                   py38_0  

wincertstore              0.2                      py38_0  

wrapt                     1.12.1                   pypi_0    pypi
xz                        5.2.5                h62dcd97_0  

zlib                      1.2.11               h62dcd97_4  

zstd                      1.4.5                h04227a9_0  

================= Configs ==================
--------- .faceswap ---------
backend:                  nvidia

--------- convert.ini ---------

[color.color_transfer]
clip:                     True
preserve_paper:           True

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.match_hist]
threshold:                99.0

[mask.box_blend]
type:                     gaussian
distance:                 11.0
radius:                   5.0
passes:                   1

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0

[scaling.sharpen]
method:                   unsharp_mask
amount:                   150
radius:                   0.3
threshold:                5.0

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
jpg_quality:              75
png_compress_level:       3

[writer.pillow]
format:                   png
draw_transparent:         False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

--------- extract.ini ---------

[global]
allow_growth:             False

[align.fan]
batch-size:               12

[detect.cv2_dnn]
confidence:               50

[detect.mtcnn]
minsize:                  20
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7
scalefactor:              0.709
batch-size:               8

[detect.s3fd]
confidence:               70
batch-size:               4

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[mask.vgg_obstructed]
batch-size:               2

--------- gui.ini ---------

[global]
fullscreen:               False
tab:                      extract
options_panel_width:      30
console_panel_height:     20
icon_size:                14
font:                     default
font_size:                9
autosave_last_session:    prompt
timeout:                  120
auto_load_model_stats:    True

--------- train.ini ---------

[global]
coverage:                 68.75
mask_type:                extended
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False
penalized_mask_loss:      True
loss_function:            mae
icnr_init:                False
conv_aware_init:          False
optimizer:                adam
learning_rate:            5e-05
reflect_padding:          False
allow_growth:             False
mixed_precision:          False

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
clipnorm:                 True
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.dlight]
features:                 best
details:                  good
output_size:              256

[model.original]
lowmem:                   False

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.unbalanced]
input_size:               128
lowmem:                   False
clipnorm:                 True
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.villain]
lowmem:                   False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4

Faceswap is up to date! Any ideas on what is going wrong here?

Top
User avatar
Mr.Man
Posts: 9
Joined: Thu Jun 25, 2020 9:28 pm
Answers: 0

Re: Training crashes before start: ‘Enabled TensorBoard Logging'

Post by Mr.Man »

Apologies for the double post. The above image that caused the crash was in the timelapse folder. I removed all images in the timelapse folder from model A and replaced them with new ones from the new extracted set. The crash log kept pointing to these photos. I replaced them again and the same error.

Now I have removed the timelapse options and the process works!

Any ideas on what is preventing the timelapse process from running alongside? I'm just happy the training process is working!

Top
User avatar
torzdf
Posts: 2796
Joined: Fri Jul 12, 2019 12:53 am
Answers: 160
Has thanked: 142 times
Been thanked: 650 times

Re: Training crashes before start: ‘Enabled TensorBoard Logging'

Post by torzdf »

Timelapse images must come from your training set. Assuming that you are just putting random faces in there which are not from your training set, then this will be the issue.

My word is final

Top
Locked

Return to “Training Support”