Status Failed - train.py

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
glantz
Posts: 8
Joined: Thu Aug 15, 2019 8:03 am

Status Failed - train.py

Post by glantz »

I installed everything using this guide
https://faceswap.dev/forum/viewtopic.php?f=4&t=68

Everything looks good.

When I am trying to train. It fails (attached screenshot)

Thanks ahead, and thanks for all your help.

by torzdf » Tue Aug 20, 2019 5:30 pm

Ok, 2 things:

1) Remove your system wide installed Cuda. How to do this will depend on how you installed Cuda in the first place. Basically, it will probably conflict with the Conda installed version.

2) Update your Nvidia Drivers.

Go to full post
User avatar
torzdf
Posts: 2665
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: Status Failed - train.py

Post by torzdf »

No screenshot.

Please post the crash_report from your Faceswap folder (if it exists)

My word is final

User avatar
glantz
Posts: 8
Joined: Thu Aug 15, 2019 8:03 am

Re: Status Failed - train.py

Post by glantz »

How do I export this file?
In the faceapp folder the crash_report .log file is empty...

I have a GeForce GTX 1060 btw.

Thanks

Last edited by glantz on Tue Aug 20, 2019 4:51 pm, edited 1 time in total.
User avatar
torzdf
Posts: 2665
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: Status Failed - train.py

Post by torzdf »

Ok.

There is still no screenshot here, or text, so I have no idea about your crash.

My word is final

User avatar
glantz
Posts: 8
Joined: Thu Aug 15, 2019 8:03 am

Re: Status Failed - train.py

Post by glantz »

Sorry, I attached the error that I saw in the GUI.

Thanks

Attachments
screenshot
screenshot
Screenshot from 2019-08-20 13-42-51.png (30.78 KiB) Viewed 8051 times
User avatar
torzdf
Posts: 2665
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: Status Failed - train.py

Post by torzdf »

Ok. This tells me nothing :(

Can you post the output of tools menu > Show System Information

My word is final

User avatar
glantz
Posts: 8
Joined: Thu Aug 15, 2019 8:03 am

Re: Status Failed - train.py

Post by glantz »

When I pressed the Tools-> System info, I got an error:

Code: Select all

Error obtaining system info: No module named 'yaml'

I pip installed the module and it now it works!!

How can I make sure that I use the GPU while training?

Inside the console I see

Code: Select all

Setting Faceswap backend to NVIDIA

Also attached the output from "nvidia-smi" command.

Thanks a lot!!
Tal

Attachments
nvidia-smi
nvidia-smi
Screenshot from 2019-08-20 20-20-38.png (67.46 KiB) Viewed 8050 times
User avatar
torzdf
Posts: 2665
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: Status Failed - train.py

Post by torzdf »

If that's a screengrab from training, then it's not using your GPU.

I would still need the output of tools > system info

My word is final

User avatar
glantz
Posts: 8
Joined: Thu Aug 15, 2019 8:03 am

Re: Status Failed - train.py

Post by glantz »

Yes it is screen from training... :(

Here is the info:

Code: Select all

============ System Information ============
encoding:            UTF-8
git_branch:          master
git_commits:         6c1a97a add git to dockerfiles (#839). 86fc951 Bugfix: Manual tool for AMD updates. 4a50493 Suppress multiple backend messages. f77cf1f AMD Backend typofix. b8598be More seamless AMD Integration
gpu_cuda:            10.0
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: GeForce GTX 1060
gpu_devices_active:  GPU_0
gpu_driver:          410.104
gpu_vram:            GPU_0: 6078MB
os_machine:          x86_64
os_platform:         Linux-4.15.0-58-generic-x86_64-with-debian-buster-sid
os_release:          4.15.0-58-generic
py_command:          faceswap.py gui
py_conda_version:    conda 4.7.10
py_implementation:   CPython
py_version:          3.6.9
py_virtual_env:      True
sys_cores:           8
sys_processor:       x86_64
sys_ram:             Total: 32010MB, Available: 20820MB, Used: 10206MB, Free: 292MB

=============== Pip Packages ===============
absl-py==0.7.1
astor==0.8.0
certifi==2019.6.16
cloudpickle==1.2.1
cycler==0.10.0
cytoolz==0.10.0
dask==2.3.0
decorator==4.4.0
fastcluster==1.1.25
ffmpy==0.2.2
future==0.17.1
gast==0.2.2
grpcio==1.22.0
h5py==2.9.0
imageio==2.5.0
imageio-ffmpeg==0.3.0
joblib==0.13.2
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
Markdown==3.1.1
matplotlib==2.2.2
mkl-fft==1.0.14
mkl-random==1.0.2
mkl-service==2.0.2
mock==3.0.5
networkx==2.3
numpy==1.16.2
nvidia-ml-py3==7.352.1
olefile==0.46
opencv-python==4.1.0.25
pathlib==1.0.1
Pillow==6.1.0
protobuf==3.9.1
psutil==5.6.3
pyparsing==2.4.2
python-dateutil==2.8.0
pytz==2019.2
PyWavelets==1.0.3
PyYAML==5.1.2
scikit-image==0.15.0
scikit-learn==0.21.2
scipy==1.3.1
six==1.12.0
tensorboard==1.13.1
tensorflow==1.13.1
tensorflow-estimator==1.13.0
termcolor==1.1.0
toolz==0.10.0
toposort==1.5
tornado==6.0.3
tqdm==4.32.1
Werkzeug==0.15.5

============== Conda Packages ==============
# packages in environment at /home/tal/miniconda3/envs/faceswap:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_tflow_select 2.1.0 gpu
absl-py 0.7.1 py36_0
astor 0.8.0 py36_0
blas 1.0 mkl
c-ares 1.15.0 h7b6447c_1001
ca-certificates 2019.5.15 1
certifi 2019.6.16 py36_1
cloudpickle 1.2.1 py_0
cudatoolkit 10.0.130 0
cudnn 7.6.0 cuda10.0_0
cupti 10.0.130 0
cycler 0.10.0 py36_0
cytoolz 0.10.0 py36h7b6447c_0
dask-core 2.3.0 py_0
dbus 1.13.6 h746ee38_0
decorator 4.4.0 py36_1
expat 2.2.6 he6710b0_0
fontconfig 2.13.0 h9420a91_0
freetype 2.9.1 h8a8886c_1
gast 0.2.2 py36_0
glib 2.56.2 hd408876_0
grpcio 1.16.1 py36hf8bcb03_1
gst-plugins-base 1.14.0 hbbd80ab_1
gstreamer 1.14.0 hb453b48_1
h5py 2.9.0 py36h7918eee_0
hdf5 1.10.4 hb1b8bf9_0
icu 58.2 h9c2bf20_1
imageio 2.5.0 py36_0
intel-openmp 2019.4 243
joblib 0.13.2 py36_0
jpeg 9b h024ee3a_2
keras-applications 1.0.8 py_0
keras-preprocessing 1.1.0 py_1
kiwisolver 1.1.0 py36he6710b0_0
libedit 3.1.20181209 hc058e9b_0
libffi 3.2.1 hd88cf55_4
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
libpng 1.6.37 hbc83047_0
libprotobuf 3.8.0 hd408876_0
libstdcxx-ng 9.1.0 hdf63c60_0
libtiff 4.0.10 h2733197_2
libuuid 1.0.3 h1bed415_2
libxcb 1.13 h1bed415_1
libxml2 2.9.9 hea5a465_1
markdown 3.1.1 py36_0
matplotlib 2.2.2 py36hb69df0a_2
mkl 2019.4 243
mkl-service 2.0.2 py36h7b6447c_0
mkl_fft 1.0.14 py36ha843d7b_0
mkl_random 1.0.2 py36hd81dba3_0
mock 3.0.5 py36_0
ncurses 6.1 he6710b0_1
networkx 2.3 py_0
numpy 1.16.2 py36h7e9f1db_0
numpy-base 1.16.2 py36hde5b4d6_0
olefile 0.46 py36_0
opencv-python 4.1.0.25 pypi_0 pypi openssl 1.1.1c h7b6447c_1
pcre 8.43 he6710b0_0
pillow 6.1.0 py36h34e0f95_0
pip 19.2.2 py36_0
protobuf 3.8.0 py36he6710b0_0
pyparsing 2.4.2 py_0
pyqt 5.9.2 py36h05f1152_2
python 3.6.9 h265db76_0
python-dateutil 2.8.0 py36_0
pytz 2019.2 py_0
pywavelets 1.0.3 py36hdd07704_1
pyyaml 5.1.2 pypi_0 pypi qt 5.9.7 h5867ecd_1
readline 7.0 h7b6447c_5
scikit-image 0.15.0 py36he6710b0_0
scikit-learn 0.21.2 py36hd81dba3_0
scipy 1.3.1 py36h7c811a0_0
setuptools 41.0.1 py36_0
sip 4.19.8 py36hf484d3e_0
six 1.12.0 py36_0
sqlite 3.29.0 h7b6447c_0
tensorboard 1.13.1 py36hf484d3e_0
tensorflow 1.13.1 gpu_py36h3991807_0
tensorflow-base 1.13.1 gpu_py36h8d69cac_0
tensorflow-estimator 1.13.0 py_0
tensorflow-gpu 1.13.1 h0d30ee6_0
termcolor 1.1.0 py36_1
tk 8.6.8 hbc83047_0
toolz 0.10.0 py_0
tornado 6.0.3 py36h7b6447c_0
tqdm 4.32.1 py_0
werkzeug 0.15.5 py_0
wheel 0.33.4 py36_0
xz 5.2.4 h14c3975_4
zlib 1.2.11 h7b6447c_3
zstd 1.3.7 h0b5b093_0
User avatar
torzdf
Posts: 2665
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 131 times
Been thanked: 625 times

Re: Status Failed - train.py

Post by torzdf »

Ok, 2 things:

1) Remove your system wide installed Cuda. How to do this will depend on how you installed Cuda in the first place. Basically, it will probably conflict with the Conda installed version.

2) Update your Nvidia Drivers.

My word is final

User avatar
glantz
Posts: 8
Joined: Thu Aug 15, 2019 8:03 am

Re: Status Failed - train.py

Post by glantz »

Will try tomorrow morning, and will post an update...

Thanks!!

Locked