Encountered CRITICAL An unexpected crash has occurred.

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
I'mForKnights
Posts: 2
Joined: Fri Nov 08, 2019 10:15 pm

Encountered CRITICAL An unexpected crash has occurred.

Post by I'mForKnights »

I have no idea how I can fix this mess

Code: Select all

11/08/2019 17:10:37 MainProcess     _run_1          training_data   initialize                DEBUG    Initializing constants. training_size: 256
11/08/2019 17:10:37 MainProcess     _run_1          training_data   initialize                DEBUG    Initialized constants: {'clahe_base_contrast': 2, 'tgt_slices': slice(40, 216, None), 'warp_mapx': '[[[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]]', 'warp_mapy': '[[[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]]', 'warp_pad': 80, 'warp_slices': slice(8, -8, None), 'warp_lm_edge_anchors': '[[[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]]', 'warp_lm_grids': '[[[  0.   0.   0. ...   0.   0.   0.]\n  [  1.   1.   1. ...   1.   1.   1.]\n  [  2.   2.   2. ...   2.   2.   2.]\n  ...\n  [253. 253. 253. ... 253. 253. 253.]\n  [254. 254. 254. ... 254. 254. 254.]\n  [255. 255. 255. ... 255. 255. 255.]]\n\n [[  0.   1.   2. ... 253. 254. 255.]\n  [  0.   1.   2. ... 253. 254. 255.]\n  [  0.   1.   2. ... 253. 254. 255.]\n  ...\n  [  0.   1.   2. ... 253. 254. 255.]\n  [  0.   1.   2. ... 253. 254. 255.]\n  [  0.   1.   2. ... 253. 254. 255.]]]'}
11/08/2019 17:10:37 MainProcess     training_0      _base           largest_face_index        DEBUG    0
11/08/2019 17:10:40 MainProcess     training_0      library         _logger_callback          INFO     Analyzing Ops: 270 of 520 operations complete
11/08/2019 17:10:52 MainProcess     training_0      _base           compile_sample            DEBUG    Compiling samples: (side: 'b', samples: 14)
11/08/2019 17:10:53 MainProcess     training_0      _base           show_sample               DEBUG    Showing sample
11/08/2019 17:10:53 MainProcess     training_0      _base           get_predictions           DEBUG    Getting Predictions
11/08/2019 17:11:00 MainProcess     training_0      _base           get_predictions           DEBUG    Returning predictions: {'a_a': (14, 64, 64, 3), 'b_a': (14, 64, 64, 3), 'a_b': (14, 64, 64, 3), 'b_b': (14, 64, 64, 3)}
11/08/2019 17:11:00 MainProcess     training_0      _base           to_full_frame             DEBUG    side: 'a', number of sample arrays: 2, prediction.shapes: [(14, 64, 64, 3), (14, 64, 64, 3)])
11/08/2019 17:11:00 MainProcess     training_0      _base           frame_overlay             DEBUG    full_size: 256, target_size: 176, color: (0, 0, 255)
11/08/2019 17:11:00 MainProcess     training_0      _base           frame_overlay             DEBUG    Overlayed background. Shape: (14, 256, 256, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'a' shape: (14, 176, 176, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'a' shape: (14, 176, 176, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'a' shape: (14, 176, 176, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           overlay_foreground        DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           overlay_foreground        DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           overlay_foreground        DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'a' shape: (14, 128, 128, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'a' shape: (14, 128, 128, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'a' shape: (14, 128, 128, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           get_headers               DEBUG    side: 'a', width: 128
11/08/2019 17:11:00 MainProcess     training_0      _base           get_headers               DEBUG    height: 32, total_width: 384
11/08/2019 17:11:00 MainProcess     training_0      _base           get_headers               DEBUG    texts: ['Original (A)', 'Original > Original', 'Original > Swap'], text_sizes: [(72, 9), (116, 9), (102, 9)], text_x: [28, 134, 269], text_y: 20
11/08/2019 17:11:00 MainProcess     training_0      _base           get_headers               DEBUG    header_box.shape: (32, 384, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           to_full_frame             DEBUG    side: 'b', number of sample arrays: 2, prediction.shapes: [(14, 64, 64, 3), (14, 64, 64, 3)])
11/08/2019 17:11:00 MainProcess     training_0      _base           frame_overlay             DEBUG    full_size: 256, target_size: 176, color: (0, 0, 255)
11/08/2019 17:11:00 MainProcess     training_0      _base           frame_overlay             DEBUG    Overlayed background. Shape: (14, 256, 256, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'b' shape: (14, 176, 176, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'b' shape: (14, 176, 176, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'b' shape: (14, 176, 176, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           overlay_foreground        DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           overlay_foreground        DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           overlay_foreground        DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'b' shape: (14, 128, 128, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'b' shape: (14, 128, 128, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'b' shape: (14, 128, 128, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           get_headers               DEBUG    side: 'b', width: 128
11/08/2019 17:11:00 MainProcess     training_0      _base           get_headers               DEBUG    height: 32, total_width: 384
11/08/2019 17:11:00 MainProcess     training_0      _base           get_headers               DEBUG    texts: ['Swap (B)', 'Swap > Swap', 'Swap > Original'], text_sizes: [(59, 9), (87, 9), (102, 9)], text_x: [34, 148, 269], text_y: 20
11/08/2019 17:11:00 MainProcess     training_0      _base           get_headers               DEBUG    header_box.shape: (32, 384, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           duplicate_headers         DEBUG    side: a header.shape: (32, 384, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           duplicate_headers         DEBUG    side: b header.shape: (32, 384, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           stack_images              DEBUG    Stack images
11/08/2019 17:11:00 MainProcess     training_0      _base           get_transpose_axes        DEBUG    Even number of images to stack
11/08/2019 17:11:00 MainProcess     training_0      _base           stack_images              DEBUG    Stacked images
11/08/2019 17:11:00 MainProcess     training_0      _base           show_sample               DEBUG    Compiled sample
11/08/2019 17:11:00 MainProcess     training_0      _base           save_models               DEBUG    Backing up and saving models
11/08/2019 17:11:00 MainProcess     training_0      _base           get_save_averages         DEBUG    Getting save averages
11/08/2019 17:11:00 MainProcess     training_0      _base           get_save_averages         DEBUG    Average losses since last save: {'a': 0.11725123971700668, 'b': 0.1210833340883255}
11/08/2019 17:11:00 MainProcess     training_0      _base           check_loss_drop           DEBUG    Loss for 'a' has dropped
11/08/2019 17:11:00 MainProcess     training_0      _base           check_loss_drop           DEBUG    Loss for 'b' has dropped
11/08/2019 17:11:00 MainProcess     training_0      _base           should_backup             DEBUG    Lowest historical save iteration loss average: {'a': 0.12577251677817486, 'b': 0.13642760413758298}
11/08/2019 17:11:00 MainProcess     training_0      _base           should_backup             DEBUG    Updating lowest save iteration average for 'a': 0.11725123971700668
11/08/2019 17:11:00 MainProcess     training_0      _base           should_backup             DEBUG    Updating lowest save iteration average for 'b': 0.1210833340883255
11/08/2019 17:11:00 MainProcess     training_0      _base           should_backup             DEBUG    Backing up: True
11/08/2019 17:11:00 MainProcess     training_0      _base           save_models               INFO     Backing up models...
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_0 backup_restore  backup_model              VERBOSE  Backing up: 'E:\John\Documents\Deepfake\Page ASMR model\original_decoder_A.h5' to 'E:\John\Documents\Deepfake\Page ASMR model\original_decoder_A.h5.bk'
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_1 backup_restore  backup_model              VERBOSE  Backing up: 'E:\John\Documents\Deepfake\Page ASMR model\original_decoder_B.h5' to 'E:\John\Documents\Deepfake\Page ASMR model\original_decoder_B.h5.bk'
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_2 backup_restore  backup_model              VERBOSE  Backing up: 'E:\John\Documents\Deepfake\Page ASMR model\original_encoder.h5' to 'E:\John\Documents\Deepfake\Page ASMR model\original_encoder.h5.bk'
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_3 _base           save                      DEBUG    Saving State
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_3 backup_restore  backup_model              VERBOSE  Backing up: 'E:\John\Documents\Deepfake\Page ASMR model\original_state.json' to 'E:\John\Documents\Deepfake\Page ASMR model\original_state.json.bk'
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_0 _base           save                      DEBUG    Saving model: 'E:\John\Documents\Deepfake\Page ASMR model\original_decoder_A.h5'
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_2 _base           save                      DEBUG    Saving model: 'E:\John\Documents\Deepfake\Page ASMR model\original_encoder.h5'
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_3 serializer      save                      DEBUG    filename: E:\John\Documents\Deepfake\Page ASMR model\original_state.json, data type: <class 'dict'>
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_3 serializer      _check_extension          DEBUG    Original filename: 'E:\John\Documents\Deepfake\Page ASMR model\original_state.json', final filename: 'E:\John\Documents\Deepfake\Page ASMR model\original_state.json'
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_1 _base           save                      DEBUG    Saving model: 'E:\John\Documents\Deepfake\Page ASMR model\original_decoder_B.h5'
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_3 serializer      marshal                   DEBUG    data type: <class 'dict'>
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_3 serializer      marshal                   DEBUG    returned data type: <class 'bytes'>
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_3 _base           save                      DEBUG    Saved State
11/08/2019 17:11:02 MainProcess     training_0      _base           save_models               INFO     [Saved models] - Average since last save: face_loss_A: 0.11725, face_loss_B: 0.12108
11/08/2019 17:12:31 MainProcess     _run_1          multithreading  run                       DEBUG    Error in thread (_run_1): tuple index out of range
11/08/2019 17:12:34 MainProcess     training_0      multithreading  check_and_raise_error     DEBUG    Thread error caught: [(<class 'IndexError'>, IndexError('tuple index out of range',), <traceback object at 0x000002B387EFE808>)]
11/08/2019 17:12:34 MainProcess     training_0      multithreading  run                       DEBUG    Error in thread (training_0): tuple index out of range
11/08/2019 17:12:34 MainProcess     _run_0          multithreading  run                       DEBUG    Error in thread (_run_0): tuple index out of range
11/08/2019 17:12:34 MainProcess     MainThread      train           monitor                   DEBUG    Thread error detected
11/08/2019 17:12:34 MainProcess     MainThread      train           monitor                   DEBUG    Closed Monitor
11/08/2019 17:12:34 MainProcess     MainThread      train           end_thread                DEBUG    Ending Training thread
11/08/2019 17:12:34 MainProcess     MainThread      train           end_thread                CRITICAL Error caught! Exiting...
11/08/2019 17:12:34 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Threads: 'training'
11/08/2019 17:12:34 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Thread: 'training_0'
11/08/2019 17:12:34 MainProcess     MainThread      multithreading  join                      ERROR    Caught exception in thread: 'training_0'
11/08/2019 17:12:34 MainProcess     MainThread      plaidml_tools   initialize                DEBUG    PlaidML already initialized
11/08/2019 17:12:34 MainProcess     MainThread      plaidml_tools   get_supported_devices     DEBUG    [<plaidml._DeviceConfig object at 0x000002B362B3A9B0>]
11/08/2019 17:12:34 MainProcess     MainThread      plaidml_tools   get_all_devices           DEBUG    Experimental Devices: [<plaidml._DeviceConfig object at 0x000002B362B3AF98>]
11/08/2019 17:12:34 MainProcess     MainThread      plaidml_tools   get_all_devices           DEBUG    [<plaidml._DeviceConfig object at 0x000002B362B3AF98>, <plaidml._DeviceConfig object at 0x000002B362B3A9B0>]
11/08/2019 17:12:34 MainProcess     MainThread      plaidml_tools   __init__                  DEBUG    Initialized: PlaidMLStats
11/08/2019 17:12:34 MainProcess     MainThread      plaidml_tools   supported_indices         DEBUG    [1]
11/08/2019 17:12:34 MainProcess     MainThread      plaidml_tools   supported_indices         DEBUG    [1]
Traceback (most recent call last):
  File "E:\John\Documents\faceswap\lib\cli.py", line 128, in execute_script
    process.process()
  File "E:\John\Documents\faceswap\scripts\train.py", line 109, in process
    self.end_thread(thread, err)
  File "E:\John\Documents\faceswap\scripts\train.py", line 135, in end_thread
    thread.join()
  File "E:\John\Documents\faceswap\lib\multithreading.py", line 117, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "E:\John\Documents\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "E:\John\Documents\faceswap\scripts\train.py", line 160, in training
    raise err
  File "E:\John\Documents\faceswap\scripts\train.py", line 150, in training
    self.run_training_cycle(model, trainer)
  File "E:\John\Documents\faceswap\scripts\train.py", line 232, in run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "E:\John\Documents\faceswap\plugins\train\trainer\_base.py", line 211, in train_one_step
    raise err
  File "E:\John\Documents\faceswap\plugins\train\trainer\_base.py", line 179, in train_one_step
    loss[side] = batcher.train_one_batch(do_preview)
  File "E:\John\Documents\faceswap\plugins\train\trainer\_base.py", line 273, in train_one_batch
    batch = self.get_next(do_preview)
  File "E:\John\Documents\faceswap\plugins\train\trainer\_base.py", line 294, in get_next
    batch = next(self.feed)
  File "E:\John\Documents\faceswap\lib\multithreading.py", line 152, in iterator
    self.check_and_raise_error()
  File "E:\John\Documents\faceswap\lib\multithreading.py", line 84, in check_and_raise_error
    raise error[1].with_traceback(error[2])
  File "E:\John\Documents\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "E:\John\Documents\faceswap\lib\multithreading.py", line 141, in _run
    for item in self.generator(*self._gen_args, **self._gen_kwargs):
  File "E:\John\Documents\faceswap\lib\training_data.py", line 200, in _minibatch
    yield self._process_batch(img_paths, side)
  File "E:\John\Documents\faceswap\lib\training_data.py", line 222, in _process_batch
    batch = self._processing.color_adjust(batch)
  File "E:\John\Documents\faceswap\lib\training_data.py", line 513, in color_adjust
    batch = batch_convert_color(batch, "BGR2LAB")
  File "E:\John\Documents\faceswap\lib\image.py", line 256, in batch_convert_color
    batch = batch.reshape((original_shape[0] * original_shape[1], *original_shape[2:]))
IndexError: tuple index out of range

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         ffd3829 Added allow_growth argument for preview
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: Advanced Micro Devices, Inc. - Ellesmere (experimental), GPU_1: Advanced Micro Devices, Inc. - Ellesmere (supported)
gpu_devices_active:  GPU_0, GPU_1
gpu_driver:          ['2906.10', '2906.10']
gpu_vram:            GPU_0: 8192MB, GPU_1: 8192MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.18362-SP0
os_release:          10
py_command:          E:\John\Documents\faceswap\faceswap.py train -A E:/John/Documents/Deepfake/replace faces -B E:/John/Documents/Deepfake/page faces -m E:/John/Documents/Deepfake/Page ASMR model -t original -bs 66 -it 1000000 -s 100 -ss 25000 -ps 50 -L INFO -gui
py_conda_version:    conda 4.7.12
py_implementation:   CPython
py_version:          3.6.9
py_virtual_env:      True
sys_cores:           8
sys_processor:       Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
sys_ram:             Total: 16349MB, Available: 8048MB, Used: 8300MB, Free: 8048MB

=============== Pip Packages ===============
absl-py==0.8.1
astor==0.8.0
certifi==2019.9.11
cffi==1.13.2
cloudpickle==1.2.2
cycler==0.10.0
cytoolz==0.10.0
dask==2.6.0
decorator==4.4.1
enum34==1.1.6
fastcluster==1.1.25
ffmpy==0.2.2
gast==0.3.2
grpcio==1.16.1
h5py==2.9.0
imageio==2.5.0
imageio-ffmpeg==0.3.0
joblib==0.14.0
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
Markdown==3.1.1
matplotlib==2.2.2
mkl-fft==1.0.15
mkl-random==1.1.0
mkl-service==2.3.0
networkx==2.4
numpy==1.16.2
nvidia-ml-py3==7.352.1
olefile==0.46
opencv-python==4.1.1.26
pathlib==1.0.1
Pillow==6.1.0
plaidml==0.6.4
plaidml-keras==0.6.4
protobuf==3.9.2
psutil==5.6.3
pycparser==2.19
pyparsing==2.4.2
pyreadline==2.1
python-dateutil==2.8.1
pytz==2019.3
PyWavelets==1.1.1
pywin32==223
PyYAML==5.1.2
scikit-image==0.15.0
scikit-learn==0.21.3
scipy==1.3.1
six==1.12.0
tensorboard==1.14.0
tensorflow==1.14.0
tensorflow-estimator==1.14.0
termcolor==1.1.0
toolz==0.10.0
toposort==1.5
tornado==6.0.3
tqdm==4.36.1
Werkzeug==0.16.0
wincertstore==0.2
wrapt==1.11.2

============== Conda Packages ==============
# packages in environment at C:\Users\John\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
_tflow_select             2.3.0                       mkl  
absl-py 0.8.1 py36_0
astor 0.8.0 py36_0
blas 1.0 mkl
ca-certificates 2019.9.11 hecc5488_0 conda-forge certifi 2019.9.11 py36_0 conda-forge cffi 1.13.2 pypi_0 pypi cloudpickle 1.2.2 py_0
cycler 0.10.0 py36h009560c_0
cytoolz 0.10.0 py36he774522_0
dask-core 2.6.0 py_0
decorator 4.4.1 py_0
enum34 1.1.6 pypi_0 pypi fastcluster 1.1.25 py36he350917_1000 conda-forge ffmpeg 4.2 h6538335_0 conda-forge ffmpy 0.2.2 pypi_0 pypi freetype 2.9.1 ha9979f8_1
gast 0.3.2 py_0
grpcio 1.16.1 py36h351948d_1
h5py 2.9.0 py36h5e291fa_0
hdf5 1.10.4 h7ebc959_0
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha66f8fd_1
imageio 2.5.0 py36_0
imageio-ffmpeg 0.3.0 py_0 conda-forge intel-openmp 2019.4 245
joblib 0.14.0 py_0
jpeg 9b hb83a4c4_2
keras 2.2.4 pypi_0 pypi keras-applications 1.0.8 py_0
keras-preprocessing 1.1.0 py_1
kiwisolver 1.1.0 py36ha925a31_0
libmklml 2019.0.5 0
libpng 1.6.37 h2a8f88b_0
libprotobuf 3.9.2 h7bd577a_0
libtiff 4.1.0 h56a325e_0
markdown 3.1.1 py36_0
matplotlib 2.2.2 py36had4c4a9_2
mkl 2019.4 245
mkl-service 2.3.0 py36hb782905_0
mkl_fft 1.0.15 py36h14836fe_0
mkl_random 1.1.0 py36h675688f_0
networkx 2.4 py_0
numpy 1.16.2 py36h19fb1c0_0
numpy-base 1.16.2 py36hc3f5095_0
nvidia-ml-py3 7.352.1 pypi_0 pypi olefile 0.46 py36_0
opencv-python 4.1.1.26 pypi_0 pypi openssl 1.1.1d hfa6e2cd_0 conda-forge pathlib 1.0.1 py36_1
pillow 6.1.0 py36hdc69c19_0
pip 19.3.1 py36_0
plaidml 0.6.4 pypi_0 pypi plaidml-keras 0.6.4 pypi_0 pypi protobuf 3.9.2 py36h33f27b4_0
psutil 5.6.3 py36he774522_0
pycparser 2.19 pypi_0 pypi pyparsing 2.4.2 py_0
pyqt 5.9.2 py36h6538335_2
pyreadline 2.1 py36_1
python 3.6.9 h5500b2f_0
python-dateutil 2.8.1 py_0
pytz 2019.3 py_0
pywavelets 1.1.1 py36he774522_0
pywin32 223 py36hfa6e2cd_1
pyyaml 5.1.2 pypi_0 pypi qt 5.9.7 vc14h73c81de_0
scikit-image 0.15.0 py36ha925a31_0
scikit-learn 0.21.3 py36h6288b17_0
scipy 1.3.1 py36h29ff71c_0
setuptools 41.6.0 py36_0
sip 4.19.8 py36h6538335_0
six 1.12.0 py36_0
sqlite 3.30.1 he774522_0
tensorboard 1.14.0 py36he3c9ec2_0
tensorflow 1.14.0 mkl_py36hb88db5b_0
tensorflow-base 1.14.0 mkl_py36ha978198_0
tensorflow-estimator 1.14.0 py_0
termcolor 1.1.0 py36_1
tk 8.6.8 hfa6e2cd_0
toolz 0.10.0 py_0
toposort 1.5 py_3 conda-forge tornado 6.0.3 py36he774522_0
tqdm 4.36.1 py_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.16.27012 hf0eaf9b_0
werkzeug 0.16.0 py_0
wheel 0.33.6 py36_0
wincertstore 0.2 py36h7fe50ca_0
wrapt 1.11.2 py36he774522_0
xz 5.2.4 h2fa13f4_4
zlib 1.2.11 h62dcd97_3
zstd 1.3.7 h508b16e_0 =============== State File ================= { "name": "original", "sessions": { "1": { "timestamp": 1573212352.3809888, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 64, "iterations": 1, "config": { "learning_rate": 5e-05 } }, "2": { "timestamp": 1573212545.8027182, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 64, "iterations": 1, "config": { "learning_rate": 5e-05 } }, "3": { "timestamp": 1573245738.4151528, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 64, "iterations": 17, "config": { "learning_rate": 5e-05 } }, "4": { "timestamp": 1573247961.1000285, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 64, "iterations": 1, "config": { "learning_rate": 5e-05 } }, "5": { "timestamp": 1573248331.4945996, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 64, "iterations": 1, "config": { "learning_rate": 5e-05 } }, "6": { "timestamp": 1573249049.5537572, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 48, "iterations": 1, "config": { "learning_rate": 5e-05 } }, "7": { "timestamp": 1573250829.66932, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 66, "iterations": 48, "config": { "learning_rate": 5e-05 } }, "8": { "timestamp": 1573251016.1346729, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 66, "iterations": 1, "config": { "learning_rate": 5e-05 } } }, "lowest_avg_loss": { "a": 0.11725123971700668, "b": 0.1210833340883255 }, "iterations": 71, "inputs": { "face_in:0": [ 64, 64, 3 ] }, "training_size": 256, "config": { "coverage": 68.75, "mask_type": null, "mask_blur": false, "icnr_init": false, "conv_aware_init": false, "subpixel_upscaling": false, "reflect_padding": false, "penalized_mask_loss": true, "loss_function": "mae", "learning_rate": 5e-05, "lowmem": false } } ================= Configs ================== --------- .faceswap --------- backend: amd --------- convert.ini --------- [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [color.match_hist] threshold: 99.0 [mask.box_blend] type: gaussian distance: 11.0 radius: 5.0 passes: 1 [mask.mask_blend] type: normalized radius: 3.0 passes: 4 erosion: 0.0 [scaling.sharpen] method: unsharp_mask amount: 150 radius: 0.3 threshold: 5.0 [writer.ffmpeg] container: mp4 codec: libx264 crf: 23 preset: medium tune: none profile: auto level: auto [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.opencv] format: png draw_transparent: False jpg_quality: 75 png_compress_level: 3 [writer.pillow] format: png draw_transparent: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate --------- extract.ini --------- [global] allow_growth: False [align.fan] batch-size: 12 [detect.cv2_dnn] confidence: 50 [detect.mtcnn] minsize: 20 threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 scalefactor: 0.709 batch-size: 8 [detect.s3fd] confidence: 50 batch-size: 4 [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 2 --------- gui.ini --------- [global] fullscreen: False tab: extract options_panel_width: 30 console_panel_height: 20 font: default font_size: 9 --------- train.ini --------- [global] coverage: 68.75 mask_type: none mask_blur: False icnr_init: False conv_aware_init: False subpixel_upscaling: False reflect_padding: False penalized_mask_loss: True loss_function: mae learning_rate: 5e-05 [model.dfl_h128] lowmem: False [model.dfl_sae] input_size: 128 clipnorm: True architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False [model.dlight] features: best details: good output_size: 256 [model.original] lowmem: False [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.unbalanced] input_size: 128 lowmem: False clipnorm: True nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.villain] lowmem: False [trainer.original] preview_images: 14 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4

Tags:
User avatar
torzdf
Posts: 2649
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 128 times
Been thanked: 622 times

Re: Encountered CRITICAL An unexpected crash has occurred.

Post by torzdf »

It looks like a problem with one of your images.

Are you able to zip up your facesets and share somewhere so I can see if I can diagnose the issue?

My word is final

User avatar
I'mForKnights
Posts: 2
Joined: Fri Nov 08, 2019 10:15 pm

Re: Encountered CRITICAL An unexpected crash has occurred.

Post by I'mForKnights »

Nvm fixed it myself

Locked