Encountered CRITICAL An unexpected crash has occurred.

Training your model
Forum rules
Read the FAQs and search the forum before posting a new topic.

Please mark any answers that fixed your problems so others can find the solutions.
Locked
User avatar
I'mForKnights
Posts: 2
Joined: Fri Nov 08, 2019 10:15 pm

Encountered CRITICAL An unexpected crash has occurred.

Post by I'mForKnights »

I have no idea how I can fix this mess

Code: Select all

11/08/2019 17:10:37 MainProcess     _run_1          training_data   initialize                DEBUG    Initializing constants. training_size: 256
11/08/2019 17:10:37 MainProcess     _run_1          training_data   initialize                DEBUG    Initialized constants: {'clahe_base_contrast': 2, 'tgt_slices': slice(40, 216, None), 'warp_mapx': '[[[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]\n\n [[ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]\n  [ 40.  84. 128. 172. 216.]]]', 'warp_mapy': '[[[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]\n\n [[ 40.  40.  40.  40.  40.]\n  [ 84.  84.  84.  84.  84.]\n  [128. 128. 128. 128. 128.]\n  [172. 172. 172. 172. 172.]\n  [216. 216. 216. 216. 216.]]]', 'warp_pad': 80, 'warp_slices': slice(8, -8, None), 'warp_lm_edge_anchors': '[[[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]\n\n [[  0   0]\n  [  0 255]\n  [255 255]\n  [255   0]\n  [127   0]\n  [127 255]\n  [255 127]\n  [  0 127]]]', 'warp_lm_grids': '[[[  0.   0.   0. ...   0.   0.   0.]\n  [  1.   1.   1. ...   1.   1.   1.]\n  [  2.   2.   2. ...   2.   2.   2.]\n  ...\n  [253. 253. 253. ... 253. 253. 253.]\n  [254. 254. 254. ... 254. 254. 254.]\n  [255. 255. 255. ... 255. 255. 255.]]\n\n [[  0.   1.   2. ... 253. 254. 255.]\n  [  0.   1.   2. ... 253. 254. 255.]\n  [  0.   1.   2. ... 253. 254. 255.]\n  ...\n  [  0.   1.   2. ... 253. 254. 255.]\n  [  0.   1.   2. ... 253. 254. 255.]\n  [  0.   1.   2. ... 253. 254. 255.]]]'}
11/08/2019 17:10:37 MainProcess     training_0      _base           largest_face_index        DEBUG    0
11/08/2019 17:10:40 MainProcess     training_0      library         _logger_callback          INFO     Analyzing Ops: 270 of 520 operations complete
11/08/2019 17:10:52 MainProcess     training_0      _base           compile_sample            DEBUG    Compiling samples: (side: 'b', samples: 14)
11/08/2019 17:10:53 MainProcess     training_0      _base           show_sample               DEBUG    Showing sample
11/08/2019 17:10:53 MainProcess     training_0      _base           get_predictions           DEBUG    Getting Predictions
11/08/2019 17:11:00 MainProcess     training_0      _base           get_predictions           DEBUG    Returning predictions: {'a_a': (14, 64, 64, 3), 'b_a': (14, 64, 64, 3), 'a_b': (14, 64, 64, 3), 'b_b': (14, 64, 64, 3)}
11/08/2019 17:11:00 MainProcess     training_0      _base           to_full_frame             DEBUG    side: 'a', number of sample arrays: 2, prediction.shapes: [(14, 64, 64, 3), (14, 64, 64, 3)])
11/08/2019 17:11:00 MainProcess     training_0      _base           frame_overlay             DEBUG    full_size: 256, target_size: 176, color: (0, 0, 255)
11/08/2019 17:11:00 MainProcess     training_0      _base           frame_overlay             DEBUG    Overlayed background. Shape: (14, 256, 256, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'a' shape: (14, 176, 176, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'a' shape: (14, 176, 176, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'a' shape: (14, 176, 176, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           overlay_foreground        DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           overlay_foreground        DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           overlay_foreground        DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'a' shape: (14, 128, 128, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'a' shape: (14, 128, 128, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'a', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'a' shape: (14, 128, 128, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           get_headers               DEBUG    side: 'a', width: 128
11/08/2019 17:11:00 MainProcess     training_0      _base           get_headers               DEBUG    height: 32, total_width: 384
11/08/2019 17:11:00 MainProcess     training_0      _base           get_headers               DEBUG    texts: ['Original (A)', 'Original > Original', 'Original > Swap'], text_sizes: [(72, 9), (116, 9), (102, 9)], text_x: [28, 134, 269], text_y: 20
11/08/2019 17:11:00 MainProcess     training_0      _base           get_headers               DEBUG    header_box.shape: (32, 384, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           to_full_frame             DEBUG    side: 'b', number of sample arrays: 2, prediction.shapes: [(14, 64, 64, 3), (14, 64, 64, 3)])
11/08/2019 17:11:00 MainProcess     training_0      _base           frame_overlay             DEBUG    full_size: 256, target_size: 176, color: (0, 0, 255)
11/08/2019 17:11:00 MainProcess     training_0      _base           frame_overlay             DEBUG    Overlayed background. Shape: (14, 256, 256, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'b' shape: (14, 176, 176, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'b' shape: (14, 176, 176, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 64, 64, 3), target_size: 176, scale: 2.75)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'b' shape: (14, 176, 176, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           overlay_foreground        DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           overlay_foreground        DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           overlay_foreground        DEBUG    Overlayed foreground. Shape: (14, 256, 256, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'b' shape: (14, 128, 128, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'b' shape: (14, 128, 128, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resizing sample: (side: 'b', sample.shape: (14, 256, 256, 3), target_size: 128, scale: 0.5)
11/08/2019 17:11:00 MainProcess     training_0      _base           resize_sample             DEBUG    Resized sample: (side: 'b' shape: (14, 128, 128, 3))
11/08/2019 17:11:00 MainProcess     training_0      _base           get_headers               DEBUG    side: 'b', width: 128
11/08/2019 17:11:00 MainProcess     training_0      _base           get_headers               DEBUG    height: 32, total_width: 384
11/08/2019 17:11:00 MainProcess     training_0      _base           get_headers               DEBUG    texts: ['Swap (B)', 'Swap > Swap', 'Swap > Original'], text_sizes: [(59, 9), (87, 9), (102, 9)], text_x: [34, 148, 269], text_y: 20
11/08/2019 17:11:00 MainProcess     training_0      _base           get_headers               DEBUG    header_box.shape: (32, 384, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           duplicate_headers         DEBUG    side: a header.shape: (32, 384, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           duplicate_headers         DEBUG    side: b header.shape: (32, 384, 3)
11/08/2019 17:11:00 MainProcess     training_0      _base           stack_images              DEBUG    Stack images
11/08/2019 17:11:00 MainProcess     training_0      _base           get_transpose_axes        DEBUG    Even number of images to stack
11/08/2019 17:11:00 MainProcess     training_0      _base           stack_images              DEBUG    Stacked images
11/08/2019 17:11:00 MainProcess     training_0      _base           show_sample               DEBUG    Compiled sample
11/08/2019 17:11:00 MainProcess     training_0      _base           save_models               DEBUG    Backing up and saving models
11/08/2019 17:11:00 MainProcess     training_0      _base           get_save_averages         DEBUG    Getting save averages
11/08/2019 17:11:00 MainProcess     training_0      _base           get_save_averages         DEBUG    Average losses since last save: {'a': 0.11725123971700668, 'b': 0.1210833340883255}
11/08/2019 17:11:00 MainProcess     training_0      _base           check_loss_drop           DEBUG    Loss for 'a' has dropped
11/08/2019 17:11:00 MainProcess     training_0      _base           check_loss_drop           DEBUG    Loss for 'b' has dropped
11/08/2019 17:11:00 MainProcess     training_0      _base           should_backup             DEBUG    Lowest historical save iteration loss average: {'a': 0.12577251677817486, 'b': 0.13642760413758298}
11/08/2019 17:11:00 MainProcess     training_0      _base           should_backup             DEBUG    Updating lowest save iteration average for 'a': 0.11725123971700668
11/08/2019 17:11:00 MainProcess     training_0      _base           should_backup             DEBUG    Updating lowest save iteration average for 'b': 0.1210833340883255
11/08/2019 17:11:00 MainProcess     training_0      _base           should_backup             DEBUG    Backing up: True
11/08/2019 17:11:00 MainProcess     training_0      _base           save_models               INFO     Backing up models...
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_0 backup_restore  backup_model              VERBOSE  Backing up: 'E:\John\Documents\Deepfake\Page ASMR model\original_decoder_A.h5' to 'E:\John\Documents\Deepfake\Page ASMR model\original_decoder_A.h5.bk'
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_1 backup_restore  backup_model              VERBOSE  Backing up: 'E:\John\Documents\Deepfake\Page ASMR model\original_decoder_B.h5' to 'E:\John\Documents\Deepfake\Page ASMR model\original_decoder_B.h5.bk'
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_2 backup_restore  backup_model              VERBOSE  Backing up: 'E:\John\Documents\Deepfake\Page ASMR model\original_encoder.h5' to 'E:\John\Documents\Deepfake\Page ASMR model\original_encoder.h5.bk'
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_3 _base           save                      DEBUG    Saving State
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_3 backup_restore  backup_model              VERBOSE  Backing up: 'E:\John\Documents\Deepfake\Page ASMR model\original_state.json' to 'E:\John\Documents\Deepfake\Page ASMR model\original_state.json.bk'
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_0 _base           save                      DEBUG    Saving model: 'E:\John\Documents\Deepfake\Page ASMR model\original_decoder_A.h5'
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_2 _base           save                      DEBUG    Saving model: 'E:\John\Documents\Deepfake\Page ASMR model\original_encoder.h5'
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_3 serializer      save                      DEBUG    filename: E:\John\Documents\Deepfake\Page ASMR model\original_state.json, data type: <class 'dict'>
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_3 serializer      _check_extension          DEBUG    Original filename: 'E:\John\Documents\Deepfake\Page ASMR model\original_state.json', final filename: 'E:\John\Documents\Deepfake\Page ASMR model\original_state.json'
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_1 _base           save                      DEBUG    Saving model: 'E:\John\Documents\Deepfake\Page ASMR model\original_decoder_B.h5'
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_3 serializer      marshal                   DEBUG    data type: <class 'dict'>
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_3 serializer      marshal                   DEBUG    returned data type: <class 'bytes'>
11/08/2019 17:11:00 MainProcess     ThreadPoolExecutor-16_3 _base           save                      DEBUG    Saved State
11/08/2019 17:11:02 MainProcess     training_0      _base           save_models               INFO     [Saved models] - Average since last save: face_loss_A: 0.11725, face_loss_B: 0.12108
11/08/2019 17:12:31 MainProcess     _run_1          multithreading  run                       DEBUG    Error in thread (_run_1): tuple index out of range
11/08/2019 17:12:34 MainProcess     training_0      multithreading  check_and_raise_error     DEBUG    Thread error caught: [(<class 'IndexError'>, IndexError('tuple index out of range',), <traceback object at 0x000002B387EFE808>)]
11/08/2019 17:12:34 MainProcess     training_0      multithreading  run                       DEBUG    Error in thread (training_0): tuple index out of range
11/08/2019 17:12:34 MainProcess     _run_0          multithreading  run                       DEBUG    Error in thread (_run_0): tuple index out of range
11/08/2019 17:12:34 MainProcess     MainThread      train           monitor                   DEBUG    Thread error detected
11/08/2019 17:12:34 MainProcess     MainThread      train           monitor                   DEBUG    Closed Monitor
11/08/2019 17:12:34 MainProcess     MainThread      train           end_thread                DEBUG    Ending Training thread
11/08/2019 17:12:34 MainProcess     MainThread      train           end_thread                CRITICAL Error caught! Exiting...
11/08/2019 17:12:34 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Threads: 'training'
11/08/2019 17:12:34 MainProcess     MainThread      multithreading  join                      DEBUG    Joining Thread: 'training_0'
11/08/2019 17:12:34 MainProcess     MainThread      multithreading  join                      ERROR    Caught exception in thread: 'training_0'
11/08/2019 17:12:34 MainProcess     MainThread      plaidml_tools   initialize                DEBUG    PlaidML already initialized
11/08/2019 17:12:34 MainProcess     MainThread      plaidml_tools   get_supported_devices     DEBUG    [<plaidml._DeviceConfig object at 0x000002B362B3A9B0>]
11/08/2019 17:12:34 MainProcess     MainThread      plaidml_tools   get_all_devices           DEBUG    Experimental Devices: [<plaidml._DeviceConfig object at 0x000002B362B3AF98>]
11/08/2019 17:12:34 MainProcess     MainThread      plaidml_tools   get_all_devices           DEBUG    [<plaidml._DeviceConfig object at 0x000002B362B3AF98>, <plaidml._DeviceConfig object at 0x000002B362B3A9B0>]
11/08/2019 17:12:34 MainProcess     MainThread      plaidml_tools   __init__                  DEBUG    Initialized: PlaidMLStats
11/08/2019 17:12:34 MainProcess     MainThread      plaidml_tools   supported_indices         DEBUG    [1]
11/08/2019 17:12:34 MainProcess     MainThread      plaidml_tools   supported_indices         DEBUG    [1]
Traceback (most recent call last):
  File "E:\John\Documents\faceswap\lib\cli.py", line 128, in execute_script
    process.process()
  File "E:\John\Documents\faceswap\scripts\train.py", line 109, in process
    self.end_thread(thread, err)
  File "E:\John\Documents\faceswap\scripts\train.py", line 135, in end_thread
    thread.join()
  File "E:\John\Documents\faceswap\lib\multithreading.py", line 117, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "E:\John\Documents\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "E:\John\Documents\faceswap\scripts\train.py", line 160, in training
    raise err
  File "E:\John\Documents\faceswap\scripts\train.py", line 150, in training
    self.run_training_cycle(model, trainer)
  File "E:\John\Documents\faceswap\scripts\train.py", line 232, in run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "E:\John\Documents\faceswap\plugins\train\trainer\_base.py", line 211, in train_one_step
    raise err
  File "E:\John\Documents\faceswap\plugins\train\trainer\_base.py", line 179, in train_one_step
    loss[side] = batcher.train_one_batch(do_preview)
  File "E:\John\Documents\faceswap\plugins\train\trainer\_base.py", line 273, in train_one_batch
    batch = self.get_next(do_preview)
  File "E:\John\Documents\faceswap\plugins\train\trainer\_base.py", line 294, in get_next
    batch = next(self.feed)
  File "E:\John\Documents\faceswap\lib\multithreading.py", line 152, in iterator
    self.check_and_raise_error()
  File "E:\John\Documents\faceswap\lib\multithreading.py", line 84, in check_and_raise_error
    raise error[1].with_traceback(error[2])
  File "E:\John\Documents\faceswap\lib\multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "E:\John\Documents\faceswap\lib\multithreading.py", line 141, in _run
    for item in self.generator(*self._gen_args, **self._gen_kwargs):
  File "E:\John\Documents\faceswap\lib\training_data.py", line 200, in _minibatch
    yield self._process_batch(img_paths, side)
  File "E:\John\Documents\faceswap\lib\training_data.py", line 222, in _process_batch
    batch = self._processing.color_adjust(batch)
  File "E:\John\Documents\faceswap\lib\training_data.py", line 513, in color_adjust
    batch = batch_convert_color(batch, "BGR2LAB")
  File "E:\John\Documents\faceswap\lib\image.py", line 256, in batch_convert_color
    batch = batch.reshape((original_shape[0] * original_shape[1], *original_shape[2:]))
IndexError: tuple index out of range

============ System Information ============
encoding:            cp1252
git_branch:          master
git_commits:         ffd3829 Added allow_growth argument for preview
gpu_cuda:            No global version found. Check Conda packages for Conda Cuda
gpu_cudnn:           No global version found. Check Conda packages for Conda cuDNN
gpu_devices:         GPU_0: Advanced Micro Devices, Inc. - Ellesmere (experimental), GPU_1: Advanced Micro Devices, Inc. - Ellesmere (supported)
gpu_devices_active:  GPU_0, GPU_1
gpu_driver:          ['2906.10', '2906.10']
gpu_vram:            GPU_0: 8192MB, GPU_1: 8192MB
os_machine:          AMD64
os_platform:         Windows-10-10.0.18362-SP0
os_release:          10
py_command:          E:\John\Documents\faceswap\faceswap.py train -A E:/John/Documents/Deepfake/replace faces -B E:/John/Documents/Deepfake/page faces -m E:/John/Documents/Deepfake/Page ASMR model -t original -bs 66 -it 1000000 -s 100 -ss 25000 -ps 50 -L INFO -gui
py_conda_version:    conda 4.7.12
py_implementation:   CPython
py_version:          3.6.9
py_virtual_env:      True
sys_cores:           8
sys_processor:       Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
sys_ram:             Total: 16349MB, Available: 8048MB, Used: 8300MB, Free: 8048MB

=============== Pip Packages ===============
absl-py==0.8.1
astor==0.8.0
certifi==2019.9.11
cffi==1.13.2
cloudpickle==1.2.2
cycler==0.10.0
cytoolz==0.10.0
dask==2.6.0
decorator==4.4.1
enum34==1.1.6
fastcluster==1.1.25
ffmpy==0.2.2
gast==0.3.2
grpcio==1.16.1
h5py==2.9.0
imageio==2.5.0
imageio-ffmpeg==0.3.0
joblib==0.14.0
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
Markdown==3.1.1
matplotlib==2.2.2
mkl-fft==1.0.15
mkl-random==1.1.0
mkl-service==2.3.0
networkx==2.4
numpy==1.16.2
nvidia-ml-py3==7.352.1
olefile==0.46
opencv-python==4.1.1.26
pathlib==1.0.1
Pillow==6.1.0
plaidml==0.6.4
plaidml-keras==0.6.4
protobuf==3.9.2
psutil==5.6.3
pycparser==2.19
pyparsing==2.4.2
pyreadline==2.1
python-dateutil==2.8.1
pytz==2019.3
PyWavelets==1.1.1
pywin32==223
PyYAML==5.1.2
scikit-image==0.15.0
scikit-learn==0.21.3
scipy==1.3.1
six==1.12.0
tensorboard==1.14.0
tensorflow==1.14.0
tensorflow-estimator==1.14.0
termcolor==1.1.0
toolz==0.10.0
toposort==1.5
tornado==6.0.3
tqdm==4.36.1
Werkzeug==0.16.0
wincertstore==0.2
wrapt==1.11.2

============== Conda Packages ==============
# packages in environment at C:\Users\John\MiniConda3\envs\faceswap:
#
# Name                    Version                   Build  Channel
_tflow_select             2.3.0                       mkl  
absl-py 0.8.1 py36_0
astor 0.8.0 py36_0
blas 1.0 mkl
ca-certificates 2019.9.11 hecc5488_0 conda-forge certifi 2019.9.11 py36_0 conda-forge cffi 1.13.2 pypi_0 pypi cloudpickle 1.2.2 py_0
cycler 0.10.0 py36h009560c_0
cytoolz 0.10.0 py36he774522_0
dask-core 2.6.0 py_0
decorator 4.4.1 py_0
enum34 1.1.6 pypi_0 pypi fastcluster 1.1.25 py36he350917_1000 conda-forge ffmpeg 4.2 h6538335_0 conda-forge ffmpy 0.2.2 pypi_0 pypi freetype 2.9.1 ha9979f8_1
gast 0.3.2 py_0
grpcio 1.16.1 py36h351948d_1
h5py 2.9.0 py36h5e291fa_0
hdf5 1.10.4 h7ebc959_0
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha66f8fd_1
imageio 2.5.0 py36_0
imageio-ffmpeg 0.3.0 py_0 conda-forge intel-openmp 2019.4 245
joblib 0.14.0 py_0
jpeg 9b hb83a4c4_2
keras 2.2.4 pypi_0 pypi keras-applications 1.0.8 py_0
keras-preprocessing 1.1.0 py_1
kiwisolver 1.1.0 py36ha925a31_0
libmklml 2019.0.5 0
libpng 1.6.37 h2a8f88b_0
libprotobuf 3.9.2 h7bd577a_0
libtiff 4.1.0 h56a325e_0
markdown 3.1.1 py36_0
matplotlib 2.2.2 py36had4c4a9_2
mkl 2019.4 245
mkl-service 2.3.0 py36hb782905_0
mkl_fft 1.0.15 py36h14836fe_0
mkl_random 1.1.0 py36h675688f_0
networkx 2.4 py_0
numpy 1.16.2 py36h19fb1c0_0
numpy-base 1.16.2 py36hc3f5095_0
nvidia-ml-py3 7.352.1 pypi_0 pypi olefile 0.46 py36_0
opencv-python 4.1.1.26 pypi_0 pypi openssl 1.1.1d hfa6e2cd_0 conda-forge pathlib 1.0.1 py36_1
pillow 6.1.0 py36hdc69c19_0
pip 19.3.1 py36_0
plaidml 0.6.4 pypi_0 pypi plaidml-keras 0.6.4 pypi_0 pypi protobuf 3.9.2 py36h33f27b4_0
psutil 5.6.3 py36he774522_0
pycparser 2.19 pypi_0 pypi pyparsing 2.4.2 py_0
pyqt 5.9.2 py36h6538335_2
pyreadline 2.1 py36_1
python 3.6.9 h5500b2f_0
python-dateutil 2.8.1 py_0
pytz 2019.3 py_0
pywavelets 1.1.1 py36he774522_0
pywin32 223 py36hfa6e2cd_1
pyyaml 5.1.2 pypi_0 pypi qt 5.9.7 vc14h73c81de_0
scikit-image 0.15.0 py36ha925a31_0
scikit-learn 0.21.3 py36h6288b17_0
scipy 1.3.1 py36h29ff71c_0
setuptools 41.6.0 py36_0
sip 4.19.8 py36h6538335_0
six 1.12.0 py36_0
sqlite 3.30.1 he774522_0
tensorboard 1.14.0 py36he3c9ec2_0
tensorflow 1.14.0 mkl_py36hb88db5b_0
tensorflow-base 1.14.0 mkl_py36ha978198_0
tensorflow-estimator 1.14.0 py_0
termcolor 1.1.0 py36_1
tk 8.6.8 hfa6e2cd_0
toolz 0.10.0 py_0
toposort 1.5 py_3 conda-forge tornado 6.0.3 py36he774522_0
tqdm 4.36.1 py_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.16.27012 hf0eaf9b_0
werkzeug 0.16.0 py_0
wheel 0.33.6 py36_0
wincertstore 0.2 py36h7fe50ca_0
wrapt 1.11.2 py36he774522_0
xz 5.2.4 h2fa13f4_4
zlib 1.2.11 h62dcd97_3
zstd 1.3.7 h508b16e_0 =============== State File ================= { "name": "original", "sessions": { "1": { "timestamp": 1573212352.3809888, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 64, "iterations": 1, "config": { "learning_rate": 5e-05 } }, "2": { "timestamp": 1573212545.8027182, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 64, "iterations": 1, "config": { "learning_rate": 5e-05 } }, "3": { "timestamp": 1573245738.4151528, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 64, "iterations": 17, "config": { "learning_rate": 5e-05 } }, "4": { "timestamp": 1573247961.1000285, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 64, "iterations": 1, "config": { "learning_rate": 5e-05 } }, "5": { "timestamp": 1573248331.4945996, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 64, "iterations": 1, "config": { "learning_rate": 5e-05 } }, "6": { "timestamp": 1573249049.5537572, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 48, "iterations": 1, "config": { "learning_rate": 5e-05 } }, "7": { "timestamp": 1573250829.66932, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 66, "iterations": 48, "config": { "learning_rate": 5e-05 } }, "8": { "timestamp": 1573251016.1346729, "no_logs": false, "pingpong": false, "loss_names": { "a": [ "face_loss" ], "b": [ "face_loss" ] }, "batchsize": 66, "iterations": 1, "config": { "learning_rate": 5e-05 } } }, "lowest_avg_loss": { "a": 0.11725123971700668, "b": 0.1210833340883255 }, "iterations": 71, "inputs": { "face_in:0": [ 64, 64, 3 ] }, "training_size": 256, "config": { "coverage": 68.75, "mask_type": null, "mask_blur": false, "icnr_init": false, "conv_aware_init": false, "subpixel_upscaling": false, "reflect_padding": false, "penalized_mask_loss": true, "loss_function": "mae", "learning_rate": 5e-05, "lowmem": false } } ================= Configs ================== --------- .faceswap --------- backend: amd --------- convert.ini --------- [color.color_transfer] clip: True preserve_paper: True [color.manual_balance] colorspace: HSV balance_1: 0.0 balance_2: 0.0 balance_3: 0.0 contrast: 0.0 brightness: 0.0 [color.match_hist] threshold: 99.0 [mask.box_blend] type: gaussian distance: 11.0 radius: 5.0 passes: 1 [mask.mask_blend] type: normalized radius: 3.0 passes: 4 erosion: 0.0 [scaling.sharpen] method: unsharp_mask amount: 150 radius: 0.3 threshold: 5.0 [writer.ffmpeg] container: mp4 codec: libx264 crf: 23 preset: medium tune: none profile: auto level: auto [writer.gif] fps: 25 loop: 0 palettesize: 256 subrectangles: False [writer.opencv] format: png draw_transparent: False jpg_quality: 75 png_compress_level: 3 [writer.pillow] format: png draw_transparent: False optimize: False gif_interlace: True jpg_quality: 75 png_compress_level: 3 tif_compression: tiff_deflate --------- extract.ini --------- [global] allow_growth: False [align.fan] batch-size: 12 [detect.cv2_dnn] confidence: 50 [detect.mtcnn] minsize: 20 threshold_1: 0.6 threshold_2: 0.7 threshold_3: 0.7 scalefactor: 0.709 batch-size: 8 [detect.s3fd] confidence: 50 batch-size: 4 [mask.unet_dfl] batch-size: 8 [mask.vgg_clear] batch-size: 6 [mask.vgg_obstructed] batch-size: 2 --------- gui.ini --------- [global] fullscreen: False tab: extract options_panel_width: 30 console_panel_height: 20 font: default font_size: 9 --------- train.ini --------- [global] coverage: 68.75 mask_type: none mask_blur: False icnr_init: False conv_aware_init: False subpixel_upscaling: False reflect_padding: False penalized_mask_loss: True loss_function: mae learning_rate: 5e-05 [model.dfl_h128] lowmem: False [model.dfl_sae] input_size: 128 clipnorm: True architecture: df autoencoder_dims: 0 encoder_dims: 42 decoder_dims: 21 multiscale_decoder: False [model.dlight] features: best details: good output_size: 256 [model.original] lowmem: False [model.realface] input_size: 64 output_size: 128 dense_nodes: 1536 complexity_encoder: 128 complexity_decoder: 512 [model.unbalanced] input_size: 128 lowmem: False clipnorm: True nodes: 1024 complexity_encoder: 128 complexity_decoder_a: 384 complexity_decoder_b: 512 [model.villain] lowmem: False [trainer.original] preview_images: 14 zoom_amount: 5 rotation_range: 10 shift_range: 5 flip_chance: 50 color_lightness: 30 color_ab: 8 color_clahe_chance: 50 color_clahe_max_size: 4

Tags:

User avatar
torzdf
Posts: 996
Joined: Fri Jul 12, 2019 12:53 am
Answers: 127
Has thanked: 28 times
Been thanked: 191 times

Re: Encountered CRITICAL An unexpected crash has occurred.

Post by torzdf »

It looks like a problem with one of your images.

Are you able to zip up your facesets and share somewhere so I can see if I can diagnose the issue?

My word is final


User avatar
I'mForKnights
Posts: 2
Joined: Fri Nov 08, 2019 10:15 pm

Re: Encountered CRITICAL An unexpected crash has occurred.

Post by I'mForKnights »

Nvm fixed it myself


Locked