Sudden error in the middle of training session

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Locked
User avatar
aolvera27
Posts: 27
Joined: Thu May 27, 2021 3:53 am
Answers: 1
Has thanked: 5 times
Been thanked: 6 times

Sudden error in the middle of training session

Post by aolvera27 »

(I'm reposting here 'cause I just noticed I had posted this on the wrong forum) I'm getting an error in the middle of training. Everything seems to be running just fine, then about a couple of hundreds of iterations into the session it gives back an "ERROR Caught exception in thread: '_training_0'", and a "ERROR Got Exception on main handler..."

I checked to see if I added a wrong (non-extracted) file by mistake, but I couldn't find any. I just added some thousand new faces to set B after 50k iterations, if that information is of any utility.

Code: Select all

09/16/2021 00:06:43 MainProcess     _training_0                    _base           _save                          DEBUG    Backing up and saving models
09/16/2021 00:06:43 MainProcess     _training_0                    _base           _get_save_averages             DEBUG    Getting save averages
09/16/2021 00:06:43 MainProcess     _training_0                    _base           _get_save_averages             DEBUG    Average losses since last save: [0.038943224345284584, 0.03432052828952629]
09/16/2021 00:06:43 MainProcess     _training_0                    _base           _should_backup                 DEBUG    Should backup: False
09/16/2021 00:06:50 MainProcess     _training_0                    _base           save                           DEBUG    Saving State
09/16/2021 00:06:50 MainProcess     _training_0                    serializer      save                           DEBUG    filename: /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json, data type: <class 'dict'>
09/16/2021 00:06:50 MainProcess     _training_0                    serializer      _check_extension               DEBUG    Original filename: '/content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json', final filename: '/content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json'
09/16/2021 00:06:50 MainProcess     _training_0                    serializer      marshal                        DEBUG    data type: <class 'dict'>
09/16/2021 00:06:50 MainProcess     _training_0                    serializer      marshal                        DEBUG    returned data type: <class 'bytes'>
09/16/2021 00:06:50 MainProcess     _training_0                    _base           save                           DEBUG    Saved State
09/16/2021 00:06:50 MainProcess     _training_0                    _base           _save                          INFO     [Saved models] - Average loss since last save: face_a: 0.03894, face_b: 0.03432
09/16/2021 00:06:50 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada1.mp4_06910_0.png', 'Screen Shot 2021-08-23 at 11.05.16 PM_0.png', 'Screen Shot 2021-08-24 at 10.33.40 PM_0.png', 'jada1.mp4_03167_0.png', 'Screen Shot 2021-08-12 at 12.04.44 PM_0.png', 'Screen Shot 2021-08-19 at 1.36.59 AM_0.png', 'Screen Shot 2021-08-18 at 11.49.01 PM_0.png', 'jada3_000469_0.png']
09/16/2021 00:06:50 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (1225).png', 'step2_3 (1803).png', 'step2_3 (2060).png', 'step2_3 (2735).png', 'step2_3 (1347).png', 'step2_3 (3434).png', 'step2_3 (1470).png', 'step2_3 (1549).png']
09/16/2021 00:06:55 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada2.mp4_15552_0.png', 'jada1.mp4_15219_0.png', 'jada2.mp4_07780_0.png', 'Screen Shot 2021-08-24 at 10.27.07 PM_0.png', 'jada1.mp4_16136_0.png', 'Screen Shot 2021-08-12 at 12.05.18 PM_0.png', 'jada1.mp4_01568_0.png', 'Screen Shot 2021-08-12 at 10.23.16 AM_0.png']
09/16/2021 00:06:55 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (1460).png', 'step2_3 (883).png', 'step2_3 (1352).png', 'step2_3 (1421).png', 'step2_3 (2242).png', 'step2_3 (1658).png', 'step2_3 (3397).png', 'step2_3 (2849).png']
09/16/2021 00:06:59 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-19 at 12.16.44 AM_0.png', 'jada1.mp4_01451_0.png', 'Screen Shot 2021-08-24 at 10.45.36 PM_0.png', 'jada2.mp4_06824_0.png', 'Screen Shot 2021-08-11 at 7.46.31 PM_0.png', 'Screen Shot 2021-08-12 at 7.14.13 AM_0.png', 'jada2.mp4_09537_0.png', 'jada1.mp4_08598_0.png']
09/16/2021 00:06:59 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (75).png', 'step2_3 (1050).png', 'step2_3 (1723).png', 'step2_3 (1448).png', 'step2_3 (1408).png', 'step2_3 (64).png', 'step2_3 (348).png', 'step2_3 (250).png']
09/16/2021 00:07:04 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-11 at 7.53.08 PM_0.png', 'Screen Shot 2021-08-12 at 11.50.46 AM_0.png', 'Screen Shot 2021-08-12 at 10.34.06 AM_0.png', 'Screen Shot 2021-08-20 at 8.02.02 AM_0.png', 'Screen Shot 2021-08-12 at 12.08.27 PM_0.png', 'Screen Shot 2021-08-19 at 12.58.37 AM_0.png', 'jada3_004843_0.png', 'jada3_002485_0.png']
09/16/2021 00:07:04 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (92).png', 'step2_3 (332).png', 'step2_3 (362).png', 'step2_3 (3225).png', 'step2_3 (2278).png', 'step2_3 (2186).png', 'step2_3 (3201).png', 'step2_3 (2323).png']
09/16/2021 00:07:08 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada1.mp4_18369_0.png', 'Screen Shot 2021-08-12 at 10.25.37 AM_0.png', 'jada2.mp4_06553_0.png', 'jada2.mp4_07783_0.png', 'jada1.mp4_07405_0.png', 'Screen Shot 2021-08-19 at 1.23.56 AM_0.png', 'Screen Shot 2021-08-18 at 11.53.13 PM_0.png', 'jada1.mp4_08555_0.png']
09/16/2021 00:07:08 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (2340).png', 'step2_3 (1645).png', 'step2_3 (3429).png', 'step2_3 (333).png', 'step2_3 (227).png', 'step2_3 (2225).png', 'step2_3 (1299).png', 'step2_3 (620).png']
09/16/2021 00:07:12 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada1.mp4_03337_0.png', 'Screen Shot 2021-08-24 at 10.41.58 PM_0.png', 'Screen Shot 2021-08-24 at 10.54.41 PM_0.png', 'Screen Shot 2021-08-20 at 7.54.14 AM_0.png', 'Screen Shot 2021-08-12 at 1.45.06 PM_0.png', 'Screen Shot 2021-08-12 at 11.51.13 AM_0.png', 'Screen Shot 2021-08-18 at 11.52.53 PM_0.png', 'jada1.mp4_03668_0.png']
09/16/2021 00:07:12 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (874).png', 'step2_3 (2531).png', 'step2_3 (60).png', 'step2_3 (1106).png', 'step2_3 (908).png', 'step2_3 (2337).png', 'step2_3 (162).png', 'step2_3 (2063).png']
09/16/2021 00:07:17 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-23 at 11.15.27 PM_0.png', 'Screen Shot 2021-08-19 at 12.00.28 AM_0.png', 'Screen Shot 2021-08-12 at 7.41.49 AM_0.png', 'jada1.mp4_14930_0.png', 'jada3_001981_0.png', 'Screen Shot 2021-08-12 at 11.49.24 AM_0.png', 'Screen Shot 2021-08-18 at 11.54.08 PM_0.png', 'jada2.mp4_15606_0.png']
09/16/2021 00:07:17 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (345).png', 'step2_3 (584).png', 'step2_3 (2216).png', 'step2_3 (175).png', 'step2_3 (1220).png', 'step2_3 (1505).png', 'step2_3 (2872).png', 'step2_3 (2094).png']
09/16/2021 00:07:21 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada2.mp4_08545_0.png', 'Screen Shot 2021-08-24 at 10.04.04 PM_0.png', 'Screen Shot 2021-08-19 at 12.45.38 AM_0.png', 'jada1.mp4_01675_0.png', 'Screen Shot 2021-08-24 at 10.52.26 PM_0.png', 'Screen Shot 2021-08-19 at 1.05.40 AM_0.png', 'Screen Shot 2021-08-23 at 11.00.30 PM_0.png', 'jada1.mp4_11315_0.png']
09/16/2021 00:07:21 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (3295).png', 'step2_3 (1326).png', 'step2_3 (3327).png', 'step2_3 (2726).png', 'step2_3 (937).png', 'step2_3 (2329).png', 'step2_3 (13).png', 'step2_3 (1824).png']
09/16/2021 00:07:26 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada3_003421_0.png', 'Screen Shot 2021-08-12 at 11.54.24 AM_0.png', 'Screen Shot 2021-08-23 at 11.29.24 PM_0.png', 'jada1.mp4_15745_0.png', 'jada1.mp4_12057_0.png', 'jada1.mp4_14893_0.png', 'Screen Shot 2021-08-19 at 1.22.23 AM_0.png', 'jada1.mp4_10019_0.png']
09/16/2021 00:07:26 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (3212).png', 'step2_3 (2620).png', 'step2_3 (1091).png', 'step2_3 (25).png', 'step2_3 (1805).png', 'side (10).png', 'step2_3 (2280).png', 'step2_3 (1619).png']
09/16/2021 00:07:30 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada1.mp4_04349_0.png', 'Screen Shot 2021-08-12 at 11.42.19 AM_0.png', 'jada1.mp4_11897_0.png', 'Screen Shot 2021-08-12 at 11.59.15 AM_0.png', 'jada2.mp4_14424_0.png', 'jada2.mp4_10104_0.png', 'jada1.mp4_00573_0.png', 'Screen Shot 2021-08-24 at 10.06.20 PM_0.png']
09/16/2021 00:07:30 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (1743).png', 'step2_3 (2157).png', 'step2_3 (3093).png', 'step2_3 (2741).png', 'step2_3 (2701).png', 'step2_3 (3394).png', 'step2_3 (2485).png', 'step2_3 (310).png']
09/16/2021 00:07:34 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-12 at 7.04.34 AM_0.png', 'Screen Shot 2021-08-19 at 12.41.04 AM_0.png', 'jada2.mp4_15342_0.png', 'jada2.mp4_13956_0.png', 'Screen Shot 2021-08-20 at 8.00.13 AM_0.png', 'jada2.mp4_06836_0.png', 'Screen Shot 2021-08-24 at 10.38.52 PM_0.png', 'Screen Shot 2021-08-23 at 11.27.10 PM_0.png']
09/16/2021 00:07:34 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (2859).png', 'step2_3 (1856).png', 'step2_3 (1309).png', 'step2_3 (335).png', 'step2_3 (1280).png', 'step2_3 (454).png', 'step2_3 (270).png', 'step2_3 (1232).png']
09/16/2021 00:07:39 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-24 at 10.34.53 PM_0.png', 'Screen Shot 2021-08-12 at 11.50.07 AM_0.png', 'jada1.mp4_00657_0.png', 'Screen Shot 2021-08-23 at 11.36.24 PM_0.png', 'jada1.mp4_06883_0.png', 'Screen Shot 2021-08-19 at 1.19.23 AM_0.png', 'Screen Shot 2021-08-23 at 11.12.10 PM_0.png', 'jada2.mp4_06345_0.png']
09/16/2021 00:07:39 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (1291).png', 'step2_3 (1518).png', 'step2_3 (58).png', 'step2_3 (1273).png', 'step2_3 (1834).png', 'step2_3 (2607).png', 'step2_3 (1001).png', 'step2_3 (2180).png']
09/16/2021 00:07:43 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-24 at 10.24.47 PM_0.png', 'jada1.mp4_04230_0.png', 'jada1.mp4_03528_0.png', 'jada2.mp4_15864_0.png', 'jada1.mp4_11714_0.png', 'jada3_007705_0.png', 'jada1.mp4_18059_0.png', 'jada2.mp4_10073_0.png']
09/16/2021 00:07:43 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (2716).png', 'step2_3 (477).png', 'step2_3 (3210).png', 'step2_3 (1338).png', 'step2_3 (2431).png', 'step2_3 (2493).png', 'step2_3 (3418).png', 'step2_3 (153).png']
09/16/2021 00:07:48 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-23 at 11.19.50 PM_0.png', 'jada1.mp4_11685_0.png', 'jada1.mp4_00054_0.png', 'jada2.mp4_08770_0.png', 'jada2.mp4_08667_0.png', 'jada2.mp4_09669_0.png', 'jada1.mp4_12886_0.png', 'jada1.mp4_16431_0.png']
09/16/2021 00:07:48 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (690).png', 'step2_3 (2532).png', 'step2_3 (3027).png', 'step2_3 (783).png', 'step2_3 (940).png', 'step2_3 (1040).png', 'step2_3 (956).png', 'step2_3 (1126).png']
09/16/2021 00:07:52 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-24 at 10.44.22 PM_0.png', 'jada2.mp4_14850_0.png', 'jada1.mp4_01485_0.png', 'jada1.mp4_11736_0.png', 'Screen Shot 2021-08-20 at 7.59.06 AM_0.png', 'Screen Shot 2021-08-24 at 10.26.50 PM_0.png', 'jada2.mp4_14406_0.png', 'Screen Shot 2021-08-19 at 12.24.02 AM_0.png']
09/16/2021 00:07:52 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (3111).png', 'step2_3 (944).png', 'step2_3 (2262).png', 'step2_3 (2430).png', 'step2_3 (131).png', 'step2_3 (872).png', 'step2_3 (2847).png', 'step2_3 (3080).png']
09/16/2021 00:07:56 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada1.mp4_00215_0.png', 'jada3_003277_0.png', 'Screen Shot 2021-08-24 at 10.33.49 PM_0.png', 'jada1.mp4_18084_0.png', 'Screen Shot 2021-08-19 at 1.10.20 AM_0.png', 'jada3_005077_0.png', 'Screen Shot 2021-08-24 at 10.57.17 PM_0.png', 'Screen Shot 2021-08-19 at 12.27.14 AM_0.png']
09/16/2021 00:07:56 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (1544).png', 'step2_3 (2610).png', 'step2_3 (3363).png', 'step2_3 (192).png', 'step2_3 (2760).png', 'step2_3 (1441).png', 'step2_3 (1708).png', 'step2_3 (2852).png']
09/16/2021 00:08:01 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada2.mp4_06477_0.png', 'jada1.mp4_06899_0.png', 'jada1.mp4_00707_0.png', 'Screen Shot 2021-08-12 at 10.22.03 AM_0.png', 'jada1.mp4_03650_0.png', 'Screen Shot 2021-08-12 at 11.50.18 AM_0.png', 'Screen Shot 2021-08-12 at 1.40.41 PM_0.png', 'jada1.mp4_04407_0.png']
09/16/2021 00:08:01 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (2161).png', 'step2_3 (508).png', 'step2_3 (2817).png', 'step2_3 (878).png', 'step2_3 (2971).png', 'step2_3 (3362).png', 'step2_3 (54).png', 'step2_3 (3043).png']
09/16/2021 00:08:05 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-19 at 1.22.55 AM_0.png', 'Screen Shot 2021-08-19 at 12.20.01 AM_0.png', 'Screen Shot 2021-08-24 at 10.05.31 PM_0.png', 'jada2.mp4_12876_1.png', 'jada1.mp4_11856_0.png', 'jada2.mp4_09680_0.png', 'jada3_003799_0.png', 'jada1.mp4_04479_0.png']
09/16/2021 00:08:05 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (463).png', 'step2_3 (2174).png', 'step2_3 (1583).png', 'step2_3 (2386).png', 'step2_3 (2743).png', 'step2_3 (2331).png', 'step2_3 (114).png', 'step2_3 (340).png']
09/16/2021 00:08:09 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-19 at 1.15.05 AM_0.png', 'Screen Shot 2021-08-23 at 11.05.53 PM_0.png', 'jada2.mp4_08989_0.png', 'Screen Shot 2021-08-12 at 11.42.04 AM_0.png', 'Screen Shot 2021-08-19 at 12.53.56 AM_0.png', 'jada3_000361_0.png', 'jada1.mp4_00056_0.png', 'Screen Shot 2021-08-24 at 10.35.34 PM_0.png']
09/16/2021 00:08:09 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (1064).png', 'step2_3 (2082).png', 'step2_3 (1572).png', 'step2_3 (1084).png', 'step2_3 (1274).png', 'step2_3 (1458).png', 'step2_3 (2513).png', 'step2_3 (3182).png']
09/16/2021 00:08:14 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada2.mp4_06317_0.png', 'Screen Shot 2021-08-19 at 12.36.23 AM_0.png', 'Screen Shot 2021-08-24 at 10.51.49 PM_0.png', 'Screen Shot 2021-08-19 at 1.20.56 AM_0.png', 'jada2.mp4_08822_0.png', 'jada2.mp4_09604_0.png', 'Screen Shot 2021-08-19 at 12.42.16 AM_0.png', 'jada1.mp4_03711_0.png']
09/16/2021 00:08:14 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (1485).png', 'step2_3 (2026).png', 'step2_3 (2188).png', 'step2_3 (1283).png', 'step2_3 (3148).png', 'step2_3 (2846).png', 'step2_3 (211).png', 'step2_3 (1111).png']
09/16/2021 00:08:18 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada2.mp4_08792_0.png', 'Screen Shot 2021-08-11 at 7.47.20 PM_0.png', 'jada2.mp4_14910_0.png', 'Screen Shot 2021-08-12 at 12.02.32 PM_0.png', 'jada1.mp4_14544_0.png', 'Screen Shot 2021-08-24 at 10.25.49 PM_0.png', 'jada1.mp4_18185_0.png', 'Screen Shot 2021-08-12 at 11.54.56 AM_0.png']
09/16/2021 00:08:18 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (3195).png', 'step2_3 (1577).png', 'step2_3 (1595).png', 'step2_3 (1207).png', 'step2_3 (1415).png', 'step2_3 (207).png', 'step2_3 (2466).png', 'step2_3 (236).png']
09/16/2021 00:08:23 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada1.mp4_03560_0.png', 'jada2.mp4_12732_0.png', 'jada2.mp4_15882_0.png', 'jada2.mp4_10002_0.png', 'Screen Shot 2021-08-23 at 10.54.51 PM_0.png', 'jada1.mp4_18349_0.png', 'jada3_000181_0.png', 'jada2.mp4_15900_0.png']
09/16/2021 00:08:23 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (3270).png', 'step2_3 (3085).png', 'step2_3 (1687).png', 'step2_3 (1857).png', 'step2_3 (1115).png', 'step2_3 (796).png', 'step2_3 (1488).png', 'step2_3 (2525).png']
09/16/2021 00:08:27 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada3_004375_0.png', 'jada3_001819_0.png', 'Screen Shot 2021-08-19 at 12.51.40 AM_0.png', 'jada1.mp4_00964_0.png', 'jada3_002683_0.png', 'Screen Shot 2021-08-23 at 10.57.42 PM_0.png', 'jada1.mp4_14146_0.png', 'Screen Shot 2021-08-18 at 11.53.40 PM_0.png']
09/16/2021 00:08:27 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (3236).png', 'step2_3 (1713).png', 'step2_3 (2914).png', 'step2_3 (2458).png', 'step2_3 (2765).png', 'step2_3 (1372).png', 'step2_3 (2758).png', 'step2_3 (500).png']
09/16/2021 00:08:32 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-12 at 1.43.01 PM_0.png', 'jada2.mp4_14874_0.png', 'jada1.mp4_10291_0.png', 'jada2.mp4_10717_0.png', 'Screen Shot 2021-08-19 at 12.13.14 AM_0.png', 'jada1.mp4_04193_0.png', 'Screen Shot 2021-08-19 at 1.00.05 AM_0.png', 'jada2.mp4_09611_0.png']
09/16/2021 00:08:32 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (41).png', 'step2_3 (401).png', 'step2_3 (2099).png', 'step2_3 (1248).png', 'step2_3 (925).png', 'step2_3 (3380).png', 'step2_3 (2958).png', 'step2_3 (2043).png']
09/16/2021 00:08:36 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-19 at 12.57.18 AM_0.png', 'Screen Shot 2021-08-20 at 7.58.42 AM_0.png', 'jada1.mp4_12212_0.png', 'Screen Shot 2021-08-24 at 10.28.20 PM_0.png', 'jada2.mp4_06952_0.png', 'jada2.mp4_06173_0.png', 'jada1.mp4_09895_0.png', 'jada1.mp4_15077_0.png']
09/16/2021 00:08:36 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (2642).png', 'step2_3 (2204).png', 'step2_3 (1481).png', 'step2_3 (2169).png', 'step2_3 (2771).png', 'step2_3 (2056).png', 'step2_3 (2700).png', 'step2_3 (1189).png']
09/16/2021 00:08:40 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-24 at 10.31.04 PM_0.png', 'Screen Shot 2021-08-19 at 12.57.51 AM_0.png', 'Screen Shot 2021-08-19 at 12.38.22 AM_0.png', 'Screen Shot 2021-08-23 at 10.53.07 PM_0.png', 'Screen Shot 2021-08-12 at 7.53.39 AM_0.png', 'jada2.mp4_14922_0.png', 'jada1.mp4_12000_0.png', 'Screen Shot 2021-08-19 at 12.51.55 AM_0.png']
09/16/2021 00:08:40 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (221).png', 'step2_3 (1277).png', 'step2_3 (1063).png', 'step2_3 (1061).png', 'step2_3 (731).png', 'step2_3 (1227).png', 'step2_3 (581).png', 'step2_3 (182).png']
09/16/2021 00:08:45 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-12 at 11.43.31 AM_0.png', 'Screen Shot 2021-08-19 at 12.46.09 AM_0.png', 'jada2.mp4_10055_0.png', 'jada1.mp4_03661_0.png', 'Screen Shot 2021-08-12 at 10.40.11 AM_0.png', 'jada2.mp4_15378_0.png', 'Screen Shot 2021-08-24 at 10.41.53 PM_0.png', 'Screen Shot 2021-08-23 at 11.20.48 PM_0.png']
09/16/2021 00:08:45 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (2745).png', 'step2_3 (1801).png', 'step2_3 (6).png', 'step2_3 (1862).png', 'step2_3 (1788).png', 'step2_3 (1166).png', 'step2_3 (1095).png', 'step2_3 (3297).png']
09/16/2021 00:08:49 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-24 at 10.40.14 PM_0.png', 'Screen Shot 2021-08-23 at 11.33.37 PM_0.png', 'Screen Shot 2021-08-23 at 11.18.46 PM_0.png', 'Screen Shot 2021-08-23 at 11.01.15 PM_0.png', 'jada2.mp4_07487_0.png', 'jada1.mp4_04486_0.png', 'Screen Shot 2021-08-23 at 11.29.35 PM_0.png', 'Screen Shot 2021-08-19 at 1.24.44 AM_0.png']
09/16/2021 00:08:49 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (3088).png', 'step2_3 (1833).png', 'step2_3 (3328).png', 'step2_3 (2536).png', 'step2_3 (626).png', 'step2_3 (2646).png', 'step2_3 (873).png', 'step2_3 (2366).png']
09/16/2021 00:08:53 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada2.mp4_10301_0.png', 'jada1.mp4_10085_0.png', 'Screen Shot 2021-08-12 at 11.46.29 AM_0.png', 'jada1.mp4_01157_0.png', 'Screen Shot 2021-08-12 at 10.30.44 AM_0.png', 'Screen Shot 2021-08-19 at 12.23.30 AM_0.png', 'jada2.mp4_12516_1.png', 'jada2.mp4_09969_0.png']
09/16/2021 00:08:53 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (216).png', 'step2_3 (474).png', 'step2_3 (1478).png', 'step2_3 (2753).png', 'step2_3 (1183).png', 'step2_3 (1885).png', 'step2_3 (2938).png', 'step2_3 (3315).png']
09/16/2021 00:08:58 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada1.mp4_08395_0.png', 'Screen Shot 2021-08-12 at 11.49.10 AM_0.png', 'Screen Shot 2021-08-12 at 11.46.09 AM_0.png', 'jada3_004249_0.png', 'jada2.mp4_08393_0.png', 'Screen Shot 2021-08-12 at 11.45.46 AM_0.png', 'Screen Shot 2021-08-12 at 10.43.39 AM_0.png', 'Screen Shot 2021-08-23 at 11.06.10 PM_0.png']
09/16/2021 00:08:58 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (3202).png', 'step2_3 (1081).png', 'step2_3 (961).png', 'step2_3 (3389).png', 'step2_3 (3431).png', 'step2_3 (2162).png', 'step2_3 (1942).png', 'step2_3 (2901).png']
09/16/2021 00:09:02 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-19 at 12.37.58 AM_0.png', 'Screen Shot 2021-08-12 at 1.41.15 PM_0.png', 'jada2.mp4_09361_0.png', 'jada2.mp4_15450_0.png', 'Screen Shot 2021-08-19 at 1.37.14 AM_0.png', 'Screen Shot 2021-08-19 at 1.21.50 AM_0.png', 'Screen Shot 2021-08-19 at 1.06.03 AM_0.png', 'jada2.mp4_14490_0.png']
09/16/2021 00:09:02 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (1447).png', 'step2_3 (1286).png', 'step2_3 (2949).png', 'step2_3 (322).png', 'step2_3 (1043).png', 'step2_3 (820).png', 'step2_3 (1682).png', 'step2_3 (765).png']
09/16/2021 00:09:07 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-12 at 7.29.48 AM_0.png', 'jada1.mp4_14015_0.png', 'Screen Shot 2021-08-12 at 10.42.43 AM_0.png', 'jada1.mp4_00909_0.png', 'Screen Shot 2021-08-12 at 12.03.42 PM_0.png', 'Screen Shot 2021-08-19 at 1.38.06 AM_0.png', 'jada1.mp4_03384_0.png', 'Screen Shot 2021-08-19 at 1.22.40 AM_0.png']
09/16/2021 00:09:07 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (1995).png', 'step2_3 (379).png', 'step2_3 (1499).png', 'step2_3 (1006).png', 'step2_3 (2794).png', 'step2_3 (2230).png', 'side (1).png', 'step2_3 (1503).png']
09/16/2021 00:09:11 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-12 at 7.06.51 AM_0.png', 'jada1.mp4_12497_0.png', 'Screen Shot 2021-08-23 at 11.06.53 PM_0.png', 'jada1.mp4_17135_0.png', 'Screen Shot 2021-08-24 at 10.34.37 PM_0.png', 'Screen Shot 2021-08-12 at 7.18.31 AM_0.png', 'Screen Shot 2021-08-12 at 11.36.25 AM_0.png', 'Screen Shot 2021-08-12 at 12.07.34 PM_0.png']
09/16/2021 00:09:11 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (304).png', 'step2_3 (1129).png', 'step2_3 (1265).png', 'step2_3 (1491).png', 'step2_3 (862).png', 'step2_3 (2059).png', 'step2_3 (1853).png', 'step2_3 (573).png']
09/16/2021 00:09:15 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada2.mp4_10599_1.png', 'Screen Shot 2021-08-19 at 12.11.44 AM_0.png', 'jada1.mp4_03853_0.png', 'Screen Shot 2021-08-12 at 10.34.53 AM_0.png', 'Screen Shot 2021-08-12 at 7.30.10 AM_0.png', 'jada1.mp4_01392_0.png', 'jada1.mp4_03749_0.png', 'Screen Shot 2021-08-11 at 7.42.39 PM_0.png']
09/16/2021 00:09:15 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (2305).png', 'step2_3 (1442).png', 'step2_3 (2739).png', 'step2_3 (1871).png', 'step2_3 (224).png', 'step2_3 (1948).png', 'step2_3 (2652).png', 'step2_3 (2259).png']
09/16/2021 00:09:20 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-12 at 12.11.49 PM_0.png', 'jada3_003223_0.png', 'Screen Shot 2021-08-12 at 11.50.24 AM_0.png', 'Screen Shot 2021-08-24 at 10.41.48 PM_0.png', 'jada1.mp4_08129_0.png', 'Screen Shot 2021-08-12 at 11.57.37 AM_0.png', 'Screen Shot 2021-08-23 at 11.09.38 PM_0.png', 'jada1.mp4_05908_0.png']
09/16/2021 00:09:20 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (356).png', 'step2_3 (2020).png', 'step2_3 (397).png', 'step2_3 (2882).png', 'step2_3 (2130).png', 'step2_3 (2718).png', 'step2_3 (2930).png', 'step2_3 (2413).png']
09/16/2021 00:09:24 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada1.mp4_04526_0.png', 'jada1.mp4_04093_0.png', 'jada2.mp4_14418_0.png', 'Screen Shot 2021-08-18 at 11.43.56 PM_0.png', 'jada2.mp4_13506_0.png', 'jada1.mp4_04539_0.png', 'jada1.mp4_06044_0.png', 'Screen Shot 2021-08-24 at 10.11.59 PM_0.png']
09/16/2021 00:09:24 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (3063).png', 'step2_3 (2275).png', 'step2_3 (1241).png', 'step2_3 (2856).png', 'step2_3 (1046).png', 'step2_3 (953).png', 'step2_3 (1114).png', 'step2_3 (506).png']
09/16/2021 00:09:29 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-20 at 7.59.43 AM_0.png', 'Screen Shot 2021-08-20 at 8.01.32 AM_0.png', 'Screen Shot 2021-08-24 at 10.05.22 PM_0.png', 'jada1.mp4_15942_0.png', 'jada3_003205_0.png', 'Screen Shot 2021-08-19 at 1.08.50 AM_0.png', 'jada1.mp4_00920_0.png', 'jada3_005959_0.png']
09/16/2021 00:09:29 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (2025).png', 'step2_3 (1173).png', 'step2_3 (580).png', 'step2_3 (2603).png', 'step2_3 (1208).png', 'step2_3 (2221).png', 'step2_3 (3072).png', 'step2_3 (1184).png']
09/16/2021 00:09:33 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada1.mp4_05357_0.png', 'jada1.mp4_00435_0.png', 'jada2.mp4_15678_0.png', 'Screen Shot 2021-08-19 at 12.59.13 AM_0.png', 'Screen Shot 2021-08-12 at 11.35.15 AM_0.png', 'Screen Shot 2021-08-24 at 10.31.42 PM_0.png', 'jada2.mp4_13848_0.png', 'jada1.mp4_04008_0.png']
09/16/2021 00:09:33 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['step2_3 (1733).png', 'step2_3 (3033).png', 'step2_3 (1244).png', 'step2_3 (2731).png', 'step2_3 (2024).png', 'step2_3 (244).png', 'step2_3 (2860).png', 'step2_3 (3303).png']
09/16/2021 00:09:37 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['jada2.mp4_14820_0.png', 'jada2.mp4_13290_0.png', 'Screen Shot 2021-08-19 at 1.15.34 AM_0.png', 'Screen Shot 2021-08-20 at 7.56.47 AM_0.png', 'jada2.mp4_09096_0.png', 'Screen Shot 2021-08-12 at 12.06.05 PM_0.png', 'Screen Shot 2021-08-24 at 10.47.11 PM_0.png', 'jada1.mp4_07394_0.png']
09/16/2021 00:09:40 MainProcess     _run_0                         multithreading  run                            DEBUG    Error in thread (_run_0): 'NoneType' object is not subscriptable
09/16/2021 00:09:42 MainProcess     _run_1                         generator       cache_metadata                 DEBUG    All metadata already cached for: ['Screen Shot 2021-08-23 at 10.58.49 PM_0.png', 'Screen Shot 2021-08-24 at 10.52.59 PM_0.png', 'jada1.mp4_00912_0.png', 'jada2.mp4_07431_0.png', 'Screen Shot 2021-08-23 at 10.55.51 PM_0.png', 'Screen Shot 2021-08-19 at 1.18.48 AM_0.png', 'jada3_004357_0.png', 'jada1.mp4_00292_0.png']
09/16/2021 00:09:42 MainProcess     _training_0                    multithreading  check_and_raise_error          DEBUG    Thread error caught: [(<class 'TypeError'>, TypeError("'NoneType' object is not subscriptable"), <traceback object at 0x7f3830413370>)]
09/16/2021 00:09:42 MainProcess     _training_0                    multithreading  run                            DEBUG    Error in thread (_training_0): 'NoneType' object is not subscriptable
09/16/2021 00:09:42 MainProcess     _run_1                         multithreading  run                            DEBUG    Error in thread (_run_1): 'NoneType' object is not subscriptable
09/16/2021 00:09:42 MainProcess     MainThread                     train           _monitor                       DEBUG    Thread error detected
09/16/2021 00:09:42 MainProcess     MainThread                     train           _monitor                       DEBUG    Closed Monitor
09/16/2021 00:09:42 MainProcess     MainThread                     train           _end_thread                    DEBUG    Ending Training thread
09/16/2021 00:09:42 MainProcess     MainThread                     train           _end_thread                    CRITICAL Error caught! Exiting...
09/16/2021 00:09:42 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Threads: '_training'
09/16/2021 00:09:42 MainProcess     MainThread                     multithreading  join                           DEBUG    Joining Thread: '_training_0'
09/16/2021 00:09:42 MainProcess     MainThread                     multithreading  join                           ERROR    Caught exception in thread: '_training_0'
Traceback (most recent call last):
  File "/content/faceswap/lib/cli/launcher.py", line 182, in execute_script
    process.process()
  File "/content/faceswap/scripts/train.py", line 190, in process
    self._end_thread(thread, err)
  File "/content/faceswap/scripts/train.py", line 230, in _end_thread
    thread.join()
  File "/content/faceswap/lib/multithreading.py", line 121, in join
    raise thread.err[1].with_traceback(thread.err[2])
  File "/content/faceswap/lib/multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "/content/faceswap/scripts/train.py", line 252, in _training
    raise err
  File "/content/faceswap/scripts/train.py", line 242, in _training
    self._run_training_cycle(model, trainer)
  File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
    trainer.train_one_step(viewer, timelapse)
  File "/content/faceswap/plugins/train/trainer/_base.py", line 191, in train_one_step
    model_inputs, model_targets = self._feeder.get_batch()
  File "/content/faceswap/plugins/train/trainer/_base.py", line 416, in get_batch
    batch = next(self._feeds[side])
  File "/content/faceswap/lib/multithreading.py", line 156, in iterator
    self.check_and_raise_error()
  File "/content/faceswap/lib/multithreading.py", line 84, in check_and_raise_error
    raise error[1].with_traceback(error[2])
  File "/content/faceswap/lib/multithreading.py", line 37, in run
    self._target(*self._args, **self._kwargs)
  File "/content/faceswap/lib/multithreading.py", line 145, in _run
    for item in self.generator(*self._gen_args, **self._gen_kwargs):
  File "/content/faceswap/lib/training/generator.py", line 598, in _minibatch
    yield self._process_batch(img_paths, side)
  File "/content/faceswap/lib/training/generator.py", line 621, in _process_batch
    batch = self._face_cache.cache_metadata(filenames)
  File "/content/faceswap/lib/training/generator.py", line 224, in cache_metadata
    self._validate_version(meta, filename)
  File "/content/faceswap/lib/training/generator.py", line 287, in _validate_version
    alignment_version = png_meta["source"]["alignments_version"]
TypeError: 'NoneType' object is not subscriptable

============ System Information ============
encoding:            UTF-8
git_branch:          Not Found
git_commits:         Not Found
gpu_cuda:            11.0
gpu_cudnn:           No global version found
gpu_devices:         GPU_0: Tesla K80
gpu_devices_active:  GPU_0
gpu_driver:          460.32.03
gpu_vram:            GPU_0: 11441MB
os_machine:          x86_64
os_platform:         Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
os_release:          5.4.104+
py_command:          faceswap/faceswap.py train -A face_a -B face_b -m /content/drive/My Drive/colab_files/faceswap/models/JdKQry -t villain -bs 8 -it 100000 -s 360 -ss 25000 -tia face_a -tib face_b -to /content/drive/My Drive/colab_files/faceswap/output/timelapse
py_conda_version:    N/A
py_implementation:   CPython
py_version:          3.7.11
py_virtual_env:      False
sys_cores:           2
sys_processor:       x86_64
sys_ram:             Total: 12991MB, Available: 10595MB, Used: 3980MB, Free: 168MB

=============== Pip Packages ===============
absl-py==0.12.0
alabaster==0.7.12
albumentations==0.1.12
altair==4.1.0
appdirs==1.4.4
argcomplete==1.12.3
argon2-cffi==21.1.0
arviz==0.11.2
astor==0.8.1
astropy==4.3.1
astunparse==1.6.3
atari-py==0.2.9
atomicwrites==1.4.0
attrs==21.2.0
audioread==2.1.9
autograd==1.3
Babel==2.9.1
backcall==0.2.0
beautifulsoup4==4.6.3
bleach==4.0.0
blis==0.4.1
bokeh==2.3.3
Bottleneck==1.3.2
branca==0.4.2
bs4==0.0.1
CacheControl==0.12.6
cached-property==1.5.2
cachetools==4.2.2
catalogue==1.0.0
certifi==2021.5.30
cffi==1.14.6
cftime==1.5.0
chardet==3.0.4
charset-normalizer==2.0.4
clang==5.0
click==7.1.2
cloudpickle==1.3.0
cmake==3.12.0
cmdstanpy==0.9.5
colorcet==2.0.6
colorlover==0.3.0
community==1.0.0b1
contextlib2==0.5.5
convertdate==2.3.2
coverage==3.7.1
coveralls==0.5
crcmod==1.7
cufflinks==0.17.3
cupy-cuda101==9.1.0
cvxopt==1.2.6
cvxpy==1.0.31
cycler==0.10.0
cymem==2.0.5
Cython==0.29.24
daft==0.0.4
dask==2.12.0
datascience==0.10.6
debugpy==1.0.0
decorator==4.4.2
defusedxml==0.7.1
descartes==1.1.0
dill==0.3.4
distributed==1.25.3
dlib @ file:///dlib-19.18.0-cp37-cp37m-linux_x86_64.whl
dm-tree==0.1.6
docopt==0.6.2
docutils==0.17.1
dopamine-rl==1.0.5
earthengine-api==0.1.278
easydict==1.9
ecos==2.0.7.post1
editdistance==0.5.3
en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.5/en_core_web_sm-2.2.5.tar.gz
entrypoints==0.3
ephem==4.0.0.2
et-xmlfile==1.1.0
fa2==0.3.5
fastai==1.0.61
fastcluster==1.2.4
fastdtw==0.3.4
fastprogress==1.0.0
fastrlock==0.6
fbprophet==0.7.1
feather-format==0.4.1
ffmpy==0.2.3
filelock==3.0.12
firebase-admin==4.4.0
fix-yahoo-finance==0.0.22
Flask==1.1.4
flatbuffers==1.12
folium==0.8.3
future==0.16.0
gast==0.3.3
GDAL==2.2.2
gdown==3.6.4
gensim==3.6.0
geographiclib==1.52
geopy==1.17.0
gin-config==0.4.0
glob2==0.7
google==2.0.3
google-api-core==1.26.3
google-api-python-client==1.12.8
google-auth==1.34.0
google-auth-httplib2==0.0.4
google-auth-oauthlib==0.4.5
google-cloud-bigquery==1.21.0
google-cloud-bigquery-storage==1.1.0
google-cloud-core==1.0.3
google-cloud-datastore==1.8.0
google-cloud-firestore==1.7.0
google-cloud-language==1.2.0
google-cloud-storage==1.18.1
google-cloud-translate==1.5.0
google-colab @ file:///colabtools/dist/google-colab-1.0.0.tar.gz
google-pasta==0.2.0
google-resumable-media==0.4.1
googleapis-common-protos==1.53.0
googledrivedownloader==0.4
GPUtil==1.4.0
graphviz==0.10.1
greenlet==1.1.1
grpcio==1.32.0
gspread==3.0.1
gspread-dataframe==3.0.8
gym==0.17.3
h5py==2.10.0
HeapDict==1.0.1
hijri-converter==2.1.3
holidays==0.10.5.2
holoviews==1.14.5
html5lib==1.0.1
httpimport==0.5.18
httplib2==0.17.4
httplib2shim==0.0.3
humanize==0.5.1
hyperopt==0.1.2
ideep4py==2.0.0.post3
idna==2.10
imageio==2.9.0
imageio-ffmpeg==0.4.5
imagesize==1.2.0
imbalanced-learn==0.4.3
imblearn==0.0
imgaug==0.2.9
importlib-metadata==4.6.4
importlib-resources==5.2.2
imutils==0.5.4
inflect==2.1.0
iniconfig==1.1.1
intel-openmp==2021.3.0
intervaltree==2.1.0
ipykernel==4.10.1
ipython==5.5.0
ipython-genutils==0.2.0
ipython-sql==0.3.9
ipywidgets==7.6.3
itsdangerous==1.1.0
jax==0.2.19
jaxlib @ https://storage.googleapis.com/jax-releases/cuda110/jaxlib-0.1.70+cuda110-cp37-none-manylinux2010_x86_64.whl
jdcal==1.4.1
jedi==0.18.0
jieba==0.42.1
Jinja2==2.11.3
joblib==1.0.1
jpeg4py==0.1.4
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.3.5
jupyter-console==5.2.0
jupyter-core==4.7.1
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.0
kaggle==1.5.12
kapre==0.3.5
keras==2.6.0
Keras-Preprocessing==1.1.2
keras-vis==0.4.1
kiwisolver==1.3.1
korean-lunar-calendar==0.2.1
librosa==0.8.1
lightgbm==2.2.3
llvmlite==0.34.0
lmdb==0.99
LunarCalendar==0.0.9
lxml==4.2.6
Markdown==3.3.4
MarkupSafe==2.0.1
matplotlib==3.2.2
matplotlib-inline==0.1.2
matplotlib-venn==0.11.6
missingno==0.5.0
mistune==0.8.4
mizani==0.6.0
mkl==2019.0
mlxtend==0.14.0
more-itertools==8.8.0
moviepy==0.2.3.5
mpmath==1.2.1
msgpack==1.0.2
multiprocess==0.70.12.2
multitasking==0.0.9
murmurhash==1.0.5
music21==5.5.0
natsort==5.5.0
nbclient==0.5.4
nbconvert==5.6.1
nbformat==5.1.3
nest-asyncio==1.5.1
netCDF4==1.5.7
networkx==2.6.2
nibabel==3.0.2
nltk==3.2.5
notebook==5.3.1
numba==0.51.2
numexpr==2.7.3
numpy==1.19.5
nvidia-ml-py==11.470.66
nvidia-ml-py3==7.352.0
oauth2client==4.1.3
oauthlib==3.1.1
okgrade==0.4.3
opencv-contrib-python==4.1.2.30
opencv-python==4.5.3.56
openpyxl==2.5.9
opt-einsum==3.3.0
osqp==0.6.2.post0
packaging==21.0
palettable==3.3.0
pandas==1.1.5
pandas-datareader==0.9.0
pandas-gbq==0.13.3
pandas-profiling==1.4.1
pandocfilters==1.4.3
panel==0.12.1
param==1.11.1
parso==0.8.2
pathlib==1.0.1
patsy==0.5.1
pep517==0.11.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.3.2
pip-tools==6.2.0
plac==1.1.3
plotly==4.4.1
plotnine==0.6.0
pluggy==0.7.1
pooch==1.4.0
portpicker==1.3.9
prefetch-generator==1.0.1
preshed==3.0.5
prettytable==2.1.0
progressbar2==3.38.0
prometheus-client==0.11.0
promise==2.3
prompt-toolkit==1.0.18
protobuf==3.17.3
psutil==5.8.0
psycopg2==2.7.6.1
ptyprocess==0.7.0
py==1.10.0
pyarrow==3.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycocotools==2.0.2
pycparser==2.20
pyct==0.4.8
pydata-google-auth==1.2.0
pydot==1.3.0
pydot-ng==2.0.0
pydotplus==2.0.2
PyDrive==1.3.1
pyemd==0.5.1
pyerfa==2.0.0
pyglet==1.5.0
Pygments==2.6.1
pygobject==3.26.1
pymc3==3.11.2
PyMeeus==0.5.11
pymongo==3.12.0
pymystem3==0.2.0
PyOpenGL==3.1.5
pyparsing==2.4.7
pyrsistent==0.18.0
pysndfile==1.3.8
PySocks==1.7.1
pystan==2.19.1.1
pytest==3.6.4
python-apt==0.0.0
python-chess==0.23.11
python-dateutil==2.8.2
python-louvain==0.15
python-slugify==5.0.2
python-utils==2.5.6
pytz==2018.9
pyviz-comms==2.1.0
PyWavelets==1.1.1
PyYAML==3.13
pyzmq==22.2.1
qdldl==0.1.5.post0
qtconsole==5.1.1
QtPy==1.10.0
regex==2019.12.20
requests==2.23.0
requests-oauthlib==1.3.0
resampy==0.2.2
retrying==1.3.3
rpy2==3.4.5
rsa==4.7.2
scikit-image==0.16.2
scikit-learn==0.24.2
scipy==1.4.1
screen-resolution-extra==0.0.0
scs==2.1.4
seaborn==0.11.1
semver==2.13.0
Send2Trash==1.8.0
setuptools-git==1.2
Shapely==1.7.1
simplegeneric==0.8.1
six==1.15.0
sklearn==0.0
sklearn-pandas==1.8.0
smart-open==5.1.0
snowballstemmer==2.1.0
sortedcontainers==2.4.0
SoundFile==0.10.3.post1
spacy==2.2.4
Sphinx==1.8.5
sphinxcontrib-serializinghtml==1.1.5
sphinxcontrib-websupport==1.2.4
SQLAlchemy==1.4.22
sqlparse==0.4.1
srsly==1.0.5
statsmodels==0.10.2
sympy==1.7.1
tables==3.4.4
tabulate==0.8.9
tblib==1.7.0
tensorboard==2.6.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorflow==2.4.0
tensorflow-datasets==4.0.1
tensorflow-estimator==2.4.0
tensorflow-gcs-config==2.6.0
tensorflow-gpu==2.4.0
tensorflow-hub==0.12.0
tensorflow-metadata==1.2.0
tensorflow-probability==0.13.0
termcolor==1.1.0
terminado==0.11.0
testpath==0.5.0
text-unidecode==1.3
textblob==0.15.3
Theano-PyMC==1.1.2
thinc==7.4.0
threadpoolctl==2.2.0
tifffile==2021.8.30
toml==0.10.2
tomli==1.2.1
toolz==0.11.1
torch @ https://download.pytorch.org/whl/cu102/torch-1.9.0%2Bcu102-cp37-cp37m-linux_x86_64.whl
torchsummary==1.5.1
torchtext==0.10.0
torchvision @ https://download.pytorch.org/whl/cu102/torchvision-0.10.0%2Bcu102-cp37-cp37m-linux_x86_64.whl
tornado==5.1.1
tqdm==4.62.0
traitlets==5.0.5
tweepy==3.10.0
typeguard==2.7.1
typing-extensions==3.7.4.3
tzlocal==1.5.1
uritemplate==3.0.1
urllib3==1.24.3
vega-datasets==0.9.0
wasabi==0.8.2
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==1.0.1
widgetsnbextension==3.5.1
wordcloud==1.5.0
wrapt==1.12.1
xarray==0.18.2
xgboost==0.90
xkit==0.0.0
xlrd==1.1.0
xlwt==1.3.0
yellowbrick==0.9.1
zict==2.0.0
zipp==3.5.0

=============== State File =================
{
  "name": "villain",
  "sessions": {
    "1": {
      "timestamp": 1631431573.2035065,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 3960,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "2": {
      "timestamp": 1631485930.8518226,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 3960,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "3": {
      "timestamp": 1631503174.5515265,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 3600,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "4": {
      "timestamp": 1631541392.1880405,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 7560,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "5": {
      "timestamp": 1631568333.871523,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "6": {
      "timestamp": 1631568527.213653,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "7": {
      "timestamp": 1631572690.7077212,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "8": {
      "timestamp": 1631573832.957988,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 61,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "9": {
      "timestamp": 1631574142.30032,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "10": {
      "timestamp": 1631576717.8594546,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 3600,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "11": {
      "timestamp": 1631592607.7613473,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1800,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "12": {
      "timestamp": 1631621477.0965884,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 42,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "13": {
      "timestamp": 1631628807.7275484,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 5798,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "14": {
      "timestamp": 1631647215.6946719,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1440,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "15": {
      "timestamp": 1631663237.8258858,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 5811,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "16": {
      "timestamp": 1631707637.9922519,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 7364,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "17": {
      "timestamp": 1631730602.151861,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1168,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "18": {
      "timestamp": 1631743803.4545527,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 3583,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "19": {
      "timestamp": 1631763981.4230247,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 250,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "20": {
      "timestamp": 1631765029.4450378,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "21": {
      "timestamp": 1631767416.9511135,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 16,
      "iterations": 1,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    },
    "22": {
      "timestamp": 1631767965.2077034,
      "no_logs": false,
      "loss_names": [
        "total",
        "face_a",
        "face_b"
      ],
      "batchsize": 8,
      "iterations": 360,
      "config": {
        "learning_rate": 5e-05,
        "epsilon_exponent": -7,
        "allow_growth": false,
        "nan_protection": true,
        "convert_batchsize": 16,
        "eye_multiplier": 3,
        "mouth_multiplier": 2
      }
    }
  },
  "lowest_avg_loss": {
    "a": 0.035034529864788055,
    "b": 0.03087991289794445
  },
  "iterations": 50363,
  "config": {
    "centering": "face",
    "coverage": 68.75,
    "optimizer": "adam",
    "learning_rate": 5e-05,
    "epsilon_exponent": -7,
    "allow_growth": false,
    "mixed_precision": false,
    "nan_protection": true,
    "convert_batchsize": 16,
    "loss_function": "ssim",
    "mask_loss_function": "mse",
    "l2_reg_term": 100,
    "eye_multiplier": 3,
    "mouth_multiplier": 2,
    "penalized_mask_loss": true,
    "mask_type": "extended",
    "mask_blur_kernel": 3,
    "mask_threshold": 4,
    "learn_mask": false,
    "lowmem": false
  }
}

================= Configs ==================
--------- train.ini ---------

[global]
centering:                face
coverage:                 68.75
icnr_init:                False
conv_aware_init:          False
optimizer:                adam
learning_rate:            5e-05
epsilon_exponent:         -7
reflect_padding:          False
allow_growth:             False
mixed_precision:          False
nan_protection:           True
convert_batchsize:        16

[global.loss]
loss_function:            ssim
mask_loss_function:       mse
l2_reg_term:              100
eye_multiplier:           3
mouth_multiplier:         2
penalized_mask_loss:      True
mask_type:                extended
mask_blur_kernel:         3
mask_threshold:           4
learn_mask:               False

[trainer.original]
preview_images:           14
zoom_amount:              5
rotation_range:           10
shift_range:              5
flip_chance:              50
color_lightness:          30
color_ab:                 8
color_clahe_chance:       50
color_clahe_max_size:     4

[model.dfaker]
output_size:              128

[model.phaze_a]
output_size:              128
shared_fc:                none
enable_gblock:            True
split_fc:                 True
split_gblock:             False
split_decoders:           False
enc_architecture:         fs_original
enc_scaling:              40
enc_load_weights:         True
bottleneck_type:          dense
bottleneck_norm:          none
bottleneck_size:          1024
bottleneck_in_encoder:    True
fc_depth:                 1
fc_min_filters:           1024
fc_max_filters:           1024
fc_dimensions:            4
fc_filter_slope:          -0.5
fc_dropout:               0.0
fc_upsampler:             upsample2d
fc_upsamples:             1
fc_upsample_filters:      512
fc_gblock_depth:          3
fc_gblock_min_nodes:      512
fc_gblock_max_nodes:      512
fc_gblock_filter_slope:   -0.5
fc_gblock_dropout:        0.0
dec_upscale_method:       subpixel
dec_norm:                 none
dec_min_filters:          64
dec_max_filters:          512
dec_filter_slope:         -0.45
dec_res_blocks:           1
dec_output_kernel:        5
dec_gaussian:             True
dec_skip_last_residual:   True
freeze_layers:            keras_encoder
load_layers:              encoder
fs_original_depth:        4
fs_original_min_filters:  128
fs_original_max_filters:  1024
mobilenet_width:          1.0
mobilenet_depth:          1
mobilenet_dropout:        0.001

[model.realface]
input_size:               64
output_size:              128
dense_nodes:              1536
complexity_encoder:       128
complexity_decoder:       512

[model.villain]
lowmem:                   False

[model.dfl_h128]
lowmem:                   False

[model.dfl_sae]
input_size:               128
clipnorm:                 True
architecture:             df
autoencoder_dims:         0
encoder_dims:             42
decoder_dims:             21
multiscale_decoder:       False

[model.original]
lowmem:                   False

[model.unbalanced]
input_size:               128
lowmem:                   False
clipnorm:                 True
nodes:                    1024
complexity_encoder:       128
complexity_decoder_a:     384
complexity_decoder_b:     512

[model.dlight]
features:                 best
details:                  good
output_size:              256

--------- convert.ini ---------

[writer.ffmpeg]
container:                mp4
codec:                    libx264
crf:                      23
preset:                   medium
tune:                     none
profile:                  auto
level:                    auto
skip_mux:                 False

[writer.pillow]
format:                   png
draw_transparent:         False
optimize:                 False
gif_interlace:            True
jpg_quality:              75
png_compress_level:       3
tif_compression:          tiff_deflate

[writer.gif]
fps:                      25
loop:                     0
palettesize:              256
subrectangles:            False

[writer.opencv]
format:                   png
draw_transparent:         False
jpg_quality:              75
png_compress_level:       3

[scaling.sharpen]
method:                   none
amount:                   150
radius:                   0.3
threshold:                5.0

[mask.mask_blend]
type:                     normalized
kernel_size:              3
passes:                   4
threshold:                4
erosion:                  0.0

[mask.box_blend]
type:                     gaussian
distance:                 11.0
radius:                   5.0
passes:                   1

[color.match_hist]
threshold:                99.0

[color.manual_balance]
colorspace:               HSV
balance_1:                0.0
balance_2:                0.0
balance_3:                0.0
contrast:                 0.0
brightness:               0.0

[color.color_transfer]
clip:                     True
preserve_paper:           True

--------- extract.ini ---------

[global]
allow_growth:             False

[align.fan]
batch-size:               12

[mask.bisenet_fp]
batch-size:               8
include_ears:             False
include_hair:             False
include_glasses:          True

[mask.vgg_obstructed]
batch-size:               2

[mask.unet_dfl]
batch-size:               8

[mask.vgg_clear]
batch-size:               6

[detect.s3fd]
confidence:               70
batch-size:               4

[detect.mtcnn]
minsize:                  20
scalefactor:              0.709
batch-size:               8
threshold_1:              0.6
threshold_2:              0.7
threshold_3:              0.7

[detect.cv2_dnn]
confidence:               50

--------- .faceswap ---------
backend:                  nvidia
by aolvera27 » Thu Sep 16, 2021 8:46 pm

Sorry for double posting. I think I've solved it, and I guess I should post this in case someone has similar issues.

I checked my dataset again, and I noticed two files were modified after extraction. I was preparing a set for fit training with no-flip, and it seems I edited the extracts by mistake, instead of flipping a copy of the original sources. And then I included those edited extracted faces for the training session. I got rid of those files and now training is running smoothly again.

Go to full post
User avatar
aolvera27
Posts: 27
Joined: Thu May 27, 2021 3:53 am
Answers: 1
Has thanked: 5 times
Been thanked: 6 times

Re: Sudden error in the middle of training session

Post by aolvera27 »

Sorry for double posting. I think I've solved it, and I guess I should post this in case someone has similar issues.

I checked my dataset again, and I noticed two files were modified after extraction. I was preparing a set for fit training with no-flip, and it seems I edited the extracts by mistake, instead of flipping a copy of the original sources. And then I included those edited extracted faces for the training session. I got rid of those files and now training is running smoothly again.

Locked