Sudden error in the middle of training session
Posted: Thu Sep 16, 2021 7:01 am
(I'm reposting here 'cause I just noticed I had posted this on the wrong forum) I'm getting an error in the middle of training. Everything seems to be running just fine, then about a couple of hundreds of iterations into the session it gives back an "ERROR Caught exception in thread: '_training_0'", and a "ERROR Got Exception on main handler..."
I checked to see if I added a wrong (non-extracted) file by mistake, but I couldn't find any. I just added some thousand new faces to set B after 50k iterations, if that information is of any utility.
Code: Select all
09/16/2021 00:06:43 MainProcess _training_0 _base _save DEBUG Backing up and saving models
09/16/2021 00:06:43 MainProcess _training_0 _base _get_save_averages DEBUG Getting save averages
09/16/2021 00:06:43 MainProcess _training_0 _base _get_save_averages DEBUG Average losses since last save: [0.038943224345284584, 0.03432052828952629]
09/16/2021 00:06:43 MainProcess _training_0 _base _should_backup DEBUG Should backup: False
09/16/2021 00:06:50 MainProcess _training_0 _base save DEBUG Saving State
09/16/2021 00:06:50 MainProcess _training_0 serializer save DEBUG filename: /content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json, data type: <class 'dict'>
09/16/2021 00:06:50 MainProcess _training_0 serializer _check_extension DEBUG Original filename: '/content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json', final filename: '/content/drive/My Drive/colab_files/faceswap/models/JdKQry/villain_state.json'
09/16/2021 00:06:50 MainProcess _training_0 serializer marshal DEBUG data type: <class 'dict'>
09/16/2021 00:06:50 MainProcess _training_0 serializer marshal DEBUG returned data type: <class 'bytes'>
09/16/2021 00:06:50 MainProcess _training_0 _base save DEBUG Saved State
09/16/2021 00:06:50 MainProcess _training_0 _base _save INFO [Saved models] - Average loss since last save: face_a: 0.03894, face_b: 0.03432
09/16/2021 00:06:50 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada1.mp4_06910_0.png', 'Screen Shot 2021-08-23 at 11.05.16 PM_0.png', 'Screen Shot 2021-08-24 at 10.33.40 PM_0.png', 'jada1.mp4_03167_0.png', 'Screen Shot 2021-08-12 at 12.04.44 PM_0.png', 'Screen Shot 2021-08-19 at 1.36.59 AM_0.png', 'Screen Shot 2021-08-18 at 11.49.01 PM_0.png', 'jada3_000469_0.png']
09/16/2021 00:06:50 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (1225).png', 'step2_3 (1803).png', 'step2_3 (2060).png', 'step2_3 (2735).png', 'step2_3 (1347).png', 'step2_3 (3434).png', 'step2_3 (1470).png', 'step2_3 (1549).png']
09/16/2021 00:06:55 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada2.mp4_15552_0.png', 'jada1.mp4_15219_0.png', 'jada2.mp4_07780_0.png', 'Screen Shot 2021-08-24 at 10.27.07 PM_0.png', 'jada1.mp4_16136_0.png', 'Screen Shot 2021-08-12 at 12.05.18 PM_0.png', 'jada1.mp4_01568_0.png', 'Screen Shot 2021-08-12 at 10.23.16 AM_0.png']
09/16/2021 00:06:55 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (1460).png', 'step2_3 (883).png', 'step2_3 (1352).png', 'step2_3 (1421).png', 'step2_3 (2242).png', 'step2_3 (1658).png', 'step2_3 (3397).png', 'step2_3 (2849).png']
09/16/2021 00:06:59 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-19 at 12.16.44 AM_0.png', 'jada1.mp4_01451_0.png', 'Screen Shot 2021-08-24 at 10.45.36 PM_0.png', 'jada2.mp4_06824_0.png', 'Screen Shot 2021-08-11 at 7.46.31 PM_0.png', 'Screen Shot 2021-08-12 at 7.14.13 AM_0.png', 'jada2.mp4_09537_0.png', 'jada1.mp4_08598_0.png']
09/16/2021 00:06:59 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (75).png', 'step2_3 (1050).png', 'step2_3 (1723).png', 'step2_3 (1448).png', 'step2_3 (1408).png', 'step2_3 (64).png', 'step2_3 (348).png', 'step2_3 (250).png']
09/16/2021 00:07:04 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-11 at 7.53.08 PM_0.png', 'Screen Shot 2021-08-12 at 11.50.46 AM_0.png', 'Screen Shot 2021-08-12 at 10.34.06 AM_0.png', 'Screen Shot 2021-08-20 at 8.02.02 AM_0.png', 'Screen Shot 2021-08-12 at 12.08.27 PM_0.png', 'Screen Shot 2021-08-19 at 12.58.37 AM_0.png', 'jada3_004843_0.png', 'jada3_002485_0.png']
09/16/2021 00:07:04 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (92).png', 'step2_3 (332).png', 'step2_3 (362).png', 'step2_3 (3225).png', 'step2_3 (2278).png', 'step2_3 (2186).png', 'step2_3 (3201).png', 'step2_3 (2323).png']
09/16/2021 00:07:08 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada1.mp4_18369_0.png', 'Screen Shot 2021-08-12 at 10.25.37 AM_0.png', 'jada2.mp4_06553_0.png', 'jada2.mp4_07783_0.png', 'jada1.mp4_07405_0.png', 'Screen Shot 2021-08-19 at 1.23.56 AM_0.png', 'Screen Shot 2021-08-18 at 11.53.13 PM_0.png', 'jada1.mp4_08555_0.png']
09/16/2021 00:07:08 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (2340).png', 'step2_3 (1645).png', 'step2_3 (3429).png', 'step2_3 (333).png', 'step2_3 (227).png', 'step2_3 (2225).png', 'step2_3 (1299).png', 'step2_3 (620).png']
09/16/2021 00:07:12 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada1.mp4_03337_0.png', 'Screen Shot 2021-08-24 at 10.41.58 PM_0.png', 'Screen Shot 2021-08-24 at 10.54.41 PM_0.png', 'Screen Shot 2021-08-20 at 7.54.14 AM_0.png', 'Screen Shot 2021-08-12 at 1.45.06 PM_0.png', 'Screen Shot 2021-08-12 at 11.51.13 AM_0.png', 'Screen Shot 2021-08-18 at 11.52.53 PM_0.png', 'jada1.mp4_03668_0.png']
09/16/2021 00:07:12 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (874).png', 'step2_3 (2531).png', 'step2_3 (60).png', 'step2_3 (1106).png', 'step2_3 (908).png', 'step2_3 (2337).png', 'step2_3 (162).png', 'step2_3 (2063).png']
09/16/2021 00:07:17 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-23 at 11.15.27 PM_0.png', 'Screen Shot 2021-08-19 at 12.00.28 AM_0.png', 'Screen Shot 2021-08-12 at 7.41.49 AM_0.png', 'jada1.mp4_14930_0.png', 'jada3_001981_0.png', 'Screen Shot 2021-08-12 at 11.49.24 AM_0.png', 'Screen Shot 2021-08-18 at 11.54.08 PM_0.png', 'jada2.mp4_15606_0.png']
09/16/2021 00:07:17 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (345).png', 'step2_3 (584).png', 'step2_3 (2216).png', 'step2_3 (175).png', 'step2_3 (1220).png', 'step2_3 (1505).png', 'step2_3 (2872).png', 'step2_3 (2094).png']
09/16/2021 00:07:21 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada2.mp4_08545_0.png', 'Screen Shot 2021-08-24 at 10.04.04 PM_0.png', 'Screen Shot 2021-08-19 at 12.45.38 AM_0.png', 'jada1.mp4_01675_0.png', 'Screen Shot 2021-08-24 at 10.52.26 PM_0.png', 'Screen Shot 2021-08-19 at 1.05.40 AM_0.png', 'Screen Shot 2021-08-23 at 11.00.30 PM_0.png', 'jada1.mp4_11315_0.png']
09/16/2021 00:07:21 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (3295).png', 'step2_3 (1326).png', 'step2_3 (3327).png', 'step2_3 (2726).png', 'step2_3 (937).png', 'step2_3 (2329).png', 'step2_3 (13).png', 'step2_3 (1824).png']
09/16/2021 00:07:26 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada3_003421_0.png', 'Screen Shot 2021-08-12 at 11.54.24 AM_0.png', 'Screen Shot 2021-08-23 at 11.29.24 PM_0.png', 'jada1.mp4_15745_0.png', 'jada1.mp4_12057_0.png', 'jada1.mp4_14893_0.png', 'Screen Shot 2021-08-19 at 1.22.23 AM_0.png', 'jada1.mp4_10019_0.png']
09/16/2021 00:07:26 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (3212).png', 'step2_3 (2620).png', 'step2_3 (1091).png', 'step2_3 (25).png', 'step2_3 (1805).png', 'side (10).png', 'step2_3 (2280).png', 'step2_3 (1619).png']
09/16/2021 00:07:30 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada1.mp4_04349_0.png', 'Screen Shot 2021-08-12 at 11.42.19 AM_0.png', 'jada1.mp4_11897_0.png', 'Screen Shot 2021-08-12 at 11.59.15 AM_0.png', 'jada2.mp4_14424_0.png', 'jada2.mp4_10104_0.png', 'jada1.mp4_00573_0.png', 'Screen Shot 2021-08-24 at 10.06.20 PM_0.png']
09/16/2021 00:07:30 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (1743).png', 'step2_3 (2157).png', 'step2_3 (3093).png', 'step2_3 (2741).png', 'step2_3 (2701).png', 'step2_3 (3394).png', 'step2_3 (2485).png', 'step2_3 (310).png']
09/16/2021 00:07:34 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-12 at 7.04.34 AM_0.png', 'Screen Shot 2021-08-19 at 12.41.04 AM_0.png', 'jada2.mp4_15342_0.png', 'jada2.mp4_13956_0.png', 'Screen Shot 2021-08-20 at 8.00.13 AM_0.png', 'jada2.mp4_06836_0.png', 'Screen Shot 2021-08-24 at 10.38.52 PM_0.png', 'Screen Shot 2021-08-23 at 11.27.10 PM_0.png']
09/16/2021 00:07:34 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (2859).png', 'step2_3 (1856).png', 'step2_3 (1309).png', 'step2_3 (335).png', 'step2_3 (1280).png', 'step2_3 (454).png', 'step2_3 (270).png', 'step2_3 (1232).png']
09/16/2021 00:07:39 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-24 at 10.34.53 PM_0.png', 'Screen Shot 2021-08-12 at 11.50.07 AM_0.png', 'jada1.mp4_00657_0.png', 'Screen Shot 2021-08-23 at 11.36.24 PM_0.png', 'jada1.mp4_06883_0.png', 'Screen Shot 2021-08-19 at 1.19.23 AM_0.png', 'Screen Shot 2021-08-23 at 11.12.10 PM_0.png', 'jada2.mp4_06345_0.png']
09/16/2021 00:07:39 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (1291).png', 'step2_3 (1518).png', 'step2_3 (58).png', 'step2_3 (1273).png', 'step2_3 (1834).png', 'step2_3 (2607).png', 'step2_3 (1001).png', 'step2_3 (2180).png']
09/16/2021 00:07:43 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-24 at 10.24.47 PM_0.png', 'jada1.mp4_04230_0.png', 'jada1.mp4_03528_0.png', 'jada2.mp4_15864_0.png', 'jada1.mp4_11714_0.png', 'jada3_007705_0.png', 'jada1.mp4_18059_0.png', 'jada2.mp4_10073_0.png']
09/16/2021 00:07:43 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (2716).png', 'step2_3 (477).png', 'step2_3 (3210).png', 'step2_3 (1338).png', 'step2_3 (2431).png', 'step2_3 (2493).png', 'step2_3 (3418).png', 'step2_3 (153).png']
09/16/2021 00:07:48 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-23 at 11.19.50 PM_0.png', 'jada1.mp4_11685_0.png', 'jada1.mp4_00054_0.png', 'jada2.mp4_08770_0.png', 'jada2.mp4_08667_0.png', 'jada2.mp4_09669_0.png', 'jada1.mp4_12886_0.png', 'jada1.mp4_16431_0.png']
09/16/2021 00:07:48 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (690).png', 'step2_3 (2532).png', 'step2_3 (3027).png', 'step2_3 (783).png', 'step2_3 (940).png', 'step2_3 (1040).png', 'step2_3 (956).png', 'step2_3 (1126).png']
09/16/2021 00:07:52 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-24 at 10.44.22 PM_0.png', 'jada2.mp4_14850_0.png', 'jada1.mp4_01485_0.png', 'jada1.mp4_11736_0.png', 'Screen Shot 2021-08-20 at 7.59.06 AM_0.png', 'Screen Shot 2021-08-24 at 10.26.50 PM_0.png', 'jada2.mp4_14406_0.png', 'Screen Shot 2021-08-19 at 12.24.02 AM_0.png']
09/16/2021 00:07:52 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (3111).png', 'step2_3 (944).png', 'step2_3 (2262).png', 'step2_3 (2430).png', 'step2_3 (131).png', 'step2_3 (872).png', 'step2_3 (2847).png', 'step2_3 (3080).png']
09/16/2021 00:07:56 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada1.mp4_00215_0.png', 'jada3_003277_0.png', 'Screen Shot 2021-08-24 at 10.33.49 PM_0.png', 'jada1.mp4_18084_0.png', 'Screen Shot 2021-08-19 at 1.10.20 AM_0.png', 'jada3_005077_0.png', 'Screen Shot 2021-08-24 at 10.57.17 PM_0.png', 'Screen Shot 2021-08-19 at 12.27.14 AM_0.png']
09/16/2021 00:07:56 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (1544).png', 'step2_3 (2610).png', 'step2_3 (3363).png', 'step2_3 (192).png', 'step2_3 (2760).png', 'step2_3 (1441).png', 'step2_3 (1708).png', 'step2_3 (2852).png']
09/16/2021 00:08:01 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada2.mp4_06477_0.png', 'jada1.mp4_06899_0.png', 'jada1.mp4_00707_0.png', 'Screen Shot 2021-08-12 at 10.22.03 AM_0.png', 'jada1.mp4_03650_0.png', 'Screen Shot 2021-08-12 at 11.50.18 AM_0.png', 'Screen Shot 2021-08-12 at 1.40.41 PM_0.png', 'jada1.mp4_04407_0.png']
09/16/2021 00:08:01 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (2161).png', 'step2_3 (508).png', 'step2_3 (2817).png', 'step2_3 (878).png', 'step2_3 (2971).png', 'step2_3 (3362).png', 'step2_3 (54).png', 'step2_3 (3043).png']
09/16/2021 00:08:05 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-19 at 1.22.55 AM_0.png', 'Screen Shot 2021-08-19 at 12.20.01 AM_0.png', 'Screen Shot 2021-08-24 at 10.05.31 PM_0.png', 'jada2.mp4_12876_1.png', 'jada1.mp4_11856_0.png', 'jada2.mp4_09680_0.png', 'jada3_003799_0.png', 'jada1.mp4_04479_0.png']
09/16/2021 00:08:05 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (463).png', 'step2_3 (2174).png', 'step2_3 (1583).png', 'step2_3 (2386).png', 'step2_3 (2743).png', 'step2_3 (2331).png', 'step2_3 (114).png', 'step2_3 (340).png']
09/16/2021 00:08:09 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-19 at 1.15.05 AM_0.png', 'Screen Shot 2021-08-23 at 11.05.53 PM_0.png', 'jada2.mp4_08989_0.png', 'Screen Shot 2021-08-12 at 11.42.04 AM_0.png', 'Screen Shot 2021-08-19 at 12.53.56 AM_0.png', 'jada3_000361_0.png', 'jada1.mp4_00056_0.png', 'Screen Shot 2021-08-24 at 10.35.34 PM_0.png']
09/16/2021 00:08:09 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (1064).png', 'step2_3 (2082).png', 'step2_3 (1572).png', 'step2_3 (1084).png', 'step2_3 (1274).png', 'step2_3 (1458).png', 'step2_3 (2513).png', 'step2_3 (3182).png']
09/16/2021 00:08:14 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada2.mp4_06317_0.png', 'Screen Shot 2021-08-19 at 12.36.23 AM_0.png', 'Screen Shot 2021-08-24 at 10.51.49 PM_0.png', 'Screen Shot 2021-08-19 at 1.20.56 AM_0.png', 'jada2.mp4_08822_0.png', 'jada2.mp4_09604_0.png', 'Screen Shot 2021-08-19 at 12.42.16 AM_0.png', 'jada1.mp4_03711_0.png']
09/16/2021 00:08:14 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (1485).png', 'step2_3 (2026).png', 'step2_3 (2188).png', 'step2_3 (1283).png', 'step2_3 (3148).png', 'step2_3 (2846).png', 'step2_3 (211).png', 'step2_3 (1111).png']
09/16/2021 00:08:18 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada2.mp4_08792_0.png', 'Screen Shot 2021-08-11 at 7.47.20 PM_0.png', 'jada2.mp4_14910_0.png', 'Screen Shot 2021-08-12 at 12.02.32 PM_0.png', 'jada1.mp4_14544_0.png', 'Screen Shot 2021-08-24 at 10.25.49 PM_0.png', 'jada1.mp4_18185_0.png', 'Screen Shot 2021-08-12 at 11.54.56 AM_0.png']
09/16/2021 00:08:18 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (3195).png', 'step2_3 (1577).png', 'step2_3 (1595).png', 'step2_3 (1207).png', 'step2_3 (1415).png', 'step2_3 (207).png', 'step2_3 (2466).png', 'step2_3 (236).png']
09/16/2021 00:08:23 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada1.mp4_03560_0.png', 'jada2.mp4_12732_0.png', 'jada2.mp4_15882_0.png', 'jada2.mp4_10002_0.png', 'Screen Shot 2021-08-23 at 10.54.51 PM_0.png', 'jada1.mp4_18349_0.png', 'jada3_000181_0.png', 'jada2.mp4_15900_0.png']
09/16/2021 00:08:23 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (3270).png', 'step2_3 (3085).png', 'step2_3 (1687).png', 'step2_3 (1857).png', 'step2_3 (1115).png', 'step2_3 (796).png', 'step2_3 (1488).png', 'step2_3 (2525).png']
09/16/2021 00:08:27 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada3_004375_0.png', 'jada3_001819_0.png', 'Screen Shot 2021-08-19 at 12.51.40 AM_0.png', 'jada1.mp4_00964_0.png', 'jada3_002683_0.png', 'Screen Shot 2021-08-23 at 10.57.42 PM_0.png', 'jada1.mp4_14146_0.png', 'Screen Shot 2021-08-18 at 11.53.40 PM_0.png']
09/16/2021 00:08:27 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (3236).png', 'step2_3 (1713).png', 'step2_3 (2914).png', 'step2_3 (2458).png', 'step2_3 (2765).png', 'step2_3 (1372).png', 'step2_3 (2758).png', 'step2_3 (500).png']
09/16/2021 00:08:32 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-12 at 1.43.01 PM_0.png', 'jada2.mp4_14874_0.png', 'jada1.mp4_10291_0.png', 'jada2.mp4_10717_0.png', 'Screen Shot 2021-08-19 at 12.13.14 AM_0.png', 'jada1.mp4_04193_0.png', 'Screen Shot 2021-08-19 at 1.00.05 AM_0.png', 'jada2.mp4_09611_0.png']
09/16/2021 00:08:32 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (41).png', 'step2_3 (401).png', 'step2_3 (2099).png', 'step2_3 (1248).png', 'step2_3 (925).png', 'step2_3 (3380).png', 'step2_3 (2958).png', 'step2_3 (2043).png']
09/16/2021 00:08:36 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-19 at 12.57.18 AM_0.png', 'Screen Shot 2021-08-20 at 7.58.42 AM_0.png', 'jada1.mp4_12212_0.png', 'Screen Shot 2021-08-24 at 10.28.20 PM_0.png', 'jada2.mp4_06952_0.png', 'jada2.mp4_06173_0.png', 'jada1.mp4_09895_0.png', 'jada1.mp4_15077_0.png']
09/16/2021 00:08:36 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (2642).png', 'step2_3 (2204).png', 'step2_3 (1481).png', 'step2_3 (2169).png', 'step2_3 (2771).png', 'step2_3 (2056).png', 'step2_3 (2700).png', 'step2_3 (1189).png']
09/16/2021 00:08:40 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-24 at 10.31.04 PM_0.png', 'Screen Shot 2021-08-19 at 12.57.51 AM_0.png', 'Screen Shot 2021-08-19 at 12.38.22 AM_0.png', 'Screen Shot 2021-08-23 at 10.53.07 PM_0.png', 'Screen Shot 2021-08-12 at 7.53.39 AM_0.png', 'jada2.mp4_14922_0.png', 'jada1.mp4_12000_0.png', 'Screen Shot 2021-08-19 at 12.51.55 AM_0.png']
09/16/2021 00:08:40 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (221).png', 'step2_3 (1277).png', 'step2_3 (1063).png', 'step2_3 (1061).png', 'step2_3 (731).png', 'step2_3 (1227).png', 'step2_3 (581).png', 'step2_3 (182).png']
09/16/2021 00:08:45 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-12 at 11.43.31 AM_0.png', 'Screen Shot 2021-08-19 at 12.46.09 AM_0.png', 'jada2.mp4_10055_0.png', 'jada1.mp4_03661_0.png', 'Screen Shot 2021-08-12 at 10.40.11 AM_0.png', 'jada2.mp4_15378_0.png', 'Screen Shot 2021-08-24 at 10.41.53 PM_0.png', 'Screen Shot 2021-08-23 at 11.20.48 PM_0.png']
09/16/2021 00:08:45 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (2745).png', 'step2_3 (1801).png', 'step2_3 (6).png', 'step2_3 (1862).png', 'step2_3 (1788).png', 'step2_3 (1166).png', 'step2_3 (1095).png', 'step2_3 (3297).png']
09/16/2021 00:08:49 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-24 at 10.40.14 PM_0.png', 'Screen Shot 2021-08-23 at 11.33.37 PM_0.png', 'Screen Shot 2021-08-23 at 11.18.46 PM_0.png', 'Screen Shot 2021-08-23 at 11.01.15 PM_0.png', 'jada2.mp4_07487_0.png', 'jada1.mp4_04486_0.png', 'Screen Shot 2021-08-23 at 11.29.35 PM_0.png', 'Screen Shot 2021-08-19 at 1.24.44 AM_0.png']
09/16/2021 00:08:49 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (3088).png', 'step2_3 (1833).png', 'step2_3 (3328).png', 'step2_3 (2536).png', 'step2_3 (626).png', 'step2_3 (2646).png', 'step2_3 (873).png', 'step2_3 (2366).png']
09/16/2021 00:08:53 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada2.mp4_10301_0.png', 'jada1.mp4_10085_0.png', 'Screen Shot 2021-08-12 at 11.46.29 AM_0.png', 'jada1.mp4_01157_0.png', 'Screen Shot 2021-08-12 at 10.30.44 AM_0.png', 'Screen Shot 2021-08-19 at 12.23.30 AM_0.png', 'jada2.mp4_12516_1.png', 'jada2.mp4_09969_0.png']
09/16/2021 00:08:53 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (216).png', 'step2_3 (474).png', 'step2_3 (1478).png', 'step2_3 (2753).png', 'step2_3 (1183).png', 'step2_3 (1885).png', 'step2_3 (2938).png', 'step2_3 (3315).png']
09/16/2021 00:08:58 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada1.mp4_08395_0.png', 'Screen Shot 2021-08-12 at 11.49.10 AM_0.png', 'Screen Shot 2021-08-12 at 11.46.09 AM_0.png', 'jada3_004249_0.png', 'jada2.mp4_08393_0.png', 'Screen Shot 2021-08-12 at 11.45.46 AM_0.png', 'Screen Shot 2021-08-12 at 10.43.39 AM_0.png', 'Screen Shot 2021-08-23 at 11.06.10 PM_0.png']
09/16/2021 00:08:58 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (3202).png', 'step2_3 (1081).png', 'step2_3 (961).png', 'step2_3 (3389).png', 'step2_3 (3431).png', 'step2_3 (2162).png', 'step2_3 (1942).png', 'step2_3 (2901).png']
09/16/2021 00:09:02 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-19 at 12.37.58 AM_0.png', 'Screen Shot 2021-08-12 at 1.41.15 PM_0.png', 'jada2.mp4_09361_0.png', 'jada2.mp4_15450_0.png', 'Screen Shot 2021-08-19 at 1.37.14 AM_0.png', 'Screen Shot 2021-08-19 at 1.21.50 AM_0.png', 'Screen Shot 2021-08-19 at 1.06.03 AM_0.png', 'jada2.mp4_14490_0.png']
09/16/2021 00:09:02 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (1447).png', 'step2_3 (1286).png', 'step2_3 (2949).png', 'step2_3 (322).png', 'step2_3 (1043).png', 'step2_3 (820).png', 'step2_3 (1682).png', 'step2_3 (765).png']
09/16/2021 00:09:07 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-12 at 7.29.48 AM_0.png', 'jada1.mp4_14015_0.png', 'Screen Shot 2021-08-12 at 10.42.43 AM_0.png', 'jada1.mp4_00909_0.png', 'Screen Shot 2021-08-12 at 12.03.42 PM_0.png', 'Screen Shot 2021-08-19 at 1.38.06 AM_0.png', 'jada1.mp4_03384_0.png', 'Screen Shot 2021-08-19 at 1.22.40 AM_0.png']
09/16/2021 00:09:07 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (1995).png', 'step2_3 (379).png', 'step2_3 (1499).png', 'step2_3 (1006).png', 'step2_3 (2794).png', 'step2_3 (2230).png', 'side (1).png', 'step2_3 (1503).png']
09/16/2021 00:09:11 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-12 at 7.06.51 AM_0.png', 'jada1.mp4_12497_0.png', 'Screen Shot 2021-08-23 at 11.06.53 PM_0.png', 'jada1.mp4_17135_0.png', 'Screen Shot 2021-08-24 at 10.34.37 PM_0.png', 'Screen Shot 2021-08-12 at 7.18.31 AM_0.png', 'Screen Shot 2021-08-12 at 11.36.25 AM_0.png', 'Screen Shot 2021-08-12 at 12.07.34 PM_0.png']
09/16/2021 00:09:11 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (304).png', 'step2_3 (1129).png', 'step2_3 (1265).png', 'step2_3 (1491).png', 'step2_3 (862).png', 'step2_3 (2059).png', 'step2_3 (1853).png', 'step2_3 (573).png']
09/16/2021 00:09:15 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada2.mp4_10599_1.png', 'Screen Shot 2021-08-19 at 12.11.44 AM_0.png', 'jada1.mp4_03853_0.png', 'Screen Shot 2021-08-12 at 10.34.53 AM_0.png', 'Screen Shot 2021-08-12 at 7.30.10 AM_0.png', 'jada1.mp4_01392_0.png', 'jada1.mp4_03749_0.png', 'Screen Shot 2021-08-11 at 7.42.39 PM_0.png']
09/16/2021 00:09:15 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (2305).png', 'step2_3 (1442).png', 'step2_3 (2739).png', 'step2_3 (1871).png', 'step2_3 (224).png', 'step2_3 (1948).png', 'step2_3 (2652).png', 'step2_3 (2259).png']
09/16/2021 00:09:20 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-12 at 12.11.49 PM_0.png', 'jada3_003223_0.png', 'Screen Shot 2021-08-12 at 11.50.24 AM_0.png', 'Screen Shot 2021-08-24 at 10.41.48 PM_0.png', 'jada1.mp4_08129_0.png', 'Screen Shot 2021-08-12 at 11.57.37 AM_0.png', 'Screen Shot 2021-08-23 at 11.09.38 PM_0.png', 'jada1.mp4_05908_0.png']
09/16/2021 00:09:20 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (356).png', 'step2_3 (2020).png', 'step2_3 (397).png', 'step2_3 (2882).png', 'step2_3 (2130).png', 'step2_3 (2718).png', 'step2_3 (2930).png', 'step2_3 (2413).png']
09/16/2021 00:09:24 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada1.mp4_04526_0.png', 'jada1.mp4_04093_0.png', 'jada2.mp4_14418_0.png', 'Screen Shot 2021-08-18 at 11.43.56 PM_0.png', 'jada2.mp4_13506_0.png', 'jada1.mp4_04539_0.png', 'jada1.mp4_06044_0.png', 'Screen Shot 2021-08-24 at 10.11.59 PM_0.png']
09/16/2021 00:09:24 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (3063).png', 'step2_3 (2275).png', 'step2_3 (1241).png', 'step2_3 (2856).png', 'step2_3 (1046).png', 'step2_3 (953).png', 'step2_3 (1114).png', 'step2_3 (506).png']
09/16/2021 00:09:29 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-20 at 7.59.43 AM_0.png', 'Screen Shot 2021-08-20 at 8.01.32 AM_0.png', 'Screen Shot 2021-08-24 at 10.05.22 PM_0.png', 'jada1.mp4_15942_0.png', 'jada3_003205_0.png', 'Screen Shot 2021-08-19 at 1.08.50 AM_0.png', 'jada1.mp4_00920_0.png', 'jada3_005959_0.png']
09/16/2021 00:09:29 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (2025).png', 'step2_3 (1173).png', 'step2_3 (580).png', 'step2_3 (2603).png', 'step2_3 (1208).png', 'step2_3 (2221).png', 'step2_3 (3072).png', 'step2_3 (1184).png']
09/16/2021 00:09:33 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada1.mp4_05357_0.png', 'jada1.mp4_00435_0.png', 'jada2.mp4_15678_0.png', 'Screen Shot 2021-08-19 at 12.59.13 AM_0.png', 'Screen Shot 2021-08-12 at 11.35.15 AM_0.png', 'Screen Shot 2021-08-24 at 10.31.42 PM_0.png', 'jada2.mp4_13848_0.png', 'jada1.mp4_04008_0.png']
09/16/2021 00:09:33 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['step2_3 (1733).png', 'step2_3 (3033).png', 'step2_3 (1244).png', 'step2_3 (2731).png', 'step2_3 (2024).png', 'step2_3 (244).png', 'step2_3 (2860).png', 'step2_3 (3303).png']
09/16/2021 00:09:37 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['jada2.mp4_14820_0.png', 'jada2.mp4_13290_0.png', 'Screen Shot 2021-08-19 at 1.15.34 AM_0.png', 'Screen Shot 2021-08-20 at 7.56.47 AM_0.png', 'jada2.mp4_09096_0.png', 'Screen Shot 2021-08-12 at 12.06.05 PM_0.png', 'Screen Shot 2021-08-24 at 10.47.11 PM_0.png', 'jada1.mp4_07394_0.png']
09/16/2021 00:09:40 MainProcess _run_0 multithreading run DEBUG Error in thread (_run_0): 'NoneType' object is not subscriptable
09/16/2021 00:09:42 MainProcess _run_1 generator cache_metadata DEBUG All metadata already cached for: ['Screen Shot 2021-08-23 at 10.58.49 PM_0.png', 'Screen Shot 2021-08-24 at 10.52.59 PM_0.png', 'jada1.mp4_00912_0.png', 'jada2.mp4_07431_0.png', 'Screen Shot 2021-08-23 at 10.55.51 PM_0.png', 'Screen Shot 2021-08-19 at 1.18.48 AM_0.png', 'jada3_004357_0.png', 'jada1.mp4_00292_0.png']
09/16/2021 00:09:42 MainProcess _training_0 multithreading check_and_raise_error DEBUG Thread error caught: [(<class 'TypeError'>, TypeError("'NoneType' object is not subscriptable"), <traceback object at 0x7f3830413370>)]
09/16/2021 00:09:42 MainProcess _training_0 multithreading run DEBUG Error in thread (_training_0): 'NoneType' object is not subscriptable
09/16/2021 00:09:42 MainProcess _run_1 multithreading run DEBUG Error in thread (_run_1): 'NoneType' object is not subscriptable
09/16/2021 00:09:42 MainProcess MainThread train _monitor DEBUG Thread error detected
09/16/2021 00:09:42 MainProcess MainThread train _monitor DEBUG Closed Monitor
09/16/2021 00:09:42 MainProcess MainThread train _end_thread DEBUG Ending Training thread
09/16/2021 00:09:42 MainProcess MainThread train _end_thread CRITICAL Error caught! Exiting...
09/16/2021 00:09:42 MainProcess MainThread multithreading join DEBUG Joining Threads: '_training'
09/16/2021 00:09:42 MainProcess MainThread multithreading join DEBUG Joining Thread: '_training_0'
09/16/2021 00:09:42 MainProcess MainThread multithreading join ERROR Caught exception in thread: '_training_0'
Traceback (most recent call last):
File "/content/faceswap/lib/cli/launcher.py", line 182, in execute_script
process.process()
File "/content/faceswap/scripts/train.py", line 190, in process
self._end_thread(thread, err)
File "/content/faceswap/scripts/train.py", line 230, in _end_thread
thread.join()
File "/content/faceswap/lib/multithreading.py", line 121, in join
raise thread.err[1].with_traceback(thread.err[2])
File "/content/faceswap/lib/multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "/content/faceswap/scripts/train.py", line 252, in _training
raise err
File "/content/faceswap/scripts/train.py", line 242, in _training
self._run_training_cycle(model, trainer)
File "/content/faceswap/scripts/train.py", line 327, in _run_training_cycle
trainer.train_one_step(viewer, timelapse)
File "/content/faceswap/plugins/train/trainer/_base.py", line 191, in train_one_step
model_inputs, model_targets = self._feeder.get_batch()
File "/content/faceswap/plugins/train/trainer/_base.py", line 416, in get_batch
batch = next(self._feeds[side])
File "/content/faceswap/lib/multithreading.py", line 156, in iterator
self.check_and_raise_error()
File "/content/faceswap/lib/multithreading.py", line 84, in check_and_raise_error
raise error[1].with_traceback(error[2])
File "/content/faceswap/lib/multithreading.py", line 37, in run
self._target(*self._args, **self._kwargs)
File "/content/faceswap/lib/multithreading.py", line 145, in _run
for item in self.generator(*self._gen_args, **self._gen_kwargs):
File "/content/faceswap/lib/training/generator.py", line 598, in _minibatch
yield self._process_batch(img_paths, side)
File "/content/faceswap/lib/training/generator.py", line 621, in _process_batch
batch = self._face_cache.cache_metadata(filenames)
File "/content/faceswap/lib/training/generator.py", line 224, in cache_metadata
self._validate_version(meta, filename)
File "/content/faceswap/lib/training/generator.py", line 287, in _validate_version
alignment_version = png_meta["source"]["alignments_version"]
TypeError: 'NoneType' object is not subscriptable
============ System Information ============
encoding: UTF-8
git_branch: Not Found
git_commits: Not Found
gpu_cuda: 11.0
gpu_cudnn: No global version found
gpu_devices: GPU_0: Tesla K80
gpu_devices_active: GPU_0
gpu_driver: 460.32.03
gpu_vram: GPU_0: 11441MB
os_machine: x86_64
os_platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
os_release: 5.4.104+
py_command: faceswap/faceswap.py train -A face_a -B face_b -m /content/drive/My Drive/colab_files/faceswap/models/JdKQry -t villain -bs 8 -it 100000 -s 360 -ss 25000 -tia face_a -tib face_b -to /content/drive/My Drive/colab_files/faceswap/output/timelapse
py_conda_version: N/A
py_implementation: CPython
py_version: 3.7.11
py_virtual_env: False
sys_cores: 2
sys_processor: x86_64
sys_ram: Total: 12991MB, Available: 10595MB, Used: 3980MB, Free: 168MB
=============== Pip Packages ===============
absl-py==0.12.0
alabaster==0.7.12
albumentations==0.1.12
altair==4.1.0
appdirs==1.4.4
argcomplete==1.12.3
argon2-cffi==21.1.0
arviz==0.11.2
astor==0.8.1
astropy==4.3.1
astunparse==1.6.3
atari-py==0.2.9
atomicwrites==1.4.0
attrs==21.2.0
audioread==2.1.9
autograd==1.3
Babel==2.9.1
backcall==0.2.0
beautifulsoup4==4.6.3
bleach==4.0.0
blis==0.4.1
bokeh==2.3.3
Bottleneck==1.3.2
branca==0.4.2
bs4==0.0.1
CacheControl==0.12.6
cached-property==1.5.2
cachetools==4.2.2
catalogue==1.0.0
certifi==2021.5.30
cffi==1.14.6
cftime==1.5.0
chardet==3.0.4
charset-normalizer==2.0.4
clang==5.0
click==7.1.2
cloudpickle==1.3.0
cmake==3.12.0
cmdstanpy==0.9.5
colorcet==2.0.6
colorlover==0.3.0
community==1.0.0b1
contextlib2==0.5.5
convertdate==2.3.2
coverage==3.7.1
coveralls==0.5
crcmod==1.7
cufflinks==0.17.3
cupy-cuda101==9.1.0
cvxopt==1.2.6
cvxpy==1.0.31
cycler==0.10.0
cymem==2.0.5
Cython==0.29.24
daft==0.0.4
dask==2.12.0
datascience==0.10.6
debugpy==1.0.0
decorator==4.4.2
defusedxml==0.7.1
descartes==1.1.0
dill==0.3.4
distributed==1.25.3
dlib @ file:///dlib-19.18.0-cp37-cp37m-linux_x86_64.whl
dm-tree==0.1.6
docopt==0.6.2
docutils==0.17.1
dopamine-rl==1.0.5
earthengine-api==0.1.278
easydict==1.9
ecos==2.0.7.post1
editdistance==0.5.3
en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.5/en_core_web_sm-2.2.5.tar.gz
entrypoints==0.3
ephem==4.0.0.2
et-xmlfile==1.1.0
fa2==0.3.5
fastai==1.0.61
fastcluster==1.2.4
fastdtw==0.3.4
fastprogress==1.0.0
fastrlock==0.6
fbprophet==0.7.1
feather-format==0.4.1
ffmpy==0.2.3
filelock==3.0.12
firebase-admin==4.4.0
fix-yahoo-finance==0.0.22
Flask==1.1.4
flatbuffers==1.12
folium==0.8.3
future==0.16.0
gast==0.3.3
GDAL==2.2.2
gdown==3.6.4
gensim==3.6.0
geographiclib==1.52
geopy==1.17.0
gin-config==0.4.0
glob2==0.7
google==2.0.3
google-api-core==1.26.3
google-api-python-client==1.12.8
google-auth==1.34.0
google-auth-httplib2==0.0.4
google-auth-oauthlib==0.4.5
google-cloud-bigquery==1.21.0
google-cloud-bigquery-storage==1.1.0
google-cloud-core==1.0.3
google-cloud-datastore==1.8.0
google-cloud-firestore==1.7.0
google-cloud-language==1.2.0
google-cloud-storage==1.18.1
google-cloud-translate==1.5.0
google-colab @ file:///colabtools/dist/google-colab-1.0.0.tar.gz
google-pasta==0.2.0
google-resumable-media==0.4.1
googleapis-common-protos==1.53.0
googledrivedownloader==0.4
GPUtil==1.4.0
graphviz==0.10.1
greenlet==1.1.1
grpcio==1.32.0
gspread==3.0.1
gspread-dataframe==3.0.8
gym==0.17.3
h5py==2.10.0
HeapDict==1.0.1
hijri-converter==2.1.3
holidays==0.10.5.2
holoviews==1.14.5
html5lib==1.0.1
httpimport==0.5.18
httplib2==0.17.4
httplib2shim==0.0.3
humanize==0.5.1
hyperopt==0.1.2
ideep4py==2.0.0.post3
idna==2.10
imageio==2.9.0
imageio-ffmpeg==0.4.5
imagesize==1.2.0
imbalanced-learn==0.4.3
imblearn==0.0
imgaug==0.2.9
importlib-metadata==4.6.4
importlib-resources==5.2.2
imutils==0.5.4
inflect==2.1.0
iniconfig==1.1.1
intel-openmp==2021.3.0
intervaltree==2.1.0
ipykernel==4.10.1
ipython==5.5.0
ipython-genutils==0.2.0
ipython-sql==0.3.9
ipywidgets==7.6.3
itsdangerous==1.1.0
jax==0.2.19
jaxlib @ https://storage.googleapis.com/jax-releases/cuda110/jaxlib-0.1.70+cuda110-cp37-none-manylinux2010_x86_64.whl
jdcal==1.4.1
jedi==0.18.0
jieba==0.42.1
Jinja2==2.11.3
joblib==1.0.1
jpeg4py==0.1.4
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.3.5
jupyter-console==5.2.0
jupyter-core==4.7.1
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.0
kaggle==1.5.12
kapre==0.3.5
keras==2.6.0
Keras-Preprocessing==1.1.2
keras-vis==0.4.1
kiwisolver==1.3.1
korean-lunar-calendar==0.2.1
librosa==0.8.1
lightgbm==2.2.3
llvmlite==0.34.0
lmdb==0.99
LunarCalendar==0.0.9
lxml==4.2.6
Markdown==3.3.4
MarkupSafe==2.0.1
matplotlib==3.2.2
matplotlib-inline==0.1.2
matplotlib-venn==0.11.6
missingno==0.5.0
mistune==0.8.4
mizani==0.6.0
mkl==2019.0
mlxtend==0.14.0
more-itertools==8.8.0
moviepy==0.2.3.5
mpmath==1.2.1
msgpack==1.0.2
multiprocess==0.70.12.2
multitasking==0.0.9
murmurhash==1.0.5
music21==5.5.0
natsort==5.5.0
nbclient==0.5.4
nbconvert==5.6.1
nbformat==5.1.3
nest-asyncio==1.5.1
netCDF4==1.5.7
networkx==2.6.2
nibabel==3.0.2
nltk==3.2.5
notebook==5.3.1
numba==0.51.2
numexpr==2.7.3
numpy==1.19.5
nvidia-ml-py==11.470.66
nvidia-ml-py3==7.352.0
oauth2client==4.1.3
oauthlib==3.1.1
okgrade==0.4.3
opencv-contrib-python==4.1.2.30
opencv-python==4.5.3.56
openpyxl==2.5.9
opt-einsum==3.3.0
osqp==0.6.2.post0
packaging==21.0
palettable==3.3.0
pandas==1.1.5
pandas-datareader==0.9.0
pandas-gbq==0.13.3
pandas-profiling==1.4.1
pandocfilters==1.4.3
panel==0.12.1
param==1.11.1
parso==0.8.2
pathlib==1.0.1
patsy==0.5.1
pep517==0.11.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==8.3.2
pip-tools==6.2.0
plac==1.1.3
plotly==4.4.1
plotnine==0.6.0
pluggy==0.7.1
pooch==1.4.0
portpicker==1.3.9
prefetch-generator==1.0.1
preshed==3.0.5
prettytable==2.1.0
progressbar2==3.38.0
prometheus-client==0.11.0
promise==2.3
prompt-toolkit==1.0.18
protobuf==3.17.3
psutil==5.8.0
psycopg2==2.7.6.1
ptyprocess==0.7.0
py==1.10.0
pyarrow==3.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycocotools==2.0.2
pycparser==2.20
pyct==0.4.8
pydata-google-auth==1.2.0
pydot==1.3.0
pydot-ng==2.0.0
pydotplus==2.0.2
PyDrive==1.3.1
pyemd==0.5.1
pyerfa==2.0.0
pyglet==1.5.0
Pygments==2.6.1
pygobject==3.26.1
pymc3==3.11.2
PyMeeus==0.5.11
pymongo==3.12.0
pymystem3==0.2.0
PyOpenGL==3.1.5
pyparsing==2.4.7
pyrsistent==0.18.0
pysndfile==1.3.8
PySocks==1.7.1
pystan==2.19.1.1
pytest==3.6.4
python-apt==0.0.0
python-chess==0.23.11
python-dateutil==2.8.2
python-louvain==0.15
python-slugify==5.0.2
python-utils==2.5.6
pytz==2018.9
pyviz-comms==2.1.0
PyWavelets==1.1.1
PyYAML==3.13
pyzmq==22.2.1
qdldl==0.1.5.post0
qtconsole==5.1.1
QtPy==1.10.0
regex==2019.12.20
requests==2.23.0
requests-oauthlib==1.3.0
resampy==0.2.2
retrying==1.3.3
rpy2==3.4.5
rsa==4.7.2
scikit-image==0.16.2
scikit-learn==0.24.2
scipy==1.4.1
screen-resolution-extra==0.0.0
scs==2.1.4
seaborn==0.11.1
semver==2.13.0
Send2Trash==1.8.0
setuptools-git==1.2
Shapely==1.7.1
simplegeneric==0.8.1
six==1.15.0
sklearn==0.0
sklearn-pandas==1.8.0
smart-open==5.1.0
snowballstemmer==2.1.0
sortedcontainers==2.4.0
SoundFile==0.10.3.post1
spacy==2.2.4
Sphinx==1.8.5
sphinxcontrib-serializinghtml==1.1.5
sphinxcontrib-websupport==1.2.4
SQLAlchemy==1.4.22
sqlparse==0.4.1
srsly==1.0.5
statsmodels==0.10.2
sympy==1.7.1
tables==3.4.4
tabulate==0.8.9
tblib==1.7.0
tensorboard==2.6.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorflow==2.4.0
tensorflow-datasets==4.0.1
tensorflow-estimator==2.4.0
tensorflow-gcs-config==2.6.0
tensorflow-gpu==2.4.0
tensorflow-hub==0.12.0
tensorflow-metadata==1.2.0
tensorflow-probability==0.13.0
termcolor==1.1.0
terminado==0.11.0
testpath==0.5.0
text-unidecode==1.3
textblob==0.15.3
Theano-PyMC==1.1.2
thinc==7.4.0
threadpoolctl==2.2.0
tifffile==2021.8.30
toml==0.10.2
tomli==1.2.1
toolz==0.11.1
torch @ https://download.pytorch.org/whl/cu102/torch-1.9.0%2Bcu102-cp37-cp37m-linux_x86_64.whl
torchsummary==1.5.1
torchtext==0.10.0
torchvision @ https://download.pytorch.org/whl/cu102/torchvision-0.10.0%2Bcu102-cp37-cp37m-linux_x86_64.whl
tornado==5.1.1
tqdm==4.62.0
traitlets==5.0.5
tweepy==3.10.0
typeguard==2.7.1
typing-extensions==3.7.4.3
tzlocal==1.5.1
uritemplate==3.0.1
urllib3==1.24.3
vega-datasets==0.9.0
wasabi==0.8.2
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==1.0.1
widgetsnbextension==3.5.1
wordcloud==1.5.0
wrapt==1.12.1
xarray==0.18.2
xgboost==0.90
xkit==0.0.0
xlrd==1.1.0
xlwt==1.3.0
yellowbrick==0.9.1
zict==2.0.0
zipp==3.5.0
=============== State File =================
{
"name": "villain",
"sessions": {
"1": {
"timestamp": 1631431573.2035065,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 3960,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"2": {
"timestamp": 1631485930.8518226,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 3960,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"3": {
"timestamp": 1631503174.5515265,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 3600,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"4": {
"timestamp": 1631541392.1880405,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 7560,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"5": {
"timestamp": 1631568333.871523,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 1,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"6": {
"timestamp": 1631568527.213653,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 1,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"7": {
"timestamp": 1631572690.7077212,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 1,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"8": {
"timestamp": 1631573832.957988,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 61,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"9": {
"timestamp": 1631574142.30032,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 1,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"10": {
"timestamp": 1631576717.8594546,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 3600,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"11": {
"timestamp": 1631592607.7613473,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 1800,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"12": {
"timestamp": 1631621477.0965884,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 42,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"13": {
"timestamp": 1631628807.7275484,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 5798,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"14": {
"timestamp": 1631647215.6946719,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 1440,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"15": {
"timestamp": 1631663237.8258858,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 5811,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"16": {
"timestamp": 1631707637.9922519,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 7364,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"17": {
"timestamp": 1631730602.151861,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 1168,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"18": {
"timestamp": 1631743803.4545527,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 3583,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"19": {
"timestamp": 1631763981.4230247,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 250,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"20": {
"timestamp": 1631765029.4450378,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 1,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"21": {
"timestamp": 1631767416.9511135,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 16,
"iterations": 1,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
},
"22": {
"timestamp": 1631767965.2077034,
"no_logs": false,
"loss_names": [
"total",
"face_a",
"face_b"
],
"batchsize": 8,
"iterations": 360,
"config": {
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"nan_protection": true,
"convert_batchsize": 16,
"eye_multiplier": 3,
"mouth_multiplier": 2
}
}
},
"lowest_avg_loss": {
"a": 0.035034529864788055,
"b": 0.03087991289794445
},
"iterations": 50363,
"config": {
"centering": "face",
"coverage": 68.75,
"optimizer": "adam",
"learning_rate": 5e-05,
"epsilon_exponent": -7,
"allow_growth": false,
"mixed_precision": false,
"nan_protection": true,
"convert_batchsize": 16,
"loss_function": "ssim",
"mask_loss_function": "mse",
"l2_reg_term": 100,
"eye_multiplier": 3,
"mouth_multiplier": 2,
"penalized_mask_loss": true,
"mask_type": "extended",
"mask_blur_kernel": 3,
"mask_threshold": 4,
"learn_mask": false,
"lowmem": false
}
}
================= Configs ==================
--------- train.ini ---------
[global]
centering: face
coverage: 68.75
icnr_init: False
conv_aware_init: False
optimizer: adam
learning_rate: 5e-05
epsilon_exponent: -7
reflect_padding: False
allow_growth: False
mixed_precision: False
nan_protection: True
convert_batchsize: 16
[global.loss]
loss_function: ssim
mask_loss_function: mse
l2_reg_term: 100
eye_multiplier: 3
mouth_multiplier: 2
penalized_mask_loss: True
mask_type: extended
mask_blur_kernel: 3
mask_threshold: 4
learn_mask: False
[trainer.original]
preview_images: 14
zoom_amount: 5
rotation_range: 10
shift_range: 5
flip_chance: 50
color_lightness: 30
color_ab: 8
color_clahe_chance: 50
color_clahe_max_size: 4
[model.dfaker]
output_size: 128
[model.phaze_a]
output_size: 128
shared_fc: none
enable_gblock: True
split_fc: True
split_gblock: False
split_decoders: False
enc_architecture: fs_original
enc_scaling: 40
enc_load_weights: True
bottleneck_type: dense
bottleneck_norm: none
bottleneck_size: 1024
bottleneck_in_encoder: True
fc_depth: 1
fc_min_filters: 1024
fc_max_filters: 1024
fc_dimensions: 4
fc_filter_slope: -0.5
fc_dropout: 0.0
fc_upsampler: upsample2d
fc_upsamples: 1
fc_upsample_filters: 512
fc_gblock_depth: 3
fc_gblock_min_nodes: 512
fc_gblock_max_nodes: 512
fc_gblock_filter_slope: -0.5
fc_gblock_dropout: 0.0
dec_upscale_method: subpixel
dec_norm: none
dec_min_filters: 64
dec_max_filters: 512
dec_filter_slope: -0.45
dec_res_blocks: 1
dec_output_kernel: 5
dec_gaussian: True
dec_skip_last_residual: True
freeze_layers: keras_encoder
load_layers: encoder
fs_original_depth: 4
fs_original_min_filters: 128
fs_original_max_filters: 1024
mobilenet_width: 1.0
mobilenet_depth: 1
mobilenet_dropout: 0.001
[model.realface]
input_size: 64
output_size: 128
dense_nodes: 1536
complexity_encoder: 128
complexity_decoder: 512
[model.villain]
lowmem: False
[model.dfl_h128]
lowmem: False
[model.dfl_sae]
input_size: 128
clipnorm: True
architecture: df
autoencoder_dims: 0
encoder_dims: 42
decoder_dims: 21
multiscale_decoder: False
[model.original]
lowmem: False
[model.unbalanced]
input_size: 128
lowmem: False
clipnorm: True
nodes: 1024
complexity_encoder: 128
complexity_decoder_a: 384
complexity_decoder_b: 512
[model.dlight]
features: best
details: good
output_size: 256
--------- convert.ini ---------
[writer.ffmpeg]
container: mp4
codec: libx264
crf: 23
preset: medium
tune: none
profile: auto
level: auto
skip_mux: False
[writer.pillow]
format: png
draw_transparent: False
optimize: False
gif_interlace: True
jpg_quality: 75
png_compress_level: 3
tif_compression: tiff_deflate
[writer.gif]
fps: 25
loop: 0
palettesize: 256
subrectangles: False
[writer.opencv]
format: png
draw_transparent: False
jpg_quality: 75
png_compress_level: 3
[scaling.sharpen]
method: none
amount: 150
radius: 0.3
threshold: 5.0
[mask.mask_blend]
type: normalized
kernel_size: 3
passes: 4
threshold: 4
erosion: 0.0
[mask.box_blend]
type: gaussian
distance: 11.0
radius: 5.0
passes: 1
[color.match_hist]
threshold: 99.0
[color.manual_balance]
colorspace: HSV
balance_1: 0.0
balance_2: 0.0
balance_3: 0.0
contrast: 0.0
brightness: 0.0
[color.color_transfer]
clip: True
preserve_paper: True
--------- extract.ini ---------
[global]
allow_growth: False
[align.fan]
batch-size: 12
[mask.bisenet_fp]
batch-size: 8
include_ears: False
include_hair: False
include_glasses: True
[mask.vgg_obstructed]
batch-size: 2
[mask.unet_dfl]
batch-size: 8
[mask.vgg_clear]
batch-size: 6
[detect.s3fd]
confidence: 70
batch-size: 4
[detect.mtcnn]
minsize: 20
scalefactor: 0.709
batch-size: 8
threshold_1: 0.6
threshold_2: 0.7
threshold_3: 0.7
[detect.cv2_dnn]
confidence: 50
--------- .faceswap ---------
backend: nvidia