Search found 714 matches

by bryanlyon
Sat Dec 10, 2022 5:35 am
Forum: Training Discussion
Topic: Shorter Training=Smaller Chances of NaNs; A hypothesis by an Uneducated Idiot
Replies: 2
Views: 33

Re: Shorter Training=Smaller Chances of NaNs; A hypothesis by an Uneducated Idiot

Very interesting. If someone else can try and verify these results I'd appreciate it. I don't typically get many NaNs so cannot test this well. This may justify an option in the training script if others can replicate.

by bryanlyon
Sat Dec 10, 2022 4:03 am
Forum: Hardware
Topic: Hardware best practices
Replies: 71
Views: 91767

Re: Hardware best practices

Thank you for the reply. How are the RAM temps on your 3090? I read that VRAM intensive applications (e.g., ETH mining) caused dangerous temperature (>100 C memory junction temperature) levels on many 3090s, and required additional cooling to the back of the card. I wonder if FS has a similar type ...
by bryanlyon
Sun Dec 04, 2022 11:19 am
Forum: Training Discussion
Topic: [Tip] Why you need more sources (with examples)
Replies: 2
Views: 13322

Re: [Tip] Why you need more sources (with examples)

The length of the video doesn't matter at all. You're training on faces, not a video. A short video doesn't take any less time than a long one since you'll have to train a model with sufficient data before you can use it to convert.

by bryanlyon
Sun Dec 04, 2022 12:16 am
Forum: Convert Support
Topic: Can a model be used on two different clips of the same person?
Replies: 1
Views: 104

Re: Can a model be used on two different clips of the same person?

When you train a model, you train it on 2 identities. You can then swap ANY videos with those identities. People like to "fine tune" a model on the swap video to make sure it's got the right exposure to that type of content, it's not required though.

by bryanlyon
Fri Dec 02, 2022 5:37 am
Forum: Training Support
Topic: Bicubic vs. Nearest Neighbor
Replies: 4
Views: 202

Re: Bicubic vs. Nearest Neighbor

by bryanlyon
Fri Nov 25, 2022 10:51 pm
Forum: Training Discussion
Topic: Nan Question I haven't seen yet
Replies: 6
Views: 278

Re: Nan Question I haven't seen yet

The short answer is "probably not but maybe". One thing that we do with our training process is we don't save out the optimizer weights. The Optimizer is a part of the model, but to save it's weights can triple the size of the model. Because of that, each time we start the model the weight...
by bryanlyon
Sun Nov 20, 2022 8:25 pm
Forum: Training Discussion
Topic: Thoughts on Auto-Training
Replies: 3
Views: 187

Re: Thoughts on Auto-Training

We welcome PRs, as like I said, the mechanics are easy but this is unlikely to be done by us since we don't know how we'd set up the options.

by bryanlyon
Sun Nov 20, 2022 6:37 pm
Forum: Training Discussion
Topic: Training freezes after 1100 iterations
Replies: 2
Views: 156

Re: Training freezes after 1100 iterations

Hmmm, can't be sure, but it looks like a storage access problem. ml.p3.2xlarge doesn't have local storage and it's synchronized to the network. Not sure why the hangup is happening though. Try a much higher save interval, not a lower one. That sets how many iterations between saving out your models.
by bryanlyon
Sun Nov 20, 2022 9:07 am
Forum: Training Discussion
Topic: Thoughts on Auto-Training
Replies: 3
Views: 187

Re: Thoughts on Auto-Training

Yes, this is possible, and the mechanics of it are quite easy. The hard part is the heuristics. When you roll back, how much you roll back, how much you lower LR and all of that are completely unknowable and so we leave it as an exercise to the user.

by bryanlyon
Sat Nov 19, 2022 10:48 am
Forum: General Discussion
Topic: Where are the biases?
Replies: 3
Views: 229

Re: Where are the biases?

Biases exist intrinsically in dense (Fully connected) layers but are generally not included in convolutional layers because convolutions work in a different way and don't really need them. HOWEVER, "weights" are generally taken to mean all variables used in the model. For example, convolut...
by bryanlyon
Sat Nov 19, 2022 12:42 am
Forum: Extract Discussion
Topic: Is it a good idea to extract from your faces folder to consolidate the alignments file?
Replies: 1
Views: 161

Re: Is it a good idea to extract from your faces folder to consolidate the alignments file?

No, you don't need to combine multiple extracts for one alignment file. The alignment file is only needed for the convert step, and since each time you extract a video it creates a matching alignment file you don't need to combine them at all anymore.

by bryanlyon
Sat Nov 12, 2022 6:40 pm
Forum: Extract Discussion
Topic: Cleaning Alignments - Error
Replies: 3
Views: 159

Re: Cleaning Alignments - Error

Check the file at 'C:\Users\chris\Desktop\NIKBIT\spiderman0 (40).png' first. That filename doesn't look like an extracted file to me. If that's not it, I'd suggest starting first with the Tools -> Alignments -> Rename to make sure they're all named with the original name and check if they all match....
by bryanlyon
Sat Nov 12, 2022 6:31 pm
Forum: Training Discussion
Topic: Experiments in pretraining
Replies: 15
Views: 611

Re: Experiments in pretraining

I tried raising the learning rate and it appears to destroy the model and progress. I'm assuming that is because it is now broadening the learning landscape and "resetting." But again I'm going to let it run for fun as an experiment. Either way it turns out it's going to teach me a good l...
by bryanlyon
Sat Nov 12, 2022 11:48 am
Forum: Training Discussion
Topic: Experiments in pretraining
Replies: 15
Views: 611

Re: Perfect 0.0 Loss value achieved !

Identity leakage is a subtle thing. It is easier to see in some contexts over others but it's there when you re-use a decoder. If you're training with FFHQ or similar dataset, the recommended method is to train a model with those, then copy the encoder over using the "Load Weights" when cr...
by bryanlyon
Sat Nov 12, 2022 10:01 am
Forum: Training Discussion
Topic: Experiments in pretraining
Replies: 15
Views: 611

Re: Perfect 0.0 Loss value achieved !

I am not blasting you. Just pointing out Dunning Kruger effect. The simple matter is that I don't have to train to 100 billion to know that FaceSwap can never become an "instant swapper" for the same reason that I know that a car cannot fly if you give it high enough octane fuel. It's not ...
by bryanlyon
Sat Nov 12, 2022 8:39 am
Forum: Training Discussion
Topic: Experiments in pretraining
Replies: 15
Views: 611

Re: Perfect 0.0 Loss value achieved !

Perfect loss is mathematically impossible. It's not just an infeasible thing, it's complete nonsense. Like claiming that some invention creates more power than it takes in -- It's simply impossible. That said, what you're doing is even worse in many ways. Effectively you're massively overtraining yo...
by bryanlyon
Sat Nov 12, 2022 2:48 am
Forum: Extract Discussion
Topic: Cleaning Alignments - Error
Replies: 3
Views: 159

Re: Cleaning Alignments - Error

I've never seen this. Are you sure that only images from that fsa are in the folder?

by bryanlyon
Thu Oct 27, 2022 1:21 am
Forum: General Discussion
Topic: PCIe 2.0 and 4.0
Replies: 2
Views: 259

Re: PCIe 2.0 and 4.0

On a single GPU your link speed is unlikely to be a major bottleneck.

by bryanlyon
Thu Oct 20, 2022 4:00 am
Forum: Training Discussion
Topic: Let me see if I have this right regarding training time...
Replies: 3
Views: 352

Re: Let me see if I have this right regarding training time...

No. Or more accurately mostly no. The model will train at the same speed if the data is good. That's not the whole story though. If you have a very few images then you'll "overtrain" before the model reaches it's maximum quality. That may show you what looks like high quality faces but it'...
by bryanlyon
Mon Oct 17, 2022 5:07 pm
Forum: Training Discussion
Topic: okay this totally threw me...
Replies: 2
Views: 232

Re: okay this totally threw me...

Check if your GPU is overclocked (factory overclocks too) and remove them. Often overclocking leads to slight instability that you wont notice in a video game but can be catastrophic to machine learning. If that still doesn't fix it, try power cycling your device. Turn it off. Unplug it, and leave i...