What's a reasonable speed? Also, error message

If training is failing to start, and you are not receiving an error message telling you what to do, tell us about it here


Forum rules

Read the FAQs and search the forum before posting a new topic.

This forum is for reporting errors with the Training process. If you want to get tips, or better understand the Training process, then you should look in the Training Discussion forum.

Please mark any answers that fixed your problems so others can find the solutions.

Post Reply
User avatar
hullo
Posts: 16
Joined: Wed Aug 23, 2023 8:30 am
Been thanked: 1 time

What's a reasonable speed? Also, error message

Post by hullo »

I'm up and running now, no crashes [knock on wood]. It's going at about one iteration per second. Realface method, everything default iirc. Is that a reasonable rate or should it be going much faster?

I'm also getting a different error message at the start than last time, so if the above speed is abnormally slow, maybe there's a clue here:

Code: Select all

WARNING:tensorflow:Mixed precision compatibility check (mixed_float16): WARNING11/03/2023 04:09:20 INFO     No existing state file found. Generating.

Your GPU may run slowly with dtype policy mixed_float16 because it does not have compute capability of at least 7.0. Your GPU:11/03/2023 04:09:20 INFO     Storing Mixed Precision compatible layers. Please ignore any following warnings about using mixed precision.

METAL, no compute capability (probably not an Nvidia GPU)
11/03/2023 04:09:20 WARNING  Mixed precision compatibility check (mixed_float16): WARNINGSee https://developer.nvidia.com/cuda-gpus for a list of GPUs and their compute capabilities.

If you will use compatible GPU(s) not attached to this host, e.g. by running a multi-worker model, you can ignore this warning. This message will only be logged onceYour GPU may run slowly with dtype policy mixed_float16 because it does not have compute capability of at least 7.0. Your GPU:

  METAL, no compute capability (probably not an Nvidia GPU)
See https://developer.nvidia.com/cuda-gpus for a list of GPUs and their compute capabilities.
If you will use compatible GPU(s) not attached to this host, e.g. by running a multi-worker model, you can ignore this warning. This message will only be logged once
User avatar
torzdf
Posts: 2687
Joined: Fri Jul 12, 2019 12:53 am
Answers: 159
Has thanked: 135 times
Been thanked: 628 times

Re: What's a reasonable speed? Also, error message

Post by torzdf »

Those message are standard, as we have to store mixed-precision compatible layers at start up. As it looks like you are using Apple Silicon, then that message is expected.

I can't speak for speed, as I don't have an Apple computer, however, if you run training with VERBOSE logging, it should tell you what device is being used.

My word is final

Post Reply