Error during extract - CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered

[dancloud1234]

I installed using Nvidia CPU, my Nvidia is (GeForce GTX 1070) after finish installation no issue, continue to open the file and able to use the software.

After selecting the video and clicking extract i got an error below, and the same as well with S3fd

Code: Select all

Setting Faceswap backend to NVIDIA
10/28/2020 18:35:58 INFO Log level set to: INFO
10/28/2020 18:36:00 INFO Output Directory: C:\Users\Daniel-PC\Videos\Model B

10/28/2020 18:36:00 INFO Loading Detect from Mtcnn plugin...
10/28/2020 18:36:00 INFO Loading Align from Fan plugin...
10/28/2020 18:36:00 INFO Loading Mask from Components plugin...
10/28/2020 18:36:00 INFO Loading Mask from Extended plugin...
10/28/2020 18:36:00 INFO Starting, this may take a while...
10/28/2020 18:36:00 INFO Initializing MTCNN (Detect)...
10/28/2020 18:36:02 INFO Initialized MTCNN (Detect) with batchsize of 8
10/28/2020 18:36:02 INFO Initializing FAN (Align)...

2020-10-28 18:36:04.204035: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:725] failed to record completion event; therefore, failed to create inter-stream dependency
2020-10-28 18:36:04.204401: E tensorflow/stream_executor/stream.cc:334] Error recording event in stream: Error recording CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered; not marking stream as bad, as the Event object may be at fault. Monitor for further errors.
2020-10-28 18:36:04.204967: E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2020-10-28 18:36:04.205426: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:273] Unexpected Event status: 1
Process exited.

10/28/2020 18:43:50 INFO Loading Detect from S3Fd plugin...
10/28/2020 18:43:50 INFO Loading Align from Fan plugin...
10/28/2020 18:43:50 INFO Loading Mask from Components plugin...
10/28/2020 18:43:50 INFO Loading Mask from Extended plugin...
10/28/2020 18:43:50 INFO Starting, this may take a while...
10/28/2020 18:43:50 INFO Initializing S3FD (Detect)...
10/28/2020 18:43:51 INFO Initialized S3FD (Detect) with batchsize of 4
10/28/2020 18:43:51 INFO Initializing FAN (Align)...

2020-10-28 18:43:54.494508: E tensorflow/stream_executor/cuda/cuda_driver.cc:910] failed to synchronize the stop event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2020-10-28 18:43:54.494821: E tensorflow/stream_executor/gpu/gpu_timer.cc:55] Internal: Error destroying CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2020-10-28 18:43:54.495199: E tensorflow/stream_executor/gpu/gpu_timer.cc:60] Internal: Error destroying CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2020-10-28 18:43:54.495492: E tensorflow/stream_executor/stream.cc:5485] Internal: Failed to enqueue async memset operation: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2020-10-28 18:43:54.495879: E tensorflow/stream_executor/cuda/cuda_driver.cc:575] failed to load PTX text as a module: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2020-10-28 18:43:54.496149: E tensorflow/stream_executor/cuda/cuda_driver.cc:580] error log buffer (1024 bytes):
2020-10-28 18:43:54.496344: E tensorflow/stream_executor/cuda/cuda_driver.cc:575] failed to load PTX text as a module: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2020-10-28 18:43:54.496580: E tensorflow/stream_executor/cuda/cuda_driver.cc:580] error log buffer (1024 bytes):
2020-10-28 18:43:54.521217: F .\tensorflow/core/kernels/random_op_gpu.h:232] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: an illegal memory access was encountered

I am still new to this and please help, i tried uninstalling and reinstall but still the same issue.

[torzdf]

This is hardware related. Most likely you have a factory overclock on your GPU, and you should remove this. Otherwise, it may be power supply related.

[dancloud1234]

Thank for replying, any info on how to remove this factory overclocked? or the suggestion mentioned?

[torzdf]

It depends from vendor to vendor. You should be able to google around "overclock settings" for your specific GPU.

[abigflea]

While I won't swear you're having the same issues I've had, one of my 2070's is factory overclocked a whopping (sarcasm) 50 MHz and gives me this error after a couple hours training.
I just underclock them both to 1700mhz and it stops.

Maybe it is a factory overclock, maybe not, butI agree with Torzdf it's somewhere hardware related .

[bryanlyon]

If using Windows I suggest MSI afterburner. It'll work on any Nvidia cards.
https://www.msi.com/Landing/afterburner

If using Linux you have to first activate coolbits then you can change the clock in the nvidiai control panel.
https://wiki.archlinux.org/index.php/NV ... nd_cooling

In either case, you simply change the clock speed to the speed that Nvidia.com says that card should have. This will remove any factory overclocks and run your card at the speed that the chip's designer recommends.