I've been getting this error alot. The model runs for a while and this then spits out.
AMD 3800x
32 GB ram
2060 Super
Any Ideas. This same hardware setup used to run like a top on the same (or heavier) models. No crashlog is created,
2021-04-12 04:43:11.527466: F ./tensorflow/core/kernels/conv_2d_gpu.h:459] Non-OK-status: GpuLaunchKernel(ShuffleInTensor3Simple<T, 1, 2, 0>, config.block_count, config.thread_per_block, 0, d.stream(), config.virtual_thread_count, in.data(), combined_dims, out.data()) status: Internal: an illegal memory access was encountered
2021-04-12 04:43:11.527467: E tensorflow/stream_executor/dnn.cc:613] CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/stream_executor/cuda/cuda_dnn.cc(3176): 'cudnnConvolutionBackwardData( cudnn.handle(), alpha, filter_nd.handle(), filter_data.opaque(), output_nd.handle(), output_data.opaque(), conv.handle(), ToConvBackwardDataAlgo(algorithm_desc), scratch_memory.opaque(), scratch_memory.size(), beta, input_nd.handle(), input_data.opaque())'