Hi, thanks for your great work. I am planning on training your model with custom dataset. I encounter following error:
""""CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)"""""
I tried multiple ways to solve this issue such as reducing batch size to 1, reducing number of gpus to 1 and reducing resolution of images to 64X64. I am training on NVIDIA Titan Xp GPUs with 12GB RAM. I didn't find any luck yet!
Can you help me resolve this issue?
Hi, thanks for your great work. I am planning on training your model with custom dataset. I encounter following error:
""""CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)"""""
I tried multiple ways to solve this issue such as reducing batch size to 1, reducing number of gpus to 1 and reducing resolution of images to 64X64. I am training on NVIDIA Titan Xp GPUs with 12GB RAM. I didn't find any luck yet!
Can you help me resolve this issue?