cuda sub matrix - cuda

Question

problem:
I have 4 matrix (64x64) of single precision numbers. need to do calculation like:
R = A * sin(B) + C * cos(D)
idea:
to speed up calculation use shared memory. since each block of threads has (in case of my GPU) 16KB shared memory and size of float is 4 there can by stored 4000 floating point numbers in shared memory. so for each matrix use 1000 elements which is 31 elements per dimension.
so each matrix shoud be devided in 16 submatrix (16x16).
dim3 dimBlock(16, 16, 1)
dim3 dimGrid(4, 4, 1)
kernel:
int Tx = threadIdx.x;
int Ty = threadIdx.y;
int Bx = blockIdx.x;
int By = blockIdx.y;
int idx = Bx * blockDim.x + Tx;
int idy = By * blockDim.y + Ty;
__shared__ float s_A[16*16];
__shared__ float s_B[16*16];
__shared__ float s_C[16*16];
__shared__ float s_D[16*16];
// I am not sure how to write this part
s_A[(Tx * blockDim.x + Ty + By) + Bx] = A[idx * 64 + idy];
s_B[(Tx * blockDim.x + Ty + By) + Bx] = B[idx * 64 + idy];
s_C[(Tx * blockDim.x + Ty + By) + Bx] = C[idx * 64 + idy];
s_D[(Tx * blockDim.x + Ty + By) + Bx] = D[idx * 64 + idy];
R[idx * 64 + idy] = s_A[(Tx * blockDim.x + Ty + By) + Bx] * sin(s_B[(Tx * blockDim.x + Ty + By) + Bx]) + s_C[(Tx * blockDim.x + Ty + By) + Bx] * cos(s_D[(Tx * blockDim.x + Ty + By) + Bx]);
How to devide original matrix to submatrixs so each block has own 4 submatrix and calculate on them.

Answer 1

Unless I have misinterpreted your question, you don't need to and shouldn't use shared memory for this operation. Shared memory is useful for sharing and resuing data between threads within the same block, and for facilitating coalesced memory access. Your operation seems to required neither of those things to work correctly. Using shared memory in the way you propose would probably be slower than just reading from global memory directly. Also, because you are only worried about element wise operations, the indexing scheme of your kernel can be greatly simplified -- the fact that A, B, C and D are "matrices" is irrelevant to the calculations as I understand your question.
As a result, an near optimal version of your kernel could be written as simply as
__global__ void kernel(const float *A, const float *B, const float *C,
const float *D, const int n, float *R)
{
int tidx = threadIdx.x + blockIdx.x * blockDim.x;
int stride = blockDim.x * gridDim.x;
while(tidx < n) {
R[tidx] = A[idx] * sinf(B[idx]) + C[idx]*cosf(D[idx]);
tidx += stride
}
}
In this code, you would launch as many blocks as would reach peak throughput of your GPU, and each thread will process more than one input/output value if the size of the array exceeds the size of the optimal 1D grid you have launched. Of course this is pretty academic if you are only processing 4096 elements in total -- that is probably about 2 orders of magnitude too small to get any benefit from using a GPU.

Answer 2

You have a problem here that your operation/transfer ratio is of order 1. You might have a hard time actually getting any decent speed from your GPU because of a bandwidth bottleneck between the thread and global memory and not having a way to reduce that.
A shared memory solution is usually best when there is some data called repeatedly from global memory. Instead of loading this data repeatedly from the low bandwidth, high latency global memory, you load it once from there, and do subsequent loads from the higher bandwidth, lower latency shared memory. Note, that's higher and lower, not high and low. There is still a performance penalty from using shared memory.
You your case, since elements aren't called several times from global memory, storing them in shared memory will only add the bandwidth limitations and latency that comes with shared memory usage. So, in effect, this solution will just add on the latency of access from shared memory to your data loading.
Now, if you have several calculations to perform, and some of these matrices are used in them too, then combining them into one kernel might give you a speed boost, since you might be able to load these once for the whole thing instead of once per operation. If that's not the case, and you can't increase your operation/transfer ratio, then you'll have a hard time getting some decent speeds, and might be better off doing these calculations on the CPU.
You might even get some decent results from multithreading on the CPU.