Reduction in OpenACC - cuda

Question

Here is a Fortran subroutine for matrix-vector multiply. It is probably old-fashioned and inefficient in a number of ways, but right now I am just trying to get it to work with OpenACC directives, and I'm trying to figure out how reduction works:
subroutine matrmult(matrix,invec,outvec,n)
integer:: n
real*8, intent(in):: matrix(n,n), invec(n)
real*8, intent(out) :: outvec(n)
real*8 :: tmpmat(n,n)
real*8 :: tmpscl
integer :: i,j,k
!$acc declare create(matrix, invec, outvec, tmpmat)
outvec = 0.d0
!$acc update device(matrix, invec, tmpmat, outvec)
!$acc parallel
!$acc loop gang
do j=1,n
!$acc loop vector
do i=1,n
tmpmat(i,j) = matrix(i,j)*invec(j)
enddo
enddo
!$acc loop vector reduction(+:tmpsclr)
do j=1,n
tmpsclr = 0.d0
do i=1,n
tmpsclr = tmpsclr+tmpmat(j,i)
enddo
outvec(j) = tmpsclr
enddo
!$acc end parallel
!$acc update host(outvec)
end subroutine
This code actually gives correct results. But when I try a gang/vector combination on the last loops, like so:
!$acc loop gang reduction(+:tmpsclr)
do j=1,n
tmpsclr = 0.d0
!$acc loop vector
do i=1,n
tmpsclr = tmpsclr+tmpmat(j,i)
enddo
outvec(j) = tmpsclr
enddo
the results come back all wrong. It looks like the summation is incomplete for most, but not all, of the elements of outvec. This is the case no matter where I put the reduction clause, whether with the gang or the vector. Changing the location changes the results, but never gives correct results.
The results I am getting in a simple test are like the following. matrix is 10x10 and all 1's, and invec is 1,2,3,...10. So the elements of outvec should each just be the sum of the elements in invec, 55. If I run the gang/vector version of the code, each element of outvec is 1, not 55. If I put the reduction with the vector, well, then I get the right answer, 55. And this continues to work until I get past 90 elements. When I get to 91, every element of outvec should be equal to 4186. But only the last one is, and all the rest are equal to 4095 (the sum of 1 to 90). As the number of elements get bigger the variation of values and the discrepancy from the correct answer gets worse.
I clearly don't understand how the reduction works. Can anyone explain?

Answer 1

The reduction clause needs to be on loop where the reduction occurs, i.e. the vector loop. I'd also recommend using the "kernels" directive here since "parallel" will create one kernel launch for the two loops, while "kernels" will create two kernels, one for each loop.
For example:
subroutine foo(n,matrix,invec,outvec)
integer n
real*8, intent(in) :: matrix(n,n)
real*8, intent(in) :: invec(n)
real*8, intent(out) :: outvec(n)
real*8 :: tmpmat(n,n)
real*8 :: tmpscl
integer :: i,j,k
!$acc declare create(matrix, invec, outvec, tmpmat)
outvec = 0.d0
!$acc update device(matrix, invec, tmpmat, outvec)
!$acc kernels
!$acc loop gang
do j=1,n
!$acc loop vector
do i=1,n
tmpmat(i,j) = matrix(i,j)*invec(j)
enddo
enddo
!$acc loop gang
do j=1,n
tmpsclr = 0.d0
!$acc loop vector reduction(+:tmpsclr)
do i=1,n
tmpsclr = tmpsclr+tmpmat(j,i)
enddo
outvec(j) = tmpsclr
enddo
!$acc end kernels
!$acc update host(outvec)
end subroutine foo
% pgf90 -c -acc -Minfo=accel test2.f90
foo:
11, Generating create(matrix(:,:),invec(:),outvec(:))
15, Generating update device(outvec(:),tmpmat(:,:),invec(:),matrix(:,:))
20, Loop is parallelizable
22, Loop is parallelizable
Accelerator kernel generated
Generating Tesla code
20, !$acc loop gang, vector(4) ! blockidx%y threadidx%y
22, !$acc loop gang, vector(32) ! blockidx%x threadidx%x
28, Loop is parallelizable
Accelerator kernel generated
Generating Tesla code
28, !$acc loop gang ! blockidx%x
31, !$acc loop vector(128) ! threadidx%x
Sum reduction generated for tmpsclr
31, Loop is parallelizable
39, Generating update host(outvec(:))
Hope this helps,
Mat