Marking up code for GPU compilation

[hiker]

We need to find a way to 'mark-up' subroutine that need to be compiled for GPU (i.e. subroutines called from a DSL kernel). While (kernel module) inlining might work, some functions are called from many different kernels, which will result in a (significant??) increase of the size of the compiled binary.

This is an issue that affects both the build system and PSyclone (probably more the build system), but I open the discussion here since we are all dealing with this issue, and it affects especially our GPU coverage for LFRic.

My approach was guided by the following requirements:

• It needs to be able to run in parallel (i.e. PSyclone typically runs in parallel, so we can't e.g. write data to one file, or at least we would need synchronisation)
• It should not closely connect PSyclone and Fab (or any other build system), i.e. should potentially be useful with other build systems as well, and PSyclone should have no direct dependency on something in Fab.
• We don't want additional dependencies to be added to PSyclone (e.g. adding sqlite)

Given this, my current suggested solution works like this:

• Any PSyclone script (ie. controlled by the user, not part of PSyclone) can write a dependency-information file. This file can for example store the list of 'secondary' subroutines to be marked up in yaml format. PSyclone might provide a convenience function to convert the static call tree information into yaml format (which a user script can use to simplify the work).
• The filename might be provided as parameter (e.g. environment variable, or Parameters for transformation scripts #2757 ), or a script might use its own name and location to derive a name.
• The build system will check these files once the PSyclone run is finished. It can merge the information contained in these files (Fab might contain a convenience function for that, which might be usable stand-alone by other build systems), and so get the list of files to mark up. This can then be done in a separate phase of the build system with a custom script.

The advantage of this approach:

• no need to manually mark-up source files before, i.e. it follows the PSyclone philosophy.
• it minimises the number of files to me compiled for GPU (and CPU) to the ones really required.
• it leaves control of the markup to PSyclone

I am aware that Sergi had a different approach in mind, I would be interested to get more detailed in this discussion, and see if we can combine these, or if one solution is better, or ..

@arporter , @LonelyCat124 , @sergisiso feedback welcome

[sergisiso]

The approach I had in mine was a bottom-up approach in terms of dependencies. I was thinking to support any build system (not only those adapted to psyclone/psykal) which typically means that the dependencies (files listed in use statement) is compiled/processed before the dependent (file that has the use statement). The idea is:

• While processing files annotate subroutines that can be on the GPU, this can be listed by name, try to do all, or some domain/application pattern, ...
• Parse imports with --keep-directives (this will not only assist with psyclone annotated subroutines but also understand manually annotated code)
• Update the existing call.is_available_on_device to reflect if the subroutine has an annotation.

I can see how the top-down could be more constrained in automatically only annotating the needed calls, but I don't see how the dependency-information file can be use in build systems that are not designed/aware of psyclone or with code that has pre-existing manual directive in some files. So maybe we need both?

[hiker]

The approach I had in mine was a bottom-up approach in terms of dependencies. I was thinking to support any build system (not only those adapted to psyclone/psykal) which typically means that the dependencies (files listed in use statement) is compiled/processed before the dependent (file that has the use statement). The idea is:

I assume you basically think of using the 'psyclone compiler wrapper'? I.e. you just change the compiler, and can then pre-process files just before compilation - is that correct?

Indeed, that's quite different from what I suggest here, and I would agree, we might need both (though, your approach seems to be mostly already possible - keep directives, and maybe add a custom directive to be added to source files that will then trigger to add the correct 'compile-for-GPU' behaviour).

• While processing files annotate subroutines that can be on the GPU, this can be listed by name, try to do all, or some domain/application pattern, ...
• Parse imports with --keep-directives (this will not only assist with psyclone annotated subroutines but also understand manually annotated code)
• Update the existing call.is_available_on_device to reflect if the subroutine has an annotation.

I admit I am not fully aware of the is_available_on_device function ;) But yes, I get the 'bottom up' approach here.

I can see how the top-down could be more constrained in automatically only annotating the needed calls, but I don't see how the dependency-information file can be use in build systems that are not designed/aware of psyclone or with code that has pre-existing manual directive in some files. So maybe we need both?

We could provide a script that would extract the dependency-information file 'somehow' (e.g. into an environment variable or so), which could be picked up by any build system. But tbh, I don't think we have time (or need) to go there :)

I think both approaches can co-exist quite easily, and I will try to implement the solution for LFRic with minimal assumptions, and no real change to PSyclone (and maybe some examples scripts ... not sure where they would be stored though :) ).

[LonelyCat124]

Parse imports with --keep-directives (this will not only assist with psyclone annotated subroutines but also understand manually annotated code)

I can look at being able to turn whichever directive we need into a real directive node as part of this too.

[sergisiso]

Its:
omp delcate target to OMPDeclareTargetDirective
and
acc routine to ACCRoutineDirective

[LonelyCat124]

Both are standalone directives so I'll have a try.

[sergisiso]

This is probably a separate issue but I have manually annotated the LFRic infrastructure and end up needed a few "declaration annotations", such as:

real(kind=r_def)    :: chi2xyz_rot_mat(3,3)
!$omp declare target (chi2xyz_rot_mat)
real(kind=r_def)    :: xyz2chi_rot_mat(3,3)
!$omp declare target (xyz2chi_rot_mat)
real(kind=r_def)    :: stretch_factor
!$omp declare target (stretch_factor)
logical(kind=l_def) :: to_rotate
!$omp declare target (to_rotate)
logical(kind=l_def) :: to_stretch
!$omp declare target (to_stretch)

Do we have any chance to capture these?

[hiker]

That should be easy, the driver extraction (which is based on the static call tree) detects and stores these variables in the driver data files (and reads them in).