Load rocSOLVER with dlopen at runtime

[daineAMD]

Don't have access to a windows machine so this hasn't been tested on windows yet.

See #896 for the request. Similar changes for hipSOLVER have been done at hipSOLVER #302

This change allows hipBLAS to build without rocSOLVER available at buildtime. By default, hipBLAS will try to find rocSOLVER at runtime using dlopen(). If rocSOLVER cannot be found, the rocsolver-based functions will return HIPBLAS_STATUS_NOT_IMPLEMENTED.

The -n flags has been changed to mean "don't build rocSOLVER tests".
The --solver-buildtime flags has been added to find rocSOVLER at build time.

Also added some flags for jenkins so will be testing the various flags on CI.

[TorreZuk]

Is the static build without solver_buildtime the default as that creates an invalid static library?

[TorreZuk]

Is this just for testing or default to not link? Default to not link may may need to review effect on rpath or package consumption documentation.

[daineAMD]

Default will be to not link. I think it'll be ok but I'll test out consuming the hipblas package now to make sure it find rocsolver.

[cgmb]

Is the static build without solver_buildtime the default as that creates an invalid static library?

It's valid. It's just unlikely that you'd want a static hipblas and a dynamic rocsolver.

[cgmb]

I believe it's just rocsolver.dll.

[daineAMD]

Testing this out on a windows machine now.

[cgmb]

Please load librocsolver.so.0. librocsolver.so is only provided by the -dev/-devel package on Debian, Ubuntu and Fedora. If you want to fall back to librocsolver.so in the case that loading librocsolver.so.0 fails, that's fine.

The unsuffixed symlink goes in the -dev package on those distros because the maintainers wish to ensure that both librocsolver.so.0 and the hypothetical future librocsolver.so.1 can both be installed at the same time without having a conflict caused by both packages containing a file with the same name. The ability to install the two versions side-by-side makes for a smoother migration during the package update process.

[cgmb]

You may want to adjust the CMake that defines the package dependencies to recommend and/or suggest the rocsolver package to users that install hipblas.

[TorreZuk]

Is the static build without solver_buildtime the default as that creates an invalid static library?

It's valid. It's just unlikely that you'd want a static hipblas and a dynamic rocsolver.

Invalid in the sense that neither a customer nor developer would want it (we don't want failing tests for bad configs), let alone be default. Maybe a conditional option should be used?

[cgmb]

Invalid in the sense that neither a customer nor developer would want it (we don't want failing tests for bad configs), let alone be default. Maybe a conditional option should be used?

Keep in mind that we're not distinguishing between 'maybe rocsolver at runtime' and 'no rocsolver'. The configuration of 'static hipblas with no rocsolver' is something that developers may want.

[TorreZuk]

Invalid in the sense that neither a customer nor developer would want it (we don't want failing tests for bad configs), let alone be default. Maybe a conditional option should be used?

Keep in mind that we're not distinguishing between 'maybe rocsolver at runtime' and 'no rocsolver'. The configuration of 'static hipblas with no rocsolver' is something that developers may want.

The flags are confusing me maybe keeping -n --no-solver as never any solver is simpler (both shared and static). What this seems to be is an architeture option to delays .so loading to make it optional. Should discuss an overall roadmap feature request for this change for hip libraries.

[cgmb]

What prompted this change?

[daineAMD]

I believe this is needed to find the rocsolver library if hipblas is built with the install script --rocsolver-path option

[cgmb]

Keep in mind that rocsolver might not actually even exist on the system at build-time. Personally, I wouldn't bother mucking around with the RPATH, as it can always be handled after the build by setting the LD_LIBRARY_PATH.

If you prefer to set the RPATH, it should be guarded to only be set when rocsolver exists and --rocsolver-path has been used. The hardcoded lib may be a problem, as the library directory can sometimes have other names in third-party builds, like lib64 or <target-triple>/lib. You could use ${CMAKE_INSTALL_LIBDIR} instead, but I think it would be more robust to extract the path from the target:

if(DEFINED ROCSOLVER_PATH)
  find_package(rocsolver QUIET PATHS "${ROCSOLVER_PATH}" NO_DEFAULT_PATH)
  if(rocsolver_FOUND)
    get_target_property(rocsolver_location roc::rocsolver LOCATION)
    get_filename_component(rocsolver_directory ${rocsolver_location} DIRECTORY)
    set_property(TARGET hipblas APPEND PROPERTY BUILD_RPATH ${rocsolver_directory})
  endif()
endif()

With all that said, I prefer less code to more code. IMO, it's fine to expect users to set LD_LIBRARY_PATH if rocSOLVER is not in a system directory or /opt/rocm.