Separate different libs for ray backend code.

[guoyuhong]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
• Ray installed from (source or binary):
• Ray version:
• Python version:
• Exact command to reproduce:

Describe the problem

We'd better separate the libs for backend code instead of put many c++ files cross different directories into one big library. Currently, we only have separate libs for object manager, raylet, raylet client. As the project becomes big and big, there could be more directories to represent different functionalities. We need to do this as early as possible.

One example is that, @micafan is adding the ray metric code in #3664. There is no separated libs for submodules, so she added the metric code as part of raylib. What she needs is the logging code in ray, but logging is not separated from raylib. Then the metric lib dependency is raylib, all the tests executables should be linked after raylib is finished. Actually, when she changed the metric code, only the metric libs should be built again, but current the whole raylib is built again and all the dependencies are build again, too.

Separating the libs will bring us following benefits:

1. Reducing unnecessary relinking after some code change (as the example showed).
2. Make several building processes parallel.
3. Make lib structure clearer.
4. Remove unnecessary dependencies.

I propose the following lib structure.

• src/ray/common: libray_common.a
• src/ray/gcs: libray_gcs.a
  • src/ray/gcs/redis_module: libray_redis_module.so
• src/ray/object_manager: libray_object_manager.a
• src/ray/raylet: libraylet.a
• src/ray/util: libray_util.a
• id.cc/status.cc: libray_basics.a or to libray_common.a?

How do you think about this change?

Source code / logs

[micafan]

@guoyuhong Awesome. This feature has been implemented in bazel, you can refer to how the library in bazel is split(BUILD.bazel).

[guoyuhong]

@micafan Thanks for the information. I will implement the cmake change accordingly.

[robertnishihara]

@guoyuhong let's not spend too much time making changes to cmake since it should all be replaced by bazel soon (targeting next two weeks). I'd suggest making any improvements directly to the bazel build.

[raulchen]

@robertnishihara Didn't know bazel would come this soon. Is @pcmoritz driving the effort right now?

[robertnishihara]

Two weeks is on the optimistic side, but yes, @pcmoritz is driving the effort.

[guoyuhong]

I will close this issue since bazel building meets the requirement and bazel will replace cmake soon.