Add symlinking ability to lbuild

[asmfreak]

I'm developing library.
I'd like to have ability to symlink files instead of copying for debug. Something in lines of:

def build(env):
    env.outbasepath = "some/dir/to/output"
    env.link("some_file.hpp")

I can achieve this behaviour using the following script, but it's way too clunky:

def build(env):
    env.outbasepath = "some/dir/to/output"
    for fn in ["some_file.hpp"]:
        ofn = env.real_outpath(fn)
        env.copy(fn)
        os.unlink(ofn)
        os.symlink(env.localpath(fn), ofn)

[salkinium]

Yeah, this came up a few times already, there's already a branch for that: https://github.com/modm-io/lbuild/tree/feature/symlink
This allows lbuild build --symlink, which symlinks every env.copy, but not env.template or env.archive for obvious reasons.

I hesitate to merge this, because it trains people to change the generated files. That's fine as long as you know which files are copied/linked and which are generated. The time will come when you make changes to a generated file and it gets overwritten.

Is there a use-case for exposing symlinking via env.link in lbuild? The issue is that it creates a dependency on the original lbuild repo to be available, which isn't something we want.

[salkinium]

The time will come when you make changes to a generated file and it gets overwritten.

Hm, on the other hand, there are a few issues open on tracking the content of the generated files (with hashing) and to generate warnings about what files were modified and are about to be overwritten: like #33.

With that I would be more comfortable to add this features.

[salkinium]

Let me know if this makes your life easier, and i'll just merge it with a big fat warning.
You can install this branch via:

pip3 uninstall lbuild
pip3 install git+https://github.com/modm-io/lbuild@feature/symlink

[asmfreak]

Is this possible on per-module basis?

[asmfreak]

Something in lines of:

def init(module):
    # ...
    module.symlink_without_copy = True
    # ...

This is false by default (if not with --symlink)

[salkinium]

Hm, definitely not a fan of changing the module API, I was thinking more of adding module names via lbuild build --symlink repo:module1 --symlink repo:module2:submodule etc, like lbuild build -m repo:module works today.

Can you explain the use-case for this? Is there an issue with symlinking all modules? Why just a few modules?

[asmfreak]

I thought of opt-in approach for debug purposes only.
I can't think of any issues with symlinking all modules.

[asmfreak]

I tried enabling symlinking in my modm project.
Well, it didn't went too well:

Compiling C++·· build/release/modm/src/modm/io/iostream.o
<REDACTED>/modm/src/modm/io/iostream.cpp:16:10: fatal error: iostream.hpp: No such file or directory
   16 | #include "iostream.hpp"
      |          ^~~~~~~~~~~~~~
compilation terminated.

[salkinium]

Is this an issue with using symlinks with GCC on Windows/NTFS?

[asmfreak]

Mixing copied and templated code does not work.
I'm using Arch Linux, arm g++ 9.2.0.
If I understand correctly, compiler is deducing current directory of the original file, but not symlink. It fails to find templated code in the source directory.

[salkinium]

I can reproduce this, why is GCC following symlinks now? I'm still on ARM GCC v8.3.1. Is there a flag we can pass to GCC?

[salkinium]

Ok, so it follows the symlink to the real file, which is in the modm repository, but of course the include file isn't there because it's a template iostream.hpp.in. Unless there is a flag to turn this behavior off, this feature cannot work at all.

[salkinium]

So… this issue is dead, right? Or did you find any ways that you can make this work for GCC?

[asmfreak]

Wait. GCC is not the culprit here. It's actually lbuild.
This is what lbuild generated in modm/SConscipt:

files = [
    env.File(r"../../../modm/src/modm/architecture/interface/can.cpp"),
    env.File(r"../../../modm/src/modm/architecture/interface/can_message.cpp"),
    env.File(r"../../../modm/src/modm/debug/error_report.cpp"),
    env.File(r"../../../modm/src/modm/debug/logger/hosted/default_style.cpp"),
    env.File(r"../../../modm/src/modm/driver/io/terminal.cpp"),
    env.File(r"../../../modm/src/modm/io/iostream.cpp"),
    env.File(r"src/modm/io/iostream_printf.cpp"),
    env.File(r"../../../modm/src/modm/math/utils/bit_operation.cpp"),
    env.File(r"../../../modm/src/modm/math/utils/pc/operator.cpp"),
    env.File(r"src/modm/platform/core/assert.cpp"),
    env.File(r"src/modm/platform/core/clock.cpp"),
    env.File(r"../../../modm/src/modm/platform/core/hosted/memory.cpp"),
    env.File(r"../../../modm/src/modm/utils/dummy.cpp"),
]

The ../../../ - is where the actual modm source repo is located.
No wonder I got those include errors.

[asmfreak]

What can be different from your setup?
I have:

$ lbuild --version
lbuild 1.12.4
$ python --version
Python 3.7.4

[asmfreak]

And this is the real culprit (os.path.realpath):

    def reloutpath(self, path, relative=None):
        #  ....
        return os.path.relpath(os.path.realpath(path), os.path.realpath(relative))

which is called from here in modm:

        env.template("resources/SConscript.in", "SConscript",
                     filters={"flags_format": flags_format,
                              "relocate": lambda p: env.relative_outpath(p, repo)})

which was called from template:

    env.File(r"{{ file | relocate | modm.windowsify(escape_level=0) }}"),

[asmfreak]

So symlinks are actually a viable solution, if this issue with relocate can be somehow resolved in lbuild or modm

[salkinium]

Oh, that's bad! I don't know why I used realpath there, this may not be intended behavior.

[salkinium]

I fixed that issue in the lbuild branch and this fixes compilation for me! So I'll open a PR with the feature.