Migrate message_data.model.transport

[khaeru]

This pull request migrates MESSAGEix-Transport and its data/commit history from the private message_data repo.

After #208 is completed, approved, and merged into this branch, CI checks should pass and this PR can be approved and merged.

Preparation

The process roughly uses the migration process from the docs, starting from iiasa/message_data@fff251a4e3c2403f978e99d78e3b64108b357b83 (dev branch). I used the attached replacements.txt…

and the following settings in migrate.py:

#: Common args for calls to git-filter-repo in `target_repo`
COMMON = [
    "--force",  # Ignore non-clean history
    # "--debug",  # Show debug information
    "--prune-empty=always",
    # "--prune-degenerate=always",  # COMMENTED
]

…

# -- Configuration ---------------------------------------------------------------------
# Do not edit outside of this section. See the documentation for examples.

S = SOURCE = RepoInfo(
    url="[email protected]:iiasa/message_data.git",
    branch="dev",
)

T = TARGET = RepoInfo(
    url="[email protected]:iiasa/message-ix-models.git",
    branch="main",
)

MOD = "transport"

BATCH = (
    # Use --path-rename to rename several paths and files under them
    dict(
        args=[
            # Module docs
            f"--path-rename={S.base}/model/{MOD}/doc/:doc/{MOD}/",
            # Module data
            f"--path-rename=data/{MOD}/:{T.base}/data/{MOD}/",
            # Module code
            f"--path-rename={S.base}/model/{MOD}/:{T.base}/model/{MOD}/",
            f"--path-rename={S.base}/model/workflow.py:{T.base}/model/workflow.py",
            f"--path-rename={S.base}/projects/engage/workflow.py:{T.base}/project/engage/workflow.py",
            f"--path-rename={S.base}/projects/navigate/__init__.py:{T.base}/project/navigate/__init__.py",
            f"--path-rename={S.base}/projects/transport_futures.py:{T.base}/project/transport_futures.py",
            # Module tests
            f"--path-rename={S.base}/tests/model/{MOD}/cache/:{T.base}/tests/model/{MOD}/",
            f"--path-rename={S.base}/tests/model/{MOD}/:{T.base}/tests/model/{MOD}/",
            # Apply replacements
            "--replace-message=../replacements.txt",
        ],
        message_callback=message_callback,
    ),
    # Use --path to keep only a subset of files and directories
    dict(
        args=[
            "--path=.github/workflows/transport.yaml",
            f"--path=doc/{MOD}/",
            f"--path={T.base}",
        ]
    ),
    # Use --invert-paths to *remove* some specific files
    dict(
        args=[
            "--invert-paths",
            # Proprietary data
            f"--path={T.base}/data/transport/Energyefficiencyindicators_2020-extended.xlsx",
            f"--path={T.base}/data/transport/GEAM_TRP_techinput.xlsx",
            f"--path={T.base}/data/transport/R11/ldv-cost-efficiency.xlsx",
            f"--path={T.base}/data/transport/RoadmapResults_2017.xlsx",
        ],
    ),
)

…plus some other improvements that I will commit separately. This yields rebase-todo.txt. The rebase is performed with git rebase --interactive main, i.e. without --empty=drop. This is to keep the commits that link the history to the many PRs in message_data that developed the code.

How to review

• See Manual adjustments for .model.transport migration #208 and its reviews.
• Note that CI checks all pass.
• Mention any items to be addressed in follow-up issues and/or PRs.

PR checklist

• Continuous integration checks all ✅
• Add or expand tests; coverage checks both ✅
• Add, expand, or update documentation.
• Update doc/whatsnew.

[codecov]

Codecov Report

Attention: Patch coverage is 79.53911% with 293 lines in your changes missing coverage. Please review.

Project coverage is 58.8%. Comparing base (4d95695) to head (96c0418).
Report is 1879 commits behind head on main.

Files with missing lines	Patch %	Lines
message_ix_models/tools/messagev.py	20.4%	97 Missing ⚠️
message_ix_models/model/transport/report.py	48.9%	72 Missing ⚠️
message_ix_models/model/transport/plot.py	74.1%	32 Missing ⚠️
message_ix_models/model/transport/operator.py	79.3%	18 Missing ⚠️
message_ix_models/model/transport/workflow.py	78.9%	16 Missing ⚠️
...sage_ix_models/tests/model/transport/test_build.py	54.2%	16 Missing ⚠️
message_ix_models/model/transport/cli.py	40.0%	12 Missing ⚠️
message_ix_models/model/transport/build.py	91.0%	8 Missing ⚠️
message_ix_models/model/transport/ikarus.py	68.1%	7 Missing ⚠️
message_ix_models/model/transport/structure.py	89.5%	5 Missing ⚠️
... and 6 more

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #207     +/-   ##
=======================================
+ Coverage   52.3%   58.8%   +6.5%     
=======================================
  Files        145     194     +49     
  Lines      11357   15159   +3802     
=======================================
+ Hits        5943    8922   +2979     
- Misses      5414    6237    +823     

Files with missing lines	Coverage Δ
message_ix_models/cli.py	93.4% <ø> (ø)
message_ix_models/model/transport/CHN_IND.py	30.3% <100.0%> (ø)
message_ix_models/model/transport/__init__.py	100.0% <ø> (ø)
message_ix_models/model/transport/callback.py	28.5% <ø> (ø)
message_ix_models/model/transport/data.py	94.2% <100.0%> (ø)
message_ix_models/model/transport/demand.py	100.0% <100.0%> (ø)
message_ix_models/model/transport/emission.py	95.2% <100.0%> (ø)
message_ix_models/model/transport/factor.py	90.9% <100.0%> (ø)
message_ix_models/model/transport/freight.py	100.0% <100.0%> (ø)
message_ix_models/model/transport/groups.py	97.0% <100.0%> (ø)
... and 41 more

... and 3 files with indirect coverage changes

[khaeru]

The test coverage of the added code is about 80%; compare with .material and .water here at about 10% each.

[glatterf42]

Overall, this PR looks great to me :)
I've noted some changes I would like to see, most of which are fine in follow-up PRs, though. The one thing I would like to request is that we should really avoid importing from message_data with code in here.
Once that is done and no tests break because of it, I'll approve :)

[glatterf42]

Thanks for addressing my comments so quickly. With several follow-up issues opened and the intention to address them somewhat soon, I'm fine with approving this.

[khaeru]

Thanks for the review!