Collect pieces for SSP 2024/ScenarioMIP

[khaeru]

This PR is to track the working branch ssp-dev, which exists to help with work on the SSP 2024/ScenarioMIP project by collecting or pointing to all the code that is actually used to execute the workflow.

The goal is to be complete and up-to-date as possible for that purpose; other aspects (code quality, cleanliness, or readiness for main) are less important.

How to use

• Per Collect pieces for SSP 2024/ScenarioMIP workflows #234, it is not necessary to use this branch. If you can make an atomic PR for individual, small changes, do that first. This branch will be regularly rebased on latest main, so will include those updates.
• Update this branch, e.g. by cherry-picking commits or copying code from various other working branches used for this project.
• Make a PR that targets into this branch, and then request that someone review it. Continue work on this branch, rather than the PR source branch.
• DO NOT use the button “Update branch” → "Update with merge commit” to keep this branch up to date with main.
  • If necessary, use “Update branch” → “Update with rebase”, or ask in Slack for the branch to be rebased.

How to review

TBD: this PR may not eventually be merged.

PR checklist

• Continuous integration checks all ✅ TBD if this will be required.
• Add or expand tests; coverage checks both ✅
• Add, expand, or update documentation.
• Update doc/whatsnew.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 191 lines in your changes missing coverage. Please review.

Project coverage is 60.9%. Comparing base (0645fd5) to head (1a26e6c).

Files with missing lines	Patch %	Lines
...essage_ix_models/project/ssp/script/util/shares.py	0.0%	100 Missing ⚠️
...age_ix_models/project/ssp/script/util/functions.py	0.0%	91 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (0645fd5) and HEAD (1a26e6c). Click for more details.

HEAD has 78 uploads less than BASE

Flag	BASE (0645fd5)	HEAD (1a26e6c)
	87	9

Additional details and impacted files

@@           Coverage Diff            @@
##            main    #235      +/-   ##
========================================
- Coverage   76.5%   60.9%   -15.6%     
========================================
  Files        203     205       +2     
  Lines      15546   15737     +191     
========================================
- Hits       11896    9597    -2299     
- Misses      3650    6140    +2490     

Files with missing lines	Coverage Δ
...age_ix_models/project/ssp/script/util/functions.py	0.0% <0.0%> (ø)
...essage_ix_models/project/ssp/script/util/shares.py	0.0% <0.0%> (ø)

... and 53 files with indirect coverage changes

[khaeru]

@GamzeUnlu95 @macflo8 —there now appears to be a conflict in a particular materials data file. I think this is the same one modified in #247 (FYI also @glatterf42 who reviewed there).

Can you please say whether these two changes to the file are perhaps the same, or different? Does one supersede the other, or are they parallel/non-overlapping?

This will allow to rebase and update the branch, which we need to do (urgently) in order for it to be usable in the ScenarioMIP process.

[khaeru]

Can you please say whether these two changes to the file are perhaps the same, or different? Does one supersede the other, or are they parallel/non-overlapping?

The reply (in Slack) was that the changes in #247 supersede the ones that were on this branch.
Accordingly, I did:

• git switch ssp-dev && git pull --rebase update my local copy of the branch.
• git rebase -i main initiate interactive rebase.
• Chose edit for the commit with subject "Move co2_trans_disp parametrization to file" —the only commit on this branch that modified the file.
• Removed the changes to that file from the commit:

  git restore --staged message_ix_models/data/material/steel/Global_steel_MESSAGE.xlsx 
  git restore message_ix_models/data/material/steel/Global_steel_MESSAGE.xlsx 
  

• git rebase --continue
• git push --force

[khaeru]

@measrainsey @ywpratama I think there is probably an issue to be resolved here.

• In the earlier-merged commits on this branch, an additional column low_medium was added to this table, leading to 6 total columns with numeric values.
• In parallel, the 3 rows for dac_lt etc. were added with only 5 numeric values.
• Thus for instance the value 0.5 appears to be in the low_medium column, when the DAC-related commit added them to a medium column, and so on for the rest of the row.

I was able to deconflict the branch history, but the latter rows probably also need values (or NaNs) for low_medium inserted so that the entered values appear in the right place.

Please let me know if that's unclear or if you need help making the adjustment.

[measrainsey]

gotcha! @ywpratama could you add low_medium values for the DAC technologies? since i wouldn't know the assumptions to make here

[khaeru]

I see the branch is showing conflicts, so I will rebase manually and force-push.

[khaeru]

I see the branch is showing conflicts, so I will rebase manually and force-push.

Now done.

[macflo8]

I will drop my 3 most recent commits and force-push because they break the model build . The commits will be added later again when the industry revisions are fully functional.

[khaeru]

Rebased after merge of #255.

[khaeru]

• Rebased after merge of Transport improvements for 2024-W36 #225.
• Cherry-picked a single commit by @adrivinca from Change in LED setup after group feedback #261.

[khaeru]

Rebased after the merge of #258.

In doing so, there was a conflict in message_ix_models/data/costs/materials/tech_map.csv. I resolved this in favour of the version on main per that other PR, which is more recent.

FYI @measrainsey if you are making a PR into main and happen to notice that it touches files also modified on this branch, please feel free to cherry-pick particular commits from this branch into a smaller PR branch. That way, they can be reviewed and merged, and then they will vanish from this branch when it is later rebased. In the long run this can shrink the work we must eventually do to incorporate the contents of this branch in main.

[macflo8]

After clarifying with @ywpratama, we figured out that the last two commits were pushed accidentally by him. I will drop them now and force-push.

[khaeru]

I'm going to rebase the branch again on main to avoid the "Merge branch 'main' into ssp-dev" that we see at the end here. Should be done in about an hour; please no one touch the branch in the meantime.

This is now done. I've also updated the issue description to be clear that no one should use the "Update with merge" feature offered by GitHub.

@measrainsey @ywpratama in doing this rebase, I notice that an issue I raised 3 weeks ago is still present. Concretely, the file contains this (comments added):

message_technology,very_low,low,low_medium,medium,high,very_high  # 7 fields
…
wind_res4,0.15,0.3,0.53,0.53,0.65,0.75  # 7 fields
dac_lt,0.1,0.3,0.5,0.7,0.9  # 6 fields
dac_hte,0.1,0.3,0.5,0.7,0.9
dac_htg,0.1,0.3,0.5,0.7,0.9

or in other words:

message_technology	very_low	low	low_medium	medium	high	very_high
wind_res4	0.15	0.3	0.53	0.53	0.65	0.75
dac_lt	0.1	0.3	0.5	0.7	0.9	EMPTY
dac_hte	0.1	0.3	0.5	0.7	0.9	EMPTY
dac_htg	0.1	0.3	0.5	0.7	0.9	EMPTY

I don't know what .tools.costs does in the current case when the very_high column contains NaNs. In any case, my guess is that you actually intend something like:

message_technology	very_low	low	low_medium	medium	high	very_high
wind_res4	0.15	0.3	0.53	0.53	0.65	0.75
dac_lt	0.1	0.3	—???—	0.5	0.7	0.9
dac_hte	0.1	0.3	—???—	0.5	0.7	0.9
dac_htg	0.1	0.3	—???—	0.5	0.7	0.9

If, on the other hand, the former/current file contents is actually what should be there, please let me know and I will stop complaining about it. It might help to add a comment to the CSV file so others aren't confused in the future:

message_technology,very_low,low,low_medium,medium,high,very_high
…
wind_res4,0.15,0.3,0.53,0.53,0.65,0.75
#
# The following rows deliberately have NaNs in
# the 'very_high'  column because [reasons]
dac_lt,0.1,0.3,0.5,0.7,0.9,
dac_hte,0.1,0.3,0.5,0.7,0.9,
dac_htg,0.1,0.3,0.5,0.7,0.9,