Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate message_data.model.transport #207

Merged
merged 1,567 commits into from
Aug 1, 2024
Merged
Show file tree
Hide file tree
Changes from 250 commits
Commits
Show all changes
1567 commits
Select commit Hold shift + click to select a range
6639eef
Handle model-specific years in .ldv.prepare_computer()
khaeru Feb 6, 2024
3cf4a83
Move common transport keys to a common module
khaeru Feb 6, 2024
9068e73
Add docstrings to some .transport.key
khaeru Feb 6, 2024
400eeec
Test .transport.files using more complete r12 set
khaeru Feb 6, 2024
51fc544
Correct expected dims of ldv-new-capacity.csv
khaeru Feb 6, 2024
859c559
Add "transport other .*" technologies
khaeru Feb 7, 2024
e35fc29
Add transport.test_build.test_debug()
khaeru Feb 7, 2024
0976bac
Add data/transport/r12/energy-other.csv
khaeru Feb 7, 2024
0f2fba9
Specify "other transport" techs explicitly
khaeru Feb 7, 2024
c321d33
Add config for preparing energy-other.csv from iea eweb
khaeru Feb 7, 2024
0395e08
Correct input commodity of "transport other gas"
khaeru Feb 15, 2024
60533b4
Add energy-other.csv to .transport.files
khaeru Feb 15, 2024
6b9967d
Add .transport.non_ldv.other()
khaeru Feb 15, 2024
7362689
Improve output of .transport.test_build.test_debug()
khaeru Feb 15, 2024
9f71c00
Apply updated data for r12_chn per @r-aneeque / iiasa/message_data#538
khaeru Feb 15, 2024
86302aa
Align dims in energy-other.csv, .files, .non_ldv
khaeru Feb 15, 2024
60b79f0
Refresh transport docs
khaeru Feb 15, 2024
6fce642
Transcribe source info from iiasa/message_data#538 to docs
khaeru Feb 15, 2024
8794d97
Adjust data/transport/r12/mode-share/default.csv
khaeru Feb 15, 2024
500ad1b
Adjust data/transport/r12/ldv-activity.csv
khaeru Feb 15, 2024
1a0f391
Use message-ix-models main branch in "transport" ci workflow
khaeru Feb 15, 2024
f585f27
Proof .non_ldv.other() usage against incomplete data
khaeru Feb 15, 2024
06f4aee
Adjust transport tests for iiasa/message_data#535
khaeru Feb 15, 2024
dd478ff
Add coal, fueloil to .transport.data.dummy_supply()
khaeru Feb 15, 2024
a43e9dd
Merge pull request iiasa/message_data#535 from iiasa/transport/2024-w06
khaeru Feb 15, 2024
72614de
Distinguish artifact names in "transport" ci workflow
khaeru Feb 16, 2024
6fc7f01
Add "merge" job to "transport" ci workflow
khaeru Feb 16, 2024
8381e30
Add ethanol to .transport.data.dummy_supply()
khaeru Feb 16, 2024
d321c68
Assign data for t=drail_pub from ikarus source
khaeru Feb 16, 2024
8ba22cb
Add .transport.non_ldv.bound_activity_lo()
khaeru Feb 16, 2024
621f1cc
Give iis when test_build_bare_res() fails to solve
khaeru Feb 16, 2024
49bee45
Use genno.keyseq in .transport.base
khaeru Feb 19, 2024
3f9b0e4
Add minimum activity values to data/transport/r12/config.yaml
khaeru Feb 19, 2024
d96c329
Read minimum activity values to .transport.config
khaeru Feb 19, 2024
2c5316b
Read values from config in .non_ldv.bound_activity_lo
khaeru Feb 19, 2024
3284af7
Temporarily run only ssp2 in "transport" ci workflow
khaeru Feb 19, 2024
30b08f2
Add 5 values for minimum activity in r12
khaeru Feb 20, 2024
1a0797f
Adjust test_get_non_ldv_data
khaeru Feb 20, 2024
da1d819
Hand-tune minimum activity values for r12
khaeru Feb 20, 2024
e292bb3
Add a case to test_build.test_debug()
khaeru Feb 20, 2024
0396dcc
Merge pull request iiasa/message_data#542 from iiasa/transport/2024-w07
khaeru Feb 20, 2024
5e9151d
Bump webfactory/ssh-agent v0.8.0 → v0.9.0
khaeru Feb 22, 2024
fc4a2f2
Merge pull request iiasa/message_data#533 from iiasa/enh/support-pyth…
khaeru Feb 22, 2024
f921e0d
Store a version of r12/energy-other.csv with pipeline included
khaeru Feb 21, 2024
16646a3
Add .transport.operator.where()
khaeru Feb 21, 2024
5ddfb0a
Use "other" as a target name in .groups_iea_eweb()
khaeru Feb 21, 2024
5875540
Use where, not clip, on stage 1 scale factor
khaeru Feb 21, 2024
2a6df18
Set submodules: false for checkout in "transport" ci workflow
khaeru Feb 21, 2024
4b8eaf0
Force upgrade matplotlib in "transport" ci workflow
khaeru Feb 21, 2024
bf35b4c
Add sums to cap:*:non-ldv+units
khaeru Feb 22, 2024
cfd42b3
Place transport plots in url-specific subdir
khaeru Feb 22, 2024
3e424fe
Show scenario url in transport plot subtitle
khaeru Feb 22, 2024
900ce97
Add .transport.plot.energycmdty1
khaeru Feb 22, 2024
a01f3cd
Reenable ssp1 and 3–5 in "transport" ci workflow
khaeru Feb 22, 2024
2e1fa88
Set gams/cplex time limit in "transport" workflow
khaeru Feb 22, 2024
e5101aa
Adjust where() condition for scale-1
khaeru Feb 22, 2024
2644fd6
Update r12/ldv-new-capacity.csv with 2005 values
khaeru Feb 27, 2024
3e420f0
Fix grouping in {fix,var}-cost transport plots
khaeru Feb 27, 2024
2e2658f
Quiet matplotlib logging in .transport.plot
khaeru Feb 27, 2024
0278447
Add cgs from spec, not build info, in add_structure()
khaeru Feb 27, 2024
3287620
Select out:*:ldv and apply units
khaeru Feb 27, 2024
3693812
Refresh "out-share-t{,-cg}-ldv transport plots
khaeru Feb 27, 2024
84167c1
Select historical_new_capacity:*:ldv
khaeru Feb 27, 2024
2952d14
Add cap-new-t-ldv transport plot
khaeru Feb 27, 2024
b68e4e9
Tidy report internals
khaeru Feb 27, 2024
c39e6f4
Refresh demand{,-share} transport plots
khaeru Feb 27, 2024
eedc876
Move {apply,assign}_units to .transport.report
khaeru Feb 27, 2024
0d027f2
Remove transport/report.yaml
khaeru Feb 28, 2024
26feb33
Type hint .transport.operator.make_output_path()
khaeru Feb 28, 2024
e0f83b7
Add reporting labels for all ldv technologies
khaeru Feb 28, 2024
c5fca45
Relax growth_activity_up for ldv technologies
khaeru Feb 28, 2024
51f2039
Use computer class, not str, to type add_iamc_store_write()
khaeru Feb 28, 2024
7fddfad
Simplify path construction in .transport.report
khaeru Feb 28, 2024
75f3861
Allow missing scenario in plot.ggtitle()
khaeru Feb 28, 2024
4a9dd9c
Use "sum" str in .aggregate() to silence pandas warning
khaeru Mar 1, 2024
3ca815c
Add plots across multiple scenarios
khaeru Mar 1, 2024
1c98ceb
Address pandas warning in .transport.operator
khaeru Mar 1, 2024
b40417e
Adjust test_report_bare_solved()
khaeru Mar 1, 2024
7fe598a
Merge pull request iiasa/message_data#543 from iiasa/transport/2024-w08
khaeru Mar 1, 2024
55d3df3
Combine transport gen-activity/debug with "run"
khaeru Mar 5, 2024
e7a4677
Exclude 2020 from "pdt non-active" ssp factor
khaeru Mar 5, 2024
6706c8d
Specify year for .transport.config.fixed_gdp
khaeru Mar 5, 2024
3be3de6
Use keys for pdt per capita check
khaeru Mar 5, 2024
8c3b9ff
Update .transport.cli docstring and docs
khaeru Mar 5, 2024
9575483
Use gha-compatible paths in add_debug()
khaeru Mar 6, 2024
7624f1b
Output processed iea eweb data in add_debug()
khaeru Mar 6, 2024
0476ae4
Simplify add_debug()
khaeru Mar 6, 2024
c33e01f
Also generate debug files when building transport
khaeru Mar 7, 2024
074e884
Update, document share_weight()
khaeru Mar 8, 2024
cd688ec
Use python operators for genno.operator.{div,mul}
khaeru Mar 8, 2024
77e842e
Delete share weight convergence for r12_nam
khaeru Mar 8, 2024
699f7c5
Add plot demand-exo-capita-gdp.pdf
khaeru Mar 8, 2024
ea96d85
Always add all plots from .transport.plot
khaeru Mar 8, 2024
200636c
Include demand-exo-capita-gdp in build debug
khaeru Mar 8, 2024
3829f18
Add gdp-ppp-cap.csv to transport build debug outputs
khaeru Mar 11, 2024
b0febc4
Add .transport.plot.comparepdtcap1
khaeru Mar 11, 2024
4a0dc68
Allow transport build debug w/o base scenario
khaeru Mar 11, 2024
6ea0898
Write transport debug plots in per-scenario directory
khaeru Mar 11, 2024
ca3ddfb
Move .transport.emission tests to parallel module
khaeru Mar 11, 2024
6685e98
Improve ef_for_input()
khaeru Mar 11, 2024
3047e8a
Ignore non-numeric values in sum in build.main()
khaeru Mar 12, 2024
e2bb377
Return an exact set of columns from ef_for_input()
khaeru Mar 12, 2024
671660c
Simplify base_model_data_header() with utils
khaeru Mar 12, 2024
463cc36
Simplify usage of ef_for_input
khaeru Mar 12, 2024
b33b719
Don't write an svg graph in testing.configure_build()
khaeru Mar 12, 2024
d5bf1cd
Improve use of temp platform/scenario in transport workflow
khaeru Mar 12, 2024
f4f394b
Update tests of "debug build" cli usage
khaeru Mar 12, 2024
0913dd9
Apply `ruff format` v0.3.2 to 33 files (whitespace only)
khaeru Mar 12, 2024
7c10671
Shift all ci jobs to self-hosted runner
khaeru Mar 12, 2024
2a4f450
Adjust "pytest" workflow for self-hosted runner
khaeru Mar 12, 2024
9be6d7e
Make _to_csv(…, write_kw=…) optional
khaeru Mar 18, 2024
b6687fd
Consolidate checks for base scenario w/ solution
khaeru Mar 18, 2024
db10bd0
Mark demandexocap1 as a build-stage plot
khaeru Mar 18, 2024
6cc3d8f
Simplify .transport.data.mertoppp
khaeru Mar 18, 2024
5ec6fea
Add .non_ldv.growth_new_capacity() and setting
khaeru Mar 19, 2024
97ad76e
Use non-interactive matplotlib in workflow
khaeru Mar 19, 2024
b04112d
Add initial_new_capacity_up, growth_activity_lo for non-ldvs
khaeru Mar 19, 2024
6866257
Allow "build debug" --from="ssp.* debug build"
khaeru Mar 21, 2024
570cbed
Add (commented) config for debug build
khaeru Mar 21, 2024
6a92be0
Override non-ldv constraints for c=electr techs
khaeru Mar 21, 2024
ab770d5
Report bound_activity_lo-projected.csv
khaeru Mar 21, 2024
62bfa83
Blend scale-1 factor to 1.0 in 2050
khaeru Mar 21, 2024
d3d172a
Adjust typing for genno 1.25.0
khaeru Mar 26, 2024
91b4547
Update .transport.operator.votm() for latest genno
khaeru Mar 25, 2024
1ef95e2
Update test_cli for iiasa/message-ix-models#156
khaeru Mar 26, 2024
e94c616
Adjust expected non-ldv data in tests
khaeru Mar 27, 2024
6ea086f
Temporarily skip 2 transport tests
khaeru Mar 27, 2024
e4a06b5
Merge pull request iiasa/message_data#545 from iiasa/transport/2024-w10
khaeru Mar 27, 2024
77a183b
Use "mix-models fetch" in ci workflows
khaeru Apr 2, 2024
e4eddf9
Use message-ix-models `main` in ci workflows
khaeru Apr 2, 2024
db1ed84
Merge pull request iiasa/message_data#548 from iiasa/fix/ssp-pooch
khaeru Apr 2, 2024
7a87eba
Remove hard-coded structure in dummy_supply()
khaeru Apr 2, 2024
80b7bc6
Remove completed todos in .transport.workflow.generate()
khaeru Apr 2, 2024
71fd0a8
Simplify get_techs()
khaeru Apr 2, 2024
4713a71
Remove skip and xfail marks in transport tests
khaeru Apr 2, 2024
d8cb7ea
Replace "transport build info" context key
khaeru Apr 2, 2024
3347629
Replace "transport spec" context key
khaeru Apr 2, 2024
737f87c
Collect model structure code in .transport.structure
khaeru Apr 2, 2024
ec5d32e
Remove .transport.config.set attribute
khaeru Apr 2, 2024
f2eca84
Adjust use of .config.{set,spec}
khaeru Apr 2, 2024
1444e3a
Update .ldv.usage_data()
khaeru Apr 2, 2024
603562a
Update test_build
khaeru Apr 2, 2024
96bb972
Type hint get_ikarus_data()
khaeru Apr 2, 2024
213ac7c
Use genno.keyseq in .ikarus.prepare_computer()
khaeru Apr 2, 2024
99119f4
Exclude plots in .transport.testing.configure_build
khaeru Apr 2, 2024
68c1420
Sort attributes in .transport.config
khaeru Apr 2, 2024
1cd6472
Format and comment gfei data
khaeru Apr 3, 2024
dbdf979
Remove .tools.gfei_fuel_economy
khaeru Apr 3, 2024
24b3c5e
Allow context.set_scenario() to update url
khaeru Apr 3, 2024
e0d3485
Remove todo in .transport.workflow.generate()
khaeru Apr 3, 2024
6edea03
Remove .projects.shape, .tools.{gea,iea_eei}
khaeru Apr 3, 2024
1c20995
Annotate ldv techs with 'gfei-2017-technology'
khaeru Apr 8, 2024
03ac201
Install dask[dataframe] via message-ix-models[iea-web]
khaeru Apr 10, 2024
ded13fa
Fix use of "transport iamc" keys in "transport all"
khaeru Apr 11, 2024
bb1e785
Inline static data clone in "transport" ci workflow
khaeru Apr 11, 2024
4c1c161
Disable debug quantities in iamc structured report
khaeru Apr 11, 2024
c8561eb
Update r12/ldv-activity.csv per @th-hara calcs
khaeru Apr 16, 2024
e725c89
Update .operator.distance_nonldv() for iiasa/message-ix-models#162
khaeru Apr 16, 2024
9d4f1ef
Add "plot base-fe-intensity-gdp" and file
khaeru Apr 16, 2024
2fa55be
Increase .config.fixed_gdp 300 → 1000 kusd_2005/a
khaeru Apr 18, 2024
815bde9
Xfail gea and shape cases for "debug build" cli
khaeru Apr 18, 2024
ae59a95
Update test_iamc_simulated
khaeru Apr 18, 2024
de20b39
Update docs for iiasa/message_data#549
khaeru Apr 18, 2024
4ed330f
Exclude import from mypy for iiasa/message_data#549
khaeru Apr 19, 2024
fe1c915
Temporarily xfail test_iamc_simulated
khaeru Apr 19, 2024
688eb7b
Merge pull request iiasa/message_data#549 from iiasa/transport/2024-w14
khaeru Apr 20, 2024
d02fe0b
Revise share_weight() handling for ref regions
khaeru Apr 22, 2024
b86239c
Write units header for transport build debug csvs
khaeru Apr 22, 2024
bc9e42d
Skip comments in .plot.read_csvs()
khaeru Apr 23, 2024
1d221d9
Document computations in .demand
khaeru Apr 23, 2024
3b16f0e
Satisfy mypy with pandas-stubs installed
khaeru Apr 23, 2024
5079c81
Compute "calibrate-fe.csv"
khaeru Apr 23, 2024
b29f9db
Rename ldv-input-ref.csv
khaeru Apr 24, 2024
b05ea3b
Construct ldv-input-adj.csv
khaeru Apr 24, 2024
d247e23
Add/use exogenousdatafile.add_tasks()
khaeru Apr 26, 2024
8207e7f
Reference `config` in add_exogenous_data()
khaeru Apr 26, 2024
6beb0dc
Simplify construction of "n:n:ex world"
khaeru Apr 26, 2024
505e4b9
Rename config.fixed_demand → .fixed_pdt
khaeru Apr 26, 2024
d710d71
Add .demand.pdt_per_capita
khaeru Apr 26, 2024
e5b6ed8
Move tests of pdt_per_capita()
khaeru Apr 26, 2024
8647e09
Add .transport.config.policy
khaeru Apr 27, 2024
e79e24f
Add "transport run --base=[auto|bare]" cli option
khaeru Apr 27, 2024
62aa0c6
Add data/transport/base-scenario-url.json
khaeru Apr 27, 2024
6cf1f6c
Improve .transport.workflow
khaeru Apr 27, 2024
1c5f27f
Parametrize "transport" ci workflow on policy setting
khaeru Apr 27, 2024
d193ad5
Remove cli default for --scenario-extra
khaeru Apr 27, 2024
ec247c3
Reverse-transform adjusted gdp/capita
khaeru Apr 29, 2024
48486b8
Use adjusted gdp/capita in mode share calcs
khaeru Apr 29, 2024
0c4aebc
Pin "transport/report" ci jobs to gh-runner-l2
khaeru Apr 29, 2024
ef5fa4d
Update "usage" in transport docs
khaeru Apr 29, 2024
04fdf26
Update transport/doc/input.rst
khaeru Apr 29, 2024
b8b71bd
Pin "transport/report" ci jobs to iiasa-self-hosted-l2
khaeru Apr 29, 2024
a0ec467
Omit both 2020 *and* 2025 from ssp factors
khaeru May 1, 2024
9a36ede
Update doc reference to ldv input files
khaeru May 1, 2024
b063d94
Add data/transport/r12/speed.csv
khaeru May 1, 2024
3409d76
Add "indexers:scenario" via .build.add_structure()
khaeru May 1, 2024
6aee316
Interpolate, select, and use speed from .csv
khaeru May 1, 2024
95cc303
Add (scenario, y) dimensions to load-factor-ldv.csv
khaeru May 1, 2024
65d60a6
Interpolate and select load-factor-ldv.csv
khaeru May 1, 2024
186d362
Apply ldv-input-adj.csv to ldv input intensity
khaeru May 1, 2024
03fd36e
Update share weight convergence in r12/config.yaml
r-aneeque May 17, 2024
66bf77c
Update pdt-elasticity.csv
r-aneeque May 17, 2024
f235e38
Add mode speed differentiation
r-aneeque May 17, 2024
e6b32bc
Update demand.py
r-aneeque May 17, 2024
30a0138
Update transport config.fixed_{gdp,pdt}
r-aneeque May 17, 2024
a6d32fc
Update speed.csv
r-aneeque May 17, 2024
16c2bfd
Update pdt-elasticity.csv
r-aneeque May 17, 2024
ab9dab7
Update ssp base model and policy scenarios for transport
r-aneeque May 21, 2024
d5ebfc3
Remove 1100 and replace with 2100
r-aneeque May 21, 2024
13ad51c
Fix spelling of 'dimensionless'
r-aneeque May 21, 2024
68e7103
Set explicit lpmethod=4, scaind=1 in .transport.workflow
r-aneeque May 22, 2024
1d18231
Update transport base scenarios to ssp(2024) v0.14
r-aneeque Jun 5, 2024
a914ae6
Set growth_activity_up for aviation technologies
khaeru Jun 6, 2024
2c32f42
Extend dynamic constraints for non-ldv techs
khaeru Jun 7, 2024
a2dacaa
Update base year pdt values
r-aneeque Jun 7, 2024
ab24fbe
Tidy formatting in r12/{pdt-elasticity,speed}.csv
khaeru Jul 1, 2024
ab0a2b4
Add placeholder transport data
khaeru Jul 2, 2024
f63a3e1
Add r12/speed.csv data for ssp(2017).2
khaeru Jul 2, 2024
dad2827
Change default transport.config.ssp to ssp(2024).2
khaeru Jul 2, 2024
a66f36f
Adjust transport tests, marks for iiasa/message_data#551
khaeru Jul 2, 2024
fc37472
Merge pull request iiasa/message_data#551 from iiasa/transport/2024-w17
khaeru Jul 2, 2024
c9335ef
Output ue-share.csv from transport
khaeru Jul 2, 2024
cd30b8a
Replace hard-coded 2020 with y₀
khaeru Jul 19, 2024
264f901
Output ue-share-{max,min}.csv from transport
khaeru Jul 19, 2024
9509ba2
Add .transport.operator.{max,min}
khaeru Jul 19, 2024
11119dc
'smooth' transport demand for base model
khaeru Jul 25, 2024
a713c88
Satisfy mypy 1.11 in .transport
khaeru Jul 26, 2024
0cd2675
Merge pull request iiasa/message_data#560 from iiasa/transport/2024-w27
khaeru Jul 26, 2024
dc37a89
Adjust first-party imports in .model.transport
khaeru Jul 29, 2024
a9b2570
Expand branch pattern for "pytest" CI workflow
khaeru Jul 29, 2024
828cd22
Migrate message_data.tools.messagev
khaeru Jul 30, 2024
e094104
Migrate message_data.testing.assert_units
khaeru Jul 30, 2024
bb24906
Adjust first-party imports in .model.transport tests
khaeru Jul 29, 2024
0e232c5
Use transport data/config files from message_ix_models
khaeru Jul 30, 2024
2f96dd3
Migrate pytest markers and options from message_data
khaeru Jul 30, 2024
199599a
Add .transport to .cli.submodules
khaeru Jul 30, 2024
931d3be
Mark tests that fail without non-public data
khaeru Jul 30, 2024
ccbe284
Adjust test marks for #207
khaeru Jul 30, 2024
a6d4fd5
Adjust docs for #207
khaeru Jul 30, 2024
58371b0
Add transport to doc/index
khaeru Jul 29, 2024
4cebc8c
Migrate pyproject.toml entries for .model.transport
khaeru Jul 30, 2024
15282c0
Isolate transport reporting tests
khaeru Jul 30, 2024
dbed4f0
Mark minimum message_ix versions for .transport
khaeru Jul 30, 2024
88a2120
Also xfail on RuntimeError in minimum_version()
khaeru Jul 30, 2024
0ed2260
Type .base.smooth() with Computer class, not str
khaeru Jul 30, 2024
cd2ddce
Add .util.graphviz.HAS_GRAPHVIZ
khaeru Jul 31, 2024
67aabf2
Mark tests that fail w/o graphviz on GHA/macos-13
khaeru Jul 31, 2024
ab0686d
Add #207, #208 to doc/whatsnew
khaeru Jul 29, 2024
f41823a
Merge pull request #208 from iiasa/migrate/transport-adj
khaeru Jul 31, 2024
96c0418
Use GitHub secrets in "transport" CI workflow
khaeru Aug 1, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/pytest.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ name: Test

on:
pull_request:
branches: [ main , "migrate-*"]
branches: [ main , "migrate**"]
schedule:
# 05:00 UTC = 06:00 CET = 07:00 CEST
- cron: "0 5 * * *"
Expand Down Expand Up @@ -123,7 +123,7 @@ jobs:
- name: Run test suite using pytest
run: |
pytest message_ix_models \
-m "not snapshot" \
-m "not (ece_db or snapshot)" \
-rA --verbose --color=yes --durations=20 \
--cov-report=term-missing --cov-report=xml \
--numprocesses=auto \
Expand Down
212 changes: 212 additions & 0 deletions .github/workflows/transport.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,212 @@
name: MESSAGEix-Transport

env:
# Starting point of the workflow.
# Use this value to build from a certain scenario:
# base: --url="ixmp://ixmp-dev/MESSAGEix-GLOBIOM 1.1-R12/baseline_DEFAULT#21"
# Use this value to allow the workflow to determine model & scenario names
# and versions:
base: --platform=ixmp-dev

# Set this to a particular step to truncate the workflow
from-step: ""

on:
# Uncomment these lines for debugging, but leave them commented on 'main'/'dev'
# pull_request:
# branches: [ main, dev ]
# push:
# branches: [ main, dev ]
schedule:
- cron: "0 1 * * *" # 01:00 UTC = 02:00 CET = 03:00 CEST
workflow_dispatch: {}

# Cancel previous runs that have not completed
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true

jobs:
report:
name: MESSAGEix-Transport ${{ matrix.SSP }}${{ matrix.policy }}
# Distribute jobs across any available self-hosted runner(s)
# runs-on: self-hosted
# Use one or a few specific runner(s)
runs-on: [ self-hosted, iiasa-self-hosted-l2 ]

strategy:
matrix:
# These correspond to short labels used for workflow step names
SSP:
- SSP1
- SSP2
- SSP3
- SSP4
- SSP5
policy:
- " policy"
- ""

steps:
- name: Clean workspace
run: |
rm -rf ${{ github.workspace }}/report/*

- uses: webfactory/[email protected]
with:
ssh-private-key: |
${{ secrets.MESSAGE_BUILDINGS_PRIVATE_KEY }}
${{ secrets.MESSAGE_TRADE_PRIVATE_KEY }}
${{ secrets.NAVIGATE_WORKFLOW_PRIVATE_KEY }}

- uses: actions/checkout@v4
with:
path: message_data
lfs: true
# On GitHub Actions
# submodules: true
# On self-hosted runners
submodules: false

# NB Needed for self-hosted runner, not for GitHub
- name: Check out Git LFS files
working-directory: message_data
run: git lfs checkout

# Other repositories required for testing
- name: Check out message-static-data
uses: actions/checkout@v4
with:
repository: iiasa/message-static-data
token: ${{ secrets.MESSAGE_STATIC_DATA_PAT }}
path: message-static-data
lfs: true
# Only check out the following files to limit bandwidth usage.
# Respectively
# 1. IEA Extended World Energy Balances, OECD source, 2022 edition
sparse-checkout: |
iea/372f7e29-en.zip
sparse-checkout-cone-mode: false

- name: Connect static data to message_data
run: cp -rsv $(realpath message-static-data)/* message_data/data/

- uses: actions/checkout@v4
with:
repository: iiasa/MESSAGE_Buildings
path: buildings
ssh-key: ${{ secrets.MESSAGE_BUILDINGS_PRIVATE_KEY }}

# NB Needed for self-hosted runner, not for GitHub
- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: "17"

- uses: actions/setup-python@v5
with:
# Latest version testable on GitHub Actions
python-version: "3.12"
cache: pip
cache-dependency-path: "**/pyproject.toml"

- uses: iiasa/actions/setup-gams@main
with:
version: 35.2.0
license: ${{ secrets.GAMS_LICENSE }}

# NB the IIASA-hosted runner does not have permissions to do "sudo apt-get",
# so the following doesn't work. Rather, the necessary packages
# (graphviz libgraphviz-dev pkg-config) are manually installed on the
# host system
# - uses: ts-graphviz/setup-graphviz@v2

# By default, the next step installs dependencies from PyPI.
# To run against unreleased code (on `main`, or other branches for open
# PRs), uncomment or modify the lines below as appropriate.
# - genno and message-ix-models are released frequently and *should* be
# installed from PyPI, with only temporary exceptions.
# - ixmp and message-ix are released less frequently and *may* be installed
# from `main`
- name: Force upgrade to unreleased dependencies
run: |
# pip install --upgrade "genno @ git+https://github.com/khaeru/genno.git@main"
pip install --upgrade "ixmp @ git+https://github.com/iiasa/ixmp.git@main"
pip install --upgrade "message_ix @ git+https://github.com/iiasa/message_ix.git@main"
# Temporary, pending a release that includes iiasa/message-ix-models#161, #166
pip install --upgrade "message-ix-models[iea-web] @ git+https://github.com/iiasa/message-ix-models.git@main"

# Upgrade matplotlib
# pyam pins matplotlib < 3.7.1, which conflicts with plotnine per
# https://github.com/has2k1/plotnine/pull/751. Override.
pip install --upgrade matplotlib

- name: Install package; upgrade dependencies
working-directory: message_data
run: pip install --upgrade --upgrade-strategy=eager --editable .[dl,scgen,tests,transport]

- name: Configure
run: |
from pathlib import Path

# Ensure keys are defined
import message_data.model.buildings
from ixmp import config

config.add_platform(
glatterf42 marked this conversation as resolved.
Show resolved Hide resolved
"ixmp-dev",
"jdbc",
"oracle",
"x8oda.iiasa.ac.at:1521/PIXMP2.iiasa.ac.at",
"ixmp_dev",
"ixmp_dev",
jvmargs="-Xmx16G",
)

p = Path("${{ github.workspace }}")
config.set("message local data", p / "report")
config.set("message solve options", {"iis": 1, "tilim": 60 * 30})
config.set("message buildings dir", p / "buildings")
config.save()

# Equivalent to "(ixmp|message-ix|mix-models) config show"
print(config.path.read_text())
shell: python

- name: Fetch data
run: |
mix-models fetch "SSP-Update-3.0"
mix-models fetch "SSP-Update-3.0.1"

- name: Run MESSAGEix-Transport workflow "${{ env.from-step }}" → "${{ matrix.SSP }}${{ matrix.policy }} reported"
run: |
mix-models \
${{ env.base }} \
transport run \
--base=auto --nodes=R12 --model-extra="ci nightly" \
--from="${{ env.from-step }}" \
"${{ matrix.SSP }}${{ matrix.policy }} reported" \
--go
continue-on-error: true

- name: Upload results as a build artifact
uses: actions/upload-artifact@v4
with:
name: MESSAGEix-Transport ${{ matrix.SSP }}${{ matrix.policy }}
path: |
${{ github.workspace }}/report/
!${{ github.workspace }}/report/cache/
if-no-files-found: error
continue-on-error: true

collate:
name: Collate outputs
needs: report
runs-on: self-hosted

steps:
- name: Merge artifacts
uses: actions/upload-artifact/merge@v4
with:
name: MESSAGEix-Transport
delete-merged: true
5 changes: 5 additions & 0 deletions .readthedocs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,11 @@ build:
os: ubuntu-22.04
tools:
python: "3.10"
jobs:
post_install:
# Remove requests-cache, which appears to cause segfaults on RTD workers
- pip uninstall --yes requests-cache


# Build documentation in the docs/ directory with Sphinx
sphinx:
Expand Down
6 changes: 5 additions & 1 deletion conftest.py
Original file line number Diff line number Diff line change
@@ -1,2 +1,6 @@
# Modules containing fixtures to use in testing
pytest_plugins = ["ixmp.testing", "message_ix_models.testing"]
pytest_plugins = [
"genno.testing",
"ixmp.testing",
"message_ix_models.testing",
]
2 changes: 1 addition & 1 deletion doc/api/report/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Elsewhere:
- Reporting for specific model variants:

- :mod:`.water.reporting`
- (Private) :doc:`Reporting of message_data.model.transport <m-data:reference/model/transport/report>`
- :doc:`transport/output` of :mod:`.model.transport`

- :doc:`‘Legacy’ reporting <legacy>`.

Expand Down
54 changes: 54 additions & 0 deletions doc/api/tools-messagev.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
MESSAGE V input files
*********************

This document describes some of the file formats for the pre-:mod:`.ixmp` MESSAGE model, a.k.a. **MESSAGE V**, and code in :mod:`message_ix_models.tools.messagev` that reads these formats.

.. note:: See also the earlier :doc:`import_from_msgV_sqlite` for similar code/descriptions.

.. contents::
:local:

``easemps3.free``: soft dynamic constraints
-------------------------------------------

Each constraint is specified by a single row with the following, space-separated entries:

1. **Constraint type.** `mpa` (constraint on activity) or `mpc` (constraint on capacity).
2. **Technology name.** Four-letter internal name/code of a technology, e.g. `uHap`.
3. **Lower/upper bound.** Either `LO` (decline constraint) or `UP` (growth constraint).
4. **Cost type.** One of:

- `lev`: levelized costs.
- `abs`: absolute costs.
- `var`: variable costs.

5. **Growth rate for step 1.** Percentage points of growth/decline at which the constraint becomes active.

6. **Additional cost for step 1.** Additional cost applied to activity/capacity growth or decline beyond the rate in #5. Depending on #4, specified:

- `lev`: in percentage points of the levelized cost of the technology.
- `abs`, `var`: in absolute monetary value.

7. **Up to 4 additional pairs of 5 and 6.** Growth rates for successive constraints are cumulative.

An example:

.. code::
mpa uEAp UP lev 5 50 15 300000
Here the constraint relates to a growth constraint (UP) for activities (mpa) and the technology for which the constraint is to be extended is uEAp.
The allowed rate of growth is increased by 5 %-points and each additional unit of output that can be produced costs 50 % of the levelized costs additional on top of the normal costs (i.e. the costs that result from building and using the additional capacity required for the additional production).

The second step increases the maximum growth rate further, by 15 %-points, but the costs are prohibitive (300000).

Soft constraints can be set for each technology individually. This can be done globally ("regions = all -glb") or for each region separately ("regions = cpa").


API reference
-------------

.. currentmodule:: message_ix_models.tools.messagev

.. automodule:: message_ix_models.tools.messagev
:members:
2 changes: 1 addition & 1 deletion doc/global/energy/enduse/index.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
Energy end-use
=================
MESSAGEix distinguishes three energy end-use sectors, i.e. transport, residential/commercial (also referred to as the buildings sector) and industry. Given the long-term nature of the scenarios, the model version used for the SSPs, represents these end-use sectors in a stylized way. For more detailed short-term analysis, a model version with a more detailed transport sector module that distinguishes different transport modes, vehicle classes and consumer types exists (McCollum et al., 2016 :cite:`mccollum_transport_2016`).
MESSAGEix distinguishes three energy end-use sectors, i.e. transport, residential/commercial (also referred to as the buildings sector) and industry. Given the long-term nature of the scenarios, the model version used for the SSPs, represents these end-use sectors in a stylized way. For more detailed short-term analysis, a model version with a more detailed transport sector module that distinguishes different transport modes, vehicle classes and consumer types exists (McCollum et al., 2017 :cite:`mccollum-2016`).

.. toctree::
:maxdepth: 1
Expand Down
2 changes: 2 additions & 0 deletions doc/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ Commonly used classes may be imported directly from :mod:`message_ix_models`.
api/report/index
api/tools
api/tools-costs
api/tools-messagev
api/data-sources
api/util
api/testing
Expand All @@ -80,6 +81,7 @@ Commonly used classes may be imported directly from :mod:`message_ix_models`.

global/index
material/index
transport/index
water/index

.. toctree::
Expand Down
29 changes: 23 additions & 6 deletions doc/main.bib
Original file line number Diff line number Diff line change
Expand Up @@ -1496,12 +1496,29 @@ @techreport{iea_investment_2014
year = {2014}
}

@article{mccollum_transport_2016,
author = {McCollum, David L. and Wilson, Charlie and Pettifor, Hazel and Ramea, Kalai and Krey, Volker and Riahi, Keywan and Bertram, Christoph and Lin, Zhenhong and Edelenbosch, Oreane Y. and Fujisawa, Sei},
journal = {Transportation Research Part D: Transport and Environment},
title = {Improving the behavioral realism of global integrated assessment models: An application to consumers’ vehicle choices},
type = {Journal Article},
year = {2016}
@article{mccollum-2017,
author = {McCollum, David L. and Wilson, Charlie and Pettifor, Hazel and Ramea, Kalai and Krey, Volker and Riahi, Keywan and Bertram, Christoph and Lin, Zhenhong and Edelenbosch, Oreane Y. and Fujisawa, Sei},
doi = {10.1016/j.trd.2016.04.003},
issn = {13619209},
journal = {Transportation Research Part D: Transport and Environment},
keywords = {Climate change mitigation; Consumer choice; Human behavior; Light-duty vehicles; Transport},
pages = {322--342},
publisher = {Elsevier Ltd},
title = {{Improving the behavioral realism of global integrated assessment models: An application to consumers' vehicle choices}},
volume = {55},
year = {2017}
}

@article{mccollum-2018,
author = {McCollum, David L. and Wilson, Charlie and Bevione, Michela and Carrara, Samuel and Edelenbosch, Oreane Y. and Emmerling, Johannes and Guivarch, C{\'e}line and Karkatsoulis, Panagiotis and Keppo, Ilkka and Krey, Volker and Lin, Zhenhong and Broin, Eoin {\'O} and Paroussos, Leonidas and Pettifor, Hazel and Ramea, Kalai and Riahi, Keywan and Sano, Fuminori and Rodriguez, Baltazar Solano and van Vuuren, Detlef P.},
doi = {10.1038/s41560-018-0195-z},
issn = {2058-7546},
journal = {Nature Energy},
number = {8},
pages = {664–673},
title = {Interaction of consumer preferences and climate policies in the global transition to low-carbon vehicles},
volume = {3},
year = {2018},
}

@article{mcjeon_gas_2014,
Expand Down
Loading
Loading