Skip to content

Implement currency data update workflow#61

Open
Wegatriespython wants to merge 6 commits intoIAMconsortium:mainfrom
Wegatriespython:feat/currency-data-update
Open

Implement currency data update workflow#61
Wegatriespython wants to merge 6 commits intoIAMconsortium:mainfrom
Wegatriespython:feat/currency-data-update

Conversation

@Wegatriespython
Copy link
Copy Markdown

This replaces the stubbed iam_units.update.currency() with a working generator and expands currency support from one hardcoded entry (EXC, EUR, 2005) to all five OECD Table 4 methods (EXC, EXCE, PPPGDP, PPPPRC, PPPP41) across five periods (2005, 2010, 2015, 2020, 2024) for EUR. This partly addresses #25.

The original stub in update.py specified three requirements for an implementation:

  • Use sdmx1 to query either the World Bank or OECD SDMX API.
  • Confirm the different data sources give the same results; if not, expose source selection. Cross-source validation against World Bank was tested manually for two data points (EXC and PPPGDP matched for DEU 2005), but the WDI SDMX endpoint was unreliable (403/404, timeouts), so WB is not wired in as a source. This can be revisited in a follow-up.
  • Write to a simple text format readable without dependencies, in iam_units/data/currency/, one file per (method, period).

The inline NB comments in currency.py noted two extensions: loading from file instead of embedded constants, and extending to cover other currencies. This PR addresses the first; EUR remains the only non-USD currency.

The current PR:

  • Implements currency() in the update submodule using sdmx1 to query OECD Table 4 (DSD_NAMAIN10@DF_TABLE4), following the pattern established for emissions data.
  • Writes plain-text data files to iam_units/data/currency/, committed as package data and loaded at runtime without any dependency on sdmx1.
  • Guards configure_currency() against silent wrong answers when called twice with different methods for the same currency/period pair — now raises ValueError. Same-method calls are idempotent.
  • Documents the update workflow in DEVELOPING.rst and the expanded capabilities in the README.

EUR rows are derived from the DEU series in Table 4. For exchange rates this is exact; for PPP methods the values are Germany-specific. This follows the approach in #60 which switched EUR deflators to Eurozone-wide values, but PPP is inherently country-level — the choice is documented in code and DEVELOPING.rst.

@khaeru
Copy link
Copy Markdown
Contributor

khaeru commented Mar 11, 2026

From the description this is great, thanks for the contribution!

One knee-jerk question prior to a real review: the 25 added data files each have 1 data point in them. Did you consider / would it be worse/better to follow an approach similar to the generation of emissions.py? IOW, generate a Python source file containing a data structure with all relevant data. This could be more compact.

@byersiiasa
Copy link
Copy Markdown

wow @Wegatriespython didn't expect you to address this within 2 hours. Thank you so much! ;)

@danielhuppmann
Copy link
Copy Markdown
Member

Thanks @Wegatriespython for the PR.

@khaeru, do you have time to review?

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Mar 27, 2026

Codecov Report

❌ Patch coverage is 78.35052% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.97%. Comparing base (1b3f6d8) to head (d0701e1).

Files with missing lines Patch % Lines
iam_units/update.py 39.39% 20 Missing ⚠️
iam_units/currency.py 96.15% 1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (1b3f6d8) and HEAD (d0701e1). Click for more details.

HEAD has 153 uploads less than BASE
Flag BASE (1b3f6d8) HEAD (d0701e1)
162 9
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #61      +/-   ##
==========================================
- Coverage   96.07%   87.97%   -8.11%     
==========================================
  Files           5        5              
  Lines         204      291      +87     
==========================================
+ Hits          196      256      +60     
- Misses          8       35      +27     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@khaeru
Copy link
Copy Markdown
Contributor

khaeru commented Mar 27, 2026

I was waiting/hoping to discuss the question I asked above.

@Wegatriespython
Copy link
Copy Markdown
Author

@khaeru Hey Paul, the 1 file per method period was following the spec in the update stub. It does look quite ugly agreed, but I think if we were to consolidate, 1 txt file will do just as well for brevity. I don't see the added value of making it a python file, unless I'm missing something

@khaeru
Copy link
Copy Markdown
Contributor

khaeru commented Mar 27, 2026

the 1 file per method period was following the spec in the update stub

OK, thanks for clarifying. I guess the lesson is that a stub is a stub because it's only half-conceived, and should be taken with a grain of salt/viewed critically.

I don't see the added value of making it a python file, unless I'm missing something

Two things:

  • A Python file doesn't need additional, format-specific loading/reading code, like _load_currency_data(), so this could be discarded.
  • A Python file would, on first use, get compiled into bytecode and stored in __pycache__, which would then be reused afterwards. Any performance gain is probably not noticeable in most contexts where iam-units is used, but it's still a good practice/habit to take advantage of this.

Having written this out, I think I'd prefer this approach, but if you are time-limited please just let me know and I'll add a commit or two to the branch to make the change.

@Wegatriespython
Copy link
Copy Markdown
Author

I see, I can pick this up but it would be around EOD. Does that work?

@khaeru
Copy link
Copy Markdown
Contributor

khaeru commented Mar 27, 2026

Yes, from my POV there's no hurry. Thanks!

Follow emissions pattern
@Wegatriespython
Copy link
Copy Markdown
Author

@khaeru Done! As requested there is now a single generated Python module (currency_data.py) instead of the 25 text files. The format-specific loader is gone — configure_currency() reads directly from the imported dict. The updater writes the module in the same pattern as emissions.py.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants