Skip to content

Quota-Aware Import Policy with Targeted Retry

Ticket #143: Quota-Aware Import Policy with Targeted Retry
Type: Feature / Reliability / Cost Control
Affected Component: code_source_simule/pipeline.py, tests/test_pipeline.py, config.ini, README.md

1. Context

The ETL pipeline enriches portfolio data by calling the Marketstack API for each ticker. With a monthly limit of 10,000 requests, there was no mechanism to prevent the pipeline from consuming its entire quota uncontrolled, nor any protection against unnecessary retry loops when prices were already available.

Additionally, the Marketstack /v1/account endpoint intended to return real-time quota usage does not exist.

2. Objective

Implement a quota-aware import policy that:

  • Classifies each pipeline run into one of four modes based on quota usage: normal, eco, protection, or blocked
  • Blocks API enrichment entirely when quota is critically low
  • Limits retry to a single pass, targeting only tickers still missing a price
  • Falls back gracefully when the quota API endpoint is unavailable
  • Allows threshold configuration without redeployment

3. Implementation

3.1 — Quota mode classification

A QuotaMode class classifies runs using three configurable thresholds (default: 65 / 80 / 95 %):

Usage Mode Behavior
< 65 % normal Full enrichment + retry enabled
65–79 % eco Enrichment enabled, retry disabled
80–94 % protection Enrichment enabled, retry disabled, operator alert
≥ 95 % blocked Enrichment cancelled entirely

3.2 — Targeted retry

In normal mode only, a single retry pass enriches only tickers that remain without a price after the first pass — never re-submitting already-enriched tickers.

3.3 — Local quota fallback counter

Since /v1/account does not exist, the pipeline maintains a local JSON counter at logs/marketstack_quota_usage.json. The counter:

  • resets automatically on the first call of each calendar month
  • increments per API call made during enrichment
  • is read as the quota source when the API endpoint is unavailable

3.4 — Configurable thresholds

All thresholds and the log path are read from config.ini. Missing or invalid values fall back to safe defaults (65 / 80 / 95, logs/pipeline.log) with a logged warning.

3.5 — Structured observability

Every quota decision, mode transition, operator alert, and enrichment block is written as a structured JSON event to logs/pipeline.log.

3.6 — Additional hardening delivered

  • Enforced secure Marketstack requests over HTTPS for quota and EOD calls
  • Confirmed MARKETSTACK_API_KEY as the only allowed API key source (no fallback secret)
  • Completed documentation parity updates across README.md, docs/tests/pipeline.md, and docs/fr/tests/pipeline.md
  • Completed final validation scenarios (normal, protection, blocked) and non-regression run

4. Test coverage

  • 11 new test cases added (tc-pipe-quota01 through tc-pipe-quota11)
  • Coverage for the pipeline module: 60% → 83%
  • All 49 tests pass (24 pipeline + 3 config + 22 other)

5. Execution reports refreshed

  • report.xml regenerated from the full test run (49 passed)
  • report.jsonl regenerated from the same run for detailed execution traces

6. Result

The pipeline now operates within its monthly quota envelope automatically. Operators can adjust thresholds in config.ini without touching code. The local fallback counter ensures protection even if the Marketstack account endpoint is inexistant.