Skip to content

Import Pipeline

Context

This workflow explains how import data moves from source input to enriched results, including quota-aware decisions and fallback behavior.

Details

High-level flow: 1. Load source dataset. 2. Evaluate quota mode (normal, eco, protection, blocked). 3. Resolve portfolio currency from the ticker (TSE: => CAD, otherwise USD). 4. Convert only a current USD price when the portfolio currency differs; keep 52-week thresholds raw. 5. Persist results and operational state, including FX state when a rate is retrieved. 6. Emit logs and optional alerts.

Exchange-rate behavior: - reuse a recent rate from logs/fx_rate_state.json before calling the API; - call APILayer Exchange Rates Data only when the local state is absent or expired; - after a successful call, update both the FX state file and config.ini (usd_to_cad_rate); - on API failure, fall back to the last known rate and then the configured rate so import remains fail-open.

Display and totals contract: - native holding pages and 52-week indicators use valeur and devise only; - raw an_haut and an_bas thresholds are never converted; - cad_value is prohibited from native displays and 52-week comparisons; it is reserved for consolidated totals, historical charting, and consolidated performance; - dashboard totals expose CAD, USD, and consolidated CAD values. The consolidated amount is the sum of persisted cad_value × quantite from the latest snapshot, so it matches the chart exactly; - the dashboard rate is the rate applied to that latest snapshot, with the snapshot date. It is unavailable only when no USD conversion can be derived from the snapshot.

Operational checkpoints: - verify input freshness before running import tasks; - confirm quota mode before large enrichment operations; - validate output integrity and log entries after completion. - when portfolio synchronization removes a holding, delete its transaction rows before its snapshots and holding row so foreign-key checks remain valid; - apply schema migrations only from the application pre-start process before Gunicorn; the deployment workflow must not run a second migration process concurrently.

When something fails: - use retry strategy for transient API issues; - keep fail-open behavior for alerting to avoid blocking core business flow; - consult logs for exact failure stage.