Skip to content

Missing market data on holdings

Ticket #206: Missing market data on holdings
Type: Debugging / Bug Fix / Pipeline Reliability
Affected Component: code_source_simule/pipeline.py, data/tipranks_raw.csv, database (titres, historique), Marketstack v2/eod

1. Context and Symptoms

For several days, the application had stopped showing recent prices in the UI. The import pipeline was still running, but expected data did not appear in the historique table.

The objective of this intervention was to isolate the exact origin of the issue:

  1. API calls (Marketstack),
  2. import-date logic,
  3. database insertion,
  4. UI rendering.

2. Investigation and Causal Chain

I conducted the investigation in short sequences with systematic hypothesis validation.

2.1 Initial observations

Because this intervention took place on a Saturday, weekend blocking prevented imports, which is expected behavior. After temporarily suspending that guardrail for debugging, the pipeline did consume API calls (quota increased). Despite those calls, processing still reported massive missing prices on the non-business test date. The database confirmed the absence of exploitable new rows for that date.

2.2 Technical checks

Following those initial observations, I ran targeted technical checks to isolate each link in the chain. First, I validated the API on reference symbols (KEY, AAL) and business dates: closing data was available on the provider side. Then I executed a targeted database import for those same symbols, with successful insertion and values clearly visible in history.

After this localized check, I expanded the test to broader imports across several business dates (2026-04-02, 2026-04-07, 2026-04-10, 2026-04-15); imports were globally successful. This phase also identified one recurring provider case (TIXT.XTSE) with missing prices on multiple dates. Finally, I performed a data-quality check and removed one duplicate detected on a ticker/date pair.

2.3 Causality with project history

I correlated the incident with the changes described in a previous report. That intervention solved a previous outage but introduced time-logic behavior that was not aligned with the current business need, which explains this bug.

The final diagnosis is therefore a business-logic regression on the import target date, not a global API outage and not a break in the database insertion chain.

3. Actions Completed

3.1 Return to pre-March logic

I reverted to the logic that existed before the March intervention on the critical point: target the previous market day instead of the raw current calendar date.

3.2 Improvements beyond a simple rollback

  1. Explicit temporal policy:
  2. block on Sunday;
  3. block on Monday by default;
  4. allow Monday-morning catch-up only after a recorded failure.
  5. Persist execution state (success/failure) with reason to drive catch-up decisions reliably.
  6. Record unexpected exceptions explicitly as exploitable failures for the next decision cycle.
  7. Add automated validation for new critical scenarios (Monday catch-up, Monday block without prior failure, unexpected-failure persistence).
  8. Update technical and operational documentation (chapters 2 and 3, FR/EN), pipeline test documentation, execution reports, and generated documentation site.
  9. Align cron strategy: keep system scheduling simple and enforce business guardrails in code.

4. Operational Result

The nominal state has been restored and strengthened:

  1. The API -> pipeline -> database chain is confirmed functional on business days.
  2. The main UI-facing issue came from a valuation-date misalignment, not from a global API outage.
  3. Business logic is now aligned with real usage: morning consultation of previous-day performance.
  4. Execution rules are explicit, tested, and driven by persistent state.
  5. The environment is back to a stable condition, with protections re-enabled and stronger robustness.

The incident is therefore closed from an operational perspective, with a stronger reliability level than before.

5. Lessons Learned

  1. An incident fix can create delayed regressions; full business scenarios must be tested, not just the initial symptom.
  2. A technically "green" pipeline does not guarantee delivered business value: the effective valuation date must be observable.
  3. Rollback alone is insufficient: explicit guardrails (execution rules + persistent state) are essential.
  4. Documentation, tests, and code must remain synchronized; otherwise maintenance becomes fragile and ambiguous.