Skip to content

Chapter 2: Technical Architecture

2.1. Architectural Overview

The architecture of the "Augmented Analyst" project was designed to ensure robustness, reliability, and automation. It is based on a clear separation of responsibilities between three distinct environments: the local development environment, the Continuous Integration/Continuous Deployment (CI/CD) platform, and the production environment.

The standard workflow is as follows:

  1. Development and initial testing are carried out in the local environment.
  2. Each validated and "pushed" change to the Git repository triggers the process in the CI/CD environment, which acts as a quality control gateway.
  3. If, and only if, quality is validated, the code is automatically deployed to the production environment, where it is accessible to the end-user.

This structure ensures that no changes can reach production without being tested and validated, thus ensuring maximum service stability.

2.2. The Main Application (Backend)

The logical core of the application is a backend developed in Python, orchestrated by the Flask micro-framework.

2.2.1. Framework

The application is built with Flask. The code follows the Application Factory design pattern (via the create_app function). This approach involves encapsulating the creation and configuration of the application in a function, which offers several advantages:

  • Testability: allows for the creation of different application instances for different contexts (production, testing) with distinct configurations.
  • Organization: avoids global application variables and promotes a cleaner, more modular code structure.

2.2.2. Models and Database

Interaction with the database is managed by the SQLAlchemy ORM. The schema consists of three main tables:

  • user: stores authentication information.
  • titres (stocks): a reference table containing the list of assets. It includes:
    • ticker (VARCHAR(255)): unique stock symbol (e.g., AAPL, BRK.B). Sized to accommodate current and future market tickers.
    • devise_portefeuille (portfolio_currency): the stock's native currency within the portfolio, derived from the ticker prefix (TSE: => CAD, otherwise USD). This is the single source of truth used by the application for native display.
    • Columns for 52-week thresholds:
      • an_haut, an_bas (52w_high, 52w_low): store the raw threshold value, exactly as provided by the source file, with no conversion.
      • an_haut_cad, an_bas_cad (52w_high_cad, 52w_low_cad): kept for schema compatibility; they contain the same raw value as an_haut/an_bas. The 52-week thresholds are never converted, regardless of the stock's currency.
  • historique (history): records a "snapshot" of each stock for each day. It contains:
    • valeur, devise (value, currency): the stock price as displayed to the user, and its actual currency. USD-to-CAD conversion is applied only when the stock is quoted in USD while its portfolio currency (devise_portefeuille) differs from USD; in every other case, valeur remains the native price and devise reflects the portfolio currency.
    • cad_value: the CAD equivalent captured with each snapshot. It is used only for consolidated CAD totals, the portfolio-evolution chart, and consolidated performance; it is never used for a stock's native price or 52-week thresholds.
    • fx_rate_used: the exchange rate explicitly applied to a targeted USD-to-CAD price conversion, or NULL when that targeted conversion was not needed.
    • quantite (quantity): the quantity held.
  • transactions: a separate ledger of the administrator's purchase and sale decisions. Each row belongs to a holding and stores a date, type, shares, unit price and a two-decimal total. The database preserves up to four decimals for legacy prices and quantities from the supplied file; new entries remain limited to two decimals. This table does not change price history or market indicators.

2.2.2bis. Holding Transactions

Each holding detail page now provides two complementary views:

  • Snapshot History: what the market reported over time, such as observed prices by date.
  • Transactions: what the administrator actually did on the holding, such as buying or selling shares on a specific date.

This separation is intentional. Market snapshots support performance analysis; transactions explain the user's own buy and sell decisions. Editing a transaction never rewrites market history and never changes market indicators.

What a transaction contains

Each transaction stores:

  • an operation date;
  • an operation type: BUY for a purchase or SELL for a sale;
  • the number of shares;
  • the unit price;
  • the total amount.

The total amount is never entered manually. The application calculates it on save: shares × unit price. This prevents a typing error from creating a total that contradicts the visible fields.

Add and edit rules

The administrator can add or correct a transaction from the holding detail page, but the application guarantees transaction consistency:

  • the first transaction by date must always be a BUY, because a holding cannot start with a sale;
  • a SELL cannot be added with a date earlier than the holding's first BUY;
  • a BUY can be added with a date earlier than the first existing BUY, because that can represent a legitimate correction to the history;
  • a transaction is rejected if, on its date, it would make the share balance negative;
  • new entries created through the UI remain limited to two decimal places for share quantities and unit prices.

These rules apply even when an operation is entered later with a past date. The application therefore checks the real chronological order, not only the order in which records were typed.

When multiple operations carry exactly the same date, BUY transactions are counted before SELL transactions. A purchase made during the day can therefore cover a sale on that same day, regardless of the order in which the rows were entered.

Delete rules

The table must keep at least one BUY transaction as long as the holding exists. This avoids showing a holding with no founding purchase.

When the administrator clicks Delete:

  • if the deleted transaction is not the holding's only remaining BUY transaction, deletion can be confirmed normally;
  • if deleting a BUY would make total Shares negative because of later SELL transactions, deletion is refused and a specific message explains why;
  • if the administrator tries to delete the only BUY while one or more SELL transactions still exist, the application asks them to delete all sales first;
  • if the only BUY is the remaining structural operation and no sale exists, the application warns that the full holding will be deleted;
  • in that final case, Cancel stops the action and nothing changes; Confirm deletes the holding, its snapshot history and its remaining transaction.

If several requests try to change the same holding at once, the application processes them one at a time. Each request checks the balance left by the previous one, preventing their combined result from producing a negative share balance.

UI ordering

Transactions are displayed from newest to oldest. This keeps the most decision-relevant operations at the top while preserving the full ledger in the same table.

Public demo

The public demo follows the same logic, but only with isolated data stored in the visitor's session. It allows visitors to view, add, edit and delete transactions without touching secure data.

If the visitor confirms deletion of the only remaining BUY for a demo holding, that holding is temporarily hidden in the session. The result is visible once, and the next refresh restores the initial demo baseline. No change is written to the secure database.

2.2.2bis. Stock Detail Chart: Enriched Demo History and Adaptive Density

  • The public demo (/demo/titre/<ticker>) now exposes 10 Snapshot History entries per stock: the 2 existing real entries (June 28 and July 5, 2026) plus 8 weekly entries generated through deterministic linear interpolation (no random values), so the chart shows a credible price trend while staying within the existing 10-entry privacy cap per stock.
  • In the secure application, the stock detail chart (/titre/<id>) no longer plots one point per raw entry. A dedicated function (bucket_history_for_chart in code_source_simule/portfolio_metrics.py) groups entries by how old they are relative to today:
    • entries from a fully completed year: one point per quarter;
    • entries from the current year but a completed quarter: one point per month;
    • entries from the current quarter but a completed week: one point per week;
    • entries from the current week: one point per available day.
  • When a grouped period contains several entries, only the most recent one is kept as the representative point. The "Snapshot History" table below the chart still displays 100% of the raw entries; only the chart rendering is reduced.

2.2.3. Authentication Management

Access security is ensured by two key Flask extensions:

  • Flask-Login: manages the user session lifecycle (login, logout, protection of "private" routes).
  • Flask-Bcrypt: ensures the secure storage of passwords by saving only their cryptographic "hash," never the password in plain text.

2.3. The Data Pipeline (ETL)

The system is powered by a robust ETL (Extract, Transform, Load) pipeline, orchestrated by the pipeline.py script. This pipeline is designed to be independent of its execution time.

2.3.1. Temporal Logic: Previous Market Day

The pipeline processes the previous market day relative to the current Montreal date/time, so morning runs always target the latest available closing session.

  • get_current_montreal_datetime(): the script first determines the current date/time in the Montreal time zone (America/Montreal).
  • date_a_traiter = previous_market_day: this date becomes the source of truth for all pipeline operations (API calls, database insertions).
  • Operational gate:
    • Sunday imports are blocked.
    • Monday imports are blocked by default.
    • Monday morning catch-up is allowed only if the previous recorded run failed.
  • Persistent run state: each run writes a status (success/failure) and reason to logs/import_run_state.json (or configured path) to drive Monday catch-up authorization.

2.3.2. Orchestration via run_full_pipeline

The entry point of the process is the run_full_pipeline function, which accepts a boolean argument fetch_market_data.

  1. fetch_market_data=True mode (behavior for the daily cronjob):

    • Extraction: The pipeline reads the reference CSV and calls the Marketstack /v2/eod endpoint to retrieve closing prices for date_a_traiter.
    • Objective: Insert only API-backed prices for the execution date. Missing API prices are not inserted and trigger operator alerting.

  2. fetch_market_data=False mode (for portfolio synchronization):

    • Extraction: The pipeline only reads the CSV file. No call to Marketstack is made.
    • Objective: Allows immediate synchronization of the database structure after a portfolio modification (addition/removal of stocks). It can insert temporary CSV-based records for date_a_traiter.

2.3.3. Transformation (Transform)

  • The read_and_clean_csv function standardizes column names, handles empty or corrupted files, and parses complex information (like the "52 Week Range" column).
  • Conversion business rule: currency conversion is targeted and limited to the current market price. It applies only when the stock's currency (marketstack_currency) is USD while its portfolio currency (derived from devise_portefeuille(ticker)) differs from USD. In every other case, the price stays displayed in its native currency. The 52-week thresholds (an_haut, an_bas) are never converted: they remain strictly faithful to the source file, regardless of the stock's currency.

2.3.3bis. Robust Exchange-Rate Handling (Fail-Open)

The USD-to-CAD rate used for the targeted conversion is obtained via get_effective_fx_rate, following logic designed to respect a limited API call quota:

  1. Local cache: if a rate was already retrieved recently (state persisted in logs/fx_rate_state.json, or a configured path), it is reused without any API call.
  2. Call to the Exchange Rates Data API (APILayer): made only when the cache is missing or expired, via fetch_exchange_rate().
  3. Automatic configuration update: on every successful API call, the retrieved rate is persisted both in the local state (fx_rate_state.json) and in the usd_to_cad_rate value of config.ini.
  4. Cascading fallback (fail-open): if the API fails, the pipeline falls back to the last known rate, then, as a last resort, to the usd_to_cad_rate value already present in config.ini. Imports are never blocked by an exchange-rate service outage.

2.3.4. Loading (Load)

Transformed data is inserted into the MariaDB database via UPSERT queries (INSERT ... ON DUPLICATE KEY UPDATE).

  • The unique key (titre_id, date_releve) of the historique table is fundamental. It ensures that there can only be one record per stock and per day.
  • Update Scenario: if an execution in fetch_market_data=False mode created an entry for a date with a temporary price, and a later run targets the same date with official API data, UPSERT will not create duplicates. It will update the existing entry.

2.4. The Production Infrastructure (VPS)

The application is hosted on a Virtual Private Server (VPS) under Ubuntu, with a modern architecture based on containerization.

2.4.1. Containerization (Docker)

  • app service: a custom-built Docker container (via a Dockerfile) that encapsulates the Python application. The application is served by a production WSGI server, Gunicorn, optimized to handle multiple simultaneous requests.
  • db service: an official MariaDB 10.6 container, ensuring a stable and isolated database environment.
  • Orchestration: all services are defined, configured, and linked by Docker Compose (via the docker-compose.yml file), which acts as the infrastructure's orchestrator.
  • Data Persistence: to ensure no database data is lost during updates or restarts, MariaDB's files are stored in a named Docker volume, which is independent of the container's lifecycle.

2.4.2. Web Server

Nginx is used as the main web server. It acts as a reverse proxy:

  • It receives all incoming web requests on ports 80 (HTTP) and 443 (HTTPS).
  • It forwards requests to the application running in the Docker container on port 8000.
  • It handles SSL/TLS termination, serving certificates managed by Certbot to ensure a secure connection (HTTPS).

2.5. The Quality and Deployment System (CI/CD)

Automation is at the heart of the project, managed by a CI/CD pipeline hosted on GitHub Actions.

2.5.1. Workflow (deploy.yml)

  • Trigger: the workflow is automatically launched with every push to the main branch.
  • test job (Continuous Integration): before any deployment, the code is retrieved in an ephemeral Linux environment. A full test suite is executed with Pytest. To ensure realistic validation, this job launches its own temporary MariaDB database service. If a single test fails, the entire pipeline stops.
  • deploy job (Continuous Deployment): only if the test job succeeds, the workflow connects to the production VPS via SSH. It then executes a script that orchestrates the deployment:
    1. git pull to retrieve the validated code.
    2. docker compose up -d --build to rebuild the application image with the new code and restart the services without major interruption.

2.5.2. Testing Strategy

  • Framework: Pytest, for its simplicity and power, as well as pytest-mock to simulate external calls.
  • Test Environments: Use of an in-memory SQLite database for fast local tests, and a real MariaDB database in CI for maximum reliability.
  • TDD Philosophy: Development follows a Test-Driven Development approach.
  • Test Management and Traceability: The project includes a "docs-as-code" test management system:
    • Each test case is a Markdown file stored in the test_cases/ directory.
    • A Python script (scripts/sync_tests.py) ensures traceability by linking each test case to its pytest implementation via a @pytest.mark.test_id marker.
    • A functional coverage report website is automatically generated with MkDocs, providing a clear view of the validation status of each feature.