Skip to content

Chapter 3: Operational Guide

This chapter explains how to develop, maintain, and administer the AI (Augmented Investor) application. It follows a change from defining the need through release, then introduces the operations required for day-to-day use.

Start with the development workflow to understand how a change is prepared and verified. The following sections distinguish data maintenance, server administration, and dependency and documentation management. Finally, the debugging plan applies the same framework to the specific case of a bug fix.

3.1. The Ideal Development Workflow

Any change to the project, whether it's a bug fix, a new feature, or a simple text update, must follow this rigorous process to ensure production stability.

3.1.1. Create a Working Branch

Never work directly on the main branch. Every new task begins by creating a dedicated branch from the most up-to-date version of main.

# Ensure the local `main` branch is up-to-date
git checkout main
git pull origin main

# Create and switch to a new descriptive branch
git checkout -b <change-type>/<short-description>
Branch name examples: feature/add-new-chart, fix/login-page-bug.

3.1.2. Local Code Modification

With the working branch in place, make all necessary code changes in your local development environment (e.g., VS Code).

3.1.3. Local Test Validation

Once the changes are ready, run the entire test suite to ensure your changes have not introduced any regressions. This command must be run from the root of the application repository.

# From the project root
python -m pytest -v
All tests must pass (or be intentionally skipped).

3.1.4. Save Your Work (Commit)

After verification, save your changes to the Git history with a clear and concise commit message, following the "Conventional Commits" convention.

# Add the modified files
git add .

# Create the commit
git commit -m "type(scope): description of the change"
Examples: feat(dashboard): Add 52-week analysis panel, fix(pipeline): Correct currency conversion logic.

3.1.5. Share and Review (Pull Request)

The commit preserves the work locally; a Pull Request makes it available for review. Push your working branch to the GitHub repository and open a "Pull Request" (PR).

git push --set-upstream origin <your-branch-name>
On GitHub, create the Pull Request targeting the main branch. The PR is the opportunity to describe your changes and allow the CI/CD system to run the tests one last time in a clean environment.

3.1.6. Run the AI Pull Request Review

The Pull Request gives the AI review a precise scope: the differences between the working branch and main. Before launching that review, verify these prerequisites:

  1. The PR is open on GitHub (source branch pushed, target = main).
  2. The PR is not in Draft mode and the diff matches the expected scope.
  3. Local tests passed on your branch (python -m pytest -v).
  4. Critical CI checks already available on the PR are not in a blocking failed state.
  5. Your working context is clean: no uncommitted local changes on feature files.

Execution procedure (from VS Code, inside the project repository):

  1. Open Copilot chat in the context of the PR branch.
  2. Run the agent command: /speckit.review.
  3. Wait for the review report to be generated in the session terminal.

Expected outcomes:

  • The review generates a structured report for merge decision-making.
  • The verdict is explicit: approved or blocked with rationale.
  • If the review is re-run on the same PR, the most recent report becomes the decision reference.
  • No comment, APPROVE status or REQUEST_CHANGES status is automatically published to GitHub.

Final control after execution:

  1. Read the structured report and confirm blocking points (if any) are actionable.
  2. Record the review decision in the PR (manual comment if needed).
  3. If review blocks the PR, fix issues, push updates, then run /speckit.review again.

Summary (Executable Checklist):

  • [ ] Finalize changes on the working branch.
  • [ ] Run local tests.
  • [ ] Commit the work.
  • [ ] Push the branch to GitHub.
  • [ ] Open the Pull Request to main.
  • [ ] Run /speckit.review.

3.1.7. Merging

Once the Pull Request is approved (review completed and CI tests passed), merge it into main.

3.2. Data Maintenance Procedures

The preceding workflow concerns changes to the project. Day-to-day portfolio maintenance requires distinguishing transactions recorded in the application from market data supplied by imports. The procedures below explain which tool to use in each case, followed by how to check or troubleshoot the result.

3.2.1. Update Portfolio Composition

  • When: When you buy or sell stocks, or modify quantities.
  • Procedure:
    1. Open the data/tipranks_raw.csv file on your local machine.
    2. Modify, add, or delete the necessary rows.
    3. Crucial: For any new row, ensure you fill in the Marketstack_Ticker and Marketstack_Currency mapping columns after researching them on the Marketstack website.
    4. Follow the Ideal Development Workflow (commit, push, merge). The data pipeline will process the changes from the CSV.

3.2.2. Manually Run the Data Pipeline

Whereas a transaction updates the holding's position, the pipeline refreshes market data. Use the following procedure when that refresh must be rerun without waiting for the next scheduled execution.

  • When: For a one-time data refresh or to force processing for a specific day after an error.
  • Procedure:
    1. SSH into the VPS.
    2. Navigate to the project directory: cd /var/www/qa-automated-pipeline.
    3. Run the command: docker compose exec app python code_source_simule/import_data.py.
  • Important Behavior: This command targets the previous market day. Sunday runs are blocked; Monday runs are blocked by default and allowed only as a morning catch-up when the previously recorded run failed.

3.2.3. Configure SMTP Alerts

To monitor these executions without routinely reading their logs, configure email notifications. They report pipeline problems, but their own unavailability must not interrupt the import.

  • When: Before enabling production operator email notifications.
  • Procedure: Configure these environment variables on the target runtime (local shell, VPS, or CI secret store):
    • SMTP_HOST
    • SMTP_PORT
    • SMTP_USER
    • SMTP_PASSWORD
    • ALERT_EMAIL_TO
    • SMTP_SECURITY (starttls, ssl, none) — default starttls
    • SMTP_TIMEOUT_SECONDS — default 10
  • Recommended profiles:
    • 587 + SMTP_SECURITY=starttls
    • 465 + SMTP_SECURITY=ssl
  • Fail-open behavior: if SMTP is misconfigured or unreachable, the pipeline continues and logs a warning/event for diagnosis.

3.2.4. Troubleshoot Missing API Prices (Marketstack)

When an alert or a portfolio review reveals missing prices, first determine whether the problem lies in the scheduled execution or the provider's response. This distinction helps target the fix rather than changing code before reproducing the failure.

  • When: UI history stops updating or alerts report a high number of missing API prices.
  • Procedure:
    1. Confirm cron timing first (crontab -l) and verify the last run log (tail -n 80 /var/log/cron-pipeline.log).
    2. Reproduce the failing date directly against provider endpoints in Postman or curl.
    3. Compare v1/eod and v2/eod for the same symbols and date range before any code change.
    4. If v2/eod returns valid data and v1/eod does not, treat this as endpoint compatibility and align pipeline endpoint usage.
    5. Re-run one manual import and verify DB/UI refresh before closing the incident.
  • Rule: For external API incidents, provider-side endpoint/version reproduction is mandatory before implementing mitigations.

3.2.5. Run User E2E/API Commands (local Windows)

After maintenance or a fix, workflow tests check what the user sees in the application. The commands below cover login and the dashboard; API commands separately check exchanges with Marketstack.

  • When: To manually validate application login, reusable authentication state, dashboard rendering, and Marketstack API scenarios.
  • Standard procedure:
    1. Open PowerShell in e2e/.
    2. Save secrets once (or after credential changes):
      • npm run secrets:save:windows
    3. Rebuild login session state:
      • npm run test:auth:setup:windows
    4. Validate access without re-entering login:
      • npm run test:skip-login:windows
    5. If only the Marketstack API key changes:
      • npm run secrets:save:windows:marketstack
    6. Run secure Marketstack API tests:
      • npm run test:marketstack:windows
    7. Run the dedicated dashboard suite without opening the browser:
      • npm run test:dashboard
    8. Run the same dashboard suite with the browser visible:
      • npm run test:dashboard:headed
  • Useful shortcuts:
    • Full E2E suite: npm run test:e2e
    • Local CI subset aligned with the Marketstack business suite: npm run test:e2e:ci
    • Local Marketstack CI subset: npm run test:marketstack:ci
  • Security rule: never store MARKETSTACK_API_KEY in plaintext in the repository; use only local Windows secrets (DPAPI) and GitHub secrets in CI.

These commands use protected credentials, but TOTP login also requires a temporary code. For a remote login, the Windows launcher requests this code without echo and retains neither the code nor the permanent factor in the vault. Automatic generation is restricted to the isolated loopback server; login suites disable screenshots and traces containing inputs. Marketstack tests remain separate: they do not validate TOTP.

3.3. Production Server Administration (VPS)

Maintenance procedures depend on the services hosting the application working correctly. This section therefore moves from data to its runtime environment: observing services, intervening when necessary, checking scheduling, and administering private access.

3.3.1. Check Application Status

  • View active containers: docker ps (should show qa-automated-pipeline-app-1 and qa-automated-pipeline-db-1 with status Up).
  • Application logs (Gunicorn/Flask): cd /var/www/qa-automated-pipeline && docker compose logs -f app (-f to follow in real-time).
  • Web server error logs: sudo tail -f /var/log/nginx/error.log.

3.3.2. Restart Services

If the preceding diagnosis warrants a restart, select the affected service rather than restarting everything by default.

  • Full restart (App + DB): cd /var/www/qa-automated-pipeline && docker compose restart.
  • Restart the application only: cd /var/www/qa-automated-pipeline && docker compose restart app.
  • Restart Nginx: sudo systemctl restart nginx.

3.3.3. Manage the Automated Pipeline (cron)

A running service does not guarantee that imports start at the expected time. Also check the pipeline schedule and the log of its latest execution.

  • List scheduled tasks: crontab -l.
  • Edit scheduled tasks: crontab -e.
    • Reference schedule: on the VPS configured as Etc/UTC, run at 06:00 UTC Monday through Saturday, which is 02:00 in Montreal during daylight time (EDT) and 01:00 during standard time (EST). The previous-weekday rule then targets Friday during Saturday's run.
    • Check the timezone with timedatectl show -p Timezone and use the versioned entry in scripts/pipeline.crontab. Replace only the pipeline entry; do not overwrite other tasks with crontab scripts/pipeline.crontab.
    • Pipeline guardrails block Sunday and allow Monday morning catch-up only after a recorded failure for the same market date.
  • Check logs of the last execution: cat /var/log/cron-pipeline.log.

3.3.4. Interact with the Database

When logs are insufficient for diagnosis, the MariaDB client lets you examine the recorded data directly.

  1. SSH into the VPS.
  2. Navigate to the project directory: cd /var/www/qa-automated-pipeline.
  3. Load environment variables: source .env.
  4. Launch the MariaDB client: docker compose exec db mysql -u"$DB_PROD_USER" -p"$DB_PROD_PASSWORD" "$DB_PROD_NAME".

3.3.5. Activate or replace the TOTP factor

The authentication code becomes mandatory after activation on your account. The encryption key configured on the server protects TOTP secrets, but does not automatically register your phone. To activate this protection, follow the steps below from your SSH terminal on the server, with your phone nearby. No application restart is required.

1. Prepare the phone and terminal

Install a TOTP-compatible authenticator application and enable automatic date and time on the phone. Then connect to the VPS over SSH and open the project directory:

cd /var/www/qa-automated-pipeline

The commands use the application container and its existing configuration; first check the database and keys using the environment preparation described below. Replace YOUR_ACCOUNT with your website login username, not your SSH account.

2. Check the current status

Before preparing a new factor, check the account status:

docker compose exec app python scripts/manage_totp.py status --username YOUR_ACCOUNT

Status reveals no secret. If you see Activé : False, continue with initial activation. If you see Activé : True, the factor is already active: do not run begin. To change phones, use the replacement procedure below. The administrative tool displays its messages in French.

3. Register the key in the phone application

For an inactive account, start preparation:

docker compose exec app python scripts/manage_totp.py begin --username YOUR_ACCOUNT

A secret key appears in your terminal. On the phone, choose Add an account → Enter a key manually, give the account a recognizable name and enter this key. Choose time-based mode; if the application asks for parameters, use SHA-1, six digits and thirty seconds.

Do not share the key in a chat or report, take terminal screenshots or use an online QR generator. Preparation is valid for ten minutes. If you run begin again during this period, explicitly answer oui to resume the same pending key; after expiration, restart preparation and update the key on the phone.

4. Confirm activation

Once the account is registered on the phone, confirm before expiration:

docker compose exec app python scripts/manage_totp.py confirm --username YOUR_ACCOUNT

Enter the six-digit code displayed by your application. Your input does not appear in the terminal: this is normal. Never pass a password, code or secret as an argument; activation and replacement actions require an interactive terminal.

The expected message is: Facteur confirmé. Les anciennes sessions sont révoquées. This means the factor is confirmed and previous sessions are revoked. Then verify the result:

docker compose exec app python scripts/manage_totp.py status --username YOUR_ACCOUNT

Status must display Activé : True. Protection becomes active only after this successful confirmation.

5. Verify login

Wait for the phone to display the next code, because the confirmation code has already been consumed. Then open the website login page over HTTPS, preferably in a private window:

  1. Check once that the password alone is rejected.
  2. Log in with your username, password and the new code.
  3. Verify dashboard access. In the previous browser, also check that a session opened before activation no longer grants private access.

Replace the factor after changing or losing a phone

Retain your administrative SSH access: it allows factor replacement without disabling protection. From the same project directory, run:

docker compose exec app python scripts/manage_totp.py replace --username YOUR_ACCOUNT

Answer oui to authorize replacement, then register the new key on the phone as in step 3. Next run confirm, check status and test login with the next code, as in steps 4 and 5. The old factor remains active until the new one is confirmed; temporary deactivation is not permitted.

If the web server refuses to start, the administrative command remains independent of its startup guard. If the encryption key is lost, recovery requires a valid server key and confirmation of a new factor; it never authorizes password-only access.

Isolated local validation. To check this workflow without touching the real portfolio, use the TOTP commands and prerequisites: AUTH_LOCAL_MODE=1, an absolute file-based SQLite path outside the repository (E2E_DB_PATH), independent keys and ephemeral credentials in the environment. The scripts/start_e2e_test_server.py server and administrative command target the same database. The server listens only on 127.0.0.1 and retains transactions, the factor and limits after restart. Never supply production secrets or the production .env.

Environment preparation. Isolated tests check the workflow; release also requires checking the environment that protects it. Check migration and concurrency on MariaDB, workflows in Chrome, Nginx, HTTPS and remote closure of port 8000; prepare and separately back up the persistent key and database. SQLite success does not replace MariaDB checks. GitHub branch, PR and manual checks do not deploy: only a push to main can trigger delivery after successful checks.

Safe rollback. Finally, if an intervention requires returning to a previous version, access protection must remain effective. Retain the schema and factors. After activation, never re-expose a version that ignores TOTP: deny private access, diagnose and fix forward. A key or database failure permits no bypass. Without server access, recovery requires hosting support; no backup code is provided.

3.4. Dependency Management

Beyond service configuration, some changes require a new library. Installing it locally and declaring it in the project serve complementary purposes: the first enables development, while the second makes the environment reproducible.

To add a new Python library (e.g., new-library): 1. On your local machine, with the venv activated, install the library: pip install new-library. 2. Update the requirements.txt file. This is the most important command to ensure reproducibility. 

pip freeze > requirements.txt
3. Verify that the application still works by running local tests (pytest -v). 4. Follow the Ideal Development Workflow (commit, push, etc.). The deployment process should rebuild the Docker image with the new library (--build), making the dependency available in production.

3.5. Project Documentation Management

To keep these procedures usable, documentation must evolve alongside the code and its dependencies. Previewing it supports review before publication, while test-case sources provide traceability for the checks.

The project documentation (the chapters you are currently reading), the test coverage report, and the architect's log are managed with MkDocs.

3.5.1. Build and View Documentation Locally

The documentation source files live directly in the docs/ directory. To preview locally, run the MkDocs development server from the project root:

mkdocs serve

You can now open your browser to http://127.0.0.1:8000 to see the live preview.

Note: You must have the documentation dependencies installed: pip install -r docs/docs-requirements.txt.

3.5.2. Update the Documentation

  • To modify the main documentation (like this guide), edit the Markdown files located in docs/.
  • To update the functional test cases, edit the Markdown files in test_cases/.
  • Changes are reflected immediately in the local preview server.

3.6. Formalized Debugging Plan

When a defect is observed, the development workflow introduced at the start of this chapter still applies. The following plan makes it specific to a fix: start with the symptom, reproduce it in a test, then verify and document the solution.

To ensure consistent, reliable, and well-documented bug fixes, all bugs must follow this standardized debugging workflow:

3.6.1. Overview of the Standardized Plan

  1. Create a GitHub Issue — Brief statement of the problem (constat initial) without diagnosis details.
  2. Create a Dedicated Branch — Branch naming: fix/<short-issue-description>.
  3. Develop TDD Tests — Write tests that validate the expected behavior or expose the bug.
  4. Implement the Fix — Make code changes to pass the new tests.
  5. Test the Fix — Run new tests and validate the fix works.
  6. Execute Regression Tests — Ensure all existing tests still pass.
  7. Update Test Coverage — Generate and update coverage reports.
  8. Document in an Activity Report — Create a report following the established format (see examples in docs/activity_report/).
  9. Update Related Documentation — Review and update chapters 1, 2, and 3 if the fix impacts architecture or procedures.
  10. Formalize as a Project Rule — Verify this plan remains codified and updated in Chapter 3.

3.6.2. GitHub Issue Template

Every issue should be concise and state only the initial observation (no investigation or diagnosis):

## Bug Title

### Constat

Brief description of the symptom or failure observed in production.
Include error messages, dates, and reproducibility information if available.

---

(Investigation and solution details will be documented in the PR and activity report.)

3.6.3. Branch Naming Convention

  • Bug fixes: fix/<short-description> (e.g., fix/ticker-column-length-bug)
  • Features: feature/<short-description>
  • Refactors: refactor/<short-description>

3.6.4. Writing TDD Tests

New tests should be added to the appropriate test file and: - Use descriptive test IDs (e.g., tc-pipe-ticker01). - Include a docstring explaining what is being validated. - Validate both the happy path and edge cases (e.g., maximum length, empty values, invalid types).

Example structure: 

@pytest.mark.test_id("tc-pipe-ticker01")
def test_insert_ticker_with_reasonable_length(self, clean_db):
    """Validates that a ticker of reasonable length inserts successfully."""
    # GIVEN: setup
    # WHEN: action
    # THEN: assertion

Once the tests are written, their documentation inventory lets readers find the scenarios covered. To synchronize authentication only, without rewriting other inventories:

python scripts\sync_tests.py --feature Authentication --language fr
python scripts\sync_tests.py --feature Authentication --language en

Sources are in test_cases\authentication\; title_en contains translated titles. Generated pages are docs\fr\tests\authentication.md and docs\tests\authentication.md. Active status means a matching pytest marker, not an execution result or coverage percentage. Validate tests separately; skipped MariaDB variants are not successes.

3.6.6. Checklist Before Merging

This final list brings together the evidence and documentation produced during diagnosis, correction, and review.

  • GitHub issue created with clear constat
  • Dedicated branch created and pushed
  • TDD tests written and passing
  • Bug fix implemented and working
  • All regression tests pass (40+ tests)
  • Test coverage reports updated
  • Activity report documented
  • Related documentation chapters reviewed and updated
  • Commit messages follow "Conventional Commits" standard
  • Pull Request reviewed and approved
  • Merged into main and scheduled for production deployment