Skip to content

Documentation test coverage report

Ticket #111: Creating the test coverage report for the project documentation
Type: Documentation / Quality / Governance
Affected Component: docs/tests/documentation.md, docs/fr/tests/documentation.md, docs/tests/index.md, docs/fr/tests/index.md, mkdocs.yml, tests/test_docs_config.py, tests/test_activity_report_governance.py

1. Context and objective

The site already published coverage reports for all major application features (authentication, dashboard, demo, pipeline, stock detail), but a dedicated coverage report for the project documentation was missing.

The goal of this session was to close that gap with a rigorous and traceable approach:

  • inventory existing tests that cover the documentation;
  • distinguish tests exclusively targeting documentation from mixed tests;
  • create a dedicated documentation coverage report following the existing template;
  • integrate the new report into the EN/FR site navigation;
  • recalculate and update the global coverage rate after adding this newly covered feature.

2. Work performed

Inventory and categorization of documentation tests

Two test blocks were identified as relevant to /docs:

  • tests/test_docs_config.py (6 tests) covering index structure, report ordering, MkDocs configuration, tags, and header consistency;
  • tests/test_activity_report_governance.py (1 parameterized test) covering tag governance rules across all activity reports.

Categorization result:

  • documentation-only tests: 7/7;
  • mixed tests (documentation + application logic): none.

Creation of the documentation coverage report

Two dedicated pages were created following the same format as existing reports:

  • docs/tests/documentation.md;
  • docs/fr/tests/documentation.md.

The content lists 7 active cases out of 7 defined, yielding a functional rate of 100% for documentation.

Site integration and navigation

The new report was added:

  • to the EN and FR test report indexes (docs/tests/index.md, docs/fr/tests/index.md);
  • to the EN and FR main navigation in mkdocs.yml.

Global coverage rate update

After consolidating all covered features, the global rate was recalculated and updated from 68% to 88% in both EN and FR indexes.

Legend harmonization

The "Obsolete" legend entry was removed from all EN/FR test reports, in line with the governance decision: obsolete tests are retired rather than kept in a passive status.

3. Validation and outcome

Operational validation performed on the updated artifacts:

  • documentation report present in EN and FR;
  • links added in both test report indexes;
  • navigation entries present in mkdocs.yml (EN + FR);
  • global coverage rate aligned at 88%;
  • legends unified without obsolete status.

Business outcome:

  • documentation is now treated as a fully tested feature in its own right;
  • the coverage picture is complete and more readable for project oversight;
  • quality governance is better aligned with the actual practice of test maintenance.

4. Lessons learned (including AI agent evaluation)

What worked well with the AI agent

  • Upfront framing (reformulation + validation before execution) reduced ambiguities from the start.
  • The agent enabled a fast, structured inventory of tests touching /docs, with immediately usable categorization.
  • Documentary changes (creation + integration + harmonization) were executed sequentially, with intermediate verification.

Improvement areas observed

  • Some decisions evolved during the session (scope and treatment of mixed tests), adding avoidable iterations.
  • The AI coach agent, at the time of evaluation, lacked enough detailed session events to produce a fine-grained token efficiency diagnosis.
  • The distinction between "drafting the report" and "publishing to navigation" could have been stated earlier to reduce clarification exchanges.

Adjustments retained for future sessions

  1. Formalize the brief with 4 mandatory fields upfront: objective, scope, exclusions, expected deliverable.
  2. Maintain explicit hypothesis validation before execution for multi-file topics.
  3. Continue using the coach agent in session-by-session mode, but progressively enrich useful events (prompt, action, validation) to improve coaching quality.