Mamta Motwani
Independent ProjectSystems

Curriculum QA Suite

Pedagogical rules turned into pass-or-fail release checks

Checks
61
Runs against
The finished product, not the material behind it
Products locked
83, read-only once sold
Defects caught
2 that would have shipped silently

What this case study claims

Design decisions, intended behaviour and the reasoning behind them, plus counts verified against the built product. It does not claim learner outcomes, adoption or improvement: those have not been measured, and describing an intention as a result would be the easiest thing on this page to get wrong.

Overview

Sixty-one automated checks that turn curriculum rules into pass-or-fail tests and run them against the finished pages a child will actually be served, rather than against the material those pages were made from. The checks gate releases: a unit that fails one does not ship. They cover every question the practice activities can produce, exercised at volume; a grade-level ceiling with each rule tied to a specific curriculum expectation; and checks on arithmetic, the promises a page makes about itself, content overlap between units, layout accessibility, product completeness and page weight.

Problem

A curriculum product ships as hundreds of pages produced automatically, and manual proofreading does not scale to that volume. It also does not find the defects that carry the most risk: a question the practice activity can produce but the game cannot accept, content that drifts above the target grade level, a page that states a promise the product does not deliver. Each of these looks correct on the page where it appears and is only detectable across the full set. Review capacity was the binding constraint, and adding reviewers would not have changed the class of defect being missed.

Audience

Internal. This is the production pipeline standing behind every product in this portfolio — but it is also the clearest statement of how I think curriculum work and machine generation should fit together.

My role

  • Designing what a curriculum rule looks like when it is written as a pass-or-fail check
  • Setting the grade-level ceiling and defining what counts as being above it
  • Deciding which failures stop a release and which are advisory
  • Writing the rules against the Ontario expectations they encode

Goals

What the design set out to achieve. These are objectives, not results.

  • Make curriculum constraints checkable rather than merely stated
  • Check the artefact the child sees, not the source that produced it
  • Find the defect that appears once in five hundred rounds, before a child does
  • Fail the release rather than the reviewer’s attention

How it works

Every check runs against the finished product rather than the material behind it. The practice activities are exercised at volume — over a hundred thousand rounds each, at every difficulty level — and what is read back is the question as it appears on the page, not the intention behind it. The grade-level ceiling is an explicit list of what Grade 3 work should not contain — inequality symbols beyond “not equal to”, whole numbers above one thousand, nested loops in the coding strand — and every item on that list carries the curriculum expectation it is derived from, so the reading behind it can be argued with.

Design decisions

Check the finished product, not the material behind it

The material can be right and the finished page still wrong — a digit lost somewhere between the two — and only the page shows it. Checking the material verifies what was meant; checking the page verifies what was sold.

Curriculum rules written as pass-or-fail checks

“Grade 3 does not use inequality symbols” is a reading of the curriculum, not a fact stated in the Ministry document. Writing it down as a check with the expectation code attached makes the reading explicit, reviewable and arguable — which is much better than leaving it as an assumption in someone’s head.

Volume instead of inspection

Over a hundred thousand rounds per activity, because a defect that surfaces in one round in five hundred will certainly reach a child and will never reach a reviewer. Sampling is not a strategy against material that is made fresh each time.

Pages are held to their own promises

A check reads what a page says about itself — no decimals, whole numbers only, no regrouping — and verifies the page keeps it. Promises are the most common thing to break during a revision and the least likely thing to be re-read.

The sales copy is checked against the product

One unit passed every check on the product itself and still could not be released, because the description was wrong: it claimed support for any browser and for tablets when the unit’s own testing report said one browser and no tablet hardware, described a one-page adult guide that ran to three pages, and carried page counts two rebuilds out of date. That is the wording a dissatisfied reader has every right to complain about. So every number and platform claim in the accompanying copy is now re-derived from the build rather than trusted — because each of those was a sentence typed once and never checked again.

Gating, not reporting

The suite blocks a release rather than producing a report someone might read. A quality process that depends on attention will eventually be run on a day when attention is elsewhere.

AI’s role

This suite is the clearest statement in the portfolio of what I think AI is actually for in curriculum work: produce at volume, then verify mechanically, and never let the thing that made the material be the thing that judges it. AI did a substantial share of the work of building the checks. It is deliberately not trusted to be the check — what each one asserts is a curriculum judgement made by hand, and the checks are run against the real product rather than by asking a model whether the output looks right.

Human judgment

Deciding what the grade-level ceiling actually is. The Ontario expectations do not say “no inequality symbols in Grade 3”; that is a reading of them, it is mine, and it has to be defensible to a teacher who disagrees. The same applies to which defects should stop a release and which are worth noting: that is a judgement about what a buyer would consider broken, and it is not something the checks can decide for themselves.

Experience

  • validate_gens
  • check_grade_boundary
  • check_claims
  • check_arithmetic
  • check_overlap
  • check_layout_a11y

61 checks · run against shipped HTML · a failure stops the release

This one has no interface. It runs at the command line and its output is a pass or a refusal to release, so there is nothing to photograph. The checks themselves are the artefact.

Challenges

False confidence is the real hazard. A green suite tells you only that the things you thought to check are fine, and the interesting failures are by definition the ones not on the list. The grade-level ceiling is a list of things I thought of. Keeping that honest means adding a check every time something gets through, and resisting the feeling that a passing run is evidence of quality rather than evidence of coverage.

Iteration

The suite grew defect-first rather than from a coverage plan, and each check records why it exists. Three of the defects it was built around are worth naming, because they are concrete and not one of them was ever going to be found by reading: an activity that showed the child measurements in decimal centimetres in 88% of rounds, on a page that promised whole numbers only; a game with a control that play could never actually reach; and a release that recorded one version of a product while shipping another, so the record and the product did not match.

Reflection

This is the one place in the portfolio with something close to a measured result, and it is worth being precise about what it is. Defects caught are counted, not estimated — those three are real and they were found by the suite. What is not measured is anything downstream: whether the products are better in a classroom for having been gated this way. The defensible claim is that specific, identified defects were caught by machine rather than by reading. That is a smaller claim than “higher quality”, and it is the one the evidence supports.

Skills demonstrated

  • Quality design for generated content
  • High-volume automated checking
  • Turning curriculum rules into checkable requirements
  • Verifying the finished product rather than the source
  • Release gating and quality discipline
  • Accessibility checking

Read next