Appendix D — Capstone Brief and Rubric
D.1 Scope
The core system requirement: a working system that calls at least one real tool. Real means it reaches something outside the model — an API, a database, a file system, a search index. A tool that returns a hard-coded string is a mock, and a project built entirely on mocks cannot demonstrate the thing this course is about. (Your tests should absolutely use fixtures — that is what makes them fast and repeatable. The requirement is on the demo and the eval run, not on every code path.)
Does it have to be an agent loop? The course’s learning objective is tool-using systems you can evaluate, so an agent loop is the default and the shape most projects take. A fixed pipeline qualifies when the control flow is genuinely not data-dependent — Chapter 23’s own criterion — and you say so and defend it. That is the honest version of the argument this book makes; building a loop you do not need, to satisfy a rule, is the failure mode the requirement exists to prevent. Clear it with your instructor in week 5 rather than discovering it at the demo.
Everything else is negotiable, and smaller is better. One task done properly beats three demonstrated.
Explicit non-goals, because over-scoping is the single most common way a capstone fails:
- You are not training a model. Fine-tuning is allowed and never required. No marks are attached to it.
- You are not building a product. No authentication, no deployment, no user interface beyond what you need to demonstrate the system.
- You are not expected to beat a benchmark. A measured negative result, honestly reported, scores as well as a positive one.
- You are not writing a survey. The deliverable is a system and an argument about it, not a literature review.
Team size is one to three. Solo projects are allowed and are graded on the same rubric with the same expectations — the scope should shrink accordingly, and shrinking it is your decision to make and to justify.
D.2 Academic integrity and LLM use
You may use LLM assistance, and you must declare it. This is a course about building systems on models; pretending you did not use one would be absurd. What is graded is your reasoning, and reasoning you did not do cannot be graded.
Every submission carries an AI-USE.md at the repository root, one row per use:
| what | which tool | what you checked afterwards |
|---|---|---|
| drafted the retrieval wrapper | name it | ran it, read every line, wrote the two tests below |
Permitted, subject to the declaration requirement above. Code completion, refactoring, debugging, explaining an error, drafting boilerplate, translating prose, formatting tables.
Allowed with declaration. Generating candidate eval items — you must then label them yourself and say what fraction you rewrote — and drafting report sections you subsequently edited.
Not allowed. Generating your ground-truth labels and using them as ground truth; reporting measurements you did not run; writing the analysis of numbers you did not read. A number you did not produce is not evidence you can defend, and the demo will ask.
Every team member must be able to explain every line of the submission. Being unable to explain a component is graded as that component being absent, which is a mark deduction. Fabricated measurements are a different thing entirely and go to the department under the university’s academic honesty rules.
D.3 Deliverables
Four artefacts, submitted as one repository plus a short demo.
1. The repository. A layout that someone else can run:
README.md what it does, how to run it, what it needs
src/ the system
evals/ the eval set and the harness that runs it
report.md the written argument (below)
requirements.txt or equivalent, with versions pinned2. The evaluation harness. The second thing every project must have, and it is not a test suite for your code. It is a set of cases with known answers, a scorer, and a script that runs them and prints a number (Chapter 30, Chapter 35). Ship it alongside the system, not afterwards.
Size it deliberately, and say what your size can and cannot see. Chapter 22’s arithmetic is the guide: against an 80%-accurate system, with two independent samples at 80% power, 30 items cannot reach that power even for a perfect replacement, 100 items detects 13.4 points and 1,000 detects 4.8. A paired design on the same items can do better, by an amount that depends on how often the two versions disagree. So there are two legitimate things to build and you must say which one you built:
- A small suite — up to about fifty items, with forty as the week-8 minimum — is a failure-mode finder. A real and useful artefact — read every output by hand and you will find categories of wrongness no aggregate would surface. It can sometimes expose a large paired difference too, but report it as what it was built for: this suite finds broken; it was not sized to rank two versions.
- A suite sized by a power analysis and scored paired — the same items through both versions — is what lets you claim a regression or an improvement of the size you care about. For a few points on a paired design that is usually a hundred items or more; base the number on your metric and the disagreement rate you expect, not on this line.
Full marks on the evaluation axis need your report to name which of the two you built and to state the smallest difference it could have detected.
3. The report. Three to five pages, and its structure is the rubric below: what problem, what you built and what you rejected, what you measured, what it says, and what you would do next. Include your non-goals.
4. The demo. Ten minutes: the system doing the thing, one failure case, and your headline number. Live or recorded — recorded is safer and equally acceptable.
How to submit. One private repository, with the instructor and assistant added as collaborators; make it public only after grades are posted and only if every team member agrees. Submit the repository URL and the demo link through the course’s usual channel, before the deadline your instructor announces for week 14. Recorded demos go somewhere the link works without an account.
Late work. The three intermediate milestones cannot be made up: they exist to stop a project failing at week 13, and a late one has already failed at that job. The final submission is graded down per started day at whatever rate your instructor sets — confirm it in week 1 rather than in week 14.
D.4 Rubric
Non-CS students are graded on reasoning, not on code elegance. A clear argument for a simple system, with a measurement that could have refuted it, beats an elaborate system nobody can evaluate.
| axis | weight | what full marks look like |
|---|---|---|
| Problem framing | 15% | The task is stated precisely enough to be wrong. Non-goals are written down. You can say who would use this and what they would do instead. |
| Data and ground truth | 20% | You can say where every item came from and how you know its correct answer. Annotation disagreement is measured, not assumed away (Chapter 5). |
| Evaluation design | 25% | A real eval set, a scorer you validated on a known-answer input, a baseline to compare against, and uncertainty reported (Chapter 22). |
| Engineering | 20% | It runs from the README on a machine that is not yours. Tools are real. Failure modes are handled rather than hidden. |
| Write-up | 20% | The argument is followable, the claims match the evidence, and the negative results are in there. |
That column describes full marks only, which is not a scoring instrument — it is the failure Chapter 5 warns about, applied to the thing doing the grading. So each axis is scored in four observable bands. The heaviest axis in full, and the others follow the same shape:
| band | evaluation design, 25% |
|---|---|
| 23–25 | A paired eval set sized by a stated power analysis, or a 40–50-item failure-mode suite with the limitation stated explicitly. Scorer validated on a known-answer input with the log included. A baseline reported. An interval, or a stated minimum detectable difference. |
| 18–22 | The eval set is real and the scorer is validated. A baseline is present. Uncertainty is mentioned but not quantified. |
| 13–17 | An eval set exists, but there is no baseline or no scorer validation, and the number is reported as though it were exact. |
| 0–12 | No eval set — or a set written after the system, whose items were chosen because the system passes them. |
Two markers score each submission independently on these bands. Disagreements of more than one band are resolved by discussion, and the disagreement rate is reported back to the class — because that is exactly the standard Chapter 5 asks of you, and a rubric nobody validated is a metric nobody validated.
Four things that cost marks disproportionately, all of them avoidable:
- A metric nobody validated. Feed your scorer a perfect prediction and confirm it returns a perfect score. Then feed it the cases that actually break scorers: a wrong answer, an empty string, a missing field, unparseable output, and — if your data is Turkish — a correct answer with
ı,şandİin it. A scorer that silently returns zero for unparseable output and a scorer that crashes are two different systems, and you should know which you have. Chapter 16 is a chapter’s worth of reasons. - A claim without an interval. “Our agent resolves 80% of tickets” needs a set size and a definition of resolved.
- No baseline. Compared against nothing, any number is unreadable. A majority-class floor or a keyword rule takes an hour.
- A demo instead of a measurement. Three impressive examples is an anecdote.
Two things that earn marks more than students expect: reporting what did not work, and finding a bug in your own instrument. Both are the behaviour the whole book argues for.
D.5 Timeline
Anchored to the week numbers in Appendix E so the appendix survives a change of semester.
| when | what is due | what it should contain |
|---|---|---|
| Week 5 | Proposal, one page | The task, the tool you will call, where ground truth comes from, and your non-goals |
| Week 8 | Eval set, submitted | Forty items minimum with known answers, the scorer, evidence you validated the scorer, and one sentence naming the smallest difference this set can resolve |
| Week 11 | Mid-point check-in | A working end-to-end path, however crude, and a first number |
| Week 14 | Repository, report, demo | Everything above |
The week-8 deadline is deliberately before most systems work. Submitting the eval set early keeps the project focused and makes progress assessable — in this course’s experience it is what separates a project that knows whether it is working from one that finds out in week 13 — and it is the habit Chapter 39 argues for at production scale.
D.6 Project shapes that work
Anonymised, and described by shape rather than by result so you can see whether yours rhymes with one.
A support agent over a real document set. Retrieval over a company’s own policy PDFs, one lookup tool, evaluated on questions written by someone who knew the answers. Worked because the ground truth existed before the system did.
A structured-extraction pipeline for Turkish forms. Invoices to JSON, scored per field against hand-checked values. Worked because per-field scoring showed which field was failing, which is the difference between a number and a finding. Note the shape: this one is a pipeline rather than a loop, and it cleared the requirement above by calling a real document store and by arguing — in the report — that nothing in the task branches on what an earlier step returned.
A literature triage agent. Given a research question, search, filter, summarise. Worked because the student measured retrieval separately from summarisation and could say which half was the problem.
A code-review assistant on a real repository. Worked because the baseline — a linter — was honest and hard to beat, and the report said where it was not beaten.
And two shapes that did not work, with the reason:
A multi-agent system with five roles, none of which could be evaluated separately. There was no way to tell whether the orchestration helped, because there was nothing to compare it against.
A chatbot with no eval set, demonstrated on examples chosen after the fact. The system may well have been good; nothing in the submission could establish it.
Both failed the same way, and it is the failure this book was written against: a system nobody can measure is a system nobody can improve, defend, or trust.