19. Exercises, submissions and grades
Because every sentence and cell has a permanent address, the answer to a verification exercise is an address, and an address can be graded.
The three formats of this chapter, exercise, submission and grade, are provisional (0.4): no course has used them yet, and the first that does will shape them.
19.1 Exercises#
An exercise is an annotation of kind exercise whose exercise property holds a scienting-exercise/1: an exercise kind, a prompt, a key and a rule.
| Kind | The student | The key |
|---|---|---|
anchor | Finds the sentence or cell that supports a statement. | elements: each a set of acceptable anchors, with points and feedback. |
check | Decides whether a cited passage supports the citing sentence, and anchors the passage. | A verdict with points, and elements. |
trace | Follows a value back to its original measurement. | A chain: an element per link, in order. |
reproduce | Digitizes a plot and recomputes a value. | A value, a tolerance, a unit, the confirmed digitization it comes from, and points. |
verify | Checks a model's summary, written with addresses, in which errors were planted. | errors: each a set of anchors and the correct verdict, with points. |
The rule's match is same-node (the student's anchor resolves to the node a key anchor resolves to), overlap (and, where the key has a span, the student's span covers at least min_overlap of it; default 0.5), or chain (each link matches in order, with credit per link).
Feedback is {hit, miss} per element.
19.2 Verdicts#
| Verdict | Means | As claim evidence, if a person confirms it |
|---|---|---|
supports | The passage says what the citing sentence claims. | supports |
partly | It says some of it, or with a qualification the citing sentence drops. | qualifies |
does-not | It does not say it, or says the opposite. | disputes, or no record when merely irrelevant; the reviewer chooses. |
cannot-tell | The passage, or the student's access to it, is not enough. | No record. |
19.3 Submissions#
A submission, scienting-submission/1, holds the exercise's ID, the document and revision, a pseudonymous student ID, and answers: anchors, a verdict, a chain of anchor lists, a value, or flags (each an anchor and a verdict), as the kind needs.
The student's name never enters a submission; the mapping from pseudonym to student stays in the course's records.
19.4 Grading#
Before grading, every anchor in the key is resolved, and if any is stale nothing is graded with it.
Each element of the key earns its points when any of the student's anchors matches any of its anchors under the rule; a check earns its verdict's points when the verdicts are equal; a reproduce earns its points when the value is within the tolerance; a verify earns an error's points when a flag matches it with the correct verdict.
Points are decimal strings, summed exactly.
The result is a scienting-grade/1: the submission, the exercise, the score, the max, and an item per element with its points, its maximum and its feedback.
grading.jsonl grades a hit, a miss and an overlap.
Student submissions are educational records, so they live in the course's private store, never on a public log.