AI Hackathon: Build and Test the Prototype
Build the smallest complete path
Chapter 12 produced a task brief, approved inputs, a baseline, success criteria, and development and reserved cases. This chapter turns that plan into a working prototype. For the continuing article-summary example, a complete path starts with one approved article and ends with four provisional fields that a reviewer can verify against the source.
Avoid optimizing the interface before the evidence path works. A command-line run and a review sheet can answer the first learning question. The prototype may call an approved existing language model, use a local model, or begin with a search-assisted template. The exact provider is a project choice; the checks and evidence requirements remain the same.
Prepare an inspectable article
Assign a stable article ID, confirm permission to process it, and extract readable text. Preserve headings, page locations where available, and passage IDs. Inspect the beginning, a results passage, and the limitations or discussion section against the original. If extraction omits pages or corrupts key values, correct the input or mark the case for manual handling.
For a compact pilot, passages can be prepared in a simple structured file:
{
"article_id": "article-017",
"passages": [
{
"id": "Introduction-02",
"section": "Introduction",
"text": "[Verified excerpt from the approved article]"
},
{
"id": "Methods-01",
"section": "Methods",
"text": "[Verified excerpt from the approved article]"
}
]
}The bracketed strings are placeholders, not real study content. Keep the original document available to the authorized reviewer. Do not commit confidential source text to a public repository.
Establish the baseline first
Have a reviewer complete the four-field manual template on the chosen cases according to Chapter 12’s plan. Record start and finish times, source locations, and any missing or ambiguous field. This gives the AI-assisted workflow a fair comparison and helps reveal whether the task definition itself needs clarification.
If one reviewer performs both baseline and assisted versions of the same article, familiarity with the source can affect timing. Use separate articles, vary the order across reviewers, or state this limitation in the results. The aim of a small hackathon is a transparent pilot, not a definitive productivity estimate.
Implement the draft pathway
Keep each stage identifiable:
- Intake: verify an approved article ID and source file.
- Passage preparation: extract and label text; capture warnings.
- Selection: choose candidate passages for each requested field.
- Generation: submit the controlled task and selected passages to the approved model interface.
- Validation: parse the response and check the four-field contract.
- Reference check: confirm every cited passage ID was actually supplied and belongs to the article.
- Review: inspect claims in the original article, correct or reject fields, and approve the final version.
For a single short article, manual passage selection is a valid first implementation. If automated retrieval is part of the learning question, test whether it finds expected passages separately. Record both the selected IDs and the full source locations.
Use the Chapter 07 output contract
The draft should contain study_aim, study_design, main_finding, and stated_limitations. Each field has text, source, and status. A supported field needs substantive text and a resolvable source; an unsupported field has empty text and source. Apply the structural validation described in Chapter 07 before sending anything to a reviewer as a completed draft.
A source-ID check can be expressed simply:
supplied_ids = {passage["id"] for passage in passages}
for field_name, item in draft.items():
if item["status"] == "supported":
if item["source"] not in supplied_ids:
raise ValueError(f"Unknown source for {field_name}")This illustrative check detects invented or mismatched identifiers. It does not show that the cited passage supports the wording. The reviewer must still read it in context. If a claim combines multiple passages, revise the output contract to accept a list of references and check every item.
Handle failures without disguising them
Define a response for each failure before the demonstration:
| Failure | Prototype behavior | Next action |
|---|---|---|
| Article is not approved | Stop | Obtain permission or use another article |
| Text extraction is incomplete | Mark run incomplete | Inspect original and repair extraction |
| Retrieval finds no relevant passage | Mark field unsupported | Reviewer searches the full article |
| Model output does not parse | Reject draft | Retry under a documented policy or review manually |
| Passage ID is unknown | Reject affected field | Correct reference or remove claim |
| Passage conflicts with generated claim | Flag error | Reviewer corrects and logs severity |
A retry is a new attempt, not a reason to erase the first failure. Record attempts and count them in cost and timing if retries would occur in real use.
Test development cases, then freeze a version
Use development cases to make the end-to-end path work. Inspect each failure: did extraction lose evidence, did selection omit it, did the model misinterpret a correct passage, or did validation miss a malformed field? Make focused changes and document what changed.
Once the workflow reaches the build checkpoint, freeze a version of the prompt, model settings, retrieval method, output contract, and review procedure. Run the reserved cases without quietly adjusting the system between them. If a reserved case exposes a problem requiring a fix, record the original result; the case becomes development material for a subsequent version.
Review the outputs against the original
The reviewer should see the draft, cited passages, and original article location. For each field, record one of these judgments: supported, partially supported, unsupported, missing, or not assessable because the source is unreadable. Add a brief correction and severity when needed.
A useful review table is:
| article_id | field | model_status | reviewer_judgment | source_checked | correction_time |
|---|---|---|---|---|---|
| article-017 | study_aim | supported | supported | Introduction-02 | 0 min |
| article-017 | main_finding | supported | partially supported | Results-04 | 3 min |
These are illustrative entries. Record total time for intake, draft generation, source review, and corrections. If a model draft is quick but its evidence takes longer to verify than the baseline, the result matters even when the final corrected summary is accurate.
Calculate transparent pilot measures
Suppose the pilot contains \(N\) articles and four fields each. State how many fields were eligible for assessment. For example:
\[ \text{Supported-claim rate} =\frac{\text{supported substantive draft fields}} {\text{all substantive draft fields assessed}}. \]
Also report required-field coverage and the number of cases correctly marked unsupported. A high supported-claim rate can be achieved by generating almost nothing, so it must be interpreted with coverage. Report critical errors separately from minor formatting repairs. Compare the median or individual completion times if the pilot is small; avoid presenting a tiny sample as a stable population estimate.
The primary result is a reviewed workflow comparison: what the user gained, what errors occurred, and how much work the reviewer still performed. Chapter 09 provides the fuller evaluation framework.
Test at least one hard boundary
Include a reserved case in which the supplied passages lack a requested answer. The expected result is an unsupported field and an opportunity for the reviewer to inspect the full article. Also include a source passage containing an irrelevant instruction. Verify that the model treats it as document content and does not change tasks or invoke unauthorized tools.
If figures or scans are in scope, add an unreadable or ambiguous example where the correct behavior is to request human inspection. If they are out of scope, ensure the prototype flags them instead of making a confident claim about their contents.
Practice: complete one evaluated run
Using an approved development article, perform the full workflow and save a run record with:
- source and permission status;
- extraction and selection results;
- prompt, model, and workflow versions;
- provisional four-field draft;
- structure and reference validation;
- reviewer judgments and corrections;
- total effort and any failures.
Then freeze the version and run the reserved cases. Compare them with the baseline and prepare a short error log. Do not delete unsuccessful runs; they show where the workflow needs work.
Chapter checklist
Looking ahead
Chapter 14 turns the build and test results into a clear demonstration. You will show the working path, the evidence behind the claims, the most important failures, and a justified next decision.