Run a human LLM evaluation campaign

Choose the decision the evaluation supports

For a support assistant, a useful decision is whether a new prompt produces replies that follow the supplied support policy. Select representative tasks, keep the policy text with each task, and record the model and prompt versions that generated the candidate responses.

Use a fixed task set for the comparison. Keep it out of the training data and avoid repeatedly tuning on its results. A small pilot reveals confusing instructions; it is not sufficient evidence of general model quality.

Use a rubric with observable criteria

For policy adherence, define three levels: 1 means the answer contradicts or invents policy; 2 means the answer omits information needed for the decision; 3 means the answer follows the supplied policy and answers the request. This is an illustrative rubric, not a validated benchmark.

Assess correctness and writing style separately when both matter. A fluent answer can still be wrong. Let a reviewer say they cannot judge when the provided evidence is insufficient, and inspect those cases rather than turning them into low scores.

Tellseed supports a rating for a single dimension or a flat structured response with fields such as an enum judgement and a written reason. Preview the controls before publishing. Long documents must be shortened to fit the item context rather than silently truncated by your preparation process.

Collect A/B preferences without hiding the setup

For a simple preference task, put response A and response B in the same item context. Use single-choice answers A, B, Tie and Neither, and tell contributors whether they should prefer correctness, helpfulness or another criterion.

Tellseed can collect these choices using its standard campaign controls. It does not provide a native side-by-side comparison widget, blind model assignment or automatic A/B order balancing. Prepare and record the assignment yourself if those controls are required by your study.

Store the original prompt, both candidate responses and their model versions in your own source table keyed by the campaign item identifier. A preference label by itself is not a complete DPO training pair. Ties and Neither need an explicit handling policy; do not silently convert them into a winner.

Calibrate with people before inviting everyone

Ask a small group with relevant expertise to judge the same examples independently. Discuss disagreements after the independent judgements are recorded. Improve ambiguous definitions and run a fresh pilot when the instructions materially change.

Recruit for the task. A technically complex answer may need domain expertise; a wording preference may need people from the intended audience. Tellseed supplies the collection page, not a vetted participant panel.

After preview and human approval, publish the campaign, complete a test round and verify its answers in the owner view. Share only the contributor link with your evaluators.

Report what the collected evidence supports

Report the number of source items, submitted judgements, unique contributors, skipped items and usable reviewed answers separately. Compare model versions on equivalent tasks and examine disagreements and failure categories alongside aggregate scores.

Tellseed provides campaign statistics and exports. Inter-rater agreement, confidence intervals, model-comparison analysis and significance testing require your analysis workflow. Multiple judgements on one item are not automatically independent observations.

Keep the collection instructions, source table, campaign version and export manifest with the evaluation report. If you later use the judgements for training, reserve a new independent evaluation set.