Collect human training data for AI

Define one learning task

Suppose a support assistant sends refund requests to the wrong queue. The first data requirement might be human topic labels for support messages. If routing is already correct but its replies are poor, you may instead need expert-written replies. These are different datasets and should have different instructions.

Write down the input, the expected output, the intended population and how you will evaluate success. Use examples from material you have permission to process. A synthetic pilot tests the collection interface; it does not establish performance on real customer traffic.

• Classification: support message in, an allowed topic label out.
• Demonstration: scenario and relevant policy in, a human-written answer out.
• Correction: scenario and candidate answer in, an expert revision out.
• Evaluation: task and model output in, a judgement against a rubric out.

Make the labelling instructions testable

For a routing dataset, define Billing, Technical problem, Feature request and Other. Explain how to label a message that mentions both an invoice and a broken feature. Include a few boundary cases with explanations in your internal instructions, then ask pilot contributors to explain their interpretation.

Let people skip items they cannot judge. Decide whether ambiguity should become an explicit label, a review flag or an exclusion. A forced confident answer can hide a task-design problem.

For expert-written answers, state the facts the writer may rely on, the audience, acceptable length and what to do when information is missing. Ask contributors to declare AI assistance if that matters to the project. A declaration records what they report; it does not prove unaided authorship.

Build a pilot through your connected agent

Use this prompt for a synthetic interface pilot. Replace the fictional examples with your own authorised source items before gathering production labels. Keep stable item identifiers so you can join labels back to their inputs.

Plan coverage, not just an answer count

Choose the languages, topics and difficult cases that matter to deployment. Keep a record of where the source items came from and which groups are underrepresented. A thousand labels on near-identical examples may add less useful coverage than a smaller, varied set.

Tellseed separates the item pool from the number each contributor sees. You can use a larger pool with short rounds and choose sequential, shuffled or coverage-weighted ordering. Coverage-weighted ordering favours less-answered items; it is not a guarantee of a fixed number of independent annotators per item.

For overlapping judgements, inspect per-item coverage and recruit enough independent contributors. Distinguish collection labels from the later review decisions on each submission. Two reviewers assessing one answer are not the same as two people independently labelling the original item.

Review the pilot before scaling

Have someone with the relevant expertise inspect errors and disagreements. Check whether the instructions, the source material or the contributor’s understanding caused each problem. Revise the draft if needed, approve the new version and retain the version references when analysing results.

Estimate the workload using submitted answers and review effort separately. Twenty people answering ten items produce up to 200 submitted answers. That does not mean 200 unique source items or 200 accepted examples.

Before the wider collection, explain the intended use of contributions, minimise unnecessary personal data and establish your permissions and retention process. Do not infer training or redistribution rights just because a response was submitted.

Keep evidence with the dataset

Export the raw data for audit and the reviewed data for the next preparation stage. Preserve the campaign version, item identifiers, review provenance and manifest. Record which material is synthetic and which responses came from real contributors.

Document your recruitment method, task instructions, acceptance criteria, exclusions and intended uses. Hugging Face dataset cards provide a useful structure for explaining these decisions. A downloadable template below gives you a starting point; complete it with evidence from your own campaign.

Tellseed collects and exports human input. Model training, provider-specific formatting and evaluation of any resulting model happen in your own pipeline.

• Dataset card guidance from Hugging Face
• Campaign brief template
• Dataset card template