Annotation Tools: Argilla, Label Studio, Scripts

~20 min · annotation, argilla, label-studio, human-review

Level 0Observer

0 XP0/43 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Pick the smallest tool that solves the problem

Three sane options, ordered from heavy to light.

Argilla (recommended for LLM data)

Purpose-built for LLM data. First-class support for preference ranking (DPO/ORPO), text classification, span tagging, and model-suggested labels. Argilla v2 organizes around datasets with configurable fields and questions, hosted as a Hugging Face Space or self-hosted.

Label Studio

General-purpose annotation tool covering text, image, audio, video, time-series. Heavier than Argilla, but the right pick when one team annotates several modalities.

Python script + CLI prompt

For projects under a few hundred examples, a 30-line CLI tool that lets one human approve, reject, or rewrite each example is more practical than a full annotation platform. The right tool is the one your reviewer will actually use.

Code

Argilla v2 dataset setup for LLM SFT review·python

import argilla as rg

client = rg.Argilla(api_url="http://localhost:6900", api_key="admin.apikey")

dataset = rg.Dataset(
    name="sft-review",
    workspace="default",
    settings=rg.Settings(
        fields=[
            rg.TextField(name="system"),
            rg.TextField(name="user"),
            rg.TextField(name="assistant"),
        ],
        questions=[
            rg.LabelQuestion(name="verdict", labels=["keep", "rewrite", "reject"]),
            rg.RatingQuestion(name="quality", values=[1, 2, 3, 4, 5]),
            rg.TextQuestion(name="improved_assistant", required=False),
        ],
    ),
)
dataset.create()

records = [
    rg.Record(fields={"system": ex["messages"][0]["content"],
                       "user":   ex["messages"][1]["content"],
                       "assistant": ex["messages"][2]["content"]})
    for ex in raw_examples
]
dataset.records.log(records)

30-line CLI reviewer for the in-between case·python

import json

def review(in_path: str, out_path: str) -> None:
    with open(in_path) as f:
        examples = [json.loads(line) for line in f]
    approved = []
    for i, ex in enumerate(examples, 1):
        print(f"\n--- {i}/{len(examples)} ---")
        for m in ex["messages"]:
            print(f"[{m['role']}] {m['content'][:240]}")
        choice = input("(a)pprove / (r)eject / (e)dit / (q)uit: ").strip().lower()
        if choice == "a":
            approved.append(ex)
        elif choice == "e":
            new = input("Rewrite the assistant response:\n")
            ex["messages"][-1]["content"] = new
            approved.append(ex)
        elif choice == "q":
            break
    with open(out_path, "w") as f:
        for ex in approved:
            f.write(json.dumps(ex) + "\n")
    print(f"approved {len(approved)}")

External links

Exercise

Pick the approach that fits your project size and stand it up end to end with 30 real examples. If you choose Argilla, deploy it locally or as a Space and invite one teammate to annotate. Record the time to review all 30 — that's your throughput estimate for scaling.