How to use Scorra

Scorra is a quality evaluation platform for AI applications. It helps teams turn human and AI judgment into measurable, auditable evidence — so every model release comes with numbers, not vibes.

You upload datasets of prompts, create evaluation tasks, and collect structured judgments: single scores, pairwise comparisons, or rankings. Results feed analytics, agreement metrics, and exports that plug into your existing workflows.

When you sign up, Scorra creates an organization for you. An organization is the top-level container for your datasets, tasks, members, and evaluation results.

You can create multiple organizations if you work across different teams or clients. Use the organization switcher in the sidebar to move between them.

Organization owners can invite members and assign roles: Owners manage the organization and billing, Admins manage members and tasks, and Evaluators complete evaluation tasks.

Every evaluation in Scorra follows the same core loop:

1. 01Upload a dataset — a CSV or tab-separated file with prompts and, optionally, model responses you want judged.
2. 02Create an evaluation task — define what you are measuring: a single score per item, a pairwise A-vs-B comparison, or a ranking of several responses.
3. 03Set your criteria — choose score dimensions like accuracy, tone, or helpfulness, and optional rubric guidance for evaluators.
4. 04Invite evaluators — internal team members or external contractors receive items to judge one at a time.
5. 05Review results — analytics show score distributions, inter-rater agreement, and per-evaluator consistency.
6. 06Export and act — download results as CSV and feed them into your model iteration or deployment approval process.

Scorra supports three evaluation types, chosen when you create a task:

• Single scoring — an evaluator assigns scores to one response per dimension (e.g., accuracy 1-5, tone 1-5). Best for absolute quality judgments.
• Pairwise comparison — an evaluator sees two responses side by side and picks the better one, optionally scoring each dimension. Best for head-to-head model comparisons.
• Ranking — an evaluator orders three or more responses from best to worst. Best for tuning among several candidate models.

Dimensions are the aspects you want evaluators to judge. Common choices are accuracy, relevance, tone, safety, and helpfulness. Each dimension can have its own scale (e.g., 1-5 or 0-10) and optional rubric text that guides evaluators on what each score means.

Rubrics matter. A dimension called "accuracy" without guidance produces noisy data. A dimension with a one-line rubric per score level produces data you can trust and compare across evaluators.

When creating a task you can configure:

• Title and description — what the evaluators will see.
• Evaluation type — single, pairwise, or ranking.
• Score dimensions — what is being measured.
• Items per evaluator — how many items each evaluator receives (used for distributing work and measuring agreement).
• Auto-activation — whether new items are released to evaluators automatically as they finish.

Once a task is active, evaluators pick up items one at a time from their dashboard. You can pause a task at any time to stop new work from being submitted.

The task progress view shows completion percentage, throughput per evaluator, and how many items remain. The disagreement view highlights items where evaluators diverged — useful for finding ambiguous prompts or unclear rubrics.

When the task is complete, results are available for analytics and export.

The organization-level analytics dashboard gives you a snapshot of activity: how many tasks are running, how many evaluations have been completed, and how evaluators are performing.

Agreement metrics tell you whether your evaluators are consistent. Scorra computes agreement scores per task so you can spot dimensions or items where evaluators diverge — often a sign that the rubric needs clarification.

Score distribution charts show how responses are distributed across each dimension's scale. Shifts in the distribution over time can indicate model improvement or regression.

Evaluator throughput and consistency metrics help you manage your reviewer pool — who is keeping up, who may need more guidance, and whether agreement is holding across the team.

All API requests go to https://scorra.onrender.com/api/v1. The API is versioned via the URL path — currently v1.

The full interactive API reference is available at the Swagger docs.

The API supports two authentication methods:

• Bearer token (JWT) — included in the Authorization: Bearer <token> header. Obtain a token by logging in via the Auth API.
• API key — included in the X-API-Key header. API keys are useful for server-to-server integration and long-running jobs.

Most endpoints require a valid Bearer token. Some invitation endpoints are public and do not require authentication.

• POST /auth/register — Create a new user and organization.
• POST /auth/login — Log in with email and password, receive a JWT pair.
• POST /auth/refresh — Refresh an access token using a valid refresh token.
• POST /auth/logout — Revoke the current tokens.
• GET /auth/me — Get the currently authenticated user.
• POST /auth/switch-org/:orgId — Switch the active organization for the session.

• GET /organizations — List organizations for the current user.
• POST /organizations — Create a new organization.
• GET /organizations/:id — Get organization details.
• PATCH /organizations/:id — Update organization settings.
• GET /organizations/:id/members — List members.
• PATCH /organizations/:id/members/:userId — Change a member's role.
• DELETE /organizations/:id/members/:userId — Remove a member.
• POST /organizations/:id/invitations — Invite a user by email.
• POST /organizations/:id/invitations/:invitationId/resend — Resend an invitation.
• DELETE /organizations/:id/invitations/:invitationId — Revoke an invitation.

• POST /datasets — Create a new dataset.
• GET /datasets — List datasets in the organization.
• GET /datasets/:id — Get a dataset by ID.
• PATCH /datasets/:id — Update a dataset.
• DELETE /datasets/:id — Delete a dataset.
• POST /datasets/:id/upload — Upload items to a dataset (file upload).

• POST /evaluations/tasks — Create an evaluation task.
• GET /evaluations/tasks — List tasks in the organization.
• GET /evaluations/tasks/:taskId — Get a task by ID.
• PATCH /evaluations/tasks/:taskId — Update a task.
• POST /evaluations/tasks/:taskId/activate — Activate a task.
• POST /evaluations/tasks/:taskId/pause — Pause a task.
• GET /evaluations/tasks/:taskId/progress — Get completion progress.
• GET /evaluations/tasks/:taskId/disagreement — Analyze evaluator disagreement.
• GET /evaluations/tasks/:taskId/summary — Summarize evaluator feedback.
• GET /evaluations/tasks/:taskId/next — Get the next item to evaluate.
• POST /evaluations/submit — Submit an evaluation.
• POST /evaluations/auto-label — Auto-label an evaluation comment with AI.
• POST /evaluations/:evaluationId/ai-suggestions — Request AI suggestions.
• GET /evaluations/tasks/:taskId/results — Get all evaluations for a task.

• GET /comparisons/:taskId/next — Get the next uncompared pair.
• POST /comparisons — Submit a pairwise comparison verdict.
• GET /comparisons/:taskId/results — Get aggregated comparison results.
• GET /comparisons/:taskId/comparisons — List individual comparisons.
• GET /rankings/:taskId/next — Get the next unranked set.
• POST /rankings — Submit a ranking.
• GET /rankings/:taskId/rankings — List individual rankings.

• GET /analytics/dashboard — Org-level analytics overview.
• GET /analytics/tasks/:taskId/agreement — Inter-rater agreement metrics.
• GET /analytics/tasks/:taskId/scores — Score distribution.
• GET /analytics/evaluators — Evaluator throughput and consistency.
• POST /exports — Request a new export.
• GET /exports — List exports for the organization.
• GET /exports/:id/download — Download a ready export file.

Here is a minimal example of creating a single-scoring evaluation task:

const res = await fetch("https://scorra.onrender.com/api/v1/evaluations/tasks", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "Authorization": `Bearer ${token}`,
  },
  body: JSON.stringify({
    title: "GPT-4 vs Claude — help quality",
    description: "Score helpfulness and accuracy for each response.",
    datasetId: "abc123",
    evaluationType: "single",
    dimensions: [
      { name: "helpfulness", scale: 1, upperBound: 5, rubric: "How well does the response help the user?" },
      { name: "accuracy", scale: 1, upperBound: 5, rubric: "Is the information correct and complete?" },
    ],
  }),
});

const task = await res.json();

The response includes the task ID and configuration. Use GET /evaluations/tasks/:taskId/next to fetch items for evaluators, and POST /evaluations/submit to submit their judgments.