Testing Platform

Testing Platform

Ship AI agents with confidence. Define test cases with expected inputs and outputs, run automated suites from the CLI or your CI/CD pipeline, and track results over time with full deployment traceability.

Core Concepts

Test Cases

Define individual test scenarios with input messages and expected response criteria. Group related cases into test suites.

Test Environments

Configure isolated environments with specific agent versions and settings for consistent, reproducible testing.

Test Runs

Execute test suites against specific deployments. View pass/fail results, response times, and detailed output comparisons.

Deployments

Track agent deployments across environments. Link test runs to specific deployment versions for traceability.

CLI in Action

The th CLI — a single Rust binary — manages and reports test runs directly from your terminal or CI/CD pipeline. Testing commands live under th api testing.

th api testing
$ th api login
Authenticated as [email protected]
$ th api testing cases list
ID Name Status
case_abc123 Greeting Test active
case_def456 Order Lookup active
case_ghi789 Escalation Flow active
case_jkl012 Edge Case: Empty Input active
$ th api testing runs report ./ctrf-report.json --environment staging --deployment-id v2.1.0
Submitting report... done
Run ID: run_abc123
Running 4 test cases...
✓ Greeting Test passed (1.2s)
✓ Order Lookup passed (2.1s)
✓ Escalation Flow passed (1.8s)
✗ Edge Case: Empty Input failed (0.4s)
Results: 3/4 passed, 1 failed
curl -fsSL https://raw.githubusercontent.com/SmooAI/smooth/main/install.sh | sh

Multi-Language Support: The th CLI is a single static binary that works from any language or environment. The @smooai/testing SDK is currently TypeScript-only. For programmatic access from other languages, you can use the REST API to integrate test reporting from any language or framework that supports CTRF.

CTRF Integration

Smoo Testing natively supports the Common Test Report Format (CTRF). Test results from any CTRF-compatible runner automatically flow into the Smoo AI dashboard with zero configuration.

import { SmooTestingClient } from '@smooai/testing';

const client = new SmooTestingClient({
    token: process.env.SMOO_ACCESS_TOKEN!,
});

// Upload a CTRF report from any test runner
await client.uploadCtrfReport({
    environmentId: 'env_staging',
    deploymentId: 'deploy_v2.1.0',
    report: './ctrf-report.json',
});

Supported Test Runners

Any runner with a CTRF reporter: Playwright, Jest, Vitest, Mocha, pytest, JUnit, and more. Results are automatically parsed and displayed in the Smoo AI dashboard.

GitHub Integration

Connect your GitHub repository for one-click sync of deployments, environments, and test results. Every push creates a deployment record, and test runs link back to the commit that triggered them.

Auto Deployments

Pushes to configured branches automatically create deployment records with commit metadata.

PR Status Checks

Test results post directly to pull requests as GitHub status checks with pass/fail indicators.

Actions Integration

Drop-in GitHub Action for running tests on every push, PR, or scheduled interval.

SmooTestingClient SDK

The TypeScript SDK provides programmatic access to the full testing API. Create test cases, trigger runs, and fetch results from your application code.

import { SmooTestingClient } from '@smooai/testing';

const testing = new SmooTestingClient({
    token: process.env.SMOO_ACCESS_TOKEN!,
    organizationId: 'org_abc123',
});

// Create a test case
const testCase = await testing.createTestCase({
    name: 'Order Status Inquiry',
    input: 'What is the status of order #12345?',
    expectedOutput: 'Includes order status and estimated delivery date',
    agentId: 'agent_xyz',
});

// Start a test run
const run = await testing.createTestRun({
    environmentId: 'env_staging',
    deploymentId: 'deploy_v2.1.0',
    testCaseIds: [testCase.id],
});

// Poll for results
const results = await testing.getTestRun(run.id);
console.log(results.summary);
// { total: 1, passed: 1, failed: 0, duration: '1.2s' }

CI/CD Pipeline

Run tests automatically on every deployment. Here's a GitHub Actions workflow that runs your test suite after deploying to staging, then reports the results to Smoo AI with the th CLI.

name: Test AI Agent
on:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to staging
        run: pnpm deploy:staging

      - name: Install th CLI
        run: curl -fsSL https://raw.githubusercontent.com/SmooAI/smooth/main/install.sh | sh

      - name: Run tests & report to Smoo AI
        env:
          SMOOAI_CLIENT_ID: ${{ secrets.SMOOAI_CLIENT_ID }}
          SMOOAI_CLIENT_SECRET: ${{ secrets.SMOOAI_CLIENT_SECRET }}
        run: |
          th api login
          th api testing runs report ./ctrf-report.json \
            --environment staging \
            --deployment-id ${{ github.sha }}

th api testing runs report Flags

  • --environment — Associate the run with a test environment
  • --deployment-id — Link the run to a specific deployment
  • --junit — Treat the report as JUnit XML and convert it to CTRF before submitting
  • --tags — Comma-separated tags for filtering runs in the dashboard

REST API Examples

Create a Test Case

curl -X POST https://api.smoo.ai/organizations/{org_id}/test-cases \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Greeting Test",
    "description": "Verify the agent responds with a friendly greeting",
    "input": "Hello, I need help with my order",
    "expectedOutput": "Contains a greeting and asks for order details",
    "agentId": "agent_id"
  }'

Create a Test Environment

curl -X POST https://api.smoo.ai/organizations/{org_id}/test-environments \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Staging",
    "description": "Pre-production testing environment"
  }'

Start a Test Run

curl -X POST https://api.smoo.ai/organizations/{org_id}/test-runs \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "testEnvironmentId": "env_id",
    "deploymentId": "deployment_id",
    "testCaseIds": ["case_1", "case_2", "case_3"]
  }'

Get Test Run Results

curl https://api.smoo.ai/organizations/{org_id}/test-runs/{run_id} \
  -H "Authorization: Bearer YOUR_TOKEN"

Typical Workflow

  1. 1

    Define test cases

    Create test cases that cover your agent's expected behavior: greetings, FAQs, edge cases, and escalation scenarios.

  2. 2

    Set up environments

    Create test environments that mirror your deployment stages (development, staging, production).

  3. 3

    Run tests on each deployment

    Execute test suites after each deployment to catch regressions. Use the CLI or GitHub Actions for automation.

  4. 4

    Review results and iterate

    Analyze test run results in the dashboard, identify failing cases, and refine your agent's knowledge base and prompts.