E2E Behavior Validation for Frontend Modifications

Core Principle: Test Product Behavior, Not UI States

CRITICAL: Tests must verify that product features WORK correctly, not just that UI elements render.

What NOT to test (UI States):

• ❌ "Dropdown opens when clicked"
• ❌ "Modal appears after button click"
• ❌ "Loading spinner shows during request"
• ❌ "Form fields are visible"
• ❌ "Sidebar collapses"

What TO test (Product Behavior):

• ✅ "Selecting an LLM provider configures the agent to use that provider"
• ✅ "Creating a new agent persists it and shows in the agents list"
• ✅ "Running a tool with parameters returns the expected output"
• ✅ "Chat messages stream correctly and maintain conversation context"
• ✅ "Workflow execution triggers tools in the correct order"

BDD Structure (REQUIRED)

Every E2E spec MUST follow the same BDD shape as the MSW tests. In packages/playground, e2e-bdd/test-needs-when-describe enforces this shape.

The structure has exactly three levels:

1. Outer test.describe = the unit under test (one page or feature per file).
2. Inner test.describe('when …') = exactly ONE precondition. The title MUST start with when.
3. Each test = exactly ONE observable outcome.

import { test, expect } from '@playwright/test';
import { resetStorage } from '../__utils__/reset-storage';

test.describe('Tools list page', () => {
  // the unit
  test.afterEach(async () => {
    await resetStorage();
  });

  test.describe('when a registered tool is clicked', () => {
    // ONE precondition (starts with "when")
    test('navigates to that tool detail page', async ({ page }) => {
      // ONE outcome
      await page.goto('/tools');
      await page.locator('text=Get current weather for a location').click();
      await expect(page).toHaveURL(/\/tools\/weatherInfo$/);
    });

    test('shows the tool name as the page heading', async ({ page }) => {
      // ONE outcome
      await page.goto('/tools');
      await page.locator('text=Get current weather for a location').click();
      await expect(page.locator('h2')).toHaveText('weatherInfo');
    });
  });
});

Rules:

• One outer test.describe per file naming the unit.
• Every leaf test lives inside a test.describe('when …') precondition group. No top-level flat test().
• Split a multi-assertion test() only where assertions represent distinct outcomes; keep tightly-coupled assertions that prove a single outcome together. Never drop an assertion.
• Place beforeEach/afterEach in the narrowest describe scope that needs them.

Prerequisites

Requires Playwright MCP server. If the browser_navigate tool is unavailable, instruct the user to add it:

claude mcp add playwright -- npx @playwright/mcp@latest

Step 1: Understand the Feature Intent

Before writing ANY test, answer these questions:

1. What user problem does this feature solve?
2. What is the expected outcome when the feature works correctly?
3. What data flows through the system? (user input → API → state → UI)
4. What should persist after page reload?
5. What downstream effects should this action have?

Document these answers as comments in your test file.

Step 2: Build and Start

pnpm build:cli
cd packages/playground/e2e/kitchen-sink && pnpm dev

Verify server at http://localhost:4111

Step 3: Map Feature to Behavior Tests

Feature-to-Test Mapping Guide

Feature Category	What to Test	Example Assertion
Agent Configuration	Config changes affect agent behavior	Send message → verify response uses selected model
LLM Provider Selection	Selected provider is used in requests	Intercept API call → verify provider in request payload
Tool Execution	Tool runs with correct params & returns result	Execute tool → verify output matches expected transformation
Workflow Execution	Steps execute in order, data flows between steps	Run workflow → verify each step's output feeds next step
Chat/Streaming	Messages persist, context maintained across turns	Multi-turn conversation → verify context awareness
MCP Server Tools	Server tools are callable and return data	Call MCP tool → verify response structure and content
Memory/Persistence	Data survives page reload	Create item → reload → verify item exists
Error Handling	Errors surface correctly to user	Trigger error condition → verify error message + recovery

Step 4: Write Behavior-Focused Tests

Test Structure Template

import { test, expect, Page } from '@playwright/test';
import { resetStorage } from '../__utils__/reset-storage';
import { selectFixture } from '../__utils__/select-fixture';
import { nanoid } from 'nanoid';

/**
 * FEATURE: [Name of feature]
 * USER STORY: As a user, I want to [action] so that [outcome]
 * BEHAVIOR UNDER TEST: [Specific behavior being validated]
 */

test.describe('[Feature Name] - Behavior Tests', () => {
  let page: Page;

  test.beforeEach(async ({ browser }) => {
    const context = await browser.newContext();
    page = await context.newPage();
  });

  test.afterEach(async () => {
    await resetStorage(page);
  });

  test.describe('when [the single precondition for these outcomes]', () => {
    test('[verb describing the single observable outcome]', async () => {
      // ARRANGE: Set up preconditions
      // - Navigate to the feature
      // - Configure any required state
      // ACT: Perform the user action that triggers the behavior
      // ASSERT: Verify the OUTCOME, not the UI state
      // - Check data persistence
      // - Verify downstream effects
      // - Confirm API calls made correctly
    });
  });
});

Behavior Test Patterns

Pattern 1: Configuration Affects Behavior

test.describe('when a different LLM provider is selected', () => {
  test('uses that provider for agent responses', async () => {
    // ARRANGE
    await page.goto('/agents/my-agent/chat');

    // Intercept API to verify provider
    let capturedProvider: string | null = null;
    await page.route('**/api/chat', route => {
      const body = JSON.parse(route.request().postData() || '{}');
      capturedProvider = body.provider;
      route.continue();
    });

    // ACT: Select a different provider
    await page.getByTestId('provider-selector').click();
    await page.getByRole('option', { name: 'OpenAI' }).click();

    // Send a message to trigger the agent
    await page.getByTestId('chat-input').fill('Hello');
    await page.getByTestId('send-button').click();

    // ASSERT: Verify the selected provider was used
    await expect.poll(() => capturedProvider).toBe('openai');
  });
});

Pattern 2: Data Persistence

test.describe('when a new agent is created', () => {
  test('persists after page reload', async () => {
    // ARRANGE
    await page.goto('/agents');
    const agentName = `Test Agent ${nanoid()}`;

    // ACT: Create new agent
    await page.getByTestId('create-agent-button').click();
    await page.getByTestId('agent-name-input').fill(agentName);
    await page.getByTestId('save-agent-button').click();

    // Wait for creation to complete
    await expect(page.getByText(agentName)).toBeVisible();

    // ASSERT: Verify persistence
    await page.reload();
    await expect(page.getByText(agentName)).toBeVisible({ timeout: 10000 });
  });
});

Pattern 3: Tool Execution Produces Correct Output

test.describe('when the weather tool is executed with a city', () => {
  test('returns formatted weather data for that city', async () => {
    // ARRANGE
    await selectFixture(page, 'weather-success');
    await page.goto('/tools/weather-tool');

    // ACT: Execute tool with parameters
    await page.getByTestId('param-city').fill('San Francisco');
    await page.getByTestId('execute-tool-button').click();

    // ASSERT: Verify OUTPUT content, not just that output appears
    const output = page.getByTestId('tool-output');
    await expect(output).toContainText('temperature');
    await expect(output).toContainText('San Francisco');

    // Verify structured data if applicable
    const outputText = await output.textContent();
    const outputData = JSON.parse(outputText || '{}');
    expect(outputData).toHaveProperty('temperature');
    expect(outputData).toHaveProperty('conditions');
  });
});

Pattern 4: Workflow Step Chaining

test.describe('when a multi-step workflow is run', () => {
  test('passes data between steps correctly', async () => {
    // ARRANGE
    await selectFixture(page, 'workflow-multi-step');
    const sessionId = nanoid();
    await page.goto(`/workflows/data-pipeline?session=${sessionId}`);

    // ACT: Trigger workflow execution
    await page.getByTestId('workflow-input').fill('test input data');
    await page.getByTestId('run-workflow-button').click();

    // ASSERT: Verify each step received correct input from previous step
    // Wait for completion
    await expect(page.getByTestId('workflow-status')).toHaveText('completed', { timeout: 30000 });

    // Check step outputs show data transformation chain
    const step1Output = await page.getByTestId('step-1-output').textContent();
    const step2Output = await page.getByTestId('step-2-output').textContent();

    // Verify step 2 received step 1's output as input
    expect(step2Output).toContain(step1Output);
  });
});

Pattern 5: Streaming Chat with Context

test.describe('when a multi-turn conversation is held', () => {
  test('maintains conversation context across messages', async () => {
    // ARRANGE
    await selectFixture(page, 'contextual-chat');
    const chatId = nanoid();
    await page.goto(`/agents/assistant/chat/${chatId}`);

    // ACT: Multi-turn conversation
    await page.getByTestId('chat-input').fill('My name is Alice');
    await page.getByTestId('send-button').click();
    await expect(page.getByTestId('assistant-message').last()).toBeVisible({ timeout: 20000 });

    await page.getByTestId('chat-input').fill('What is my name?');
    await page.getByTestId('send-button').click();

    // ASSERT: Verify context was maintained
    const response = page.getByTestId('assistant-message').last();
    await expect(response).toContainText('Alice', { timeout: 20000 });
  });
});

Pattern 6: Error Recovery

test.describe('when the API fails during tool execution', () => {
  test('shows an actionable error and allows a successful retry', async () => {
    // ARRANGE: Set up failure fixture
    await selectFixture(page, 'api-failure');
    await page.goto('/tools/flaky-tool');

    // ACT: Trigger the error
    await page.getByTestId('execute-tool-button').click();

    // ASSERT: Error is shown with recovery option
    await expect(page.getByTestId('error-message')).toContainText('failed');
    await expect(page.getByTestId('retry-button')).toBeVisible();

    // Switch to success fixture and retry
    await selectFixture(page, 'api-success');
    await page.getByTestId('retry-button').click();

    // Verify recovery worked
    await expect(page.getByTestId('tool-output')).toBeVisible({ timeout: 10000 });
    await expect(page.getByTestId('error-message')).not.toBeVisible();
  });
});

Step 5: Update Existing Tests

When a test file already exists:

1. Read the existing tests to understand current coverage
2. Identify if tests are UI-focused or behavior-focused
3. Refactor UI-focused tests to verify behavior instead:

Refactoring Example

BEFORE (UI-focused):

test('dropdown opens when clicked', async () => {
  await page.getByTestId('model-dropdown').click();
  await expect(page.getByRole('listbox')).toBeVisible();
});

AFTER (Behavior-focused + BDD nesting):

test.describe('when a model is selected from the dropdown', () => {
  test('updates and persists the agent configuration', async () => {
    // Open dropdown and select model
    await page.getByTestId('model-dropdown').click();
    await page.getByRole('option', { name: 'GPT-4' }).click();

    // Verify the selection persists and affects behavior
    await page.reload();
    await expect(page.getByTestId('model-dropdown')).toHaveText('GPT-4');

    // Optionally: verify the model is used in actual requests
    // (via request interception or checking response metadata)
  });
});

Step 6: Kitchen-Sink Fixtures for Behavior Testing

Fixtures should represent realistic scenarios, not just mock data:

Fixture Naming Convention

<feature>-<scenario>.fixture.ts

Examples:
- agent-with-tools.fixture.ts
- chat-multi-turn-context.fixture.ts
- workflow-parallel-execution.fixture.ts
- tool-validation-error.fixture.ts
- mcp-server-timeout.fixture.ts

Fixture Content Requirements

Each fixture must define:

1. Scenario description (what behavior it enables testing)
2. Expected outcomes (what assertions should pass)
3. Edge cases covered (error states, empty states, etc.)

// fixtures/agent-provider-switch.fixture.ts
export const agentProviderSwitch = {
  name: 'agent-provider-switch',
  description: 'Tests that switching LLM providers changes agent behavior',

  // Mock responses for different providers
  responses: {
    openai: { content: 'Response from OpenAI', model: 'gpt-4' },
    anthropic: { content: 'Response from Anthropic', model: 'claude-3' },
  },

  expectedBehavior: {
    // When provider is switched, subsequent messages use new provider
    providerSwitchAffectsNextMessage: true,
    // Provider selection persists across page reload
    providerPersistsOnReload: true,
  },
};

Step 7: Run and Validate

cd packages/playground && pnpm test:e2e

Test Quality Checklist

Before considering tests complete, verify:

• Each test has a clear user story comment
• One outer test.describe names the unit under test
• Every test is nested in a test.describe('when …') precondition block (no flat top-level test())
• Each test asserts exactly ONE observable outcome
• Tests verify OUTCOMES, not intermediate UI states
• Tests would FAIL if the feature broke (not just if UI changed)
• Persistence is verified via page.reload() where applicable
• Error scenarios are covered
• Tests use appropriate timeouts for async operations
• Fixtures represent realistic usage scenarios

Quick Reference

Step	Command/Action
Build	pnpm build:cli
Start	cd packages/playground/e2e/kitchen-sink && pnpm dev
App URL	http://localhost:4111
Routes	@packages/playground/src/App.tsx
Run tests	cd packages/playground && pnpm test:e2e
Test dir	packages/playground/e2e/tests/
Fixtures	packages/playground/e2e/kitchen-sink/fixtures/

Anti-Patterns to Avoid

❌ Don't	✅ Do Instead
Test that modal opens	Test that modal action completes and persists
Test that button is clickable	Test that clicking button produces expected result
Test loading spinner appears	Test that loaded data is correct
Test form validation message shows	Test that invalid form cannot submit AND valid form succeeds
Test dropdown has options	Test that selecting option changes system behavior
Test sidebar navigation works	Test that navigated page has correct data/functionality
Assert element is visible	Assert element contains expected data/state
Top-level flat test() with no precondition describe	Nest every test in a test.describe('when …') block
One test() asserting several unrelated outcomes	One test() per observable outcome