Email verification and OTP tests fail unpredictably in CI for reasons that never appear locally. What works perfectly on your laptop races, times out, or breaks silently when running in parallel CI jobs. This isn't a flaky test—it's a predictable outcome of seven avoidable mistakes.

Each mistake below includes the failure mode, why it happens, and the fix with working OTPBox API examples.

1. Sharing One Inbox Across Tests and CI Jobs

The mistake: Reusing the same email address (test@example.com or a single disposable inbox) across all tests and parallel CI jobs.

Why it flakes: Multiple tests race for the same inbox. Job A triggers a signup OTP, Job B triggers a password-reset OTP—both poll the same inbox and grab whichever code arrives first. Sequential retries are equally broken: a test fails, retries, and reads the stale OTP from the first attempt instead of the fresh one.

The fix: Create a unique disposable inbox per test run. Services like OTPBox make this trivial:

# Each test gets a fresh inbox with unique email + token
RESPONSE=$(curl -s -X POST https://otpbox.app/api/inbox)
TOKEN=$(echo "$RESPONSE" | jq -r '.token')
EMAIL=$(echo "$RESPONSE" | jq -r '.email')

echo "Test inbox: $EMAIL"

The email address is at the *@box.otpbox.app domain, and the token provides isolated access. No test can accidentally read another test's OTP. Parallel CI jobs work without coordination, and retries don't see stale messages.

Related reading: CI Email OTP Checklist explains why inbox isolation is the foundation of reliable email tests.

2. Using Fixed Sleeps Instead of Bounded Polling

The mistake: Hard-coding sleep 5 or sleep 10 after triggering an email send, then fetching the OTP once.

Why it flakes: Email delivery latency varies from under 1 second to 15+ seconds depending on SMTP queue depth, provider throttling, and network conditions. A fixed sleep 5 either wastes 4 seconds when email arrives in 1 second or fails prematurely when delivery takes 7 seconds. You're guessing—CI doesn't tolerate guesses.

The fix: Poll with a bounded timeout budget. Check every 2–3 seconds for up to 30 seconds. If no email arrives, fail the test—this signals a real problem, not transient slowness.

MAX_ATTEMPTS=10
SLEEP_SECONDS=3

for i in $(seq 1 $MAX_ATTEMPTS); do
  RESPONSE=$(curl -s "https://otpbox.app/api/inbox/${TOKEN}/latest")
  OTP=$(echo "$RESPONSE" | jq -r '.otp // empty')
  
  if [ -n "$OTP" ] && [ "$OTP" != "null" ]; then
    echo "✓ OTP received: $OTP"
    break
  fi
  
  echo "  Waiting... (attempt $i/$MAX_ATTEMPTS)"
  sleep $SLEEP_SECONDS
done

if [ -z "$OTP" ] || [ "$OTP" = "null" ]; then
  echo "✗ Timeout: No OTP after ${MAX_ATTEMPTS} attempts"
  exit 1
fi

The /latest endpoint returns 404 when the inbox exists but has no messages yet. This isn't an error—it means "keep polling."

Why 30 seconds? Typical SMTP delivery ranges from 1–10 seconds. A 30-second timeout catches 99%+ of normal deliveries while failing fast enough to detect misconfigurations (wrong SMTP credentials, DNS issues) before they waste CI minutes.

3. Scraping HTML with Regex Instead of Using Structured Extraction

The mistake: Fetching raw email bodies and running regex like grep -oP 'code:\s*\K\d{6}' or parsing HTML with sed to find the OTP.

Why it flakes: Email templates change—marketing rewrites the welcome email, and your regex no longer matches. Multipart MIME emails include both text and HTML bodies; parsing requires base64 decoding and content-type handling. Verification patterns vary: "Your code is 123456", "Verification: 123456", or <button>123456</button>. Every variation breaks your regex silently.

The fix: Use an API that extracts OTP codes for you. The OTPBox /latest endpoint returns extracted codes automatically:

{
  "otp": "123456",
  "link": "https://app.com/verify?token=abc...",
  "from": "noreply@yourapp.com",
  "subject": "Verify your email",
  "receivedAt": 1726377600000
}

Both otp (4–8 digit codes) and link (common verification URL patterns) are extracted via regex patterns maintained by the service. Template changes don't break your tests—the extraction logic adapts.

// Playwright example: works for OTP codes or magic links
const response = await fetch(`https://otpbox.app/api/inbox/${token}/latest`);
const { otp, link } = await response.json();

if (link) {
  // Magic link flow
  await page.goto(link);
} else if (otp) {
  // OTP code flow
  await page.fill('input[name="code"]', otp);
  await page.click('button:has-text("Verify")');
}

Local alternative: For local development, Mailpit provides a local SMTP catcher with an API. Use it locally; use cloud disposable email like OTPBox for CI E2E tests.

Related reading: Disposable Email for CI OTP Testing covers when to use local SMTP versus cloud disposable inboxes.

4. Reusing the Same Inbox on CI Retry

The mistake: Creating an inbox once at the start of a test suite, then reusing it when the test retries after failure.

Why it flakes: The retry reads the stale OTP from the first failed attempt instead of waiting for the fresh one triggered by the retry. If your app doesn't send duplicate emails to the same address within a short window (sensible spam prevention), the test hangs forever waiting for a message that will never arrive.

The fix: Either create a brand-new inbox per retry, or trigger a fresh OTP without recreating the inbox (depending on your app's behavior). Most reliably: create the inbox at test start and ensure retries trigger a new verification flow with a new OTP.

# Correct: create inbox once, trigger FRESH verification each time
create_inbox() {
  RESPONSE=$(curl -s -X POST https://otpbox.app/api/inbox)
  export TOKEN=$(echo "$RESPONSE" | jq -r '.token')
  export EMAIL=$(echo "$RESPONSE" | jq -r '.email')
}

trigger_and_verify() {
  # Trigger signup (generates fresh OTP)
  curl -X POST https://yourapp.com/api/signup \
    -H "Content-Type: application/json" \
    -d "{\"email\": \"$EMAIL\"}"
  
  # Poll for the latest OTP
  poll_for_otp "$TOKEN"
}

# Create inbox ONCE
create_inbox

# Retry logic calls trigger_and_verify again
# App sends a NEW OTP to the same inbox
for attempt in {1..3}; do
  if trigger_and_verify; then
    break
  fi
  echo "Retry $attempt failed, retrying..."
done

Rate limit note: OTPBox free tier allows 10 inbox creates per hour per IP. Polling is unrestricted. Reusing the inbox across retries conserves your creation quota.

5. Matching "Latest Email" by Subject Only

The mistake: Polling for "any email with subject 'Verify your email'" without checking sender, timestamp, or inbox isolation.

Why it flakes: If you're sharing an inbox (see Mistake #1) or testing multiple flows, the "latest email" might be from a different test entirely. Subject-only matching gives you password-reset links when you expected signup OTPs, or vice versa.

The fix: Isolate inboxes per test (Mistake #1), then fetch the /latest message. The isolation guarantees the latest message is your message. If you need additional filtering, check the from and subject fields returned by the API:

async function pollForEmail(token, expectedFrom) {
  const maxAttempts = 10;
  
  for (let i = 0; i < maxAttempts; i++) {
    const response = await fetch(`https://otpbox.app/api/inbox/${token}/latest`);
    
    if (response.ok) {
      const data = await response.json();
      
      // Validate sender if needed (but isolation already guarantees relevance)
      if (data.from === expectedFrom) {
        return data;
      }
    }
    
    await new Promise(resolve => setTimeout(resolve, 3000));
  }
  
  throw new Error('Timeout: No matching email received');
}

const email = await pollForEmail(token, 'noreply@yourapp.com');

Better approach: Trust the isolation. If each test has its own inbox, the latest message is always the one you triggered.

Related reading: How to Test Email OTP in CI covers isolation and polling patterns in depth.

6. Creating a New Inbox Inside the Polling Loop

The mistake: Recreating the disposable inbox on every poll attempt or retry.

Why it flakes: You hit rate limits. OTPBox free tier permits 10 inbox creates per hour per IP. If you create a new inbox every 3 seconds for 30 seconds, that's 10 creates in 30 seconds—you've exhausted your hourly quota in half a minute. The test fails with 429 Too Many Requests.

The fix: Create the inbox once before triggering the email send. Poll the same inbox using its token. Polling endpoints are unrestricted—you can poll every second if needed.

# ✓ CORRECT: Create once, poll many times
RESPONSE=$(curl -s -X POST https://otpbox.app/api/inbox)
TOKEN=$(echo "$RESPONSE" | jq -r '.token')
EMAIL=$(echo "$RESPONSE" | jq -r '.email')

# Trigger email
curl -X POST https://yourapp.com/api/signup -d "{\"email\": \"$EMAIL\"}"

# Poll unrestricted
for i in {1..20}; do
  curl -s "https://otpbox.app/api/inbox/${TOKEN}/latest"
  sleep 2
done

# ✗ WRONG: Creating inbox inside polling loop
for i in {1..20}; do
  # This hits rate limits after ~10 iterations
  RESPONSE=$(curl -s -X POST https://otpbox.app/api/inbox)
  TOKEN=$(echo "$RESPONSE" | jq -r '.token')
  curl -s "https://otpbox.app/api/inbox/${TOKEN}/latest"
  sleep 2
done

If you hit 429: Your test suite is creating too many inboxes. Consolidate test runs, reuse inboxes across related assertions within a single test, or throttle parallel CI jobs.

7. Not Testing Failure Paths or Logging the Test Email

The mistake (Part A): Only testing the happy path—signup → OTP → success. No tests for expired codes, wrong codes, or rate limits.

Why it causes pain: Production bugs hide in edge cases. You discover your app accepts any 6-digit code (not just the valid OTP) when a security researcher reports it. Or expired OTPs crash your verification handler because you never tested that path.

The fix (Part A): Write explicit tests for failure scenarios:

// Test expired OTP
test('expired OTP is rejected', async ({ page }) => {
  const { email, token } = await createInbox();
  
  await page.goto('https://yourapp.com/signup');
  await page.fill('input[name="email"]', email);
  await page.click('button:has-text("Sign Up")');
  
  const { otp } = await pollForOTP(token);
  
  // Simulate expiration (adjust to your app's timeout)
  await new Promise(resolve => setTimeout(resolve, 300000)); // 5 minutes
  
  await page.fill('input[name="code"]', otp);
  await page.click('button:has-text("Verify")');
  
  await expect(page.locator('text=Code expired')).toBeVisible();
});

// Test wrong code
test('wrong OTP is rejected', async ({ page }) => {
  const { email, token } = await createInbox();
  
  await page.goto('https://yourapp.com/signup');
  await page.fill('input[name="email"]', email);
  await page.click('button:has-text("Sign Up")');
  
  // Don't fetch the real OTP; submit a fake one
  await page.fill('input[name="code"]', '000000');
  await page.click('button:has-text("Verify")');
  
  await expect(page.locator('text=Invalid code')).toBeVisible();
});

The mistake (Part B): Not logging the test email address. When a CI test fails with "No OTP received," you have no way to check whether the app sent email at all.

The fix (Part B): Always log the test inbox at test start:

- name: Create inbox
  id: inbox
  run: |
    RESPONSE=$(curl -s -X POST https://otpbox.app/api/inbox)
    TOKEN=$(echo "$RESPONSE" | jq -r '.token')
    EMAIL=$(echo "$RESPONSE" | jq -r '.email')
    echo "token=$TOKEN" >> $GITHUB_OUTPUT
    echo "email=$EMAIL" >> $GITHUB_OUTPUT
    echo "::notice::Test inbox: $EMAIL"

Now when the test fails, you can:

Check your app's send logs for that email address
Inspect bounce/reject reports
Manually trigger a verification email to the same inbox to reproduce the issue

Anti-pattern: Creating inboxes silently, then spending 20 minutes debugging because you don't know which email the test used.

Related reading: Email OTP Testing FAQ covers debugging strategies and failure-path testing in Q&A format.

Bonus Mistake: Using Mocks for End-to-End Tests

The mistake: Mocking your sendEmail() function in CI E2E tests to "speed things up" or "avoid flakiness."

Why it defeats the purpose: Mocking removes the entire point of E2E testing. You won't catch:

Invalid SMTP credentials
Missing DNS records (SPF/DKIM)
Email template rendering bugs (broken HTML, missing variables)
Deliverability issues (bounces, spam filters)

The fix: Use real email delivery in E2E tests. Mock in unit tests where you're testing business logic, not the full flow. For CI E2E tests, let your app send email normally and use a disposable inbox service to receive it.

When to use local SMTP: For local development and unit tests, Mailpit (actively maintained) or MailHog (largely unmaintained) run a local SMTP server with a web UI and API. Fast, offline, no external dependency. But for CI E2E tests where you want to validate real-world email delivery, use cloud disposable email.

Decision matrix:

Scenario	Use Local SMTP	Use Cloud Disposable
Local dev feedback	✓
Unit tests (logic only)	✓
CI E2E tests		✓
Parallel CI jobs		✓
Validate deliverability		✓

Related reading: Magic Link Testing Without Polluting Real Inboxes explains when disposable email is essential versus nice-to-have.

Putting It All Together

Avoiding these seven mistakes boils down to three principles:

Isolate state: Unique inbox per test, no shared email addresses
Handle async correctly: Bounded polling with deadlines, not fixed sleeps
Use the right tools: Structured extraction APIs, real SMTP in E2E tests, logged email addresses for debugging

A complete GitHub Actions example fixing all seven mistakes:

name: E2E Email Verification

on: [push, pull_request]

jobs:
  test-otp:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        flow: [signup, password-reset]
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Create unique inbox (fixes Mistakes 1, 4, 6)
        id: inbox
        run: |
          RESPONSE=$(curl -s -X POST https://otpbox.app/api/inbox)
          echo "token=$(echo "$RESPONSE" | jq -r '.token')" >> $GITHUB_OUTPUT
          echo "email=$(echo "$RESPONSE" | jq -r '.email')" >> $GITHUB_OUTPUT
          echo "::notice::Test inbox: $(echo "$RESPONSE" | jq -r '.email')"
      
      - name: Trigger ${{ matrix.flow }}
        run: |
          curl -X POST https://yourapp.com/api/${{ matrix.flow }} \
            -H "Content-Type: application/json" \
            -d "{\"email\": \"${{ steps.inbox.outputs.email }}\"}"
      
      - name: Poll with bounded timeout (fixes Mistakes 2, 3, 5)
        id: otp
        run: |
          MAX_ATTEMPTS=10
          for i in $(seq 1 $MAX_ATTEMPTS); do
            RESPONSE=$(curl -s "https://otpbox.app/api/inbox/${{ steps.inbox.outputs.token }}/latest")
            OTP=$(echo "$RESPONSE" | jq -r '.otp // empty')
            
            if [ -n "$OTP" ] && [ "$OTP" != "null" ]; then
              echo "otp=$OTP" >> $GITHUB_OUTPUT
              exit 0
            fi
            
            sleep 3
          done
          
          echo "✗ Timeout after 30s"
          exit 1
      
      - name: Verify OTP
        run: |
          curl -X POST https://yourapp.com/api/verify \
            -d "{\"code\": \"${{ steps.otp.outputs.otp }}\"}"
      
      - name: Test failure path (fixes Mistake 7)
        run: |
          # Test wrong code
          curl -X POST https://yourapp.com/api/verify \
            -d "{\"code\": \"000000\"}" \
            | grep -q "Invalid code"

This workflow runs two parallel jobs (signup, password-reset), each with isolated inboxes, bounded polling, structured extraction, logged email addresses, and failure-path tests. No shared state, no fixed sleeps, no HTML scraping, no rate limit issues.

CI Email OTP Checklist — 10-step checklist for reliable OTP tests in CI
How to Test Email OTP in CI — Deep dive into email OTP testing patterns and framework integration
Email OTP Testing FAQ — Practical FAQ for CI engineers testing email OTP and magic-link flows
Disposable Email for CI OTP Testing — Practical patterns for parallel testing and state isolation

Resources

OTPBox Home — Create a test inbox and try it manually
OTPBox API Documentation — Full REST API reference for CI automation
Mailpit — Actively maintained local SMTP catcher for development

Email OTP tests don't have to flake. Avoid these seven mistakes—shared inboxes, fixed sleeps, HTML scraping, stale retry inboxes, subject-only matching, inbox creation spam, and missing failure tests—and your CI suite becomes predictable, fast, and maintainable.

7 Mistakes That Make Email OTP Tests Flake in CI