Email OTP Testing FAQ: Answers for Flaky CI Suites

Testing email verification and magic-link flows in CI pipelines raises recurring questions about isolation, polling strategies, extraction methods, and rate limits. This FAQ addresses the most common challenges CI engineers face when automating email OTP tests.

Why do email OTP tests flake in CI?

Email OTP tests typically flake for three reasons:

1. Race conditions from shared inboxes — Multiple test runs or parallel CI jobs using the same email address collide: one test's OTP leaks into another's assertion, or an old message from a retry confuses the current run.

2. Fixed sleeps instead of bounded polling — A hard-coded sleep 5 either times out prematurely when email takes 6 seconds or wastes 4 seconds when it arrives in 1 second. Email delivery latency varies.

3. Brittle HTML parsing — Tests that fetch raw email bodies and run grep -oP 'code:\s*\K\d{6}' break silently when marketing changes the email template. Multipart MIME parsing, HTML stripping, and regex maintenance add fragility.

The root cause is treating email as a static resource instead of an asynchronous, variable-latency channel. Robust tests isolate inboxes, poll with deadlines, and extract codes via API instead of parsing.

---

Should I use a unique disposable inbox per test or share one inbox?

Always create a unique disposable inbox per test run. Shared inboxes are the primary source of CI flakiness in email testing.

Why unique inboxes matter:

• Parallel CI jobs run simultaneously. If Job A and Job B both use test@example.com, their OTP codes interleave unpredictably.
• Sequential retries leave old messages. A failed test that retries might read the stale OTP from the first attempt instead of the fresh one.
• Test isolation is fundamental. Just as you'd never share a database connection pool between parallel tests, you shouldn't share inboxes.

How to implement:

Create a fresh inbox at the start of each test. Services like OTPBox make this trivial:

# Each test gets its own inbox
RESPONSE=$(curl -s -X POST https://otpbox.app/api/inbox)
TOKEN=$(echo "$RESPONSE" | jq -r '.token')
EMAIL=$(echo "$RESPONSE" | jq -r '.email')

echo "Test inbox: $EMAIL"

The email address is at *@box.otpbox.app domain, and the token provides isolated access. No coordination between tests is needed.

Anti-pattern: Setting CI_TEST_EMAIL=shared@example.com as an environment variable that all tests reuse.

---

Should I use a fixed sleep or a bounded poll with a deadline?

Use bounded polling with a maximum time budget. Fixed sleeps are either too short (causing flakes) or too long (wasting CI time).

Why polling beats fixed sleeps:

• Email delivery latency varies from under 1 second to 10+ seconds depending on SMTP queue depth, provider throttling, and network conditions.
• A fixed sleep 5 wastes 4 seconds when email arrives in 1 second.
• A fixed sleep 3 fails prematurely when email takes 5 seconds.

How to implement:

Poll every 2–3 seconds for up to 30 seconds. If no email arrives, fail the test—this indicates a real problem, not transient slowness.

MAX_ATTEMPTS=10
SLEEP_SECONDS=3

for i in $(seq 1 $MAX_ATTEMPTS); do
  RESPONSE=$(curl -s "https://otpbox.app/api/inbox/${TOKEN}/latest")
  OTP=$(echo "$RESPONSE" | jq -r '.otp // empty')
  
  if [ -n "$OTP" ] && [ "$OTP" != "null" ]; then
    echo "✓ OTP received: $OTP"
    break
  fi
  
  echo "  Waiting... (attempt $i/$MAX_ATTEMPTS)"
  sleep $SLEEP_SECONDS
done

if [ -z "$OTP" ] || [ "$OTP" = "null" ]; then
  echo "✗ Timeout: No OTP after ${MAX_ATTEMPTS} attempts"
  exit 1
fi

Note: The /latest endpoint returns 404 when the inbox exists but has no messages yet. This is not an error—it means "keep polling."

---

Should I extract OTP via API or scrape HTML with regex?

Always use an API that extracts OTP codes for you. HTML scraping is brittle and time-consuming to maintain.

Why API extraction is superior:

• Email templates change — Marketing updates the welcome email, and suddenly your regex \bcode:\s*(\d{6})\b no longer matches. Your tests fail silently.
• Multipart MIME is complex — Real emails are multipart/alternative with text and HTML bodies. Parsing requires a library, base64 decoding, and handling edge cases.
• Verification patterns vary — Some emails say "Your code is 123456", others say "Verification code: 123456" or just "123456" in a button.

How OTPBox solves this:

The GET /api/inbox/:token/latest endpoint returns extracted codes automatically:

{
  "otp": "123456",
  "link": "https://app.com/verify?token=abc...",
  "from": "noreply@yourapp.com",
  "subject": "Verify your email",
  "receivedAt": 1726377600000
}

Both otp and link fields are extracted via regex patterns for 4–8 digit codes and common verification URL patterns. No manual parsing required.

Anti-pattern: Fetching raw email bodies, stripping HTML tags, and running regex in every test script.

---

How do I handle both OTP codes and magic links?

Check both the otp and link fields from the API response. Many apps send both; some send only one.

Why dual support matters:

• Different flows — Signup might send a numeric OTP, while password reset sends a magic link.
• Product changes — Your app might switch from OTP codes to magic links (or vice versa) in a future release. Tests shouldn't break.
• Fallback patterns — Some emails include both: "Enter code 123456 or click here to verify."

Implementation example:

// Playwright/Puppeteer test
async function getVerification(token) {
  const response = await fetch(`https://otpbox.app/api/inbox/${token}/latest`);
  const data = await response.json();
  return {
    otp: data.otp || null,
    link: data.link || null
  };
}

const { otp, link } = await getVerification(token);

if (link) {
  // Magic link flow: visit the URL directly
  await page.goto(link);
  await expect(page).toHaveURL(/dashboard/);
} else if (otp) {
  // OTP flow: fill the code input
  await page.fill('input[name="code"]', otp);
  await page.click('button:has-text("Verify")');
}

This dual-path approach keeps tests flexible as your authentication flows evolve.

---

What are the rate limits, and how do I avoid hitting them?

OTPBox free tier allows 10 inbox creates per hour per IP. Polling endpoints are unrestricted.

Key principles:

• Create inboxes sparingly — Each POST /api/inbox call counts toward the 10/hour limit. Create once per test, not inside retry loops.
• Poll as aggressively as needed — GET /api/inbox/:token/latest has no rate limit. You can poll every second if necessary (though 2–3 seconds is typically sufficient).
• Reuse inboxes within retries — If a test fails and retries, reuse the same token instead of creating a new inbox.

Implementation:

# ONE create (counts toward 10/hour limit)
RESPONSE=$(curl -s -X POST https://otpbox.app/api/inbox)
TOKEN=$(echo "$RESPONSE" | jq -r '.token')

# MANY polls (unrestricted)
for i in {1..20}; do
  curl -s "https://otpbox.app/api/inbox/${TOKEN}/latest"
  sleep 2
done

If you hit the 429 rate limit, it means your test suite is creating too many inboxes. Consolidate test runs or reuse inboxes across related assertions.

---

How long do disposable inboxes last?

OTPBox inboxes expire after 1 hour from creation. This automatic expiry prevents state leakage between test runs and eliminates cleanup overhead.

What expiry means:

• Inbox lifetime: 1 hour from the POST /api/inbox creation timestamp
• After expiry: The inbox stops accepting new emails
• API behavior: Requests to expired inboxes return 410 Gone
• No cleanup needed: Tests don't need teardown logic to delete inboxes

Error handling:

RESPONSE=$(curl -s -w "%{http_code}" "https://otpbox.app/api/inbox/${TOKEN}/latest")
HTTP_CODE=$(echo "$RESPONSE" | tail -n1)

case $HTTP_CODE in
  200) echo "✓ Email received" ;;
  404) echo "⏳ Inbox exists, no messages yet (keep polling)" ;;
  410) echo "✗ Inbox expired (test took >1 hour)" ;;
  429) echo "✗ Rate limited (too many inbox creates)" ;;
esac

The 1-hour TTL is sufficient for typical CI test suites. If your end-to-end tests take longer than 1 hour, they likely have other performance issues to address.

---

Can parallel CI jobs run email tests without coordination?

Yes, if each job creates its own unique inbox. No locking, queues, or cleanup logic is required.

Why this works:

• Isolated state — Each test has a unique email address (random-id@box.otpbox.app) and opaque access token
• No shared resources — Unlike shared databases or file systems, disposable inboxes don't conflict
• Auto-expiry — Inboxes disappear after 1 hour; no manual cleanup between parallel runs

GitHub Actions example:

strategy:
  matrix:
    test-group: [auth, signup, password-reset]

steps:
  - name: Create unique inbox
    run: |
      RESPONSE=$(curl -s -X POST https://otpbox.app/api/inbox)
      echo "token=$(echo "$RESPONSE" | jq -r '.token')" >> $GITHUB_OUTPUT
      echo "email=$(echo "$RESPONSE" | jq -r '.email')" >> $GITHUB_OUTPUT
  
  - name: Run tests
    run: ./test-${{ matrix.test-group }}.sh

All three jobs (auth, signup, password-reset) run in parallel. Each creates its own inbox. No collision is possible.

Anti-pattern: Using a single CI_TEST_EMAIL environment variable that all parallel jobs fight over, then adding advisory locks or queues to prevent conflicts.

---

When should I use local SMTP instead of a cloud disposable email service?

Use local SMTP for local development and unit tests. Use cloud disposable email for CI E2E tests.

Local SMTP (Mailpit) is best for:

• Local development — Fast feedback while coding email flows
• Offline testing — No network dependency
• Unit tests — Mock SMTP responses to test business logic

Recommendation: Mailpit is actively maintained (v1.31.0 released August 2026) and replaces the largely unmaintained MailHog. It provides a local SMTP server, web UI, and API.

Cloud disposable email (OTPBox) is best for:

• CI E2E tests — Validate real email delivery (SMTP credentials, SPF/DKIM, template rendering)
• Parallel CI jobs — Each job creates isolated inboxes without coordination
• Magic-link testing — Click verification links in browser automation frameworks
• Production-like validation — Catch deliverability issues before they reach users

Decision matrix:

Scenario	Use Local SMTP	Use Cloud Disposable
Local dev feedback	✓
Unit tests (logic only)	✓
CI integration tests		✓
E2E browser tests		✓
Parallel CI jobs		✓
Validate deliverability		✓

Both tools are complementary. Use Mailpit for fast local iteration, OTPBox for CI validation.

---

How do I test failure paths like expired or wrong OTP codes?

Write explicit tests for expired codes, wrong codes, and rate limits. Production bugs hide in these edge cases.

Common failure scenarios:

1. Expired OTP — User receives the code but waits too long to submit it
2. Wrong code — User typos the code or an attacker guesses randomly
3. Rate limits — User retries verification too many times
4. Invalid email format — User submits a malformed email address

Example: Testing expired OTP

test('expired OTP is rejected', async ({ page }) => {
  const { email, token } = await createInbox();
  
  // Trigger verification
  await page.goto('https://yourapp.com/signup');
  await page.fill('input[name="email"]', email);
  await page.click('button:has-text("Sign Up")');
  
  // Get the OTP
  const { otp } = await pollForOTP(token);
  
  // Simulate expiration by waiting (adjust to your app's timeout)
  await new Promise(resolve => setTimeout(resolve, 300000)); // 5 minutes
  
  // Attempt verification (should fail)
  await page.fill('input[name="code"]', otp);
  await page.click('button:has-text("Verify")');
  
  await expect(page.locator('text=Code expired')).toBeVisible();
});

Example: Testing wrong code

test('wrong OTP is rejected', async ({ page }) => {
  const { email, token } = await createInbox();
  
  await page.goto('https://yourapp.com/signup');
  await page.fill('input[name="email"]', email);
  await page.click('button:has-text("Sign Up")');
  
  // Don't fetch the real OTP; submit a fake one
  await page.fill('input[name="code"]', '000000');
  await page.click('button:has-text("Verify")');
  
  await expect(page.locator('text=Invalid code')).toBeVisible();
});

These tests validate that your app handles edge cases correctly before they reach production.

---

Should I log the inbox email address for debugging?

Yes, always log the test email address at test start. When a CI test fails with "No OTP received," the first debugging step is checking whether the app sent email at all.

What to log:

echo "Test inbox: $EMAIL (token: $TOKEN)"

Or in GitHub Actions:

- name: Create inbox
  id: inbox
  run: |
    RESPONSE=$(curl -s -X POST https://otpbox.app/api/inbox)
    TOKEN=$(echo "$RESPONSE" | jq -r '.token')
    EMAIL=$(echo "$RESPONSE" | jq -r '.email')
    echo "token=$TOKEN" >> $GITHUB_OUTPUT
    echo "email=$EMAIL" >> $GITHUB_OUTPUT
    echo "::notice::Test inbox: $EMAIL"

Why this helps:

• Verify send logs — Check your app's logs to confirm it attempted to send to this email
• Check bounce reports — See if the email bounced due to SMTP misconfiguration
• Manual retry — Trigger a new verification email to the same inbox to reproduce the issue

What NOT to log in production: Full OTP codes in application logs. Log "OTP sent to user@example.com" but not "OTP sent: 123456". Test logs can include the full code since they're ephemeral.

---

How does OTPBox fit into my testing workflow?

OTPBox provides three steps: create, send, poll.

Typical workflow:

1. Create a unique inbox

   RESPONSE=$(curl -s -X POST https://otpbox.app/api/inbox)
   TOKEN=$(echo "$RESPONSE" | jq -r '.token')
   EMAIL=$(echo "$RESPONSE" | jq -r '.email')
   

   Returns:

   • id: Internal identifier
   • email: The address to send to (*@box.otpbox.app)
   • token: Access token for polling (opaque, unique per inbox)
   • expiresAt: Timestamp when the inbox expires (1 hour from creation)

2. Trigger your app to send email

   curl -X POST https://yourapp.com/api/signup \
     -H "Content-Type: application/json" \
     -d "{\"email\": \"$EMAIL\"}"
   

   Your app sends the verification email to the OTPBox inbox via its normal SMTP provider.

3. Poll for the extracted OTP/link

   RESPONSE=$(curl -s "https://otpbox.app/api/inbox/${TOKEN}/latest")
   OTP=$(echo "$RESPONSE" | jq -r '.otp')
   LINK=$(echo "$RESPONSE" | jq -r '.link')
   

   The /latest endpoint returns the most recent message with extracted otp (4–8 digit codes) and link (verification URLs).

No signup required. The OTPBox API is free, privacy-focused, and designed for CI automation. Inboxes use opaque tokens for access control and expire automatically after 1 hour.

---

Related Articles

• CI Email OTP Checklist — 10-step checklist for reliable OTP tests in CI
• How to Test Email OTP in CI — Deep dive into email OTP testing patterns and framework integration
• Disposable Email for CI OTP Testing — Practical patterns for parallel testing and state isolation
• Magic Link Testing Without Polluting Real Inboxes — Test magic-link authentication flows with ephemeral inboxes

Resources

• OTPBox Home — Create a test inbox and try it manually
• OTPBox API Documentation — Full REST API reference for CI automation
• Mailpit — Actively maintained local SMTP catcher for development

---

Email OTP testing doesn't have to be flaky. Isolate inboxes, poll with deadlines, extract codes via API, respect rate limits, test failure paths, and log for debugging. Your CI suite will be faster, more reliable, and easier to maintain.