API Introduction

Overview

The SEO Crawler REST API allows you to automate link health monitoring, integrate SEO checks into your CI/CD pipeline, and build custom workflows around your website's link health data.

What you can do with the API:

• ✅ Start and manage crawls programmatically
• ✅ Retrieve crawl results and link health data
• ✅ Monitor domain status and verification
• ✅ Configure webhooks for real-time notifications
• ✅ Export reports in multiple formats
• ✅ Integrate with your development workflow

Quick Start

Get up and running with the API in 3 steps:

Navigate to [Settings → API](https://seocrawler.app/settings) and click **Create API Key**.

# Your API key will look like this:
sc_live_a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6

Test your API key by fetching your domains:

curl https://seocrawler.app/api/v1/domains \
  -H "Authorization: Bearer sc_live_your_api_key"

{
  "data": [
    {
      "id": "882baa8e-4137-4dec-bb81-cd7402f62365",
      "domain_name": "example.com",
      "verified": true,
      "created_at": "2024-12-09T10:30:00.000Z"
    }
  ],
  "meta": {
    "pagination": {
      "page": 1,
      "limit": 20,
      "total": 1,
      "totalPages": 1,
      "hasMore": false
    }
  }
}

Trigger a crawl for one of your verified domains:

curl -X POST https://seocrawler.app/api/v1/crawls \
  -H "Authorization: Bearer sc_live_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "domain_id": "882baa8e-4137-4dec-bb81-cd7402f62365",
    "options": {
      "max_urls": 500,
      "max_depth": 3
    }
  }'

{
  "data": {
    "id": "123e4567-e89b-12d3-a456-426614174000",
    "domain_id": "882baa8e-4137-4dec-bb81-cd7402f62365",
    "domain_name": "example.com",
    "status": "pending",
    "started_at": "2024-12-09T14:30:00.000Z"
  }
}

Base URL

All API requests should be made to:

https://seocrawler.app/api/v1

Authentication

Every API request requires authentication via an API key in the Authorization header using the Bearer token scheme.

Header Format

Authorization: Bearer sc_live_your_api_key

Example Request

curl https://seocrawler.app/api/v1/domains \
  -H "Authorization: Bearer sc_live_a1b2c3d4e5f6..."

API Key Types

Authentication Errors

If authentication fails, you'll receive:

{
  "error": {
    "message": "Invalid or expired API key",
    "code": "UNAUTHORIZED"
  }
}

Common causes:

• Missing Authorization header
• Invalid API key format
• Expired or deleted API key
• API key for a different environment

API Keys Management

Creating API Keys

1. Navigate to Settings → API Keys
2. Click Create API Key
3. Enter a descriptive name (e.g., "Production Server", "CI/CD Pipeline")
4. Select permissions/scopes (all or specific operations)
5. Click Create
6. Copy the key immediately - it's only shown once

Key Scopes

Control what each API key can access:

Revoking API Keys

If an API key is compromised:

1. Go to Settings → API Keys
2. Click Revoke next to the compromised key
3. The key becomes invalid immediately
4. Create a new key and update your applications

Request Format

HTTP Methods

The API uses standard HTTP methods:

Content Type

All POST and PATCH requests must include:

Content-Type: application/json

Request Body

Send data as JSON in the request body:

curl -X POST https://seocrawler.app/api/v1/crawls \
  -H "Authorization: Bearer sc_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "domain_id": "882baa8e-4137-4dec-bb81-cd7402f62365",
    "options": {
      "max_urls": 1000,
      "max_depth": 3,
      "ssl_check": true
    }
  }'

Response Format

All API responses are JSON with consistent structure.

Success Response

Single Resource:

{
  "data": {
    "id": "882baa8e-4137-4dec-bb81-cd7402f62365",
    "domain_name": "example.com",
    "verified": true,
    "created_at": "2024-12-09T10:30:00.000Z"
  }
}

List Response (Paginated):

{
  "data": [
    {
      "id": "882baa8e-4137-4dec-bb81-cd7402f62365",
      "domain_name": "example.com",
      "verified": true,
      "created_at": "2024-12-09T10:30:00.000Z"
    }
  ],
  "meta": {
    "pagination": {
      "page": 1,
      "limit": 20,
      "total": 156,
      "totalPages": 8,
      "hasMore": true
    }
  }
}

Use page and limit query parameters:

GET /api/v1/domains?page=2&limit=50

Timestamps

All timestamps are in ISO 8601 format (UTC):

{
  "created_at": "2024-12-09T10:30:00.000Z",
  "updated_at": "2024-12-09T14:45:00.000Z"
}

Error Handling

All errors follow a consistent format for easy parsing.

Error Response Structure

{
  "error": {
    "message": "Human-readable error message",
    "code": "ERROR_CODE"
  }
}

With Details (Optional):

{
  "error": {
    "message": "A crawl is already in progress for this domain",
    "code": "CRAWL_IN_PROGRESS",
    "details": {
      "existing_crawl_id": "123e4567-e89b-12d3-a456-426614174000"
    }
  }
}

HTTP Status Codes

Common Error Codes

Error Handling Example

try {
  const response = await fetch('https://seocrawler.app/api/v1/crawls', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${apiKey}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      domain_id: '882baa8e-4137-4dec-bb81-cd7402f62365',
      options: {
        max_urls: 500,
        max_depth: 3
      }
    })
  })
  
  if (!response.ok) {
    const error = await response.json()
    
    switch (error.error.code) {
      case 'DOMAIN_NOT_VERIFIED':
        console.error('Please verify domain ownership first')
        break
      case 'CRAWL_IN_PROGRESS':
        console.error('A crawl is already running for this domain')
        // Access details if available
        if (error.error.details?.existing_crawl_id) {
          console.log(`Existing crawl: ${error.error.details.existing_crawl_id}`)
        }
        break
      case 'DAILY_LIMIT_REACHED':
        console.error('Daily crawl limit exceeded')
        if (error.error.details?.usage) {
          console.log(`Usage: ${error.error.details.usage}`)
        }
        break
      case 'RATE_LIMITED':
        const resetTime = response.headers.get('X-RateLimit-Reset')
        console.error(`Rate limited. Resets at ${new Date(resetTime * 1000)}`)
        break
      default:
        console.error(`API Error: ${error.error.message}`)
    }
    
    return
  }
  
  const result = await response.json()
  console.log('Crawl started:', result.data.id)
  console.log('Status:', result.data.status) // "pending"
  
} catch (err) {
  console.error('Network error:', err)
}

Rate Limits

To ensure fair usage and system stability, the API enforces rate limits.

Limits by Resource

Rate Limit Headers

Every API response includes rate limit information:

X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 847
X-RateLimit-Reset: 1699574460

Rate Limit Exceeded

When you exceed the rate limit, you'll receive:

{
  "error": {
    "message": "Rate limit exceeded. Try again in 23 seconds.",
    "code": "RATE_LIMITED"
  }
}

HTTP Status: 429 Too Many Requests

Best Practices

Idempotency

POST requests that create resources support idempotency keys to prevent duplicate operations.

Using Idempotency Keys

Include an Idempotency-Key header with a unique identifier:

curl -X POST https://seocrawler.app/api/v1/crawls \
  -H "Authorization: Bearer sc_live_..." \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: unique-request-id-123" \
  -d '{"domain_id": "abc123"}'

Benefits:

• Safe to retry failed requests without creating duplicates
• Network issues won't cause duplicate crawls
• Same key returns the original response for 24 hours

// Generate idempotency key
const idempotencyKey = `crawl-${domainId}-${Date.now()}`

const response = await fetch('/api/v1/crawls', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${apiKey}`,
    'Content-Type': 'application/json',
    'Idempotency-Key': idempotencyKey
  },
  body: JSON.stringify({
    domain_id: domainId,
    options: {
      max_urls: 1000,
      max_depth: 3
    }
  })
})

const result = await response.json()
console.log('Crawl ID:', result.data.id)

Webhooks

Receive real-time notifications when events occur instead of polling the API.

Available Events

See the Webhooks documentation for setup instructions.

SDK & Libraries

Official SDKs

We're working on official SDKs for popular languages:

• JavaScript/TypeScript - Coming soon
• Python - Coming soon
• PHP - Coming soon
• Ruby - Coming soon

Community Libraries

Until official SDKs are available, here are community examples:

Environment Best Practices

Development

// .env.development
SEO_CRAWLER_API_KEY=sc_test_...  // Test key for development
SEO_CRAWLER_BASE_URL=https://seocrawler.app/api/v1

Production

// .env.production
SEO_CRAWLER_API_KEY=sc_live_...  // Production key
SEO_CRAWLER_BASE_URL=https://seocrawler.app/api/v1

Configuration

// config/api.js
export const apiConfig = {
  apiKey: process.env.SEO_CRAWLER_API_KEY,
  baseURL: process.env.SEO_CRAWLER_BASE_URL,
  timeout: 30000,
  retries: 3,
}

// Validate configuration on startup
if (!apiConfig.apiKey) {
  throw new Error('SEO_CRAWLER_API_KEY environment variable required')
}

if (!apiConfig.apiKey.startsWith('sc_')) {
  throw new Error('Invalid API key format')
}

Next Steps

Ready to dive deeper? Explore these resources:

Start crawls and retrieve link health data.

Manage domains and verification status.

Set up real-time event notifications.

Export crawl results in multiple formats.

Support

Need help with the API?

Check system status and uptime.

Get help from our technical team.