Domains

Overview

Domains are the foundation of SEO Crawler. Each domain represents a website you want to monitor for broken links, SEO issues, and overall link health.

A domain in SEO Crawler:

• Represents a single website or subdomain
• Requires DNS verification before crawling
• Has its own crawl history and reports
• Tracks link health independently
• Can have custom crawl settings

What Counts as a Domain?

Each unique hostname counts as one domain toward your plan limit.

Examples

Subdomains

Each subdomain is a separate domain:

mysite.com           → Domain 1
blog.mysite.com      → Domain 2
shop.mysite.com      → Domain 3
api.mysite.com       → Domain 4

If you need to monitor multiple subdomains, consider the Agency plan with unlimited domains.

Domain Lifecycle

Every domain goes through these stages:

Domain is added to your account with a unique verification token.

You add the DNS TXT record to prove ownership.

DNS record detected, domain ownership confirmed.

Regular crawls run, link health tracked, reports generated.

Domain deleted from account, all data removed.

Domain Status Indicators

Verification Status

Health Status

Domain Dashboard

The domains dashboard shows all your monitored websites at a glance.

Dashboard Columns

Quick Actions

From the domain dashboard, you can:

• Start Crawl - Immediately trigger a new scan
• View Details - See comprehensive domain information
• Crawl History - Review all previous crawls
• Export Report - Download PDF or CSV
• Settings - Configure crawl options
• Delete - Remove domain and all data

Adding Domains

Quick Add Process

1. Click "Add Domain" button
2. Enter domain name (e.g., mysite.com)
3. Click "Continue"
4. Copy verification token
5. Add DNS TXT record
6. Click "Check DNS"
7. Start crawling! ✓

Domain Input Normalization

SEO Crawler automatically cleans your input:

# All of these become: example.com
https://example.com
http://www.example.com
EXAMPLE.COM
example.com/
example.com:443
www.example.com/homepage

Domain Verification

Why Verification is Required

Domain verification ensures:

• ✅ You own or control the domain
• ✅ Prevents unauthorized crawling
• ✅ Protects website owners' privacy
• ✅ Complies with responsible crawling practices

Verification Methods

Currently supported: DNS TXT Record

Type:  TXT
Name:  @ (or subdomain name)
Value: seo-crawler-verify=your-unique-token

Coming Soon:

• HTML meta tag verification
• File upload verification
• Email verification for small sites

Re-verification

Domains require continuous verification. If you remove the DNS TXT record:

• Domain becomes unverified
• New crawls are blocked
• Existing data remains accessible
• Re-add TXT record to restore access

Domain Settings

Each domain has customizable settings for crawling behavior.

Crawl Configuration

Notification Settings

Configure how you're notified about domain issues:

Custom Settings

Agency Plan Features:

• Custom User-Agent - Set your own crawler identification
• Custom Headers - Add authentication or special headers
• IP Whitelist - Crawl from specific IP ranges
• Priority Crawling - Jump the queue for urgent scans

Managing Multiple Domains

Organizing Domains

Use these strategies to manage many domains:

Domain Limits by Plan

Domain Analytics

Per-Domain Metrics

Each domain tracks:

Historical Data

View trends over time:

• Health score graph (30/60/90 days)
• Broken link trends
• New vs. fixed issues
• Crawl duration patterns
• Response time changes

Domain Security

Access Control

Who can access your domain data:

Sharing Crawl Results

Control who sees your data:

Domain Best Practices

Before Adding a Domain

Ensure you can add TXT records to your DNS provider before adding the domain.

Check if your `robots.txt` blocks crawlers. Add our user-agent if needed:

User-agent: SEOCrawlerBot
Allow: /

Decide on max URLs and depth based on your site size and goals.

Ongoing Maintenance

• ✅ Keep DNS record active - Don't remove the verification TXT record
• ✅ Monitor health trends - Check dashboard weekly
• ✅ Fix broken links promptly - Address issues as they appear
• ✅ Update exclusions - Add new paths to skip (admin, APIs)
• ✅ Review crawl settings - Adjust as your site grows
• ✅ Use automation - Schedule regular crawls

Performance Optimization

Removing Domains

What Gets Deleted

When you delete a domain, we permanently remove:

• ❌ All crawl history and results
• ❌ All broken link data
• ❌ All reports and exports
• ❌ All automation schedules
• ❌ Verification status
• ❌ Custom settings and configurations

Before You Delete

1. Export final report - Save crawl data as PDF/CSV
2. Cancel automations - Stop scheduled crawls
3. Revoke share links - Disable public access
4. Download historical data - Archive what you need
5. Inform team members - Notify anyone with access

How to Delete

Go to the domain you want to delete.

At the bottom of the domain settings page.

Review the warning and confirm deletion.

Enter the exact domain name to confirm.

Click "Delete Permanently" - domain is immediately removed.

Re-adding a Deleted Domain

If you delete a domain and want to add it back:

• Same domain name allowed
• New verification token required
• All previous data is gone
• Starts fresh with new crawls

Troubleshooting

API Access

Manage domains programmatically with our API:

# List all domains
GET /api/v1/domains

# Get domain details
GET /api/v1/domains/{domain_id}

# Add new domain
POST /api/v1/domains

# Update domain settings
PATCH /api/v1/domains/{domain_id}

# Delete domain
DELETE /api/v1/domains/{domain_id}

Next Steps

Step-by-step guide to adding domains.

Complete DNS verification process.

Run a link health scan.

Schedule recurring crawls.