Wayback CDX API: Wildcard Query Pagination

Problem

The Wayback Machine CDX API's page=N pagination parameter returns empty results when combined with wildcard URL queries (url=domain.com/path/*), even though the same query without page= returns data. This causes scripts to incorrectly report "no results found" when there are actually hundreds of thousands of results.

Context / Trigger Conditions

• CDX query with url=*.example.com/* or url=example.com/path/* and page=0 returns 0 bytes
• Same query without page= parameter returns expected data
• Using web.archive.org/cdx/search/cdx API endpoint
• matchType=prefix + collapse=urlkey + page=N combination also fails silently
• Script works with limit=5 but fails when adding page=0

Solution

Option 1: Use showResumeKey=true (Recommended)

The most reliable pagination method. CDX appends a base64 resume token after a blank line separator in the response.

resume_key = None
while True:
    params = {
        'url': 'example.com/path/*',
        'output': 'text',
        'fl': 'original',
        'filter': 'statuscode:200',
        'limit': '10000',
        'showResumeKey': 'true',
    }
    if resume_key:
        params['resumeKey'] = resume_key

    data = fetch_cdx(params)
    if not data:
        break

    # Resume key is after the last blank line
    parts = data.rstrip().rsplit('\n\n', 1)
    if len(parts) == 2:
        data_lines = parts[0].strip().split('\n')
        resume_key = parts[1].strip()
    else:
        data_lines = parts[0].strip().split('\n')
        resume_key = None

    # Process data_lines...

    if not resume_key or len(data_lines) < limit:
        break

Option 2: Use offset=N

Works but less efficient for very large result sets.

offset = 0
limit = 10000
while True:
    params = {
        'url': 'example.com/path/*',
        'output': 'text',
        'fl': 'original',
        'limit': str(limit),
        'offset': str(offset),
    }
    data = fetch_cdx(params)
    lines = data.strip().split('\n')
    if not lines or not lines[0]:
        break
    offset += len(lines)

What NOT to do

# THIS RETURNS EMPTY for wildcard queries:
params = {
    'url': 'example.com/path/*',
    'page': '0',        # <-- INCOMPATIBLE with wildcard
    'limit': '10000',
}

# THIS ALSO FAILS from some IPs:
params = {
    'url': 'example.com/path/',
    'matchType': 'prefix',
    'collapse': 'urlkey',
    'page': '0',
}

Verification

• Query with showResumeKey=true and no page= param returns data
• Response ends with a blank line followed by a base64 token (the resume key)
• Subsequent request with resumeKey=<token> returns the next page

Example

Fetching all vine.co/oembed/* URLs (690K+ captures, 312K unique vine IDs):

# This works:
curl "https://web.archive.org/cdx/search/cdx?url=vine.co/oembed/*&output=text&fl=original&limit=10000&showResumeKey=true"

# This returns empty:
curl "https://web.archive.org/cdx/search/cdx?url=vine.co/oembed/*&output=text&fl=original&limit=10000&page=0"

Notes

• page=N works fine for non-wildcard queries (e.g., exact URL lookups)
• The showResumeKey approach is server-side cursor-based, more efficient than offset
• Keep limit at 10000 or less to avoid timeouts, especially from cloud IPs
• Always add time.sleep(3-5) between pages to be polite to the CDX server
• The CDX API has no official documentation for this incompatibility

References

• CDX API informal docs: https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server
• CDX pagination: The page API was designed for the pywb CDX server and may not be fully compatible with all query modes on the production Wayback CDX