Wayback Machine API Archive Recovery

Problem

When recovering data from defunct services, you need to know which API endpoints were archived by the Wayback Machine, and how to access them. Not all endpoints are archived - authenticated endpoints typically aren't captured.

Context / Trigger Conditions

• Recovering data from a shut-down service (Vine, old Twitter API, defunct platforms)
• Building an archive crawler for digital preservation
• Need to reconstruct engagement data (comments, likes, followers) from archives
• Getting empty results when trying to fetch archived API endpoints

Solution

Step 1: Discover What Was Archived

Use the CDX API to search for archived endpoints:

# Find all archived endpoints under a path prefix
curl "https://web.archive.org/cdx/search/cdx?url=vine.co/api/posts/&matchType=prefix&output=json&fl=original,timestamp&filter=statuscode:200&limit=100"

# Check total pages available
curl "https://web.archive.org/cdx/search/cdx?url=vine.co/api/&matchType=prefix&showNumPages=true"

Key CDX parameters:

• matchType=prefix - Match URL prefixes with wildcards
• fl=original,timestamp - Select which fields to return
• filter=statuscode:200 - Only successful captures
• collapse=urlkey - Deduplicate by URL
• limit=N - Cap results

Step 2: Understand What Gets Archived

Typically archived (public endpoints):

• Comment lists: /api/posts/{id}/comments
• User profiles: /api/users/profiles/{id}
• Timelines: /api/timelines/users/{id}
• Public feeds: /api/timelines/promoted

Typically NOT archived (require authentication):

• Like lists: /api/posts/{id}/likes
• Follower/following lists: /api/users/{id}/followers
• Private data: DMs, settings, notifications

Step 3: Fetch Archived API Responses

Use the id_ modifier to get raw JSON without the Wayback toolbar:

# Without id_ - may return HTML wrapper
curl "https://web.archive.org/web/20161209032217/https://vine.co/api/posts/123/comments"

# With id_ - returns raw JSON
curl "https://web.archive.org/web/20161209032217id_/https://vine.co/api/posts/123/comments"

Step 4: Extract Embedded Data as Fallback

When API endpoints weren't archived, check if data was embedded in archived web pages:

# Many SPAs embed initial data in script tags
import re
import json

html = fetch_archived_page(url)

# Look for embedded JSON (common patterns)
patterns = [
    r'window\.POST_DATA\s*=\s*({.*?});',      # Vine
    r'window\.__INITIAL_STATE__\s*=\s*({.*?});',  # Redux apps
    r'<script type="application/json"[^>]*>({.*?})</script>',
]

for pattern in patterns:
    match = re.search(pattern, html, re.DOTALL)
    if match:
        data = json.loads(match.group(1))

Caveat: Embedded data is often partial (e.g., only first 3 comments rendered).

Step 5: Infer Missing Data from Interactions

When follower/following lists aren't available, infer relationships:

# If A commented on B's post → A probably follows B
inferred_follows.append({
    'follower': commenter_id,
    'followee': post_creator_id,
    'inference_type': 'commented_on_post',
    'confidence': 0.7
})

# If A mentioned @B → A probably follows B
# If A and B mutually commented → bidirectional follow (collaborators)

Verification

1. CDX query returns results with status 200
2. Fetching with id_ modifier returns valid JSON
3. Parsing embedded data produces expected structure

Example

import urllib.request
import json

def find_archived_endpoints(base_url):
    """Find all archived API endpoints for a service."""
    cdx_url = f"https://web.archive.org/cdx/search/cdx?url={base_url}&matchType=prefix&output=json&fl=original,timestamp&filter=statuscode:200"

    with urllib.request.urlopen(cdx_url, timeout=60) as resp:
        data = json.loads(resp.read())

    # Skip header row, extract unique URLs
    endpoints = {}
    for row in data[1:]:
        url, timestamp = row[0], row[1]
        if url not in endpoints:
            endpoints[url] = timestamp

    return endpoints

def fetch_archived_json(url, timestamp):
    """Fetch archived JSON using id_ modifier."""
    archive_url = f"https://web.archive.org/web/{timestamp}id_/{url}"

    with urllib.request.urlopen(archive_url, timeout=30) as resp:
        return json.loads(resp.read())

# Usage
endpoints = find_archived_endpoints("vine.co/api/posts/")
print(f"Found {len(endpoints)} archived endpoints")

# Filter to comment endpoints
comments = {k: v for k, v in endpoints.items() if '/comments' in k}

Notes

• Wayback Machine rate limits requests - add delays (1+ second) between fetches
• Archived timestamps vary - same endpoint may have captures from different dates
• Paginated APIs: check for page, offset, cursor parameters in archived URLs
• Some archives return 302 redirects - follow them or check for "stale" archives
• The CDX API itself can be slow for large result sets - use pagination

References

• Wayback Machine CDX API
• Internet Archive APIs