Skip to content
cccskills
FreePublished on GitHub

wayback-machine-raw-content-id-modifier

Fix JSON parse errors when fetching archived API responses from Wayback Machine. Use when: (1) Getting "Expecting value: line 1 column 1 (char 0)" JSON decode errors from archived URLs, (2) Wayback returns HTML instead of expected JSON/raw content, (3) Crawling archived REST APIs or JSON endpoints from web.archive.org. The `id_` modifier returns raw content without the Wayback toolbar wrapper.

The English summary is being prepared. Showing the original description.

divinevideo/divine-mobile
See how to install

Files included (1)

  • SKILL.md3.7 KB

SKILL.md (original)

Review the instructions the agent will receive before you install.

Wayback Machine Raw Content with id_ Modifier

Problem

When fetching archived JSON API endpoints from the Wayback Machine, you get HTML-wrapped content with the Wayback toolbar instead of the raw JSON response. This causes JSON parse errors like "Expecting value: line 1 column 1 (char 0)" because the response starts with <!DOCTYPE html> instead of valid JSON.

Context / Trigger Conditions

  • Fetching archived API endpoints from web.archive.org/web/{timestamp}/{url}
  • JSON parsing fails with "Expecting value: line 1 column 1 (char 0)"
  • Response content starts with HTML instead of expected JSON
  • Crawling archived REST APIs, JSON feeds, or any non-HTML content from Wayback
  • Using Python json.loads(), response.json(), or similar JSON parsing

Solution

Add id_ after the timestamp in the Wayback URL to get raw content:

Default (HTML-wrapped):

https://web.archive.org/web/20170112012313/https://vine.co/api/users/profiles/123

Raw content (add id_):

https://web.archive.org/web/20170112012313id_/https://vine.co/api/users/profiles/123

Code Fix Pattern

# BEFORE (broken - returns HTML)
url = f"https://web.archive.org/web/{timestamp}/https://example.com/api/data"

# AFTER (works - returns raw JSON)
url = f"https://web.archive.org/web/{timestamp}id_/https://example.com/api/data"

Other Wayback Modifiers

  • id_ - Raw/identity (no modifications, returns original content)
  • if_ - Iframe embed mode
  • js_ - JavaScript rewriting mode
  • cs_ - CSS rewriting mode
  • im_ - Image mode

Verification

  1. Test the URL with curl to see actual response:

    # Without id_ - shows HTML with Wayback toolbar
curl -s "https://web.archive.org/web/20170112012313/https://example.com/api/data" | head -5

# With id_ - shows raw JSON
curl -s "https://web.archive.org/web/20170112012313id_/https://example.com/api/data" | head -5

  2. Verify JSON parsing works:

    import json
import urllib.request

url = f"https://web.archive.org/web/{timestamp}id_/{api_url}"
with urllib.request.urlopen(url) as resp:
    data = json.loads(resp.read())  # Should work now

Example

Crawling archived Vine API profiles:

WAYBACK_BASE = "https://web.archive.org/web"

def fetch_profile(user_id: str, timestamp: str):
    # Use id_ modifier to get raw JSON instead of HTML-wrapped content
    url = f"{WAYBACK_BASE}/{timestamp}id_/https://vine.co/api/users/profiles/{user_id}"

    with urllib.request.urlopen(url) as resp:
        data = json.loads(resp.read())
        return data['data']  # Now works correctly

Notes

  • The id_ modifier works for any content type, not just JSON (images, CSS, JS, etc.)
  • Some archived content may still fail if it was never properly captured
  • CDX API queries (for finding archived URLs) don't need the modifier
  • Rate limit requests to archive.org (5+ seconds between requests recommended)
  • Empty responses (0 bytes) indicate the archive entry exists but content wasn't captured

References

Reviews

No reviews yet. Tell others what you thought.

Comments

No login needed. Share how it worked for you, ask a question, or note where it did not run.

No comments yet. Be the first to write one.

Comments are published right away. Do not include personal information such as email addresses or phone numbers. Up to 2 URLs.

0 / 1,000 characters

This form is protected by reCAPTCHA, and the Google Privacy Policy and Terms of Service apply.

More skills from this repository

argocd-externalsecret-namespace-permission

Free

Fix ArgoCD ExternalSecret deployment failing with "namespace X is not permitted in project Y". Use when: (1) ExternalSecret shows OutOfSync in ArgoCD but won't sync, (2) ArgoCD application status shows "namespace X is not permitted in project 'infrastructure'", (3) ExternalSecret targets a namespace managed by a different ArgoCD project, (4) Using apps-of-apps pattern with separate infrastructure and application projects.

The English summary is being prepared. Showing the original description.

divinevideo/divine-mobile266Updated Oct 19, 2026

art-direct

Free

Art direction for any content — reads text, PDF, Word, HTML, PPT, then proposes 2-3 creative directions with photography style, mood, and visual language. After selection, generates AI image prompts and visual briefs section-by-section. Use when the user shares content and needs visual direction, image sourcing, or creative direction for any material.

The English summary is being prepared. Showing the original description.

divinevideo/divine-mobile266Updated Oct 19, 2026

async-await-null-race-condition

Free

Fix "Null check operator used on a null value" errors when an object is set to null during an async await. Use when: (1) Object reference is nullified while awaiting, (2) Code accesses object with ! after await returns, (3) Cancel/dispose operations run concurrently with async operations on same object. Solution: capture local reference before await.

The English summary is being prepared. Showing the original description.

divinevideo/divine-mobile266Updated Oct 19, 2026

aws-v4-signing-custom-headers-gcs

Free

Add custom metadata headers (x-amz-meta-*) to AWS v4 signed requests for GCS S3-compatible API. Use when: (1) Adding custom metadata to GCS uploads via S3 API, (2) Getting signature mismatch errors after adding new headers, (3) x-amz-meta-* headers being ignored or causing 403 errors. Custom headers MUST be included in canonical headers and signed headers list.

The English summary is being prepared. Showing the original description.

divinevideo/divine-mobile266Updated Oct 19, 2026

bash-herestring-newline-secrets

Free

Fix password/secret authentication failures caused by trailing newlines when creating Google Cloud secrets (or similar) with bash here-strings. Use when: (1) Password authentication fails with correct password, (2) Secret created with `<<< "value"` syntax, (3) Error like "password authentication failed" or "invalid token" despite correct value. Bash here-strings (`<<<`) add a trailing newline that corrupts secrets.

The English summary is being prepared. Showing the original description.

divinevideo/divine-mobile266Updated Oct 19, 2026

blossom-content-addressed-url-filtering

Free

Fix silent video/media processing failures caused by URL extraction code that filters on file extensions (.mp4, .webm, .webp). Use when: (1) Media moderation, transcoding, or analysis silently skips files from Blossom or content-addressed storage servers, (2) URL extraction from Nostr event tags (imeta, r tags) drops URLs without recognized extensions, (3) CDN fallback URLs append .mp4 but the actual server uses extensionless content-addressed paths like /{sha256}. Common in Nostr video events (kind 34236) where different clients use different URL formats.

The English summary is being prepared. Showing the original description.

divinevideo/divine-mobile266Updated Oct 19, 2026

See all skills by divinevideo

Report a problem with this skill