markdown-new-crawl

---

name: markdown.new-crawl

description: Use `https://markdown.new/crawl/{target_url}` endpoints to recursively crawl a site section and return markdowns. Trigger this skill when the user asks for multi-page extraction, whole-docs crawl, link-depth crawling, or job-based crawl polling from a URL. Prefer local terminal access (`curl`) with `/crawl`, `/crawl/status/{jobId}`, and `/crawl/{url}` before other browsing methods.

---

Markdown.New Crawl Local-First Access

Use `markdown.new/crawl` for multi-page crawling and async Markdown generation.

Required Behavior

1. Prefer local API calls with `curl` (or any suitable alternative tools).

2. Assume no authentication is required unless the service behavior changes.

3. Start with `POST /crawl` to get a job ID.

4. Poll `GET /crawl/status/{jobId}` until crawl completes.

5. Default to Markdown output; use `?format=json` only when structured output is needed.

6. Keep crawl scope explicit (`limit`, `depth`, subdomain/external toggles, include/exclude patterns).

7. State default crawl behavior: same-domain links only, up to `500` pages per job.

8. If `/crawl` fails (network, timeout, blocked host), fall back to another method and state the fallback.

Core Commands

curl -X POST "https://markdown.new/crawl" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://docs.example.com","limit":50}'

curl "https://markdown.new/crawl/status/<jobId>"

curl "https://markdown.new/crawl/status/<jobId>?format=json"

curl "https://markdown.new/crawl/https://docs.example.com"

curl -X DELETE "https://markdown.new/crawl/status/<jobId>"

API Coverage

Common Crawl Options

Output Notes

Operational Limits