Extract Site Content

Crawls a public website in a real browser and saves what each page says, word for word: one readable text file and one data file per page, plus site-wide details, structured data and a manifest of what was captured. Pages drawn by JavaScript and content hidden in collapsed sections are captured too.

Nothing is summarised or invented. A page that could not be read is reported as a failure, never filled in. Paths the site's robots rules disallow are not fetched and are listed. Up to 150 pages can be captured; the default is 100.

Everything goes into one output folder that the tool owns, and a re-run overwrites that folder, so hand-written files should never be kept there. The crawl currently runs on the Syvon render service.