There are five main features WebCopy (and Sitemap Creator) need based on user feedback and our own observations. In no particular order, these are making the product easier to use, supporting multiple downloads at once, being able to pause and resume a copy, JavaScript support and authentication. The current plan is to address three of the five in WebCopy 1.7, starting with authentication.
Since the earliest days of WebCopy, it has supported challenge authentication (where a web browser prompts you for credentials) and form based authentication (where you enter credentials into a web page). Almost all web sites use the latter approach and with WebCopy this can either be tricky to configure or impossible due to websites using interactive methods such as authenticators or captcha codes.
In order to resolve this, WebCopy 1.7 includes a new option named Log in use web browser, found in the Passwords property page.
When this new option is set, WebCopy will display a browser window when you copy a website to allow you to authenticate with the web site. Once authenticated, the cookies associated with the site are applied to WebCopy's crawler and copying will commence.
This new feature is currently available in 1.7 nightly builds, please let us know if this feature helps or hinders!
You can learn more about this feature and any caveats on the documentation page.
Update History
- 2018-10-31 - First published
- 2020-11-23 - Updated formatting
Like what you're reading? Perhaps you like to buy us a coffee?
Comments
The founder of Cyotek, Richard enjoys creating new blog content for the site. Much more though, he likes to develop programs, and can often found writing reams of code. A long term gamer, he has aspirations in one day creating an epic video game - but until that time comes, he is mostly content with adding new bugs to WebCopy and the other Cyotek products.