WebCopy 1.7 - web browser authentication

There are five main features WebCopy (and Sitemap Creator) need based on user feedback and our own observations. In no particular order, these are making the product easier to use, supporting multiple downloads at once, being able to pause and resume a copy, JavaScript support and authentication. The current plan is to address three of the five in WebCopy 1.7, starting with authentication.

Since the earliest days of WebCopy, it has supported challenge authentication (where a web browser prompts you for credentials) and form based authentication (where you enter credentials into a web page). Almost all web sites use the latter approach and with WebCopy this can either be tricky to configure or impossible due to websites using interactive methods such as authenticators or captcha codes.

In order to resolve this, WebCopy 1.7 includes a new option named Log in use web browser, found in the Passwords property page.

The new option to enable using a web browser for authentication
The new option to enable using a web browser for authentication

When this new option is set, WebCopy will display a browser window when you copy a website to allow you to authenticate with the web site. Once authenticated, the cookies associated with the site are applied to WebCopy's crawler and copying will commence.

Using an embedded web browser to authenticate with the website to be copied
Using an embedded web browser to authenticate with the website to be copied

This new feature is currently available in 1.7 nightly builds, please let us know if this feature helps or hinders!

You can learn more about this feature and any caveats on the documentation page.

Update History

  • 2018-10-31 - First published
  • 2020-11-23 - Updated formatting

Comments

We'll never share your email with anyone else Styling with Markdown is supported