This is a preview version of Cyotek's blog and may be missing functionality and/or unstable. Please visit https://www.cyotek.com/blog for the current version of the blog.
If you encounter any problems using this preview site, please contact us with the details.

WebCopy 1.8 - JavaScript Support

Cyotek WebCopy 3 Comments

One of the long standing requests/complaints is for WebCopy to support JavaScript enabled websites, e.g. modern SPA's where JavaScript is used to build the page. Traditionally this is something I have always put onto the furthest of back burners as in order to support this natively I'd have to essentially write half a browser, something that would be a full time job and a half and not something I'm interested in doing. Other solutions did exist but I never really looked into them.

It recently occurred to me however, that I'd put into place all the building blocks I needed to have WebCopy support JavaScript execution (in a limited fashion, more on this later) using Internet Explorer. And it was easy, in fact, the hardest part was sorting out threading issues - despite the fact that WebCopy currently only crawls on a single thread, it does run on a different thread to the UI in order not to freeze it, which COM can have a problem with.

Read More

WebCopy 1.8 - New Project Wizard

Cyotek WebCopy 0 Comments

In my previous post regarding WebCopy 1.8, I briefly covered a general grab-bag of some of the new features in this version. This post is dedicated to another new feature, the New Project Wizard.

Whilst you can still create a new blank project as with previous version of WebCopy, there's also a new GUI that will ask a series of questions and create a neatly configured project.

Read More

Introducing WebCopy 1.8

Cyotek WebCopy 0 Comments

It's been over two months since the last CI build of WebCopy was made, and during this time I've been working quite hard on some major internal refactoring and adding a long requested feature. It hope it's worth the wait, I need a break!

WebCopy 1.8 nightly builds are now available for download and so this series of posts will describe some of the changes and new functionality that have been made to the software. This first post will cover a grab bag of smaller changes.

Read More

Website history gallery

website 1 Comments

The Cyotek (and Ariad Software) websites have gone through a lot of iterations over the years. The current website software dates from 2011, with the design from 2013, and both are in drastic need of an overhaul. As I plan on redeveloping the site using ASP.NET Core, I'm also going to introduce a new look at the same time. With that in mind, I thought it might be interesting (for myself at least!) to put together a rogues gallery of some of the iterations.

Fortunately I still have the archived source code for the oldest versions on this page, and the Internet Archive was able to supply snapshots for iterations between version 10 and 12. Some of the designs I forgot were ever a thing!

Read More

WebCopy 1.7 - local file name generation

Cyotek WebCopy 0 Comments

As part of WebCopy 1.7's mission to reduce user confusion and make the product more appealing, a pair of new options for controlling local file name generation have been introduced, as well as correcting a potentially confusing bug.

By default, WebCopy will name local files to match their content type. For example, if you download the homepage of a website which is named index.php, WebCopy will save a local file named index.html - end users would probably very confused trying to open a .php file and either the operating system doesn't know how to handle it, or it executes the PHP runtime.

Read More

WebCopy 1.7 - tls/ssl invalid certificate handling

Cyotek WebCopy 1 Comments

You should add an option to ignore checking for an SSL certificate.

The above quote is the last part of a piece of uninstallation feedback I received about WebCopy on Friday. This isn't the first time I've had an anonymous feedback about ignoring SSL errors and each time it has happened it has been frustrating and even bewildering as the option is already there and has been since 2013!

Read More

WebCopy 1.7 - web browser authentication

Cyotek WebCopy Cyotek Sitemap Creator 0 Comments

There are five main features WebCopy (and Sitemap Creator) need based on user feedback and our own observations. In no particular order, these are making the product easier to use, supporting multiple downloads at once, being able to pause and resume a copy, JavaScript support and authentication. The current plan is to address three of the five in WebCopy 1.7, starting with authentication.

Since the earliest days of WebCopy, it has supported challenge authentication (where a web browser prompts you for credentials) and form based authentication (where you enter credentials into a web page). Almost all web sites use the latter approach and with WebCopy this can either be tricky to configure or impossible due to websites using interactive methods such as authenticators or captcha codes.

Read More

WebCopy 1.5 beta released

0 Comments

A new beta of WebCopy has been released, one I hope will resolve some of the plethora of uninstall feedback I've been receiving recently relating to users having problems logging into web sites using WebCopy.

When WebCopy posts a form, it downloads the existing form, pulls out all the values and merges these with the specific values the user has configured. That way, you can easily post login forms that include hidden parameters containing anti-forgery tokens or other meta data that may be generated dynamically.

Read More

WebCopy 1.4 beta released

Cyotek WebCopy

A beta version of WebCopy 1.4 complete with a fundamental change to how rules are ran, various performance improvements, UI tweaks and miscellaneous bug fixes has been released.

In previous versions of WebCopy, rule processing would stop as soon as the first rule was matched. This made it impossible to do standard tasks like exclude all HTML pages from being downloaded (but still scan them) and only download image resources, as an example.

Read More

On Product Feedback

All Products 1 Comments

It's been a few months since the installation programs for our products were updated to request feedback on when the affected product is uninstalled. Since percolating through our different release channels we've been receiving more and more feedback, particularly regarding WebCopy.

What surprises me is that while the vast majority is anonymous "reason code" feedback without context, there is quite a lot of quality and useful feedback describing problems and shortcomings. Which is really helpful, and I wish I had thought to put this in years ago. (It was only after a third party tool I use prompted for feedback and then fixed the issue that my feedback related to that I belatedly thought "Oh, wouldn't this be a good thing to do".)

Read More