wget –spider = Head Request

In my Tasks Pro™ and Tasks documentation, I have example code for setting up CRON jobs to load URLs. In the case of my software, those URLs kick off the e-mail reminders and activate recurring tasks.

In the example code, I included a ‘–spider’ option for wget. This option keeps wget from saving the downloaded page as a file – handy for not cluttering up your home directory.

Unfortunately, the –spider option means that wget only does a head request for the file which may not cause the code in the file to be executed. I had two virtually identical commands set up (with –spider) – one worked and one didn’t.

I’ve updated the online versions of the documentation to exclude the –spider option (good thing the distributed docs also include a link to the online version of the page).

I’ve received the following recommendations:

  1. For wget, pass in a filename with -O

    wget -q -O temp.txt http://example.com....

    so that each run will overwrite the same file.

  2. For curl, pipe the output to /dev/null

I’m not a server geek, so concrete examples (especially using the examples in the doc pages) would be quite welcome.

This post is part of the project: Tasks Pro™. View the project timeline for more context on this post.