grabbing http files with perl cgi

[colulus]

I have looked through the Perl Cookbook for how to grab files from the web and drew a blank. There were directions on how to write a script to log into an ftp server and get things from it, but no instructions on simply grabbing a file from an http server.

Any suggestions? A search on terms "cgi perl download" or somesuch in the search engines is understandably unproductive. It seems the http module must have something on how to do this but I havent found anything yet. I've found lots on info on using the http module to simulate a browser, but not on just downloading a file.

TIA,
JF

[bturner]

Look into the LWP module. It's simple to use.

For example, here's a short script to download the yahoo.com home page, and save it as a local file yahoo.html (just saves the HTML code, no images, but you may be able to rewrite the script to get the images too.)

#!/usr/bin/perl

use LWP::Simple;

mirror ('http://www.yahoo.com/','yahoo.html');

That's it.

The LWP Cookbook has many more examples as well: http://www.perldoc.com/perl5.6.1/lib/lwpcook.html

Have fun!
Bill

[colulus]

Thanks! That's what I was looking for.

I would think, for the example of saving an html file in particular, that saying which browser you are would be important. I can see from http://www.perldoc.com/perl5.6.1/lib/lwpcook.html that LWP::UserAgent lets you do that as well.

For what I need, though, LWP::Simple is just right.

Thanks again!

[Daytripper_MI6]

I use 1st Page 2000 to do some html work and under the "file" menu it has an option to "open from web" an HTML document. So I typed in http://www.yahoo.com and it opened it right in the program. Edit view of the source code and preview of the page itself. Thought it was kinda neat, don't know if that was what you were trying to do, but I thought I'd just throw in my 2 cents worth cha-ching!

[colulus]

Hi Daytripper,
That's not what I had in mind, but thanks.

What I had in mind, which can be implemented with Perl's LWP module as described above, is to grab files from one site directly into another.

For example, copyright infringement aside, I could download an image from www.lotsaphotoshere.com directly into my website and display it in a page on my website (perhaps as the background for the page), just by filling out a simple form on my site. This would obviate the need to download the picture to my desktop first and then upload it to my site.

Another example would be that I write a script, ie homepage.cgi, and then give it to friends for their websites. A related script, updater.cgi, could be set to run each day on their sites by a cron job to look for updated versions of homepage.cgi at a download section of my website; if updater.cgi notes an update has occurred, it could directly download and install the latest version. In this way, my friends wouldnt have to worry about the backend of their site staying up to date, nor would I need to have their password to update it for them. All I'd have to do to update their sites is place an updated version of homepage.cgi, or perhaps a zipped version of it, in the pre-arranged download directory. This would be good for distributing bug-fixes and security updates for the script, as well as feature enhancements.

[MikeGnitecki]

Originally posted by colulus:
Another example would be that I write a script, ie homepage.cgi, and then give it to friends for their websites. A related script, updater.cgi, could be set to run each day on their sites by a cron job to look for updated versions of homepage.cgi at a download section of my website; if updater.cgi notes an update has occurred, it could directly download and install the latest version. In this way, my friends wouldnt have to worry about the backend of their site staying up to date, nor would I need to have their password to update it for them. All I'd have to do to update their sites is place an updated version of homepage.cgi, or perhaps a zipped version of it, in the pre-arranged download directory. This would be good for distributing bug-fixes and security updates for the script, as well as feature enhancements.

Sorry if I'm getting too off-topic, and I realize that you just used this is as a theoretical example, but I wanted to comment on this. I'd really recommend that people avoid creating auto-updating scripts or even using auto-updating scripts. In my opinion, the security risks outweigh the benefits.

There are two main supposed benefits. The first one is that the end user would never have to update the program manually. The second one is that the author of the script could use the auto-update to patch security holes quickly. But in order for the main benefits to exist, the author must be actively developing the program. In addition, the author must quickly respond to and patch security issues. And here's what could go wrong:
[list=a][*]The end user might have customized to the program. After an auto-update, the customizations would be lost. In addition, if the customizations involved changing the format of databases in some way, the program could stop running and the databases could possibly be corrupted.[*]The author could mistakenly add a bug into his program, which could add additional security issues or stop the program from running. In addition, the bug could stop auto-update from working again, leaving the end user with a broken program until a manual update was done.[*]Someone could hack into the author's web-site and insert a trojan / virus into the program. All of the end users would be affected within a short period of time. This could be used to stop the program from running, give the hacker passwords, or permanently corrupt databases. In addition, the end user could be forced to manually update to get rid of the trojan / virus.[/list=a]

Some might argue that the chance of the program being targeted is unlikely. But if a large web-site were to use the program, hacking the author's web-site and adding a trojan / virus to the program could be the best way to hurt the large web-site.

Some might also argue about how much damage could be done to the web-site. But imagine if the trojan / virus was in forum software. What if the forums for a large web-site stopped running? What if the databases were intentionally corrupted? What if every member's username and password was e-mailed to the hacker? What if the hacker granted himself administrator rights on the forums and harassed users or deleted posts?

In conclusion, I recommend that people stay away from any auto-update features at the current time. There's too much at stake to risk security for a little convenience.

-Mike Gnitecki

[Note: I haven't studied auto-update features in depth recently, so it is possible that security measures were developed. I apologize in advance if auto-update systems have found a way to stop the above examples from happening.]

[colulus]

Hi Mike,
The example I noted was for non-tech friends. It would not be a public, open-source solution, so shouldnt attract any hacker attention, at least not as much as something on sourceforge could. Also, rather than have an auto-updater, I think it would be better to have an admin page with a button to push for updating the backend of the system. It would then run the updater script to download and install the material.
I'd prefer to open an SSH connection from site to site so the files could be stored off the web (ie, outsite the www directory in FQ), but I dont know how to do that with perl cgi. I guess that's another thread.

[colulus]

Actually, re the SSH issue, I wouldnt really want to allow a script to SSH into my whole site, but it would be nice to access some protected subdirectory that way. If one of the secondary ftp accounts could be used for connecting via SSH as well, for ssh-piped ftp with read-only capability, that would be ideal.

[MikeGnitecki]

The example I noted was for non-tech friends. It would not be a public, open-source solution, so shouldnt attract any hacker attention, at least not as much as something on sourceforge could. Also, rather than have an auto-updater, I think it would be better to have an admin page with a button to push for updating the backend of the system. It would then run the updater script to download and install the material.

I'd prefer to open an SSH connection from site to site so the files could be stored off the web (ie, outsite the www directory in FQ), but I dont know how to do that with perl cgi. I guess that's another thread.

colulus:

A privately-distributed program would definitely attract less attention. The push-button-updating would be safer for your end users, too.

After reading your next post about what exactly you want the SSH to do, I don't think I can help you much there. I've never tried SSH for file transfers -- I've only used SSH in the past as a replacement for telnet. However, I found the following threads on Usenet that look interesting:

http://groups.google.com/groups?hl=e...Edu%26rnum%3D8

http://groups.google.com/groups?hl=e...com%26rnum%3D5

http://groups.google.com/groups?q=ss...ome.com&rnum=4

http://groups.google.com/groups?hl=e...net%26rnum%3D2

http://groups.google.com/groups?hl=e...com%26rnum%3D1

I'm not sure how much the threads will help you, but they might give you some ideas.

-Mike Gnitecki