Output limit for command initiated by CGI script

[Wassercrats]

My Perl CGI script does a command line diff (using backticks), but for large files, some of the output is missing. When I do the same diff directly from the command line, I see all of the output. Through CGI, it looks like the output gets cut off at about 200K. It's not a time-out of the script, because after the diff, other parts are executed and more data is added and saved. It looks like there's a time or output limit that's lower for commands issued through CGI scripts than for CGI-only or command-line-only usage.

Should I use Perl's system or exec function and have a command save the output for the script to read rather than have the data passed to the script, or would it be truncated then too? Or maybe have the command execute another script to read the data to keep the script's run-time down?

[Terra]

Using backticks causes the entire output to be slurped in and stored into a variable, which is bad for large output... Also, using backticks is bad for security if you are passing any variables set from the outside world into the backtick command...

The general idiom for a Safe Pipe Open is:

open(P, '-|') or exec("script);

Then instead of slurping it in all at once, walk the output line by line...

while (my $line = <P>) {
chomp($line);
do_something_interesting($line);
}
close(P);


--
Terra
sysAdmin
FutureQuest

[Wassercrats]

I tried that idiom (there's an explanation of it here (http://www.cgisecurity.com/lib/sips.html) that I might bookmark), and I thought it didn't work, but then I realized that I limited the file size of each file to 100,000 K when they're fetched with LWP, which caused the truncation. I saw that someone diffed files larger than 100,000 K, but they must have been uploaded, and I set the $CGI::POST_MAX to 1,000,000. The uploaded files caused a timeout though. I'll tweak my limits.