View Full Version : PERL: flock not working...
Aaron
06-25-1999, 03:09 AM
Hi,
[nbsp][nbsp][nbsp][nbsp]I have somewhat of a counter cgi script that utilized the flock function of Perl...but on heavy days, the count gets reset back to zero.[nbsp][nbsp]Below is a snip of the script.
Any ideas?[nbsp][nbsp]Does flock work to a certain extent?[nbsp][nbsp]Is there a limitation on how "bombarded" a file can be and still be flocked properly?[nbsp][nbsp]Is there a solution?
------------------
open(GETTOTAL, "/path/to/file/filename.txt");
flock(GETTOTAL,2);
$total = <GETTOTAL>;
flock(GETTOTAL,8);
close(GETTOTAL);
$total++;
open(TOTAL, ">/path/to/file/filename.txt");
flock(TOTAL,2);
print TOTAL "$total";
flock(TOTAL,8);
close(TOTAL);
------------------
Thanks for any help!
Aaron
Terra
06-25-1999, 03:31 AM
This is an excellent reference for File Locking issues:
http://www.singlesheaven.com/stas/TULARC/webmaster/myfaq.html#1
Also, I would drop the explicit 'flock(TOTAL,8);' and let close(TOTAL); handle it implicitly...
It seems there may be a very small race condition between unlocking, and closing the filehandle... (close will flush the buffer as well)
--
Terra
--No Trespassing!--
FutureQuest
Aaron
06-25-1999, 03:39 AM
Thanks Terra,
[nbsp][nbsp] I'll try that and see if that solves the problem.
-Aaron
I had that problem as well, using the same code you use, so I added a simple lockfile, instead of flocking the counter-file.
I had no resets after this:
#!/usr/bin/perl
$basedir="/home/user/"; # set it to your root
$flock=1; # Depends on your system
$lockfile = "$basedir/Data/lockfile";
if ($flock) {
[nbsp]while (-e $lockfile) {sleep 1};
[nbsp]open (LOCK, ">$lockfile");
[nbsp]flock(LOCK, 2);
[nbsp]print LOCK $$;
[nbsp]flock(LOCK, 8);
[nbsp]close(LOCK);
};
open (COUNTER, "$basedir/Data/counter.dat");
@count=<counter>;
close(COUNTER);
print "Content-type: text/html\n\n";
for (@count) {
[nbsp]chop($_);
[nbsp]($key, $value)=split(",",$_,2);
[nbsp]$counthash{$key}=$value;
};
# Determine which counter to update
$ENV{'DOCUMENT_URI'} =~ s/[^\w]/_/g;
printf ("%06d", ++$counthash{$ENV{'DOCUMENT_URI'}});
open (COUNTER, ">$basedir/Data/counter");
foreach $key (keys(%counthash)) {
[nbsp]print COUNTER $key.",".$counthash{$key}."\n"
};
close(COUNTER);
# remove lockfile
if ($flock) {
[nbsp]$cmd="/bin/rm $lockfile";
[nbsp]system($cmd);
};
--
Pier
Justin
06-25-1999, 10:04 AM
I do mine similar, but there's always the slight possibility that someone could cancel the operation prematurely, causing a lock file to remain...
What I do is print the time in the lock file, and if the script comes accross a dormant lock file it deletes it if the time is more than 30 seconds old. Otherwise it knows another process is using it and will wait for it to disappear.
I also pass the name of the file the script needs to access - no sense in blocking another instance of the script if it's writing to a different file all together (the UBB is quite guilty of this, considering it uses so many files):
##########
sub lock {
[nbsp][nbsp] my $file = shift;
[nbsp][nbsp] my $endtime = time + 15;
[nbsp][nbsp] if (-e "$file.lock") {
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]open (LOCKFILE,"$file.lock");
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]my $temp = <LOCKFILE>;
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]close (LOCKFILE);
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]chomp ($temp);
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]if ($temp < (time - 30)) {
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp] unlink ("$file.lock");
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]}
[nbsp][nbsp] }
[nbsp][nbsp] while (-e "$file.lock" && time < $endtime) {
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]sleep(1);
[nbsp][nbsp] }
[nbsp][nbsp] if (-e "$file.lock") {
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]print "Server too busy - can't lock $file for reading/writing.";
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]exit;
[nbsp][nbsp] } else {
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]open (LOCKFILE, ">$file.lock");
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]print LOCKFILE time;
[nbsp][nbsp] }
}
############
sub unlock {
[nbsp][nbsp] my $file = shift;
[nbsp][nbsp] close (LOCKFILE);
[nbsp][nbsp] unlink ("$file.lock");
}
So if you need to access counter.dat, for example, you just say:
&lock ("counter.dat");
And it will create counter.dat.lock, effectively only locking that file, allowing other processes to access other files - although a counter usually only uses one file... This is pulled right from my banner rotation script.
------------------
Justin Nelson
FutureQuest Support<!-- NO_AUTO_LINK -->
[This message has been edited by Justin (edited 06-25-99)]
Yups, I know about the hanging problem.
That's why normally there's a counter running with it
(what is effectivly the same :) )...
Pier
[ Sometimes I even dream in perl... ]
Aaron
06-25-1999, 02:36 PM
So Justin, would it look something like this (with the subs in my code as well)?
open(GETTOTAL, "/path/to/file/filename.txt");
&lock("/path/to/file/filename.txt");
$total = <GETTOTAL>;
&unlock("/path/to/file/filename.txt");
close(GETTOTAL);
$total++;
open(TOTAL, ">/path/to/file/filename.txt");
&lock("/path/to/file/filename.txt");
print TOTAL "$total";
&unlock("/path/to/file/filename.txt");
close(TOTAL);
Or would I not use the &unlock sub?[nbsp][nbsp]And do I have to lock the file when it reads filename.txt? or just when it is writing to it?
Thanks for your help!
Aaron
OK, i'm not Justin, but I also might give a try :)
[nbsp]- You don't HAVE to use &unlock, but I sure would. &unlock removes the lock (doh), if you don't use it, &lock will unlink (eq remove) the lockfile 'after' 30 seconds, but with a busy site, you'll have lotsa server-busy errors.
[nbsp]- Also, you don't have to lock it while reading, but it can be possible for two processes to read the same value, and so you miss a thing.
Your code seems alright, though.
--
Pier
[ Who just saw The Phantom Menace. No I really want to see it when it arrives here in Holland (in september) ]
Justin
06-25-1999, 04:45 PM
Almost - you will want to lock the file before opening it - and unlock it after you close the file. Also, I always lock while reading to, just in case another process happens to be writing to it at that exact moment - like Pier said, it can miss something...
Another reason: Let's take a counter script. In it's life cycle, the script opens the file, reads all of the info into an array, changes one of the values, and rewrites the info to the file. Note that in doing this, it is not appending, but clearing the file and writing the updated info.
So for a split second, that file is empty (the moment it opens with the ">" flag).
So script A opens the file, reads the data, changes it (in memory), and is getting ready to write.
- tick -
Script B opens the file, and reads the data. There is no data. Script A just wiped it but the clock ticked before it could write the data.
- tick -
Script A writes it's data back to the file and closes it.
- tick -
Script B now opens the file for writing and writes it's 0 bytes... effectively wiping out the data... oops!
So yes, lock before reading also, as this ensures that only one instance of the script is using the file at a time.
BTW - This locking scheme is far superior to the scheme used in the UBB. I didn't like it, so I put it to the test. I wrote a script who's job was to simulate 1000 simultaneous posts to the same forum.
The stock UBB borked more than one file, resulting in some really odd looking things - and only 47 of the posts made it (although they were posted by 04-19-99 and stuff like that).
My rewritten locking scheme prevailed with no corrupted files at all, and 376 of the posts made it (my PC was completely busy, hard drive going nuts, for like 5 minutes). Now I know that's not all 1000 posts but either a) the simulation script timed out, or b) the UBB (thus my locking scheme) returned about 624 "Server Busy" errors :)
BTW - my banner script, before I came up with that locking scheme above, kept losing all of it's banners... I was a little ticked, but since it logs impressions and clicks, and each page (at that time) had 2 banners, therefore always calling the script twice in rapid succession.
Since I came up with that scheme (in March), I haven't had a problem, after over 100,000 impressions (and thus lock/open/do stuff/close/unlock's :))
------------------
Justin Nelson
FutureQuest Support
Aaron
06-25-1999, 06:10 PM
Thanks Justin and Pier.[nbsp][nbsp]Your help is really appreciated! :)
So the code should look like this?
&lock("/path/to/file/filename.txt");
open(GETTOTAL, "/path/to/file/filename.txt");
$total = <GETTOTAL>;
close(GETTOTAL);
&unlock("/path/to/file/filename.txt");
$total++;
&lock("/path/to/file/filename.txt");
open(TOTAL, ">/path/to/file/filename.txt");
print TOTAL "$total";
close(TOTAL);
&unlock("/path/to/file/filename.txt");
Thanks!
Aaron
Aaron
06-27-1999, 06:13 PM
Ok Justin,
[nbsp][nbsp]I installed the code yesterday, and it was working to some extent.[nbsp][nbsp]Before I tried your code, it would keep going back to zero after about 7000 hits.[nbsp][nbsp]Now with your code, it got all the way up to around 40,000 hits (you can see the demand the script must handle with over 40,000 hits a day...thus I need something accurate and able to handle it.), and then all of the sudden at about 1:00 PM today, it went back to zero. :([nbsp][nbsp]I was getting high hopes for this code, but when it reset back to zero... :(
[nbsp][nbsp]Could there be any other reason why the log would reset back to zero?[nbsp][nbsp]Is there a limitation on how many lockings it can handle at one time?[nbsp][nbsp]I know there has to be some way of doing this...[nbsp][nbsp]Huge sites can handle thousands of hits at the same time, and still keep an accurate count.
[nbsp][nbsp]Any help would be appreciated...
Thanks!
Aaron
Justin
06-27-1999, 08:13 PM
I don't see the need to unlock just to lock it right back up - I got bored, so try this:
#!/usr/bin/perl
# Set this to your data file
my $file = "/file.txt";
# Content header - required I think if you use SSI...?
print "Content-type: text/html\n\n";
# Lock the file
&lock ($file);
# Open the file or print ??? instead of continuing
unless (open FILE, $file) {
[nbsp][nbsp][nbsp][nbsp]print "???";
[nbsp][nbsp][nbsp][nbsp]exit;
}
my $num = <FILE> + 1;
close FILE;
open FILE, ">$file";
print $num;
print FILE $num;
close FILE;
&unlock ($file);
exit;
##########
sub lock {
[nbsp][nbsp] my $file = shift;
[nbsp][nbsp] my $endtime = time + 5;
[nbsp][nbsp] if (-e "$file.lock") {
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]open (LOCKFILE,"$file.lock");
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]my $temp = <LOCKFILE>;
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]close (LOCKFILE);
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]chomp ($temp);
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]if ($temp < (time - 10)) {
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp][nbsp] unlink ("$file.lock");
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]}
[nbsp][nbsp] }
[nbsp][nbsp] while (-e "$file.lock" && time < $endtime) {
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]sleep(1);
[nbsp][nbsp] }
[nbsp][nbsp] if (-e "$file.lock") {
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]print "???";
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]exit;
[nbsp][nbsp] } else {
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]open (LOCKFILE, ">$file.lock");
[nbsp][nbsp][nbsp][nbsp][nbsp][nbsp]print LOCKFILE time;
[nbsp][nbsp] }
}
############
sub unlock {
[nbsp][nbsp] my $file = shift;
[nbsp][nbsp] close (LOCKFILE);
[nbsp][nbsp] unlink ("$file.lock");
}
I think you're problem is that you have no error control if the open fails - meaning the number would be zero, and would effectively reset the counter.
Try the above for a few hundred thousand iterations - note that if the open or lock fails, the counter will read ??? instead of resetting to zero.
Lemme know if/how it works (the above code has been tested).
<edit>
Changed the timeout to only 5 seconds, so as not to delay loading of the page the counter is on if there are counter problems.
It also now deletes the lock file after 10 seconds (to be extremely safe) instead of 30.
</edit>
------------------
Justin Nelson
FutureQuest Support<!-- NO_AUTO_LINK -->
[This message has been edited by Justin (edited 06-27-99)]
JRepici
03-30-2002, 10:32 AM
HI,
Mind if I throw a couple of cents in here?
The problem I see with the above code snippet is that it isn't atomic. That is, there are many opportunities for another thread to preempt, do the test, and create the file between the time the test is done and the time the file is created.
let me see if I can represent what I'm talking about in code:
if (-e "$file.lock") {
# thread A does test: finds no file
print "???";
exit;
} else {
#
# Thread A gets preempted here and waits
# while thread B runs and does -e test. Thread B finds
# no file as well, and continues on to OPEN
# THE LOCK FILE...
#
# THEN, thread A returns. Since it's already
# done the test and restarts from HERE, it
# ALSO opens the lock file... OOOPS!
#
open (LOCKFILE, ">$file.lock")
print LOCKFILE time;
}
We need to find some operation that we are relatively confident the OS handles atomically, and then exploit that atomicity. Here I am pinning my hopes on a call to mkdir() which (i'm guessing) is so central to the OS's core function that it will surely surround it with critical section ("don't preempt this") calls.
sub LockSem
{
my($sem) = @_;
my($n) = 0;
while(mkdir("$sstLockDirectory/$sem",0777) == 0 ) {
$! == 17 ||
&AbortScript(
"can't make $sstLockDirectory/$sem: $!\n"
);
# Error 17 is 'directory already exists'
if($n++ >= $LockTOSecs) {
# LockTOSecs: really num of tries
if($LockRelOnTO) {
&ReleaseSem($sem); # remove dir
}
&AbortScript(
"timed out with [$!] waiting for $sstLockDirectory/$sem\n"
);
}
sleep(1); # that's seconds
}
}
If you're getting bunches of hits you'll want to adjust the sleep to a smaller number (say, .5 or .25) as well as perhaps the number of retries. If you set $LockRelOnTO to something other than 0 this will automatically purge semaphores that persist longer than tries*sleep-seconds (almost a requirement on web pages) and return NO SEMAPHORE to the requesting process (i.e. it will abort it).
-John
P.S. Can somebody please tell me how to get code to format like the above? ucode tags leave everything flush left.
deja vu :)
I also stated in another thread that I believe John's directory create/delete method to be a reliable method of simulating the flock function.
I should also point out though that if you don't use open(FH,">$file") with flock (or any other technique that clobbers the file before you lock it) then you won't have to worry about a <*cough*> "broken" flock.
vBulletin® v3.6.8, Copyright ©2000-2013, Jelsoft Enterprises Ltd.