Hi Folks,
For several weeks I have been running FreeBSD 5.1 Release (was on 4.7 previously).
I am using the ports collection of iftop 0.14.
What am am finding that that after a day or two I have thousands of open files on the OS. If I exit iftop, that number drops back down to 200 or so, and after I start iftop again it starts climbing again at a rate of about 3 or 4 files per second.
I only noticed because after a week it is using all 7000 available file handles, and I get a kernel file table full errors, and everything stops.
Anyone else seem this behaviour?
Cheers, Stuart Whelan Technical Lead Simulation Hardware LTD
DDI: +64 3 3778866 Mobile: +64 27 2828074
--- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.535 / Virus Database: 330 - Release Date: 1/11/2003
On Tue, Nov 04, 2003 at 08:09:21PM +1300, iftop@simhardware.co.nz wrote:
Hi Folks,
For several weeks I have been running FreeBSD 5.1 Release (was on 4.7 previously).
I am using the ports collection of iftop 0.14.
What am am finding that that after a day or two I have thousands of open files on the OS. If I exit iftop, that number drops back down to 200 or so, and after I start iftop again it starts climbing again at a rate of about 3 or 4 files per second.
I only noticed because after a week it is using all 7000 available file handles, and I get a kernel file table full errors, and everything stops.
Hmm. Which resolver is it using? Does strace show anything useful (try strace -eopen,socket iftop)? Can you use lsof or similar to identify files the process has open?
Hmm. Which resolver is it using? Does strace show anything useful (try strace -eopen,socket iftop)? Can you use lsof or similar to identify files the process has open?
Hi Chris,
Thanks for the prompt reply. :)
Judging from the make output of the port it is using the forking resolver:
checking how to call gethostbyaddr_r... no idea; dropping back to the
forking resolver
I just tried to build strace onto my system, but it is having problems, I will get back to you on that one.
lsof provides interesting output:
There are the normal open files, but the interesting lines (current 174... Opps, make that 176 lines of them) are similar to:
iftop 21553 21552 root 6u KQUEUE 0xc2a08300 count=0, state=0 iftop 21553 21552 root 8u KQUEUE 0xc2987900 count=0, state=0 iftop 21554 21552 root 8u KQUEUE 0xc29e9300 count=0, state=0 iftop 21554 21552 root 9u KQUEUE 0xc2a06400 count=0, state=0 iftop 21553 21552 root 10u KQUEUE 0xc29ee600 count=0, state=0 iftop 21553 21552 root 11u KQUEUE 0xc2a11200 count=0, state=0 iftop 21554 21552 root 11u KQUEUE 0xc2b38400 count=0, state=0 iftop 21553 21552 root 12u KQUEUE 0xc29a1700 count=0, state=0 iftop 21554 21552 root 12u KQUEUE 0xc29e9d00 count=0, state=0
I tied disabling name resolving (using -n) to see if that makes a difference, and it did.
Only 18 open files, and not growing.
Do you still wish to have the strace output? I will attempt to get it running.
Cheers, Stuart Whelan Technical Lead Simulation Hardware LTD
DDI: +64 3 3778866 Mobile: +64 27 2828074
--- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.535 / Virus Database: 330 - Release Date: 1/11/2003
On Wed, Nov 05, 2003 at 09:43:43AM +1300, Stuart Whelan - Simulation Hardware LTD wrote:
Hmm. Which resolver is it using? Does strace show anything useful (try strace -eopen,socket iftop)? Can you use lsof or similar to identify files the process has open?
Hi Chris,
Thanks for the prompt reply. :)
Judging from the make output of the port it is using the forking resolver:
checking how to call gethostbyaddr_r... no idea; dropping back to the
forking resolver
OK.
Do you still wish to have the strace output? I will attempt to get it running.
-- that would be handy, but not essential. I think I know where the problem lies but not exactly what it is yet.
On Wed, Nov 05, 2003 at 09:43:43AM +1300, Stuart Whelan - Simulation Hardware LTD wrote:
Hmm. Which resolver is it using? Does strace show anything useful (try strace -eopen,socket iftop)? Can you use lsof or similar to identify files the process has open?
Hi Chris,
Thanks for the prompt reply. :)
Judging from the make output of the port it is using the forking resolver:
checking how to call gethostbyaddr_r... no idea; dropping back to the
forking resolver
I just tried to build strace onto my system, but it is having problems, I will get back to you on that one.
ok... I can't reproduce this here (on Linux), so it may be a FreeBSD issue. Could you try applying the following patch, which will make iftop print some debugging information from the resolver, and then running iftop with a command line like
iftop -i whatever ... 2> iftop.log
-- and then telling me what the contents of iftop.log look like? (Obviously you have to direct stderr into a file since otherwise it'll mess up the display....)
diff -u -r1.18 resolver.c --- resolver.c 6 Sep 2003 13:52:36 -0000 1.18 +++ resolver.c 5 Nov 2003 12:46:04 -0000 @@ -320,7 +320,7 @@ workerinfo = xmalloc(sizeof *workerinfo); pthread_setspecific(worker_key, workerinfo); workerinfo->fd = p[0]; - + switch (workerinfo->child = fork()) { case 0: close(p[0]); @@ -332,6 +332,7 @@ return NULL;
default: +fprintf(stderr, "New child is %d, we have fd %d, he has fd %d\n", workerinfo->child, p[0], p[1]); close(p[1]); } } @@ -340,11 +341,13 @@ if (write(workerinfo->fd, addr, sizeof *addr) != sizeof *addr || read(workerinfo->fd, name, NAMESIZE) != NAMESIZE) { /* Something went wrong. Just kill the child and get on with it. */ +fprintf(stderr, "Protocol error (%s) talking to child %d\n", strerror(errno), workerinfo->child); kill(workerinfo->child, SIGKILL); wait(); close(workerinfo->fd); xfree(workerinfo); pthread_setspecific(worker_key, NULL); + *name = 0; } if (!*name) return NULL;
-
Chris Lightfoot -
iftop＠simhardware.co.nz -
Stuart Whelan - Simulation Hardware LTD