[tpop3d-discuss] Memory leak?

Marc Lewis marc at blarg.net
Wed, 8 May 2002 15:04:42 -0700

On Wed, May 08, 2002 at 09:32:46PM +0100, Chris Lightfoot wrote:
> On Wed, May 08, 2002 at 01:07:34PM -0700, Marc Lewis wrote:
> > On Wed, May 08, 2002 at 06:38:07PM +0100, Chris Lightfoot wrote:
> > > On Wed, May 08, 2002 at 10:12:52AM -0700, Marc Lewis wrote:
> > > > On Wed, May 08, 2002 at 11:40:56AM +0100, Chris Lightfoot wrote:
>     [...]
> > I've been running some tests using the auth-ldap module, and it appears
> > that the leak is still there.
> > 
> > When first starting up the daemon:
> > 
> > root     10829  0.0  0.1  4788 1900 pts/1    S    12:18   0:00 /usr/src/tpop3d-1.4.1/tpop3d -f /etc/tpop3d-test.conf -d -v
> > 
> > After about a hundred or so mail checks (on a different port):
> > 
> > root     10829  0.0  0.1  4836 1948 pts/1    S    12:18   0:00 /usr/src/tpop3d-1.4.1/tpop3d -f /etc/tpop3d-test.conf -d -v
> > 
> > Looks like it has grown by about 48K.  So, it appears to be a small leak,
> > but there none the less.
> Errm. I'd want to run the test for a bit longer than that
> to be sure-- a change of 48k could just be `noise'. How
> much does it grow after a few thousand checks?

Well, its better, but its still leaking a bit:

root     10829  0.0  0.2  5196 2312 pts/1    S    12:18   0:05 /usr/src/tpop3d-1.4.1/tpop3d -f /etc/tpop3d-test.conf -d -v

I have an expect script pounding on it generating about 20-30
authentications per minute for the last 30 minutes or so.

> > I know this is hackish sounding, but what is the motivation for doing all
> > of the auth and such before forking?  When I've written simple daemons and
> When I originally wrote tpop3d, I wanted a POP server
> which would interact with a relational database in a
> sensible way; this means not setting up a database
> connection for every time a user connects to the service.
> Hence this choice of design. I agree with your comments
> about forking servers in general, but you do take a
> significant performance hit from this.

That makes perfect sense to me, but since the main server doesn't seem to
maintain a persistant connection to the authentication source (at least in
LDAP), the overhead of binding to the server per authentication still seems
to be there.  Not criticizing, just trying to understand.

> > I will probably end up hacking in a new option to tpop3d to make it not
> > detach from the controlling TTY and not send out debugging information to
> > stderr.  This is how we ran our old pop3 server (cucipop) before switching
> > to Maildir format.  Dropping it into inittab made sure that even if the
> > parent died, it would be restarted.
> tpop3d -d 2> /dev/null ?

Excuse me while I kick myself.  The simple solutions are often so elusive.

> > We hadn't seen any timeouts like this when we were using Courier POP, but
> > it often corrupts attachments over 500K (the first reason we switched to
> > tpop3d).  We also have a lot of IMAP usage for our Webmail services and
> > haven't seen any timeouts at all (big LDAP in-memory caches are setup).
> I take it that the Courier IMAP server doesn't suffer from
> a memory leak of this type? Its PAM implementation looks
> much like that in tpop3d, but it's possible that it forks
> after accept, in which case this wouldn't be apparent.

Correct, it does fork right after accepting the connection.  So far, tpop3d
is the only application we've tested with that suffers from this type of
behaviour.  It may not even be in tpop3d, but possibly in one of the
supporting libraries, like one of the LDAP libraries.  

In thinking about this, it is possible that it is a problem in the OpenLDAP
libraries.  In my first build of tpop3d, I only configured in Maildir and
PAM support.  Since we are using PAM to access LDAP, its possible the leak
was caused by PAM's accesses to the LDAP server, since the program never
exited, things kept growing, but because of access to libpam.  Building
against LDAP directly lessons the leak, but it does definatly appear to
still be there.  Others using different non-LDAP authentication methods do
not appear to have these leaks.  Hmmm.

> Yep. I did some work on a Postgres-based solution for this
> (to me, LDAP looks more like a buzzword than a solution,
> but I imagine this puts me in a minority...).

The one advantage that I have found in using LDAP over anything else is not
an elegant design, or good documentation, or anything else nice like that.
The advantage is that it is central and secure and everything supports it.
The problem with SQL based solutions is that everyone has their own
implementation and database schemas, with LDAP things are pretty strictly
laid out.

Personally, I would much rather use Postgres or MySQL, but there are no
standard Apache mods, PAM mods, RADIUS mods, etc, etc, etc.  If we were
ever to change software, we would have to spend dozens of man-hours
redoing whatever we had previously had setup.  "At least its consistent
across applications and services" is the prime motivation for using LDAP.


 - Marc

Marc Lewis
Network Administrator
Blarg! Online Services, Inc.