My access map is configured for hash and it is basically at 300 K
entries.  It takes about 15 minutes to build as I suspect that the
number of elements (fixed at 4096) is too small.  When there were only
150 K entries it built in about 5 seconds.  Since the number of elements
is not configurable, would I be better off switching to the btree map
type or changing the number of elements in makemap are re-building it?

[ Post a follow-up to this message ]

Doug Hardie <bc979@lafn.org> writes:

> My access map is configured for hash and it is basically at 300 K
> entries.  It takes about 15 minutes to build as I suspect that the
> number of elements (fixed at 4096) is too small.  When there were only
> 150 K entries it built in about 5 seconds.  Since the number of elements
> is not configurable, would I be better off switching to the btree map
> type or changing the number of elements in makemap are re-building it?

1) Use "-c size" option to make makemap use bigger cache.
The default cache size is 1048576 (1 Mega Byte).
Try "-c 4194304" (4MB) and find out if it makes a difference.

Remember: it will help to decrease creation times.
Lookup times may be "longer than usually".

2) If "1" does not help then consider creating btree map from
*presorted* input file.
[ from ancient discussion about virtusertable with 25E6 entries :-) ]

PS
I personally suggest "2" with some (perl) script doing presorting.

--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Open-Sendmail: http://open-sendmail.sourceforge.net/
We are not loved by our friends for what we are; rather, we are loved in
spite of what we are.
-- Victor Hugo
----
http://groups.google.com/groups?sel....hobby-site.com

[ Post a follow-up to this message ]

In article <87fxxotfw5@francisco.fsf.hobby-site.com>,
Andrzej Adam Filip <anfi@onet.eu> wrote:

> Doug Hardie <bc979@lafn.org> writes:
> 
>
> 1) Use "-c size" option to make makemap use bigger cache.
>    The default cache size is 1048576 (1 Mega Byte).
>    Try "-c 4194304" (4MB) and find out if it makes a difference.
>
>    Remember: it will help to decrease creation times.
>              Lookup times may be "longer than usually".
>
> 2) If "1" does not help then consider creating btree map from
>    *presorted* input file.
>    [ from ancient discussion about virtusertable with 25E6 entries :-)
 ]
>
> PS
> I personally suggest "2" with some (perl) script doing presorting.

Wow - what a difference.  I ran the following tests on an otherwise idle
machine.

The reference test - the production approach.

time /usr/sbin/makemap hash /etc/mail/accessnew < /etc/mail/access.tmp
8.490u 43.003s 5:28.96 15.6%  76+1558k 1497+336io 0pf+0w

Changing the cache size.

time /usr/sbin/makemap -c 4194304 hash /etc/mail/accessnew <
/etc/mail/access.tmp
7.916u 31.640s 5:10.50 12.7%  76+5441k 1321+356io 0pf+0w

Using btree with unsorted input.

time /usr/sbin/makemap btree /etc/mail/accessnew < /etc/mail/access.tmp
6.161u 10.630s 2:10.71 12.8%  76+2810k 0+676io 0pf+0w

Using btree with sorted input.

time /usr/sbin/makemap btree /etc/mail/accessnew < /etc/mail/access.sort
2.317u 1.217s 0:04.19 84.0%   77+2789k 0+368io 0pf+0w

Btree with sorted input is incredibly fast.  Since I am generating most
of the access file from a database it becomes real easy to have it
sorted.  Even without the sort its faster though.  I believe all I need
to do to switch to btree format is recreate the access file and then
change sendmail.cf from

Kaccess hash /etc/mail/access

to

Kaccess btree /etc/mail/access

[ Post a follow-up to this message ]

Doug Hardie <bc979@lafn.org> writes:

> In article <87fxxotfw5@francisco.fsf.hobby-site.com>,
>  Andrzej Adam Filip <anfi@onet.eu> wrote:
> 
>
> Wow - what a difference.  I ran the following tests on an otherwise idle
> machine.
>
> The reference test - the production approach.
>
> time /usr/sbin/makemap hash /etc/mail/accessnew < /etc/mail/access.tmp
> 8.490u 43.003s 5:28.96 15.6%  76+1558k 1497+336io 0pf+0w
>
> Changing the cache size.
>
> time /usr/sbin/makemap -c 4194304 hash /etc/mail/accessnew <
> /etc/mail/access.tmp
> 7.916u 31.640s 5:10.50 12.7%  76+5441k 1321+356io 0pf+0w
>
> Using btree with unsorted input.
>
> time /usr/sbin/makemap btree /etc/mail/accessnew < /etc/mail/access.tmp
> 6.161u 10.630s 2:10.71 12.8%  76+2810k 0+676io 0pf+0w
>
> Using btree with sorted input.
>
> time /usr/sbin/makemap btree /etc/mail/accessnew < /etc/mail/access.sort
> 2.317u 1.217s 0:04.19 84.0%   77+2789k 0+368io 0pf+0w
>
> Btree with sorted input is incredibly fast.  Since I am generating most
> of the access file from a database it becomes real easy to have it
> sorted.  Even without the sort its faster though.  I believe all I need
> to do to switch to btree format is recreate the access file and then
> change sendmail.cf from
>
> Kaccess hash /etc/mail/access
>
> to
>
> Kaccess btree /etc/mail/access

FYI: I have suggested (once) to allow (as an option) to make access use
keys format optimized for btree performance (increasing chances of
hitting cached page):

connect:xxx.example.com -> com.example.xxx:connect
xxx.example.com ->  com.example.xxx:
connect:1.2.3.4 -> 1.2.3.4:connect
connect:5.6.7 -> 5.6.7:connect
8.9 -> 8.9:

Sendmail can do 20-30 access lookups when processing single incoming
message so reduction of number of I/O operations for *BIG* maps make
difference.

--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Open-Sendmail: http://open-sendmail.sourceforge.net/
"We dedicated ourselves to a powerful idea -- organic law rather than naked
power.  There seems to be universal acceptance of that idea in the nation."
-- Supreme Court Justice Potter Steart
----
http://groups.google.com/groups?sel....hobby-site.com

[ Post a follow-up to this message ]