Hi,

Any reason why LDAP never defined application level subtree specification
mechanisms?  Right now the subentry is used
with the a operational usage for the main subtreeSpecification attribute.
Also the base is AP position relative.  Why not
have an application space specification and use that for dynamic grouping?

Any thoughts?

Alex

[ Post a follow-up to this message ]

> Any reason why LDAP never defined application level subtree specification
> mechanisms?  Right now the subentry is used
> with the a operational usage for the main subtreeSpecification attribute.
> Also the base is AP position relative.  Why not
> have an application space specification and use that for dynamic grouping?
>
>

When it comes to a 'local' directory how is this different from having
application specific attributes and ACLs that control who can see
what?

Oracle Virtual Directory has similar feature, but as a virtual
directory it limits what 'adapters' can be accessed by a client
connection.  Since virtual directories often create application
specific 'views' of data this feature makes it easier to use a single
virtual directory for multiple applications.  Maybe I don't understand
your use case?

Marc

[ Post a follow-up to this message ]

On 9/20/07, Marc Boorshtein <mboorshtein-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:[vbc
ol=seagreen]
> 
> specification 
> attribute. 
> grouping?
>
> When it comes to a 'local' directory how is this different from having
> application specific attributes and ACLs that control who can see
> what?[/vbcol]


Not sure if we're talking about the same thing here.

Oracle Virtual Directory has similar feature, but as a virtual
> directory it limits what 'adapters' can be accessed by a client
> connection.  Since virtual directories often create application
> specific 'views' of data this feature makes it easier to use a single
> virtual directory for multiple applications.  Maybe I don't understand
> your use case?
>

Creating sets of entries is the topic at hand - just wondering why SS was
not
used there.  I think we're not on the same page but no worries. Thanks
anyway.

Alex

[ Post a follow-up to this message ]

On 9/21/07, Alex Karasulu <akarasulu-1oDqGaOF3Lkdnm+yROfE0A@public.gmane.org> wrote:
>
> Hi,
>
> Any reason why LDAP never defined application level subtree specification
> mechanisms?  Right now the subentry is used
> with the a operational usage for the main subtreeSpecification attribute.
> Also the base is AP position relative.  Why not
> have an application space specification and use that for dynamic grouping?


I think netscape family implements Roles (like dynamic groups) using
Subentries. As far as I know, OpenDS implements subtreeSpecifications with
RootDSE as the base relative position. But none of these are standard.

Any thoughts?
>
> Alex
>
>


--
Ersin Er
http://www.ersin-er.name

[ Post a follow-up to this message ]

Hi,

It seems I was not very clear on what I was asking.  Let me give it another
shot.

We have subentries which contain subtreeSpecifications (SS).  The SS is a
very powerful mechanism
for selecting entries in the DIT.  It's essentially a means to group entries
together and much more powerful
than what is currently used in practice for dynamic groups: dynamic groups
uses an LDAP URL to
dynamically select the users for inclusion in the group.

I was wondering if a application specific form of this could be used for
dynamic groups instead of using a
simple LDAP URL.

The problem with an SS is that it's USAGE is a directoryOperation and it's
base is relative to the position
of the Administrative Point (AP).  What if we defined a means for
applications to group together entries based
on this concept.  Say we have the following objectClass for a
subtreeSelector:


objectclass ( 1.3.6.1.4.1.18060.0.4.1.3.11 NAME 'subtreeSelector'
DESC 'application level mechanism for specifying subtrees with specified
base'
SUP top
STRUCTURAL
MAY ( selectorFilter $ minimum $ maximum $ chopBefore $ chopAfter )
MUST ( cn & selectorBase )
)

I'm sure you can figure out what the may and must attributes correspond to
along with their
characteristics: i.e. chopBefore is a distinguishedName syntax multivalued
attribute etc.

So why are we (LDAP community) not leveraging something this powerful
instead of using
a simple URL to define dynamic groups.

Alex

On 9/21/07, Ersin Er <ersin.er-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>
>
> On 9/21/07, Alex Karasulu <akarasulu-1oDqGaOF3Lkdnm+yROfE0A@public.gmane.o
rg> wrote: 
>
>
> I think netscape family implements Roles (like dynamic groups) using
> Subentries. As far as I know, OpenDS implements subtreeSpecifications with
> RootDSE as the base relative position. But none of these are standard.
>
> Any thoughts? 
>
>
> --
> Ersin Er
> http://www.ersin-er.name

[ Post a follow-up to this message ]

>It's essentially a means to group entries
> together and much more powerful
> than what is currently used in practice for dynamic groups: dynamic groups
> uses an LDAP URL to
> dynamically select the users for inclusion in the group.
>

Ok, I think I better understand what you are saying now.  I've seen a
lot of different group structures in directories.  From the simplest
static group to overly complex combinations of custom group entries
and custom az modules for applications.  The issue (or non issue
depending on how you look at it) is that a group is simply a directory
object.  What gives a group power is how the application deals with
the information.  The moral here being that if you have some alternate
idea of how to specify a group, i'm sure there are infinite ways of
specifying how to group users together but can the application
interpret that information?

For instance a dynamic group is much easier to manage in many ways
then a static group but applications don't generally know how to
compare a user against an ldap url.  Thats why many virtual
directories have dynamic group plugins to make dynamic groups work and
act like static groups.  So if you were to pair this group
specification idea with an interceptor that made it work like a static
group you might be on to something.

The answer to 'why dont we do it' is that applications need to be able
to consume it.  There is no az mechanism in a directory that can hide
the implementation.  Authorization is left to the applications that
are accessing the directory.


Marc

[ Post a follow-up to this message ]

Hi Marc,

On 9/21/07, Marc Boorshtein <mboorshtein-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:[vbc
ol=seagreen]
> 
> groups 
>
> Ok, I think I better understand what you are saying now.  I've seen a
> lot of different group structures in directories.  From the simplest
> static group to overly complex combinations of custom group entries
> and custom az modules for applications.  The issue (or non issue
> depending on how you look at it) is that a group is simply a directory
> object.  What gives a group power is how the application deals with
> the information.  The moral here being that if you have some alternate
> idea of how to specify a group, i'm sure there are infinite ways of
> specifying how to group users together but can the application
> interpret that information?[/vbcol]


Yep the app needs to either be aware or the server has to present the
dynamic group as a static group in structure.  Meaning inserting the
information into a static representation of the group on the applications
behalf.

Another avenue for it is to enable a membership extended operation to
ask simple questions like is this entry a member of a group.  As for the
following questions which return a set of entries:

i). list the groups an entry is a member of
ii). list the members of a group

I think a control on search would be best because you want to iterate
through
it instead of caching a result set and returning it in a single response.
Obviously
this is due to the potentially massive result set you could get with some
groups.
So it is best to stream it as you do with a db cursor on a result set. This
is how
search works in ApacheDS - it pulls records from Partitions as clients ask
for the
next one.  Hence the reason why a control would be best.

(case ii)
But the question remains how do you handle the semantics of such a search.
You might have a client that reads a dynamic group with the list members
control
as an object scoped search request.  The client should then expect to see
not
one result but several since it asked for all members with a specific
control (client
can't be that ignorant).  So instead of getting a single result it handles
several
results as if they executed the search for several entries.  This may be an
OK route
to take.  Don't know for sure but perhaps we should talk about it.

(case i)
This one can be really easy. First let me revert and say ok we can just
stuff all
the groups associated with the entry into a single entry result return and
not stream
since the number of groups an entry is far far less than the number of
members in
some groups.  So I'm doubling back here.

This control should just insert a groups dynamic attribute into the returned
entry
so the application that read the entry has this information when requesting
it via
the control.  This makes sense for identity management and I think some
servers
already have something like this in place.  SUN?  It makes sense to get all
the
groups of a user for example when reading the user entry in one shot instead
of
performing several searches to figure out this information which you may not
be
able to do as the user you bind as.


For instance a dynamic group is much easier to manage in many ways
> then a static group but applications don't generally know how to
> compare a user against an ldap url.


Exactly!

Thats why many virtual
> directories have dynamic group plugins to make dynamic groups work and
> act like static groups.


Yes this is one such approach but it scares me in terms of the size of
entries that
will be returned.  You're going to be caching a lot in memory.  Consider a
dynamic
group with 1M members.  You now have to stuff your entry to be returned with
1M
values for the [unique]member attribute.  If the average size for a memb
er's
internationalized
DN is 512 bytes then we're going to return an entry which is 512M+ in size.
I don't want
to keep this entry in memory as the server builds it to send it out the
door.

Ok this is a corner case but consider the count to be 10K members.  Still
this is a 5MB
entry.  I am not sure I want to cache this either.  We make requests for
membership information
quite frequently with large groups a lot of the server's memory is going to
be sucked up to
handle lookups of groups.

So if you were to pair this group
> specification idea with an interceptor that made it work like a static
> group you might be on to something.


I'd love to do that but for the reasons above I'm constrained.  The search
control based
approach is perhaps the best way to do this but then again the client has to
get a bit
smarter.  Perhaps both options can exist and when the size of the group is
beyond some
threshold the static group like lookup fails?

This get's me to another point.  LDAP needs a partial result return protocol
response
which could allow streaming large amounts of attributes back to clients to
prevent
excessive memory consumption.

The answer to 'why dont we do it' is that applications need to be able
> to consume it.


Yep this is the biggest problem but applications need to change and so does
LDAP
otherwise we're stuck doing stupid things like HTTP did with statelessness
for decades.

There is no az mechanism in a directory that can hide
> the implementation.  Authorization is left to the applications that
> are accessing the directory.
>

I'd like to approach this is as a general grouping mechanism discussion
independent of
the application of groups.  Yes you can use it for Authorization but for now
let's just
keep talking just about grouping.

Incidentally Authorization is already handled correctly by the X.500 model
which is being
specified in LDAP with the work being done by Steven Legg here:

http://tools.ietf.org/html/draft-legg-ldap-acm-bac-04

Authorization does not need this after this draft becomes an RFC which shows
a strong
liklihood.  ApacheDS uses this same model - this is why we stuck so close to
the X.500
standards in our AC implementation.

Getting back to groups.  This proposed mechanism for dynamic groups has it's
basis in X.500.
The excellent points regarding client consumption that you sited can be
ameliorated using some
existing protocol features such as extended operations and search controls
but we must explore
the impact of them on clients and on the server.  I smell a potential draft
for a new specification
here.  Interested?

Alex

[ Post a follow-up to this message ]

Ok got another idea on the matter ...

Let's think about the inverse.  You have this subtreeSelector and when you
read the entry
of that objectClass with object (base) scope without any controls you can
get the following
results:

(1) a single entry with all the member attribute values injected into it
(2) multiple entries if the member attribute values exceed a certain
threshold

If a management control is present then you see the entry without any of the
magic going
on.

Now perhaps we need to make the subtreeSelector an AUXILIARY objectClass so
it can
be added to static groups.  Once that happens the injection is handled by
the server and
clients see a static group with all it's members.

Also we will need a threshold parameter in the subtreeSelector to be used as
the trigger
to switch from mode 1 to mode 2 based on the size of the result set.

WDYT?

Alex

[ Post a follow-up to this message ]

Alex,
 
>
> Yes this is one such approach but it scares me in terms of the size of
> entries that
> will be returned.  You're going to be caching a lot in memory.  Consider a
> dynamic
> group with 1M members.  You now have to stuff your entry to be returned wi
th
> 1M
> values for the [unique]member attribute.  If the average size for a me
mber's
> internationalized
> DN is 512 bytes then we're going to return an entry which is 512M+ in size
.
> I don't want
> to keep this entry in memory as the server builds it to send it out the
> door.
>

If the application were pulling all members back then yes, this would
be a major issue.  It is not however a requirement to bring all the
members into the virtual directory and then return them to the client
to test a group membership.  Generally a group test will look like:

ldapsearch ... -b "cn=my group,ou=groups,dc=domain,dc=com" -s base
" (uniqueMember=cn=me,ou=people,dc=domain,
dc=com") 1.1

If the user is a member of the group then the group dn will be
returned as the only entry.  While there are definitely really badly
written applications that will retrieve the entire group and then do
an evaluation thats pretty rare.  Even if it does happen you would
have the same problem with a static group.  I've actualy deployed this
solution with Oracle's virtual directory where the static members were
150k-500k members (and is a 256mb entry any better then a 512mb entry?
:-D) and it worked very well.

I'm still not sure what the advantage of your solution is over a
dynamic group, but I'm probably missing something.

> I'd love to do that but for the reasons above I'm constrained.  The search
> control based
> approach is perhaps the best way to do this but then again the client has 
to
> get a bit
> smarter.  Perhaps both options can exist and when the size of the group is
> beyond some
> threshold the static group like lookup fails?
>

well, applications just aren't written well.  1/2 the commercial LDAP
applications out there have poorly written LDAP modules that were
built for a specific customer and were just 'released'.

> This get's me to another point.  LDAP needs a partial result return protoc
ol
> response
> which could allow streaming large amounts of attributes back to clients to
> prevent
> excessive memory consumption.
> 
>
> Yep this is the biggest problem but applications need to change and so doe
s
> LDAP
> otherwise we're stuck doing stupid things like HTTP did with statelessness
> for decades.
>

Good luck to you there...I don't know that there will ever be an
LDAPv4 but I suppose thats another discussion.

> Incidentally Authorization is already handled correctly by the X.500 model
> which is being
> specified in LDAP with the work being done by Steven Legg here:
>
>     http://tools.ietf.org/html/draft-legg-ldap-acm-bac-04
>
> Authorization does not need this after this draft becomes an RFC which sho
ws
> a strong
> liklihood.  ApacheDS uses this same model - this is why we stuck so close 
to
> the X.500
> standards in our AC implementation.
>

Well, I'm not a huge X.500 fan but you say tomatoe....I'm curios to
check out the spec though.  Thanks for the pointer.

> Getting back to groups.  This proposed mechanism for dynamic groups has it
's
> basis in X.500.
> The excellent points regarding client consumption that you sited can be
> ameliorated using some
> existing protocol features such as extended operations and search controls
> but we must explore
> the impact of them on clients and on the server.  I smell a potential draf
t
> for a new specification
> here.  Interested?
>

Always glad to be involved and bounce ideas around.  I think I take a
different tact but thats probably OK too.

Extended operations & controls are great and all but applications just
don't use them.  If I were to design my super group it would have 4
requirements:

1.  Members can be dynamic
2.  Members can be static
3.  Members can be other groups
4.  I can test a group membership with a single ldap search call

Now how to do this under the covers I bet you are far better at
answering then me.  Again I can do this with a virtual directory now
(MyVirtualDirectory has a DynamicGroups insert and a EmbeddedGroups
insert that does this).  Maybe ApaceDS can do it better because it has
all of its data locally (actually I'm positive it could).

I do say I'd be very interested to get the other Open Source ldap
proejects (opends, openldap, penrose & fedorads) opinions on the
matter.

Marc

[ Post a follow-up to this message ]

Hi Marc,

On 9/21/07, Marc Boorshtein <mboorshtein-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:[vbc
ol=seagreen]
>
> Alex,
> 
> a 
> with 
> member's 
> size. 
>
> If the application were pulling all members back then yes, this would
> be a major issue.  It is not however a requirement to bring all the
> members into the virtual directory and then return them to the client
> to test a group membership.  Generally a group test will look like:
>
> ldapsearch ... -b "cn=my group,ou=groups,dc=domain,dc=com" -s base
> " (uniqueMember=cn=me,ou=people,dc=domain,
dc=com") 1.1[/vbcol]


Very true but I was specifically talking about reading the entire group.
But you're
right this would work as well as a Compare request which is even cheaper to
perform
on most servers.


> If the user is a member of the group then the group dn will be
> returned as the only entry.  While there are definitely really badly
> written applications that will retrieve the entire group and then do
> an evaluation thats pretty rare.


Yeah this is what I am afraid of.  I know of many apps like this
unfortunately.  Some
are really big commercial applications yuk!

Even if it does happen you would
> have the same problem with a static group.


Yep this is the reason why static groups are useless when the set increases
in size.

I've actualy deployed this
> solution with Oracle's virtual directory where the static members were
> 150k-500k members (and is a 256mb entry any better then a 512mb entry?
> :-D) and it worked very well.


Haa!  That's crazy. Try doing that under heavy load.  I'd like to see how
the VD then handles OOM
exceptions.

I'm still not sure what the advantage of your solution is over a
> dynamic group, but I'm probably missing something.


Hmmm are you up to date on the X.500 admin model?  SubtreeSpecifications and
what not?
The whole point is to leverage the same concepts rather then the standard
dynamic group
being based on the LDAP URL.  SS is much more flexible.

> I'd love to do that but for the reasons above I'm constrained.  The search
 
> has to 
> is 
>
> well, applications just aren't written well.  1/2 the commercial LDAP
> applications out there have poorly written LDAP modules that were
> built for a specific customer and were just 'released'.


Aye!

> This get's me to another point.  LDAP needs a partial result return
> protocol 
> to 
> does 
> statelessness 
>
> Good luck to you there...I don't know that there will ever be an
> LDAPv4 but I suppose thats another discussion.


As with everything it's up to the community.  I don't know either and it's
probably a pipe dream
but oh what a good dream it is.

> Incidentally Authorization is already handled correctly by the X.500 model
 
> shows 
> close to 
>
> Well, I'm not a huge X.500 fan but you say tomatoe....I'm curios to
> check out the spec though.  Thanks for the pointer.


Well you're going to need to be.  I know reading X.500 specs make the best
ambien replacement in the world but LDAP is based on X.500 and is moving
closer to it.  IMO LDAP was too lightweight in an adverse reaction to the
OSI
weight of X.500.  So now people realize we have to embrace X.500 concepts
and in particular the admin model.  Lookie here ..

http://www.ietf.org/rfc/rfc3672.txt

There's much more coming down the pike.  I've been dreaming about the day
this would happen.  The admin model is going take the ad hoc crap out of
LDAP
implementations today.  Standards are the only way to reach true
interoperability
across these ideas here.

> Getting back to groups.  This proposed mechanism for dynamic groups has
> it's 
> controls 
> draft 
>
> Always glad to be involved and bounce ideas around.  I think I take a
> different tact but thats probably OK too.


Sure definitely.

Extended operations & controls are great and all but applications just
> don't use them.  If I were to design my super group it would have 4
> requirements:
>
> 1.  Members can be dynamic
> 2.  Members can be static
> 3.  Members can be other groups
> 4.  I can test a group membership with a single ldap search call


I like these items here especially #3 which did not occur to me until now.
Also note you can use a compare
op too but you're right most apps don't have a freakin clue about how to do
a compare either.   The name of
this game is to cater to the dumb clients already out there that won't
change.

Now how to do this under the covers I bet you are far better at
> answering then me.  Again I can do this with a virtual directory now
> (MyVirtualDirectory has a DynamicGroups insert and a EmbeddedGroups
> insert that does this).  Maybe ApaceDS can do it better because it has
> all of its data locally (actually I'm positive it could).


We do have our class of problems but we'll solve them with time.  My whole
angle with this stuff
is to do it while extending (through IETF) and complying with standards and
expected semantics.
ADS is trying to achive these feature carefully without being another ad hoc
concoction to respond
to the market.  We want to outlast the fads.

In fact I am certain virtualization can have a solid theory behind it
through the X.500 admin model.
I'm in the process of defining conceptually views for LDAP using the admin
model and stored
procedures.  This however is future work.

I do say I'd be very interested to get the other Open Source ldap
> proejects (opends, openldap, penrose & fedorads) opinions on the
> matter.
>

Hmmm as you know I've been intricately involved with the Penrose peeps
having helped them use
apacheds under the hood.  I withdrew from them for various reasons but one
was because they
don't have a clue about the protocol and X.500.  Penrose is just an ad hoc
concoction to make money
quick. Nothing wrong with that but it bastardizes the technology with
one-offs jammed in to make
it to market.  You need a balance.

The other efforts are legitimate IMO but these folks are best engaged
through an open specification
process.

Alex

[ Post a follow-up to this message ]