Apache Directory Project - [LDAP DAS] Efficient Updating of Persisted Objects

Author

[LDAP DAS] Efficient Updating of Persisted Objects

Ole Ersoy

2007-03-21, 1:11 am

Hey Guys,

Just wanted to see if anyone had any thoughts on handling updates
to Java beans (Service Data Objects - but basically the same thing)
persisted with ADS.

With Service Data Objects we create a datagraph that is then disconnected
from it's persistence source and we can mutate it. Then later we want to
persist the graph. Each object in the graph has a change summary, that
stores the fields that were updated.

This makes it possible to only update objects that have been changed, and
we only need to update the fields that were changed.

However, I think the DirContext will overwrite the entire
object during the bind operation, rather than updating specific fields
on the object.

Initially I was thinking that the object's attributes (primitive
properties - not references to other objects)
would be serialized and made into directory attributes. But I think a
LDAP ObjectClass schema that corresponds to the
object's class (The class of the object we are persisting) would have to
be generated and stored along with the instance.

This might lead to performance improvments, if doable...?

Thoughts?

Thanks,
- Ole

Emmanuel Lecharny

2007-03-21, 7:11 am

Interesting point. the problem is that when you modify an entry, you do it
by grabbing the entire entry from the disk (or from the cache, but this is
the same problem if the entry is not cached).

Changing an object is done through a ModRequest operation, in which you give
the list of modified attributes. If the entry is not too big (I mean, no
JpegPhoto in it), then it's likely to be less than 1kbytes to read from the
disk. Reading 500 bytes compared to read 1kbytes makes no difference, as the
underlying system will read some 4Kbytes block (let's assume it's 4Kbytes,
even if it's not the real size in your system). At this point, oprtimizing
the server to read only the modified attributes will not bring enormous
gain, if any, but for sure will add some complexity.

However, if we switch to another backend (ie RDBMS), this might be something
to consider, as we may have to read attributes spreaded all over a big
table. I don't know exactly what would be the pros and cons right now, but
as teh RDBMS backend is not written (and we are still considering
implementing it), this is perfect timing to push new ideas.

Emmanuel

On 3/21/07, Ole Ersoy <ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> Hey Guys,
>
> Just wanted to see if anyone had any thoughts on handling updates
> to Java beans (Service Data Objects - but basically the same thing)
> persisted with ADS.
>
> With Service Data Objects we create a datagraph that is then disconnected
> from it's persistence source and we can mutate it. Then later we want to
> persist the graph. Each object in the graph has a change summary, that
> stores the fields that were updated.
>
> This makes it possible to only update objects that have been changed, and
> we only need to update the fields that were changed.
>
> However, I think the DirContext will overwrite the entire
> object during the bind operation, rather than updating specific fields
> on the object.
>
> Initially I was thinking that the object's attributes (primitive
> properties - not references to other objects)
> would be serialized and made into directory attributes. But I think a
> LDAP ObjectClass schema that corresponds to the
> object's class (The class of the object we are persisting) would have to
> be generated and stored along with the instance.
>
> This might lead to performance improvments, if doable...?
>
> Thoughts?
>
> Thanks,
> - Ole
>
>
>

--
Cordialement,
Emmanuel Lécharny
www.iktek.com

David Jencks

2007-03-21, 1:11 pm

On Mar 21, 2007, at 1:06 AM, Ole Ersoy wrote:

> Hey Guys,
>
> Just wanted to see if anyone had any thoughts on handling updates
> to Java beans (Service Data Objects - but basically the same thing)
> persisted with ADS.

Maybe you documented this somewhere I missed, but could we back up a
little bit and see if we agree or understand what the goals are?

For this to be useful to me in triplesec it needs to be able to do
what the framework I wrote does:

- map single valued attributes in an entry to an object field
- map single and multiple valued attributes to an object reference or
collection/map of object references respectively
- map multiple valued attributes to collection valued fields of
simple types
- map the natural ldap tree structure to object tree structure
- leave out layers of the ldap tree that convey type information
rather than instance data (such as "ou=Application")

those are the important ones I can remember... I hope I didn't leave
anything too critical out.

Also I don't know much of anything about SDO.... do you write your
own POJOs and then map/enhance/code generate/?? to relate them to the
backend data store or are classes generated from a description of the
data store or are there generic data structures like a map of
attribute name to attribute value?

>
> With Service Data Objects we create a datagraph that is then
> disconnected
> from it's persistence source and we can mutate it. Then later we
> want to
> persist the graph. Each object in the graph has a change summary,
> that
> stores the fields that were updated.

Tracking change information is usually desirable whether or not you
support disconnection.

>
> This makes it possible to only update objects that have been
> changed, and
> we only need to update the fields that were changed.
>
> However, I think the DirContext will overwrite the entire
> object during the bind operation, rather than updating specific
> fields on the object.
>
> Initially I was thinking that the object's attributes (primitive
> properties - not references to other objects)
> would be serialized and made into directory attributes. But I
> think a LDAP ObjectClass schema that corresponds to the
> object's class (The class of the object we are persisting) would
> have to be generated and stored along with the instance.
>

I don't know what you mean by this. Could you provide a simple
concrete example? Probably my lack of knowledge about ldap :-)

thanks
david jencks

> This might lead to performance improvments, if doable...?
>
> Thoughts?
>
> Thanks,
> - Ole
>
>

Emmanuel Lecharny

2007-03-21, 1:11 pm

On 3/21/07, David Jencks <david_jencks-/E1597aS9LQAvxtiuMwx3w@public.gmane.org> wrote:
>
>
>
>
> Tracking change information is usually desirable whether or not you
> support disconnection.

In a ldap server, changed informations are done through ModificationItem
elements. A ModificationItem (
http://java.sun.com/products/jndi/1...cationItem.html)
element contains the operation (add, remove, change) and the attribute to be
modifed. If it's an add or remove, this is obvious ( with some special case
like non existing attribute for remove op, or existing attribute for add
op). If it's a modify, then it's a little bit complex because we should take
care of many cases, like non-existing attributes or values, doing an union
of values, etc... Internal affairs ...

This is handled more or less correctly in the server, but what we do is to
upload the entire entry being modified, not each attribute of this entry one
by one, because the entry is serialized as a single object.

The reason is that an entry could have dozens of attributes, and grabbing
all of them could cost much more than reading a single big object. (of
course, as I stated in my previous mail, it depends on the object size : why
would we read a 1Mb JpegPhoto if it has not been modified ?)

>

Bind operation just do one thing : authenticate a user, create a session for
him, and store this session. Authentication is just a lookup in the base to
check that the user credentials are correct. No object are written to the
backend during this operation. And no updating occurs, too.
[vbcol=seagreen]
>

There are two cases :
1) the attribute is indexed : we stroe a normalized form of the value(s)
into a specific file.
2) general case (including indexed attributes) : we store *all* the entry
into a big file, called Mastertable.

Nothing else right now.

But I[vbcol=seagreen]
>
> I don't know what you mean by this. Could you provide a simple
> concrete example? Probably my lack of knowledge about ldap :-)

The schema is like metaData in a RDBMS : it's always present, and we don't
have to link an attribute to it : the AttributeType is a key to get all the
needed informations from the schema, which is always loaded in memory when
the server is started.

For instance, "uid = jsmith" is an Attribute with a value, which is case
sensitive. The "uid" attributeType is known by the schema, and it has an OID
and also an alias : "uid" <=> "userid" <=> 0.9.2342.19200300.100.1.1. It's
enough to have this key to get all what we need to know from the schema, we
don't have to store any other piece of information within the instance.

Hope it helps.

--
Cordialement,
Emmanuel Lécharny
www.iktek.com

Ole Ersoy

2007-03-21, 1:11 pm

Emmanuel,

I in-more comments, but am adding a little more context first (It's a
bit long though :-) ):

The DAS will do CRUD operations for us.

CREATE
READ
UPDATE
DELETE

So if we were to look at this from an Application point of view
we can imagine the following scenario:

SCENARIO-CREATE:
There is nothing in ADS.

Our application creates an empty SDO model instance, an address for example.

Then it stores the address under
DN: |uid=address,ou=users,ou=system|

So now we can fetch that address instance from there.

SCENARIO-READ
We want the address object so we grab it.

SCENARIO-UPDATE
This is the interesting part.

We we are storing the Address object
as a whole, we can update say only
its street attribute, and then overwrite
the entire persisted object again.

But in reality we really only wish to overwrite
the street attribute of the Java object.

Suppose there was an ObjectClass
defined in ADS called
Address.

Then when we did the read operation instead of
deserializing/unmarshaling the object,
we would grab the list of attributes on the entry, pass them to the DAS,
and the DAS would instantiate
the corresponding object for us.

(The DAS would then use the setters on the object
to set those attributes per a predefined map, using
annotations on the object or an xml configuration file.)

So now when we update street, we would only send that updated
attribute back to ADS.

This leads me to the next scenario which is outside of CRUD:

SCENARIO-CREATE-SCHEMA:
This is similar to the creation of a RDB Schema or XML Schema,
but instead of creating these, it's a LDAP Schema defining
new ObjectClasses that correspond to the objects we are persisting.

These would be written to ADS so that attributes of the objects
could be stored the same way we store instances of ObjectClasses
at some DN location.

Does that make sense?

We are now using ADS as an RDB essentially.

More like an Object Oriented Database actually.

Emmanuel Lecharny wrote:
> Interesting point. the problem is that when you modify an entry, you
> do it by grabbing the entire entry from the disk (or from the cache,
> but this is the same problem if the entry is not cached).
>
Yes - So if objects were broken down into individual attributes we could
let the DAS reconstitute the object from the attributes.

Then when the update is written, the DAS would only need to send the
updated attributes.

> Changing an object is done through a ModRequest operation, in which
> you give the list of modified attributes. If the entry is not too big
> (I mean, no JpegPhoto in it), then it's likely to be less than 1kbytes
> to read from the disk. Reading 500 bytes compared to read 1kbytes
> makes no difference, as the underlying system will read some 4Kbytes
> block (let's assume it's 4Kbytes, even if it's not the real size in
> your system). At this point, oprtimizing the server to read only the
> modified attributes will not bring enormous gain, if any, but for sure
> will add some complexity.
>
> However, if we switch to another backend (ie RDBMS), this might be
> something to consider, as we may have to read attributes spreaded all
> over a big table. I don't know exactly what would be the pros and cons
> right now, but as teh RDBMS backend is not written (and we are still
> considering implementing it), this is perfect timing to push new ideas.
>
> Emmanuel

So this is silly - but just for example:
Imagine imagine a Terabyte of addresses stored in ADS.

We wanted to process all the zips to add the new 4 digit extension at
the end,
so mine would go from
60004

to

60004-4232

So I only really wish to update the zip attribute of Address SDO object,
and the SDO change summary has the capability to enable this, but
if we store the entire object on ADS, then we have to write the entire
thing back,
instead of just the changed zip.

So if we were writing to a database, we would just update the zip field
of the database table.
This is also possible with ADS, but we need to generate and store the
ObjectClass for the
address (or could be FooBoo Class, just some random class that we wish
to store).

Thoughts?

Thanks,
- Ole

>
> On 3/21/07, *Ole Ersoy* <ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> <mailto:ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> wrote:
>
> Hey Guys,
>
> Just wanted to see if anyone had any thoughts on handling updates
> to Java beans (Service Data Objects - but basically the same thing)
> persisted with ADS.
>
> With Service Data Objects we create a datagraph that is then
> disconnected
> from it's persistence source and we can mutate it. Then later we
> want to
> persist the graph. Each object in the graph has a change summary,
> that
> stores the fields that were updated.
>
> This makes it possible to only update objects that have been
> changed, and
> we only need to update the fields that were changed.
>
> However, I think the DirContext will overwrite the entire
> object during the bind operation, rather than updating specific fields
> on the object.
>
> Initially I was thinking that the object's attributes (primitive
> properties - not references to other objects)
> would be serialized and made into directory attributes. But I think a
> LDAP ObjectClass schema that corresponds to the
> object's class (The class of the object we are persisting) would
> have to
> be generated and stored along with the instance.
>
> This might lead to performance improvments, if doable...?
>
> Thoughts?
>
> Thanks,
> - Ole
>
>
>
>
>
> --
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com <http://www.iktek.com>

Ole Ersoy

2007-03-21, 1:11 pm

Hey David,

I just sent out a longer thang with more context, so maybe that will
help a little WRT some of your question below.

I'm going to publish an Eclipse Plugin with design documentation recipes
soon, I'm just getting a feel for whether
I should target generation of schema's for Classes right out of the
gate, or save that for later in case it requires a lot
of modification in ADS.

Either way the DAS will do all the things you are wondering about below.

You could hand write SDO's.

But you will most likely want to generate them from some model.
With EMF you could generate them from Java Interfaces, XML Schema,
or an Ecore Model.

If you check out
http://www.eclipse.org/modeling/emf/docs/

There's a lot of good articles.

I would probably try to understand how EMF works first, with respect to
generating
a model (Javabeans) from an xml schema perhaps, and then look at some of
the SDO articles.

Also, if you look at the pom.model.v400
project under rpm.factory.all in my sandbox
you will see that I regenerated the entire
Maven POM model from the Maven POM XML Schema.

The model classes generated are not SDOs.

But the EMF generator has a switch that you flip, making the classes
generated SDO classes.

Please let me know if I left anything out between this email and the one
I just sent out,
and I'll elaborate further.

Thanks,
- Ole

David Jencks wrote:
>
> On Mar 21, 2007, at 1:06 AM, Ole Ersoy wrote:
>
>
> Maybe you documented this somewhere I missed, but could we back up a
> little bit and see if we agree or understand what the goals are?
>
> For this to be useful to me in triplesec it needs to be able to do
> what the framework I wrote does:
>
> - map single valued attributes in an entry to an object field
> - map single and multiple valued attributes to an object reference or
> collection/map of object references respectively
> - map multiple valued attributes to collection valued fields of simple
> types
> - map the natural ldap tree structure to object tree structure
> - leave out layers of the ldap tree that convey type information
> rather than instance data (such as "ou=Application")
>
> those are the important ones I can remember... I hope I didn't leave
> anything too critical out.
>
> Also I don't know much of anything about SDO.... do you write your own
> POJOs and then map/enhance/code generate/?? to relate them to the
> backend data store or are classes generated from a description of the
> data store or are there generic data structures like a map of
> attribute name to attribute value?
>
>
> Tracking change information is usually desirable whether or not you
> support disconnection.
>
>
> I don't know what you mean by this. Could you provide a simple
> concrete example? Probably my lack of knowledge about ldap :-)
>
> thanks
> david jencks
>
>
>

Emmanuel Lecharny

2007-03-21, 1:11 pm

Ole,

Comments inside your mail...

On 3/21/07, Ole Ersoy <ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> <snip/>
>

SCENARIO-UPDATE
> <snip/>
> Suppose there was an ObjectClass
> defined in ADS called
> Address.
>
> Then when we did the read operation instead of
> deserializing/unmarshaling the object,
> we would grab the list of attributes on the entry, pass them to the DAS,
> and the DAS would instantiate
> the corresponding object for us.

This is something we might consider. Right now, we don't do that because it
does not make a lot of sense. Again, an entry is generaly small, so grabing
one attribute or the whole entry makes no difference compared to the IO cost
(reading some object from disk will be thousand time slower than doing any
in-memory manipulation). If we have bigger objects, with bigger values
(JPegPhoto for instance), then, yes, we have to do something. But we are
already thinking about serializing such objects in streams which will be
separated from the entries.

(The DAS would then use the setters on the object
> to set those attributes per a predefined map, using
> annotations on the object or an xml configuration file.)

We should avoid reflection as much as possible. Even in Java 1.5 it's damn
costly. We have our own serialization of objects, and we are using it. Now,
we want to extend this mechanism to all the attributeTypes, because the gain
is enormous (I have been able to speedup the server by 50% just using the
serializer for Attributes)

If we switch to another internal structuration for objects (have a look at
http://cwiki.apache.org/confluence/...xSRVx11/Backend), then using
some other mechanism may be interesting.

So now when we update street, we would only send that updated
> attribute back to ADS.
>
> This leads me to the next scenario which is outside of CRUD:
>
> SCENARIO-CREATE-SCHEMA:
> This is similar to the creation of a RDB Schema or XML Schema,
> but instead of creating these, it's a LDAP Schema defining
> new ObjectClasses that correspond to the objects we are persisting.

In fact, in Ldap, this is exactly what we have : you can't create an object
if the schema does not contain the attributeTypes and ObjectClasses. The new
system, available in 1.5.0, allow you to define a new ObjectClass, or
AttributeType, store it into ADS itself, and then create instances of thise
newly created ObjectClass

These would be written to ADS so that attributes of the objects
> could be stored the same way we store instances of ObjectClasses
> at some DN location.

Don't understand what you mean.

Does that make sense?
>
> We are now using ADS as an RDB essentially.

ADS is really close to a RDB. It has a master table storing all the entries,
and many BTrees indexing all the needed attributes.

More like an Object Oriented Database actually.

Hierarchical databes is much closer to the reality.

--
Cordialement,
Emmanuel Lécharny
www.iktek.com

Emmanuel Lecharny

2007-03-21, 1:11 pm

I just wanted to make something really clear, but I don't want to be
misunderstood :

There are two kinds of developpement :
1) Client side developpement, where you try to improve maintenability of the
code, because performance is not really the main issue (generally speaking,
when it comes to performance, you easily find out that the team was simply
not able to use the tools).
2) Server side developpement, where you have to deal with many different
clients, and the internal performance of the server is priority #1, of
course in pair with quality of developpement

Sometime, it's better to ditch a good tool if the order of performance you
get out of it is simply inacceptable. Even if you have the best
architecture, if performance sucks, your server sucks. People are not really
using DB, App servers or Ldap server with 10 000 req/s (we are not always
working with companies like E-bay or Amazon), but even then, they are asking
for those kind of performances. You can see this all over the news, company
building servers are doing speed tests, and are giving numbers which does
not have any sense ... That's simply a fact.

So, now, to be sure that I'm not discouraging anyone : this is *not* because
performance matters than we should *not* think about using new tools, and
new technos. Simply, when you think about a new way to do things, for ADS,
you will have to think about performance.

The best solution would certainly to go a little bit deeper, and try to
build some kind of small prototypes. Sometime, the advantage of using a
layer which slow down the server but ease a *lot* the implementation and
evolution of the server is much more valuable than a 2 or 3 ms gain on an
operation

Btw, now that we have almost closed the 1.5.0 version, I will run some perfs
tests like I did one year ago. I hope to be able to profile the server, and
to find some interesting bottleneck.

Guys, let's keep on the good works, don't stand up on the brakes, sometime
ideas will be dismissed, and sometime they will be so brigt that they may
take time to go through our half-dead brains ;)

Emmanuel

--
Cordialement,
Emmanuel Lécharny
www.iktek.com

Ole Ersoy

2007-03-21, 7:11 pm

Emmanuel,

I think my squirrel brain is starting to put together what you are
saying :-)

Let me see if I'm getting it.

ATTRIBUTES VS. ENTIRE OBJECT
Whether we pass an entire object or an individual attribute to ADS when
updating,
performance wise there's probably no difference since it's such a small
chunk of info.

This applies at when ADS serialized to disk and when the Application
sends data to ADS
via the directory context per the criteria that object's size is below a
certain number of KB.

So from a "Passing the Baton" point of view, it does not matter whether
it is a attribute or
an object...since their size different is typically so small.

Since this is the case, then the DAS implementation will be really
straight forward I think.

I'll just skip commenting on the rest of your in lined comments, if I
understand correctly,
since the rest is not really important anyways.

JUST A SIDE NOTE:

RDB Backend for ApacheDS
If we did this, then passing an attribute instead of an Object might
make sense, if I understand correctly.

From what I understand rear ends like Prevayler
are thousands of times faster than any RDB, even if the entire RDB were
stored in main memory (Like with hsql),
so would there ever be a point in using an RDB?

I mention this because Tuscany also has a DAS for RDB.

So an SDO model could serve as a middle tier between RDB persistance and
LDAP persistance.

Applications that need an RDB rear end, could pull info out of ADS using
the
DAS for LDAP, and store in in any RDB using DAS for RDB...

Anyways, should probably put that in a different thread or JIRA or
something...or the ADS
design document that I need to get started...

- Ole

Emmanuel Lecharny wrote:
> Ole,
>
> Comments inside your mail...
>
> On 3/21/07, *Ole Ersoy* <ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> <mailto:ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> wrote:
>
> <snip/>
>
>
>
> SCENARIO-UPDATE
> <snip/>
> Suppose there was an ObjectClass
> defined in ADS called
> Address.
>
> Then when we did the read operation instead of
> deserializing/unmarshaling the object,
> we would grab the list of attributes on the entry, pass them to
> the DAS,
> and the DAS would instantiate
> the corresponding object for us.
>
>
> This is something we might consider. Right now, we don't do that
> because it does not make a lot of sense. Again, an entry is generaly
> small, so grabing one attribute or the whole entry makes no difference
> compared to the IO cost (reading some object from disk will be
> thousand time slower than doing any in-memory manipulation). If we
> have bigger objects, with bigger values (JPegPhoto for instance),
> then, yes, we have to do something. But we are already thinking about
> serializing such objects in streams which will be separated from the
> entries.
>
> (The DAS would then use the setters on the object
> to set those attributes per a predefined map, using
> annotations on the object or an xml configuration file.)
>
>
> We should avoid reflection as much as possible. Even in Java 1.5 it's
> damn costly. We have our own serialization of objects, and we are
> using it. Now, we want to extend this mechanism to all the
> attributeTypes, because the gain is enormous (I have been able to
> speedup the server by 50% just using the serializer for Attributes)
>
> If we switch to another internal structuration for objects (have a
> look at
> http://cwiki.apache.org/confluence/...xSRVx11/Backend), then
> using some other mechanism may be interesting.
>
> So now when we update street, we would only send that updated
> attribute back to ADS.
>
> This leads me to the next scenario which is outside of CRUD:
>
> SCENARIO-CREATE-SCHEMA:
> This is similar to the creation of a RDB Schema or XML Schema,
> but instead of creating these, it's a LDAP Schema defining
> new ObjectClasses that correspond to the objects we are persisting.
>
>
> In fact, in Ldap, this is exactly what we have : you can't create an
> object if the schema does not contain the attributeTypes and
> ObjectClasses. The new system, available in 1.5.0, allow you to define
> a new ObjectClass, or AttributeType, store it into ADS itself, and
> then create instances of thise newly created ObjectClass
>

> These would be written to ADS so that attributes of the objects
> could be stored the same way we store instances of ObjectClasses
> at some DN location.
>
>
> Don't understand what you mean.
>
> Does that make sense?
>
> We are now using ADS as an RDB essentially.
>
>
> ADS is really close to a RDB. It has a master table storing all the
> entries, and many BTrees indexing all the needed attributes.
>
> More like an Object Oriented Database actually.
>
>
> Hierarchical databes is much closer to the reality.
>
>
>
> --
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com <http://www.iktek.com>

Emmanuel Lecharny

2007-03-21, 7:11 pm

On 3/21/07, Ole Ersoy <ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> Emmanuel,
>
> I think my squirrel brain is starting to put together what you are
> saying :-)
>
> Let me see if I'm getting it.
>
> ATTRIBUTES VS. ENTIRE OBJECT
> Whether we pass an entire object or an individual attribute to ADS when
> updating,
> performance wise there's probably no difference since it's such a small
> chunk of info.

In fact, we pass attributes and we update them in a whole object before
storing it. To be able to do that, we first have to get the whole object out
of the backend. For instance, if you want to add a userPassord to the
uid=oersoy, dc=apache, dc=com entry, you will just pass a modifyRequest with
the DN, and the userPassord attribute with the value to add. Then the server
will search for the corresponding entry from its DN, and if found, it will
read it entirely. Then it will add the attribute into the entry, and save it
into the backend.

This applies at when ADS serialized to disk and when the Application
> sends data to ADS
> via the directory context per the criteria that object's size is below a
> certain number of KB.

Well, the criteria must still be implemented ... It's now 2 years we know we
have to deal with such a criteria, but we didn't had time to implement it
yes... 1.5.2 maybe

So from a "Passing the Baton" point of view, it does not matter whether
> it is a attribute or
> an object...since their size different is typically so small.

We can say that

Since this is the case, then the DAS implementation will be really
> straight forward I think.

Well, it's pretty much done in two classes only, AttributesSerializerUtils
and AttributeSerializerUtils. Only 700 lines of code, javadoc included

I'll just skip commenting on the rest of your in lined comments, if I
> understand correctly,
> since the rest is not really important anyways.
>
> JUST A SIDE NOTE:
>
> RDB Backend for ApacheDS
> If we did this, then passing an attribute instead of an Object might
> make sense, if I understand correctly.

yep.

From what I understand rear ends like Prevayler
> are thousands of times faster than any RDB, even if the entire RDB were
> stored in main memory (Like with hsql),
> so would there ever be a point in using an RDB?

There are many, including high volumes, reliability, replications,
marketing, "We already have Or*cle/I*m db*/MySql/PostGresql" (TM), ...

I mention this because Tuscany also has a DAS for RDB.

We have had discussion with Tuscany guys in Austin about including ADS into
tuscany.

So an SDO model could serve as a middle tier between RDB persistance and
> LDAP persistance.

That may be a very good idea, because then it may abstract totally the
plumbery between ADS and the backend. At the moment, this plubery is not
really satisfactory.

Applications that need an RDB rear end, could pull info out of ADS using
> the
> DAS for LDAP, and store in in any RDB using DAS for RDB...

Yep.

Anyways, should probably put that in a different thread or JIRA or
> something...or the ADS
> design document that I need to get started...

We are seriously thinking about it for a 2.0 version of ADS. We might
discuss this in may, during ApacheCon (if the number of beers we will
absorb, not to mention the strange substances we will smoke, keep our brain
productive

--
Cordialement,
Emmanuel Lécharny
www.iktek.com

Ole Ersoy

2007-03-21, 7:11 pm

OK - Cool - So for LDAP DAS version 1.0
I'll just write the entire SDO datagraph to ADS with each
SDO object being an entry.

Also, (Just wanted to make sure I understand) are you saying that
having a RDB backend because of:

High volumes
Reliability
Replications
Marketing

Makes sense?

Emmanuel Lecharny wrote:
>
>
> On 3/21/07, *Ole Ersoy* <ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> <mailto:ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> wrote:
>
> Emmanuel,
>
> I think my squirrel brain is starting to put together what you are
> saying :-)
>
> Let me see if I'm getting it.
>
> ATTRIBUTES VS. ENTIRE OBJECT
> Whether we pass an entire object or an individual attribute to ADS
> when
> updating,
> performance wise there's probably no difference since it's such a
> small
> chunk of info.
>
>
> In fact, we pass attributes and we update them in a whole object
> before storing it. To be able to do that, we first have to get the
> whole object out of the backend. For instance, if you want to add a
> userPassord to the uid=oersoy, dc=apache, dc=com entry, you will just
> pass a modifyRequest with the DN, and the userPassord attribute with
> the value to add. Then the server will search for the corresponding
> entry from its DN, and if found, it will read it entirely. Then it
> will add the attribute into the entry, and save it into the backend.
>
> This applies at when ADS serialized to disk and when the Application
> sends data to ADS
> via the directory context per the criteria that object's size is
> below a
> certain number of KB.
>
>
> Well, the criteria must still be implemented ... It's now 2 years we
> know we have to deal with such a criteria, but we didn't had time to
> implement it yes... 1.5.2 maybe
>
> So from a "Passing the Baton" point of view, it does not matter
> whether
> it is a attribute or
> an object...since their size different is typically so small.
>
>
> We can say that
>
> Since this is the case, then the DAS implementation will be really
> straight forward I think.
>
>
> Well, it's pretty much done in two classes only,
> AttributesSerializerUtils and AttributeSerializerUtils. Only 700 lines
> of code, javadoc included

>
> I'll just skip commenting on the rest of your in lined comments, if I
> understand correctly,
> since the rest is not really important anyways.
>
> JUST A SIDE NOTE:
>
> RDB Backend for ApacheDS
> If we did this, then passing an attribute instead of an Object might
> make sense, if I understand correctly.
>
>
> yep.
>
> From what I understand rear ends like Prevayler
> are thousands of times faster than any RDB, even if the entire RDB
> were
> stored in main memory (Like with hsql),
> so would there ever be a point in using an RDB?
>
>
> There are many, including high volumes, reliability, replications,
> marketing, "We already have Or*cle/I*m db*/MySql/PostGresql" (TM), ...
>
> I mention this because Tuscany also has a DAS for RDB.
>
>
> We have had discussion with Tuscany guys in Austin about including ADS
> into tuscany.
>
> So an SDO model could serve as a middle tier between RDB
> persistance and
> LDAP persistance.
>
>
> That may be a very good idea, because then it may abstract totally the
> plumbery between ADS and the backend. At the moment, this plubery is
> not really satisfactory.
>
> Applications that need an RDB rear end, could pull info out of ADS
> using
> the
> DAS for LDAP, and store in in any RDB using DAS for RDB...
>
>
> Yep.
>
> Anyways, should probably put that in a different thread or JIRA or
> something...or the ADS
> design document that I need to get started...
>
>
> We are seriously thinking about it for a 2.0 version of ADS. We might
> discuss this in may, during ApacheCon (if the number of beers we will
> absorb, not to mention the strange substances we will smoke, keep our
> brain productive

>
>
> --
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com <http://www.iktek.com>

Luciano Resende

2007-03-21, 7:11 pm

Sorry if I'm just jumping on the discussion, but I'd like to understand it a
little more. Is the idea here to have client performing LDAP operations,
using a SDO/DAS layer that can be easily switched back and forth between a
pure LDAP repository and a RDB repository ?

---> LDAP Respository
Client -> LDAP -> SDO/DAS ->
---> RDB Repository

Or it's going to be two ways to access the repository, the current way using
LDAP, and also a new way using SDO/DAS ?

--
Luciano Resende
http://people.apache.org/~lresende

On 3/21/07, Ole Ersoy <ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> OK - Cool - So for LDAP DAS version 1.0
> I'll just write the entire SDO datagraph to ADS with each
> SDO object being an entry.
>
> Also, (Just wanted to make sure I understand) are you saying that
> having a RDB backend because of:
>
> High volumes
> Reliability
> Replications
> Marketing
>
> Makes sense?
>
>
> Emmanuel Lecharny wrote:
> I
>
>

Ole Ersoy

2007-03-21, 7:11 pm

Shoot - Goota run - I'm off to Canada for a week for a wedding - So I'll
reply more when I get back...

Ciao,
- Ole

Luciano Resende wrote:
> Sorry if I'm just jumping on the discussion, but I'd like to
> understand it a little more. Is the idea here to have client
> performing LDAP operations, using a SDO/DAS layer that can be easily
> switched back and forth between a pure LDAP repository and a RDB
> repository ?
>
> ---> LDAP Respository
> Client -> LDAP -> SDO/DAS ->
> ---> RDB Repository
>
>
> Or it's going to be two ways to access the repository, the current way
> using LDAP, and also a new way using SDO/DAS ?
>
> --
> Luciano Resende
> http://people.apache.org/~lresende <http://people.apache.org/%7Elresende>
>
> On 3/21/07, *Ole Ersoy* <ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> <mailto:ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> wrote:
>
> OK - Cool - So for LDAP DAS version 1.0
> I'll just write the entire SDO datagraph to ADS with each
> SDO object being an entry.
>
> Also, (Just wanted to make sure I understand) are you saying that
> having a RDB backend because of:
>
> High volumes
> Reliability
> Replications
> Marketing
>
> Makes sense?
>
>
> Emmanuel Lecharny wrote:
> <mailto:ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
> you are
> to ADS
> such a
> just
> Application
> size is
> time to
> really
> lines
> comments, if I
> Object might
> entire RDB
> (TM), ...
> including ADS
> totally the
> of ADS
> JIRA or
> might
> we will
> keep our
>
>
>
>

Emmanuel Lecharny

2007-03-21, 7:11 pm

Luciano Resende a écrit :

> Sorry if I'm just jumping on the discussion, but I'd like to
> understand it a
> little more. Is the idea here to have client performing LDAP operations,
> using a SDO/DAS layer that can be easily switched back and forth
> between a
> pure LDAP repository and a RDB repository ?
>
> ---> LDAP Respository
> Client -> LDAP -> SDO/DAS ->
> ---> RDB Repository

This is another project. We can call it a proxy. What we were discussing
about is the connexion between the ldap engine and the backend.

>
>
> Or it's going to be two ways to access the repository, the current way
> using
> LDAP, and also a new way using SDO/DAS ?

This is it. But the first point is also something we are thinking about.

Emmanuel

Ole Ersoy

2007-03-22, 1:11 am

OK - Flight got canceled so I'm back in action until tomorrow morning.

Emmanuel Lecharny wrote:[vbcol=seagreen]
> Luciano Resende a écrit :
>
Yes.

Let me just give a concrete example of what I mean so that it's clear.

Someone creates a webapp, that stores account information.
So theres a browser client that talks to Tomcat.
Tomcat runs a servlet talks to an integration layer consisting of SDOs.

When a user changes account information Tomcat uses the webapps's
integration layer to update the persistance tier, consisting either of
an RDB or LDAP server.

Thus the SDO instances that hold the account data could retrieve their
state either
from an RDB or from ADS.

If they get the account information from the RDB, they would use the RDB
DAS.

If they want to get the info from ADS, they would use the LDAP DAS.

FOR READING
SDOs request state
Client(SDOs) -- > LDAP DAS --> ADS
Client(SDOs) <-- LDAP DAS <-- ADS

SDOs need to update ADS
Client(SDOs) -- > LDAP DAS --> ADS

So that's what I'm thinking.

I think we got on the track of discussing ADS's persistance layer as
well due to me wondering
whether whether it would be faster if an LDAP jndi client
sent individual attributes to be persisted, rather than an entire
object. However I did not use the wording LDAP jndi client,
so it could have been miss understood.

So are we cleared up?

Thanks for jumping in Luciano :-)

- Ole

Alex Karasulu

2007-03-27, 1:11 pm

Ole,

On 3/21/07, Ole Ersoy <ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Hey Guys,
>
> Just wanted to see if anyone had any thoughts on handling updates
> to Java beans (Service Data Objects - but basically the same thing)
> persisted with ADS.
>
> With Service Data Objects we create a datagraph that is then disconnected
> from it's persistence source and we can mutate it. Then later we want to
> persist the graph. Each object in the graph has a change summary, that
> stores the fields that were updated.

This change summary is very interesting. I had experimented with something
similar which David Jencks did not like too much. Basically the modifier
pattern
was being used to track modifications to attributes of entities. It was
tracking
the set of modify add, remove, and replace operations to perform on each
attribute.

> This makes it possible to only update objects that have been changed, and
> we only need to update the fields that were changed.

Exactly this is what I was doing in this one admin API I had in triplesec.

> However, I think the DirContext will overwrite the entire
> object during the bind operation, rather than updating specific fields
> on the object.

Hmmm with heirarchical services in JNDI you should not be using bind(). You
should be using the createSubcontext() and modifyAttributes() methods
instead. You
might want to go through the JNDI tutorial for LDAP just to get a good feel
for how to
work with non-flat namespaces using JNDI. Namely with LDAP you don't need
to
rebind the object with a modification to an attribute. This is what the
modify
operations are for.

http://bsd.cs.cofc.edu/Java/Javadoc...odifyAttributes(javax.naming.Name,
int, javax.naming.directory.Attributes)

> Initially I was thinking that the object's attributes (primitive
> properties - not references to other objects)
> would be serialized and made into directory attributes. But I think a
> LDAP ObjectClass schema that corresponds to the
> object's class (The class of the object we are persisting) would have to
> be generated and stored along with the instance.
>
> This might lead to performance improvments, if doable...?
>
> Thoughts?

Hmmm I think some of your premisses in this question may be due to
considering the use
of bind() instead of using modifyAttributes() and createSubcontext(). If
you use these methods
I think there is no further preformance issue to consider. WDYT?

Alex

David Jencks

2007-03-27, 1:11 pm

On Mar 27, 2007, at 11:28 AM, Alex Karasulu wrote:

> Ole,
>
> On 3/21/07, Ole Ersoy <ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hey Guys,
>
> Just wanted to see if anyone had any thoughts on handling updates
> to Java beans (Service Data Objects - but basically the same thing)
> persisted with ADS.
>
> With Service Data Objects we create a datagraph that is then
> disconnected
> from it's persistence source and we can mutate it. Then later we
> want to
> persist the graph. Each object in the graph has a change summary,
> that
> stores the fields that were updated.
>
> This change summary is very interesting. I had experimented with
> something
> similar which David Jencks did not like too much. Basically the
> modifier pattern
> was being used to track modifications to attributes of entities.
> It was tracking
> the set of modify add, remove, and replace operations to perform on
> each attribute.

What I didn't like was that you were keeping track of the
modifications outside the POJO-like data object itself in (IMO) very
hard to use helper objects, and you didn't write a framework. I took
essentially the same idea and came up with something pretty similar
to jpa/jdo, where you have things that look like POJOs to the outside
world, but inside they keep track of how they relate to what's in the
persistent store. The main differences to the ideas of jpa are that
I don't support disconnection and you have to do the enhancement
yourself. In a non-locking not-transactional environment I'm not
sure exactly what disconnection means so this might not really be
accurate. IIRC this stuff is all in the sandbox/triplesec-jacc2 branch.

In the past I've tried to describe the kinds of ldap <> object
mappings I need and support in my framework but haven't understood
from Ole whether the DAS will offer similar capabilities.

thanks
david jencks

>
>
> This makes it possible to only update objects that have been
> changed, and
> we only need to update the fields that were changed.
>
> Exactly this is what I was doing in this one admin API I had in
> triplesec.
>
>
> However, I think the DirContext will overwrite the entire
> object during the bind operation, rather than updating specific fields
> on the object.
>
> Hmmm with heirarchical services in JNDI you should not be using bind
> (). You
> should be using the createSubcontext() and modifyAttributes()
> methods instead. You
> might want to go through the JNDI tutorial for LDAP just to get a
> good feel for how to
> work with non-flat namespaces using JNDI. Namely with LDAP you
> don't need to
> rebind the object with a modification to an attribute. This is
> what the modify
> operations are for.
> http://bsd.cs.cofc.edu/Java/Javadoc...ming/directory/
> DirContext.html#modifyAttributes(javax.naming.Name , int,
> javax.naming.directory.Attributes)
>
>
>
> Initially I was thinking that the object's attributes (primitive
> properties - not references to other objects)
> would be serialized and made into directory attributes. But I think a
> LDAP ObjectClass schema that corresponds to the
> object's class (The class of the object we are persisting) would
> have to
> be generated and stored along with the instance.
>
> This might lead to performance improvments, if doable...?
>
> Thoughts?
>
> Hmmm I think some of your premisses in this question may be due to
> considering the use
> of bind() instead of using modifyAttributes() and createSubcontext
> (). If you use these methods
> I think there is no further preformance issue to consider. WDYT?
>
> Alex
>
>
>

Alex Karasulu

2007-03-27, 7:11 pm

Yeah I think it will be very similar to what you have done or what I have
tried to do.

Alex

On 3/27/07, David Jencks <david_jencks-/E1597aS9LQAvxtiuMwx3w@public.gmane.org> wrote:
>
>
> On Mar 27, 2007, at 11:28 AM, Alex Karasulu wrote:
>
> Ole,
>
> On 3/21/07, Ole Ersoy <ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>
>
> This change summary is very interesting. I had experimented with
> something
> similar which David Jencks did not like too much. Basically the modifier
> pattern
> was being used to track modifications to attributes of entities. It was
> tracking
> the set of modify add, remove, and replace operations to perform on each
> attribute.
>
>
>
> What I didn't like was that you were keeping track of the modifications
> outside the POJO-like data object itself in (IMO) very hard to use helper
> objects, and you didn't write a framework. I took essentially the same idea
> and came up with something pretty similar to jpa/jdo, where you have things
> that look like POJOs to the outside world, but inside they keep track of how
> they relate to what's in the persistent store. The main differences to the
> ideas of jpa are that I don't support disconnection and you have to do the
> enhancement yourself. In a non-locking not-transactional environment I'm
> not sure exactly what disconnection means so this might not really be
> accurate. IIRC this stuff is all in the sandbox/triplesec-jacc2 branch.
>
>
> In the past I've tried to describe the kinds of ldap <> object mappings I
> need and support in my framework but haven't understood from Ole whether the
> DAS will offer similar capabilities.
>
>
> thanks
> david jencks
>
>
>
>
>
>
>
>
> Exactly this is what I was doing in this one admin API I had in triplesec.
>
>
>
>
>
> Hmmm with heirarchical services in JNDI you should not be using bind().
> You
> should be using the createSubcontext() and modifyAttributes() methods
> instead. You
> might want to go through the JNDI tutorial for LDAP just to get a good
> feel for how to
> work with non-flat namespaces using JNDI. Namely with LDAP you don't need
> to
> rebind the object with a modification to an attribute. This is what the
> modify
> operations are for.
>
> http://bsd.cs.cofc.edu/Java/Javadoc...odifyAttributes(javax.naming.Name
> , int, javax.naming.directory.Attributes)
>
>
>
>
>
> Hmmm I think some of your premisses in this question may be due to
> considering the use
> of bind() instead of using modifyAttributes() and createSubcontext(). If
> you use these methods
> I think there is no further preformance issue to consider. WDYT?
>
> Alex
>
>
>
>
>

Ole Ersoy

2007-03-28, 1:11 am

Alex Karasulu wrote:
> Ole,
>
> On 3/21/07, *Ole Ersoy* <ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> <mailto:ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> wrote:
>
> Hey Guys,
>
> Just wanted to see if anyone had any thoughts on handling updates
> to Java beans (Service Data Objects - but basically the same thing)
> persisted with ADS.
>
> With Service Data Objects we create a datagraph that is then
> disconnected
> from it's persistence source and we can mutate it. Then later we
> want to
> persist the graph. Each object in the graph has a change summary,
> that
> stores the fields that were updated.
>
>
> This change summary is very interesting. I had experimented with
> something
> similar which David Jencks did not like too much. Basically the
> modifier pattern
> was being used to track modifications to attributes of entities. It
> was tracking
> the set of modify add, remove, and replace operations to perform on
> each attribute.
>
>
>
> This makes it possible to only update objects that have been
> changed, and
> we only need to update the fields that were changed.
>
>
> Exactly this is what I was doing in this one admin API I had in triplesec.
>
>
>
> However, I think the DirContext will overwrite the entire
> object during the bind operation, rather than updating specific
> fields
> on the object.
>
> Hmmm with heirarchical services in JNDI you should not be using
> bind(). You
> should be using the createSubcontext() and modifyAttributes() methods
> instead.

Oh OK. My impression was that I would use createSubcontext() when
creating a new context for a serialized Java object for example.

Let me give an exampel - I'll be starting the DAS "serious" work tomorrow,
so if I'm off on this, I'll eventually find out :-), but this may help
facilitate
the conversation with respect to updating attributes vs. the entire object.

Suppose my initial context is:

|uid=ole,ou=users, ou=system
|

And I wanted to add a sub context for an Instance of MyClass
with reference named myClassInstance

I could do something like

ctx.createSubcontext("cn=myClassInstance");

Which would create the context:

|cn=myClassInstance, uid=ole, ou=users, ou=system|

Then to store a my serialized myClassInstance under this dn in a
javaSerializedData attribute, I would do something like this:

|||String bindContext = "cn=|myClassInstance|,uid=ole,ou=users,
ou="system";|

|ctx.bind( bindContext, ||nameOfMyObject||)

I'm also guessing that the bind operation would take care
of the createSubcontext() step for me, ... but anyways...

|

> You
> might want to go through the JNDI tutorial for LDAP just to get a good
> feel for how to
> work with non-flat namespaces using JNDI. Namely with LDAP you don't
> need to
> rebind the object with a modification to an attribute. This is what
> the modify
> operations are for.

Just want to make sure we are talking about the same thing here.
We may have some attributes stored in ADS under:
|DN: uid=ole,ou=users, ou="system

This DN has the Person ObjectClass associated with it.
So I may wish to change one of the attributes (The password attribute
for instance)
that are associated with the Person ObjectClass.
Then I could see using a modify operation.

Is that also the case when storing entire javaObjects
in a javaSerializedData attribute?

|I just want to make sure we are clearly separating Java object
attributes (Primitive Java class members)
from LDAP attributes.

Let me stop here in case I'm making any sense?

If not hopefully once I'm done writing the design guide and get a
terminology and concept section going
things will be a little clearer.

> http://bsd.cs.cofc.edu/Java/Javadoc...odifyAttributes(javax.naming.Name
> <http://bsd.cs.cofc.edu/Java/Javadoc...vax.naming.Name>,
> int, javax.naming.directory.Attributes)
>
>
Cool - I'll have a looksee - I've been using the articles on the
ApacheDS website as a reference so far, but the more material the better.

>
>
> Initially I was thinking that the object's attributes (primitive
> properties - not references to other objects)
> would be serialized and made into directory attributes. But I think a
> LDAP ObjectClass schema that corresponds to the
> object's class (The class of the object we are persisting) would
> have to
> be generated and stored along with the instance.
>
> This might lead to performance improvments, if doable...?
>
> Thoughts?
>
>
> Hmmm I think some of your premisses in this question may be due to
> considering the use
> of bind() instead of using modifyAttributes() and createSubcontext().
> If you use these methods
> I think there is no further preformance issue to consider. WDYT?

Ahh - This is the part where I need clarification.

If we serialize an entire object to ApacheDS, then we have to get it
from ApacheDS,
update it, and send it back to be stored in a javaSerializedData
attribute right?

For instance I may have a Class called UserClass with a String member
userName.

I create an instance of UserClass called userClass.

The I do
userClass.setUserName("ole");

Now I want to store userClass in ADS here:
|DN: cn=|userClass|,ou=users, ou="system

So I do this:

||String bindContext = "cn=|myClassInstance|,ou=users, ou="system";|

|ctx.bind( bindContext, ||userClass||);

Now userClass is in ADS stored as a javaSerializedData LDAP attribute value,
hanging off of the bindContext I specified.

Later when I want to update
the userName member of the UserClass instance I serialized to ADS,
I need to use JNDI to load the instance again right?

Then update userName. Then serialize userClass back to ADS again?

|

WDYT?

I mentioned some stuff about generating an ObjectClass schema that is
the LDAP schema of the Java Class UserClass,
so that the primitive members (That have Java primitive types) would be
stored as LDAP atttributes rather than storing the entire
object as a single javaSerializedData attribute attached to a DN. I'll
hold off on commenting more in case I'm getting warmer.
>
> Alex
- Ole
>
>
>

Ole Ersoy

2007-03-28, 1:11 am

David,

The LDAP DAS should work exactly the same way an RDB DAS would work.

So from the SDO point of view (Or your Java bean point of view) it does
not really
matter where the data is being persisted.

It could go to an XML file(s), an RDB, ADS, ERP, ..... It just depends on
which DAS the SDO datagraph is being serialized with.

I'm going to start writing the DAS design docs tomorrow, starting with
conventions used to perform CRUD against ADS for an SDO datagraph,
so hang in there. I'm 99.9% sure that it does everything you want it to
and more,
but the proof is in the pudding, and I would like you to be able to
judge for yourself,
so I'll hurry it up.

Cheers,
- Ole

David Jencks wrote:
>
> On Mar 27, 2007, at 11:28 AM, Alex Karasulu wrote:
>
>
> What I didn't like was that you were keeping track of the
> modifications outside the POJO-like data object itself in (IMO) very
> hard to use helper objects, and you didn't write a framework. I took
> essentially the same idea and came up with something pretty similar to
> jpa/jdo, where you have things that look like POJOs to the outside
> world, but inside they keep track of how they relate to what's in the
> persistent store. The main differences to the ideas of jpa are that I
> don't support disconnection and you have to do the enhancement
> yourself. In a non-locking not-transactional environment I'm not sure
> exactly what disconnection means so this might not really be
> accurate. IIRC this stuff is all in the sandbox/triplesec-jacc2 branch.
>
> In the past I've tried to describe the kinds of ldap <> object
> mappings I need and support in my framework but haven't understood
> from Ole whether the DAS will offer similar capabilities.
>
> thanks
> david jencks
>
>

Stefan Seelmann

2007-03-28, 7:11 pm

Ole,

Ole Ersoy schrieb:
>
> Just want to make sure we are talking about the same thing here.
> We may have some attributes stored in ADS under:
> |DN: uid=ole,ou=users, ou="system
>
> This DN has the Person ObjectClass associated with it.
> So I may wish to change one of the attributes (The password attribute
> for instance)
> that are associated with the Person ObjectClass.
> Then I could see using a modify operation.
>
>
> Is that also the case when storing entire javaObjects
> in a javaSerializedData attribute?
>
> |I just want to make sure we are clearly separating Java object
> attributes (Primitive Java class members)
> from LDAP attributes.
>
> Let me stop here in case I'm making any sense?
>
> If not hopefully once I'm done writing the design guide and get a
> terminology and concept section going
> things will be a little clearer.

What kind of storage are you thinking of? Saving the entire SDO in the
javaSerializedData or saving the attributes of the SDO?

When storing entire Java objects using the javaSerializedData attribute
the whole serialized object is transmitted over the wire and stored as
binary (BLOB). This is a generic way to save any type of serializable
object into a directory. But only Java programs could read this object
from directory again, other programming languages could not handle these
objects. Sun's JNDI LDAP provider uses the javaObject object class and
the javaSerializedData, see RFC2713.

To allow interoperability with other languages is would be better to
save objects as entries with attributes, maybe using a custom schema
with special object classes and attribute types. Then of course a
mapping between the Java classes and attributes and the LDAP object
classes and attribute types is needed.

>
> Ahh - This is the part where I need clarification.
>
> If we serialize an entire object to ApacheDS, then we have to get it
> from ApacheDS,
> update it, and send it back to be stored in a javaSerializedData
> attribute right?

Yes.

>
> For instance I may have a Class called UserClass with a String member
> userName.
>
> I create an instance of UserClass called userClass.
>
> The I do
> userClass.setUserName("ole");
>
> Now I want to store userClass in ADS here:
> |DN: cn=|userClass|,ou=users, ou="system
>
> So I do this:
>
> ||String bindContext = "cn=|myClassInstance|,ou=users, ou="system";|
>
> |ctx.bind( bindContext, ||userClass||);
>
> Now userClass is in ADS stored as a javaSerializedData LDAP attribute
> value,
> hanging off of the bindContext I specified.
>
> Later when I want to update
> the userName member of the UserClass instance I serialized to ADS,
> I need to use JNDI to load the instance again right?
>
> Then update userName. Then serialize userClass back to ADS again?
>
> |
>
> WDYT?
>
> I mentioned some stuff about generating an ObjectClass schema that is
> the LDAP schema of the Java Class UserClass,
> so that the primitive members (That have Java primitive types) would be
> stored as LDAP atttributes rather than storing the entire
> object as a single javaSerializedData attribute attached to a DN. I'll
> hold off on commenting more in case I'm getting warmer.

Yeah, I think that would be the better approach.

I also think that the LDAP DAS isn't limited to ApacheDS as backend. Any
LDAP directory could be used as backend.

Regards,
Stefan

Emmanuel Lecharny

2007-03-28, 7:11 pm

Stefan Seelmann a écrit :

><SNIP/>
>To allow interoperability with other languages is would be better to
>save objects as entries with attributes, maybe using a custom schema
>with special object classes and attribute types. Then of course a
>mapping between the Java classes and attributes and the LDAP object
>classes and attribute types is needed.
>
>
Stefan,

just to inform you that we have this meta-schema in ADS 1.5.0, it's
described in
http://cwiki.apache.org/confluence/...ystem+Redesign.

As a matter of fact, this is very interesting that you have interested
in this thread, because this is something we would like to add to
LdapStudio : a way to handle this meta-schema. Atm, when using the
browser, the informations stored in cn=schema are just totally cryptic.
e have had a convo with pam today about a new plugin we would like to
have to manage the newly implemented schema.

Emmanuel.

Ole Ersoy

2007-03-28, 7:11 pm

Yes!

That's precisely what I'd like to be able to do.

Define a Custom ObjectClass (Schema) for a Java Class,
then store the members of the Java Class that are not references to
other objects
using the custom ObjectClass.

I just sent out another question on how to do this a little while ago, so
I'll keep it brief here.

Cheers,
- Ole

Emmanuel Lecharny wrote:
> Stefan Seelmann a écrit :
>
> Stefan,
>
> just to inform you that we have this meta-schema in ADS 1.5.0, it's
> described in
> http://cwiki.apache.org/confluence/...ystem+Redesign.
>
>
> As a matter of fact, this is very interesting that you have interested
> in this thread, because this is something we would like to add to
> LdapStudio : a way to handle this meta-schema. Atm, when using the
> browser, the informations stored in cn=schema are just totally
> cryptic. e have had a convo with pam today about a new plugin we would
> like to have to manage the newly implemented schema.
>
> Emmanuel.
>

Ole Ersoy

2007-03-28, 7:11 pm

OK - I just answered the one Emmanuel answered before this, anyways...
I think we are all thinking the same thing...

Stefan Seelmann wrote:
> Ole,
>
>
> Ole Ersoy schrieb:
>
>
> What kind of storage are you thinking of? Saving the entire SDO in the
> javaSerializedData or saving the attributes of the SDO?
>
> When storing entire Java objects using the javaSerializedData attribute
> the whole serialized object is transmitted over the wire and stored as
> binary (BLOB). This is a generic way to save any type of serializable
> object into a directory. But only Java programs could read this object
> from directory again, other programming languages could not handle these
> objects. Sun's JNDI LDAP provider uses the javaObject object class and
> the javaSerializedData, see RFC2713.
>
> To allow interoperability with other languages is would be better to
> save objects as entries with attributes, maybe using a custom schema
> with special object classes and attribute types. Then of course a
> mapping between the Java classes and attributes and the LDAP object
> classes and attribute types is needed.
>
>
>
>
> Yes.
>
>
>
> Yeah, I think that would be the better approach.
>
> I also think that the LDAP DAS isn't limited to ApacheDS as backend. Any
> LDAP directory could be used as backend.
>
> Regards,
> Stefan
>
>

Alex Karasulu

2007-03-29, 1:11 am

Hi Ole,

Excuse the late response ... I've been lazy while on vacation. More inline
....

On 3/28/07, Ole Ersoy <ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> Alex Karasulu wrote:

SNIP ...
[vbcol=seagreen]
> Hmmm with heirarchical services in JNDI you should not be using
>
>
> Oh OK. My impression was that I would use createSubcontext() when
> creating a new context for a serialized Java object for example.

Oh sorry I thought you were not storing a serialized object. Then bind()
would
help you accomplish that I think. However this is not very efficient
because
of the reasons you mention: you would in essence be updating the entire
object
when just one member changes. And you're dealing with a blob of serialized
data which is bigger. Meaning the transfers are bigger for a small change
in the
overall object.

My impression was that you were converting an object into an entry with
attributes
for it's properties which is more like what you do when you persist an
object in a
record in a database. For example you have a User object and you convert
that
into a set of attributes like so:

givenName: johny
surName: walker
commonName: wiskey
age: 12

etc ...

Then this way you would call createSubcontext with the attributes and that
would
persist the entry. If you change the age to 15 then there is a single
update operation
on that attribute. Also this way indexing can be taken advantage of for
attributes for
speeding up search. I have no idea how you would search for serialized
objects in
a directory efficiently.

Let me give an exampel - I'll be starting the DAS "serious" work tomorrow,
> so if I'm off on this, I'll eventually find out :-), but this may help
> facilitate
> the conversation with respect to updating attributes vs. the entire
> object.
>
> Suppose my initial context is:
>
> |uid=ole,ou=users, ou=system
> |
>
> And I wanted to add a sub context for an Instance of MyClass
> with reference named myClassInstance
>
> I could do something like
>
> ctx.createSubcontext("cn=myClassInstance");
>
> Which would create the context:
>
> |cn=myClassInstance, uid=ole, ou=users, ou=system|

I think you were right in using bind() where you provide the object to
serialize. This createSubcontext() operation will merely create a
javaContainer I think. There's an
RFC on this which I implemented a while back and you can find more info in
the
JNDI tutorial about this as well.

However again I don't think using this serialization mechanism is such a
good idea
because of the search implications. How would you search a blob of binary
data
effectively?

Then to store a my serialized myClassInstance under this dn in a
> javaSerializedData attribute, I would do something like this:
>
> |||String bindContext = "cn=|myClassInstance|,uid=ole,ou=users,
> ou="system";|
>
> |ctx.bind( bindContext, ||nameOfMyObject||)
>
>
> I'm also guessing that the bind operation would take care
> of the createSubcontext() step for me, ... but anyways...

Yes you were right. Actually you would not need to createSubcontext()
operation at all just use bind().

I recommend just going through the JNDI tutorial on this stuff and testing
these operations on the server just to see what happens.

|
>
>
> Just want to make sure we are talking about the same thing here.
> We may have some attributes stored in ADS under:
> |DN: uid=ole,ou=users, ou="system
>
> This DN has the Person ObjectClass associated with it.
> So I may wish to change one of the attributes (The password attribute
> for instance)
> that are associated with the Person ObjectClass.
> Then I could see using a modify operation.

Yeah we're mixing object serialization here with converting that object to
an entry with attributes.

Is that also the case when storing entire javaObjects
> in a javaSerializedData attribute?

Technically you can use modify on javaSerializedData but you're replacing
the blob of serialized bytes representing the serialized object.

> |I just want to make sure we are clearly separating Java object
> attributes (Primitive Java class members)
> from LDAP attributes.

Basically my advice would be to stay clear of using Java serialization in
the directory.

Let me stop here in case I'm making any sense?
>
> If not hopefully once I'm done writing the design guide and get a
> terminology and concept section going
> things will be a little clearer.
>
>
> http://bsd.cs.cofc.edu/Java/Javadoc...odifyAttributes(javax.naming.Name
> http://bsd.cs.cofc.edu/Java/Javadoc...vax.naming.Name
> Cool - I'll have a looksee - I've been using the articles on the
> ApacheDS website as a reference so far, but the more material the better.
>
> a
>
> Ahh - This is the part where I need clarification.
>
> If we serialize an entire object to ApacheDS, then we have to get it
> from ApacheDS,
> update it, and send it back to be stored in a javaSerializedData
> attribute right?

Right exactly! This is why Java LDAP serialization is not a good idea for
your DAS thingy.

For instance I may have a Class called UserClass with a String member
> userName.
>
> I create an instance of UserClass called userClass.
>
> The I do
> userClass.setUserName("ole");
>
> Now I want to store userClass in ADS here:
> |DN: cn=|userClass|,ou=users, ou="system
>
> So I do this:
>
> ||String bindContext = "cn=|myClassInstance|,ou=users, ou="system";|
>
> |ctx.bind( bindContext, ||userClass||);
>
> Now userClass is in ADS stored as a javaSerializedData LDAP attribute
> value,
> hanging off of the bindContext I specified.
>
> Later when I want to update
> the userName member of the UserClass instance I serialized to ADS,
> I need to use JNDI to load the instance again right?

Yep unless you write the code to resusitate the serialized object manually.

Then update userName. Then serialize userClass back to ADS again?

Yep.

|
>
> WDYT?
>
> I mentioned some stuff about generating an ObjectClass schema that is
> the LDAP schema of the Java Class UserClass,
> so that the primitive members (That have Java primitive types) would be
> stored as LDAP atttributes rather than storing the entire
> object as a single javaSerializedData attribute attached to a DN.

Yes this would be best in my opinion.

Regards,
Alex

Ole Ersoy

2007-03-29, 1:11 am

OK - I just sent another reply - that sort of makes this on irrelevant,
but I'll send it anyways for context....

Alex Karasulu wrote:
> Hi Ole,
>
> Excuse the late response ... I've been lazy while on vacation. More
> inline ...

Yeah - The Balls - Vacationing and Sheeeet :-)

Welcome back!
>
> On 3/28/07, *Ole Ersoy* <ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> <mailto:ole.ersoy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>> wrote:
>
> Alex Karasulu wrote:
>
>
> SNIP ...
>
> methods
>
>
> Oh OK. My impression was that I would use createSubcontext() when
> creating a new context for a serialized Java object for example.
>
>
>
> Oh sorry I thought you were not storing a serialized object. Then
> bind() would
> help you accomplish that I think. However this is not very efficient
> because
> of the reasons you mention: you would in essence be updating the
> entire object
> when just one member changes. And you're dealing with a blob of
> serialized
> data which is bigger. Meaning the transfers are bigger for a small
> change in the
> overall object.

Yeah I was actually going to break the the object graph up
so that the entire DataGraph would not get stored in one entry,
but instead I would store proxies that only had the non object
references stored, and then a proxy would be used to load the reference,
but even this is somewhat non-sexy.

>
>
> My impression was that you were converting an object into an entry
> with attributes
> for it's properties which is more like what you do when you persist an
> object in a
> record in a database. For example you have a User object and you
> convert that
> into a set of attributes like so:

Yes - That is what I want to do.

So first I want to create an ObjectClass that has a 1:1 relationship
with a Java class.

This is somewhat similar to creating doing a CREATE TABLE TABLE_NAME in SQL,
but different when there is only a single instance of the class's type
in the model.
>
> givenName: johny
> surName: walker
> commonName: wiskey
> age: 12
>
> etc ...
>
> Then this way you would call createSubcontext with the attributes and
> that would
> persist the entry.

Yes - And then I would assign the custom ObjectClass to this entry.

> If you change the age to 15 then there is a single update operation
> on that attribute.

It's a beautiful thing!

> Also this way indexing can be taken advantage of for attributes for
> speeding up search. I have no idea how you would search for
> serialized objects in
> a directory efficiently.

Fabulous!
>
> Let me give an exampel - I'll be starting the DAS "serious" work
> tomorrow,
> so if I'm off on this, I'll eventually find out :-), but this may
> help
> facilitate
> the conversation with respect to updating attributes vs. the
> entire object.
>
> Suppose my initial context is:
>
> |uid=ole,ou=users, ou=system
> |
>
> And I wanted to add a sub context for an Instance of MyClass
> with reference named myClassInstance
>
> I could do something like
>
> ctx.createSubcontext("cn=myClassInstance");
>
> Which would create the context:
>
> |cn=myClassInstance, uid=ole, ou=users, ou=system|
>
>
> I think you were right in using bind() where you provide the object to
> serialize. This createSubcontext() operation will merely create a
> javaContainer I think. There's an
> RFC on this which I implemented a while back and you can find more
> info in the
> JNDI tutorial about this as well.
>
> However again I don't think using this serialization mechanism is such
> a good idea
> because of the search implications. How would you search a blob of
> binary data
> effectively?

Indeed.

>
> Then to store a my serialized myClassInstance under this dn in a
> javaSerializedData attribute, I would do something like this:
>
> |||String bindContext = "cn=|myClassInstance|,uid=ole,ou=users,
> ou="system";|
>
> |ctx.bind( bindContext, ||nameOfMyObject||)
>
>
> I'm also guessing that the bind operation would take care
> of the createSubcontext() step for me, ... but anyways...
>
>
> Yes you were right. Actually you would not need to createSubcontext()
> operation at all just use bind().
>
> I recommend just going through the JNDI tutorial on this stuff and
> testing these operations on the server just to see what happens.
>
> |
>
> a good
> don't
>
> Just want to make sure we are talking about the same thing here.
> We may have some attributes stored in ADS under:
> |DN: uid=ole,ou=users, ou="system
>
> This DN has the Person ObjectClass associated with it.
> So I may wish to change one of the attributes (The password attribute
> for instance)
> that are associated with the Person ObjectClass.
> Then I could see using a modify operation.
>
>
> Yeah we're mixing object serialization here with converting that
> object to an entry with attributes.
>
> Is that also the case when storing entire javaObjects
> in a javaSerializedData attribute?
>
>
> Technically you can use modify on javaSerializedData but you're
> replacing the blob of serialized bytes representing the serialized
> object.
>
>
> |I just want to make sure we are clearly separating Java object
> attributes (Primitive Java class members)
> from LDAP attributes.
>
>
> Basically my advice would be to stay clear of using Java serialization
> in the directory.
>
Diddo
>
> Let me stop here in case I'm making any sense?
>
> If not hopefully once I'm done writing the design guide and get a
> terminology and concept section going
> things will be a little clearer.
>
>
> http://bsd.cs.cofc.edu/Java/Javadoc...odifyAttributes(javax.naming.Name
> <http://bsd.cs.cofc.edu/Java/Javadoc...vax.naming.Name>
> http://bsd.cs.cofc.edu/Java/Javadoc...vax.naming.Name>,
> Cool - I'll have a looksee - I've been using the articles on the
> ApacheDS website as a reference so far, but the more material the
> better.
>
> (primitive
> I think a
> would
> createSubcontext().
>
> Ahh - This is the part where I need clarification.
>
> If we serialize an entire object to ApacheDS, then we have to get it
> from ApacheDS,
> update it, and send it back to be stored in a javaSerializedData
> attribute right?
>
>
> Right exactly! This is why Java LDAP serialization is not a good idea
> for your DAS thingy.
>
> For instance I may have a Class called UserClass with a String member
> userName.
>
> I create an instance of UserClass called userClass.
>
> The I do
> userClass.setUserName("ole");
>
> Now I want to store userClass in ADS here:
> |DN: cn=|userClass|,ou=users, ou="system
>
> So I do this:
>
> ||String bindContext = "cn=|myClassInstance|,ou=users, ou="system";|
>
> |ctx.bind( bindContext, ||userClass||);
>
> Now userClass is in ADS stored as a javaSerializedData LDAP
> attribute value,
> hanging off of the bindContext I specified.
>
> Later when I want to update
> the userName member of the UserClass instance I serialized to ADS,
> I need to use JNDI to load the instance again right?
>
>
> Yep unless you write the code to resusitate the serialized object
> manually.
>
> Then update userName. Then serialize userClass back to ADS again?
>
>
> Yep.
>
> |
>
> WDYT?
>
> I mentioned some stuff about generating an ObjectClass schema that is
> the LDAP schema of the Java Class UserClass,
> so that the primitive members (That have Java primitive types)
> would be
> stored as LDAP atttributes rather than storing the entire
> object as a single javaSerializedData attribute attached to a DN.
>
>
> Yes this would be best in my opinion.

Yeeeeeeeessss!!!!!!!!!

Cool - I think we are all on the same page now.

So my question is:

Can I use my directory context to first
create a new ObjectClass that corresponds to my
Java Class?

OK - I just asked this again in the other mail I just sent, so I'll stop
here.

>
> Regards,
> Alex
>
>
>

Cheers,
- Ole

Alex Karasulu

2007-03-29, 1:11 am

>
> OK - I just asked this again in the other mail I just sent, so I'll stop
> here.

Yep I think I just responded to that.

Alex