|
Home > Archive > Apache Directory Project > September 2007 > [ApacheDS] Big Bang Cleanup
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
[ApacheDS] Big Bang Cleanup
|
|
| Alex Karasulu 2007-09-28, 7:11 pm |
| Hi all,
I just want to report on some interesting conversations on IRC and IM that
I've had recently
and the clear conclusions it has led a few of us to. First let me state the
problem:
The server is tightly coupled to JNDI from the protocol provider down into
the core. The intent
to use JNDI originally came from the idea that people are used to JNDI and
so if used as the
wrapper API around the server it would ease the learning cure. Furthermore
using JNDI interfaces
to achieve this made sense since it would reduce the transformation overhead
when embedding
the server making it more efficient. Furthermore when used in conjunction
with stored procedures
JNDI would allow procedures to be written outside of the server and (in
theory) tested outside of
the server just by switching the JNDI provider from the SUN LDAP provider to
the embedded LDAP
provider in ApacheDS.
Although the idea to use JNDI to reduce the learning curve is a good one the
implementation which
coupled the server internals with JNDI details is causing us serious
problems. The JNDI provider
would have been better implemented as a wrapper around internal API's that
are more aligned with
server side LDAP centric data structures. Because they are not one must be
aware of JNDI and the
complexities of interchanging from JNDI environment variables to and from
the core data structures.
The entropy is high and results in a lot of one offs in the code.
I have always wanted to fix this problem but never had the bandwidth to do
it. It just stuck and rooted
itself as we built upon this foundation.
Years ago Trustin Lee had expressed the desire to strip out the JNDI
coupling as well but he too ran
into the same hurdles.
Enrique Rodriguez had other issues revolving around the side effects of JNDI
in the core while dealing
with an OSGi based version of the server.
A few weeks ago David Jencks expressed his dismay over having to use JNDI to
configure the server
and wondered why we do not wire the components of the server directly.
Recently Emmanuel Lecharny and I paired up to review the authentication
subsystem for some clearly needed
changes. However at some point we realized that JNDI is excessively
complicating the simple picture
that should be there. So we stopped after some point.
Later in the day today I had a conversation with Chris Custine about these
problems on IM. We reviewed
the problems and thought about what it would take to strip out JNDI while
removing these configuration
beans getting in the way of directly wiring up the server. We reached
similar conclusions. We're going
to have to bite the bullet on this one at some point or another if we intend
to build on the architecture
without loosing time and energy dealing with the trouble that this JNDI
coupling brings with it.
Chris said he's on board with just doing it. So am I. David and Emmanuel
wants to as well but we need
some confirmation from them. With the recent discussions to delay the
2.0release I think this is a great
time to just take care of this problem once and for all. Also now we have
many more hands and minds
to do this right relatively quickly.
While doing this we're going to break many things. It's not going to be
pretty. So I recommend
switching to a temporarily branch so we're not caught with broken builds on
the trunk. Then we can
immediately merge the changes back into the trunk and delete this branch.
No need to worry about
the trunk running away from us since most of us on this branch will not be
working on it and well
the branch should last at most 2-4 weeks. I'd like to get it done sooner
but I am afraid the fallout from
the changes will be very significant.
I am going to just branch now and start working on this. If yall want to
join me then let's go to town.
I'll post the SVN URLs to this tread. And of course we can have the team
review where we go before
merging back but then again most of the ApacheDS active community will be in
this branch anyway.
Thoughts?
Alex
| |
| Emmanuel Lecharny 2007-09-28, 7:11 pm |
| I'm just +1 with all you wrote.
Let's kick JNDI out of the whole picture, except as a layer for
embedding the server.
--=20
Regards,
Cordialement,
Emmanuel L=E9charny
www.iktek.com
| |
|
|
| David Jencks 2007-09-28, 7:11 pm |
| I'm definitely in favor of this and very much want to help out. I
have some other stuff I have to get done before I can spend much time
on it... hopefully I can finish that up this weekend.
So would it be appropriate to apply my xbean-spring and no-
InterceptorConfiguration changes to the new branch?
thanks
david jencks
On Sep 28, 2007, at 3:19 PM, Alex Karasulu wrote:
> Hi all,
>
> I just want to report on some interesting conversations on IRC and
> IM that I've had recently
> and the clear conclusions it has led a few of us to. First let me
> state the problem:
>
> The server is tightly coupled to JNDI from the protocol provider
> down into the core. The intent
> to use JNDI originally came from the idea that people are used to
> JNDI and so if used as the
> wrapper API around the server it would ease the learning cure.
> Furthermore using JNDI interfaces
> to achieve this made sense since it would reduce the transformation
> overhead when embedding
> the server making it more efficient. Furthermore when used in
> conjunction with stored procedures
> JNDI would allow procedures to be written outside of the server and
> (in theory) tested outside of
> the server just by switching the JNDI provider from the SUN LDAP
> provider to the embedded LDAP
> provider in ApacheDS.
>
> Although the idea to use JNDI to reduce the learning curve is a
> good one the implementation which
> coupled the server internals with JNDI details is causing us
> serious problems. The JNDI provider
> would have been better implemented as a wrapper around internal
> API's that are more aligned with
> server side LDAP centric data structures. Because they are not one
> must be aware of JNDI and the
> complexities of interchanging from JNDI environment variables to
> and from the core data structures.
> The entropy is high and results in a lot of one offs in the code.
>
> I have always wanted to fix this problem but never had the
> bandwidth to do it. It just stuck and rooted
> itself as we built upon this foundation.
>
> Years ago Trustin Lee had expressed the desire to strip out the
> JNDI coupling as well but he too ran
> into the same hurdles.
>
> Enrique Rodriguez had other issues revolving around the side
> effects of JNDI in the core while dealing
> with an OSGi based version of the server.
>
> A few weeks ago David Jencks expressed his dismay over having to
> use JNDI to configure the server
> and wondered why we do not wire the components of the server directly.
>
> Recently Emmanuel Lecharny and I paired up to review the
> authentication subsystem for some clearly needed
> changes. However at some point we realized that JNDI is
> excessively complicating the simple picture
> that should be there. So we stopped after some point.
>
> Later in the day today I had a conversation with Chris Custine
> about these problems on IM. We reviewed
> the problems and thought about what it would take to strip out JNDI
> while removing these configuration
> beans getting in the way of directly wiring up the server. We
> reached similar conclusions. We're going
> to have to bite the bullet on this one at some point or another if
> we intend to build on the architecture
> without loosing time and energy dealing with the trouble that this
> JNDI coupling brings with it.
>
> Chris said he's on board with just doing it. So am I. David and
> Emmanuel wants to as well but we need
> some confirmation from them. With the recent discussions to delay
> the 2.0 release I think this is a great
> time to just take care of this problem once and for all. Also now
> we have many more hands and minds
> to do this right relatively quickly.
>
> While doing this we're going to break many things. It's not going
> to be pretty. So I recommend
> switching to a temporarily branch so we're not caught with broken
> builds on the trunk. Then we can
> immediately merge the changes back into the trunk and delete this
> branch. No need to worry about
> the trunk running away from us since most of us on this branch will
> not be working on it and well
> the branch should last at most 2-4 weeks. I'd like to get it done
> sooner but I am afraid the fallout from
> the changes will be very significant.
>
> I am going to just branch now and start working on this. If yall
> want to join me then let's go to town.
> I'll post the SVN URLs to this tread. And of course we can have
> the team review where we go before
> merging back but then again most of the ApacheDS active community
> will be in this branch anyway.
>
> Thoughts?
>
> Alex
>
>
>
>
>
>
| |
| Alex Karasulu 2007-09-28, 7:11 pm |
| Go on buddy get jiggy with it.
Alex
On 9/28/07, David Jencks <david_jencks-/E1597aS9LQAvxtiuMwx3w@public.gmane.org> wrote:
>
> I'm definitely in favor of this and very much want to help out. I
> have some other stuff I have to get done before I can spend much time
> on it... hopefully I can finish that up this weekend.
>
> So would it be appropriate to apply my xbean-spring and no-
> InterceptorConfiguration changes to the new branch?
>
> thanks
> david jencks
>
> On Sep 28, 2007, at 3:19 PM, Alex Karasulu wrote:
>
>
>
| |
| Emmanuel Lecharny 2007-09-29, 7:11 am |
| Hi guys,
a new branch has been created where we are supposed to introduce some
major modifications and new features. That's fine. However, I would
like everyone to be aware of some traps we should not fell in :
1) We currently have a live trunk, which will be used to release a
1.5.2 release sooner or later (a bug fix release). We have to find a
way to merge changes from the trunk to the branch _or_ from the branch
to the trunk without jeopardize the trunk
2) Even if this new branch is just a branch, at some point we may ask
ourselves if this will become the new trunk for a 2.0 (ie, will we
abandon the current trunk or not)
3) Our number of committers is big : we are more than 10, and at least
5 of us are directly working on the server. If we start pouring some
new features and doing some big modifications, we will have to
document what we are doing, and to share the information
I would like to be a little bit more precise about point (3) :
- we can have big spreaded modifications, like switching from
String/byte[] to Value
- or we can have local modification (even if they are big : like XmlBeans).
the second kind of modification can be done quite easily, but the
first kind of modification should also be done in seperate branches,
otherwise the server will become a big f*** mess quickly.
Also keep in mind that we have a roadmap, so try to stick to it : a
modification can be tagged as a JIRA fix (we just have to inject some
new JIRA, probably prefixing them with a tag like [bigbang]), making
it easier to see the consequences of a modification.
The documentation is absolutely necessary, as some of us have a deep
knowledge about the server internals, but this is not a general case.
If we introduce deep modifications without documentation, then
- people who had some knowledge about the server will lose it
- and you won't be able to share your vision, leading to some
potential bad choices.
In three words : share your vision ! Confluence Is The Way, Mailing
List Is The Messenger !
I don't want to scare you all, but we must be careful. We are not
anymore in the ancient ages of ADS where 2 or 3 persons were working
in caves with stones and wood drafting the 1.0 release... Let's try to
cut this 2.0 in a way all of our committers can jump into the common
work (I'm also thinking about Studio, which will be impacted by this
effort, and about our doc writers, which will have to understand what
are the new features about without having to crawl into the code).
Last, not least, keep in mind we are targetting VSLDAP compliance for this =
2.0.
Yes, yes, I know, it's not funny... But this is a live project !
Emmanuel
--=20
Regards,
Cordialement,
Emmanuel L=E9charny
www.iktek.com
| |
| Stefan Zoerner 2007-09-29, 7:11 am |
| Emmanuel Lecharny wrote:
> Also keep in mind that we have a roadmap, so try to stick to it : a
> modification can be tagged as a JIRA fix (we just have to inject some
> new JIRA, probably prefixing them with a tag like [bigbang]), making
> it easier to see the consequences of a modification.
How about adding a page for the roadmap in Confluence DIRxPMGT space? Or
is there already one in place?
> The documentation is absolutely necessary, as some of us have a deep
> knowledge about the server internals, but this is not a general case.
> If we introduce deep modifications without documentation, then
> - people who had some knowledge about the server will lose it
> - and you won't be able to share your vision, leading to some
> potential bad choices.
I will create a version of the Basic User Guide for the new XBean
configuration style, if the Big Bang stabilized. This is perhaps more
important than the 1.5.1 update, because it will lead to better
documentation for 2.0 if released.
> Last, not least, keep in mind we are targetting VSLDAP compliance for this 2.0.
I will move VSLDAP documentation to DIRxPMGT space as well, and will
start to create an XBean configuration for running the tests.
Greetings from Hamburg,
Stefan
---8<---
Stefan Zoerner (szoerner-1oDqGaOF3Lkdnm+yROfE0A@public.gmane.org)
Committer :: PMC Member
Apache Directory Project
http://directory.apache.org
| |
| Emmanuel Lecharny 2007-09-29, 7:11 am |
| Setafn, using the DIRxPMGT for the roadmap is an excellent idea !
I'm just wondering why I didn't had it ... May be I 'm a little bit stupid =
;)
--=20
Regards,
Cordialement,
Emmanuel L=E9charny
www.iktek.com
| |
| Emmanuel Lecharny 2007-09-29, 7:11 am |
| I have created a roadmap page :
http://cwiki.apache.org/confluence/...MGT/2.0+Roadmap
Let's move what we have on the ML to this page
On 9/29/07, Emmanuel Lecharny <elecharny-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Setafn, using the DIRxPMGT for the roadmap is an excellent idea !
>
> I'm just wondering why I didn't had it ... May be I 'm a little bit stupi=
d ;)
>
>
>
> --
> Regards,
> Cordialement,
> Emmanuel L=E9charny
> www.iktek.com
>
--=20
Regards,
Cordialement,
Emmanuel L=E9charny
www.iktek.com
| |
|
|
| David Jencks 2007-09-29, 1:11 pm |
|
On Sep 29, 2007, at 3:47 AM, Stefan Zoerner wrote:
> Emmanuel Lecharny wrote:
>
>
> How about adding a page for the roadmap in Confluence DIRxPMGT
> space? Or is there already one in place?
>
>
> I will create a version of the Basic User Guide for the new XBean
> configuration style, if the Big Bang stabilized. This is perhaps
> more important than the 1.5.1 update, because it will lead to
> better documentation for 2.0 if released.
Excellent! I may have mentioned before that there is some generated
documentation for xbean-spring, see apacheds/apacheds-xbean-spring/
target/xbean/apacheds-xbean-spring.xsd.html and *.wiki. The html
looks a bit basic in my browser, and I'm not sure if the .wiki is
confluence format. Could you take a look and see if you think this
could be made appropriate to include in the documentation? For
myself I find it much easier to write and fix little bits of javadoc
than to try to write a whole page of description at once so I think
I'd have an easier time contributing to generated documentation like
this than a whole user manual. It also gets kept up to date with the
code changes pretty easily :-)
thanks!
david jencks
>
>
> I will move VSLDAP documentation to DIRxPMGT space as well, and
> will start to create an XBean configuration for running the tests.
>
> Greetings from Hamburg,
> Stefan
>
> ---8<---
>
> Stefan Zoerner (szoerner-1oDqGaOF3Lkdnm+yROfE0A@public.gmane.org)
> Committer :: PMC Member
>
> Apache Directory Project
> http://directory.apache.org
>
| |
| Stefan Zoerner 2007-09-29, 1:11 pm |
| David Jencks wrote:
> Excellent! I may have mentioned before that there is some generated
> documentation for xbean-spring, see
> apacheds/apacheds-xbean-spring/target/xbean/apacheds-xbean-spring.xsd.html
> and *.wiki. The html looks a bit basic in my browser, and I'm not sure
> if the .wiki is confluence format. Could you take a look and see if you
> think this could be made appropriate to include in the documentation?
> For myself I find it much easier to write and fix little bits of javadoc
> than to try to write a whole page of description at once so I think I'd
> have an easier time contributing to generated documentation like this
> than a whole user manual. It also gets kept up to date with the code
> changes pretty easily :-)
I have tried whether Confluence is able to render it, and it is
(although there is HTML embedded in it).
My first impression:
We will probably have some process to re-organize the content a bit, but
it looks like a very good point to start.
http://cwiki.apache.org/confluence/...xPMGT/Demo-Docs
But I will need some more time to figure it out. Thanks for the hint, David!
Greetings from Hamburg,
Stefan
---8<---
Stefan Zoerner (szoerner-1oDqGaOF3Lkdnm+yROfE0A@public.gmane.org)
Committer :: PMC Member
Apache Directory Project
http://directory.apache.org
| |
| Alex Karasulu 2007-09-29, 1:11 pm |
| Morning all,
On 9/29/07, Emmanuel Lecharny <elecharny-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> Hi guys,
>
> a new branch has been created where we are supposed to introduce some
> major modifications and new features. That's fine. However, I would
> like everyone to be aware of some traps we should not fell in :
>
> 1) We currently have a live trunk, which will be used to release a
> 1.5.2 release sooner or later (a bug fix release).
Emmanuel this is slighly incorrect perhaps in terminology according to our
release scheme and
policy which you had a hand in establishing. Let me try to explain below
with a review of what
we voted on.
Oh 1.5.2 is a feature release not a bug fix release however we can do bug
fixes in these releases
as we get feedback.
Our model is simple.
1). Start an experimental branch as the main branch of development (x.5branch)
2). Add new features and do some "feature releases" for early access to
these features i.e. x=1
- 1.5.0
- 1.5.1
- 1.5.2
...
3). Feature Freeze: freeze the addition of new features to begin to cleanup
and stabilize
4). Release next major release candidates (x+1).0.0-RCy suffix with bug
fixes only ie.:
- 2.0.0-RC1
- 2.0.0-RC2
- 2.0.0-RC3
....
5). Release final major version i.e. 2.0.0
6). Copy this branch into branches area as 2.0 branch and do "bug fix"
(ONLY) releases
7). Designate trunk as new experimental branch in this example 2.5 with next
feature release 2.5.0 etc ...
So the cycle continues. Right now a feature freeze has not been called.
But I do agree at some point
very soon we have call it. So my focus for the time being is not on bugs
although I will fix them for sure
when I can without taking too much time.
Please take my comments below as constructive criticism: I have profound
respect for you so I don't
want you to take these comments the wrong way.
Emmanuel what I find you always doing is mixing concepts. You mix features
and bug fix releases.
I feel like you're mixing some things perhaps we did not have enough
discussion. Let me give you an
example: a while back you worked on the 1.0 branch where you changed
something freely without it
being needed to fix a bug. As a result you changed the classes serialized
into the partition entries and
this made it so 1.0.1 partitions were not compatible with 1.0.2 partitions.
I'm pointing this out because
now you're doing just the opposite with experimental releases. Here we
should have the freedom to
change as much as we like so don't hold back. This is the time to be
liberal not after the final major
release in bug fix branches.
We have to stick to what we decided and be firm about it or everything false
apart. You're a very valuable
part of this project but I need your help on this or we're just going to
fall apart making policies and not adhering
to them. Basically we're going to be spinning cycles without producing
output.
We have already started doing that by not adhering to our release management
policies.
We have to find a
> way to merge changes from the trunk to the branch _or_ from the branch
> to the trunk without jeopardize the trunk
Oh certainly I agree.
2) Even if this new branch is just a branch, at some point we may ask
> ourselves if this will become the new trunk for a 2.0 (ie, will we
> abandon the current trunk or not)
We can swap out the existing trunk for this bigbang branch or just merge
back but I guess this depends on our strategy.
It might be best for some weeks to just have us make the large changes we
need for now in this common area together
the be done with this in 2-4 weeks. Then we continue back in the trunk with
feature releases until we vote to freeze the
feature additions and start releasing RCs.
3) Our number of committers is big : we are more than 10, and at least
> 5 of us are directly working on the server. If we start pouring some
> new features and doing some big modifications, we will have to
> document what we are doing, and to share the information
+1
The documentation is absolutely necessary, as some of us have a deep
> knowledge about the server internals, but this is not a general case.
> If we introduce deep modifications without documentation, then
> - people who had some knowledge about the server will lose it
> - and you won't be able to share your vision, leading to some
> potential bad choices.
+1
Alex
| |
| Emmanuel Lecharny 2007-09-29, 7:11 pm |
| I replied to alex only ...
Alex, I don't know if it's you or your mailer, but just in case, no
need to add me on the recipient, I have subscribed to dev@a.o ;)
On 9/29/07, Emmanuel Lecharny <elecharny-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hi,
our[vbcol=seagreen]
>
> yes, I'm a little bit incorrect. I meant : bug fix + new features.
>
ures[vbcol=seagreen]
zed[vbcol=seagreen]
ons.[vbcol=seagreen]
>
> Don't worry. I sometime confusing a few things, due to some language gap,=
too.
>
h[vbcol=seagreen]
>
> I think it's what I meant with my previous sentence, even if it was not c=
lear.
>
> <snip/>
>
>
>
> --
> Regards,
> Cordialement,
> Emmanuel L=E9charny
> www.iktek.com
>
--=20
Regards,
Cordialement,
Emmanuel L=E9charny
www.iktek.com
| |
| Alex Karasulu 2007-09-29, 7:11 pm |
| Ooops np.
On 9/29/07, Emmanuel Lecharny <elecharny-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> I replied to alex only ...
>
> Alex, I don't know if it's you or your mailer, but just in case, no
> need to add me on the recipient, I have subscribed to dev@a.o ;)
>
> On 9/29/07, Emmanuel Lecharny <elecharny-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> our
> features
> serialized
> we
> gap, too.
> branch
> clear.
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
|
|
|
|
|