For an embedded DS I provide a dump method which prints the content of the j
dbm partition(s) as a form of data integrity check.

In order to implement this I think I need a reference to the bootstrapRegist
ries:

<snip>

JdbmMasterTable master = new JdbmMasterTable( recMan );
AttributeType attributeType = bootstrapRegistries.getAttributeTypeRegistry( 
 )
.lookup( "apacheUpdn" );
JdbmIndex idIndex = new JdbmIndex( attributeType,
partitionDirectory, 1000, 1000 );

NamingEnumeration list = master.listTuples(  );
StringBuffer buf = new StringBuffer(  );

while ( list.hasMore(  ) )

<snip>

This code is very similar to my old DS 1.0 code and is a match to the code i
n 1.5 server-tools:dumpcmd:DumpCommandExecutor.java

How do I get a reference to boostrapRegistries in an running, embedded direc
tory?

Doing this throws an exception:

Registries bootstrapRegistries = new DefaultRegistries( "bootstrap", 
new BootstrapSchemaLoader(), new DefaultOidRegistry());

javax.naming.NamingException: OID for name 'apacheUpdn' was not found within
 the OID registry
at org.apache.directory.server.schema.registries.DefaultOidRegistry.getOid(D
efaultOidRegistry.java:113)
at org.apache.directory.server.schema.registries.DefaultAttributeTypeRegistr
y.lookup(DefaultAttributeTypeRegistry.java:156)
<snip>


Many Thanks


Simon Temple

[ Post a follow-up to this message ]

Hi Simon,

the method org.apache.directory.server.schema.registries.DefaultOidRegistry=
.getOid()
expect to receive an OID as a parameter, not a name. You might want to
use the  SchemaConstant  constants I just committed :

AttributeType attributeType =3D
bootstrapRegistries.getAttributeTypeRegistry(  )
.lookup( SchemaConstants.APACHE_UP_DN_OID );

Not 100% sure it's enough though 


On 9/25/07, Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org
<Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org> wrote:
>
>
> For an embedded DS I provide a dump method which prints the content of th=
e
> jdbm partition(s) as a form of data integrity check.
>
> In order to implement this I think I need a reference to the
> bootstrapRegistries:
>
>             <snip>
>
>             JdbmMasterTable master =3D new JdbmMasterTable( recMan );
>             AttributeType attributeType =3D
> bootstrapRegistries.getAttributeTypeRegistry(  )
>
>  .lookup( "apacheUpdn" );
>             JdbmIndex idIndex =3D new JdbmIndex( attributeType,
>                     partitionDirectory, 1000, 1000 );
>
>             NamingEnumeration list =3D master.listTuples(  );
>             StringBuffer buf =3D new StringBuffer(  );
>
>             while ( list.hasMore(  ) )
>
>             <snip>
>
> This code is very similar to my old DS 1.0 code and is a match to the cod=
e
> in 1.5 server-tools:dumpcmd:DumpCommandExecutor.java
>
> How do I get a reference to boostrapRegistries in an running, embedded
> directory?
>
> Doing this throws an exception:
>
>                Registries bootstrapRegistries =3D new DefaultRegistries(
> "bootstrap",
>                     new BootstrapSchemaLoader(), new DefaultOidRegistry()=
);
>
> javax.naming.NamingException: OID for name 'apacheUpdn' was not found wit=
hin
> the OID registry
>  at
> org.apache.directory.server.schema.registries.DefaultOidRegistry.getOid(D=
efaultOidRegistry.java:113)
>  at
> org.apache.directory.server.schema.registries.DefaultAttributeTypeRegistr=
y.lookup(DefaultAttributeTypeRegistry.java:156)
> <snip>
>
>
> Many Thanks
>
>
> Simon Temple


--=20
Regards,
Cordialement,
Emmanuel L=E9charny
www.iktek.com

[ Post a follow-up to this message ]

Emmanuel,

Thanks but...

javax.naming.NamingException: attributeType w/ OID
1.3.6.1.4.1.18060.0.4.1.2.2 not registered!

IMHO: I think the Registries I'm working with need need to be set to
something other than the 'default'?

How do I get a reference to boostrapRegistries in an running, embedded
directory?

Thanks

Simon Temple

25 September 2007 11:35
To: "Apache Directory Developers List" <dev-aYN4UCa7k1r1N9kud6OZbmD2FQJk+8+b
@public.gmane.org>
cc:
From: "Emmanuel Lecharny" <elecharny-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
>
Subject: Re: Dumping The Partition in DS 1.5


Hi Simon,

the method
org.apache.directory.server.schema.registries.DefaultOidRegistry.getOid=
()
expect to receive an OID as a parameter, not a name. You might want to
use the  SchemaConstant  constants I just committed :

AttributeType attributeType =3D
bootstrapRegistries.getAttributeTypeRegistry(  )
.lookup( SchemaConstants.APACHE_UP_DN_OID );

Not 100% sure it's enough though 


On 9/25/07, Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org
<Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org> wrote:
>
>
> For an embedded DS I provide a dump method which prints the content o=
f
the
> jdbm partition(s) as a form of data integrity check.
>
> In order to implement this I think I need a reference to the
> bootstrapRegistries:
>
>             <snip>
>
>             JdbmMasterTable master =3D new JdbmMasterTable( recMan );=

>             AttributeType attributeType =3D
> bootstrapRegistries.getAttributeTypeRegistry(  )
>
>  .lookup( "apacheUpdn" );
>             JdbmIndex idIndex =3D new JdbmIndex( attributeType,
>                     partitionDirectory, 1000, 1000 );
>
>             NamingEnumeration list =3D master.listTuples(  );
>             StringBuffer buf =3D new StringBuffer(  );
>
>             while ( list.hasMore(  ) )
>
>             <snip>
>
> This code is very similar to my old DS 1.0 code and is a match to the=

code
> in 1.5 server-tools:dumpcmd:DumpCommandExecutor.java
>
> How do I get a reference to boostrapRegistries in an running, embedde=
d
> directory?
>
> Doing this throws an exception:
>
>                Registries bootstrapRegistries =3D new DefaultRegistri=
es(
> "bootstrap",
>                     new BootstrapSchemaLoader(), new DefaultOidRegist=
ry()
);
>
> javax.naming.NamingException: OID for name 'apacheUpdn' was not found=

within
> the OID registry
>  at
>
org.apache.directory.server.schema.registries.DefaultOidRegistry.getOid=
(DefaultOidRegistry.java:113)
>  at
>
org.apache.directory.server.schema.registries.DefaultAttributeTypeRegis=
try.lookup(DefaultAttributeTypeRegistry.java:156)
> <snip>
>
>
> Many Thanks
>
>
> Simon Temple


--
Regards,
Cordialement,
Emmanuel L=E9charny
www.iktek.com=

[ Post a follow-up to this message ]

Simon,

In 1.5 there is no longer a bootstrap registries.  Now the server bootstrap
process starts up the schema
partition and loads all the schema info from the schema partition with a
minimal set of schema data.

To help you out I need to know what your main objective is not how you're
trying to solve some part of
it.  You may just be doing it the wrong way and I might be able to suggest a
better option.

Alex

On 9/25/07, Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org <
Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org>
wrote:
>
>  For an embedded DS I provide a dump method which prints the content of
> the jdbm partition(s) as a form of data integrity check.
>
> In order to implement this I think I need a reference to the
> bootstrapRegistries:
>
>             <snip>
>
>             JdbmMasterTable master = new JdbmMasterTable( recMan );
>             AttributeType attributeType =
> bootstrapRegistries.getAttributeTypeRegistry(  )
>                                                              .lookup(
> "apacheUpdn" );
>             JdbmIndex idIndex = new JdbmIndex( attributeType,
>                     partitionDirectory, 1000, 1000 );
>
>             NamingEnumeration list = master.listTuples(  );
>             StringBuffer buf = new StringBuffer(  );
>
>             while ( list.hasMore(  ) )
>
>             <snip>
>
> This code is very similar to my old DS 1.0 code and is a match to the code
> in 1.5 server-tools:dumpcmd:DumpCommandExecutor.java
>
> How do I get a reference to boostrapRegistries in an running, embedded
> directory?
>
> Doing this throws an exception:
>
>                Registries bootstrapRegistries = new DefaultRegistries(
> "bootstrap",
>                     new BootstrapSchemaLoader(), new DefaultOidRegistry()
> );
>
> javax.naming.NamingException: OID for name 'apacheUpdn' was not found
> within the OID registry
>  at
> org.apache.directory.server.schema.registries.DefaultOidRegistry.getOid(
> DefaultOidRegistry.java:113)
>  at
> org.apache.directory.server.schema.registries.DefaultAttributeTypeRegistry
.lookup
> (DefaultAttributeTypeRegistry.java:156)
> <snip>
>
>
> Many Thanks
>
>
> Simon Temple
>

[ Post a follow-up to this message ]

Alex

When we implemented early releases of DS 1.0 we found that under certain con
ditions (e.g. uncontrolled shutdown) the jdbm store/indexes became damaged.

To enable our customer service department to determine if this was the cause
 of a reported fault, I added a method to our directory MBean to enumerate a
ll of the jdbm indexes/entries.  If it failed then we have a procedure to re
move all the jdbm store folders and restart DS with a backup LDIF.

Whereas this capability is not a runtime requirement of our systems it isan 
important operational tool as jdbm storage has proved to be a little 'fragil
e'.

So to reword my question:  How can I run an integrity check on the underlyin
g jdbm stores?


Many Thanks

- Simon

25 September 2007 17:36
To: "Apache Directory Developers List" <dev-aYN4UCa7k1r1N9kud6OZbmD2FQJk+8+b
@public.gmane.org>
cc: 
From: "Alex Karasulu" <akarasulu-1oDqGaOF3Lkdnm+yROfE0A@public.gmane.org>
Subject: Re: Dumping The Partition in DS 1.5


Simon,

In 1.5 there is no longer a bootstrap registries.  Now the server bootstrap 
process starts up the schema
partition and loads all the schema info from the schema partition with a min
imal set of schema data. 

To help you out I need to know what your main objective is not how you'retry
ing to solve some part of 
it.  You may just be doing it the wrong way and I might be able to suggest a
 better option.  

Alex 


On 9/25/07, Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org <
 Simon.Temple@saaconsultants.com> wrote:
For an embedded DS I provide a dump method which prints the content of the j
dbm partition(s) as a form of data integrity check.

In order to implement this I think I need a reference to the bootstrapRegist
ries:

<snip>

JdbmMasterTable master = new JdbmMasterTable( recMan );
AttributeType attributeType = bootstrapRegistries.getAttributeTypeRegistry( 
 )
.lookup( "apacheUpdn" );
JdbmIndex idIndex = new JdbmIndex( attributeType,
partitionDirectory, 1000, 1000 );

NamingEnumeration list = master.listTuples(  );
StringBuffer buf = new StringBuffer(  );

while ( list.hasMore(  ) )

<snip>

This code is very similar to my old DS 1.0 code and is a match to the code i
n 1.5 server-tools:dumpcmd:DumpCommandExecutor.java

How do I get a reference to boostrapRegistries in an running, embedded direc
tory?

Doing this throws an exception:

Registries bootstrapRegistries = new DefaultRegistries( "bootstrap", 
new BootstrapSchemaLoader(), new DefaultOidRegistry());

javax.naming.NamingException: OID for name 'apacheUpdn' was not found within
 the OID registry
at org.apache.directory.server.schema.registries.DefaultOidRegistry.getOid(D
efaultOidRegistry.java:113)
at org.apache.directory.server.schema.registries.DefaultAttributeTypeRegistr
y.lookup(DefaultAttributeTypeRegistry.java:156)
<snip>


Many Thanks


Simon Temple

[ Post a follow-up to this message ]

Simon,

FYI, regarding this damaged JDBM database, we have identified two issues at=
m :
- there is a race condition when stopping the server which occurs from
time to time (https://issues.apache.org/jira/browse/DIRSERVER-1047),
but I'm not 100% sure it has impact on the database
- As we are using a differed write system for entries, which default
to 15s between each flush, you may get a damaged database if the
server is brutally stopped (Kill -9, for instance)

As a consequence, we are working on a journalization mechanism which
will trace every modifications and replay them if the server is not in
a stable state when restarted. To avoid killing the server
performance, this must be a lightweight mechanism. We will implement
it through a new interceptor : changeLog
(https://issues.apache.org/jira/browse/DIRSERVER-1066)

This won't help you a lot right now, as this won't be ready in 2 days,
but I just wanted to inform you (and the dev list) about that.

Emmanuel

On 9/26/07, Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org
<Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org> wrote:
>
>
>
>
>
> Alex
>
>
>
> When we implemented early releases of DS 1.0 we found that under certain
> conditions (e.g. uncontrolled shutdown) the jdbm store/indexes became
> damaged.
>
>
>
> To enable our customer service department to determine if this was the ca=
use
> of a reported fault, I added a method to our directory MBean to enumerate
> all of the jdbm indexes/entries.  If it failed then we have a procedure t=
o
> remove all the jdbm store folders and restart DS with a backup LDIF.
>
>
>
> Whereas this capability is not a runtime requirement of our systems it is=
an
> important operational tool as jdbm storage has proved to be a little
> 'fragile'.
>
>
>
> So to reword my question:  How can I run an integrity check on the
> underlying jdbm stores?
>
>
>
>
>
> Many Thanks
>
>
>
> - Simon
>
>
> 25 September 2007 17:36
> To: "Apache Directory Developers List" <dev-aYN4UCa7k1r1N9kud6OZbmD2FQJk+8
+b@public.gmane.org>
> cc:
> From: "Alex Karasulu" <akarasulu-1oDqGaOF3Lkdnm+yROfE0A@public.gmane.org>
> Subject: Re: Dumping The Partition in DS 1.5
>
> Simon,
>
> In 1.5 there is no longer a bootstrap registries.  Now the server bootstr=
ap
> process starts up the schema
> partition and loads all the schema info from the schema partition with a
> minimal set of schema data.
>
> To help you out I need to know what your main objective is not how you're
> trying to solve some part of
> it.  You may just be doing it the wrong way and I might be able to sugges=
t a
> better option.
>
> Alex
>
>
> On 9/25/07, Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org
 <
> Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org> wrote: 
the[vbcol=seagreen]
> jdbm partition(s) as a form of data integrity check. 
> bootstrapRegistries: 
> bootstrapRegistries.getAttributeTypeRegistry(  ) 
>    .lookup( "apacheUpdn" ); 
ode[vbcol=seagreen]
> in 1.5 server-tools:dumpcmd:DumpCommandExecutor.java 
> directory? 
([vbcol=seagreen]
> "bootstrap", 
()[vbcol=seagreen]
> ); 
> within the OID registry 
> org.apache.directory.server.schema.registries.DefaultOidRegistry.getOid(D=
efaultOidRegistry.java:113) 
> org.apache.directory.server.schema.registries.DefaultAttributeTypeRegistr=
y.lookup(DefaultAttributeTypeRegistry.java:156) 
>
>


--=20
Regards,
Cordialement,
Emmanuel L=E9charny
www.iktek.com

[ Post a follow-up to this message ]

Hi Simon,

Sorry for getting to you late.  Looks like Emmanuel got to you first but let
me attempt to answer
some of your questions as well.

On 9/26/07, Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org <
Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org>
wrote:
>
>   Alex
>
>
>
> When we implemented early releases of DS 1.0 we found that under certain
> conditions (e.g. uncontrolled shutdown) the jdbm store/indexes became
> damaged.
>

Is this still the case with DS 1.5.x ?  Have you tried to reproduce?

To enable our customer service department to determine if this was the cause
> of a reported fault, I added a method to our directory MBean to enumerate
> all of the jdbm indexes/entries.  If it failed then we have a procedure to
> remove all the jdbm store folders and restart DS with a backup LDIF.
>

Ahhh I see.  I think you may be at some point interested in the change log
service we've recently started talking about
which would automatically produce this LDIF.

Whereas this capability is not a runtime requirement of our systems it is an
> important operational tool as jdbm storage has proved to be a little
> 'fragile'.
>

Oh we'd sure appreciate feedback on the problems you've found so we can make
this a bit better.

So to reword my question:  How can I run an integrity check on the
> underlying jdbm stores?
>


I'd have to have an idea of the kind of broken state you're exactly
referring to.  Perhaps if you have such a
busted partition I could look at the actual state it is left in to
understand the class of problem.

Really without having some reference point of the information you expect in
the backing store it's pretty hard.  I
need more information.

Alex

[ Post a follow-up to this message ]

Hi,

On 9/26/07, Emmanuel Lecharny <elecharny-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:[vbc
ol=seagreen]
>
> Simon,
>
> FYI, regarding this damaged JDBM database, we have identified two issues
> atm :
> - there is a race condition when stopping the server which occurs from
> time to time (https://issues.apache.org/jira/browse/DIRSERVER-1047),
> but I'm not 100% sure it has impact on the database
> - As we are using a differed write system for entries, which default
> to 15s between each flush, you may get a damaged database if the
> server is brutally stopped (Kill -9, for instance)[/vbcol]


Perhaps he can also enable sync on write for the partition.

Alex

[ Post a follow-up to this message ]