|
Home > Archive > Apache Directory Project > September 2007 > Dumping The Partition in DS 1.5
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Dumping The Partition in DS 1.5
|
|
| Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@ 2007-09-25, 7:11 am |
| For an embedded DS I provide a dump method which prints the content of the jdbm partition(s) as a form of data integrity check.
In order to implement this I think I need a reference to the bootstrapRegistries:
<snip>
JdbmMasterTable master = new JdbmMasterTable( recMan );
AttributeType attributeType = bootstrapRegistries.getAttributeTypeRegistry( )
.lookup( "apacheUpdn" );
JdbmIndex idIndex = new JdbmIndex( attributeType,
partitionDirectory, 1000, 1000 );
NamingEnumeration list = master.listTuples( );
StringBuffer buf = new StringBuffer( );
while ( list.hasMore( ) )
<snip>
This code is very similar to my old DS 1.0 code and is a match to the code in 1.5 server-tools:dumpcmd:DumpCommandExecutor.java
How do I get a reference to boostrapRegistries in an running, embedded directory?
Doing this throws an exception:
Registries bootstrapRegistries = new DefaultRegistries( "bootstrap",
new BootstrapSchemaLoader(), new DefaultOidRegistry());
javax.naming.NamingException: OID for name 'apacheUpdn' was not found within the OID registry
at org.apache.directory.server.schema.registries.DefaultOidRegistry.getOid(DefaultOidRegistry.java:113)
at org.apache.directory.server.schema.registries.DefaultAttributeTypeRegistry.lookup(DefaultAttributeTypeRegistry.java:156)
<snip>
Many Thanks
Simon Temple
| |
| Emmanuel Lecharny 2007-09-25, 7:11 am |
| Hi Simon,
the method org.apache.directory.server.schema.registries.DefaultOidRegistry=
..getOid()
expect to receive an OID as a parameter, not a name. You might want to
use the SchemaConstant constants I just committed :
AttributeType attributeType =3D
bootstrapRegistries.getAttributeTypeRegistry( )
.lookup( SchemaConstants.APACHE_UP_DN_OID );
Not 100% sure it's enough though 
On 9/25/07, Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org
<Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org> wrote:
>
>
> For an embedded DS I provide a dump method which prints the content of th=
e
> jdbm partition(s) as a form of data integrity check.
>
> In order to implement this I think I need a reference to the
> bootstrapRegistries:
>
> <snip>
>
> JdbmMasterTable master =3D new JdbmMasterTable( recMan );
> AttributeType attributeType =3D
> bootstrapRegistries.getAttributeTypeRegistry( )
>
> .lookup( "apacheUpdn" );
> JdbmIndex idIndex =3D new JdbmIndex( attributeType,
> partitionDirectory, 1000, 1000 );
>
> NamingEnumeration list =3D master.listTuples( );
> StringBuffer buf =3D new StringBuffer( );
>
> while ( list.hasMore( ) )
>
> <snip>
>
> This code is very similar to my old DS 1.0 code and is a match to the cod=
e
> in 1.5 server-tools:dumpcmd:DumpCommandExecutor.java
>
> How do I get a reference to boostrapRegistries in an running, embedded
> directory?
>
> Doing this throws an exception:
>
> Registries bootstrapRegistries =3D new DefaultRegistries(
> "bootstrap",
> new BootstrapSchemaLoader(), new DefaultOidRegistry()=
);
>
> javax.naming.NamingException: OID for name 'apacheUpdn' was not found wit=
hin
> the OID registry
> at
> org.apache.directory.server.schema.registries.DefaultOidRegistry.getOid(D=
efaultOidRegistry.java:113)
> at
> org.apache.directory.server.schema.registries.DefaultAttributeTypeRegistr=
y.lookup(DefaultAttributeTypeRegistry.java:156)
> <snip>
>
>
> Many Thanks
>
>
> Simon Temple
--=20
Regards,
Cordialement,
Emmanuel L=E9charny
www.iktek.com
| |
| Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@ 2007-09-25, 1:11 pm |
|
Emmanuel,
Thanks but...
javax.naming.NamingException: attributeType w/ OID
1.3.6.1.4.1.18060.0.4.1.2.2 not registered!
IMHO: I think the Registries I'm working with need need to be set to
something other than the 'default'?
How do I get a reference to boostrapRegistries in an running, embedded
directory?
Thanks
Simon Temple
25 September 2007 11:35
To: "Apache Directory Developers List" <dev-aYN4UCa7k1r1N9kud6OZbmD2FQJk+8+b@public.gmane.org>
cc:
From: "Emmanuel Lecharny" <elecharny-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Subject: Re: Dumping The Partition in DS 1.5
Hi Simon,
the method
org.apache.directory.server.schema.registries.DefaultOidRegistry.getOid=
()
expect to receive an OID as a parameter, not a name. You might want to
use the SchemaConstant constants I just committed :
AttributeType attributeType =3D
bootstrapRegistries.getAttributeTypeRegistry( )
.lookup( SchemaConstants.APACHE_UP_DN_OID );
Not 100% sure it's enough though
On 9/25/07, Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org
<Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org> wrote:
>
>
> For an embedded DS I provide a dump method which prints the content o=
f
the
> jdbm partition(s) as a form of data integrity check.
>
> In order to implement this I think I need a reference to the
> bootstrapRegistries:
>
> <snip>
>
> JdbmMasterTable master =3D new JdbmMasterTable( recMan );=
> AttributeType attributeType =3D
> bootstrapRegistries.getAttributeTypeRegistry( )
>
> .lookup( "apacheUpdn" );
> JdbmIndex idIndex =3D new JdbmIndex( attributeType,
> partitionDirectory, 1000, 1000 );
>
> NamingEnumeration list =3D master.listTuples( );
> StringBuffer buf =3D new StringBuffer( );
>
> while ( list.hasMore( ) )
>
> <snip>
>
> This code is very similar to my old DS 1.0 code and is a match to the=
code
> in 1.5 server-tools:dumpcmd:DumpCommandExecutor.java
>
> How do I get a reference to boostrapRegistries in an running, embedde=
d
> directory?
>
> Doing this throws an exception:
>
> Registries bootstrapRegistries =3D new DefaultRegistri=
es(
> "bootstrap",
> new BootstrapSchemaLoader(), new DefaultOidRegist=
ry()
);
>
> javax.naming.NamingException: OID for name 'apacheUpdn' was not found=
within
> the OID registry
> at
>
org.apache.directory.server.schema.registries.DefaultOidRegistry.getOid=
(DefaultOidRegistry.java:113)
> at
>
org.apache.directory.server.schema.registries.DefaultAttributeTypeRegis=
try.lookup(DefaultAttributeTypeRegistry.java:156)
> <snip>
>
>
> Many Thanks
>
>
> Simon Temple
--
Regards,
Cordialement,
Emmanuel L=E9charny
www.iktek.com=
| |
| Alex Karasulu 2007-09-25, 1:11 pm |
| Simon,
In 1.5 there is no longer a bootstrap registries. Now the server bootstrap
process starts up the schema
partition and loads all the schema info from the schema partition with a
minimal set of schema data.
To help you out I need to know what your main objective is not how you're
trying to solve some part of
it. You may just be doing it the wrong way and I might be able to suggest a
better option.
Alex
On 9/25/07, Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org <Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org>
wrote:
>
> For an embedded DS I provide a dump method which prints the content of
> the jdbm partition(s) as a form of data integrity check.
>
> In order to implement this I think I need a reference to the
> bootstrapRegistries:
>
> <snip>
>
> JdbmMasterTable master = new JdbmMasterTable( recMan );
> AttributeType attributeType =
> bootstrapRegistries.getAttributeTypeRegistry( )
> .lookup(
> "apacheUpdn" );
> JdbmIndex idIndex = new JdbmIndex( attributeType,
> partitionDirectory, 1000, 1000 );
>
> NamingEnumeration list = master.listTuples( );
> StringBuffer buf = new StringBuffer( );
>
> while ( list.hasMore( ) )
>
> <snip>
>
> This code is very similar to my old DS 1.0 code and is a match to the code
> in 1.5 server-tools:dumpcmd:DumpCommandExecutor.java
>
> How do I get a reference to boostrapRegistries in an running, embedded
> directory?
>
> Doing this throws an exception:
>
> Registries bootstrapRegistries = new DefaultRegistries(
> "bootstrap",
> new BootstrapSchemaLoader(), new DefaultOidRegistry()
> );
>
> javax.naming.NamingException: OID for name 'apacheUpdn' was not found
> within the OID registry
> at
> org.apache.directory.server.schema.registries.DefaultOidRegistry.getOid(
> DefaultOidRegistry.java:113)
> at
> org.apache.directory.server.schema.registries.DefaultAttributeTypeRegistry.lookup
> (DefaultAttributeTypeRegistry.java:156)
> <snip>
>
>
> Many Thanks
>
>
> Simon Temple
>
| |
| Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@ 2007-09-26, 7:11 am |
| Alex
When we implemented early releases of DS 1.0 we found that under certain conditions (e.g. uncontrolled shutdown) the jdbm store/indexes became damaged.
To enable our customer service department to determine if this was the cause of a reported fault, I added a method to our directory MBean to enumerate all of the jdbm indexes/entries. If it failed then we have a procedure to remove all the jdbm store folders and restart DS with a backup LDIF.
Whereas this capability is not a runtime requirement of our systems it isan important operational tool as jdbm storage has proved to be a little 'fragile'.
So to reword my question: How can I run an integrity check on the underlying jdbm stores?
Many Thanks
- Simon
25 September 2007 17:36
To: "Apache Directory Developers List" <dev-aYN4UCa7k1r1N9kud6OZbmD2FQJk+8+b@public.gmane.org>
cc:
From: "Alex Karasulu" <akarasulu-1oDqGaOF3Lkdnm+yROfE0A@public.gmane.org>
Subject: Re: Dumping The Partition in DS 1.5
Simon,
In 1.5 there is no longer a bootstrap registries. Now the server bootstrap process starts up the schema
partition and loads all the schema info from the schema partition with a minimal set of schema data.
To help you out I need to know what your main objective is not how you'retrying to solve some part of
it. You may just be doing it the wrong way and I might be able to suggest a better option.
Alex
On 9/25/07, Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org < Simon.Temple@saaconsultants.com> wrote:
For an embedded DS I provide a dump method which prints the content of the jdbm partition(s) as a form of data integrity check.
In order to implement this I think I need a reference to the bootstrapRegistries:
<snip>
JdbmMasterTable master = new JdbmMasterTable( recMan );
AttributeType attributeType = bootstrapRegistries.getAttributeTypeRegistry( )
.lookup( "apacheUpdn" );
JdbmIndex idIndex = new JdbmIndex( attributeType,
partitionDirectory, 1000, 1000 );
NamingEnumeration list = master.listTuples( );
StringBuffer buf = new StringBuffer( );
while ( list.hasMore( ) )
<snip>
This code is very similar to my old DS 1.0 code and is a match to the code in 1.5 server-tools:dumpcmd:DumpCommandExecutor.java
How do I get a reference to boostrapRegistries in an running, embedded directory?
Doing this throws an exception:
Registries bootstrapRegistries = new DefaultRegistries( "bootstrap",
new BootstrapSchemaLoader(), new DefaultOidRegistry());
javax.naming.NamingException: OID for name 'apacheUpdn' was not found within the OID registry
at org.apache.directory.server.schema.registries.DefaultOidRegistry.getOid(DefaultOidRegistry.java:113)
at org.apache.directory.server.schema.registries.DefaultAttributeTypeRegistry.lookup(DefaultAttributeTypeRegistry.java:156)
<snip>
Many Thanks
Simon Temple
| |
| Emmanuel Lecharny 2007-09-26, 7:11 am |
| Simon,
FYI, regarding this damaged JDBM database, we have identified two issues at=
m :
- there is a race condition when stopping the server which occurs from
time to time (https://issues.apache.org/jira/browse/DIRSERVER-1047),
but I'm not 100% sure it has impact on the database
- As we are using a differed write system for entries, which default
to 15s between each flush, you may get a damaged database if the
server is brutally stopped (Kill -9, for instance)
As a consequence, we are working on a journalization mechanism which
will trace every modifications and replay them if the server is not in
a stable state when restarted. To avoid killing the server
performance, this must be a lightweight mechanism. We will implement
it through a new interceptor : changeLog
(https://issues.apache.org/jira/browse/DIRSERVER-1066)
This won't help you a lot right now, as this won't be ready in 2 days,
but I just wanted to inform you (and the dev list) about that.
Emmanuel
On 9/26/07, Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org
<Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org> wrote:
>
>
>
>
>
> Alex
>
>
>
> When we implemented early releases of DS 1.0 we found that under certain
> conditions (e.g. uncontrolled shutdown) the jdbm store/indexes became
> damaged.
>
>
>
> To enable our customer service department to determine if this was the ca=
use
> of a reported fault, I added a method to our directory MBean to enumerate
> all of the jdbm indexes/entries. If it failed then we have a procedure t=
o
> remove all the jdbm store folders and restart DS with a backup LDIF.
>
>
>
> Whereas this capability is not a runtime requirement of our systems it is=
an
> important operational tool as jdbm storage has proved to be a little
> 'fragile'.
>
>
>
> So to reword my question: How can I run an integrity check on the
> underlying jdbm stores?
>
>
>
>
>
> Many Thanks
>
>
>
> - Simon
>
>
> 25 September 2007 17:36
> To: "Apache Directory Developers List" <dev-aYN4UCa7k1r1N9kud6OZbmD2FQJk+8+b@public.gmane.org>
> cc:
> From: "Alex Karasulu" <akarasulu-1oDqGaOF3Lkdnm+yROfE0A@public.gmane.org>
> Subject: Re: Dumping The Partition in DS 1.5
>
> Simon,
>
> In 1.5 there is no longer a bootstrap registries. Now the server bootstr=
ap
> process starts up the schema
> partition and loads all the schema info from the schema partition with a
> minimal set of schema data.
>
> To help you out I need to know what your main objective is not how you're
> trying to solve some part of
> it. You may just be doing it the wrong way and I might be able to sugges=
t a
> better option.
>
> Alex
>
>
> On 9/25/07, Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org <
> Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org> wrote:
the[vbcol=seagreen]
> jdbm partition(s) as a form of data integrity check.
> bootstrapRegistries:
> bootstrapRegistries.getAttributeTypeRegistry( )
> .lookup( "apacheUpdn" );
ode[vbcol=seagreen]
> in 1.5 server-tools:dumpcmd:DumpCommandExecutor.java
> directory?
([vbcol=seagreen]
> "bootstrap",
()[vbcol=seagreen]
> );
> within the OID registry
> org.apache.directory.server.schema.registries.DefaultOidRegistry.getOid(D=
efaultOidRegistry.java:113)
> org.apache.directory.server.schema.registries.DefaultAttributeTypeRegistr=
y.lookup(DefaultAttributeTypeRegistry.java:156)
>
>
--=20
Regards,
Cordialement,
Emmanuel L=E9charny
www.iktek.com
| |
| Alex Karasulu 2007-09-27, 1:11 am |
| Hi Simon,
Sorry for getting to you late. Looks like Emmanuel got to you first but let
me attempt to answer
some of your questions as well.
On 9/26/07, Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org <Simon.Temple-bNHniqfy2KDk7dXyY5I9mNBPR1lH4CV8@public.gmane.org>
wrote:
>
> Alex
>
>
>
> When we implemented early releases of DS 1.0 we found that under certain
> conditions (e.g. uncontrolled shutdown) the jdbm store/indexes became
> damaged.
>
Is this still the case with DS 1.5.x ? Have you tried to reproduce?
To enable our customer service department to determine if this was the cause
> of a reported fault, I added a method to our directory MBean to enumerate
> all of the jdbm indexes/entries. If it failed then we have a procedure to
> remove all the jdbm store folders and restart DS with a backup LDIF.
>
Ahhh I see. I think you may be at some point interested in the change log
service we've recently started talking about
which would automatically produce this LDIF.
Whereas this capability is not a runtime requirement of our systems it is an
> important operational tool as jdbm storage has proved to be a little
> 'fragile'.
>
Oh we'd sure appreciate feedback on the problems you've found so we can make
this a bit better.
So to reword my question: How can I run an integrity check on the
> underlying jdbm stores?
>
I'd have to have an idea of the kind of broken state you're exactly
referring to. Perhaps if you have such a
busted partition I could look at the actual state it is left in to
understand the class of problem.
Really without having some reference point of the information you expect in
the backing store it's pretty hard. I
need more information.
Alex
| |
| Alex Karasulu 2007-09-27, 1:11 am |
| Hi,
On 9/26/07, Emmanuel Lecharny <elecharny-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> Simon,
>
> FYI, regarding this damaged JDBM database, we have identified two issues
> atm :
> - there is a race condition when stopping the server which occurs from
> time to time (https://issues.apache.org/jira/browse/DIRSERVER-1047),
> but I'm not 100% sure it has impact on the database
> - As we are using a differed write system for entries, which default
> to 15s between each flush, you may get a damaged database if the
> server is brutally stopped (Kill -9, for instance)
Perhaps he can also enable sync on write for the partition.
Alex
|
|
|
|
|