Apache JDO Project - TestRunner remarks

Author

TestRunner remarks

Michael Bouschen

2005-06-20, 5:45 pm

Hi Michelle,

I read the TestRunner description on the wiki and it really looks good!

Some remarks: instead of using configuration files for the list of
databases and the identity types to run the JDO TCK on, we could use
properties in the project.properties file, e.g.
jdo.tck.dblist=derby oracle mysql
jdo.tck.identitytypes=applicationidentity datastoreidentity
I'm not sure about the property names. We already have an internal
property called jdo.tck.identitytype which denotes the identity type
used for the current test. So the plural version jdo.tck.identitytypes
would specify the list of identity types to run the tck on (better name?).

The user can overwrite the default settings as specified in
project.properties by using the -D flag when calling maven. E.g. 'maven
-Djdo.tck.dblist=derby runtck' would run the tck on derby only.

I try to give answers to the questions you added to the TestRunner
description on the wiki:
(1) I think we do not need separate properties for dblist and identity
types for the iut and jdori.
(2) Yes, I think we assume that the user will overwrite setting using -D
option. I think this becomes less of an issue if we skip the
configuration files db.list and identity.list as proposed above.
(3) It looks like we need separate properties for specifying what
identity types the iut supports (iut.applicationidentity.supported and
iut.datastoreidentity.supported) and for specifying what identity types
the user want to run the tck on (jdo.tck.identitytypes).

We talked about that the test case might need to know whether we are
running with application identity or datastore identity. I propose to
pass the value of jdo.tck.identitytype as a system property. JDO_Test
could read the system property and provide an access method. I have
prototyped this, you find a patch attached.
- maven.xml: pass the value of the property jdo.tck.identitytype as a
system property.
- JDO_Test: reads the system property and provides a boolean method
runsWithApplicationIdentity. I'm not sure about the method name. I did
not choose isApplicationIdentity, since this could be confused with
isApplicationIdentitySupported which returns true if the iut supports
application identity.

Regards Michael

--
Michael Bouschen Tech@Spree Engineering GmbH
mailto:mbo.tech@spree.de http://www.tech.spree.de/
Tel.:++49/30/235 520-33 Buelowstr. 66
Fax.:++49/30/2175 2012 D-10783 Berlin

Michelle Caisse

2005-06-22, 5:45 pm

Michael,

Thanks for trying out TestRunner! My responses below...

Michael Watzek wrote:

> Hi Michelle,
>
> I ran your patch on a clean workspace.
>
> I started with "maven installSchema". This installs two schemas in
> derby - one for application identity and one for application identity.
> I really appreciate that we have separate schemas for application
> identity and datastore identity. Due to this fact we only need to call
> this maven goal once as each test cleans up the database - great!
>
> Afterwards I ran "maven runtck.jdori". This runs 2 configurations -
> one for the completeness and one for all test cases. As a result, 4
> log files are written to target/logs - 2 for application identity, 2
> for datastore identity. The all test TCK runs produced the expected
> amount of errors/failures - though, they did not take as much time as
> expected. Both runs perform about 7-8 times faster on my system -
> that's great :-)! What have you changed?

Wow, that IS great! It seemed a lot faster on my system too, but I
wasn't paying much attention to performance. I have no idea what the
difference is :-\
I will try to investigate before I check in.

>
> Some questions:
>
> - You renamed maven property "jdo.tck.identitytype" to "jdo.tck.id".
> Is this a good name? To me, it sounds like the ID for the TCK.
>
> - Class JDO_Test still checks for property "jdo.tck.identitytype" in
> the constructor.

You're right; brevity is not necessarily good. I will change it back to
jdo.tck.identitytype and it will then match the name in JDO_Test.

>
> - Can we include the configuration name and the identity type in the
> log file name? Currently, it's hard to find the right file if you are
> interested in only one file.

This is a good idea. I definitely would like to get more information
into the log file or file name. Can you tell me where the log file name
is created?

Thanks again,
Michelle
[vbcol=seagreen]
>
> Regards,
> Michael
>

Michael Watzek

2005-06-23, 5:45 pm

Hi Michelle,

>
>
> Wow, that IS great! It seemed a lot faster on my system too, but I
> wasn't paying much attention to performance. I have no idea what the
> difference is :-\ I will try to investigate before I check in.
I'm also curious about the performance boost, so I spent some time in
investigating - without success:

I only get the performance boost with your patch. A clean workspace
without your patch performs 7-8 times slower. I applied two changes of
your patch to a clean workspace just to see if this makes a difference:

jdori.properties:
-org.jpox.validateTables=false # default = true
+org.jpox.validateTables=false

logging.properties:
-log4j.category.JPOX.General=INFO, A1
+log4j.category.JPOX.General=DEBUG, A1

Both changes do not give better performance. So, I ran out of ideas.
Hopefully, you have more luck in investigation.

>
>
> This is a good idea. I definitely would like to get more information
> into the log file or file name. Can you tell me where the log file name
> is created?
Class "BatchTestRunner" has a static inner class "ConsoleFileOutput"
defining a static String "fileNamePrefix":

private static String fileNamePrefix = "TCKLog-";

I suggest to append the value of a system property which contains the
configuration name and the identity type.

Regards,
Michael
--
-------------------------------------------------------------------
Michael Watzek Tech@Spree Engineering GmbH
mailto:mwa.tech@spree.de Buelowstr. 66
Tel.: ++49/30/235 520 36 10783 Berlin - Germany
Fax.: ++49/30/217 520 12 http://www.spree.de/
-------------------------------------------------------------------

Craig Russell

2005-06-23, 5:45 pm

Michelle Caisse

2005-06-23, 5:45 pm

I think that would be hard, since there is on invocation of
BatchTestRunner, necessarily, per configuration.

-- Michelle

Craig Russell wrote:

> Hi,
>
> I hadn't thought of the multiple log file issue. For what it's worth,
> I'd prefer one log file for each configuration (app/datastore id;
> database) and not one log file for completeness/all test cases. We
> have in mind that there would be many completeness/all test cases
> configs, so it will be difficult to sort them all out.
>
> Craig
>
> On Jun 23, 2005, at 8:55 AM, Michael Watzek wrote:
>
>
> Craig Russell
>
> Architect, Sun Java Enterprise System http://java.sun.com/products/jdo
>
> 408 276-5638 mailto:Craig.Russell@sun.com
>
> P.S. A good JDO? O, Gasp!
>
>

Michael Watzek

2005-06-23, 5:45 pm

Hi Craig,

> Hi,
>
> I hadn't thought of the multiple log file issue. For what it's worth,
> I'd prefer one log file for each configuration (app/datastore id;
> database) and not one log file for completeness/all test cases. We have
> in mind that there would be many completeness/all test cases configs, so
> it will be difficult to sort them all out.
A new log file is created for each TCK run. Each configuration starts a
new TCK run. Additionally, each identity type and each database starts a
new TCK run. So, if you run "runtck.jdori" currently, then you get 4 log
files:

- Completeness test for application identity on derby
- Completeness test for datastore identity on derby
- All tests for application identity on derby
- All tests test for datastore identity on derby

Regards,
Michael
>
> Craig
>
> On Jun 23, 2005, at 8:55 AM, Michael Watzek wrote:
>
>
> Craig Russell
>
> Architect, Sun Java Enterprise System http://java.sun.com/products/jdo
>
> 408 276-5638 mailto:Craig.Russell@sun.com
>
> P.S. A good JDO? O, Gasp!
>
>

--
-------------------------------------------------------------------
Michael Watzek Tech@Spree Engineering GmbH
mailto:mwa.tech@spree.de Buelowstr. 66
Tel.: ++49/30/235 520 36 10783 Berlin - Germany
Fax.: ++49/30/217 520 12 http://www.spree.de/
-------------------------------------------------------------------

Michelle Caisse

2005-06-23, 5:45 pm

"one" invocation...
Michelle Caisse wrote:

> I think that would be hard, since there is on invocation of
> BatchTestRunner, necessarily, per configuration.
>
> -- Michelle
>
> Craig Russell wrote:
>
>
>

Craig Russell

2005-06-23, 5:45 pm

Michelle Caisse

2005-06-23, 5:45 pm

Hi Michael,

The performance boost is associated with using the jpox property
org.jpox.identifier.defaultSchemaName to specify the schema in which the
tables are installed. There is a performance boost whether the tables
are installed in a named schema or the default schema when this property
is set. I see an improvement of about 5.8x.

I've attached two patches to demonstrate the changes, in case you're
interested. performancediff.patch reverses the preformance boost when
applied to the latest checkout from svn by installing into the default
schema and not setting the defaultSchemaName option.
performancediff3.patch recovers the performance boost by simply setting
defaultSchemaName to tckuser, the user name and default schema name.
(Revert maven.xml first if you apply them in sequence.)

-- Michelle

Michael Watzek wrote:

> Hi Michelle,
>
>
> I'm also curious about the performance boost, so I spent some time in
> investigating - without success:
>
> I only get the performance boost with your patch. A clean workspace
> without your patch performs 7-8 times slower. I applied two changes of
> your patch to a clean workspace just to see if this makes a difference:
>
> jdori.properties:
> -org.jpox.validateTables=false # default = true
> +org.jpox.validateTables=false
>
> logging.properties:
> -log4j.category.JPOX.General=INFO, A1
> +log4j.category.JPOX.General=DEBUG, A1
>
> Both changes do not give better performance. So, I ran out of ideas.
> Hopefully, you have more luck in investigation.
>
>
> Class "BatchTestRunner" has a static inner class "ConsoleFileOutput"
> defining a static String "fileNamePrefix":
>
> private static String fileNamePrefix = "TCKLog-";
>
> I suggest to append the value of a system property which contains the
> configuration name and the identity type.
>
> Regards,
> Michael

Michael Watzek

2005-06-24, 7:45 am

Hi Michelle,

> The performance boost is associated with using the jpox property
> org.jpox.identifier.defaultSchemaName to specify the schema in which the
> tables are installed. There is a performance boost whether the tables
> are installed in a named schema or the default schema when this property
> is set. I see an improvement of about 5.8x.
I see. The table name qualification of JPOX generated SQL statements
depends on this property. This means, the performance of Derby depends
on table name qualification.

This is curious, because Derby qualifies table names with the default
schema name in case of unqualified table names. The default schema name
equals the user name. The user name is part of the connection URL. For
this reason, all informations for Derby implicit qualification are part
of the connection - no need for a costly default schema lookup by Derby.
I looked into the tuning document of Derby, but could not find any note
about this fact.

However, I have a question:

The property "org.jpox.identifier.defaultSchemaName" is passed as system
property. This is good because the default schema depends on the
identity type. However, there are a number of other JPOX properties
which are passed as java.util.Properties. Should we document somewhere
that the default schema name property is passed as system property?

Regards,
Michael

>
> I've attached two patches to demonstrate the changes, in case you're
> interested. performancediff.patch reverses the preformance boost when
> applied to the latest checkout from svn by installing into the default
> schema and not setting the defaultSchemaName option.
> performancediff3.patch recovers the performance boost by simply setting
> defaultSchemaName to tckuser, the user name and default schema name.
> (Revert maven.xml first if you apply them in sequence.)
>
> -- Michelle
>
> Michael Watzek wrote:
>
>
>
>
>
> ------------------------------------------------------------------------
>
> Index: test/sql/derby/datastoreidentity/schema.sql
> ========================================
===========================
> --- test/sql/derby/datastoreidentity/schema.sql (revision 198769)
> +++ test/sql/derby/datastoreidentity/schema.sql (working copy)
> @@ -2,8 +2,8 @@
>
> connect 'jdbc:derby:jdotckdb;create=true' user 'tckuser' password 'tckuser';
>
> -CREATE SCHEMA datastoreidentity0;
> -SET SCHEMA datastoreidentity0;
> +--CREATE SCHEMA datastoreidentity0;
> +--SET SCHEMA datastoreidentity0;
>
> -------------------------
> -- mylib
> Index: test/sql/derby/applicationidentity/schema.sql
> ========================================
===========================
> --- test/sql/derby/applicationidentity/schema.sql (revision 198769)
> +++ test/sql/derby/applicationidentity/schema.sql (working copy)
> @@ -2,8 +2,8 @@
>
> connect 'jdbc:derby:jdotckdb;create=true' user 'tckuser' password 'tckuser';
>
> -CREATE SCHEMA applicationidentity0;
> -SET SCHEMA applicationidentity0;
> +--CREATE SCHEMA applicationidentity0;
> +--SET SCHEMA applicationidentity0;
>
> -------------------------
> -- mylib
> Index: maven.xml
> ========================================
===========================
> --- maven.xml (revision 198769)
> +++ maven.xml (working copy)
> @@ -269,8 +269,10 @@
> value="${jdo.tck.excludelist}"/>
> <sysproperty key="javax.jdo.option.Mapping"
> value="${jdo.tck.Mapping}"/>
> +
> <jvmarg line="${database.runtck.sysproperties}"/>
> <jvmarg line="${jdo.runtck.sysproperties}"/>
> <arg line="${jdo.tck.classes}"/>
>
>
> ------------------------------------------------------------------------
>
> Index: test/sql/derby/datastoreidentity/schema.sql
> ========================================
===========================
> --- test/sql/derby/datastoreidentity/schema.sql (revision 198769)
> +++ test/sql/derby/datastoreidentity/schema.sql (working copy)
> @@ -2,8 +2,8 @@
>
> connect 'jdbc:derby:jdotckdb;create=true' user 'tckuser' password 'tckuser';
>
> -CREATE SCHEMA datastoreidentity0;
> -SET SCHEMA datastoreidentity0;
> +--CREATE SCHEMA datastoreidentity0;
> +--SET SCHEMA datastoreidentity0;
>
> -------------------------
> -- mylib
> Index: test/sql/derby/applicationidentity/schema.sql
> ========================================
===========================
> --- test/sql/derby/applicationidentity/schema.sql (revision 198769)
> +++ test/sql/derby/applicationidentity/schema.sql (working copy)
> @@ -2,8 +2,8 @@
>
> connect 'jdbc:derby:jdotckdb;create=true' user 'tckuser' password 'tckuser';
>
> -CREATE SCHEMA applicationidentity0;
> -SET SCHEMA applicationidentity0;
> +--CREATE SCHEMA applicationidentity0;
> +--SET SCHEMA applicationidentity0;
>
> -------------------------
> -- mylib
> Index: maven.xml
> ========================================
===========================
> --- maven.xml (revision 198769)
> +++ maven.xml (working copy)
> @@ -192,6 +192,8 @@
> <j:expr value="${schemaname.concat(id)}"/>
> <j:expr value="${schemaname.concat(mapping)}"/>
> </j:set>
> + <j:set var="schemaname" value="tckuser"/>
> + <echo> Schemaname is ${schemaname} </echo>
> <attainGoal name="doRuntck.jdori"/>
> </j:forEach>
> </j:forEach>

--
-------------------------------------------------------------------
Michael Watzek Tech@Spree Engineering GmbH
mailto:mwa.tech@spree.de Buelowstr. 66
Tel.: ++49/30/235 520 36 10783 Berlin - Germany
Fax.: ++49/30/217 520 12 http://www.spree.de/
-------------------------------------------------------------------

Andy Jefferson

2005-06-24, 7:45 am

> However, I have a question:
>
> The property "org.jpox.identifier.defaultSchemaName" is passed as system
> property. This is good because the default schema depends on the
> identity type. However, there are a number of other JPOX properties
> which are passed as java.util.Properties. Should we document somewhere
> that the default schema name property is passed as system property?

Three comments :-

1. You should be using "javax.jdo.mapping.Schema" (and
javax.jdo.mapping.Catalog) and not this JPOX-specific property (as I
mentioned on the JPOX Forum thread about it that Michelle raised). Michelle,
the reason you were not seeing any effect before was *because* you were
passing this in as a system property and not using the JDO method of PMF
property. I added that as an accepted System property yesterday in JPOX CVS.

2. I don't remember seeing anything in the JDO spec about accepting PMF props
as System props (ok, JPOX supports it for many properties, but that is an
extension IIRC and you can't rely on it for other impls). Correct me if I'm
wrong (with a reference to the section of the JDO2 spec that defines it).

3. As I also mentioned on the JPOX Forum thread, your schema name should be in
UPPERCASE if you want schema validation to work (Derby accepts UPPERCASE
identifiers - it only accepts mixed case if they are quoted, and we don't
support validation of mixed case quoted at the moment either).

--
Andy
Java Persistent Objects - JPOX

Craig Russell

2005-06-25, 5:45 pm

Craig Russell

2005-06-25, 5:45 pm