[jdom-interest] JDOM2 and Performance.

Fri Oct 14 05:38:54 PDT 2011

just got off the train, and I've bumped up the inner iterations to 12, and
it makes no difference w/r/t the timings, but, while looking in to things I
have identified the changes to the XMLOutputter (
https://github.com/hunterhacker/jdom/commit/bf4fa33d253035edd085c5d190bd818133871742
) as being the cause for the regression in the performance between 1.1.2
and 2.x. I am going to have to figure out how the performance of the
'hamlet' XMLOutputter,output(Document) goes from 3ms in JDOM1.1.2 to 24ms
in 2.x (on my laptop). I think it may have to do with the
Element.getNamespacesIntroduced() code. I will need another train-ride to
fix that!

I am certain that a tool that 'monitors' the performance of the core JDOM
features is essential for the confidence required in any changes we make to
JDOM itself, so I am committed to making sure such a tool is available.

Right now what I have is 'just adequate', I think, but it has rough edges,
and is narrowly scoped.

The question is how much effort should we put in to the 'tool' rather than
the core code? What's the trade-off? Is there a better way of doing it?

Is anyone willing to put together a more thorough 'harness' for JDOM? I
certainly would appreciate that! Something that is:
1. easy to run so that contributors to JDOM can test their work before and
after their submissions
2. has a way to 'preserve' results in an easy fashion that makes it easy
to update a web-page
3. is more 'extensible' than what I have done so far (so that 'plugging
in' additional tests is easy....).

I considered something 'on top' of jUnit, but the 'performance' benchmark
is more important for the 'typical' usages of JDOM, whereas the jUnit tests
are more targeted at the atypical execution paths.... We need *fast* core
code, but exceptions and unusual code we don't really care about in respect
to performance.

Rolf

On Fri, 14 Oct 2011 05:54:34 -0400, Rolf Lear <jdom at tuis.net> wrote:
> Hi Noel
> 
> Thanks for that.
> 
> It comes out in the numbers, but,  for the record, I am doing something 
> very similar to that.... only the structures are slightly (very)
different.
> 
> I do a bunch of GC's, and I do one in a different thread with the 
> current thread sleeping, then I repeat the GC's until the size becomes 
> 'stable' at a change of less than 128 bytes.
>
https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R33
> 
> I do a complete once-through of the test suite to warm things up.
> Each once-through runs the code through 6 times (hmmm... I thought it 
> was 12, but that was something else I did yesterday). Each of the actual

> tests 'exercises' the code repeatedly because it's all sort of 
> loop-based code (parsing, scanning, etc.).
> 
> Anyway, the output of the 'warmup' run is always much slower than the 
> remaining 5 'real' runs, and I do the 'real' runs multiple times to 
> ensure there is some stability.
> 
> What you see in the web-page is the result of what I believe to be fully

> JIT-compiled and 'clean' and 'reliable enough' for the purposes I want.
> 
> I know that the Java VM testing is 'tricky' when it comes to 
> performance, and as such I understand that it's easy to get things 
> wrong, and I'll spend more time looking at it to ensure I'm doing the 
> reasonable thing, but, are you suggesting that the code I am running is 
> not actually getting reliable results?
> 
> The code is structured differently to what you have suggested below, 
> but, the entire 'main' loop is warmed up: 
>
https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R124
> 
> Then, the main loop is run 5 times, and I visually inspect the numbers 
> to ensure that they are consistent: 
>
https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R135
> 
> Between each 'test' I do a full GC with 'bells and whistles'
>
https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R160
> 
> It is quite obvious that the runs that come out of the 'real' loops are 
> optimized, cached, etc.
> 
> What is not clear is whether the optimizer has completely compiled out 
> some of the code. I have tried to ensure that it does not by doing some 
> sort of test on each element so that it is not completely ignored.
> now that I think about it though, maybe the 'devnull' Writer is too 
> 'light' and the optimizer may have completely skipped it entirely..... 
> and the whole XMLOUtputter code with it.... I will check.
> 
> So, I appreciate the insight, and I will play around with things to see 
> if increasing the number of warmup and actual 'real' runs changes the 
> numbers.
> 
> I'll look in to making sure that some of the code is not being optimized

> out completely.
> 
> But, my code already is doing pretty much exactly what you are 
> suggesting... (it does not calculate the deviation, but it does ignore 
> the fastest and slowest run.....).
> 
> In fact, it does more because it then repeats the exact same loops 
> multiple times to ensure the averages remain consistent over runs (as it

> happens, it essentially does 20 'runs' of the code to get the results - 
> 5 loops of 6 runs but the 6 only counts as 4 because the best and worst 
> are eliminated).
> 
> Have you got specific concerns about the code? Did you run it? Do you 
> think the results are 'wrong'?
> 
> Thanks for the insight in to the commons-math code. I'm always 
> 'discovering' more and more 'stuff' in commons code. I have some 'stuff'

> I've done at work I am trying to convince my boss (actually 
> legal&compliance) to let me use in JDOM, but it's the sort of thing that

> belongs in a 'commons' type location, not JDOM....
> 
> Rolf
> 
> On 14/10/2011 4:08 AM, Noel Grandin wrote:
>> Hi
>>
>> Performance testing on the Java VM is tricky.
>> To avoid getting caught out by cache-hot/cache-cold and JIT vs. 
>> not-JIT things, it's preferrable to do something like this in 
>> PerfTest#timeRun(Runnnable)
>>
>> // warm up the caches and get the JIT going
>> for (int i=0; i<10; i++) {
>>    runnable.run();
>> }
>>
>> // give the JIT time to run, and get GC to run - GC can be stubborn 
>> sometimes
>> for (int i=0; i<3; i++) {
>> Thread.sleep(100);
>> System.gc();
>> }
>>
>> // need 20 runs to get a decent average and standard deviation
>> ArithmeticMean mean = new ArithmeticMean(); // these two classes are 
>> in jakarata-commons-math
>> Variance deviation = new Variance();
>> for (int i=0; i<20; i++) {
>>   long time1 = System.currentTimeNanos();
>>   runnable.run();
>>   long time2 = System.currentTimeNanos();
>>   mean.increment(time2 - time1);
>>   deviation.increment(time2 - time1);
>> }
>>
>> System.out.println("result  = " + mean.getMean() + " +- " + 
>> deviation.getVariance());
>>
>> Regards, Noel Grandin
>>
>> Rolf wrote:
>>> Hi all.
>>>
>>> I have put together a 'simple' system for measuring the relative 
>>> performance of JDOM2. The idea is that I need to know whether I am 
>>> improving or breaking JDOM performance as the code evolves.
>>>
>>> Currently the metric code is only useful of you compare apples to 
>>> apples, and, in this case, it means processing a single (medium size) 
>>> XML document on my laptop, yada-yada-yada. But, it should be useful 
>>> as a tool to get a feel for what a code-change does.
>>>
>>> Already I can see that I probably have an issue in the SAXHandler 
>>> (possibly an issue in JDOM-1.1.2 actually) because 1.1.2 is 5-times 
>>> faster in that area than JDOM2.
>>>
>>> I have put together a results page here:
>>>
>>> http://hunterhacker.github.com/jdom/jdom2/performance.html
>>>
>>> It also describes what each test does. If you are interested in 
>>> seeing the code and what it does have a look here (it is not well 
>>> documented and it is still perhaps evolving):
>>>
>>>
https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb
>>>
>>>
>>>
>>>
>>> Rolf
>>> _______________________________________________
>>> To control your jdom-interest membership:
>>>
http://www.jdom.org/mailman/options/jdom-interest/youraddr@yourhost.com
>>>
>>
>>
>>
>>
------------------------------------------------------------------------
>> Disclaimer: http://www.peralex.com/disclaimer.html
>>