[jdom-interest] Performance regressions in JDOM

Wed Jun 10 03:56:56 PDT 2015

Hello Vojtech

Your results are fascinating, and your feedback is very appreciated. It 
is going to take me a little while to digest it all, but, at the same 
time, yes, it is most likely that at a minimum I will duplicate your 
setup and see how it can be integrated. I strive to produce 
high-performing code and value any tools and feedback that can help with 
that.

I am delighted that you could use JDOM as a test subject for your tool, 
and paper. Congratulations on it, by the way.

As for your results, I wish you would have contacted us before choosing 
your individual test cases. I have some concerns about them. For 
example, the one you have listed as being a regression (negative 
Improvement) is this commit here:

https://github.com/hunterhacker/jdom/commit/4e2753539ad1217775e3b79dcb4cc7d1326a7798

That commit is a major update with a huge impact on many parts of the 
code base... it is also from 2001 and is from when the API was changing 
significantly.

Most importantly, that commit did two things of note:

1. it added a Text class to the API which is what likeley does impact 
and slow down the SAXBuilder.build performance
2. the performance changes it mentions are for "FilterList" which is not 
related to the build process at all. The performance impact would not be 
seen in build().

Your tool may have been useful to see the regression in the build, but 
the developer(s) are "innocent" (or, at least there's no relevant 
evidence) when it comes to a performance-reducing commit being called a 
performance-improving one.

Also of significance with that commit, it's using Java 1.2 or 
something.... it is not like we can actually go back and reproduce the 
same performance environments that were used at the time. The commit 
message may in fact be right....

----

The first Verifier example 4ad684a ( 
https://github.com/hunterhacker/jdom/commit/4ad684aa0427ee4f8feec8bbe360d6e32e32771c 
) you use for the verifier performance is great to see confirmed, but I 
am also concerned with that because that commit is just the first of 5 
that specifically target performance in the Verifier. As a developer 
(actually, as one of the two developers in this case), I would have 
preferred to see the results of the wider range of commits. Using the 
tool I would have wanted to compare my "current" performance to the 
"before" performance, and spanned the 5 commits, instead of just the first.

---

The remaining example also offers some insight. Unfortunately it is from 
so long ago.... from 2000: 
https://github.com/hunterhacker/jdom/commit/1a057182f2d87056e729d95fd9e18adffa7048fc

----

Regardless, I believe there is some significant value in here, and using 
your tool as a day-to-day aid seems like a good approach.

Rolf

On 10/06/2015 2:32 AM, Vojtech Horky wrote:
> Hello all.
>
> TL;DR version of this e-mail is: at our research group we run some 
> tests scanning for performance regressions in JDOM. We mostly focused 
> on finding out whether assumptions stated in commit messages (such as 
> "it should be faster and consume fewer resources") were actually met 
> and whether it is possible to capture them in automatically testable way.
>
> Now, we would like to know whether someone would be interested in the 
> results; whether we should polish the tests to be able to move them to 
> contrib/ or (better) whether someone would like to push this further.
>
>
> If you are interested in this, here is the full version.
>
> Our aim is to allow developers create performance unit tests, i.e. 
> something to test performance of individual methods. For example, if 
> you (think that you have) improved performance of a certain method, it 
> would be nice if you could test that. Of course, you can run tests 
> manually but that is time consuming and rather inconvenient in 
> general. So our tool (approach) allows you to capture such assumption 
> in a Java annotation and our program - still a prototype, but working 
> pretty well - scans these annotations and runs the tests 
> automatically, reporting violations. In this sense, it is like a 
> performance equivalent of an assertion of a unit test.
>
> As a simple example, if we want to capture an assumption stating that 
> SAXBuilder.build() function is faster in version (commit) 6a49ef6 than 
> in 4e27535, we would put in the annotation the following string:
>
> SAXBuilder#build @ 6a49ef6 < SAXBuilder#build @ 4e27535
>
> and the tool would handle the rest. Well, more or less.
>
> Regarding JDOM, we went through its commits and identified almost 50 
> of them that we found interesting. Interesting commits were those that 
> mentioned that they improved performance or when the commit was a 
> "refactoring" one. We measured mostly SAXBuilder.build() and several 
> methods from the Verifier class.
>
> In the end, we found out that we were able to confirm lot of the 
> assumptions about performance but there were also cases where the 
> assumptions were not met, i.e. the developer thought that the commit 
> improved performance while the opposite was true.
>
> We published our results in a paper [1] (PDF available as well [2]); 
> detailed results are available on-line [3].
>
> Right now, the tests themselves are in a separate repository [4] and 
> the setup is rather complicated. However, if someone would find this 
> interesting and potentially useful, we would gladly refactor the tests 
> to fit the contrib/ structure and prepare a fork to be merged.
>
> Regards,
> - Vojtech Horky
>
>
> [1] http://dx.doi.org/10.1007/978-3-642-40725-3_12
> [2] http://d3s.mff.cuni.cz/~horky/papers/epew2013.pdf
> [3] http://d3s.mff.cuni.cz/software/spl/#jdom-case-study
> [4] https://sf.net/p/spl-tools/casestudy/ci/master/tree/
>
>
> _______________________________________________
> To control your jdom-interest membership:
> http://www.jdom.org/mailman/options/jdom-interest/youraddr@yourhost.com
>