[jdom-interest] How to manipulate a very large XML file? Any suggestions?

[Elliotte Rusty Harold]

At 9:22 AM -0800 2/10/04, Jason Hunter wrote:

>>  However, this could also be done with SQL and a relational 
>>database or a custom file format written in Java.
>
>Shredding XML into tables is not an enjoyable task and does not make 
>for efficient queries.  You're trying to fit a jagged peg into a 
>square hole.

It depends completely on the structure of the data and the nature of 
the problem. Relational databases work very well for many classes of 
problems. They are much more proven and reliable than native XML 
databases. I wouldn't even consider a native XML database at this 
point unless the problem domain were clearly unsuited to a relational 
database.

>And writing a custom file format in Java?  Let's compare techniques 
>sometimes, Rusty.  You'll do what you propose, and I'll use a real 
>XQuery engine.  We'll throw in about 10 Gigs of data and see who can 
>make the data dance.  :-)

Again, it depends completely on the nature of the problem. I could 
throw in ten gigs of solar astronomy data, then ask for a 
three-dimensional fast fourier transform. Not exactly the problem 
XQuery is designed to handle. With memory mapped I/O, Java (or better 
yet, Fortran) could blast through this.

-- 

   Elliotte Rusty Harold
   elharo at metalab.unc.edu
   Effective XML (Addison-Wesley, 2003)
   http://www.cafeconleche.org/books/effectivexml
   http://www.amazon.com/exec/obidos/ISBN%3D0321150406/ref%3Dnosim/cafeaulaitA