[KPhotoAlbum] Search performance
johannes at zarl-zierl.at
Thu Oct 18 21:26:16 CEST 2018
Am Donnerstag, 18. Oktober 2018, 14:48:23 CEST schrieb Robert Krawitz:
> On Thu, 18 Oct 2018 07:31:22 +0200, Martin Hoeller wrote:
> > The reason seemed to be complexity. Read this email thread from 6 years
> > ago: https://mail.kdab.com/pipermail/kphotoalbum/2012-April/005024.html
> Yes, I can well understand why.
IMO there were several problems with the approach that we took for the SQL
1) It was an *additional* backend+frontend (a new file format, as well as a
new database core). The "traditional" database still needed to be maintained
in addition to the new one. (I say "traditional" instead of "XMLDB" to avoid
confusion, because the backend has nothing to do with XML.)
2) Having a new file format prevented switching to the new format because many
people (at least if they were thinking like me) wouldn't trust the new format
right away and there was no easy way to switch back. Lack of user adoption
also means less testers, ...
3) Lack of unit testing for the database means that we can not make radical
changes with confidence.
This is, of course my personal opinion and not a criticism of Jesper, Tuomas,
Jan and Henner. It's always easier to look back and point out problems then to
build something and anticipate them ;-)
Learning from the old mistakes means that we could try to do things
1) Switch the backend fully to in-memory SQL.
2) Don't introduce a new on-disk format. Of course, we can still cache the
database on disk for performance reasons and only load from index.xml if it
3) Unit testing is still not there. Before taking on such an endeavor, we need
to get testing in place. I have done a little write-up of my plans to increase
> >> but if we want an
> >> SQL database, I think it would make more sense to store the data in
> >> one in the first place.
I should have mentioned in the first mail what I mean by "canonical file
format". I've no problems with storing the data into a persistent database for
But I still think that the index.xml format has good properties (resistant
against file corruption, easy/robust versioning, readable and writable "by
hand"). Also, many people use kphotoalbum on different machines in different
versions - with the XML format, you can easily pull that off as long as you
take some care.
If we take the caching approach, we should be able to eat our cake (index.xml
format, fast queries) and still have half of it (usually fast loading with
"slow" saving to index.xml).
> > Well, I like the idea of an SQL backend... the question is: can we manage
> > this task. As far as I know we have 3 active committers (all with limited
> > time). So as already 3 attempts were made an failed, I doubt that another
> > attempty would succeed.
> It certainly would not be an easy task; I just think that without it
> we're going to have a lot of trouble improving performance very much.
I think switching to an SQL backend is one goal that is worth investigating,
but plays very nicely with my "big picture" (if you can call it that): invest
time now in the "boring" task of increasing maintainability and leverage the
new freedom to move kphotoalbum forward.
> We're spending so much time in string functions, though, that I wonder
> whether shortening the XML tag and attribute names (to one
> character/byte, perhaps?) would significantly improve load/save
> times. It shrinks the size of my index.xml considerably, from 51 MB
> to 40 MB.
I'm not opposed to doing optimizations like that. I have to agree with you
though, that we won't get *huge* performance gains doing that.
More information about the KPhotoAlbum