[Gammaray-interest] Getting rid of "Loading..." on client side models

[Christian Gagneraud]

On Fri, 7 Jun 2019 at 02:24, Volker Krause <volker.krause at kdab.com> wrote:
>
> On Thursday, 6 June 2019 14:15:04 CEST Christian Gagneraud wrote:
> > On Thu, 6 Jun 2019 at 01:54, Christian Gagneraud <chgans at gmail.com> wrote:
> > > When programmatically using remote models to build my graphs, i have
> > > to deal with a very annoying "feature": the lazy loading.
> > >
> > > The problem is that i need to filter out these cases as a node in a
> > > graph is uniquely identified by its object ID, and this object ID is 0
> > > when the item/row is in the "Loading.." state, which breaks
> > > "everything" (I need all my data sources to be complete, fully
> > > resolved, synchronised and ordered).
> > > So far i'm unable to "stream" the graph data (changes). I need to
> > > rebuild it continuously, which is rather inefficient.
> > >
> > > Is there a mechanism to be notified of a loading -> loaded transition?
> > >
> > > Let's dream for a moment: Wouldn't it be nice to have deferred
> > > notification of  item/row/column/... inserted/removed/changed, instead
> > > of asynchronous content?
>
> Yes, the lazy loading is getting in the way for direct use. There's good
> reasons why it's there though, together with the change batching you noticed
> below as well: Performance for interactive use. We didn't have a lot of these
> optimizations initially, and it simply wasn't able to keep up. If we combine
> content retrieval and "structure" updates (ie. data() and rowCount() calls),
> we'd be downloading the data for all cells in a 10k row model, even if you
> only look at 50 at a time. Worse in tree models, where rowCount() is requested
> for every row to determine the amount of children, we don't want to trigger
> the download recursively there.
>
> The batching and delayed signal emission has similar reasons, a property model
> on QQ2 or Qt3D elements can easily change 60 times per second, enough to drown
> an embedded device in GammaRay model updates. Some of the source models
> implement event more aggregation on top of that, such as the event model.
>
> Another thing that was added in that context was the communication statistics
> under Help -> Diagnostics (when having GAMMARAY_DEVELOPERMODE=1 set in the
> environment), useful to see the cost/impact of remote model operations.

Thanks for sharing all these info.

I noticed the bandwidth stats messages on stdout, but man the GUI one
is very detailed! :)

> For your use-case this isn't helping obviously. I see two options:
> - adding a dedicated mode to the remote model for this, disabling all those
> optimizations, and with some form of done signal.

I think the optimisation are good, it's just the out-of-order and
synchronisation with non-model remote objects that is a problem.
This is out of scope for me to add a new mode, this is technically
complex and time consuming.

But after looking at the remote model and the remote model server,
I've noticed 2 interesting features:
- The sync barrier, which seems to be used only when clearing a remote
model on the client side. Not sure it could be used for other purpose
- The RemoteModelNodeState, this one is IMHO way more interesting.

If your use case is to be notified that all the datas you've requested
are now available, then it's just a matter of tracking how many nodes
haven't had their state cleared, which is done here[1] if i'm not
wrong.
A simple counter will do. Maybe node state access should be wrapped by
a seter/getter, that would maintain the counter and emit a signal when
this counter reach 0.

Maybe we could add an extra flags parameter to ObjectBroker::model,
that would be passed down to the RemoteServer ctor.
Since model() returns a pointer to QAIM, maybe the communication
mechanism could be done via a self posted custom event or dynamic
properties.
If the self posted event could contain both the counter and the goal,
you could use that info to update a progress bar, and take action once
you hit 100% synced.... :)
When i do my tests against QtCreator, it can take 3 to 4 seconds to
just get the data needed to build the initial graph.

If i'm not wrong all signals emitted by the main event loop on the
server side will be sent over the network in their order of emission.

That's all nice theory, what's your feeling about this?

[1] https://github.com/KDAB/GammaRay/blob/master/client/remotemodel.cpp#L373


> - rather than going through a model, send your entire data as a signal
> argument or synchronized property to the client, on demand. For those in-order
> delivery is guaranteed.

That's a tempting approach, i might use that if the above idea cannot
work or is not feasible.

>
> Regards,
> Volker
>
> > B/c i don't have connect notify, my approach so far was to
> > incrementally build the list of connected objects by using a sample
> > timer, and scanning all (pre-filtered) objects [1].
> > This sort of worked, but i realised that model and remote object
> > signals are received on the client side out-of order.
> > For example at sample time, server side,  the code update the
> > connection list, and then emit a "sampling done" signal.
> > On the client side, remote model changes are tracked by setting a
> > simple "need sync" flag, and then when the "sampling done"
> > notification is received, the sync is applied if needed.
> > This have gave me a lot of head-ache, as it turns out that the model
> > signals can be delayed-emitted after the "sampling done" signal.
> > It looks to me that there's some "aggregation" process going on
> > somewhere, i don't know if it's from GammaRay or Qt.
> > But what i've notived, is that if i reset a model a then insert rows,
> > and then emit the "done" signal. The server will receive, the reset
> > signal, then the "done" signal and then the aggregated "rows inserted"
> > signal.
> > This whole process is done with the object lock on, as it has to be an
> > instantaneous snapshot.
> >
> > Maybe i should create a PR with my server side code, so that you can
> > have a better picture of what i'm trying to do.
> >
> > Chris.
> >
> > [1] The original idea was to store sender/receiver objectid/threadid,
> > plus number of connections, but it turns out that it is impossible to
> > track connections this way, for example you cannot detect duplicated
> > connections w/o resetting the connection model before-hand.
> >
> > > Thanks,
> > > Chris