[Gammaray-interest] Thoughts about the problem scanner

[Volker Krause]

On Friday, 10 May 2019 13:53:40 CEST Christian Gagneraud wrote:
> [some old draft emails, sorry if it doesn't make sense]
> 
> On Mon, 25 Mar 2019 at 23:09, Volker Krause <volker.krause at kdab.com> wrote:
> > > > The signal history model is already quite expensive, you certainly
> > > > don't
> > > > want to transfer that entirely to the client just to sum up costs for
> > > > example, so doing aggregation in there makes a lot of sense. A useful
> > > > intermediate step to verify the data is all there might be to show
> > > > emission data in the connections tab of an object.
> > > 
> > > I guess, what's expensive is the list of event, that keep growing,
> > > right? And the fact that the object list never shrink as well?
> > 
> > Exactly, unbound memory consumption combined with the high frequency
> > emission of some signals (eg. from render or event loops). We already try
> > to exclude some of the worst built-in offenders, but that is of course
> > skewing the results.
> 
> Yes, I saw the exclusion of event dispatcher.
> 
> When i look at the Signal History model, i have the feeling that the
> model is wrong given what is achieved graphically.
> Basically, b/c the 3rd column is the "scope" timeline, then all the
> event data are transferred as the column data. So each time a signal
> is emitted the gui has to fetch the whole time series, when it only
> needs the missing events.
> To be network efficient, why not have a table model that represent
> signal emission as a time series, eg 4 column: timestamp, object,
> signal, duration.
> this way each time a signal is triggered one row of data is added, and
> only that single row is fetched by the GUI.
> Which means that the "scope delegate" of the signal history model
> would need to get it's data from another model... that's the "weird"
> part.
> And to keep the view update logic, the third column could instead be
> "signal emission total count".
> 
> Alongside the current history view, a "stupid" table view of the above
> table model would still be useful, eg if you want to check for signal
> emission ordering or anylise timing, ...

Yep, I agree. What Tim is currently working on for the event monitor (which 
has a similar problem with high frequency data) is going into that direction. 
We are also looking into filtering non-interesting events before recording 
already, based on user configuration. So this will hopefully provide some 
input on how to improve the signal tooling as well.

> Any way, i'm not planning to experiment with that, but for the graph
> we talked about (Connectivity map with weighted edges, and connection
> cascading), I realised that what gammaray offers simply does not fit.
> For example, if i understand correctly, i cannot use the connection
> model on arbitrary objects, there is only two connection models
> available and they point to the current selected object. On the server
> side, i've used InboundConnectionModel::inboundConnectionForObject(),
> but this is not accessible on the GUI side.
> To gather detailed connection data for all objects, i would have to
> programmatically select all object, one after another. This looks very
> hackish to me...

Right, this is something that probably needs server-side support. Since Qt5 
doesn't have callbacks for dis/connect anymore, I think the iteration will be 
unavoidable, but it should happen server-side.

> > > As well, this history model only tracks signals, no slots, are you OK
> > > if I add slot tracking too?
> > 
> > To make this really useful we need to track that too, yes. We might get to
> > the point where we need a way to limit the memory consumption though, by
> > having a fixed limit and purging older records, or by making the full
> > tracking opt-in, etc. Depends how expensive this really turns out to be I
> > guess.
> 
> Or maybe have a start/stop/restart and clear command? There is a pause
> button in the gui, but i guess it doesn't pause the recording in the
> probe, it just pauses the rendering in the GUI.

Yep, something we have added to the event monitor very early already, it 
should be in the signal view as well.

> > > > However, If you primarily care about the connection structure rather
> > > > than
> > > > the emission data, producing a graphviz dot file with code similar to
> > > > the
> > > > connection issue scanner should already be doable, maybe with some
> > > > filtering applied to hide non-connected objects to keep the result
> > > > readable.
> > > 
> > > Yes, but this is server-side code. Ideally I would prefer to generate
> > > it in the client/gui side...
> > 
> > Right, that is conceptually the cleaner approach of course. We might need
> > to look at more efficient ways to transfer the data than the commonly
> > used remote model view approach though. That's optimized for
> > interactivity rather than efficient transfers, and it's not offering a
> > convenient way for you to know you downloaded everything you need. The Qt
> > Quick remote view might offer some inspiration for alternatives.
> 
> See above for a different view of the problem.
> After a quick dive into the RemoteView code, it doesn't look that
> complicated (it never looks complicated *at first*! :))
> It seems that the "magic" of remote objects is
> ObjectBroker::registerObject() and
> Endpoint::instance()->invokeObject() on the client side.

The low-level communication isn't the tricky part, for the RemoteView it 
turned out to be the flow control. On embedded systems we managed to saturate 
the network interface with this, leading to ever increasing local network 
buffers, effects you don't see on desktop systems where you could easily debug 
this.

Regards,
Volker
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4664 bytes
Desc: not available
URL: <http://mail.kdab.com/pipermail/gammaray-interest/attachments/20190514/f21dc4fb/attachment.p7s>