Atomic/Consistent Detector Data Store

Table of Contents

• Introduction
• The naive solution
• Topic Introspection
• Detector Data Store Service
• Conclusion

Introduction

DetectorGraph removes from the detectors the responsability (and freedom) to store and serve its state alongside with performing computation. We belive that's a design advantage but that clearly leaves opens a gap of functionality when compared to our current DetectorGraph. That gap needs to be addressed. The following are our current uses of such functionality:

1. a DetectorA uses the sate of a DetectorB without actually subscribing to DetectorB
2. a DetectorA Publishes outgoing data consumed by Consumer1 (e.g. AppThread) which in turn inspects a (locked) graph for the states of DetectorA, DetectorB, .. DetectorX to form a complete opinion.
3. a DetectorA wants to save/resume it's state (desired)

The design constraints for this solution are:

1. Retrieving this state should be 'atomic' in relation to graph evaluations; this means that all values returned should be product of the same evaluation pass.
2. The retrieved state should be "the most up-to-date" possible.

This section lists a few approaches to close that gap.

The naive solution

The most simple solution to this would be to compose the needed state on the Consumer side by subscribing to all events that compose it and inpecting it when needed. This solution has a number of inefficiencies and one major flaw:

The consumer would demand awareness of the boundaries between evaluation passes and only use the state at the end of it. Say Consumer needs to know StateB when it receives an update to StateA, caching would only work if StateB updates are produced before StateA updates - so it's dependent on the order events are emitted. To solve this you'd need a AllEventsEmitted event... you get it: it gets ugly before you can say quick.

Topic Introspection

Another suggested solution is to allow a consumer to safely introspection topics in between evaluations of the graph. This appears to be the smallest effort solution given the role that topics play on the graph. This approach has a few advantages and disadvantages:

Advantages:

• The latest state is returned by design
• No extra possible points of failure or latency between what the graph sees and what the Consumer would see.
• No concepts added, logical extension of previous design.

Disadvantages:

• Introducing locking/multi-threading mechanisms into topics and/or graph adds application complexity to the basic detector graph.
• Locking the graph means the application can interfere with the performance of the graph as a computation framework.
• Topic is a sensitive class when it comes to code size due to its templated nature.
• Many of topics have very local/detector-specific nature (timeouts, internals) and exposing access to it can be seen as an anti-pattern that will mix public & private workings of the graph.

We believe the disadvantages outweighs the advantages when it comes to code quality and excellence even though this approach might seem as the smallest effort one.

Detector Data Store Service

Following ROS's inspiration, its concept of Services lands itself nicely to the idea of an entity that provides a complete snapshot of the graph's state through a atomic method call. If implemented correctly, a DetectorStateStoreService could ensure state atomicity/consistency as a single-stop shop for all needed states. This solution extends the loose coupling across detectors to consumers (e.g. app).

Conclusion

See GraphStateStore