Dflow

Real time event processing using data flow declarations.

The following has been tested using ghc 7.0.4 under Fedora 15 64-bit.

Installation

$ cabal configure $ cabal build $ cabal install

Tests

$ cabal configure --enable-test $ cabal build $ cabal test

The tests work by building random networks of event processors and checking them against an equivalent list implementation. Occasionally they seem to hit a pathological case that chews up memory and CPU time. If memory usage goes over 1GB then kill the test and try again.

Test Coverage

Edit the Dflow.cabal file and add "-fhpc" to the GHC options for the test (see comment in the file). Then do "cabal clean" and follow the instructions for "Tests" above. Then do:

$ hpc markup ArbTest.tix --destdir=coverage

The coverage report is in coverage/hpc_index.html.

Concepts

The main data types for the application programmer are:

Event: A value that occurs at a certain time. For instance an "Event Char" might represent a key press.

RTSP: The Real Time Stream Processor. A value of type "RTSP x y" takes in events of type "x" and emits events of type "y". RTSPs can be strung together into pipelines using "." (or ">>>" if you prefer your data to flow left-to-right). RTSPs are also Monoids, so you can fork your data through two parallel RTSPs and then merge the results.

RTA: A monad for building stateful RTSPs. Convert an RTA into an RTSP using "execRTA" or "accumulateRTA" depending what you want to do with pending output events when a new input event arrives.

You can test an RTSP in "fast time" (that is, without waiting for real-time delays) by using "simulateRTSP". However the argument list of input events must be finite. Then you can execute the RTSP in real time using "execRTSP" and be confident that the real time behaviour will match the fast-time behaviour.

To Do

Events are currently sorted into time order using (in effect) an O(n) insertion sort. Event streams should use something more sophisticated internally, such as a heap.

What API, if any, should be exposed for event streams? On one hand developers should be able to create new kinds of RTSP primitives using event streams, but on the other hand anything you can do with event streams can be done using RTA anyway.

Figure out some way for RTAs and RTSPs to execute non-blocking IO with timeouts, probably using continuations in some way. Hence develop true distribution by having RTSPs on different machines interact in a declarative manner.

Make state persistent, together with a replay mechanism for lost events.