Flow-based Programming

Panta rhei (Panta rhei) - Everything flows.

Relationship with NoFlo

Some readers may have arrived at Flow-Based Programming (FBP) by way of NoFlo, which is a JavaScript-based system motivated by J. Paul Morrison's book (Flow-based Programming, 2nd edition), and which uses a number of the same terms and concepts. This project, started by Henri Bergius in 2012, implements a number of the FBP concepts, and has been creating a significant buzz world-wide since then, due largely to the technical and marketing skills of Henri, Dan Tocchini IV, and their talented team at The Grid.

In Aug. 1, 2013, they started a KickStarter project to raise $100,000 for future development, and had raised $115,677 by the end of the 45-day period. NoFlo has been recognized by Forbes, O'Reilly, FastCompany, and many other companies and publications.

NoFlo is integrated with an open-ended graphics tool called NoFlo-UI, developed by The Grid.

While a lot of credit is due the NoFlo team for bringing FBP to the attention of the computer world, what the developers of NoFlo call "FBP" in fact differs in a number of respects from FBP as it has evolved over the last 40+ years. Although NoFlo shares with FBP some of its terminology and characteristics, it has a very different mindset, and targets a different application area.

FBP (sometimes called "classical" FBP) addresses data processing applications (business or scientific), typically long-running and high volume, and has a way of thinking ("non-von Neumann") that is different from conventional programming. This new paradigm can be thought of as a "data factory" mental image, where the application is expressed as a series of transforms on data streams. This paradigm requires a fundamental shift in the way the designer/developer thinks about building applications, much more akin to engineering than conventional programming, highly visual and component-oriented. Experience shows that this paradigm results in more maintainable and in fact better performing systems. One large program written using an early (green thread) implementation of FBP was running in production for almost 40 years (as of the beginning of 2014) processing millions of transactions a night, while undergoing continuous maintenance during all that time, often by people who weren't even born when it was written!

NoFlo gets its name from Node.js, a JavaScript-based, open source, cross-platform runtime environment for server-side and networking applications. Because it runs in a single process, NoFlo is naturally much more synchronous and procedural than FBP, and in fact addresses a rather different problem space from that addressed by FBP: NoFlo addresses user experience (UX), human interfaces, and its "configurable modularity" certainly gives it great flexibility. However, NoFlo's dependence on JavaScript has both advantages and disadvantages. On the one hand, with the advent of Node.js, JavaScript is now usable for both client- and server-side programming, so developers can use a single language for both client and server code, and NoFlo takes advantage of that. On the other hand, JavaScript is purely synchronous, and does not support multiple cores. Node.js does have asynchronous I/O, but does not provide support for CPU-intensive applications, so it would be better to describe NoFlo, not as "an FBP implementation", but as "an FBP-like dataflow framework for the Web and non-CPU-intensive applications".

While an FBP process is a "black box" component with its own internal environment and control thread, a NoFlo process is essentially a cloud of callbacks loosely linked by instance variables. Henri Bergius was able to simulate many FBP-like characteristics on the Node.js infrastructure, but some extremely basic FBP techniques have no obvious counterpart in NoFlo. For instance, basic FBP business functions such as "Collate" require a process to be specific about which port it wants to receive from, and be able to suspend until data arrives at that port - this would involve extensive changes to the NoFlo infrastructure, if it could be done at all. A related architectural concept in FBP, missing from NoFlo, is what we have started to call "back pressure", where an upstream process will be suspended if the connection it feeds into becomes full. In comparison, NoFlo runs in a single thread, so cannot take full advantage of multiple cores - in addition, for complex networks, the designer has to keep NoFlo's scheduling algorithm in mind, which results in a more complex mental model. While NoFlo is appealing because of its ability to support both client- and server-side processing, and because of JavaScript's close integration with HTML, it is still tied to "von Neumann thinking", and experience suggests that it will not provide the productivity gains being looked for.

Because the NoFlo people use the term FBP so prominently when talking about NoFlo, and to give them due credit for having "kickstarted" awareness of FBP in the computing industry, we frequently add the term "classical" to FBP to distinguish it from NoFlo and other FBP-like frameworks. A number of these are starting to appear, such as IBM's recent NodeRed, but these systems are different in important ways from "classical" FBP, based as they are on von Neumann thinking. However, there is clearly common ground, as both FBP and NoFlo are component-oriented, with multiple instances of components communicating indirectly via a list of connections, giving what is called "configurable modularity", and both support stepwise decomposition using "subnets". We have sometimes used the term "reactive" FBP to distinguish NoFlo and similar technologies from FBP proper, but it has been pointed out that many systems, both FBP-like and not, can be described as "reactive", so this does not seem like an appropriate term.

Because of FBP's asynchronous nature, it naturally results in components with lower granularity (coarser-grained), working with more complex data objects. In addition, FBP data objects (Information Packets or IPs) behave more like objects in the real world than variables in conventional programming (including NoFlo): they have a well-defined lifetime, from creation to destruction, and can only be owned by one process at a time, or be in transit between processes. While NoFlo allows one output port to be connected to two input ports, FBP does not allow this as this would entail magically cloning IPs (potentially even complex IP trees), and FBP designers feel this should be up to the discretion of the network designer. In addition, every IP that a process takes ownership for (by creating or receiving) must be explicitly disposed of (by sending or destroying) before that process deactivates. In fact, there is almost no "global" data in an FBP application - all data is either local to a method or held within Information Packets. This is very different from the storage management concept underlying conventional programming.

FBP networks written in different languages can communicate by way of sockets. FBP is essentially language-agnostic, but the virtual machines underlying many modern programming languages make it difficult for code written in different languages to communicate, so multiple networks running in, say, Java and C#, could function as a "super-network" combining the strengths of both languages, with sockets providing the linkages between networks written in different languages. Since these ideas can be extended up or down ("fractally" as someone called it), FBP provides a consistent application view from very high level (e.g. documents flowing between departments), down to the implementation level.

For those wishing to gain experience with "classical" FBP, there is no substitute for reading the book (Flow-based Programming, 2nd edition), and then starting to use one of the FBP implementations such as JavaFBP or C#FBP, or even the C++/Boost implementation currently under development, as described on the FBP web site.

NoFlo-UI, NoFlo's diagramming tool, while originally developed to support NoFlo, is more general, and is in process of being extended to support other FBP, and FBP-like, development environments. NoFlo-UI is well-integrated with the NoFlo environment, and provides run-time interpretive facilities to help develop NoFlo applications. There is no reason why Noflo-UI could not be adapted for use with "classical" FBP implementations, except that at this time it is lacking some features that are required for full scale FBP projects - see NoFlo-UI Issues 370-374.

For the time being, users wishing to work with "classical" FBP can either code up networks using JavaFBP, C#FBP or CppFBP by hand, or use the DrawFBP drawing tool, written using Java Swing, which is also quite general, and can in fact generate the .fbp notation used by NoFlo and CppFBP, as well as NoFlo JSON networks. While DrawFBP does not support run-time network execution, it can generate runnable networks for JavaFBP (immediately runnable) and C#FBP. DrawFBP diagrams are stored in XML format, and additional generators can be added easily, or users can build their own generators using the XML format as input.

FBP and OO

For a discussion of the differences and similarities of FBP vis à vis OO, see Comparison between FBP and Object-Oriented Programming (Chapter 25 of the 2nd edition).