Flow-based Programming

Panta rhei (Panta rhei) - Everything flows.

Relationship with NoFlo

Some readers may have arrived at Flow-Based Programming (FBP) by way of NoFlo, which is a JavaScript-based system motivated by J. Paul Morrison's book (Flow-based Programming, 2nd edition), and which uses a number of the same terms and concepts. This project, started by Henri Bergius in 2012, implements a number of the FBP concepts, and has been creating a significant buzz world-wide since then, due largely to the technical and marketing skills of Henri, Dan Tocchini IV, and their talented team at The Grid.

In August, 2013, they started a KickStarter project to raise $100,000 for future development, and had raised $115,677 by the end of the 45-day period. NoFlo has been recognized by Forbes, O'Reilly, FastCompany, and a number of other companies and publications.

NoFlo is integrated with an open-ended graphics tool called NoFlo-UI, developed by The Grid.

While a lot of credit is due the NoFlo team for bringing FBP to the attention of the computer world, what the developers of NoFlo call "FBP" in fact differs in a number of respects from FBP as it has evolved over the last 40+ years. Although NoFlo shares with FBP some of its terminology and characteristics, it has a somewhat different mindset, and targets a different application area. The essential difference is that NoFlo and "classical" FBP embody different paradigms, with "classical" FBP embodying a new/old, asynchronous paradigm, while NoFlo still conforms to the standard, synchronous, "von Neumann" paradigm. The latter is the paradigm that almost all programmers have grown up with, so that they cannot easily envision a different way of building applications. Programmers trained in the latter paradigm (the vast majority) tend to view this distinction as a purely aesthetic choice, but the difference is actually much more fundamental.

For those unfamiliar with term "von Neumann", it refers to a computer design where a single instruction counter walks through a program accessing a uniform array of non-destructive readout memory cells. This has in fact been the "classical" computer architecture for several decades, but people are increasingly finding it inadequate for today's challenges, as shown by the frequent cost and schedule overruns, weird bugs, and difficulty maintaining large applications. These problems have now been shown to derive in large part from the architecture itself, as more and more writers have started to point out.

We referred to FBP as a "new/old" paradigm, because in fact its approach and methodology has parallels with Unit Record systems, which were used for the first data processing applications, until these systems started to be replaced by today's computers. In the process, however, a lot of useful functions were lost... which FBP is now reintroducing.

It is now time to replace the von Neumann paradigm with a new programming paradigm, referred to as "Flow-Based Programming", which approaches the construction of computer applications from a very different point of view. An application built using FBP may be thought of as a "data processing factory": a network of independent "machines", communicating by means of conveyor belts, across which travel structured chunks of data, which are modified by successive "machines" until they are output to files or discarded. The various "machines" run in parallel, or intermingled, as determined by the number of processors in the machine. Of course these "machines" can be real hardware machines, or simulated on one or more hardware machines, or combinations of the above.

FBP in this form (sometimes called "classical" FBP or "cFBP", to minimize confusion) supports data processing applications (business or scientific), typically long-running and high volume, and, as we have shown, involves a way of thinking (the new "paradigm") that is fundamentally different from that of conventional programming. This paradigm is actually more similar to engineering than conventional programming. While similar models have been used for application design for a number of years, up until now there was no easy way of converting these designs into running programs. Programmers could indeed design systems using data-oriented thinking, but then had to laboriously convert these designs into procedural code. In comparison, FBP supports a seamless transition from design to implementation, and our experience with it shows that it results in more maintainable and in fact better performing systems. It also facilitates communication between designers, programmers, maintenance staff and users. One large program written using an early ("green thread") implementation of FBP was running in production for almost 40 years (as of the beginning of 2014) processing millions of transactions a night, while undergoing continuous maintenance during all that time, often by people who weren't even born when it was written!

NoFlo gets its name from Node.js, a JavaScript-based, open source, cross-platform runtime environment for server-side and networking applications. Node.js runs in a single native thread, and supports a certain degree of parallelism based on JavaScript's callback mechanism. NoFlo therefore tends to be more synchronous and procedural than FBP, and in fact addresses a rather different problem space from that addressed by FBP. However, it shares the attribute of "configurable modularity" with "classical" FBP, and this certainly gives it considerable flexibility. Both approaches are highly visual - with NoFlo being currently supported by the NoFlo-UI visual environment.

While an FBP process is a "black box" component with its own internal environment and control thread, a NoFlo process is essentially a cloud of callbacks linked by instance variables. Henri Bergius was able to simulate many FBP-like characteristics on the Node.js infrastructure, but some rather basic FBP techniques have no obvious counterpart in NoFlo. For instance, basic FBP business functions such as "Collate" require a process to be specific about which port it wants to receive from, and to be able to suspend until data arrives at that port - this would involve extensive changes to the NoFlo infrastructure, if it could be done at all. A related architectural concept in FBP, missing from NoFlo, is what we call "back pressure", where an upstream process will be suspended if the connection it feeds into becomes full. Finally, NoFlo lacks the concept of information packet (IP) "lifetimes", by which an IP is tracked from creation to destruction and can only be "owned" by a single process at a time, or be in transit between processes.

While NoFlo is appealing because of its ability to support both client- and server-side processing, thanks to Node.js, and because of JavaScript's close integration with HTML, it is still tied pretty closely to "von Neumann thinking". Also all existing implementations of "classical" FBP on J. Paul Morrison's GitHub directory can take advantage of multiple cores, with the exception of JSFBP (see below). Because of JavaScript's restriction to a single core (similarly to the early FBP "green thread" implementations) neither NoFlo nor JSFBP support CPU-intensive applications, which are in fact well supported by JavaFBP, C#FBP and C++FBP using Boost.

Because the NoFlo people use the term FBP so prominently when talking about NoFlo, and to give them due credit for having "kickstarted" awareness of FBP within the IT industry, we sometimes add the term "classical" to "FBP" when it is necessary to to distinguish it from NoFlo and other FBP-like frameworks. A number of the latter are starting to appear, such as IBM's recent NodeRed, but these systems are different in important ways from "classical" FBP, based as they are on von Neumann thinking, and there is clearly common ground, as both FBP and NoFlo are component-oriented, with multiple instances of components communicating indirectly via a list of connections, giving what is called "configurable modularity", and both supporting stepwise decomposition using "subnets". However, our experience shows that it is the above-mentioned ("classical" FBP) paradigm change that offers the most leverage for improved productivity and maintainability in application development.

Recently J. Paul Morrison, with two colleagues, has been working on a more "classical" JavaScript FBP implementation called JSFBP, based on node-fibers, developed by Marcel Laverdet, and therefore in turn on Node.js. This is actually a "green thread" implementation, as were the first two FBP implementations running on IBM mainframes. These systems supported asynchronous I/O, so performance was good enough that the first FBP implementation was able to support most of the batch processing for a major bank for over 40 years! However, "green threads" do not support multiple cores, a limitation shared by NoFlo, and JSFBP's dependence on "node-fibers" seems to suggest that it will not gain wide acceptance. At least two more implementations of FBP in JavaScript are currently being worked on, based on the advanced (main stream) JavaScript ideas of Promises and Streams, but more work needs to be done to see how well they will be able to support the basic concepts of "classical" FBP.

Because of cFBP's highly asynchronous nature, it naturally results in components with lower granularity (coarser-grained), working with more complex data objects, than NoFlo. As stated above, cFBP data objects (Information Packets or IPs) behave more like objects in the real world than variables in conventional programming (including NoFlo): they have a well-defined lifetime, from creation to destruction, and can only be owned by one process at a time, or be in transit between processes. While NoFlo allows one output port to be connected to two input ports, FBP does not allow this as this would entail magically cloning IPs (potentially even complex IP trees), and FBP designers feel this should be left to the discretion of the network designer. In addition, every IP that a process takes ownership for (by creating or receiving) must be explicitly disposed of (by sending or destroying) before that process deactivates. In fact, there is almost no "global" data in an FBP application - all data is either local to a method within a process or held within Information Packets. This is very different from the storage management concept underlying conventional programming (and NoFlo and its ilk).

A common philosophy which is shared by "classical" FBP and NoFlo is that they are both to some extent "language-agnostic". NoFlo does use JavaScript (and CoffeeScript), but its UI (NoFlo-UI) has successfully been used to generate networks in several different languages (including JavaFBP), and on several different types of hardware. In the case of "classical" FBP, the different virtual machines underlying Java and C# make it difficult for JavaFBP applications to communicate with C#FBP applications in the same network, but they can communicate by means of sockets, so multiple networks running in, say, Java and C#, could function as a "super-network" combining the strengths of both languages, with sockets providing the linkages. Since these ideas can be extended up or down ("fractally"), FBP provides a consistent application view from very high level (e.g. documents flowing between departments), down to the implementation level.

For those wishing to gain experience with "classical" FBP, there is no substitute for reading the book (Flow-based Programming, 2nd edition), and then starting to use one of the FBP implementations such as JavaFBP or C#FBP, or even the C++/Boost implementation currently under development, as described on the FBP web site. JavaFBP has the advantage of being closely integrated with a powerful diagramming tool, DrawFBP (see below).

NoFlo-UI, NoFlo's diagramming tool, while originally developed to support NoFlo, is more general, and is in process of being extended to support other FBP, and FBP-like, development environments. NoFlo-UI is well-integrated with the NoFlo environment, and provides run-time interpretive facilities to help develop NoFlo applications. An extension of NoFlo-UI called Flowhub has been adapted for use with "classical" FBP implementations (in fact one prototype has actually run a JavaFBP program), except that at this time it is lacking some features that are required for full scale FBP projects - see NoFlo-UI Issues 370-374.

For the time being, users wishing to work with "classical" FBP can either code up networks using JavaFBP, C#FBP or CppFBP by hand, or JSFBP. Alternatively, they can use the DrawFBP drawing tool, written using Java Swing, which is also quite general, and can in fact generate the .fbp notation used by NoFlo and CppFBP, as well as NoFlo JSON networks. While DrawFBP does not support run-time network execution, it can generate runnable networks for JavaFBP (immediately runnable) and C#FBP. DrawFBP diagrams are stored in XML format, and additional generators can be added easily, or users can build their own generators using the XML format as input.

FBP and OO

For a discussion of the differences and similarities of FBP vis à vis OO, see Comparison between FBP and Object-Oriented Programming (Chapter 25 of the 2nd edition).