For those of you unfamiliar with wikis, a wiki is a tree structure and you would normally follow the (underscored blue, green or mauve) links to get where you want to go, but for those who want to dive right in to a particular topic, you can either use the search facility, or click on http://www.jpaulmorrison.com/cgi-bin/wiki.pl?action=index. A more complex facility is the "links" facility: to show the links structure, click on http://www.jpaulmorrison.com/cgi-bin/wiki.pl?action=links.
The Flow-Based Programming Wiki uses the UseModWiki implementation. For information on how to edit text, see TextFormattingRules. For some background on wikis, and how they got started, see WelcomeVisitors. To see all changes, click on http://www.jpaulmorrison.com/cgi-bin/wiki.pl?action=rc&all=1&showedit=1.
The news portion of this page has now been moved into a separate page called FBPNews - it was getting too big! The most recent items will continue to be shown here (in reverse chron sequence):
More news of a totally different kind: it seems that hot discussion has been bombing along on comp.lang.lisp, comparing FBP and Cells, started by one Frank Buss, in Germany I assume, on May 23, 2008. It can be found at http://groups.google.ca/group/comp.lang.lisp/browse_thread/thread/0719e128c11d7c3c?hl=en , and I found it very interesting! I got told about it yesterday, and already someone has been prompted to try out JavaFBP, and contact me about it. 2008-05-27.
Another source of deadlocks has, I hope, been removed from http://www.jpaulmorrison.com/fbp/javaFBP-2.1.jar . There was also a problem with the deadlock detection logic - I hope this also has been fixed. Thanks to everyone who helped with these problems! I would appreciate it enormously if someone could subject this version to some thorough inspection and/or testing.
Also, I realized that deadlock detection may have been switched off on a recent promotion - this has now been corrected. Also, methods have been provided to push and pop to and from component stacks - this was always in the architecture, but for some reason was missing from the Java implementation. 2008-05-24.
I have just received a thought-provoking note from a software engineer named JustinBozonier. His very interesting blog can be found at http://www.aboutjustin.com/blog/. I have added a portion of our discussion in AsynchronismAndUIs - Justin feel free to modify it! 2008-05-19
A mini-release of JavaFBP (http://www.jpaulmorrison.com/fbp/javaFBP-2.1.jar) has been put on my web site and on SourceForge, containing a network which plays some bars of "Jingle Bells" :-) It also allows the priority of a component to be set using Java metadata. 2008-03-31
Another mini-release of C#FBP (http://www.jpaulmorrison.com/fbp/CsharpFBP-2.2.zip) has been put on my web site and on SourceForge, which includes subnet support; plus the component priority function has been changed to use attributes, e.g. [Priority(ThreadPriority?.Highest)]. 2008-03-31
A mini-release of C#FBP (http://www.jpaulmorrison.com/fbp/CsharpFBP-2.1.zip) has been put on my web site and on SourceForge, which has some rather annoying sporadic deadlocks fixed - also a simple concordance application, with key parts of the code contributed by DavidBennett, based on a suggestion by me, which displays the source text and the concordance output in TextBoxes?; plus some bars of "Jingle Bells" :-) As stated elsewhere, it uses "attributes" to specify certain characteristics of components. 2008-03-13
I am reaching a point where the current Java and C# open source FBP implementations are nearing completion, and I am looking for experts in these two languages to help me with clean-up and packaging issues. If you want to give a hand, please contact me at the above email address.
The same thing is true for a diagramming tool, written in Java, which is able to generate (almost all of) a running FBP network. However it is not clear to me whether it should be open source or not - again feedback would be welcome.
Some implementation issues that I would like answers to are listed in ImplementationIssues.
I have also been wondering whether it would be useful to indicate the "looper/non-looper" attribute of components in the metadata. This would not affect the dispatcher logic, and indeed there are intermediate states ("partial looper"), but it might be nice to be able to display this attribute in a diagramming tool. Feedback would be appreciated!
What is now known as "Flow-Based Programming" (FBP) -- it was referred to previously as Dataflow Programming, but this term was felt to be too general -- was invented/discovered in 1969-71 by PaulMorrison at IBM Canada. An early implementation of these concepts has been in continuous use at a major Canadian bank since around 1975, running some very large batch applications, but it has remained largely unknown in the industry as a whole, although it has recently started to attract more attention as a result of PaulMorrison's 1994 book [ISBN 0-442-01771-5 (amazon.com, search)], and his web site (http://www.jpaulmorrison.com/fbp/).
Flow-Based Programming is an approach to developing applications, not in terms of the old von Neumann paradigm, but based on the concept of multiple asynchronous processes communicating by means of streams of data InformationPackets? passing over BoundedBuffer connections. In addition, a process identifies its related connections via Port names, rather than directly, and the network defining an application is defined externally to the internal logic of the processes. An application is viewed as a system of data streams being transformed by processes, rather than as a single "dumb" actor doing one step after another until all the data is processed. By "system", we mean that lots of processes are proceeding concurrently - it is the BoundedBuffers that make sure that all of this can still be done using finite resources. However, this "mechanism" quickly recedes into the background, and the developer just thinks in terms of flowing data streams.
This approach involves a paradigm shift which profoundly changes the way developers look at applications, and, although this concept is similar to ideas current in the area of distributed and parallel systems, up until now it has not been recognized that it is also an extremely productive approach to improving programmer productivity and application maintainability. The power and ease of application development and maintenance using FBP is an EmergentProperty of the attributes that make up FBP.
Most programming approaches, and discussions about programming, seem to assume a VonNeumannArchitecture, although people are beginning to network separate machines by means of data streams. However, it is very difficult to build complex applications on a (single) von Neumann machine, especially large data processing applications. Conversely, it turns out that, if you break your application up into small "machines" communicating via data streams travelling across BoundedBuffer connections, these kinds of applications suddenly become much simpler.
The addition of the concept of Port in turn allows each "machine" to potentially become an individual reusable component as it doesn't know the name of the other machine(s) it is communicating with - see ConfigurableModularity. This in turn means that the application becomes essentially a list of connections, which can be generated by a graphical tool if so desired. A "connection engine" (scheduler) is used to relate port names to the real network, and to drive the individual processes.
The connections constitute a DirectedGraph, usually acyclic, but not necessarily. Here is (a subset of) the network for a small interactive application - more information about it can be found in BrokerageApplication:
Wikipedia article: Flow-based_programming - feedback would be appreciated!
C2 (Ward's) Wiki page: FlowBasedProgramming - also has a link to a comparison with the Actors model - http://www.c2.com/cgi/wiki?ActorsModel (do find on FBP).
Patterns in FBP - some examples are described in Collate, ComplementaryComponentPairs (TelegramProblem), SubNets?, DeadLocks? (an AntiPattern), and a general page is PatternsInFBP, which includes a pointer to my recent paper. See also RichardVeryard's work on patterns in http://del.icio.us/dontpanic2/pattern.
For an example of an application which is very natural in FBP and CoRoutine based technologies, but extremely difficult without them, see SameFringeProblem (do a find on FlowBasedProgramming).
Related FBP topics: AsynchronismInFBP, StorageManagementInFBP, CoordinationLanguage, PortsAndConnections, ParametrizationOfComponents, FutureDirections, BrokerageApplication, FlowBasedProgramming, ExecutionEnvironments.
Comega:Microsoft Research is now working on a derivation of C# that has a lot of attractive features, including support for streams - see http://research.microsoft.com/Comega/
Patent applied for: http://www.freshpatents.com/Method-and-system-for-improving-performance-and-scalability-of-applications-that-utilize-a-flow-based-programming-methodology-dt20051229ptan20050289505.php?type=description, dated 28 Dec., 2005, #20050289505, by StanWilliams. Good luck, Stan! Should be interesting!
I also came across http://www.freepatentsonline.com/6282697.html, filed by Wylci Fables and Jore Park, in 1998 - not sure what its current status is.
For more on FBP's legal status, see LegalStatus. Based on this, I would say that Stan's work is definitely non-obvious, and therefore very worthy of being patented. Since Stan has also extended the FBP work in novel directions that give very high performance, his work should definitely be looked at by anyone who has large volumes of data to process.
Review: Nice review of FlowBasedProgramming (the book) by LispMeister? - http://lispmeister.com/blog/books/fbp.html
Recent article: http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.html ("The Problem with Threads" by Edward A. Lee)
From the same institution, one of Bertram Ludäscher's many excellent publications: http://chess.eecs.berkeley.edu/publications/talks/04/ludaescher9-28.pdf
Just came across http://denisbider.blogspot.com/2006/09/function-calls-considered-harmful.html - wonder what happened to his "Flow" project?
For projects that seem to be related to FBP, see FlowLikeProjects.
There is some interesting stuff being written about Python, esp. StacklessPython, which seems to be developing commonalities with FlowBasedProgramming - see CarlosRibeiro?'s blog quoted in StacklessPython. However, for more reading on this topic, see GeneratorsAreNotCoroutines.
Interesting discussion of FBP and Unix pipelines in http://lambda-the-ultimate.org/node/1210.
For a pretty complete 2004 survey of the literature on dataflow programming languages: http://portal.acm.org/citation.cfm?id=1013208.1013209 .
XML stuff: http://xpipe.sourceforge.net/ - just stumbled across this one!
The following companies are selling products based on FBP concepts: http://www.daxisoft.com/, http://www.trelliswerk.com, http://www.protosw.com.
FBP and hardware: TheConvergence - thanks, SteveTraugott. Seems like these concepts are starting to pick up steam - see, e.g.
http://fractal.objectweb.org/ - very interesting! And it's been around since 2002!
FBP seems to be an implementation of a ServiceOrientedArchitecture.
Java has added some nice concurrent features - see java.util.concurrent in 5.0. Unfortunately they turned out to be too much trouble to use for FlowBasedProgramming. If you run across statements saying that I am using them - that's not true any longer (except for CountDownLatch?).
Vorlath: self-described software developer and futurist. Lots of material: check out http://my.opera.com/Vorlath/blog/ for his insights into the future of software development, especially with respect to his Project V.
See also Vorlath's comments for a free-wheeling discussion.
Some other people who IMO are tackling the fundamental questions of programming:
And now... Art Division of FlowBasedProgramming:
A couple of relevant (?) cartoons from "the old days":
See also WikiStructures for a discussion on using wikis for such things as FBP diagrams, org charts, etc. See also the description of <presub>...</presub> below.
WikiNode: FBPWiki is now officially a WikiNode - thanks TedErnst!
For answers to all your questions, visit WikiPedia's HomePage!
One difference in this wiki from the basic UseMod package is that InterWiki links are displayed in green, without the site indication (although you see them as links when you move the cursor to them). When editing them, use the convention 'site:XxxYyy', where 'site' is Wiki for the c2 wiki (Cunningham and Cunningham) (see FrontPage on c2.com), UseMod (see UseModWiki) or MeatBall with capital B (see MeatballWiki). If you wish to add other InterWiki links, please let PaulMorrison know.
I have also added a pair of tags <presub>...</presub> which behave like <pre>...</pre> except that WikiNames, italics and bold are supported. We plan to use this for various kinds of program documentation. See TelegramProblemWR for a simple example.''
Please add your name to the VisitorsList (consider using wiki-style names).
Wiki Spam is becoming a major headache. I have regretfully decided to implement an edit password as a means of defending against spambots, using a simple CaptchaTest. If someone uses this to vandalize this wiki, they will rapidly find themselves on the banned list. It will be interesting to see how many real humans continue this destructive and stupid practice - especially since it now no longer improves the Google ranking of their sites...
More information can be found on WikiSpam.
DrawFBP is a graphical FBP design tool - it can capture much of the information needed to build an FBP network definition. It has a number of icons which can be connected - in particular, a "composite" process that can be clicked on to reveal a subnet with "sticky" connections.
Within IBM, involving PaulMorrison:
Outside IBM - all available on my web site, and (except for THREADS) on SourceForge:
APIs for THREADS, JavaFBP and C#FBP: FbpApi
Actually, a lot of the discussion is taking place on Ward's Wiki. DavidHopwood has done a great job of refactoring our discussion on the differences between FBP and Actors - see ActorsModel.
(Not sure where these came from!)
![[Home]](/wiki.gif)