Paul, could you please look at Erlang? I'm just wondering if I should continue developing Fluix or start developing in Erlang. Do you think Erlang captures all the FlowBasedProgramming concepts that we want? Please advise. Thanks! -- RobbertVanDalen
I took a quick look at it, and I have a few points:
Paul, thanks for your response! I think you make very valid points (especially about the missing Port concept) so I'm more confident that I should continue developing Fluix!
Sorry for barging in, but when you mention transactions, what type of transactions do you mean? I've been part of designing Erlang-based systems that easily handle 500 transactions/second (in this case, setting up + tearing down voice or multimedia sessions). That would make some 42 million transactions/day on a fairly standard 1 GHz machine. Erlang is really quite fast in terms of handling thousands of processes with lots of message passing. Also, due to distribution transparency in message passing, you can rather easily grow an Erlang system by adding more computers.
500 transactions/second sounds pretty good! I withdraw my doubts about its performance! --PaulMorrison
Regarding interfacing different languages, Erlang talks to the outside world via "ports", which look to an Erlang program pretty much like another Erlang process. We use this a lot in testing, where we can insert an Erlang process in place of what later becomes an external program. The rest of the application will not know the difference. For this reason, Erlang is often used very successfully as a coordination and supervision language, tying together components, perhaps written in other languages. We often call it a "systems programming language". Anyway, I believe that practical experiences from Erlang programming serve to validate the concepts of FlowBasedProgramming, despite subtle differences. -- UlfWiger
By the way, Ulf, I was very pleased you barged in! Feel free to expand on this discussion, or even do some factoring! --PaulMorrison
Now I'll barge in :-) Hi Paul, I really think you should take a hard look at Erlang - it fits very nicely with your way of thinking. What probably does not come over so well in the Erlang books and papers is the notion that the Erlang universe is populated with large numbers of communicating black boxes. Erlang is a concurrent programming language, it happly supports hundreds of thousands of concurrent processes. Erlang's processes are your black boxes. An important Erlang detail is that black boxes (processes) cannot share data structures - this is impossible in Erlang. Everything takes places using asynchronous message passing. There are (basically) three primitives in the language. Spawn creates a new process. Send sends a message to a processes. Receive waits for for a messages. That's it.
Erlang also has a few extra notions, black boxes (processes) can observe each other for error handling purposes. If a black box dies (by evalutaing invalid code, example an untrapped divide by zero) a set of monitoring back boxes receive exit signals from the box that died.
The black boxes obey the principle of isolation, incorrect code in one BB should not effect what is happening in other BBs. My Ph.D. thesis (a great read :-)  describes this is mind dumbing detail.
I am now of the opinion that we need several layers when building a system. Inside Bbs we have regular programming languages. Outside BBs we nned language that describe protocols. On top of this we need a "plumbing language" that describes the component configuration. This top level is best suited with a graphic language.
Erlang is great for the bottom level langauge. My UBF is an attempt at the middle layer (I suggest you read http://www.sics.se/~joe/ubf/site/ubf.pdf) and see also  - this describes "contracts" between pairs of black boxes. Your flow language looks nice at the top-layer.
I think it would be quite fun to try and fit this stuff together -- JoeArmstrong
I have been reading your work with more and more amazement - of course you're right. On Friday I suddenly realised why you were right - this came like a blinding flash of white light - I was so happy. Let me try this argument on you to see if it fits in with your way of thinking.
You claim (and have observed) that a pure flow style of programming leads to clear easily reusable programs - I asked "why is this?" - I think the answer is as simple as it is profound.
You forbid remote procedure calls across component boundaries.
This simple observation seems to be the key to everything.
Now why is this desirable? - I think it has to do with the error and concurrency characteristics of an RPC.
Imagine a single machine N with subroutines P Q R S ... where P calls Q and R calls S etc.
If The entire machine fails in a call to S then R cannot trap the error in S because it itself has also failed.
Suppose now we make some call P' on a new machine N'. P' is the mirror of P on N'. Thus on N' we can call P on N by calling P' on N' - this is the remote proceedure call (RPC) mechanism.
On N' we marshall the arguments to P then send them to the machine N. On N we unmarshall the arguments to P, call P, and send the return value back to N'. Suppose P fails, now the calling machine will not get a return value - note that the semantics here are *completely different* to the single machine case since the calling subroutine does not crash as well (in the single machine case everything crashes)
RPCs are supposed to provide the illusion that calling a proceedure on a remote machine has the same net effect semantics as if the proceedure had been called locally. This is *only* true when there are no failures.
Failures of RPCs are a nightmare for distributed system designers, we often don't know if the communication link between a machine broke or if the remote machine broke - we don't know if the command got to the machine and was executed, or if it never got there in the first place. We have to ensure idempotence etc. - all of this is painfully difficult.
PRCs provide a "99.9%" abstraction - i.e. they are useful most of the time - but they are never 100% when they break - ouch - so in your banking system where you want 1000% accountability ("we count all the IPs") using RPCs is highly problematic.
RPCs also destroy the concurrency between components - If component A does an RPC in B then A must wait until B is ready. A failure in B before an RPC has completed will lead to a synchronisation error in A which is difficult to treat.
In a sense RPCs "go against the flow" - and messages arriving before an RPC has completed have to be queued etc. The flow based programming model avoids all these problems by construction - just don't use RPCs at all - brilliant.
The entire software industry has taken a wrong turn with java remote method evocation, Corba etc (this makes me sound like a madman here :-) [Not in the least! --pm] - just for fun I started redesigning some applications *without* rpcs at a component level - the designs suddenly simplified at the expense of slightly more components - but the nice thing was you could see in the diagrams how the data swirled through the components.
I am very excited about this and am designing a graphic front-end to Erlang to allow me to express flows.
For years I have imagined components like hardware chips on a bus - the vital key missing step is to forbid RPCs across component boundaries. Doing this makes software like hardware - we can start becoming chip users instead of chip designers.
I think I shall try to write a book about this...
I'm beginning to suspect that, actually, RpcIsFbp -- a poor implementation of FBP, but still a special case of FBP. I think the key might be in where the references to remote methods are defined -- if they're hardcoded it's RPC, if they're externally defined (and probably redefinable at runtime) then it's FBP. --SteveTraugott