* May I request some comments on MapReduce? This has caused quite a fuss, but realistically this is 100% flow based. I don't see it as functional at all. If you pass a set of data which may be infinite to a component, this will process the entire data set in a much more powerful way than map or reduce in functional languages. At each step, the data can be passed on to something else. If MapReduce is so powerful where you can take an action on every item, what does that say about flow based programming? -V
** I agree - the magic seems to be in how they assign a "map" function to different worker machines, and probably also for "reduce" (which is, as they say, less easy to parallelize). Look at the first diagram in the page called DrawFBP - treat S1, S2 and S3 as separate machines, accepting the mapped function as a callback, and you basically have the Map part of MapReduce. At a higher level, make that whole diagram a subnet, perhaps with dynamic adjustment of the number of Si machines, and you have a one-stream-in, one-stream-out "mapping" component, with a lot of the details private. What they have done IMO is to separate the high-level description further from the actual implementation level, so that users can do a lot of the design at the highest level, and trust the implementation to take care of the details. Nice, but not a paradigm shift over FBP! Thanks, Vorlath! By the way, "map" and "reduce" have exact counterparts in APL also. -pm
|
There appears to be a great convergence, decades in the making, that the computing industry is about to be caught up in, and it's going to take place in the next few years.
FlowBasedProgramming, electronic hardware design, and the features of at least one popular contemporary scripting language are all aligning quite eerily. The result is going to be a little hard to predict, but the scale is about what we saw with the transition from procedural to OO programming; OO was mostly driven by GUI development but spread everywhere; this shift is going to be driven by things like reconfigurable hardware but have the same broad reach. Better hang onto your hats folks, this is going to be a wild ride, and FBP's lined up to shoot the rapids right down the middle. It's been a long haul Paul, but you done good. ;-)
I'm not going to detail this very well on the first try. I'll tweak this over the next few months, but anyone else should feel free to dive in if you have a link or an example handy...
In no particular order:
- Open-source schematic capture software (e.g. gEDA) has gotten pretty good. It should now be possible to use existing tools to graphically create FBP applications in which FBP components are represented in the schematic the same way you'd show any other electronic components. You can then use those tools to automatically generate a SPICE netlist from the schematic, and then, with relatively simple code, convert the SPICE netlist into an FBP network description in your chosen language. Let me say that again -- you can use existing tools, right now, to graphically connect FBP components in a schematic the same way you'd draw the connections between, say, integrated circuits. Those same tools will generate a SPICE netlist for you. The only piece of code missing is a small Python or Perl script to convert from SPICE to your top-level FBP application code or configuration file. You might even write your FBP application framework so that it reads the SPICE file directly...
- Thanks for the feedback - that sounds great! By the way, did you look at DrawFBP? I added an XML generator to it, so one should be able to do something similar with that... --pm
- Yes, did look, but we're a UNIX/Linux? shop, etc. The appealing thing about the idea of using arbitrary EDA tools is that then people might pick their preferred tool from among many, for their preferred OS/hardware platform, with their preferred GUI features, and still get the same intermediate SPICE file out... --stevegt
- SPICE appears to link component ports using numbered connections, with ports specified positionally. This is similar to the first FBP implementation we ever did, but we now feel named ports and array ports are more flexible, and it's not clear to me how you would specify them in SPICE. My feeling is that XML would be a more "universal" way to go, if we could get some agreement on what tags to use... --pm
- Python's generators are just what the doctor ordered for writing FBP components. If you haven't seen these yet, there is an excellent [tutorial] in the MyHDL manual. In the Python community over the last several months there has been a groundswell of interest in generators, and the co-routines, microtasks, and other things you can do with them. Most of the kinetic energy of the stackless Python effort looks like it's going to migrate over to generators as well, since generators don't require a hacked interpreter. The one thing which all of the clamor for co-routines and stackless is missing is any sort of structure for what you do with them when you *get* them; piles of write-only code are the usual result. FBP is a great framework for dealing with this.
- Take a look at VHDL and Verilog if you haven't already, then take a look at MyHDL; the latter is a hardware design and simulation library written in, again, Python. Of special note to FBP folks, take a look at the greetings() example on the [Parameters, instances and hierarchy] page of the manual. This is a great example of a composite FBP component -- and the configuration of the subnet is as plain as day.
- FPGAs are bringing chip design to the open-source world -- see [the projects page] on opencores.org. Languages fell "open" first, with C, then peripheral architecture with the PC platform, then networking with TCP/IP, then the O/S with Linux, and the CPU is next. A new breed of hobbyist is emerging; software geeks who create hardware. They code in VHDL, Verilog, SystemC, or MyHDL, and create not only application-specific chips, but chips which can be reconfigured at runtime -- ConfigurableModularity which transcends the hardware/software interface. We know from past experience that, in this industry, this year's hobbyists are next year's professionals. The difference between hardware and software hacking is going to start getting really blurry. Who remembers when it went the other way; when Popular Electronics dried up because their readership all turned into software geeks? The pendulum is swinging back... If you really want to get FBP goosebumps, see the opencore [Data Flow Processor].
- Moore's law is tapping out, finally, and we're going to see more and more multi-core CPUs showing up over the next few years. This means parallel programming and loose coupling start getting more important for the average application developer; with FBP you get those for free. By the way, many of those multi-core designs are going to throw in an FPGA or two for good measure...
- The nice thing about FBP is the mobility of components. Have a slow-running component that's written in Perl, Java, or Python? Rewrite in C and move it out of the interpreter. Still too slow? Rewrite in VHDL and move it to a spare corner of your FPGA.
- Also converging: Games. Simulation systems. Physical modeling. Financial modeling. Lots of demand, even during the downturn. And lots of hunger for raw hardware performance. And getting very complex. But the complexity stays simple when you keep coupling loose.
- Bioinformatics, another source of demand. My own variation of FBP technique has for years used Lincoln Stein's [BoulderIO] data interchange format, which he originally developed for gene sequencing.
- Web services. Or what people want from web services, but can't get without looser coupling. FBP again.
- And last but not least, SOX!!!! Sarbanes-Oxley. Workflow. Corporate accountability. Visibility of business processes and data flow, processing, and storage. You know what? The neat thing about FBP is that, when designing a workflow system, either a human or a piece of software can be an FBP process... If none of this means anything to you, then you aren't working in a U.S. public company, you don't plan a U.S. IPO, or you've been hoping it all goes away. In the wake of Enron, MCI, et al, the government oversight that used to be reserved for corporate officers and finance has now been quadrupled, extended into the I.T. organization, and applied to in-house apps and any other means of data processing and storage -- do the books right, keep controls in place, don't lose data, and show that you're doing it. This is being phased in slowly, over the course of a few years, so most people who are affected by it are like the frog-in-boiling-water. But the overall scale is similar to Y2K in terms of the amount of work that needs to get done. And a great deal of it is application development. And again, FBP is in the sweet spot, with its ease of auditability and rapid response to changing requirements. Right now MQ has the high ground here, and there are no clear open-source alternatives. Yet. But then again there's Python...
-- SteveTraugott
- May I request some comments on MapReduce? This has caused quite a fuss, but realistically this is 100% flow based. I don't see it as functional at all. If you pass a set of data which may be infinite to a component, this will process the entire data set in a much more powerful way than map or reduce in functional languages. At each step, the data can be passed on to something else. If MapReduce is so powerful where you can take an action on every item, what does that say about flow based programming? -V
- I agree - the magic seems to be in how they assign a "map" function to different worker machines, and probably also for "reduce" (which is, as they say, less easy to parallelize). Look at the first diagram in the page called DrawFBP - treat S1, S2 and S3 as separate machines, accepting the mapped function as a callback, and you basically have the Map part of MapReduce. At a higher level, make that whole diagram a subnet, perhaps with dynamic adjustment of the number of Si machines, and you have a one-stream-in, one-stream-out "mapping" component, with a lot of the details private. What they have done IMO is to separate the high-level description further from the actual implementation level, so that users can do a lot of the design at the highest level, and trust the implementation to take care of the details. Nice, but not a paradigm shift over FBP! Thanks, Vorlath! By the way, "map" and "reduce" have exact counterparts in APL also. -pm
The following section moved to CellArchitecture.
Codito looks like they are taking aim at that spot! See http://www.codito.com/prodtech_framework.html. Watch out - India is coming up fast on the outside!
![[Home]](/wiki.gif)