John scripsit:
I am a 20 year old design and technology student attending Parsons School of Design. Over the past few years, I have been doing a lot of work creating programmatic/generative art. My tool of choice has been Max (http://www.cycling74.com/products/maxmsp.html). Max is a graphical programming environment. As my work has progressed, I have grown more and more frustrated with the limitations of Max. Seeking to create an alternative, I stumbled across Io (http://www.iolanguage.com).
To sort of cut to the chase, Max is somewhat similar to a FBP language, except it only has what amounts to a "single dumb actor". One night when contemplating how I wanted to shape my language, I thought that allowing multiple objects to process simultaneously and then output from their ports when they finish asynchronously would be a good way to take advantage of multiple processors. Lucky me, this idea could very feasibly be implemented in Io because it is an actor language with coroutines. (Io has many other features which fit closely with what I want to implement as well, namely the prototype- based aspect of it.) This realization has me rethinking my current sketch of the language, which is more of a "single dumb actor" which runs through every branch until it has nothing else to do, then starts again on the next branch from the point where they diverged. (My rough idea for implementation was that an object with nowhere to send to would tell the object above it, and if that object had no where to send to, it would tell the object above that, and so on, until an object that had data left to output was informed it should send.)
Here is a rough bit of example code. It sends out pulses at a random rate. Each object sends to the object to the right of it. Each object has a default inlet and a default outlet, and to send to another you can prefix the port name with a dot (to mean "sends to") or a comma (to mean "receives from"). Parenthesis are used for branching and merging. Objects can be named to allow for easy referencing later which can help ameliorate the describing of more complex systems. I'm quite excited with what I have so far (I've generated dozens and dozens of pages over the past few months). If you would like to see more examples, I'd be glad to show you.
object/random_banger_w_delay [
![[Home]](/wiki.gif)
in/on_off (
.set gate/g
sel[1] r (!! d | .set delay/d) g (out/random_bang | !! r)
)
in/set .set random/r
]
The syntax is tentative now (and I don't expect you to understand it), but I think I have the general concepts set. Or at least I thought I did, until I started thinking about allowing all objects to run in their own coroutine, sending forward data whenever they finished processing. While the above syntax still works for describing such a system, I am not sure as to how to go about implementing it. A bit of searching around the web brought up your website. It seems that "flow based programming" is nothing new (drats, I thought I had something there), so perhaps I should look into what has already been done to avoid reinventing the wheel. I read a good portion of your book online (and I will likely order a copy), but I am interested in other resources and/or any languages which currently implement FBP. My goal is to build upon such concepts to implement a new general purpose flow-based language with extensions for OpenGL? (see OpenGl) and audio manipulation. I think the flow-based paradigm would fit *very* well in such an environment, as you often have to deal with streams of data. I suppose the reason I'm writing you here is not so you can point me in a direction so I can find the answers to questions I have, but rather so you can direct me in a way that I will discover new questions I've not thought to consider. I am absolutely dedicated to this, and I would really like to put together something great.
> I am impressed! But a bit lost :-)
Let's see if I can clarify. :-)
> At first I thought you were > designing your own language, but then you mentioned Io, which I hadn't > heard of. Are you extending Io, building a language processor using Io, > or what?
I'm interested in building a language processor using Io essentially I suppose.
> Sorry to be dense! Also, once it is built, are you using it > to output static pictures, or does the output change over time - sort of > like kinetic sculpture? You also mention audio... I seem to remember > Nierstrasz et al. wrote a number of papers on what they called "media > objects", or "knowledge objects" (KNOs)...
It would be used for realtime video and audio synthesis (via Io's OpenGL bindings and Supercollider's scsynth over OSC (CCRMA's Open Sound Control) respectively). Such things are inherently suited to making use of parallelism, hence my interest in dataflow programming. Of course 90% of people only have single processor systems, and for parallelizable things you can always take an SIMD approach, so I suppose it is overkill. I suppose half my rationale is that "I like dataflow programming".
> Re "rather so you can direct me in a way that I will discover new > questions I've not thought to consider" , that's flattering, but I think > I first need to know how your area of endeavour differs from application > or software development (the two areas I have mostly worked in).
Basically I want to create a tool that comes with a large set of objects. The objects would abstract OpenGL commands and audio-related tasks, as well as control sorts of structures. You could then just chain up these objects to create the system. The objects themselves would be written in Io itself (utilizing C bindings when necessary/ desired). I have little to no experience though with flow-style systems where everything is done in parallel and all objects run in their own coroutines (I only know "single actor" flow systems a la Max). I'm not sure how well this will work in practice.
My main issue right now is getting the language design down as best I can (I'm sure I'll want to make some small changes later). I'm sort of struggling as well with the notion that dataflow systems may be difficult to express textually. Graphically, no doubt, it is easier. However, I like text as I want to be able to make structures that are very dynamic. I want them to be able to inspect and modify themselves. Graphically, this is tough, as you have to worry about how the new connections/objects will be represented on a 2D plane. ** It is this existing limitation of Max (amongst many other smaller things) that drives me so nuts.** With text, it is just conceptual, so it doesn't matter. Graphical languages make my wrist hurt. It is also much easier to write a textual environment than a graphical one. ;-)
Right now, 90% of my programming is done in graphical flow-based single actor (I hope I'm using that term correctly) environments. Certainly an odd case I suppose. The deterministic nature of single actor systems though makes things fairly easy, and the graphical nature is nice for representing DSP chains. It is less nice however for expressing complex control structures, and downright worthless for any complex math. I worry adding objects that all process "at the same time" and output whenever will make control structures even more difficult. I assume it will require lots of "syncing" objects to keep things in order. At the same time though, it is a very simple and elegant concept which programmer-artists would like (data comes in whenever and goes out whenever, and that's all you need to know), despite what I'd expect to be its fairly complex implications. I'm not talking chaining 10-20 objects together here. I'm talking about hundreds or thousands of them, with the ability to modify, reconnect, delete, and create new objects on the fly. Such flexibility combined with non-realtime audio/video processing could yield results not currently attainable with any existing software. The question is though, will such a powerful "language" actually be usable? With a single actor graphical flow system (and excuse me if that is an anachronism by the way, but Max/Pd? are often described as "dataflow" languages despite the lack of parallelism), I can say through experience the answer is definitely yes. With a "true" dataflow system though, I think it has the potential to become a nightmare. Then again, if all I cared about was getting work done, I'd use Supercollider (a smalltalk-based language with C-style syntax atop an audio synthesis server) and add OpenGL bindings. I want to do it in a way though that I *love*. Algol-style syntax and procedural/ traditional-OOP programming is too pragmatic/utilitarian to use for art. If the system/program itself isn't beautiful, the result likely won't be either. And I don't mean beautiful in that OpenBSD? C hackery sense either...
Ah, I think I'm failing at clarifying here... ;-)
> If people are > working on wearable computers, maybe you'll bring wearable interactive > art into an awed world... Who said "If it can be done, someone will do > it"?!
Wearable computing is a real button-pushing phrase in many art communities now. So much of it is just complete and absolute crap. The fact that you're wearing a computer doesn't mean you're using "wearable computing". A Timex watch is wearable computing. Strapping a hacked up laptop to your back is just a way of asking to get your ass kicked. I don't want an 3x5" LCD display strapped to my arm thank you.
Sorry... I've seen one too many failed experiments in that field... ;-)
'Nother note from John answering more questions from me:
> I view FBP as having two language levels: the top is basically a list of connections > of the form > > process A port P -> port Q process B > > This lends itself to graphical notations - in fact many of its patterns don't emerge > until you display the list graphically. For an example, see > diagrams in DrawFBP (text wikisized)
This is of course true. What I am wondering is if it is possible to program systems without the graphical notation. Is it too much to keep in your head? It does have some benefits to be sure (less mouse action and wrist pain, arguably quicker to use, easier implementation, more flexible in terms of dynamic creation/deletion/ reconnection, less overhead, easier to interface with other apps, etc). Of course, any graphical system I would devise would have a textual interface underneath for various reasons.
> The processes may be written in any language that can send/receive data chunks - > from/to ports, and create/destroy data chunks. Processes should know about chunk > handles, or could work (more slowly) by copying the chunks. I now think that such > languages should not do multithreading - they should all single-thread, and let the > infrastructure suspend/resume sends and receives. For one such infrastructure, see > THREADS (find THREADS in FlowBasedProgramming)
What I planned on doing was using Io's coroutines, which are essentially very lightweight non-OS threads. It a futures-based system which I find very pleasurable to work with.
> The Java implementation happens to use Java for both levels, so you can do some dynamic > modification of the network, but it has to be done carefully!
Why? What are the problems that dynamic modification of a network causes? Obviously there are issues such as the chain being temporarily broken at some point when data is supposed to pass through, causing things to hang. One solution would be to force all output to queue (or just plain freeze all network processing) when the network is being modified. Am I missing something obvious? The main issue I foresee is dynamic modification interfering with audio processing being completed before the buffer is exhausted, simply due to lack of processing power. Ideally portions of the network should be able to be reconstructed at some lower priority with the assumption that the parts being modified are not currently part of the signal chain.
> Now the processes communicate via way of streams of data chunks, being generated and > consumed asynchronously. Batch jobs start and stop, so processes on the left start and > finish early, while those over on the right start and finish later... For an > interactive application (sketch), see second diagram in DrawFBP - > this one is really transaction-oriented, but it will run forever, as long as > something injects messages in at the top left, and accepts them at the bottom left.
The problem with this sketch in terms of applying it to what I want to do is that it is very simplified and high level. The issue with this sort of asynchronous streaming of data is that sometimes things do need to happen in a specific order. While with high level objects you can work around this fact by taking care of such things within objects, if your language is to be composed entirely from a set of dataflow object primitives (which is what I would like to do), you need to be able to handle such situations elegantly. This is one of the things I'd like to ascertain from you: Is an asynchronous dataflow paradigm suitable for lower level tasks, or is it only useful for assembling very high level objects written in a traditional language? What I do not want to do is require people to write objects in some traditional language and then assemble them in a flow-style network. I want them to be able to construct programs entirely with objects that handle input asynchronously and in parallel, making use of syncing objects when necessary to handle situations when the order of execution is critical. If you do not feel this is a suitable thing to shoehorn this paradigm into, I will likely resort to my other idea of a network where only one object may send at a time, and each outgoing connection from the same "port" has a priority to resolve ambiguities if necessary. The result of such a system is complete determinism (ignoring any areas of the network where you might intentionally delay and defer messages), certainly something that's useful when programming. The downside of course is that the concept is less elegant (for what that is worth, if anything), and more importantly you lose the ability for more than one object to process at a given time (excluding objects that do not stop processing after they output data, or objects that output continuously, and even in this case, you'd need some sort of scheduler to defer to them when possible). Hm.
> Re art, I dunno! I would visualize one process that takes chunks of a picture using > randomly selected coordinates; another to invert the colours; another to do visual > transforms; another could take sampled music and turn it into colours; and the data > streams all flow from one to another undergoing various transforms: you make your > processes parameter-based, and then either hardwire the parameters, or feed them into > an input port. If this isn't exciting enough, you could periodically scramble > the connections :-)
This sort of thing is what I am doing now with Max/MSP/Jitter?: http://cycling74.com -- In fact, many other programs such as PureData work under similar principles. If you are not familiar with these, you owe it to yourself to check them out. They are domain specific (for the most part), but I think you'd still find them interesting. Besides, Max is a ton of fun to play with.
Here is a screenshot of the code for part of a synthesizer I did for amusement in Max (I am not usually a straight ahead synth sort of person, but I wanted a break from generative nonsense):
Perhaps that will help explain a bit the angle I am coming from. ;-) If you'd be interested in seeing some examples of how to do common programming tasks using single actor network dataflow primitives a la Max, let me know.
> BTW When I was talking about wearable art, I was picturing some kind of crystal that > would sit on the intersections of a wire mesh, and change colours depending on energy > flows through the mesh. Then makes clothes out of the stuff ;-) I read somewhere > that someone is working on a fabric that will turn sunlight into a trickle of > electricity, so you put some of that stuff on the sleeves ...
Given the propensity for people apt to wear such things not to go outside, that may have trouble finding its market niche. ;-)
A further note from John (turns out he and MathieuBouchard know each other):
PureData is more the thing I am trying to get away from. I do not feel that PureData meets the criteria necessary to be considered a "dataflow" system (although I may be wrong). PureData (like Max) is essentially an imperative language with a graphical representation. There is no parallelism, and the order of evaluation is definite. Objects do not queue incoming data, but rather block everything else while they do their thing. It is not similar to something like LabView (G Language) in this regard, nor is it similar to what you discuss in your book.
What I am trying to determine if a program with a similar look and interface to Pd could make use of true dataflow techniques. I made a lot of progress this week actually sorting out ideas as to how this could be done, and how you could handle different conditions where order of evaluation is critical. I'm doing this with a "dispatcher" object and outlets on every object that send out a "done" notice, which then causes the "dispatcher" to send out of its next outlet, and so on -- this odd recursive process is made easy through a big of syntactic sugar. I also have other concepts for grouping, pairing, coupling, etc objects.
Right now, I am representing chains with a text-based syntax.
For example, this structure (note to Mathieu, this doesn't behave as Max would, it is just a picture to represent an asynchronous dataflow system):
Could be represented as:
<< m,2 m,3 >2 ; g + d
The meanings of the objects are not important, and I assure you it makes a lot of sense once you get used to looking at the text representation. It is surprisingly workable (a result of far too many nights up working out a usable syntax). I have many examples written up for dataflow versions of Max example patches, but as Mathieu is a Pd man, I'm not sure these would be helpful for him to see. Here is something that takes care of repeating notes, just for the hell of it:
m@i >>
<< #
'i,2
== sel,1,0
<0 'i
<1 #
m,stop >>
'i
Hideous, I know, but after working with it for some time, I can read it as clearly as English now.
I feel that I am really making good progress here, and may have a working prototype language set atop the actor language Io within the next two months. Should this come to fruition, and should it be usable, I would then like to begin adding multimedia capabilities to the new language through dataflow-style objects wrapping OpenGL?/GLUT commands, as well as objects wrapping Open Sound Control objects which would communicate with Supercollider Server (scsynth).
I feel the possibility for a very elegant language that inherently takes advantage of multiple processors does exist, and I must've put in over 200 hours so far hashing out the details. I realize I am probably reinventing the wheel here in some ways regarding dataflow programming, and I still have to work out how queuing methods will be specified for user-created objects, but overall I think I am getting very close to a workable concept. For example, this works fine (it tests if a number is a power of another, or something... I can't even remember!), but if the inlet sends data through too soon, it can mess up the process:
<< e,$i1<2,false,$i1 sel,false
<1 f@x /,2. #
'x,2
sel,1.
<1 %,2. e,$f1==0,bang,0 sel,bang
<1 >>
<0 'x
<0 m,1 >>
<0 m,0 >>
Mathieu, if you are interested at all, I can discuss with you a bit more regarding what I have been doing. I realize this is far from clear -- I'm in a rush. Perhaps I am alone in thinking that Pd and Max are far from what we're capable of doing, but then again, perhaps others feel the same way and would like to work towards something new. I really want to *love* the tools I'm using to create art, rather than just using the best thing that happens to be out there as I am now. I definitely think if a graphical IDE was added, it could replace Pd. A lot of the hard work is already done on the sound side, as we just need wrappers for scsynth (which is very good, although I despise the language that currently sits on top of it). Certainly not an easy task, but definitely something that could be usable within two years or so.