THREADS API and Network Specification

This chapter has been excerpted from the book "Flow-Based Programming: A New Approach to Application Development" (van Nostrand Reinhold, 1994), by J.Paul Morrison.

A second edition (2010) is now available from CreateSpace eStore and Amazon.com.

The 2nd edition is also available in e-book format from Kindle (Kindle format) and Lulu (epub format).

To find out more about FBP, click on FBP.

For definitions of FBP terms, see Glossary.

The THREADS package, mentioned in the 1st edition, was converted to use Windows "fibres" and C++ a few years ago. However, fibres did not seem sufficiently general, so this package is now in process of being converted to use Boost, to take advantage of Boost multithreading. This process is ongoing (as of May 2014), and is described in CppFBP and Lua Interface. The earlier, fibres-based, version is still available as a zip file.

Material from first edition of book starts here:

In Chapter 23, we described the THREADS notation for linking components into networks - this will also be repeated below in summary form. However, perhaps even more important are the conventions for writing components, as these are what will allow independently developed components to be combined into a single network without the requirement that they be developed by the same person or even in the same location (apart from the normal requirements for consistency of IP and stream formats).

Here is a simple component written in C++ to copy IPs from IN to OUT. Let's call this rather unoriginally ThCopy.

A THREADS component is a regular C++ subroutine, so, as you might expect, the first few statements are the declares.


#include <string.h>
#include "compsvcs.h"

THRCOMP ThCopy(anchor proc_anchor)
{
	void *ptr;
	void *ptr2;
	int value;
	long size;
	char *type;
	port_ent port_tab[2];

	value = dfsdfpt(proc_anchor, 2, port_tab,"IN","OUT");

	value = dfsrecv(proc_anchor, &ptr, &port_tab[0], 0, 
                   &size, &type);
	while (value == 0) {
		value = dfscrep(proc_anchor, &ptr2, size, type);
		memcpy(ptr2,ptr,size);
		value = dfssend(proc_anchor, &ptr2, &port_tab[1], 0);  
		if (value != 0) {
			dfsdrop(proc_anchor, &ptr);
			return(1);
		}
		value = dfsrecv(proc_anchor, &ptr, &port_tab[0], 0, 
                    &size, &type);
	}

	return(0);
}

The first action is always to initialize the port table using a call to dfsdfpt - the 2nd parameter of the call to dfsdfpt must match the dimension declared for the port table. This number must also match the number of port names specified in the parameters to dfsdfpt . Note: dfsdfpt modifies the port table, so this must not be defined as constant.

The return statement causes deactivation (but not necessarily termination, as described above). The return code value on the return statement determines whether the process is willing to be reactivated: a value of 5 or greater forces termination even if data is available.

API Calls:

Here are the API calls available to components in THREADS:

Create a Packet:
  value = dfscrep(proc_anchor, &ptr, size, type);

Define ports:
  value = dfsdfpt(proc_anchor, port_count, port_tab);

Receive an IP:
  value = dfsrecv(proc_anchor, &ptr, &port_tab[port_no], 
           elem_no, &size, &type);

Send an IP:
  value = dfssend(proc_anchor, &ptr, &port_tab[port_no], 
           elem_no);

Drop an IP:
  value = dfsdrop(proc_anchor, &ptr);

Pop an IP off the stack:
  value = dfspop(proc_anchor, &ptr, &size, &type);

Push an IP onto the stack:
  value = dfspush(proc_anchor, &ptr);

Close a port element:
  value = dfsclos(proc_anchor, &port_tab[port_no], 
           elem_no);

Parameters:

Note that the ptr, size, type and port table element parameters all have &'s attached, except in the case of dfscrep , where only ptr has it. The &'s are required because these parameters may be modified, and C parameters are all passed by value (except for strings and arrays). In some cases a particular service does not set them, but for consistency &'s are used for all of them (except dfscrep ).

port_count and elem_no are all binary integer variables (int). size is binary long. dfscrep is restricted to a maximum of 64000 bytes.

proc_anchor: a variable of type anchor (defined in thxanch.h)
ptr: a void pointer used to point at IPs
size: a long variable containing the IP's size
type: a null-terminated string of up to 32 chars
port_count: parameter to dfsdfpt - must match the dimension of the defined port_tab array for this component
elem_no: this specifies the element of the appropriate port to be used in a send, receive or close. These are only required for array-type ports, where they number up from 0. For non-array ports, this parameter must be 0.
port_tab: this parameter is declared as an array of type port_ent (see below), where each element corresponds to one of the ports known to the component; the whole structure is passed to dfsdfpt, while specific elements of port_tab are passed to send, receive and close using &.
port_name_n: port name to be used by dfsdfpt; the number of these must match the port_count parameter

All pointers in the following declarations are "far" pointers.

Declare for `port_ent:`

  struct _port_ent
   {
      char port_name[32];
      void *reserved;
      int elem_count;
      int ret_code;
   };

   typedef struct _port_ent port_ent;

After a call to dfsdfpt, elem_count in each port_ent instance will be set to the number of connected elements for that port, and ret_code will be set to 0 or 2, depending on whether that port was connected or not.

Declare for anchor (`thxanch.h`):

struct _anchor {
    int (* svc_addr) ();
    void *proc_ptr;
     } ;

 typedef struct _anchor anchor;

Service return codes:

dfscrep:
	0	OK

dfsdfpt:
	0	OK - but some elements of port_tab may have their
		ret_code fields set to 2, meaning "port name not
		found"; a receive or send from or to such a 
		port_tab element will result in a return code value
		of 2.  

dfsrecv:
	0	OK
	1	port element closed (end of data)
	2	port element not defined or not connected

dfssend:
	0	OK
	1	port element closed 
	2	port element not defined or not connected

dfsdrop:
	0	OK

dfspop:
	0	OK
	2	stack empty

dfspush:
	0	OK

dfsclose:
	0	OK
	1	port element already closed 
	2	port element not defined or not connected

Limitations:

There is a limit of 4000 elements per array port.

Working storage for a component should not exceed a few thousand bytes (including the working storage of any subroutines it calls). If the component needs more than this, it should use one of C's dynamic allocation functions (e.g. malloc or calloc) to allocate the additional storage.

Network Notation

Networks are defined initially using a free-form notation, described briefly in Chapter 23. For instance,

'data.fil'->OPT Reader(THFILERD) OUT -> IN Selector(THSPLIT) MATCH -> ...,
Selector NOMATCH -> ...

The general syntax for networks is quite simple, and can be shown as follows (using a variant of the flow notation which has started to become popular for defining syntax):

I have labelled the ports above and below the connection indicator (arrow with optional capacity figure) "up-port" and "down-port" to indicate that they are upstream and downstream, respectively, of the connection.

The main network may be followed by one or more subnets, which have basically the same notation (each one starting with a label and finishing with a semi-colon). However, subnets have to have additional notation describing how their external port names relate to their internal ones. Since this association is like an equivalence, we use the symbol => to indicate this relationship. Thus,

, port-name-1 => port-name-2 process-A port-name-3,

indicates that port-name-1 is an external input port of the subnet, while port-name-2 is the corresponding input port of process-A. Similarly,

, port-name-1 process-A port-name-2 => port-name-3,

indicates that port-name-3 is an external output port of the subnet, while port-name-2 is the corresponding output port of process-A. For an example, see the example of subnets given in Chapter 23.

Other syntactic elements:

Two consecutive quotes are taken to mean a single quote.

element-number does not apply to the external port names of subnets. The asterisk indicates an automatic port (see Chapter 13).

capacity does not apply to the external port names of subnets.

The component name can be specified on any occurrence of the process name. The question mark indicates that tracing is desired.

In Chapter 13 we alluded to the fact that we can specify that a process "must run at least once" - this is specified by means of an attribute file for the component, which has the name of the component, and an extension of atr, e.g. thcount.atr. These files must be provided by the supplier of a component, and must have a specific format. So far, only the "must run" attribute has been defined - it is specified by coding one of the character strings must_run, Must_run or MUST_RUN, with optional preceding blanks, in the attribute file for a given component. If no attribute file is found for a component, default values are assumed to apply (the default for the "must run" attribute is "need not run").

As mentioned in Chapter 23, there is also a compiled (or compilable) format for specifying networks - while this is obviously not appropriate for hand-coding by human users, it is easy enough to generate by means of software. It is a data-only C program, and specifies a network, together with all referenced subnets. Its importance is that this will be the source form for networks owned by customers, so THREADS must guarantee that any enhancements to it will be upwards compatible. These guarantees are essentially encoded in the C headers which it uses: thxiip.h, thxanch.h and thxscan.h.

thxanch.c has been given above. The other two are defined as follows:

thxiip.h

struct _IIP
   {
      int IIP_len;
      char datapart[512];
   };

typedef struct _IIP IIP;

thxscan.h

   struct _cnxt_ent {
     char upstream_name[32];      /* if 1st char is !,   */
     char upstream_port_name[32]; /* connxn points at IIP */
     int upstream_elem_no;
     char downstream_name[32];
     char downstream_port_name[32];
     int downstream_elem_no;
     union cnxt_union {IIP *IIPptr; void *connxn;} gen;
     int capacity;
     };

typedef struct _cnxt_ent cnxt_ent;

struct _label_ent {
	char label[32];
     char file[10];
	struct _cnxt_ent *cnxt_ptr;
	struct _proc_ent *proc_ptr;
     char ent_type;
	int proc_count;
	int cnxt_count;
	};

typedef struct _label_ent label_ent;

   struct _proc_ent {
     char proc_name[32];
     char comp_name[10];
     int (*faddr)();
     void *proc_block;
     int label_count;
     int trace;
     int composite;
     int must_run;
     };

typedef struct _proc_ent proc_ent;

Note: the above structures are not the internal control blocks of THREADS - they are an encoding of the free-form network specification notation. This separation will allow THREADS to be extended in the future without requiring network definitions to be reprocessed.

Flow-Based Programming - AppendixTHREADS API and Network Specification