Syzygy Background

Syzygy Architecture

Syzygy has five layers, from bottom to top as follows. 1, an OS-portability layer. 2, a network communications layer. 3, communication protocols for things like graphics and audio. 4, a distributed operating system that forms things like sound players and remote scene graph renderers ("media objects"). 5, application frameworks for various styles of cluster programming.

Portability Layer

Windows, Unix, etc., differently implement threads, locks, sockets, and serial I/O. This layer hides these differences. This necessarily omits some idiosyncratic features, but that doesn't hurt Syzygy as a whole. Details are in the Portability Layer chapter.

Communications Infrastructure

This layer provides tools for constructing communications protocols that are easy to change, automatically convert data formats, and robustly connect/disconnect/reconnect in an unreliable network environment. A protocol is defined by a dictionary of message templates, which defines the semantics of the sendable messages. Data transfer objects then manage connections and data transfer. Eventually, this layer will contain an abstract interface to the underlying communications medium, allowing the same code to work over TCP, UDP, or shared memory communications pipes.

Media Protocols

Protocols for manipulating media information implement synchronization; distributing and rendering a scene graph; manipulating and playing a sound database; and communicating with input devices.

Network-based synchronization primitives provide "active stereo" swap lock across displays.

The distributed database itself is simple, while the graphics and sound implementations built on it are elaborate.

A master database accepts network connections from remote slave databases. At connection, the master database dumps itself in the slave database, creating two synchronized copies. Subsequent changes to the master database are automatically mirrored to the slave, creating consistency without programmer effort.

The general input device framework lets distinct physical devices aggregate into larger virtual ones. The data streams flowing through these devices can be filtered arbitrarily.

Distributed Operating System

One program, szgserver, manages basic services of a distributed operating system. One instance of this program runs per Syzygy installation. All Syzygy programs connect to it. To avoid bottlenecks, it only sets up connections between elements in the distributed system. Another program, szgd ("Syzygy Daemon"), runs on each computer to provide remote execution services.

The distributed operating system manages processes. Applications can be started or killed on any computer, by typing commands at any other computer. Applications send messages to one another through a standard interface for interprocess communication. A parameter database, managed by szgserver, acts as a config file maintable over the network. This replaces the text configuration files that might otherwise be scattered across many computers.

The distributed operating system supports multiple users.

Details are in the Cluster Mode chapter.

Media Objects

These objects include SoundRender, for the distributed sound database, and szgrender, for the distributed graphics database. They turn information from the network into sensory events.

Application Frameworks

See Programming and Application Frameworks

Goals

Syzygy's first web release was in September 2000. Its goal was to run a wide range of VR applications on a PC cluster, with performance matching shared-memory SMP computers like the SGI Onyx. Since then, it has become portable, robust, and easier to use. The Syzygy application model has also been adapted to diverse environments such as standalone laptops, wide-area networks, and even the older shared-memory SMP systems.

Hardware changes drive software changes. Here, developing for PC clusters requires explicit library support and an updated application model. Monolithic shared memory systems can ignore many data-sharing issues, leaving these to the applications programmer and encouraging machine-dependent code.

Consider the detailed requirements for PC cluster software. Inter-processor communication is a bottleneck. Tools are needed to efficiently manage and develop cluster applications. The developer needs an API that supports complex communications between cluster machines. These information exchanges must be fault-tolerant and conducted in a way that keeps the cluster producing frame-by-frame synchronized views of the virtual world. During the early stages of application development, the application code may be unstable, so the overall distributed system must tolerate the failure of individual components. The system needs the ability to portably launch applications across a wide range of target clusters, both homogeneous and heterogenous in nature. It also needs mechanisms for steering running applications and monitoring tools that allow users to remotely determine its state.

The goal of creating VR applications on a PC cluster is best served by increasing the level of abstraction and writing a toolkit with broader applicability. Syzygy aims to provide simple software tools for building heterogeneous distributed systems. While the system's main focus is PC clusters on high performance LANs, it is also applicable to WAN-based systems. Syzygy is designed to be a lightweight toolkit that can operate on PC clusters, SMP boxes, and laptops, while connecting meaningfully with smaller devices. This presents several challenges. Much system infrastructure is unnecessary for a standalone laptop being used to provide a portable road demo, and, to maintain ease of use, the sytem's operation should scale down gracefully. Furthermore, we need to deploy communication methods suitable for SMP, namely shared memory instead of sockets, and do this while reusing our low-level communications infrastructure. Finally, performance is very important, as many of the toolkit's functions are OS-level and must be efficient.

Syzygy solves several problems: distributed graphics, distributed sound, distributed systems management, distributed data access, and general I/O device management. There are systems out there that do each of these things, but no one system addresses them all. However, each problem relies on a similar infrastructure for its solution. Syzygy aims to identify that common infrastructure and provide an implementation. Thus, each problem can be solved more easily, and the resulting code base is well-integrated, maintainable, and smaller. In effect, Syzygy is a middleware solution for distributed multimedia.

Change Log

Changes in version 1.3

As of revision number 1794 of July '09, the environment variables for building the python bindings have changed. Please read the linked section.

Make now inserts bazaar revision number and other info into a source file, szg/src/language/arVersion.cpp. This requires both bzr and python. If they're not available, set the environment variable SZG_BZR_INFO=FALSE to skip this step.

Added support for alternate input simulators, built as shared libraries and loaded based on a database parameter (SZG_INPUTSIM/sim_type) at runtime. An example is included in szg/skeleton.

Added support for Python input devices. This required changes to the arAppLauncher to allow Python input devices in virtual computer definitions, so all programs will need to be recompiled for any setup where you want to use a Python input driver. Hence the bump to 1.3.

Added the ability to set the application frameworks' navigation matrix by copying a matrix directly from the input event stream. This allows control of navigation by e.g. a Python script.

Re-worked the Makefiles and conditional compilation such that input device drivers are built as shared libraries even if szg is built with SZG_LINKING=STATIC--except with Visual C++. This means that you can load input devices in standalone mode on Linux and MacOS.

Renamed 'python' and 'sip_python' directories to 'python_swig' and 'python_sip', respectively.

Changes in version 1.2

Syzygy 1.2 happened internally.

Previously, application launch on a virtual computer would fail if even one of the constituent computers didn't have an szgd running on it. Now the application will launch on the remaining computers (unless of course the missing szgd was on the computer that the application's trigger instance was supposed to run on).

Bugfixes

Fixed a bug in szgd that could lead to the current working directory not being set properly if an renderer and one or more services were all launched on the same computer.

Fixed a crash bug for daisy-chained input devices (arInputFactory netinput slot #).

utilities/PictureViewer now runs properly on multi-display computers.

Fixed a bug that caused certain path variables to grow over time.

Fixed the documentation stylesheet so that the docs can now be printed legibly.

We've found some bugs in the handling of multiple networks. A bug that occurred when multiple networks were used in a virtual computer, but some computers were only attached to one of them has been fixed.

Previously, if a connection between an szgserver and szgd were broken due to e.g. network unlreliability, the szgserver would notice and remove the szgd from the process table but the szgd would not notice (and hence would not attempt to reconnect). This has been fixed.

Extraction of Euler angles from matrices was incorrect & has been fixed.

Diagnostics

ar_log_error() and company now print a process ID and timestamp.

Compiling

Got things compiling with Visual Studio 8.

Ditto for g++ 4.3.1 (on Sidux).

Added support for joysticks/gamepads when compiling with MinGW. Ditto for speech synthesis. Ditto for .3ds model files.

Windows builds have been speeded up with all compilers.

Input Devices

Added ar5DTGloveDriver (5th dimension technologies dataglove).

arFOBDriver attempts to query the flock about its configuration instead of using the SZG_FOB/config database parameter.

Added a prototype wiimote driver on Linux using wiiuse-0.9 (1.0 and later won't work without a re-write).

Added some more PForth words.

Standalone Mode

Programs in standalone mode can now load input device drivers, but only if they were built with dynamic linking.

Programming

The biggest change is much improved .obj support.

obj texture maps are mip-mapped by default, resulting in much faster rendering.
Added support for the map_opacity field (transparent texture maps).
Added the activateTextures() method. Call this to download textures to the graphics card before compiling an arOBJRenderer into an OpenGL display list.
Added some diagnostics, methods, bugfixes for doing things with bounding spheres (view frustum culling).

Added the ability to set the framework's navigation matrix from a matrix of specified index in the input event stream. This behavior is toggled either by a database parameter or a 'use_nav_input_matrix' message sent by dmsg. Since input matrix events can be generated either in PForth or, now, in a Python input device, this means navigation is now scriptable for applications that use the framework-mediated navigation.

Changed default arEffector signature to allow for more than two axes.

Master/slave framework sends a response to messages sent with dmsg -r, including user messages. Scene graph framework sends a response to user messages.

Copied some methods from arVector3 to arVector2.

Python

Began the switch from SWIG to SIP. Currently two sets of bindings available, in szg/python and szg/sip_python. The new bindings are easier to read and maintain and sometimes faster, but don't yet include support for the more object-oriented version of the scene graph API.

Moved the scene graph demos out of the 'doc/python folder to python/demos/myriad and sip_python/demos/myriad. As mentioned above, the latter don't work yet, although sip_python/demos/blobby/blobby.py is a working scene-graph program.

Commands

The ouput of dex has been changed a bit, and the -v and -q (verbose and quiet) options have been added. Also, some of the application output during startup that was just being lost before is now returned to dex.

Added dkillall -a <virtual_computer> option, which kills all non-szgd processes associated with a virtual computer.

Changes in version 1.1

Syzygy 1.1. is primarily a bug-fix release (too many to list here). The most important changes:

Documentation

...Has been largely re-written for clarity. A few major messes remain, however.

Demos

Some demos (ones that were insufficiently instructive or that depended on very large data sets) have been removed from the main "szg" repostitory. We're moving towards packaging each of these individually together with their data.

Skeleton/Build Template

The new szg/skeleton directory is a directory tree template for building your own programs. Copy the directory somewhere, rename, modify the source files, etc.

Commands

szgd

szgd now takes a non-optional <base paths> argument. The full form of the command is:

szgd <base paths> [-r]

The <base paths> argument is a semicolon-delimited list of paths to directories or native executables. This allows you to restrict remote execution to particular directories or programs. On Windows the ".exe" suffix of executables should be omitted. You'll probably need to enclose the argument in quotes if it contains a semicolon. Each item must exist or szgd will exit. All SZG_EXEC and SZG_PYTHON paths and SZG_PYTHON/executable must begin with an item from the <base paths> argument or the dex command will fail.

The optional -r causes it to repeatedly attempt to reconnect to the Syzygy server on failure (instead of quitting as formerly).

dex

One variant of the dex command has been eliminated (to clarify error messages). You may no longer run an application on the local computer by typing:

dex <app> <args>

Instead, type:

dex <computer> <app> <args>

Precompiled SDKs

Are no longer available; we're too under-staffed to support them. Weren't too useful anyway.

Sound

Syzygy 1.1 uses FMOD 4 (ex) instead of FMOD 3. This change is transparent to the application programmer.

Configuration

XML parameter files

The code that parses XML config files now reports errors (quite well) instead of failing silently as it did in 1.0 (we hang our heads in shame).

Resource Layout

Native executables can now be packaged together with their data files (as Python programs could before). szgd now looks for executables in subdirectories of directories on SZG_EXEC/path, and when it finds an executable it sets the current working directory to the directory containing the executable. This means that data files that the application opens directly can be placed in the directory with the executable (or a subdirectory of it) and opened using the normal file-access functions with application-relative paths. And applications and their data can be bundled up into their own directory. See also the description of setDataBundlePath() in the Programming chapter.

Python Executable

The location of the Python executable should be specified using the database variable SZG_PYTHON/executable instead of having its directory added to SZG_EXEC/path. You can pass arguments to this executable by delimiting them with bars, e.g. c:\Python24\python.exe|<arg1>|<arg2>.

Programming

Master/Slave

Previously M/S programs could only load .obj 3D models using the scene graph. Added new arOBJRenderer class that does this much more easily, see Objects.
We're gradually removing the GLUT dependency. arMasterSlaveFramework.h no longer includes the GLUT headers, so if you use any of the GLUT drawing functions (e.g. glutSolidCube() etc.) you'll need to include "arGlut.h". Note that using any of the non-drawing GLUT functions (window-control, event-handling) will most likely crash your application.
Fixed a bug in framework.getOnButton() when called in the post-exchange in slaves.
Added ar_navBoundingBox( arVector3, arVector3 ). Call with opposite corners of a bounding-box and the standard navigation routines will not allow you to travel outside of it.
The alternate framework start...() methods (startWithoutWindowing(), startWithoutEventLoop()) have been eliminated. An alternative start() method now takes two arguments: bool start(bool useWindowing, bool useEventLoop);

Scene Graph

Is now extensible via new node types defined in shared library plugins (see szg/src/graphics/arTeapotGraphicsPlugin.cpp, called from szg/src/demo/cubes/cubes.cpp). No build template for creating these plugins yet, alas.

Python

Python programs that sub-class arPyMasterSlaveFramework can now have an interactive Python prompt if run with the "--prompt" argument. This allows you to access framework properties and call framework methods from the prompt in a running application.
An unhandled Python exception raised in a callback will now cause the program to exit with an error message.

All

Fixed bug in arGUIWindow preventing application access to window pixel dimensions. One of these days we'll even document the arGUI classes...

Fixed a bug in arTexture::readPPM() that was causing textures to be read in left/right reversed.

PForth

Added new words less, greater to work around XML parser bug ("<" and ">" don't work well in XML and the parser was choking on the escaped versions).

Changes in szg-1.0

LICENSE CHANGE! With the unanimous agreement of those who've contributed so far, we're moving to a BSD-style license for the base code and LGPL for the python bindings.
Added a logging feature, ar_log(), and partially converted the code's couts/cerrs/printfs to use it.
Added more Python examples (doc/python).
Fixed OS X problems like window resizing.
More work on the scene graph, making sure that objects can be attached via node pointers instead of node names. This simplifies scene construction.
Improved the scene graph documentation.
Added texture font code (for high resolution font bitmaps from Bitmap Font Builder).
Improved Python event processing (the input device framework). Also made it easier to create custom input simulators.
Added the ezszg.sh script for quick set-up of a programming environment from a Syzygy SDK.
Ported the Syzygy doc sources from html to txt2tags.

Changes in szg-0.8

BUMPED THE SZG MAGIC NUMBER! This is used in all network communications... and szg components will refuse to communicate with one another if it does not match. This is intended to keep components with incompatible communication protocols from intergering with one another.
IMPORTANT: It is now possible to set Cluster Mode parameter values using environment variables and to set the identity of the standalone parameter file using the environment variable SZG_PARAM.
Added "predetermined harmony" mode to the arMasterSlaveFramework. Here, each app instance operates in lock-step and there can be no late joiners.
dget and dset can now hierarchically traverse XML database entries, even following the usenamed pointers!
The arGraphicsDatabase/arGraphicsPeer code is now much better integrated with Python (more classes wrapped plus the implementation of reference couting in the C++). Also, a decent thread-safety model has been implemented making the operation of arGraphicsServer and arGraphicsPeer thread-safe!
Graphics configuration is now completely different! It is XML based (like the DeviceServer config). Config options are much more complete and multiple viewport custom windows are better supported (i.e. easier to create). SZG_SCREEN has GONE away! To be replaced with SZG_DISPLAY.
Added the arGUI API for windowing. This allows multi-threaded windowing, lets us eliminate some global framework variables, and allows some other neat tricks (like making windows from the Python prompt). All because of its thread friendliness! It also fixes a bug with standalone mode on the Mac for distributed scene graph apps.
Removed Cube-specific code from Motionstar driver and moved it to a PForth filter (see cube.txt in this directory).
Functionality Change! (That I don't think will affect anyone). In the PForth filter, the generic filter words (e.g. filter_all_matrices) are now executed before the specific ones (e.g. filter_matrix_0).
Added several new PForth words, e.g. to do element-by-element math ops on arrays.
Added words that duplicate the functionality of the arTrackCalFilter. These are used in the new cube.txt.
Added a color filter to the arGraphicsWindow to e.g. simulate how things will appear in the dim Cube on a brighter monitor.
Made a small tweak to fixed-head mode. Previously, SZG_SCREEN#/fixed_head_pos specified the position of the head sensor; it now specifies the mid-eye position. This improves the appearance of wall-to-wall transitions in the Cube noticeably.
Numerous additions to the Python bindings. Should all be backwards compatible with existing Python code.

Changes in szg-0.7

Changed the mechanism by which alternative config file and login file locations are determined for Cluster Mode. There is no longer a -t option for dname, dports, daddinterface, or ddelinterface. Instead, the SZG_CONF environment variable, if set, gives the nonstandard config file location. Similarly, SZG_LOGIN, if set, controls the location of the login files.
Master/slave preExchange callack is now only called on the master. postExchange callback is not called on a disconnected slave. postExchange callback returns void, not bool. setInitCallback has been changed to setStartCallback because the callback is invoked during arMasterSlaveFramework::start().
Changed arGraphicsWindow so it supports much more general drawing modes into a single window. Before, a few were hard-coded in, now things are fairly general.
Added an arHead class to hold head-related data (eye spacing, mid-eye offset, eye direction), automatically shared master->slaves and scene-graph controller->clients. Associated parameters are read from the SZG_HEAD group by the master and scene-graph controller only (i.e. they only need to be set on the master and trigger computers). It also holds the graphics unit conversion factor, simply because that's an application-wide constant and that was a convenient place to put it.
Deleted the arScreenObject. Its head-related info was moved into arHead, virtual screen location into the new arGraphicsScreen, and the frustum and lookat matrix computations are done by the new arVRCamera class.
Changed the way the arViewport interacts with the arGraphicsWindow and the camera in setting up the rendering context & frustum parameters. Among other changes, each viewport now has an associated eye sign and OpenGL draw buffer (e.g. left or right). This will allow Ben to mix active stereo & anaglyph in the same display (oooo!).
The discovery packet/ response packet formats have changed to allow version numbers and also bring the interior contents up to date.
TCP wrapper filtering has been improved so that lists and netmasks can be employed (essentially exactly like TCP wrappers now)
dbatch format has changed. This allows "globals" not tied to a particular computer into the parameter database. This is needed for input node descriptions. It is also needed for pforth programs.
dhunt/dlogin now work with netmasks != 255.255.255.0. And also play better with firewalls (i.e. use port 4621 on the return).
PForth program no longer live in an XML file, but are instead embedded in the input node description.
DeviceServer is now extensible via loadable modules.
Python bindings plus system support for python executables has been added (thanks Peter and Jim)!
dll-ification has happened. Syzygy library is now built as dll's and szgd has been modified to support this.
Changed the virtual computer set-up to include "location", which better allows different virtaul computers in the same setting to play nicely with one another.
Possible to set-up the virtual computer to kill everything instead of just the components determined to be incompatible at the launch of a new application.
None of the "killing" via arAppLauncher happens via names anymore but instead via services and locks.
Made the arSZGClient "stoppable" via remote kill (this fixes some weirdnesses with dkill -9. Now, the szgserver sends a "kill" packet to the client, which cleans up all pending communications. Fixed some bugs in arStructuredDataParser in the process.
Fixed all the "jamming" issues in szgd and the arAppLauncher. The szgd is now fully multithreaded.
Have a working shared library implementation now.
Added "stream playing" functionality to the sound.
Added functionality to support "graphics peers" as a generalization of arGraphicsServer and arGraphicsClient.
The Irix port again seems to be working. This was a thread-safety issue and has gone away with better Irix compilers. This issue is explored in depth elsewhere in the documentation (including a simple test program to diagnose whether your compiler is OK or not).
Fixed the FOB driver so that it can be used with more general FOB set-ups.
Added the "EASY" build style and improved the project template. There is also a "make create-install" option which is perfect for making an image of the library for development.
Added support for streams to the sound API. This is important for long sound files. Modified the SoundRender program so that it can export the waveform (needed for Cube sound visualization).
Added support for arMasterSlaveDataRouter to arMasterSlaveFramework. This is an easier way of supporting object replication for complex systems.
Added the ability to parse XML versions of arStructuredData over network connections. This allows for easy RPC to szg apps.
Added overlay and keyboard callbacks to arMasterSlaveFramework. This makes it easier to do standalone mode.
Fixed the concept of a custom camera in the arMasterSlaveFramework, allowing applications that use non-euclidean geometry, for instance.
Added "standalone" mode, which enables operation of applications without the presence of an szgserver.
Added rudimentary security to the szgserver. It is now possible to reject connections based on incoming IP adddress (similar to TCP wrappers). Added because of the increasing frequency of port scanning.

Changes in szg-0.6

IMPORTANT: The location of the data for the demos has been changed. Make sure that you reorganize your data directory for the new method.
IMPORTANT: Please re-read the basic documentation! The method of setting-up the distributed system has changed!
IMPORTANT: Cluster Mode now uses connection brokering, via the szgserver, to determine how components will connect to one another. This has many advantages, espcially in making initial set-up easier. However, YOUR OLD CONFIG FILES AND YOUR OLD INFRASTRUCTURE EXECUTABLES ARE INCOMPATIBLE WITH THE NEW STUFF! Furthermore, the new configuration resides at a different location on Windows computers, C:\szg\szg.conf. The config file location changed because Windows XP does not allow normal users to write files to the top level of the C: drive.
Because of the connection brokering, various objects, such as arNetInputSource, arNetInputSink, arBarrierServer, arGraphicsServer, arGraphicsClient, etc. have changed interfaces with respect to connections. If you based code on this layer, you'll have some small changes to make. However, code based on the arMasterSlaveFramework and arDistributedSceneGraph APIs is unaffected.
A memory leak in the navigation framework on Win32 was fixed (a critical section was being allocated over and over). Thanks CalState Hayward!
The support for the wildcat hardware framelocking was improved.
IMPORTANT: Syzygy now supports the latest fmod. It turns out that there's been an API change between the original version we supported (3.4) and today's version (3.7). Since it is impossible to find older versions, we've updated our source code. This means you'll need to download and install the latest dll's and header files if you want to compile Syzygy with sound support.
Preliminary experimentation with incorporating NVIDIA's CG with the scene graph has happened. More to come in the future! Thanks Mark Flider!
SZGUSEOLD is now unused again. Instead autodetection of gcc version includes the appropriate header.
Loading of jpeg images is now supported in the arGraphicsDatabase and elsewhere.
fmod support for Mac OS X has been added.

Changes in szg-0.55

IMPORTANT: The "name translation" (szgtrans.conf) file format has changed! Now it begins with a magic character '#' and has names that are delimited by parentheses. This allows for multi-token user names, as might be found on Windows systems. Note that this means this version of syzygy will not be able to coexist with previous versions.
A C++-compilation issue has been introduced: namely is not present on older Linux systems but it is the proper header to use. It seems to do no harm to use the old-style headers, but if you want to use the new one uniformly, go ahead and set the environment variable SZGUSEOLD to false.
Precompiled headers are now used on the Windows side, speeding up builds by a factor of 2.
Added support for Intersense trackers. Thanks Drew!
Fixed a problem with the spacepad driver. It is now returning reasonable values (before... the rotation matrix was screwy).
Added syzygy support for vrpn. Thanks to Russ Taylor for guidance.

Changes in szg-0.5

IMPORTANT: it is now necessary to set SZGHOME on Win32 for the cygwin build to succeed.
Previously, szgserver would not run under Windows 2000. Bug Fixed.
Previously, host names beginning with a numeral confused the sszgserver name resolution process on Windows. Bug Fixed.
The Cluster Mode protocol has changed greatly, mostly to allow greater handshaking (and thus determinism) between components during application launch on a cluster. It is not compatible with previous versions. See Cluster Mode for details.
The graphics database has changed, adding more primitive types and now correctly supporting OpenGL lights and materials. The new API is incompatible with that found in version szg-0.4. See this for details.
The callback signatures for arMasterSlaveFramework have changed. See this for details.
Stability and correctness have been greatly improved.
Graphics import filters and facilities for working with human avatars are much better. Importing of OBJ files has been well-tested. And 3D Studio Max files can be imported via lib3ds. We are also able to read htr files (a standard for describing human motion) and clothe these with segmented avatars. The program szgview (a general purpose viewer for all supported types) is now available.
Modifications have been made to the graphics database to allow distributed operation over a WAN (for distance telecolaboration).
Increased input device support. We now have support for the Ascension Spacepad, Ascension Flock of Birds, and the Motion Analysis optical motion tracker. See this for details.
A navigation/interaction framework has been added. This makes it easy to add navigation to your program using a standard interface. See this for details.
A extensible system for event filtering has been added, based on the forth language. See this for details.

Changes in szg-0.4

Executable names have been changed. This is true of many Cluster Mode commands, in an attempt to make them more keyboard friendly (for instance, changing szgConnect to dconnect). Also, some service programs have seen name changes, like SZGRender being changed to szgrender. See the rest of the documentation, in particular Cluster Mode, for details.
Instead of many server executables for input device drivers, a single one, DeviceServer, runs them all.
Endian conversion works. We can communicate again between the SGI and an x86 box. Also, Irix build problems are fixed.
There is a communications incompatibility between the current version and previous versions. This is largely due to the endian-conversion fix. Old and new versions cannot communicate with one another.
A way to automatically launch applications has been added. One can now define a virtual computer out of cluster components and launch an application on it.
Programming APIs have been deployed: arMasterSlaveFramework and arDistSceneGraphFramework. These simplify the work of making a VR application using either the master/slave or distributed scene graph programming paradigms.
Synchronization again occurs over TCP instead of UDP. This means that you can run distributed graphics applications over low-latency WANs.
A new sound API is fully integrated. Example code is in the various shipping demos.
Bug fixes in math code (ar_rotationMatrix).
Input devices are all VR now. You can't use a joystick to control demos anymore.
Demo navigation unified. See the Example Programs chapter for details.
New SZG_RENDER configuration options. SZG_RENDER/frustum and SZG_RENDER/lookat deprecated.

Changes in szg-0.3

IMPORTANT NOTE: Endian-conversion is currently broken! The arStructuredData::parse call breaks it. The next version of Syzygy will do endian-conversion upstream so that this is not a problem.

IMPORTANT API CHANGE: In szg-0.2, one could get the value of XXX/SZG_RENDER/size (where XXX is the local computer) via the command:
```
   value = SZGClient.getAttribute("NULL","SZG_RENDER","size")
```
This will no longer work. Instead, use:
```
   value = SZGClient.getAttribute("SZG_RENDER","size")
```
One can set defaults of "true" or "false" for a parameter "show_framerate" like so:
```
   value = SZGClient.getAttribute("SZG_RENDER","show_framerate",
                                  "|true|false|")
```
If the parameter value in the database is neither "true" nor "false" then "true" is returned. This allows for code simplification in a common case. (The first value is chosen as the default.) Note that to manipulate values in the parameter databse associated with computers other than the local one, you'll need to use four parameters, with the last parameter the empty string if you want no defaults.
```
   value = SZGClinet.getAttribute("air","SZG_RENDER","show_framerate","")
```
Note that this is a relatively uncommon case, only relevant to those doing systems programming with Cluster Mode.
szgServer set-up is now vastly different. Broadcast communications using ports 4620 and 4621 allow for automatic set-up on a LAN. See SyzygyServer.txt for details. Also, the communication protocol between client and server is different. You cannot mix programs linked with version-0.2 and version-0.3.
The szgServer and related programs now have the concept of a user. Accordingly, you need to login (szgLogin) to have command line programs work (with the exception of szgd). See SyzygyServer.txt for details.
Various demos have been added in src/demo. The distribution now gives a self-contained means of going from cluster hardware to demo very quickly.
A framework has been added for creating applications supporting synchronization across multiple application instances.
All the demos look for their input devices based on the SZG_INPUT/IP and SZG_INPUT/port parameters. This cleans up matters a little bit over the previous method of having some programs look at SZG_JOYSTICK and some programs look at SZG_INPUT. Expect more work in this area in future releases.
New environment variables that affect the build process have been added. Please check out the info about SZGEXTERNAL, SZGHOME, and SZGBIN in QuickStart.txt.

Changes in szg-0.2

You no longer need to run the BarrierServer program seperately to synchronize rendering. This is handled internally by the arGraphicsServer object now. In fact, running BarrierServer as before will cause the software to fail!
The synchronization uses UDP broadcast now for speed and scalability. In a future release, the ability to use TCP instead will be returned. There are some important consequences of the change.
- Remote rendering only works over a LAN (routers filter broadcast packets) This is one reason why we want to return TCP functionality in the future.
- Synchronization ports for different render clusters on the same LAN need to be set differently. Remember that the graphics controller sends its synchronization information via broadcast.
- Since UDP is connectionless, we use a TCP shadow connection to monitor connectivity between the master and slaves. Consequently, the synchronization software uses a block of 3 ports on each box. You cannot use your old port numbers for SZG_RENDER/sync_port if this block would overlap other ports. For instance, suppose you ran JoystickServer from port 10003 and set the SZG_RENDER/sync_port to 10001 previously. This would not work under the new set-up (a conflict at port 10003).

The graphics API has changed completely (and gotten alot simpler). See GraphicsAPI.txt for details and rationale. Note that it is a straightforward, mechanical exercise to translate your code. In both syzygy-0.1 and syzygy-0.2, you get graphics by altering an arGraphicsDatabase object via a stream of messages. In syzygy-0.1, this is handled directly by the programmer. In syzygy-0.2, a wrapper API is provided, and, in fact, the programmer is discourged from touching the database directly. In syzygy-0.1, there were two basic ways to get the data into the database, create node and alter existing node. These also exist in syzygy-0.2. Here's an example with a transform node.
```
   Create node (0.1):
   theDatabase->alter(
     theDatabase->makeTransformData("world",0,"root",theMatrix));
   Alter node (0.1):
   transformNodeID = theDatabase->getNodeID("world");
   theDataBase->alter(
     theDatabase->makeTransformData("",transformNodeID,"",theMatrix));
   CreateNode (0.2):
   transformNodeID = dgTransform("world","root",theMatrix);
   Alter Node (0.2):
   dgTransform(transformNodeID,theMatrix);
```
Other calls can be similarly mechanically translated.

Testing

Supported Configurations

Syzygy uses the following combination of technologies, influencing the platforms on which it can compile and run. As of 2004, this requirement set has become mainstream, and, because a development goal is not to expand it further, building and running Syzygy should require only trailing edge features of the one's development environment in the future, a desirable characteristic.

OS with preemptive multitasking and threading.
Thread-safe STL and standard C/C++ library.
Good support for dynamic libraries and loadable modules, along with the ability to pass STL objects across module boundaries.
Graphics support depends on OpenGL.

Syzygy is portable across a range of modern operating systems, Linux, Win32, Mac OS X, and Irix. It is stable on these systems, with the stability subjected to rigorous testing, as noted below. Furthermore, it is stable and tested across heterogenous combinations of these systems: your clusters need not be comprised of one type of machine.

Any new (as of summer 2004) system should compile and run Syzygy. Some specific information about a wider range of systems follows.

Any modern linux should compile and run Syzygy. The developers have successfully run and built Syzygy on RedHat 6.2, 6.3, 8.0, 9.0, the various Fedora releases, along with Xandros 2.0 and 3.0. Currently, the developers are compiling using gcc 3.x, so it could be that earlier versions (as would have shipped with RedHat 6.x) will no longer compile the code base. gcc 2.96 failed the thread-safe STL test mentioned above (it was a flawed release coming after the egcs merge) and, consequently, Redhat 7.1 and 7.2 should NOT be used.
Windows NT, Windows 2000, Windows XP. Windows 98 will also work in conjunction with cygwin. Visual Studio 6 works in combination with STLport (since the STL shipping with this product does not allow STL objects to cross module boundaries). Visual Studio .NET works.
Mac OS X 10.3 is required. A bug in the gcc shipped with Apple's 10.2 development tools prevents Syzygy code from being built as dylibs.
Irix 6.5 and a recent (summer 2004) version of CC is required. Previous versions of CC would not produce thread-safe STL code.

Special Notes for Windows 98

The problem with Windows 98 derives from the way in which Syzygy components automatically determine their names for connecting to the Syzygy distributed OS. On Windows 98, it is impossible for an executable to get its own name (other than in all capital letters) from argv[0] when run from cmd.exe (the default Windows shell). Since Syzygy uses the command line argv[0] (in most cases) to set the component name, this causes problems with automated operation of the distributed system, for instance when running an application on a "virtual computer", one of whose nodes is Win98. As a work-around, three components (szgd, DeviceServer, and SoundRender) manually set their component names. Thus, Win98 nodes in a virtual computer can host input devices (via DeviceServer) and can also emit sounds (via SoundRender). Finally, components can retrieve the correct name from the argv[0] when launched via the cygwin shell. Consequently, normal operation of the distributed system is assured if you only launch components from the cygwin shell when on Win98. Running a syzygy program from an incorrectly configured Win98 DOS shell can fail. If this happens, try unchecking "Prevent MS-DOS-based programs from detecting Windows".

Testing Procedures

Syzygy is stable enough that it can now be used to reliably detect bugs and misconfigurations in the hardware/OS upon which it is being run. The following tests thoroughly exercise the system, though they have yet to be automated.

Some small self-contained unit tests exist. These should run and pass on each supported platform. - TestLanguage: Tests the data marshalling/de-marshalling along with some elementary file and directory operations. Tests binary and XML data reading/writing, both in memory and to and from files. Performance information is returned. It also tests STL thread-safety of the platform. - TestLanguageServer/ TestLanguageClient: Tests send/receiving of binary and XML data records across a network connection. Can be used to test a network link's speed (10 Mbit vs. 100 Mbit vs. 1 Gbit). Performance information is returned. - TestMath: Makes sure that the math functions (simple vector and matrix algebra plus some VR-specific projection matrices) do the right thing. - phleettest: This test hammers a running szgserver from multiple threads with various requests. Running several of these simultaneously is a good test of the szgserver's stability (and immunity to memory leaks). - BarrierServer/BarrierClient: Tests synchronization performance. One BarrierServer is run along with multiple BarrierClient's, probably on different machines. Performace, in terms of number of synchronizations per second is returned. Performance should be very close to one synchronization per slowest link ping time between pairs of machines running BarrierClient and BarrierServer. These programs should be able to run indefinitely and tolerate the BarrierServer being killed and restared, along with dynamically adding and subtracting BarrierClient nodes.
Each supported platform runs szgserver and szgd. The szgserver communicates with clients on each of the supported platforms. dhunt and dlogin also work.
Successfully run a distributed scene graph application on each of the supported platforms. Connected display clients work on each supported platform.
Successfully run a master/slave application on each of the supported platforms. Slaves run on each of the supported platforms.
The following is the "stress test" currently used to validate the system (here the Syzygy Cluster Mode and its application launching and management features are stressed). Given a 3x2 video wall with some Windows and some Linux components. All of the computers have dual processors and two network cards, one connected to the internet and one connected to a private gigabit network. A seventh trigger computer also exists, a dual processor Linux computer, with similar networking capabilities. This seventh computer also runs the szgserver. A script launches applications at 3 second intervals. 4000 applications launches are required without a problem before the software is said to be sufficiently stable.
A scene graph application should run on a small cluster (say 3 rendering nodes) for a period of weeks.
Frame synchronization in a running master/slave application should satisfy certain "quality-of-service" guarantees, in terms of speed and uniformity over a long period of time. No formal definition here... just attempting to eyeball a Gantt chart measuring this in real-time over a period of some minutes.

Bugs

Documentation Errors

Need to emphasize that shared libs must be copied into the exe directory!
Should explain how to use the ar_log() API (and why to use it instead of cout/cerr).
Should explain how to use arTexFont (and especially how to make the fonts).

Setup/Building

- Thread-safety issues... The Irix port does, indeed, seem to be working. However, it seems sensitive to having an appropriately configured compiler. (or maybe a recent enough version). Specifically, if the following test program compiles and runs for a minute or two, then syzygy will run and be stable.

#include
#include
#include
using namespace std;
void* task1(void*){
  cout << "YYYY\n";
  while (true){
    string temp("yyy_test_name");
  }
  return NULL;
}
void* task2(void*){
  cout << "XXXX\n";
  while (true){
    string temp("death_by_xxx");
  }
  return NULL;
}
int main(int, char**){
  cout << "#####################################################\n";
  pthread_t ID1, ID2;
  pthread_create(&ID1,NULL,task1,NULL);
  pthread_create(&ID2,NULL,task2,NULL);
  // wait forever
  pthread_join(ID1, NULL);
}

The compile script for Irix is simple. Here, we are assuming that the program name is test.cpp.

CC -n32 -LANG:std -c test.cpp
CC -n32 -LANG:std -o test test.o -lpthread

NOTE: This would probably catch the thread safety problem on RedHat 7.x as well (?). Haven't really felt like re-installing something that ancient to find out.

The very latest fmod is NOT compatible with the headers we had previously.

Configuration/Cluster Mode

Phleet needs new methods! Specifically, we must be able to release services. Also, need to be able to get service info from the command line!
BUG: if someone is looking for a service on network X but the service is offered on network Y, the connection won't work, but the service will disappear from dpending. - Same computer name in szg.conf file on multiple computers impairs the system. We should be more robust!

Graphics/Scene Graph/Database

The node method calls are woefully inefficient for large arrays because they do not use prtIn.
dgSetGraphicsDatabase must be eliminated! (getting closer to this goal...)
The light direction seems to be wrong! (i.e. direction = (0,0,10,0) instead of direction = (0,0,-10,0)
3DS objects cannot have multiple parts! (i.e. only a single mesh element, which means only a single material). Very bad!

Miscellaneous

On OS X (at least on 10.3), there is a strange memory leak. Specifically, the statement
```
  list<arDatabaseNode*> foo = graphics_node.getChildren();
```
causes a memory leak!
q33 does NOT run successfully when built with Visual Studio 6 on Win32. Visual Studio 7 is needed.
jpeglib gotcha. IMPORTANT. On the Windows side, it seems that the jpeglib MUST be compiled by the same version of Visual Studio that you are using to compile your executables. Otherwise, the first call to the library will produce a segfault. Why this should be for a C library is unclear.
When arMotionstarDriver is run on Win32, it seems to make the Motionstar "stutter". Does not seem to be a problem on Linux.
It would be helpful to have a string event type for the I/O framework (i.e src/drivers).
Our global init means that sockets cannot be declared globally! (the solution is to eliminate the global init for WinSock, i.e. arCommunicatorHelper) and make the arSZGClient do the init in it's init OR, if no arSZGClient, we will go ahead and explicitly do it.
The arDataServer should allow specific-socket-sending via HANDLES only. This is because it actively manages the sockets (and there's the possibility of a stale pointer). Another way to deal with this would be to add an event processing API, so that socket delete occurs a little less automatically, and to add reference counting to the socket class.
A weird bug. In standalone mode, with joystick (as in the standalone landspeeder), if the pforth filter is in the old style (PForth_programs.xml) instead of the new style (*.pf in PForth) then the program segfaults with an error message "arSyncDataServer error: start called before init".