Quartz Composer based Visualizer Walls

I believe that the official sources gives a much better introduction to Quartz Composer then me coming up with something on my own. So here they are.

From http://developers.apple.com ...

"Quartz Composer is a groundbreaking graphics development environment that allows you to explore the incredible power of the graphics stack of Mac OS X Tiger. With Quartz Composer, you can easily combine the capabilities of Cocoa, Quartz 2D, Core Image, OpenGL, and QuickTime^?, all using an approachable visual programming paradigm. Use Quartz Composer to prototype Core Image filters, build engaging screen savers, create custom user-interface widgets, make data-driven visual effects, and even perform live performance animations."

From http://en.wikipedia.org/wiki/Quartz_Composer ...

"Quartz Composer is a node based visual programming language provided as part of the Xcode development environment in Mac OS X v10.4 "Tiger" for processing and rendering graphical data. Quartz Composer uses OpenGL, Core Image, Core Video, and other technologies to build an API and a developer tool around a simple visual programming paradigm. Apple has embedded Quartz technologies deeply into the operating system. Compositions created in Quartz Composer can be played standalone in any QuickTime^?-aware application (although only on Mac OS X 10.4 and later), from inside the Quartz Composer application, or can be embedded into a Cocoa or Carbon application. Because Quartz Composer makes extensive use of hardware acceleration and pixel shaders, it is recommended to have a recent graphics card with at least 32MB of VRAM."

Total images: 15

Hardware Requirements

Optimally, the hardware requirements of a QC based display wall should include

• MacPro^? Quad Core 2.66GHz to 3GHz with High end graphics card
• 1-4 Displays with dedicated graphics card
• Use 4GB RAM
• Gigabit Ethernet network or above

Although i feel that the above hardware will provide a decent per-node configuration for a display wall, i have tested the setup with ANY machine that is able to run QC and have been able to get decent results. Good enough to demonstrate the basic functionalities of QC.

Physical and Aspect ratio considerations

In order to obtain the correct aspect ratio that is required, the physical dimensions of the video wall needs to be addressed. 3 main things needs to be known.

1. The aspect ratio that you desire
2. The physical dimension of the display
3. The bezel size

With the above, it becomes possible to calculate the number of displays that is required to achieve your target size. We take for example an Apple 30 inch Cinema Display with the following attributes.

Display	Width	Height	Bezel Width	Bezel Height
30 inch	68.8cm	54.3cm	2.3cm	2.4cm

2 of the more common aspect ratios is 4:3 or 16:9.

In order to calculate the correct aspect, we have to consider 2 things

1. The bezel size. All displays WITHIN the wall has to consider the bezel as part of the display. The software therefore should handle bezel offsetting.
2. The original display aspect ratio.

In the above example of the Apple display, the aspect is apparently 16:10. This gives a weird size that we have to cope with.

Setting up the equations to calculate this is simple. We assume that the x is the width and y is the height.

Plugging in the right values above and solving for will provide the correct ratio of screens to use for the width and height of the wall for the given aspect.

Note that exactness might be compromised when doing the above math. I find that it is safe to assume 2 decimal places and have marginal error because it is actually really hard to get exactness.

Building the Quartz Composer Visualizer

The QC Visualizer applications is available in the XCode Tools installs. Once installed, it can be found in the path /Developer/Examples/Quartz Composer/Quartz Composer Visualizer. From there, open the Visualizer project using Xcode and build the example.

When completed the resulting application should be available under /Developer/Examples/Quartz Composer/Quartz Composer Visualizer/build/Release. The application can be placed anywhere in a convenient location to be launched.

Interesting Components for QC in Leopard

There are some patches in QC that is of particular interest in Leopard and it will be covered here.

Network Broadcast and Receiving

These 2 patches are by far one of the most interesting i have found so far. Simply, these patches provides a way to communicate strings from one machine to another over QC. In a single display mode, this will allow a user from another location, with network permitting, to control the visualization or data in another QC by passing it strings. This can be done either through broadcast or multicast. Which means that it should be done pretty much over a local network.

The use of these patches are pretty easy. On one side of the QC, you have the sender, and on the other side, you have the receiver. Once the ports are setup with the defined transfer method (Broadcast or Multicast), the QC would pretty much work.

• Network Sender and Receiver:
  

Controlling a QC Visualizer Composition

The purpose of exploring the Network patches is to allow the control of QC Visualizer compositions from a master computer to the slave visualization nodes. This would allow the compositions to be able to respond to certain data cues or mouse control. While on initial tests, QC Visualizers works flawlessly with drawn compositions with no required interactions, once interactions where added, severe distortions where observed and it seems that there was some problems drawing the objects.

After some investigation, it was discovered that the problems in drawing was due very much to the Depth Testing field within QC. This was something pretty hard to pin down as it will only show up when the composition is placed on the QC Visualizer. However, once it has been identified, one would just have to turn that off in order to get the QC Visualizer to display the compositions correctly.

• Parameter for Depth Testing:
  

A very simple implementation of a network controlled QC is provided here. It shows a few things as a "Receiver"

• A particle system generator
• A basic teapot with a transprent texture
• Primitive shape (Cube)
• Sphere with a world map downloaded from the internet.

By itself, it doesn't do anything interesting. But when the "Sender" QC is used, the mouse controls on the "Sender" is sent over to the "Receiver" and then manipulates the objects in pre-defined ways. It serves as basic introduction to using the Network patches to control a QC Visualizer composition.

• recv.zip: QC receiver composition
• sender.zip: QC sender composition

A screenshot of how the QC looks like is provided below.

• QC Visualizer Shot:
  

Quartz Composer and creating a Video Wall

The Quartz Composer (QC) Visualizer is a great tool that allows one to be able to distribute visualizations made in Quartz. However, it is unable to distribute VIDEOS. When one creates a QC and embeds the "Movie Loader" Patch, one would realize that they are unable to get the video to play correctly.

This is a great disappointment as one of the major things that one would think of first when looking at a large video wall, is to be able to play videos!

So after some amount of investigation, i believe that i have created a passable solution to this problem by creating a custom QC. The techniques used includes using

• View-ports
• Network sync
• NTP time sync

What this QC does is that given the dimensions of a video wall, each screen is tagged with their own position. The video file should be located such that it is accessible by ALL the machines. The machines should also have a NTP time sync such that their times should be exactly the same. The performance of each machine should also be similar. The pristine state of the machines should be load-less in terms of disk activity, network activity and CPU activity.

A quick sketch of the operating environment is show in the diagram below. The Audio of all machines should be muted other then the "Dispatcher" that controls when to start the playback and provides a monitor of the video. The audio of the video will be provided via the "Dispatcher" machine.

• QC Video wall layout:
  

However, the hardware requirements are very important as well. Every playback machine should be similarly spec-ed such that every node loads the video at around the same speed to ease synchronization of the video when it plays back.

In the QC files that i created, there are input parameters for both the "!VideoSync" and the "!VideoLoader". VideoSync will be launched within the dispatcher machine. The VideoLoader will be launched within the playback machines. Once the video files are loaded within the same location for each playback machine, the dispatcher machine will then provide the Movie Path to the playback machines, and the time to "Play Time" is also sent to the playback machines.

• VideoLoader^? Input Parameters:
  

• VideoSync^? Input Parameters:
  

If all the times of the playback machines are in sync, the video should play and sync up. A time counter at the bottom of the QC provides the playback duration for observing the sync.

Some shots of the individual QC sync ups are shown below.

• QC Video wall shot 1:
  

• QC Video wall shot 2:
  

The QC files used to create the Video wall is available here.

• VideoWall.zip: QC Video wall Files

Here is a YouTube^? Preview of a Laptop based tile in action.

• YouTube^? embed

• YouTube^? embed with proper 23" monitors for display

Reading Images from a directory for display in Quartz

Reading a set of images for displaying in Quartz is something that most people would want to achieve. Doing so would allow the user to be able to manipulate an entire set of images using Quartz's own core image filters and reduce the programming overhead. However, things might not always as simple as it seems.

The WRONG way of reading a directory of images

The most common logic of displaying a set of images is as follows.

1. Read directory for images
2. Read first/next image and prepare drawing area for image
3. Draw the first/next image
4. Return to step 2

In taking the above workflow and implementing it directly in Quartz, the result, one would notice, doesn't seem to be what you would want. An example of such a workflow is shown.

• Negative Workflow 1:
  

• Negative Workflow 2:
  

• Negative Result:
  

One would notice that for some reason, only the first image seems to be repeatedly rendered. While the logic seems completely valid, somehow, the results seems to be wrong! What seems to be the problem? Honestly, i can't say for sure, but one thing is certain, all variables that is passed to the iterator seems correct, but the enumeration of the structure WITHIN the iterator doesn't seem to run well.

So, what is the right way of doing it?

The CORRECT way of reading a directory of images

Apparently, the logic of reading the images in Quartz, is to have everything done kind of like a batch. All the images in the directory has to be preloaded and passed to the iterator for rendering. I envision a simplified view of this workflow as the following:-

1. Read the directory for images
2. Download all the images in the directory and load it up as a structure of images
3. Pass the structure to be drawn in a drawing area
4. Return to step 2

The above simple means that rather than passing the entire directory structure to be iterated, we have to "read" all the images in the directory, create it as a structure, and have that iterated instead. The workflow of that is shown here.

• Positive Workflow 1:
  

So how do we create an image structure? What is basically done is that each URL provided in the "File List" structure passed from the Directory Scanner is passed into the Image Downloader, and the resulting image is then constructed as an Array to be passed out.

The Create Image Structure is a piece of javascript that does just that. The javascript flow is described below.

var _list = null;
var _index = -1;

function (__structure imageStructure, __string imageURL) main (__structure imageList, __boolean reset, __image previousImage)
{
   var result = new Object();
   
   if (imageList != null) {
      if (reset || _index >= imageList.length) {
         _list = null;
         _index = -1;
      } else {
         if (_list == null) {
            _list = new Array(imageList.length);
         }
         _index += 1;
         if (_index < imageList.length) {
            if (_index > 0 && previousImage != null) { _list[_index-1] = previousImage; }
            result.imageURL = imageList[_index].url;
         } else {
            _list[_index-1] = previousImage;
            result.imageStructure = _list;
         }
      }
   }
   return result;
}

1. On the first image URL, the output image array is initialized to the correct length and the index counter incremented
2. The first image URL is then returned to the Image Downloader
3. On subsequent URLs, the resulting previousImage is passed into the function and allocated to the imageStructure with the relevent index.
4. The next image URL is also returned at the same time to Image Downloader

The above loop keeps running until the number of images run out, and the imageStructure then contains ALL the pre-loaded images in a form of a structure. All this is then passed off into a iterator which helps to draw all these images into a renderer.

• Positive Workflow 2:
  

Once this is done, the images would then be drawn in Quartz in the correct fashion as shown in the output below.

• Positive Result:
  

The Quartz files for both the methods are provided in the attachments within this page.

• ImageCache_Wrong.zip: Negative Workflow Quartz File
• ImageCache_Correct.zip: Positive Workflow Quartz File