Handling High-precision (float) Rendering

There are two steps to creating a plug-in that supports high-precision rendering. The first one is easy: you must tell FxFactory that the plug-in supports one of the floating point pixel formats:

For reasons that will be clear later, in most cases you will simply want to enable the RGB + alpha, 16 bit half-float option in the supported pixel formats section.

Whereas that first part is easy, it may take a lot more to determine if your plug-in is indeed capable of rendering in high-precision.

A Little Background

The bit depth supported by your plug-in does not necessarily affect how an effect is computed. Internally, all GPU-accelerated effects that rely on Core Image or OpenGL shaders are always computed using floating point numeric representations. On most modern graphics accelerators, you can assume that the precision of your computation will be comparable (if not identical) to IEEE 32-bit floats 1).

So what does the pixel format option do?

  • It enables high-quality I/O. A plug-in that supports 16-bit or 32-bit float rendering will communicate with the host using images at 16-bit integer, 16-bit half-float or 32-bit float per component.
  • It affects the bit depth of the OpenGL resources allocated for rendering. In developer parlance, your textures and contexts will be created as half-float or float primitives.

Testing Plug-Ins in High-precision

The Quartz Composer application only runs compositions at 8-bit integer precision, which means you will have to test the plug-in in a host to discover if the composition is capable of rendering at higher bit depth:

Whether or not a composition is capable of high-precision rendering depends entirely on what it is rendering. If the plug-in uses Core Image filters exclusively, and renders to a Billboard using the Native Core Image Rendering option, it will likely be able to run unmodified at 16-bit half-float and even 32-bit float bit depth. Unfortunately this simple scenario doesn't happen all that often.

Here are a few reasons a plug-in may not be able to support high-precision rendering:

  • Your plug-in uses a consumer (purple) patch that uses something other than Replace or Over as its blend mode.
  • Your plug-in uses a custom Core Image kernel or OpenGL shader that contains incorrect math. When your code is executing with 8-bit integer textures, the output RGBA values are automatically clamped to the range 0..255. When you render in high-precision, out-of-range values are not clamped, and thus are more likely to show problems with your algorithm. Forgetting to clamp the alpha values to the range 0..1 is the single most common mistake.
  • Your plug-in uses a patch that was coded to support only 8-bit textures.

All plug-ins eventually run using OpenGL, and OpenGL does not behave the same when it operates with float-based textures and contexts. The type of features available in high-precision vary greatly from one graphics card to another, and from one version of Mac OS X to another.

To make matters more complicated, most graphics cards do not support 32-bit float rendering as well as they support 16-bit half-float rendering. In light of this limitation, it is recommended that you declare your plug-ins only as capable of 16-bit half-float rendering for the time being. This should not be seen as a drawback: 32-bit float rendering provides negligible benefits over 16-bit half-float, and it requires roughly twice as much video memory for representing the same textures.

The bottom line: unless you structure the composition very conservatively, you may design an effect that will work in high-precision only on certain hardware/software combinations. It is thus best (and easier) to design and test a plug-in in high-precision from the start, than to take an existing plug-in and modify it to support high-precision rendering.

Blend Modes

As mentioned above, only the Replace and Over blend modes work identically at all bit depths. Since the Replace blend mode has little value, that means you should only be using Over for all your rendering.

How do you go about blending things together then? Use equivalent Core Image filters whenever possible. Quartz Composer lists a large number of composite modes, some of which are available as part of FxFactory. One of the advantages of Core Image is that it produces consistent results on different hardware and at all bit depths.

Compute Overlays in 8-bit

If your effect involves computing and blending an overlay with the source media, keep in mind the following: the main point behind high-precision rendering is to preserve all the color values of the source media intact. With that in mind, you could compute the overlay in 8-bit, and composite it over the 16-bit media using a Core Image patch.

How do you go about rendering certain sub-sections of your composition in 8-bit?

  1. Use a Render in Image patch whose bit depth is specifically set to 8-bit. Alternatively, use the NI Iterator patch, which supports 8-bit rendering by default.
  2. Composite the 8-bit output image from the previous step with the source media using a Core Image blend mode. The results ought to be accurate no matter what the bit depth of the original clip is.
  3. Render the result using a Billboard patch whose Native Core Image Rendering option has been turned on.
On some really old graphics hardware, fixed-point math or vendor-specific 24-bit float representations could be used.