FxFactory Developer

One of the hardest problem facing plug-in developers is designing effects that honor the native pixel aspect ratio of any source media fed to the plug-in by the host. All FxFactory plug-ins are based on Quartz Composer, which was not designed with video applications in mind. As such it always expects and generates images with a square pixel aspect ratio. The same goes for Core Image units, OpenGL shaders and any other computer graphics technology accessed through Quartz Composer.

Why is this a problem in the first place? Host applications feed images to plug-ins at their native size. For example, if your effect is used in a ProRes 1080i sequence, Premiere Pro will feed you one field at a time. The field has resolution 1440×540. That is obviously quite different from its ultimate display size of 1920×1080. If your composition draws a circle inside that 1440×540 field, by the time your circle is scaled to the final display resolution you end up with a nasty-looking oval:

You draw this…

…the viewer sees this:

Different hosts behave differently. Motion and After Effects always field-double their images, so the same field from the same source media will be fed to the plug-in as a 1440×1080 image. Final Cut Pro X supports plug-ins only when wrapped inside Motion Templates, and it thus borrows Motion’s habit of field-doubling all frames before processing.

To make effects development as simple as it can be, FxFactory has always provided an easy way out. When you enable the options “Can only render fields at full frame resolution” and “Can only render fields/frames at square pixel aspect ratio” in the Renderer section, you are asking FxFactory: “Scale input/output images for me so that I don't have to worry about pixel-aspect ratio issues”:

The “easy way out” described above is still widely used by many FxFactory plug-ins, and there will always be reasons to continue using it in the future.

In earlier versions of FxFactory, those options were on by default on any new plug-in you created. The automatic scaling of images up/down done by FxFactory behind the scenes is the source of potential filtering artifacts. But before we talk about the problems, lets talk about the situations where it is (and always will be) desirable and acceptable to use the easy way out:

• Generators: if you are generating your own content, it doesn't matter if it will get scaled down to the native field size. There are some desirable side effects too: if you generate a 1920×1080 graphic, you are super-sampling and thus the scaling process to 1440×540 will give you some nice, free anti-aliasing (at the expense of performance and higher resource usage).

• Generators (again!): if the media you use is heavily scaled/distorted or stylized, it doesn't matter if you are not rendering at the native resolution. You may actually benefit from super-sampling (again!).

• Stylized filters/transitions: if your effect heavily modifies the source, to the point where the original color information, features, etc… are replaced or barely visible, it's OK to rely on FxFactory's automatic scaling (e.g. half-toning filters).

• Very hard to do otherwise: some effects, especially those involving 3D transformations, lighting effects, etc. are very hard to design, and even harder to design with support for native pixel aspect ratio. The time you would spend in getting it all to work at native pixel aspect ratio would probably be better spent making tutorials, documentation, etc. or more plug-ins.

When designing high-quality effects, the optional, automatic scaling performed by FxFactory will introduce unwanted filtering artifacts. These are worse with interlaced media than with progressive media, and they are worse in Premiere Pro than in other hosts (due to the fact that Final Cut Pro X, Motion, After Effects always field-double their media). It’s easy to see how a plug-in that processes 1440×540 fields directly will be faster than an identical effect that is forced to process 1920×1080 images.

Notice that if your plug-in does not render any geometry (e.g. a color correction filter) and does not use filters that rely on a geometric distance (such as a blur radius), you get a free pass. You can (must!) safely disable all Renderer options discussed earlier, and your plug-in will render faster and at higher-quality: no extra scaling operations are being done on your source media or on the output generated by your composition.

You can test this problem very easily. Take a simple pass-through filter:

Enable the following Renderer properties:

This forces FxFactory to scale the input to square-pixels, and then the output from square-pixels, when your filter is applied.

Find a test graphic that is designed to help you spot filtering artifacts, such as this grid:

Apply the passthrough filter to get the following result:

Notice the unwanted softness introduced by the filter because of the automatic scaling, even if you are not actually touching the pixels inside the composition.

There is no general approach to handling native pixel aspect ratio correctly. The most important step is actually understanding what's going on behind the scenes. Then, depending on the effect you are designing, one of the following techniques may be used:

• If the effect simply involves rendering an overlay over the original media, the composition can:
  1. Figure out what the square pixels dimensions of the media are. For example, a 1440×540 field really has 1920×1080 square pixel dimensions ¹⁾
  2. Render its overlay at the square pixel dimensions.
  3. Scale the overlay down to the native frame size, and composite over the original media.

• The composition scales all inputs from their (potentially different) pixel aspect ratio to a 1:1 square pixel aspect ratio ²⁾.This technique can be summed up as adding a normalization step to your composition (click to enlarge image):
  
  
  1. All image processing is performed on scaled versions of inputs that have a square pixel aspect ratio.
  2. Point parameters that express a location in the frame using 2D coordinates need to be adjusted to find the corresponding 2D location in square-pixels space.
  3. The output is scaled back to the native dimensions of the frame.

• You are lucky enough to be working with existing Core Image units that can be tweaked to compensate for a non-square pixel aspect ratio, or you can design the effect using Noise Industries variants of Core Image units that can handle pixel aspect ratio for you (more on those in the sections below).

• You design a custom Core Image unit that is capable of rendering its geometric features with different settings for the horizontal and vertical axes, so that you can compensate for the pixel aspect ratio with a bit of math.

• You don't do any work at all! There are plug-ins that do not render geometry, or do not rely on any geometric feature (radius) and you get the extra speed and quality at no cost. Most color correction filters fall into this category.

Unfortunately, some of the stuff mentioned above can be very tricky to figure out, and it certainly comes in the way of FxFactory's goal of keeping things as simple as they can be. That's why FxFactory sports improved versions of built-in effects (Core Image units) that handle pixel aspect ratio.

The main difference between our versions and Apple's is that our units include an extra “Pixel Aspect Ratio” input. By virtue of doing that, we have created units that are capable of honoring non-square pixel aspect ratios.

Try creating, for example, a new filter in FxFactory. You'll notice that the composition you start with now looks like this:

Notice the FxFactory Gaussian Blur unit that honors the media's pixel aspect ratio. You just need to feed it the Pixel Aspect Ratio of the source, which you do by mapping the corresponding Image Info Structure inside FxFactory. When testing inside Quartz Composer, you'll only see a pixel aspect ratio of 1, so you need to test the end result in the host to make sure the approach is working.