RadLab 2 Bitmap Painter Proof-Of-Concept

One of my goals for RadLab 2 is to have a local adjustments feature – a way for users to paint an effect onto (or off of) certain areas. This is a fundamentally important part of good photo editing, and the only reason RadLab 1 is useful without it is its Photoshop integration; but RadLab 2 is to be a standalone app, so that means I need to climb that mountain. And I did. Here’s the prototype app, and a brief explanation of what it does:

For a traditional photo editor, this would mean making a bitmap paint engine – the user drags the mouse around, and as you receive mouse events, you stamp a bitmap image onto the canvas at regular intervals to create the smoothed brushstroke. In RadLab, however, it’s a bit tricker than that. RadLab’s adjustments are masks, which get used by the filter rendering chain to affect the strength of an filter in a particular area. For instance, when the user paints to strengthen a filter’s effect in a particular area, RadLab has to maintain a mask for that filter, and increase the value of the mask where the user paint. Then, when the preview image gets redrawn, the filter chain can reference that mask to determine how strong the effect should be, pixel-by-pixel. So that’s one complicating factor – RadLab doesn’t draw the mask, but rather has to draw a bunch of (sometimes computationally expensive) stuff, of which the mask is just a part.

Second, RadLab is non-destructive. We need to be able to support undo and redo, but more than that, the user should be able to pick up their work on a particular photo later on. This means we need a way to store the mask. In a Photoshop workflow, you make layer masks, and Photoshop stores those in a .PSD file as raster data. I’m pretty sure it compresses the masks, but nonetheless, a document with many layer masks ends up becoming a HUGE file on disk. This is bad. Lightroom escapes this by storing its local adjustments as a list of brush dabs – a “blueprint” for recreating the mask – and this is the same strategy RadLab 2 will employ. So the brush engine needs to be able to store and recreate raster masks from a list of brush dabs.

Finally, a related concern is that RadLab doesn’t draw its previews at full-resolution, but rather draws them at the exact size they’re displayed in the UI. The user can zoom in to get a closer look at the image, and we resample and redraw the image at the current zoom ratio. The problem here is that masks are, potentially, VERY expensive to recreate. A user who has worked for a couple minutes, with a large brush radius, can easily wind up with thousands of individual brush dabs in their mask. Redrawing this, especially at a large size, can take several seconds (or longer). So we don’t want to spend several seconds redrawing masks every time the user changes the zoom level.

With those parameters in mind, I build a proof of concept for RadLab 2’s mask engine. It’s basically a bitmap paint engine that:

  • Features adjustable brush radius, feather, and flow,
  • Can store and recreate masks from a list of brush dabs, and most crucially…
  • Can render a preview-sized mask and a full-resolution mask simultaneously and asynchronously, prioritizing the preview-sized mask, for realtime performance.

That ended up being the key – as we receive UI events, maintain the UI-sized mask for realtime updates, and in the background, maintain a full-resolution mask for when we need it. If the user zooms in, we make a reduced-size copy of the full-resolution mask, at whatever size we need, and then continue on. Say we have a UI-sized mask of 800x800px, for a full-resolution image of 4000x4000px. The user zooms in, and now the preview image is 1600x1600px. Instead of redrawing a mask from the original list of dabs at the new size, we just resample the 4000x4000px master mask down into the 1600x1600px buffer, and pick up from there. Visually, it’s identical. Computationally, it’s much cheaper, because we’ve been doing the “large mask” maintenance on a low-priority background thread while the user works.

And it works.

The other tricky bits here were finding the right formula for generating the brush bitmap, making the brush generation quick, and coming up with the right mechanics for the paint engine itself. Core Image provided the actual paint drawing code, but here are some highlights from the rest, in sexy, sexy Swift code:

Generating the brush bitmap

Full disclosure – math is not my strong suit, but I understand enough to fumble through problems like this. This uses a gaussian function (well, part of one, anyway) to generate the falloff toward the edge of the brush. I tried a handful of other methods, but they either bled too far outside the bitmap, didn’t look right, were too slow, or any combination of the above. Just creating a lookup table with the gaussian function we want to use, then applying that based on the pixel’s distance from the center (with linear interpolation), turned out to be both pretty fast and very, very pretty.

One expensive bit of this routine is calculating the distance of each pixel from the image’s center. Fortunately, we’re limiting ourselves to symmetrical brushes here, so I just render the top-left quadrant, and copy it into the other three (flipped as necessary). That doesn’t make it actually 75% faster… but close.

class BrushGenerator {
    class func generateBitmap(radius radius: Double, feather: Double, opacity: Double, color: MaskBrushColor) -> CGImage {
        // Gaussian function constants
        let c = feather * 0.3 + 0.00000000001                 // width of 1 standard dev
        let e = M_E                 // Euler's number
        let a = Double(UINT16_MAX)  // max val
        let b = Double(0)           // midpoint

        // Bitmap is 2x radius, plus the width of one standard deviation
        let bitmapDimension = Int(2 * Double(radius) * (c + 1.0))
        let bytesPerPixel: Int = sizeof(UInt16) * 2
        let bytesPerRow: Int = bitmapDimension * bytesPerPixel
        let totalPixels: Int = bitmapDimension * bitmapDimension
        let threshold: UInt16 = UInt16((UINT16_MAX * 995) / 1000)

        Swift.print("radius: \(radius), bitmapDimension: \(bitmapDimension), feather: \(feather)")

        // Create a lookup table sampling the gaussian function to 2
        // standard deviations, normalized to the range 0.0 - 1.0
        let gaussianTable = UnsafeMutablePointer<UInt16>.alloc(101)
        for i in 0 ... 100 {
            let x = Double(i) / 100

            // f(x) = a * e^-( (x-b)^2 / 2*c^2)
            let top = pow(x - b, 2)
            let bottom = 2 * pow(c, 2)
            let exp = -(top / bottom)
            let gaussVal = a - a * pow(e, exp)
            var val = UInt16(gaussVal * opacity)
            if val >= threshold { val = UInt16(a) }
            gaussianTable[100 - i] = val
        }

        let bitmapDataArray = UnsafeMutablePointer<UInt32>.alloc(totalPixels)

        // Fill the upper-left quadrant by walking each pixel in the quadrant,
        // and calculating its distance from the center of the bitmap, then
        // look that up in our gaussian table to get the output value.
        var col = 0
        var row = 0
        var rowStart = 0
        let midpoint = bitmapDimension / 2
        while row <= midpoint {
            let x = Double(abs(midpoint - col)) * 100 / Double(bitmapDimension)
            let y = Double(abs(midpoint - row)) * 100 / Double(bitmapDimension)
            var distanceFromCenter = sqrt(pow(x, 2) + pow(y, 2)) * 2
            if distanceFromCenter > 100 {distanceFromCenter = 100}

            let lowerSample = floor(distanceFromCenter)
            let upperSample = ceil(distanceFromCenter)
            let lowerSampleWeight = distanceFromCenter - lowerSample

            let lowerVal = Double(gaussianTable[Int(lowerSample)])
            let upperVal = Double(gaussianTable[Int(upperSample)])
            let valDiff = upperVal - lowerVal
            let outputVal: UInt16 = UInt16(lowerVal + (lowerSampleWeight * valDiff))
            // Premultiplied Alpha means that, in our case, the alpha and the
            // pixel value are the same, so copy the 16-bit value for this pixel
            // into the upper 16 as well
            var outputGray: UInt16
            if color == .White {
                outputGray = outputVal  // premultiplied alpha on pure white = alpha val
            } else {
                outputGray = 0          // premultiplied alpha on pure black = 0
            }

            bitmapDataArray[col + rowStart] = UInt32 (outputVal) | (UInt32(outputGray) << 16)
            col += 1
            if (col > midpoint) {
                col = 0
                row += 1
                rowStart += bitmapDimension
            }
        }

        // Copy upper-left quadrant to the upper right (flipped)
        col = 0
        row = 0
        rowStart = 0
        while row <= midpoint {
            let destCol = bitmapDimension - col - 1
            bitmapDataArray[rowStart + destCol] = bitmapDataArray[col + rowStart]
            col += 1
            if (col > midpoint) {
                col = 0
                row += 1
                rowStart += bitmapDimension
            }
        }

        // Copy the top half to the bottom half (flipped)
        col = 0
        row = 0
        rowStart = 0
        var destRowStart = (bitmapDimension - 1) * bitmapDimension
        while row <= midpoint {
            let destCol = bitmapDimension - col - 1
            bitmapDataArray[destRowStart + destCol] = bitmapDataArray[col + rowStart]
            col += 1
            if (col > bitmapDimension) {
                col = 0
                row += 1
                rowStart += bitmapDimension
                destRowStart -= bitmapDimension
            }
        }

        // Finally, produce a CGImage from our memory buffer
        let provider = CGDataProviderCreateWithData(nil, bitmapDataArray, totalPixels * bytesPerPixel, nil)
        let colorspace = NSColorSpace.genericGrayColorSpace().CGColorSpace
        let byteOrder = CFByteOrderGetCurrent()
        var bitmapInfoRawValue = CGImageAlphaInfo.PremultipliedFirst.rawValue
        if byteOrder == Int(CFByteOrderLittleEndian.rawValue) {
            bitmapInfoRawValue = bitmapInfoRawValue | CGBitmapInfo.ByteOrder16Little.rawValue
        } else {
            bitmapInfoRawValue = bitmapInfoRawValue | CGBitmapInfo.ByteOrder16Big.rawValue
        }
        let bitmapInfo = CGBitmapInfo(rawValue: bitmapInfoRawValue)
        let image = CGImageCreate(bitmapDimension, bitmapDimension, 16, bytesPerPixel * 8, bytesPerRow, colorspace, bitmapInfo, provider, nil, false, CGColorRenderingIntent.RenderingIntentDefault)

        bitmapDataArray.destroy()
        gaussianTable.destroy()

        return image!
    }
}