ASCII: Balancing Structural Similarity with an Illusory Boundary

How it's made

Our goal is to convert a target image into ASCII glyphs. For simplicity, only monospaced fonts are used, meaning every glyph is identically sized: 15x7 pixels. Based on this paper Fast Rendering of Image Mosaics and ASCII Art we use SSIM, a structural similarity score, to measure how close the ASCII rendition is to the target image. Maximizing SSIM therefore means the ASCII image is close to the target image. For simplicity, glyph selection is performed independently for every 15x7 patch of the image. For each patch, we iterate through all glyphs in the glyphset, usually ~100 characters, and select the one with the best SSIM score. This similarity scoring approach works well for greyscale images and results in ASCII that approximates gradient like features. Here's some results using just the SSIM scoring method.

Move the mouse across a panel to sweep the divider: the target image to its left, the ASCII render to its right. Scroll to zoom, drag to pan, double-click to reset.

Replacing the fuzz with an illusory boundary

The results look OK, but there's clearly undesirable artifacts. There's ASCII fuzz: characters bleeding into the background. SSIM scoring doesn't handle complete transition to white background well. The result is that sharp boundaries become fuzzy/noisy. Unfortunately, these are very salient to human perception so we try to address it with an additional scoring term.

To motivate the additional scoring term, note that human vision is remarkably good at filling in what's missing. Gaps in the contour can be closed by the human brain: illusory contours. On the other hand, it's much harder to unsee a stray character in the margin. So if we want to perceive clean boundaries, we should bias the scoring towards white. Mathematically, this is expressed by the equation

trimming = 1 − stray ink / N

This penalizes over-painting. Glyphs are told explicitly it's worse to put black pixels where the target image is white. Asymmetrically, it is somewhat ok to put white pixels where it should be black. We blend a small amount of this penalty with SSIM scoring depicted by the alpha slider. Even a slight amount of this penalty cleans up the boundary.

Suggested usage

Lean on SSIM for anything with real greyscale. e.g. photographs, pencil drawings, shaded renders. If the target image is a greyscale foreground image with sharp transition to white background, add a bit of trimming loss: alpha > 0 until the margins look clean. Adding too much alpha will result in an over-exposed look.

Lean on the trimming term for edges and for black-and-white targets. Line art, logos, hard-thresholded frames, anything where the image is almost entirely black on white, or just the line-art: there is no gradient for SSIM to match, and what you care about is that the shape reads and the background stays empty. Alpha up to 0.8-0.9 are reasonable.

Two other levers:

• Equalize will map the input image through a monotonic illumination transformation so that the distribution of greyscale colors is more uniform "equal". This typically makes both dark and bright regions read more clearly. This is best paired with SSIM based scoring where it produces a good distribution of greyscale tones.
• Scale will increase or decrease the size of the target image. Increasing the size, means more ASCII characters will be used in the conversion. Note that since images are not typically clean multiples of 15x7, there is a center-cropping step after scaling. It automatically crops the minimal amount necessary to reach the right multiples of 15x7. This cropping is hardly noticeable for the most part. Scale is a very subjective parameter to use, since if you set it too high, the image can be almost perfectly reproduced but the charm of ASCII art is lost. I typically go for the minimal scale value at which key features of the image still come through as readable.

Bad Apple

I thought a good demo of the ASCII converter and the trimming loss would be the Bad Apple music video. It's originally drawn in pure black-white so I don't actually use the SSIM loss on it. Instead SSIM is replaced by IOU loss. And I keep the trimming term at alpha ~0.9 to keep boundaries clean. This ASCII converted bad apple video is rendered by a text player here:

Watch it in the text player →

Do whatever you want

The ASCII algorithm is fully frontend. No secrets about how the algorithm works. Feel free to modify it how you like. Maybe we'll get more interesting ASCII styles that way.