Bokeh Testing


Testing Bokeh (updated 9/10/08):
I believe that testing a series of single points of light should be enough to determine the bokeh characteristics of a lens.  If you have read the “About Bokeh” section, you understand my arguments for this.  I have started testing lenses, but due to the large amount of work involved have only partially tested
one lens with the testing procedure I have outlined below.  It still needs to have several distances done, AND the procedure needs to be repeated at several other lens focal lengths. 

The sheer number of images needed to fully evaluate a lens is so huge that the procedure is very time consuming.  Therefore, I realize I CANNOT DO THIS ON MY OWN.  I only have a few lenses, I only have a minolta body, and I don’t have access to any other bodies or lenses.  I envision a site in which I could compare the bokeh characteristics of all the major brand’s 70-200 f/2.8 lenses, all of their primes, etc.  But to do this I will need help.  A LOT of help.  As such, I would be THRILLED if anyone would volunteer to help out with lens testing.  I will publish as detailed an account as possible how to do the testing.  Anyone testing lenses needs to follow the EXACT protocol to make the lens tests meaningful.

How to Test Lenses
Lets start with some definitions that i will make so people don’t get confused:

Focal Length:  a measurement of a lens, (eg 135mm for the Sony 135mm f/2.8 (t/4.5) STF)
Focal Distance:  The distance from the front of the lens to the in focus plane in front of it (ie. the target upon which you are focussing)
Point distance:  The distance from the front of the lens to the point light source that is out of focus

A general diagram showing the protocol can be seen in the figure below:

testing setup2

The specifics of the procedure are detailed below.  Camera must be tripod mounted.  In order to generate a point of light, I use a flashlight (a
Nuwai TM-301x-3 and place a NiteIze 20” AA fiber optic adapter on it, tripod mounted (I duct taped it to a popsickle stick to mount it on the tripod).  The target you use to focus can be anything, or you can just use the distance marks on the lens. 

Here is a diagram of what should be seen in the viewfinder.  The first image has the blurred circle in the very center, and the second shows proper positioning of the the blurred circle on the side.  Note that the entire blurred circle is IN the viewfinder.  In some cases, for example a very low f-stop lens with an infinite focal distance and a very small point distance (see above for these definitions), it may not be possible to get the whole image circle in the lens.  In this case, place the center of the blurred spot 1/3 of the way between the left edge and the center of the viewfinder.

Viewfinder center Viewfinder left side
    Center spot location                                              Side spot location

Exposure is easy to obtain.  Take a picture and review the histogram.  If anything is blown out, reduce exposure.  Ideally, exposure should be 1/2 stop below the point at which the first signs of highlight washout occurs.  The beauty is this-- when you have done this for one aperture, all you need to do to get the right exposure for all other apertures is to change the aperture... the time is always the same.  It seems that while decreasing the amount of light that is let in (by stopping down), you also focus the light that DOES get in into a smaller area.  So there is less light, but focussed on a smaller area.  And by luck, this seems to always have the exact same peak intensity.

I never took the time to figure out exactly what would be the BEST focal distance and point distances to best evaluate a lens.  Instead, I arbitrarily chose the distances above on the chart.  Focal distances to be tested are: 1.5 feet, 2 feet, 3 feet, 5 feet, 10 feet, 20 feet, and infinity (or as close as you can get).  If anyone has issues with this, it would be a nice time to hear from you now!

Point distances to be tested are as follows:  2 feet, 12 feet, 25 feet and 100 feet.  Does anybody think this is a problem?  Could also do binary... 3, 6, 12, 24, 48, and 96 feet, but this is a lot more work.  Truthfully, the 100 foot distance is very difficult to do with the method above.  The small point of not so bright light dissipates so much over that distance, that you would have to use the bulb setting, and the background illumination becomes a problem.  What I do instead is find a street light that is approximately 100 feet away and shoot that instead.  Yes, not as scientific, but the only way  could make it work well.

I scale the images so that the blurred circle is about the same size on every exposure, then cut and paste them into a grid system that I have made available
here.  This is an adobe illustrator chart, but should open fine in Photoshop.  Ideally, I would like the charts to be in the same format.  In addition to the charts, I would appreciate a few real world examples of shots of difficult backgrounds to really put the lens to a real world bokeh test, as well.  Also, the charts go up to f/22, but there really is no need to go that high on the f-stop.  Certainly one could stop testing at f/8 or f/11 and not be criticized.  Even getting to that point is a lot of work.

Anyone who has any suggestions, or wants to help, please write me by using the link at the end of the page.  Thanks!

Glimpses of the future
So far I have been unable to come to terms with the complexity of assigning a “score” to something that is so subjective as what constitutes good and bad Bokeh.  I know what I like, but that is not necessarily what other people do.  There are also SO MANY variables such that it is very difficult to quantify anything.  I had been pondering this when I got a letter by Terence Tay, who had a brilliant suggestion.  Since the problem is so complex, people will not be able to assign grades, it is likely.  However, people can look at a picture and have a sense of if they like the bokeh of a shot or not.  So, if we take many pictures with a lens, with different focal lengths and different distances to backgrounds, and enough people rate the pictures, we can get a sense of how good a lens’s bokeh is (of course, what kind of pictures would have to be standardized again).  The excellent part of this is as follows: with enough input, we can have a pretty good idea of how a lens performs, bokeh-wise, at various focal distances and various distances to the blurred background both for the center and side of an image.  We can then use the image circles that are being made through the above technique, and input THEM into a learning computer algorithm which then LEARNS all by itself what constitutes good or bad bokeh baed on the image circles.  With enough images and enough lenses, we should eventually be able to forgo the human testing, and just check the blurred circles, run them through the computer, and get a GRADE.

Examples of analysis
Here is some of the stuff I worked on originally, using a small flashlight that used monochromatic red light.  It turns out, monochromatic red light is NOT a good idea, as different wavelengths can be refracted differently, so you could theoretically end up with color fringing, and that would probably be good to know about.  Furthermore, monochromatic light should theoretically be more prone to both positive and negative interference.  But the idea is basically the same.  For any lens we can analyze how quickly the edge drops off, the intensity of ring artifacts, etc.  The pictures below used a Tamron 24-135 lens (at 135mm) with a light source at 25 feet and focal distance of 7 feet.  Analysis in ImageJ can only be done on monochromatic light, which presents its own problems due to image conversion, but this is just an example.

7 ft PICT77087 ft PICT7708 2
  Actual Image                       Desaturated for Analysis

Using ImageJ, we get the following analysis:

Surface Plot1

It can also be interpreted this way:
line plot

The line graph can be input to a spreadsheet for the intensity of every pixel on the line.  Analysis can then be done in a more mathematical quantitative way than the qualitative way of just examining the pictures.  I think that using these tools will allow more accurate analysis of the bokeh of an image.