POST

HAL Likes Photos

Monday August 23rd, 2010

Interesting new directions in computerized photography categorization are emerging from my hometown (and Bill’s alma mater). Penn State University researchers Jia Li and James Wang introduced an online database a couple years ago called ALIP, a way of searching the content of photographs. Li was quoted in the press release explaining the technology:

Our basic approach is to take a large number of photos—we started with 60,000 photos — and to manually tag them with a variety of keywords that describe their contents. For example, we might select 100 photos of national parks and tag them with the following keywords: national park, landscape, and tree… We then would build a statistical model to teach the computer to recognize patterns in color and texture among these 100 photos and to assign our keywords to new photos that seem to contain national parks, landscapes, and/or trees. Eventually, we hope to reverse the process so that a person can use the keywords to search the Web for relevant images.

That search system was made available under the name ALIPR, which stands for Automatic Linguistic Indexing of Pictures in Real-time. Users can help the machine learn by evaluating the accuracy of its keywords, and by uploading their own photos. Li hints at the significance of this technology in the last sentence quoted above. Image search engines like Google’s currently search the filename of the image, the link pointing to the image and the adjacent text to generate results; but ALIPR could potentially allow users to search the images themselves.

Since then the same team, along with their collaborator Ritendra Datta, has introduced another online photography robot that evaluates the quality of a photo. The new system is called ACQUINE, for Aesthetic Quality Inference System. Once again, the system gathers data from the photographs and produces an evaluation, which it compares to the ratings given by real humans on photo-sharing websites. Wang said in the press release for ACQUINE that there is “more than 80 percent consistency between the human and computer ratings.”

I decided to test these programs by uploading some photos. First, I put this photo by our Atlanta-based photographer Alex Martinez into ALIPR.

These are the keywords that ALIPR came up with (click for a bigger image):

Well, not bad. I was impressed to see that the computer was able to come up with the keywords female, supermodel, face and “ocean-animal,” which I checked off to help ALIPR learn. Excited, I pulled out another of Alex’s pictures to see if the trend would continue.

And here are ALIPR’s results (click for a bigger image):

Needless to say, this was a disappointment. No mention at all of the tasty-looking chocolates; instead, this attractive picture had been identified as a dinosaur dancing ballet.

I decided not to subject Alex to ACQUINE’s photography criticism, opting instead to do a controlled test. So I uploaded a well-known photo by Ansel Adams, and here is the result:

Though a rating of 24.8 out of 100 may not necessarily mean a bad photo, it is a little confusing. Perhaps AQCUINE prefers photographs taken by Uncle Earl.

In case you were wondering what all this software could be used for, Wired reports that designer Andrew Kupresanin has put together a new camera called Nadia that uses ACQUINE to evaluate your composition as you are taking a photo. The “camera” is actually a Nokia N73 cellphone communicating with a nearby computer through Bluetooth—but it certainly seems like a step towards cyborg photography. Here’s a video demonstration:

link