How machines see people


Millions of photos are uploaded to the Internet every day. There have long been algorithms that are able to recognize faces on them. But how does this technology work? An artist researched this – with disturbing results.

Sterling Crispin has studied the face of surveillance and printed it out on his 3-D printer. It’s a mask with a dented surface. It looks like egg white sizzling in the pan and making bubbles. But an algorithm sees it differently. If he has his way, the answer is clear: the mask is a human face, composed of the available information. It took Crispin a year before he could hold the mask in his hands. Now he says, „This is how machines see us. It’s very disturbing.”

This is disturbing for Crispin not only because the mask looks so inhuman, which – in contrast to the machine – he would never have expected. What irritates him most is the message behind it: This strange grimace is the basis for powerful surveillance systems. When police officers take tablets on the patrol in San Diego, California and use them to photograph suspicious people, the software compares this image with a database; this comparison is based on algorithms. The machine view is becoming the focus of police work.

The FBI does something similar. With its ” Next Generation Identification System ” program, the next generation of identification, the agency plans to add more than 52 million faces to its database by 2015. „They mix pictures of people with a criminal record with photos of innocent people,” says Crispin. The FBI states that there is an 85 percent chance of finding the perpetrator – if his picture is in the database at all, it will appear in the first 50 suggestions. „This rate is very low,” says Crispin, visibly indignant, „in fact, that means that 49 innocent people are proposed as potential perpetrators for a crime with which they have nothing to do with.”

The more different the images, the better the algorithm learns

Crispin is 29 years old and is studying multimedia technology at the University of California at Santa Barbara. The „Data Masks” project is his thesis , and he will submit it this Friday. Crispin spent a year analyzing exactly how an algorithm works in face recognition . You have to imagine algorithms like recipes. They go through clearly defined individual steps and thus deliver results. The variant of the algorithm that Crispin used is capable of learning. The more data she gets, the better she can refine her results.

So Crispin chose an image database with 6000 faces , which is freely available on the Internet – it contains a multitude of skin colors, genders and gaze directions. The more different the images are, the sooner the algorithm learns. Passport photos, for example, in which people traditionally look straight into the camera, usually without smiling, convey a different picture of reality than vacation photos. The database therefore offers a large selection for recognizing the special features of a human face. These faces serve as the basis for the program. It analyzes what a face normally looks like. If pictures are presented to him, the algorithm checks on this basis whether a face can be seen in the picture presented to him.

Compared to Facebook, the proverbial face book, 6000 pictures are of course little. 350 million photos are uploaded to Facebook every day and that is the status of 2013. There should be significantly more by now. 70 million photos land on Instagram, which is also part of Facebook, every day (this number also includes the uploaded videos).

A face divided into countless rectangles and pixels

The type of face recognition that Facebook also uses, however, is likely to be similar to the variant of Crispin. He developed a software himself. He took snippets of code from scientific papers and wrote a lot of them himself. The 2-D model of a face, for example, is divided into countless rectangles and the algorithm checks to what extent the end result corresponds to the results that, according to the algorithm, represent human facial expressions. It’s about details as big as a pixel and not about realizing what a nose or mouth looks like.

Since the calculation is carried out by an algorithm, it can be automated in such a way that it is a process that takes hours of comparison, with countless attempts – and correspondingly better results. „Imagine you breed 1,000 dogs for ten thousand years, but only those with long ears are allowed to pass on their genes,” says Crispin, to explain his work more clearly. „In the end, all dogs have long ears.”

„I’m not sure if it’s any good to train these autonomous systems,” says Crispin. He thinks people shouldn’t rely on these systems. If that’s the way people are analyzed by machines – as statistical patterns – it is dangerous to treat that as evidence. Data are falsifiable.

His data mask is an act of political protest through transparency. According to the motto: You have to know what exactly you are talking about. His masks are a symbolization of what is otherwise described as invisible technology. Face recognition algorithms exist and work, but are usually not visible. Humans have no idea what it is to be monitored. Crispin has worked to change that.