Ways of Seeing Algorithmically

Artists often claim to want to change the world through their work, a lofty goal, which if it happens at all, seldom does so predictably. But in 1972 a small book and a four-part television series influenced by Marxist materialism and Brechtian dramaturgy transformed the way people thought about art. Ways of Seeing, authored by the painter and writer John Berger and three collaborators, related complex and unfamiliar ideas from economics, cultural theory, and feminism to famous works of classical art, and in the process made these ideas seem not only accessible, but vitally relevant for understanding the history and meaning of these paintings. If Berger and his collaborators had one point to make, it was that seeing, what we might more specifically define as ‘active looking’, is always a political act.

Ways of Seeing has gone on to sell millions of copies in dozens of languages, and has become a staple of arts reading lists the world over. But since 1972, a seismic shift has occurred, which Berger’s book could not have anticipated, and this is that humans no longer have a monopoly on seeing, in this active sense of the word. Today, an ever-expanding number of artificial vision systems are being built and deployed in an ever-widening array of areas, from law enforcement, to agriculture, from transportation to military surveillance. Combining image capture systems with forms of artificial intelligence, these ‘vision machines’, as Paul Virilio presciently dubbed the systems that directly preceded them, have the potential to fundamentally change almost every area of life. For some, these new technologies will be liberators from the drudgery and trauma of the demand to look, for others they will be overbearing and unblinking supervisors and jailors.

In response to this, I have begun a new project that looks to rework Berger’s Ways of Seeing into a new book, intended to be a primer on the politics of artificial vision in much the same way that the original was a primer on the politics of human sight. Titled Ways of Seeing Algorithmically, it exists primarily as an augmented reality application that is driven by the same technology it is a critique of. When run on a device whose camera is pointed towards a copy of Berger’s original 1972 book, this app uses computer vision to map a new virtual book onto the physical pages of the original. This app acts as the hub, or centre of the Ways of Seeing Algorithmically project, but it also serves as the starting point for a series of other mini-projects, collaborations and creations, each dealing with a different facet of this vastly complex and rapidly evolving field. Alongside this creative work, my PhD research at the London School of Economics approaches this same subject from an empirical, social sciences perspective, exploring the cultures and ideologies that give rise to these artificial visions systems. Rejecting the determinist materialism that has often typified Marxist accounts of technology, I aim for a description of these technologies as things which both shape society and culture, and are shaped by these things.

Much attention, both artistic and scholarly, is being directed towards the more overtly disturbing implication of these technologies, for example the inherent prejudices that are often hard written into their designs, or the problematic implications of taking human agency out of decision-making processes in arenas like armed conflict or criminal justice. This focus is valid and urgent, but what concerns me is that the focus on the most obviously dire consequences of these innovations does not necessarily eliminate the causes that lie at their root, but instead drives them deeper, further out of sight, in a way which might allow us to sit comfortably and falsely console ourselves that these technologies have been neutralised and made unproblematic by our efforts. It is these subtler aspects that are the concern of Ways of Seeing Algorithmically, questions about the foundational assumptions on which systems like computer visions are built.

An example of these concerns is the impact these technologies have on our relationship with our own sight. Reflecting a deep cultural fascination with artificial life, something with origins as old as Ancient Greece, there is a strong and problematic tendency to anthropomorphise these technologies. This confers a sense of agency on them, which they in fact lack, and lends credence to a dangerous technological determinism that asks ‘what the algorithm wants’ rather than asking what we want of the algorithm, and giving us as citizens the agency to shape these technologies to our needs. It is also problematic as we attempt to understand these systems through the prism of our own biological limits, even at the same time that we are designing them in many cases precisely to exceed those human capabilities.

Sight is a particular example here, where we often think of the goal of computer vision being to create an analogue to human vision, and often speak of it in terms that compare the two. However, there is no precise consensus on how human sight works, or what the normative baseline for an artificial model of it should be. Sight is a messy, complex process of which only around a tenth occurs in the physical structure of the eye itself. Dziga Vertov might have famously called the camera a ‘mechanical eye’ but in truth that eye did little to create the meaning of his dazzling visual spectacles.

The viewer’s brain, as an extension of that mechanical eye, does most of the work, and so it would be better to speak of the human operated camera in the spirit of Donna Harraway, as a cyborg eye, technology enmeshed with the vision making structures of the human brain, and consequently also embedded in all the subjective vagaries of human biology. When we speak of ‘sight’ in a singular sense we ignore the multiplicity of biological sight, and even setting aside psychoneurological conditions like synthesia or colour blindness, within a ‘healthy’ sample of people visual experience of the world can vary widely.

But beyond these biological vagaries, it is also problematic because of the way we make sense of the things we see. Humans are beings in and of culture, and so we cannot see the world around us apart from this culture. To a human viewer a tree is never just a tree, it is a rich treasury of diverse and sometimes contradictory cultural meanings, which we can acknowledge but never fully escape from. Machine vision systems, while they can be developed to recognise specific things and specific relationships between things, have never come close to understanding this complex web of cultural significance, and if some of the AI researchers who are more critical of the field’s hyperbole are to be believed, they will never reach these levels of cognitive ability. This is important, because when we talk about ‘seeing’ in the terms that Berger and his collaborators meant, it still remains unproven that artificial vision systems can in fact be said to see at all.

In response to these and other issues, I have been working with generative artist and creative coder Matt desLauriers to develop a system which re-renders digital photographs into new representations based on the information contained in the source image. The principle here is that the new images contain broadly the same data as the original, but re- presented in different configurations which are semiotically meaningless to human eyes, but to a properly configured computer vision system provide broadly the same information as the original. The aim in part is to suggest the profound differences the ways that these systems seem images, not as we do, as a charged arrangement of symbolically significant objects, greater than the sum of their parts, but rather as a largely meaningless aggregations of spatial and tonal data.

This collaboration is one of around a dozen planned to emerge from Ways of Seeing Algorithmically over the coming years, orbiting the central hub of the augmented reality app itself. Berger ended his Ways of Seeing with an entreaty for his work to be continued by the reader, and that is very much the spirit of Ways of Seeing Algorithmically. This is not a project with a predefined end, or even with a particular ambition for what it will conclude or say. It is an inquiry into concerns and technologies both very old, and rapidly changing, and like Berger’s book it is one to be continued by the viewer.