5.4.1. Software visualization of Processing 2

As mentioned earlier, choosing a software environment or a programming language greatly influences not only the kind of images obtained, but also how we think about the media they handle. In this study, the value of an environment such as Processing was essentially in the sense of “software esthetics”.

I have argued elsewhere that software art is important because it shows how software could behave differently, mainly through ruptures of function [REY 14]. Understanding such ruptures requires us also to understand how the software operates. Esthetics of software happens when we discover and use new software. Not only it is appealing to see and to interact with its elements, it also represents the vision of another designer or artist: how she thought the names, icons, functions and which algorithms were implemented.

There is a deep and large effect underlying media art software. The more we use it, the deeper we go into its ecology and ideology. Fully adopted, software shapes us as media designers and media artists, but at the same time it also means challenging its paradigms and mode of existence. When we discover new software, it happens that the mere production of ‘Hello World’ is satisfying, but it is also of the most significant importance precisely because it embraces engagement and motivates us to continue experimenting.

Furthermore, software esthetics finds an echo in software criticism. For example, media theorist Matthew Fuller understands software as a form of digital subjectivity [FUL 03, p. 19]. The human–computer interface is seen as having a series of ideological, socio-historical and political values attached to it. Studying the HCI implies investigating power relations between the user and the way the software acts as a model of action. This idea resonates with Winograd and Flores: “We encounter the deep questions of design when we recognize that in designing tools we are designing ways of being” [WIN 86, p. ix].

Inspired by the notion of “object-oriented ontology” and their visual representation as ontographs [BOG 12, p. 51], I created views of the Processing 2 source code (Figure 5.9) using software visualization tools and techniques. The idea was to put in relation to the underlying components of the software classes and entities. I downloaded the source code from the Processing github repository, then I used inFusion for basic software metrics and to convert into Moose format, which is required by CodeCity and X-Ray applications to produce a more graphical representation.

5.4.2. Graphical interventions on Stein poems

While Processing 1.x was an exciting tool for experimenting with visual information, one of the main caveats was its support for web browser distribution. In 2008, the situation changed when computer engineer John Resig, best known for creating jQuery, wrote the first version of Processing.js to be displayed on an HTML canvas. At that time, the graphical web was still in its early stages of development, thus Processing.js was only featured in WebKit-based browsers (Firefox, Safari, Opera).

Processing.js was inspiring to explore artistic hypertext as it does not ascribe specific forms of representation to data, but rather allows for the exploration of interrelationships between the system components. In collaboration with scholar Samuel Szoniecky and digital poetry pioneer Jean-Pierre Balpe, we worked together on “Stein Poems” with the artistic goal of experimenting and provoking unexpected behaviors of conventional computing in order to challenge cultural and perceptual habits [REY 16].

Stein poems are short fragments of text that refer to life in general. In early 2015, Jean-Pierre Balpe started creating such structures as part of his extensive research and development in automatic literary text processing since the 1980s. As the first text generators developed by Balpe were created on Hypercard, it was necessary to migrate the dictionaries and rules into more recent technologies. In late 2014, Samuel Szoniecky developed a complete authoring tool for generating texts using Flex and made it available through the web.

I used the generator API to produce graphical interventions in the web browser. The idea was to exploit plastic properties of text through recent graphical web technologies. As Stein poems are short fragments and combine vocabulary from multiple domains, they can be “experienced” and “touched” as visual images and user interfaces. Figure 5.10 shows two output interfaces, one is based on Processing.js and the other is rendered as SVG text using the library D3.js⁴. Both applications are interactive and use the same text API. While the first handles the web page as 2D matrix and applies rotation and translation transformations with a GUI, the second randomizes position and allows dragging/dropping each individual letter⁵. In the end, both tools create patterns and textures of text from the Stein poems.

5.4.3. Web-based visualization of rock album covers

While text as plastic element allows exploring visual configurations, the main challenge of media visualization within a web environment is to load great amounts of images as individual elements. In 2014, I prototyped “RockViz”⁶ in an effort to design interactive web-based image plots and mosaics. For this production, I gathered the most representative albums of different rock genres according to editors of the platform AllMusic.com. I quickly obtained almost 2000 albums, from folk rock to progressive rock and heavy metal.

The metadata collected was about the artist/group name, album title, release date and album cover image. I then downloaded the album cover images and extracted their basic visual features with ImageJ. In this respect, members of the Cultural Analytics Lab have created macro scripts for ImageJ and MATLAB that help the extraction process: ImageMeasure.imj, ShapeMeasure.imj, ImageMontage.imj and ImagePlot.imj. As mentioned earlier, scripting applications are common when working in media visualization. I have contributed with a different version of ImageMontage that supports different compressed file formats (PNG and JPG) and another script to extract colors according to the RGB color model. The latter script, ImageMeasure-RGB.imj, is shown in Table 5.3.

When the database was completed, I used Open Refine to handle data, but more importantly to apply mathematical formulae to numerical visual attributes and dynamically calculate their Cartesian position. The result was a dynamically generated column in Refine that constructed a full line of CSS style for each individual image, for instance: <img style="left:70.5%; top:4.0%; visibility:visible;">

I adapted this method to create an image mosaic, ordered according to: 1) median of hue; 2) median saturation; 3) median brightness (Figure 5.11). In the image plot visualization, the horizontal axis represented years (from 1955 to 2014) and the vertical axis hue values (from 0 to 360, following the HSB color model) (Figure 5.12). In GREL scripting language, which is natively supported by Refine, my formula to calculate plot horizontal positions is (((cells["year"].value/1.0) − 1950) * 1500) / 1000 and vertical positions is (((value/1.0) − 0) * 3500) / 10000. As can be seen, the delimited space is no bigger than 1500 x 900 screen pixels.

An additional functionality I added was a text input filter. Given the amount of images, I found it useful to explore by typing a year, an artist name or an album title. To make the loading of images a little faster, I produced two versions of each image: one is scaled to 100 x 100 px. and the other to 500 x 500 px. The small version is used for visual representations and the larger appears when the user clicks on an image therefore allowing her to observe more details of a single cover. I originally used JQuery and the function getJSON to communicate with a JSON database, but the loading time is very slow for more than a few hundred images.

With the intention of producing different organizations of images and to eventually observe different patterns, I modified my original formulae. I explored some combinations of sine and cosine functions taking advantage of the fact that GREL supports trigonometric calculations. Figure 5.13 plots the formula r2 = x2 + y2 ; x = r cos(θ) ; y = r sin(θ) and Figure 5.14 plots the formula r2 = x2 + y2 ; x = r cos(θ) * cos(ι) ; y = r sin(θ) * cos(ι). Other variations are sketched in Figure 5.15.

5.4.4. Esthetics and didactics of digital design

With the more recent development of web graphics, the possibilities of HTML5, CSS3 and JavaScript facilitate the way in which we can approach data visualization. Within an educational context my endeavor has the intention to introduce digital humanities to undergraduate students. Indeed, it may be regarded from three angles: first, as the construction of tailored toolkits of digital methods for students; second, as a contribution to the analysis of material properties of cultural productions; and third, as a mid-term strategy to orient students towards design-based learning techniques.

The way in which the three perspectives connect is as follows: we consider the realm of cultural productions populated by music albums, films, comic books, TV series, video games, digital art, architecture, industrial design, etc. Now, with the emergence of various kinds of tools and scripts for analyzing media data (text, images, audio, etc.), we select and assemble several of them in a tailored toolkit for studying cultural productions. Then we use the toolkit as a teaching methodology in the classroom. In the mid/long-term, our intention is to move students from the use of tools (as it happens in undergraduate courses) to the creation and design of tools, services and processes (as it happens in postgraduate courses).

The analysis of cultural productions deals with tasks such as gathering, documenting, representing and exploring valuable data about forms, materials, contexts, techniques, themes and producers of these productions. The resulting visualizations are put together into what I call “analytical maps”, which are helpful in the processes of identification of relationships, observation, comparison, evaluation, formulation of hypothesis, verification of intuitions, elaboration of conclusions and other humanities methods.

For example, a media visualization model might emerge from an abstraction of forms and layouts of web graphics, from simple HTML element positions to data and images. I coded an interactive version of the interface shown in Figure 5.16 (top), using exclusively HTML, CSS and jQuery. The GUI controller is implemented with the dat.GUI library⁷, made available by the Google Data Arts Team. In the bottom picture, the same organization is generated from a data table saved in JSON format. This table contains 201 images of paintings by artist Paul Klee (http://ereyes.net/kleeviz/). We had previously extracted visual attributes with ImageJ: red, green, blue, hue, saturation, brightness, and some shape descriptors. Of course, I am conscious that Klee was a prolific artist whose entire production counts in the thousands; however, I limited myself to images available from the website wikiart.org.

I present to students this and other examples in order to stimulate the exploration of different manners of using the same web technologies we employ in conventional websites. Among the original results, Figure 5.17 presents an experimental visualization made by three MA students in Interface Design. They chose to represent 609 comic magazine covers from Marvel in a vertical orthogonal view⁸. Each bar stands for a cover and its most prevalent color. Bars are positioned in a chronological order and clustered by season: spring, summer, autumn and winter. Other functionalities were proposed by students themselves: a color chart containing the 10 most common colors and hexadecimal color notations.

Other examples made with students are shown in Figure 5.18 and 5.19. The first one shows a simple implementation in CSS for an image slicing. Students of the MA Analysis of Digital Uses gathered pictures from actresses who won an Oscar between 2010 and 2016. With a simple right float position and shrinking the width value, the images are depicted in chronological order. When the user passes the mouse over an image, its size changes while showing a larger preview below the analytical map⁹.

The third example presents a modification of a D3.js script. Similar to Figure 5.7, this visualization replaces nodes as circles with images themselves –in this particular case, movie posters from films by director Quentin Tarantino. My workflow to introduce and personalize D3.js examples relies on RAW Graphs¹⁰, an open-source tool created by the DensityDesign Research Lab (Politecnico di Milano). Once students have constructed a database, RAW Graphs allows them to copy/paste the data to generate and personalize a static visualization. It is interesting to note that such visualizations can be downloaded as a compressed image (PNG), a vector image (SVG), and also as a data model in JSON format. Specifically regarding network graphical models, the output structure is an object containing two arrays: nodes and links. This data structure is the same employed in D3.js, hence students can easily personalize scripts from the examples by replacing the JSON file.

5.4.5. Prototyping new graphical models for media visualization with images of paintings by Mark Rothko

I have started working on media visualization with colleagues from other disciplines, most notably semiotics and information sciences. The Paul Klee visualization that I presented earlier was the subject of a communication at the congress of the International Association for Visual Semiotics held in Liège in 2014. Klee is a peculiar artist who has attracted semiotic analyses and thus my own visualization tried to put in perspective the different paintings analyzed within his complete production.

Another attractive artist for semioticians has been Mark Rothko. For this occasion, I prototype “Rothko Viz”¹¹. I gathered 203 images of paintings from wikiart.org and followed the already discussed workflow for extracting visual attributes: shape measurements, RGB and HSB color modes together with a variety of statistical derivations (median, standard deviation, mean). The database quickly had 60 columns and I produced Principal Component Analyses (PCA) of those dimensions. Figure 5.20 shows a parallel coordinate visualization of the 60 data columns made with Mondrian, and Figure 5.21 plots images according to the first two factors of the PCA, made with ImagePlot.imj. The main difference of plotting PCA values instead of, for example, years versus hue values as we did before is that images seem closer according to similarity. As we saw in section 4.2.4, PCA is a statistical method dedicated to dimension reduction and the numerical result derives from all the data columns.

With the intention to produce new media visualization models, I explored plugins already available for ImageJ. Among such plugins, the Polar Transformer¹² (created by scholars E. Donnelly and F. Mothe in 2007) proves to be useful with our context. The idea is to apply a Cartesian 360 degree transform to an orthogonal view of a series of images. Figure 5.22 depicts an infographic in which I placed an image mosaic to situate a range of images within the polar plot.

Although it may seem rather abstract at first glance, above all because the image set does not contain an image sequence that would help to create a continuous discourse, the design of guides and labels might assist in completing the sense. To furnish one more example where radial plots can be pictured, let us come back to Figure 5.6 (visualization of Nirvana videos). If we apply our technique to both vertical and horizontal orthogonal views, we obtain the graphical patterns in Figure 5.23.

One more model we introduced using Rothko images is the 3D image plot media visualization. In its first and foremost form, it is an extension of the 2D plot, but it now takes into account three different values for each of the three axes. The RGB model defines a cube, whose vertices map the relationships between red, green and blue: magenta, cyan, yellow, black and white. Another model that can be plotted in 3D is the HSB with a cylindrical form, where the top corresponds to the 360 degrees of the hue values, the height to the brightness and the distance from the centroid to the saturation. In ImageJ, it is possible to generate these visualizations with the 3D Color Inspector¹³ plugin (developed by scholar K. Barthel in 2004). A simplified version only for RGB cubes is also available as a plugin for Firefox, the Color Inspector 3D¹⁴ (developed by scholar David Fichtmueller in 2010). I developed a simple version of 3D color plots in Processing. Figure 5.24 shows these different versions.

The latest version of the prototype for 3D media visualization in web-based environments was an adaptation of the visualization engine by graduate student Mathias Bernhard [BER 16, pp. 95–116]. Although Bernhard’s project has different research goals, I employed the basis of his engine using WebGL and three.js. For this prototype, I adapted algorithms and layouts to display 3D media visualization in three plot forms: the RGB color model, the HSV color model and the same algorithm I conceived for “Rock Viz” in Figure 5.14.

As a matter of fact, given that the algorithm has now been used in 2D and 3D graphical spaces, it can be formalized as follows:

Given the equation of sphere S as:

• – Algorithm Polar HSB: Given a set values h = hue; s = saturation; b = brightness; where 0 ≤ h ≤ 360; 0 ≤ s ≤ 100; and 0 ≤ b ≤ 100:
  1. 1) For each row in the database, get values representing h, s and b.
  2. 2) Plot values according to x, y and z in function SVn (Sphere Variation n).
  3. 3) Display source image file at point positions.
  The variation SV1 being:
  • x = x₁ + r cos hπ cos sπ
  • y = y₂ + r sin hπ cos sπ
  • z = z₃ b

The graphical result is shown in Figure 5.26.

To conclude this section about emerging web-based graphical models, I want to point to recent ideas on creating an image generator with iconic similarity to the late works by Rothko. Indeed, the idea is to employ the visual attributes existing in a determined painting style. Then, as the forms are basically abstract, we can describe basic spatial composition rules. Incorporating colors, combinations of tonalities and format (portrait or landscape), I wrote a simple generator of Rothko images. For me, the interesting aspect of this kind of visualization is that of synthesizing images proven to mislead the viewer or to evoke Rothko values; then, the descriptive rules are effective. Furthermore, it could be imagined that the generator might be useful in experimental tests and other cognitive and psychological applications. Figure 5.27 shows a screenshot of one of the Rothko generators.

5.4.6. Volumetric visualizations of images sequences

In 2011, I used the term “motion structures” to describe my ongoing approach to exploring and interacting with motion media such as film, video and animations, as volumetric media visualization. My aim is based on the idea of representing spatial and temporal transformations of an animated sequence. For this kind of visualization, I am interested in representing the shape of spatial and temporal transformations that occur within the same visual space of the frame. The final outcome thus traces those transformations in the form of a 3D model. The mode of interaction with the 3D object allows for different ways of exploration: orbiting around, zooming in and out and even exploring the inside structure of the model.

Our process pays special attention to shapes above other visual features. From a technical point of view, to create volumetric media visualization, we use a script we wrote for ImageJ. Figure 5.28 depicts the overall workflow used to generate the 3D model that can be consulted in Table 5.4. Basically, an image sequence is manipulated as a stack and several operations are applied: converting to 8-bit format, subtracting background and transforming the stack as 3D shape in the 3D Viewer window. From the 3D Viewer, it is possible to save the result as a static image, as a 360° rotation movie and to export it as a mesh surface.

The obtained 3D shape encodes the changes in the objects in a frame: the different positions, the movement traces and spatial and temporal relations. The way in which we can interact with an object is not limited to ImageJ. An exported 3D mesh can be manipulated in other 3D software applications such as Maya, Sculptris or MeshLab. Furthermore, it is also possible to export a motion structure for the web or to 3D print it; however, both techniques require destructive 3D model processing, i.e. reducing geometry by simplification, decimation or resampling. For technical records, a motion structure exported from ImageJ has an average of 500,000 vertices and more than 1 million faces, which is a very large amount compared with an optimized 2000-face model for the web, to be loaded with the library three.js.

My first experiments with motion structures, initiated in late 2011 working on the shape of CGI visual effects sequences, started with the Paris fold-over sequence from the film “Inception” (Nolan, 2010). In November 2012, I produced a second motion structure: it came from the title sequence of the TV series Game of Thrones. I took a 5-second shot of a traveling pan of the fictional city of Qarth. The image sequence of the fragment has 143 images and they were staked and converted into a 3D model. For this project, I felt compelled to 3D print the model as a physical piece, a technique that can be called “data physicalization” (Figure 5.30). As it was required to reduce geometry for printing, the final object can be seen as a map of traces. We selected white plastic as printing material in order to convey the esthetics of rapid prototyping¹⁵.

At the occasion of the re-new 2013 digital arts forum, I selected image sequences from seminal video artworks by Charles Csuri, Peter Weibel and Bill Viola. The importance of taking into account video as input media was its inherent characteristics in opposition to cinema. The introduction of video and animation technologies for recording, processing and playing back moving pictures opened a wide range of possibilities for artists to explore and experiment with the esthetics of space and time. Contrary to cinema, video was more accessible, malleable and portable for artists. It was also easier and faster to watch and project the recorded movie. Finally, the look and size of the technological image was based on lines, reproduced at a different pace than film. While video art has been in itself a rhetorical movement against the traditional representation of moving images, motion structures of video artworks are at a second degree; an esthetization of the shape of time and space. Figure 5.29 shows a volumetric visualization of Bill Viola’s “Acceptance” (2008), 02:03 minutes, equivalent to 1231 frames; Figure 5.30 of Charles Csuri’s “Hummingbird” (1967), 02:10 minutes, 1295 frames; and Figure 5.31 of Peter Weibel’s “Endless Sandwich” (1969), 00:38 seconds, 378 frames.

Finally, in a more recent prototype, I have applied the same technique to a series of image sequences from web page design. “Google Viz” contains almost 200 screen shots from the Google home page, from 1998 to 2015, obtained with the Internet Archive Wayback Machine¹⁶ (Figure 5.32).