In May 2015 I participated in the first edition of the Digital Humanities Hackathon at the University of Helsinki. During the week four multidisciplinary teams conducted small research project with different datasets. It was a super-intensive week with lot to learn – both about the methods, about coding Python, and for me as a social scientist about humanities too!
This blog post is a cross-posting from Day of Digital Humanities site and summarizes the work of our Multimodality group in the hackathon. Group members were Dragana Cvetanovic, Arja Karhumaa, Pasi Kojola, Salla-Maaria Laaksonen, Taina Laaksonen & Aaro Salosensaari, and our team was guided by Tuomo Hiippala. A summary of the whole week including other teams work can be found on here.
Background for the study: representations of Finland in Finnair’s in-flight magazines
There has emerged a strong interest in nation branding among both practitioners and academics (e.g. Aronczyk 2013). Media is one prominent environment where nation brands are built and maintained. What is notable is that media content is not exclusively linguistic. Newspapers and magazines, for instance, combine photographs, infographics and typography in a layout to communicate with the reader. This phenomenon is often referred to as multimodality.
Yet most studies have focused exclusively on the textual or visual aspects of representing a nation brand. Consequently, the joint contribution of language and images has been rarely given consideration in the study of nation branding.
During the Digital Humanities Hackathon our team adopted a multimodal approach to study how Finland and Finnishness are represented using multiple modes of communication in Blue Wings, the in-flight magazine of Finnair. We propose these articles convey an image of Finland to both business travellers and tourists. Our final research question thus was: what modes of communication are used to represent Finland in Finnair’s in-flight magazines?
Data and methods and lessons learned
Studying multimodality consumes both time and resources, because how language, images, layout and other modes of communication combine varies from page to page. We learned that when focus shifts to images or other visual content also methods get more complicated: what is easy to see for humans can be very difficult for computers. In practice, we had to code all our tools ourselves during the hackathon.
To locate the pages that mention Finland or Finnishness, we first extracted and searched the text contained in the data set. Having identified the relevant pages using a Python script, we returned to examine their layout. We then used a computer vision algorithm to identify elements on the page and applied machine learning to classify them into two categories: texts and images.
Preliminary results and exploration
First of all our preliminary analysis shows that most of the content representing Finland is in image format. In Figure one the different issues in our data set are presented in lines according to page number. The size of the bubble represents the pixel size of a particular clip on paper. Blue color is for images and black for text. As you can see textual parts concentrate to the end of each issue – on these pages in each number there is a section dedicated for in-flight information. Thus, while the visualization is hardly a research finding as such, it does imply that our computer vision algorithm is identifying images and text properly.
Figure 1. The content of ten different issues portrayed in bubble map arranged by page numbers. Visualized with RAW.
Next, to dive deeper into the textual content a LDA topic model was conducted to find different topics in the textual parts of the data. Using the algorithm we found twenty different topics. They are listed below in order of prominence.
- Life and family
- Finland/Finnair info
- Nature and ecotravel
- Business
- Cultural events
- Finnair services, aviation
- In-flight entertainment
- Politics & economy
- (broken parts of words only)
- Design
- Finnair services
- Food, eating in/out
- Sauna & other Finnish classics
- Environment
- Culture (music, artists, books)
- Customer loyalty program of Finnair
- Finnish sports
- Wellbeing
- Work, companies
- Editorial information
Finally, we wanted to look in which modes of communication these topics are represented, i.e. to study if some topics are more visual than others. During the hackathon we had only time to do preliminary exploration with some of topics just to check if the idea is applicable. Figures 2 and 3 below show some examples of the distribution between image and text in two different topics. The charts show that cultural events are presented with more and larger images than sports, and also that the layout on the cultural event pages is more scattered as the amount of clips is larger.
Future directions
After this preliminary study the developed methods could be used to study multimodality in all in-flight magazines of European or other state-owned airlines to investigate the ways how nationality and nation branding interact in these magazines. In this way we could build more comprehensive and comparative research setting.
In order to contribute to practice-based fields such as graphic design, the proposed method can be used to trace the development of design conventions, whose understanding is an important aspect of multimodal literacy. The development various genres of the magazine medium could be traced using the data available from Google Books and the National Library of Finland.
Moreover, the method could be trained to recognize specific elements in graphic design, in order to distinguish between different types of images (photographs, illustrations, information graphics), headers, captions and body text, etc.
But how to automatically examine the visual content? We could extract prominent nouns on each page and use them to retrieve a set of training images from ImageNet (www.image-net.org). ImageNet contains thousands of images for each noun, which can be used to evaluate the content to of the images found on the page.
Further reading:
- Aronczyk, M. (2013). Branding the Nation: The Global Business of National Identity. Oxford: Oxford University Press.
- Hiippala, T. (2013). Modelling the structure of a multimodal artefact. Helsinki: University of Helsinki.
Rajapinta 