The Digital Society Research Cluster of CityU’s College of Liberal Arts and Social Sciences (CLASS) hosted an online seminar entitled “Digital Humanities and Multimodality” on 10 May 2023 to examine the intersection of digital humanities and multimodality and how this emerging field can contribute to future research in humanities. The seminar featured two keynote speakers. First, Professor John BATEMAN from Bremen University presented “The Relevance of Multimodality Theory for the Future of Digital Humanities.” He discussed how the basic notions of multimodal semiotics relate to digital humanities and the humanities in general and how relevant analytical methods can be scaled up to open up further issues and discussion points. The second keynote speaker was Dr Christoph HAFNER from CityU’s Department of English (EN), who spoke on “Digital Mediation in Specialised Communication.” He explored the application of digital mediation in professional communication and how genre theory and digital genres can be used to improve pedagogical applications for specialised language teaching contexts.
This seminar was organised by Dr Esterina NERVINO, Assistant Professor of EN, and the discussion was initiated by Dr TSUI Lik-hang, Convenor of the Digital Society Research Cluster and Assistant Professor of CityU’s Department of Chinese and History. About 70 people attended this seminar on Zoom across different geographies, disciplines, and career stages.
Dr Tsui welcomed the speakers and participants with an opening speech, introducing the research cluster’s mission and the importance of interdisciplinary research in understanding the relationship between technology and society. He highlighted the cluster’s commitment to fostering interdisciplinary exchanges and initiating conversations and collaborations through seminars and training events. “A thorough understanding of the relationship between technology and society, informed by the humanities and social sciences, must involve vigorous interdisciplinary research,” he stated.
Following the opening address, Prof Bateman, who is Professor of Applied Linguistics in the Linguistics and English Departments of the Faculty of Linguistics and Literary Sciences at Bremen University, highlighted the fit between multimodality and the digital and the need for a different approach to multimodal semiotics. He presented three types of digital humanities scholars: literary, blogging, and multimodal, and emphasised the importance of understanding the semiotically complex nature of cultural artefacts. He saw the limitations of traditional views of text as well as the need to move away from the idea that written language is the core. He explored the concept of modalities and their responsibility for material distinctions, emphasising the importance of discourse semantics as a mediator in meaning-making. Prof Bateman also stressed the need to make the process explicit and discussed the methodological consequences of not knowing the semiotic mode, including the need for multi-layer descriptions and encoding necessary information into corpora. Finally, Prof Bateman addressed the importance of raising the levels of semiotic abstraction and the challenges of analysing artefacts or performances where the relationship is unclear.
Coming up next was Dr Hafner’s talk. He discussed the following points: the relevance of English for specific purposes and applied linguistics in specialised contexts, the use of genres as forms in context, the values of the community, the purpose of authors, and the processes and activities of text construction in digital contexts. He also explored the concept of mediation, cultures-of-use, and the emergence of “polycontextual” digital genres on social media. Hafner questioned the novelty of digital genres and discussed the GEM model, developed by Prof Bateman in 2008, to analyse genres more systematically. He also highlighted the pedagogical applications of specialised language teaching in digital contexts.
After the talks, Dr Nervino moderated a fireside chat and Q&A session where speakers and participants discussed the challenges of generative AI in multimodal communication and emphasised the importance of interdisciplinary collaboration and new theoretical frameworks. Dr Tsui then raised a question about the role of digital humanities in studying multimodal conversational AI. Prof Bateman highlighted the ambivalence of AI, particularly in linguistic study aspects, mentioning that it brings convenience but also gives rise to some criticism. He also pointed out that we have not yet reached a stage where AI technology can fully comprehend all aspects of language and communication. There is a need for further research in multimodal semiotics and English for specific purposes, which is a pedagogically driven enterprise. Dr Hafner suggested that broader studies are required, not limited to case studies, and that scaling up works theoretically and practically. The analysis of more complex materialities, including interactive examples such as digital games, was also emphasised.
