Unified Multi-modal Multi-task Learning Paradigm for Smart Cities

Principal Investigator:

Professor Xiangyu ZHAO
Assistant Professor, School of Data Science

Co-Principal Investigator:

Professor Zijun ZHANG
Associate Dean and Professor, School of Data Science

Project Period:  1 November 2022 - 31 October 2024

The smart city research and industry community aspires to develop a unified model that can address a large amount of urban computing tasks and modalities. The primary challenge in achieving this objective is to capture the vast variety of tasks, modalities, and training algorithms inside a single framework. To this end, we aim to proposed a unified framework to ensure architecture adaptiveness across various urban tasks and data modalities, allowing for the possibility of generalizing to new tasks inside the model. In addition, we will depict the potential smart city tasks involving several modalities in a unified paradigm. Thus, we will take into account the multi-task and multi-modal nature while designing the unified paradigm. 

The main objectives include:

1. Develop an AI system that can continually and swiftly process the volume of urban data, and design a distributed data-storage and processing infrastructure to improve the performance and efficiency of artificial intelligence and machine techniques.
2. Propose a scheme that uses heterogeneous data with multiple structures and modalities, and fuse multi-modal datasets using the following three techniques: stage-based approaches, DNNbased approaches and semantics-based approaches.
3. Design a unified paradigm that can tackle multiple smart city tasks in a single model. In general, we will devise two types of unified paradigms, i.e., automated machine learning paradigm, and pretraining finetuning paradigm.
4. Layout an experimental prototype system to demonstrate the effectiveness of our proposed paradigm, and extensive testing will be conducted by using real-world urban task datasets to evaluate our proposed research.