WIC 2014 Tutorial - Intelligent multimodal human-computer interfaces

Intelligent multimodal human-computer interfaces

Andrzej Czyżewski - Gdańsk University of Technology, Multimedia Systems Department, Gdańsk, Poland
Bożena Kostek - Gdańsk University of Technology, Audio Acoustics Lab, Gdańsk, Poland
Piotr Odya, Bartosz Kunka, Michał Lech - Gdańsk University of Technology, Multimedia Systems Department, Gdańsk, Poland


Multimodal interfaces development history will be reviewed briefly in the introduction to the tutorial. Methods for intelligent processing of audio and video will be discussed in the context of their applications to multimodal human-computer interfaces. Some examples of applications of multimodal interfaces to education software and for the disabled people will be shown, including the eye-gaze tracking system named “Cyber Eye” employed to many kinds of experiments including analysis of visual activity of patients remaining in vegetative state and their awareness evaluation. The scent emitting multimodal computer interface, playing an essential role in education and therapy of children with certain developmental disorders will serve as one more practical example of applications. The multimodal interface called Virtual-Touchpad (VTP) used for supporting medical diagnosis will be presented also. The role of multimodal computer interfaces applied to learning, therapy and everyday usage of computerized devices will be illustrated by above mentioned and by some more practical examples. Moreover, the subject of intelligent audio & video surveillance providing a special case of multimodal interfacing will be addressed and illustrated with practical application examples.


The first part of the tutorial concerns application of soft computing to dynamic and static hand gesture recognition in computer vision [1]. The problem of recognizing gestures performed in the air has been addressed considering two environments, i.e. with a back facing and front facing camera. The possible solution for the first case has been presented based on two applications developed by the Multimedia Systems Department – the Virtual Whiteboard and Gesture-based Sound Mixing System. The methodology overview begins with presentation of image processing methods utilized in a specific computer vision configuration employing a camera and multimedia projector. Next, the fuzzy rule based system applied to recognition of dynamic gestures following the motion trajectory smoothed with Kalman filters is been described. The way of using Support Vector Machines for recognizing static hand gestures is explained.

A part of the tutorial dedicated to the Cyber Eye focuses on a specific application of eye-gaze tracking technology and EEG analysis [2]. The problem of objective assessment of non-communicative patients’ consciousness state is to be presented. The calibration procedure of training the system to recognize the patient's gaze direction based on information included in the eye image is explained. The proposed methodology based on employing multimedia interfaces supporting the diagnosis and neurorehabilitation will be described. Positive influence of cognitive processes’ and polysensory stimulation utilizing multimedia approach is described.. Computer applications developed at the Multimedia System Department dedicated to patients with disorders of consciousness as well as non-communicative patients, i.e.: Consciousness Test, Multimedia Training, Agraphia and Alexia, Pictograms and Virtual Keyboard will be also presented.

A subsequent part of the tutorial presents a new approach to diagnosing Parkinson’s Disease [3]. The progression of the disease can be measured by the UPDRS (Unified Parkinson Disease Rating Scale) scale which is used to evaluate behavioral and motor symptoms of Parkinson's Disease. Hitherto the evaluation of the advancement of the disease in the UPDRS scale is made by a specialist through medical observation. A partial automation of this process using a gesture interface named Virtual Touchpad has been described. The Virtual Touchpad is based on the processing of hand image acquired by a camera. The patient’s hand is observed while performing the test and the Virtual Touchpad assigns a hand gesture to the given class utilizing the SVM classification method. The hand movement velocity is the measured and used in the diagnostic process.  

More multimodal human-computer interfaces will be discussed with a special regards to intelligent data processing techniques underlying their principle of working.  

Modern intelligent surveillance systems can serve as a special case of multimodal interfacing. Consequently, video processing and event detection methods designed to be applied in multi-camera surveillance systems will be discussed. They were designed in order to: detect moving objects in fixed camera streams, tracks them, classify them, and discover user-defined events including intrusion and wrong-direction movement, as well as specialized events, such as abandoned luggage, events occurring in a parking lot (parking time and  place identification, entering and leaving a parking lot, etc.), and traffic events (prohibited lane change, stopping in the middle of an intersection, etc.) [4]. Moreover, intelligent processing of acoustical signals can help to detect security threads through automatic recognition of gunshots, broken glass or cry for help [5].


  1. Czyżewski A., Kostek B., “Intelligent video and audio applications for learning enhancement”, Journal of Intelligent Information Systems, vol. 38, 3, pp. 555-574, 2012.
  2. Kunka B., Czyżewski A., Kwiatkowska A., “Awareness Evaluation of Patients in Vegetative State Employing Eye-Gaze Tracking System”, International Journal on Artificial Intelligence Tools, No. 2, vol. 21, pp. 1 - 11, 4.2012.
  3. Maziewski P., Suchomski P., Kostek B., Czyżewski A., An Intuitive Graphical User Interface for the Parkinson’s Disease Patients; 4th International IEEE EMBS Conference on Neural Engineering, pp. 14 - 17, Antalya, Turkey, 29.4.2009 - 2.5.2009.
  4. Czyżewski A., Lisowski K., Employing flowgraphs for forward route reconstruction in video surveillance system; Journ. of Intelligent Information Systems, pp. 1 - 15, 2013.
  5. Łopatka K., Czyżewski A., Acceleration of decision making in sound event recognition employing supercomputing cluster; Information Sciences, 2013.


Prof. ANDRZEJ CZYŻEWSKI is Head of the Multimedia Systems Department of the Technical University of Gdansk. He received his M.Sc. degree in Sound Engineering from the Gdansk University of Technology in 1982, his Ph.D. degree in 1987 and his D.Sc. degree in 1992 from the Cracov Academy of Mining and Metallurgy. In December 1999 Mr. President of Poland granted him the title of Professor. In 2002 the Senate of his University approved him to the position of Full Professor. He is author of more than 500 scientific papers in international journals and conference proceedings. He has led more than 30 R&D projects funded by the Polish Government and participated in 7 European projects. He is also an author of 10 Polish patents and 5 international patents. He has extensive experience in soft computing algorithms and image & sound processing for applications in surveillance and in acoustical monitoring of environment.

Prof. BOŻENA KOSTEK holds professorship at the Faculty of Electronics, Telecommunications and Informatics, Gdansk University of Technology (GUT), Poland. She is now Head of the Audio Acoustics Laboratory. She is a Fellow of the Audio Engineering Society. Bozena Kostek received her M.Sc. degree in Sound Engineering from the Technical University of Gdansk (1983). From 1987-1989, she Kostek studied at the Paul Sabatier University in Toulouse, France, receiving DEA d'Acoustique. In 1992, she supported her thesis devoted to the quality of the pipe organ control systems and received her Ph.D. degree with honors from the Technical University of Gdansk. In March 2000 she supported her D.Sc. degree at the Institute of Research Systems of the Polish Academy of Sciences in Warsaw. In 2005 she was granted the title of professor from the President of Poland. She is now full professor at the Faculty of Electronics, Telecommunications and Informatics, Gdansk University of Technology. In 2003 she was elected Corresponding Member of the Polish Academy of Sciences.
Dr. PIOTR ODYA was born in Gdansk in 1974. He received his M.Sc. in 1999 from the Faculty of Electronics, Telecommunications and Informatics, Technical University of Gdansk, Poland. His thesis was related to the problem of sound quality improvement in the contemporary broadcasting studio. He is interested in video editing and multichannel sound systems. The goal of Mr. Odya Ph.D. thesis concerned methods and algorithms for correcting stuttering. In 2007 Piotr Odya received his Ph.D. degree. At present, he works at the Multimedia Systems Department as an Assistant Professor.
Dr. BARTOSZ KUNKA was born in Starogard Gdanski in 1983. He received his M.Sc. degree in 2007 from the Faculty of Electronics, Telecommunications and Informatics, Technical University of Gdansk. His thesis was related to stereoscopic imaging, particularly with anaglyph movie. He realized 3D movie. In 2012 he completed his Ph.D. dissertation The eye-gaze tracking system supporting audio-visual correlations research. His scientific interests are associated with image processing, video recording and editing and telemedicine.
Dr. MICHAŁ LECH was born in Gdynia, Poland in 1983. In 2007 he graduated from the faculty of Electronics, Telecommunications and Informatics, specialization: Distributed Applications and Internet Systems. The subject of his thesis was the application for automatic pitch detection and correction of detuned singing. In June 2013, he received his Ph.D. degree. The subject of the dissertation was: "The method and algorithms for controlling sound mixing processes by hand gestures using video image analysis". He is interested in image processing and artificial intelligence algorithms.



Warsaw - The Old Town

Panorama of Warsaw

Chopin statue in Łazienki park

University of Warsaw - Library and gardens

Warsaw - Palace in Łazienki park

University of Warsaw - WIC 2014 venue

University of Warsaw - Central Campus

Glimpse of modern Warsaw

Warsaw - Castle Square

Warsaw - Downtown by night

Warsaw - Royal Castle seen from the river

Warsaw University of Technology