Plenary Lecture
On New Visualization Tools, Data Mining Methods and Mathematical Techniques in the Analysis of the Weight Space of Neural Networks Solving Complex Real World Tasks for Pattern Recognition, Control and Function Approximation
Professor Dimitrios A. Karras
Dept. Automation, Hellas
Chalkis Institute of Technology
Greece
E-mail: dakarras@ieee.org
Abstract: One of the main reasons for the slow convergence and the suboptimal generalization results of MLP Neural Networks (Multilayer Perceptrons) based on gradient descent training is the lack of proper initialization of the weights to be adjusted. Even sophisticated learning procedures are not able to compensate for bad initial values of weights, while good initial guess leads to fast convergence and/or better generalization capability even with simple gradient-based error minimization techniques. Although initial weight space in MLPs seems so critical there is no in depth study so far of its properties with regards to which regions lead to solutions or failures concerning generalization and convergence in real world problems. There exist only some preliminary studies for toy problems, like XOR. Therefore, the topological properties analysis of such Neural Network weight spaces emerges as an important issue. The first major scope of this plenary talk is to present visualization tools and techniques based on a data mining approach, involving suitable Self Organizing Feature Maps (SOFM), in order to demonstrate that a complete analysis of the MLP weight space is possible even in the case of complex real world problems. Moreover, involving the transformed space of Self Organizing Feature Maps, a deterministic chaos approach is employed to quantitatively estimate and explore the transformed weight space revealing critical topological properties.
On the other hand, research attempts on weight initialization in Neural Networks have provided interesting results and led to the development of a number of initialization procedures. In this plenary talk, after a critical overview, we present, as a second major goal, enhanced simple initialization procedures for back-propagation trained Multi Layer Perceptrons that employ nodes with sigmoid activation functions, involving simple algorithmic schemes based on interval arithmetic considerations, remarkably improving previous literature results. Such new efficient schemes based on interval analysis show the value of this important mathematical tool in nonlinear systems, like neural networks, analysis for devising new initialization and training algorithms.
The herein presented visualization tools based on data mining approach involving SOFM, the analysis of SOFM transformed MLPs weight space based on deterministic chaos calculations as well as the rich mathematical methods of interval analysis applied to weight space algorithmic estimation, for improved initialization and training procedures are considered and evaluated in a series of real world benchmarks including pattern recognition, control and function approximation tasks obtaining remarkably promising results.
Brief Biography of the Speaker: Dimitrios A. Karras received his Diploma and M.Sc. Degree in Electrical and Electronic Engineering from the National Technical University of Athens, Greece in 1985 and the Ph. Degree in Electrical Engineering, from the National Technical University of Athens, Greece in 1995, with honours. From 1990 and up to 2004 he has collaborated as visiting professor and researcher with several universities and research institutes in Greece. Since 2004, after his election, he has been with the Chalkis Institute of Technology, Automation Dept., Greece as associate professor in Digital Systems and Signal Processing as well as with the Hellenic Open University, Dept. Informatics as a visiting professor in Communication Systems (since 2002 and up to 2010). He has published more than 55 research refereed journal papers in various areas of pattern recognition, image/signal processing and neural networks as well as in bioinformatics and telecommunications and more than 155 research papers in International refereed scientific Conferences. His research interests span the fields of pattern recognition and neural networks, image and signal processing, image and signal systems, biomedical systems, communications, networking and security. He has served as program committee member in many international conferences, as well as program chair and general chair in several international workshops and conferences in the fields of signal, image and automation systems. He is, also, editor in chief of the International Journal in Signal and Imaging Systems Engineering (IJSISE), topics editor in chief of the International Journal of Digital Content Technology and its Applications (JDCTA) as well as associate editor in various scientific journals. He has been cited in more than 500 research papers, his h-index is 10 and his Erdos number is 5.