AbstractsComputer Science

This work investigates Sparse Neural Networks, which are artificial neural information processing systems restricted in their structure and dynamics to conserve resources. The concept of sparseness refers to the network"s connectivity structure, such that each neuron receives inputs from only a limited number of other neurons, and to the network"s state which describes the level of activity of the entire neural population, so that only few neurons are active at any one time. The mathematical tool to achieve these properties, a projection operator that finds the best approximation to any given vector fulfilling a pre-defined sparseness degree, is extensively analyzed and efficient algorithms for its computation are proposed. Since the target sets for the projection are non-convex, standard approaches are not applicable and new methods must be developed to solve this optimization problem. Building upon this theory, original models suited to solving classification tasks subject to explicit sparseness constraints are proposed and their characteristics studied. In doing so, novel approaches to solving the problems of inefficient inference and poor discriminatory performance of sparse representations are developed. It is demonstrated that sparseness acts as a regularizer and significantly better classification results can be obtained compared to classical non-sparse models. Moreover, the sparse data flow can be exploited to reduce the computational complexity of the application of a trained classifier by approximately one order of magnitude. [Full abstract available in main document.]