Laboratory support for the teaching of neural networks

International Journal of Electrical Engineering Education, Jan 1998 by Fulcher, John

Abstract Experiences gathered over the past six years in teaching a graduate neural networks subject are reported. Various neural network software simulators have been critically evaluated during this time, as to their suitability for supporting the laboratory component of this subject. Specific examples obtained using the NeuralWorks Professional II simulator are demonstrated.

1 BACKGROUND

In common with educators at numerous other post-secondary teaching institutions, at the University of Wollongong we have held the view for some time that effective learning cannot take place unless subjects incorporate a significant laboratory component. Such a view is consistent with the assertion of Lave and Wenger1 that people learn by doing, rather than by receiving factual knowledge. Such a `situated learning' approach has its origins in the ancient art of apprenticeship.

Thus, an important aspect of learning is students attempting to put into practice the theory learnt in lectures. In so doing, they inevitably encounter difficulties, and, indeed, make numerous mistakes of their own. Such learning by experience is a cornerstone of our Computer Science courses. This philosophy has underpinned our offerings in the following subject areas: real time computing and interfacing2-4, computer architecture5, parallel computing6 and indeed neural networks-9.

Since our primary interest in computer science is software, the emphasis with the laboratory components of these subjects is with software simulators rather than hardware per se; indeed pragmatic (budgetry) constraints often reinforce this position. The only exception in the current context is with the real time computing and interfacing subject, in which students `get their hands dirty' with real life hardware devices. Computer architecture makes use of the DLX, SPIM and Dinero simulators10,11, whereas parallel computing utilises the SPOC12 and PVM13 packages. Neural network software simulator packages are the concern of the present paper.

2 ARTIFICIAL NEURAL NETWORKS

The foundations of artificial neural networks are not new. Alan Turing used the brain as a computing paradigm during the mid 1930s, and in the mid 1940s Norbert Weiner and Jon Von Neumann thought that research into the design of brain-like (brain-inspired) computers might be interesting. McCulloch and Pitt's14 simple threshold neuron model dates from the 1940s, whereas Rosenblatt's15 'perceptron' and Widrow and Hoff's ADALINE16 were developed during the 1960s. Interest in and research funding for neural networks dried up for more than a decade following Minsky and Papert's 1969 assertion17 `our intuitive judgement is that the extension to multi-layer systems is sterile.'

A `second wave' of interest in artificial neural networks was sparked off during the early 1980s by successes reported with both the backpropagation algorithm (used with multi-layer perceptrons)18,19 and Hopfield networks20. The subsequent R&D - indeed commercial -- activity in the field has been nothing short of phenomenal, as witnessed by the huge numbers of books, international conferences and journals which have sprung up in recent times. So just what is all the fuss about?

Artificial Neural Networks - ANNs - are models of computing based on the workings of the biological (human) brain. Now, since the workings of the latter is by no means well understood, it is not surprising that the former tend to be rather simplistic in nature. Even so, the workings of ANNs cannot even be well characterised. As a result, they represent somewhat of a `black box' solution to problems which have proven difficult, if not intractable, with other approaches. As such, ANNs have met with resistance in some quarters in industry, where accompanying explanations of how the solution works are the normal expectation. The wholistic, connectionist, massively parallel, distributed nature of ANNs do not lend themselves to characterisations of the kind: `this particular neuron is responsible for that particular output behaviour'.

ANNs comprise many simple, non-linear 'neurons', often connected in various layers, with adaptive weights connecting these neurons together. A typical ANN will comprise 100s (1,000s) of neurons and 1,000s (10,000s) of weights (compared with the billions of nodes and/or weights typical of biological neural networks or brains). The resulting network constitutes a massively parallel distributed computer. Rather than being programmed in the conventional sense, such networks are 'trained' by presenting input-output pattern pairs (training exemplars), over many epochs, such that the overall error (between actual and desired network outputs) decreases (converges) to an acceptable level.

Now, although network training times are typically quite long, recognition times (of inputs not previously encountered), by contrast, are virtually instantaneous. ANNs are not suitable for solving all types of problems, but they are particularly well suited to pattern classification and/or recognition problems. Often the major problem lies not with training an ANN to solve a particular problem per se, but in preprocessing the available training data into a form more suitable for training. Numerous successes have been reported in the literature during the past decade or so; the following list highlights only some: autonomous guided vehicle, NetTalk, bomb sniffer, pap smear, handwritten character recognition, speech recognition and so forth.