Artificial Neural Networks

Artificial neural networks

An artificial neuron.
The basic behavior of an artificial neural network is determined by the dot product (weighted sum) operator in each neuron.
This also has a geometric interpretation.
The dot product of A and B is the magnitude (vector length) of vector A by the magnitude of vector B by the cosine of the angle between them.

 Dot product video.
To understand information storage in a weighted sum it helps to know the central limit theorem.

Statistical properties of the dot product.

The central limit theorem.
If you store 1 <vector,scalar> association in a weighed sum the weight vector will point in the same direction as the input vector. Store 2 <vector,scalar> associations and both input vectors will point some angle away from the weight vector. As a result the scalar output will be more sensitive to small changes in the input vectors. A reduction in an initial mild error correction capacity.
If you store too many associations you can only get approximations to the scalar values you want. They will basically be contamintated with Gaussian noise.


  1. Further information.

    The Weighted Sum:

    A frozen neural network:

    The Walsh Hadamard transform:

    The Walsh Hadamard transform (short):

    Activation weight switching (Beyond ReLU):

    Zero curvature initialisation of neural networks:

    SwitchNet4 neural network:


Post a Comment

Popular posts from this blog

Switch Net 4 (Switch Net N) Combine multiple low width neural layers with a fast transform.

Random Projections for Neural Networks

Switch Net