Switch Net 4 (Switch Net N) Combine multiple low width neural layers with a fast transform.

 Switch Net 4 

Switch Net 4.

Random sign flipping before a fast Walsh Hadamard transform results in a Random Projection. For almost any input the result is a Gaussian distributed output vector where each output element contains knowledge of all the input elements.

With Switch Net 4 the output elements are 2-way switched using (x<0?) as a switching predicate. Where x is the input to the switch. 

The switches are grouped together in units of 4 which together form a small width 4 neural. When a particualr x>=0, the pattern in the selected pattern of weights is forward projected with intensity x. When x<0 a different pattern of forward selected weights is again projected with intensity x (x being negative this time.)

If nothing was projected when x<0 then the situation would be identical to using a ReLU. You could view the situation in Switch Net 4 as using 2 ReLUs one with input x and one with input -x. 

The reason to 2-way switch (or +- ReLU) is to avoid early information loss about the input before the net has fully processed it and to allow an output to be constructed through multiple layers without information bottlenecks due to ReLU blocking fully when x<0.

Once the width 4 layers have been computed they are combined using another fast Walsh Hadamard transform.

The transform does 2 things.

First it is one-to-all. A change in a single input alters all the outputs. That makes it an excellent combining algorithm. It is a one-to-all connectionist device.

Second it is a set of orthogonal statistical summary measures. Where each switching decision and forward projected pattern alters the statistical measures in a gradual way.

Since the width 4 neural layers already provide 4-way connectivity you can truncate the later fast Walsh Hadamard transforms by 2 'layers.'

Shortened Switched Net 4 (for net width 8). WHT Truncated by 2 layers.

You can add layers by repeating the section between the red marks.

When choosing the weights you have to include a scaling factor for the WHT.
Which is 1/sqrt(n) for the full WHT and 1/sqrt(n>>>2) for the partial WHT.

Some example code is here:

Of course you could choose a different width size for the small width layers, like 2 or 8 or 16 for example.

One metric for such networks is the number of switching decisions per parameter.
Conventional dense ReLU neural networks score poorly on that basis.
Switch Net 4 scores better and is easy to train.
Switch Net scores even better but for technical reasons you may need to use a net with double or 4 times the input data width. Switch Net also seems to generalize better.
It is an engineering decision whether to use Switch Net 4 or Switch Net.
Switch Net:




Comments

  1. AI462April 15, 2023 at 4:13 PM

    A Walsh Hadamard transform library for CPUs: https://github.com/FALCONN-LIB/FFHT

    ReplyDelete
  2. AI462May 2, 2023 at 8:34 PM

    Switch Net 4 with backpropagation. Written in Processing, a Java like programming language:
    https://archive.org/details/switchnet4bp

    ReplyDelete
  3. AI462May 7, 2023 at 1:03 AM

    Switch Net 4 with backpropagation in FreeBasic:
    https://archive.org/details/switch-net-4-bpfb
    FreeBasic is nearly C, with pointers etc. It should be an easy convert if you wanted to do that. There are also optional Linux AMD64 assembly language speed-ups included.

    ReplyDelete
  4. AI462May 8, 2023 at 7:38 PM

    Switch Net 4 in JavaScript (online editor) : https://editor.p5js.org/siobhan.491/sketches/sHU0dNmWk

    ReplyDelete

Post a Comment

Popular posts from this blog

Switch Net

Image
 Switch Net Switch Net. Switch Net is a neural network based on using the fast Walsh Hadamard tranform (WHT) as a fixed set of layer weights with a parametric switching based activation function. The cost of the WHT is nlog2(n) add subtracts as opposed to n squared fused multiply adds for a conventional dense neural layer's weight calculations. A fixed random pattern of sign flips is applied to the input data so that the first WHT fairly distributes information with a Gaussian distribution across its outputs. Then a switching decision is applied to each output of the WHT and the output is multiplied by one weight or another.  The section between the red lines is basically a layer and can be repeated a number of times. A final WHT is done to combine the last lot of switched weight outputs. Some example code is here: https://editor.p5js.org/congchuatocmaydangyeu7/sketches/7ekZTwQMF One slight issue is if the switching weights for a particular output have opposite signs then the resul
Read more

2 Siding ReLU via Forward Projections

Image
2 Siding ReLU via Forward Projections The ReLU acivation function in neural networks has special properties that allow it to be considered in a different way to other activation functions. The Switching Viewpoint You can view ReLU as being non-conducting when x<=0 and fully conducting when x>0.  It is a switch which is automatically turned on when x>0 and automatically turned off when x<=0. Which is an ideal rectifier in electrical engineering terms.  Hence the term  Rectified Linear Unit. A switch isn't there when it is on from the point of view of the flowing electricity (or analogous thing.)  Electricity flows through pushed together switch contacts the same as through the wires to the switch.  All the ReLUs in a neural network that are conducting, wire together various sub-components of the network. The wiring being complete the ReLUs become essentially invisible, until one or more ReLUs change state. Since neural networks are computed in a feedforward fashion the w
Read more

Artificial Neural Networks

Image
Artificial neural networks An artificial neuron. The basic behavior of an artificial neural network is determined by the dot product (weighted sum) operator in each neuron. This also has a geometric interpretation. The dot product of A and B is the magnitude (vector length) of vector A by the magnitude of vector B by the cosine of the angle between them.  Dot product video. To understand information storage in a weighted sum it helps to know the central limit theorem. Statistical properties of the dot product. The central limit theorem. If you store 1 <vector,scalar> association in a weighed sum the weight vector will point in the same direction as the input vector. Store 2 <vector,scalar> associations and both input vectors will point some angle away from the weight vector. As a result the scalar output will be more sensitive to small changes in the input vectors. A reduction in an initial mild error correction capacity. If yo
Read more