Switch Net 4 - reducing the cost of a neural network layer.

 Switch Net 4

The layers in a fully connected artificial neural network don't scale nicely with width.

Width of a neural network.

For width n the number of multiply add operations required per layer is n squared. For a layer of width 256 the number multiply adds would be 65536 (256*256.) Even with modern hardward layers cannot be made much wider than that.

Or can they? 

A layer of width 2 only requires 4 operations, width 4 only 16 operations, width 8 only 64 operations. If you could combine k width n (n being small) layers into a new much wider layer you'd end up with a computational advantage.

For example 64 width 4 layers combined into a width 256 layer would cost 64*16=1024 multiply add operations plus the combining cost. 

A combining algorithm.

The fast Walsh Hadmard transform can be used as a combiner because a change in a single input causes all the outputs to vary.


The combining cost is n*log2(n) add subtract operations. For a layer of width 256 the combining cost is 2048 add subtract operations. In practice you don't need to do a full WHT transform and the combining cost is n*(log2(n)-2) operations starting with width 4 layers. For a final layer of width 256 (base width 4) that gives 2560 add subtract operations and 1024 multiply operations compaired to 65536 add multiply operations for a normal width 256 layer.

The larger the width the better the scaling comparison goes.

WTH example: http://md2020.eu5.org/wht1.html

The actual Switch Net 4 algorithm uses "2 Siding ReLU via its forward connections" as the activation function (or actually activation scheme.)

See: 2 Siding ReLU

An early example of Switch Net 4 is here:

https://discourse.processing.org/t/switch-net-4-neural-network/33220

A somewhat later version:

https://archive.org/details/swnet

A version using subrandom projections for pre and post-processing:

https://archive.org/details/switchnet4_new

Further Information:

https://archive.org/details/afrozenneuralnetwork

https://archive.org/details/whtebook-archive

https://archive.org/details/zero-curvatue

https://www.kdnuggets.com/2021/07/wht-simpler-fast-fourier-transform-fft.html






Comments

  1. AI462April 13, 2023 at 3:15 PM

    Online JavaScript version of Switch Net 4: https://editor.p5js.org/congchuatocmaydangyeu7/sketches/IIZ9L5fzS
    And for comparison Switch Net Wide:
    https://editor.p5js.org/congchuatocmaydangyeu7/sketches/7ekZTwQMF

    ReplyDelete

Post a Comment

Popular posts from this blog

Switch Net

Image
 Switch Net Switch Net. Switch Net is a neural network based on using the fast Walsh Hadamard tranform (WHT) as a fixed set of layer weights with a parametric switching based activation function. The cost of the WHT is nlog2(n) add subtracts as opposed to n squared fused multiply adds for a conventional dense neural layer's weight calculations. A fixed random pattern of sign flips is applied to the input data so that the first WHT fairly distributes information with a Gaussian distribution across its outputs. Then a switching decision is applied to each output of the WHT and the output is multiplied by one weight or another.  The section between the red lines is basically a layer and can be repeated a number of times. A final WHT is done to combine the last lot of switched weight outputs. Some example code is here: https://editor.p5js.org/congchuatocmaydangyeu7/sketches/7ekZTwQMF One slight issue is if the switching weights for a particular output have opposite signs then the resul
Read more

2 Siding ReLU via Forward Projections

Image
2 Siding ReLU via Forward Projections The ReLU acivation function in neural networks has special properties that allow it to be considered in a different way to other activation functions. The Switching Viewpoint You can view ReLU as being non-conducting when x<=0 and fully conducting when x>0.  It is a switch which is automatically turned on when x>0 and automatically turned off when x<=0. Which is an ideal rectifier in electrical engineering terms.  Hence the term  Rectified Linear Unit. A switch isn't there when it is on from the point of view of the flowing electricity (or analogous thing.)  Electricity flows through pushed together switch contacts the same as through the wires to the switch.  All the ReLUs in a neural network that are conducting, wire together various sub-components of the network. The wiring being complete the ReLUs become essentially invisible, until one or more ReLUs change state. Since neural networks are computed in a feedforward fashion the w
Read more

Artificial Neural Networks

Image
Artificial neural networks An artificial neuron. The basic behavior of an artificial neural network is determined by the dot product (weighted sum) operator in each neuron. This also has a geometric interpretation. The dot product of A and B is the magnitude (vector length) of vector A by the magnitude of vector B by the cosine of the angle between them.  Dot product video. To understand information storage in a weighted sum it helps to know the central limit theorem. Statistical properties of the dot product. The central limit theorem. If you store 1 <vector,scalar> association in a weighed sum the weight vector will point in the same direction as the input vector. Store 2 <vector,scalar> associations and both input vectors will point some angle away from the weight vector. As a result the scalar output will be more sensitive to small changes in the input vectors. A reduction in an initial mild error correction capacity. If yo
Read more