ReLU as a Switch

 ReLU as a Switch

The conventional view - ReLU as a function



ReLU as a function.

ReLU as a Switch

A slight confusion in computer science is that switches are purely digital, from switching gates and such. That appears to be the case because the supply voltage is fixed and that is the only thing switched.

However a light switch in you house is binary on off, yet connects or disconnects an AC sine wave voltage.  A switch therefore is a mixed digital analog device.

When on, the input signal goes through 1 to 1. When off, the output is zero.

You can then view ReLU as a switch that is 1 to 1 when on and output zero when off. Of course there is a switching decision to make. 
In your house you make the switching decision for a light. With ReLU the switching decision is based on the predicate (x>=0)? Where x is the input to the switch.
You could supply other predicates for the switching decisions if you wanted but switching at x=0 is very helpful for optimization.
A ReLU network with all the switches open. Pa, Pb and Pc are the switching predicates.

A ReLU network with one switch closed.


In the diagram with one switch closed you can see some weighted sums connect together.  The switching decision Pb is based on a simple weighted sum of the inputs. The outputs are weighted sums (with only 1 connected input each in this case) of a weighted sum. 
Weighted sums of weighted sums can be simplifed via linear algebra to an equivelent simple weighted sum. 
Each switching decision in a layer in a ReLU neural network is based on some equivalent simple weighted sum of the input values. Though of course the previous layers have decided what that simple weighted sum is via their switching decisions.
A ReLU neural network with 2 switches closed.

You can also notice some other things like each switch whose output is x projects a pattern onto the next layer through its forward connected weights of intensity x. Of course when the switch is open x=0 and nothing is forward projected.
You can ask the question whether the switch should actually be 2-way rather than 1-way. And have it project an alternative pattern through an alternative set of weights of intensity x when x<0. 
You can also view max-pooling as a from of switching.

Comments

  1. Jinc(z)October 21, 2023 at 4:58 AM

    What a fun perspective! Some people may think a ReLU as a diode, if the x is the electric current...

    ReplyDelete

Post a Comment

Popular posts from this blog

Switch Net

Image
 Switch Net Switch Net. Switch Net is a neural network based on using the fast Walsh Hadamard tranform (WHT) as a fixed set of layer weights with a parametric switching based activation function. The cost of the WHT is nlog2(n) add subtracts as opposed to n squared fused multiply adds for a conventional dense neural layer's weight calculations. A fixed random pattern of sign flips is applied to the input data so that the first WHT fairly distributes information with a Gaussian distribution across its outputs. Then a switching decision is applied to each output of the WHT and the output is multiplied by one weight or another.  The section between the red lines is basically a layer and can be repeated a number of times. A final WHT is done to combine the last lot of switched weight outputs. Some example code is here: https://editor.p5js.org/congchuatocmaydangyeu7/sketches/7ekZTwQMF One slight issue is if the switching weights for a particular output have opposite signs then the resul
Read more

Switch Net 4 (Switch Net N) Combine multiple low width neural layers with a fast transform.

Image
 Switch Net 4  Switch Net 4. Random sign flipping before a fast Walsh Hadamard transform results in a Random Projection. For almost any input the result is a Gaussian distributed output vector where each output element contains knowledge of all the input elements. With Switch Net 4 the output elements are 2-way switched using (x<0?) as a switching predicate. Where x is the input to the switch.  The switches are grouped together in units of 4 which together form a small width 4 neural. When a particualr x>=0, the pattern in the selected pattern of weights is forward projected with intensity x. When x<0 a different pattern of forward selected weights is again projected with intensity x (x being negative this time.) If nothing was projected when x<0 then the situation would be identical to using a ReLU. You could view the situation in Switch Net 4 as using 2 ReLUs one with input x and one with input -x.  The reason to 2-way switch (or +- ReLU) is to avoid early information los
Read more

Artificial Neural Networks

Image
Artificial neural networks An artificial neuron. The basic behavior of an artificial neural network is determined by the dot product (weighted sum) operator in each neuron. This also has a geometric interpretation. The dot product of A and B is the magnitude (vector length) of vector A by the magnitude of vector B by the cosine of the angle between them.  Dot product video. To understand information storage in a weighted sum it helps to know the central limit theorem. Statistical properties of the dot product. The central limit theorem. If you store 1 <vector,scalar> association in a weighed sum the weight vector will point in the same direction as the input vector. Store 2 <vector,scalar> associations and both input vectors will point some angle away from the weight vector. As a result the scalar output will be more sensitive to small changes in the input vectors. A reduction in an initial mild error correction capacity. If yo
Read more