ReLU as a Switch
ReLU as a Switch The conventional view - ReLU as a function ReLU as a function. ReLU as a Switch A slight confusion in computer science is that switches are purely digital, from switching gates and such. That appears to be the case because the supply voltage is fixed and that is the only thing switched. However a light switch in you house is binary on off, yet connects or disconnects an AC sine wave voltage. A switch therefore is a mixed digital analog device. When on, the input signal goes through 1 to 1. When off, the output is zero. You can then view ReLU as a switch that is 1 to 1 when on and output zero when off. Of course there is a switching decision to make. In your house you make the switching decision for a light. With ReLU the switching decision is based on the predicate (x>=0)? Where x is the input to the switch. You could supply other predicates for the switching decisions if you wanted but switching at x=0 is very helpful for optimization. A ReLU netwo...