Random Projections for Neural Networks

Random Projections for Neural Networks 

Variance and the CLT

For Linear Combinations of Random Variables a negative sign before a particular variable has no effect on variance.

And of course the Central Limit Theorem is in operation.

The Walsh Hadamard transform (WHT) is a set of orthogonal weighted sums (where the weights are +1 or -1 or some constant multiple of those.) 
And so the variance equation for linear combinations of random variables applies (minus signs not invalidating that), as does the Central Limit Theorem.
8-Point Walsh Hadamard transform.
Therefore applying the WHT to a sequence of random numbers from the Uniform random number distribution results in a sequence of numbers from the Gaussian (Normal) distribution.

Example code is here:

If you want to use that behavior to generate Gaussians you should remember that the WHT leaves vector magnitued (length) unchanged (except by a constant c.) 
The result is all the Gaussians are very slightly entangled through that property.

Destructuring the WHT

The reason the WHT can work as an image compression algorithm is that the underlying sequency (cf. frequency) patterns match strongly with patterns always found in natural images.
Sequency Patterns of the 8-point WHT.

However that is a property easily broken by applying a random pattern of sign flips to the input data to a WHT or applying a random permuation.

What is left then is the residual variance and central limit theorem properties.
The combination of random sign flipping and the WHT then is a Random Projection (PR) of the input data. A transformation from the coherent domain into an incoherent domain of Gaussian noise.

If you know the original pattern of sign flips you can invert the Random Projection since both the WHT and sign flipping are self-inverse.

1. The original data. 2. A Random Projection of the data. 3. The image restored from 20% of the Random Projection (dimensional reduction.) 4. The image restored from 20% of the random projection with iterative smoothing.

Code for Random Projections:

Neural Network Applications

You can use random projections for dimension reduction and dimensional increase.

Dimensional reduction and restoration via iterative smoothing is one option for processing large images with small neural networks. 
Iterative smoothing shows there is a lot more information in random projection sub-sample than you might imagine and actually there is even more since by preforce smoothing loses high frequency information.

You can use random projections as form of preprocessing for neural networks. Especially for auto-associative neural networks. 
Or you can use a random projection for fair splitting of input data between multiple small neural networks.
 
Binarizing the output of a random projection give you a Locality Sensitive Hash (LSH.) Where the output of the LSH is random but small changes in the input cause only a few bit changes in the output. 







Comments

  1. AI462April 4, 2023 at 6:56 PM

    If we remember Rader's work on the FFT algorithm in 1968:
    https://en.wikipedia.org/wiki/Rader%27s_FFT_algorithm
    He also did work on the Walsh Hadamard transform in 1969:
    https://archive.org/details/DTIC_AD0695042
    Where he used the WHT to generate random numbers with the Gaussian distribution.

    ReplyDelete

Post a Comment

Popular posts from this blog

Switch Net

Image
 Switch Net Switch Net. Switch Net is a neural network based on using the fast Walsh Hadamard tranform (WHT) as a fixed set of layer weights with a parametric switching based activation function. The cost of the WHT is nlog2(n) add subtracts as opposed to n squared fused multiply adds for a conventional dense neural layer's weight calculations. A fixed random pattern of sign flips is applied to the input data so that the first WHT fairly distributes information with a Gaussian distribution across its outputs. Then a switching decision is applied to each output of the WHT and the output is multiplied by one weight or another.  The section between the red lines is basically a layer and can be repeated a number of times. A final WHT is done to combine the last lot of switched weight outputs. Some example code is here: https://editor.p5js.org/congchuatocmaydangyeu7/sketches/7ekZTwQMF One slight issue is if the switching weights for a particular output have opposite signs then the resul
Read more

Switch Net 4 (Switch Net N) Combine multiple low width neural layers with a fast transform.

Image
 Switch Net 4  Switch Net 4. Random sign flipping before a fast Walsh Hadamard transform results in a Random Projection. For almost any input the result is a Gaussian distributed output vector where each output element contains knowledge of all the input elements. With Switch Net 4 the output elements are 2-way switched using (x<0?) as a switching predicate. Where x is the input to the switch.  The switches are grouped together in units of 4 which together form a small width 4 neural. When a particualr x>=0, the pattern in the selected pattern of weights is forward projected with intensity x. When x<0 a different pattern of forward selected weights is again projected with intensity x (x being negative this time.) If nothing was projected when x<0 then the situation would be identical to using a ReLU. You could view the situation in Switch Net 4 as using 2 ReLUs one with input x and one with input -x.  The reason to 2-way switch (or +- ReLU) is to avoid early information los
Read more

Artificial Neural Networks

Image
Artificial neural networks An artificial neuron. The basic behavior of an artificial neural network is determined by the dot product (weighted sum) operator in each neuron. This also has a geometric interpretation. The dot product of A and B is the magnitude (vector length) of vector A by the magnitude of vector B by the cosine of the angle between them.  Dot product video. To understand information storage in a weighted sum it helps to know the central limit theorem. Statistical properties of the dot product. The central limit theorem. If you store 1 <vector,scalar> association in a weighed sum the weight vector will point in the same direction as the input vector. Store 2 <vector,scalar> associations and both input vectors will point some angle away from the weight vector. As a result the scalar output will be more sensitive to small changes in the input vectors. A reduction in an initial mild error correction capacity. If yo
Read more