Factorized Convolutional Layers

It is possible to apply low-rank tensor factorization to convolution kernels to compress the network and reduce the number of parameters.

In TensorLy-Torch, you can easily try factorized convolutions: first, let’s import the library:

```import tltorch
import torch
```

Let’s now create some random data to try our modules: we can choose the size of the convolutions.

```device='cpu'
input_channels = 16
output_channels = 32
kernel_size = 3
batch_size = 2
size = 24
order = 2

input_shape = (batch_size, input_channels) + (size, )*order
kernel_shape = (output_channels, input_channels) + (kernel_size, )*order
```

We can create some random input data:

```data = torch.randn(input_shape, dtype=torch.float32, device=device)
```

Creating Factorized Convolutions

From Random

In PyTorch, you would create a convolution as follows:

```conv = torch.nn.Conv2d(input_channels, output_channels, kernel_size)
```

In TensorLy Torch, it is exactly the same except that factorized convolutions are by default of any order: either you specify the kernel size or your specify the order

```conv = tltorch.FactorizedConv(input_channels, output_channels, kernel_size, order=2, rank='same', factorization='cp')
```
```conv = torch.nn.Conv2d(input_channels, output_channels, kernel_size=3)
```

In TensorLy-Torch, factorized convolutions can be of any order, so you have to specify the order at creation (in Pytorch, you specify it through the class name, e.g. Conv2d or Conv3d):

```fact_conv = tltorch.FactorizedConv(input_channels, output_channels, kernel_size=3, order=2, rank='same')
```

Or, you can specify the order directly by passing a tuple as kernel_size (in which case, `order = len(kernel_size)` is used).

```fact_conv = tltorch.FactorizedConv(input_channels, output_channels, kernel_size=(3, 3), rank='same')
```

From an existing Convolution

You can create a Factorized convolution from an existing (PyTorch) convolution:

```fact_conv = tltorch.FactorizedConv.from_conv(conv, rank=0.5, decompose_weights=True, factorization='tucker')
```

Efficient Convolutional Blocks

If you compress a convolutional kernel, you can get efficient convolutional blocks by applying tensor factorization. For instance, if you apply CP decomposition, you can get a MobileNet-v2 block:

```fact_conv = tltorch.FactorizedConv.from_conv(conv, rank=0.5, factorization='cp', implementation='mobilenet')
```

Similarly, if you apply Tucker decomposition, you can get a ResNet BottleNeck block:

```fact_conv = tltorch.FactorizedConv.from_conv(conv, rank=0.5, factorization='tucker', implementation='factorized')
```