Tensor Regression Layers
In deep neural networks, while convolutional layers map between high-order activation tensors, the output is still obtained through linear regression: first the activation is flattened before being passed through linear layers.
This approach has several drawbacks:
Linear regression discards topological (e.g. spatial) information.
Very large number of parameters (product of the dimensions of the input tensor times the size of the output)
Lack of robustness
A Tensor Regression Layer (TRL) generalizes the concept of linear regression to higher-order but alleviates the above issues. It allows to preserve and leverage multi-linear structure while being parsimonious in terms of number of parameters. The low-rank constraints also acts as an implicit reguralization on the model, typically leading to better sample efficiency and robustness.
TRL in TensorLy-Torch
Now, let’s see how to do this in code with TensorLy-Torch
Let’s first see how to create and train a TRL from scratch
import tltorch import torch from torch import nn import numpy as np
input_shape = (4, 5) output_shape = (6, 2) batch_size = 2 device = 'cpu' x = torch.randn((batch_size,) + input_shape, dtype=torch.float32, device=device)
trl = tltorch.TRL(input_shape, output_shape, rank='same')
result = trl(x)