tltorch.TTMLinear

class tltorch.TTMLinear(in_features, out_features, tensorized_shape, rank='same', bias=True)[source]

Tensorized Fully-Connected Layers in the TT-Matrix format [1]

The weight matrice is tensorized to a tensor of size tensorized_shape. That tensor is expressed as a low-rank TT-Matrix by jointly compressing inputs and outputs. During inference, the full tensor is reconstructed, and unfolded back into a matrix, used for the forward pass in a regular linear layer.

Parameters:	in_featuresint out_featuresint tensorized_shapeint tuple should be left_shape + right_shape correponsding to a weight matrix of size left x right rankint tuple or str, default is ‘same’ biasbool, default is True

See also

TuckerLinear

CPLinear

TTLinear

Notes

This layer permutes the dimensions of the weight matrix after it has been reshaped into a higher-order tensor of shape tensorized_shape.

For a linear layer with out_feature = O_1 * O_2 * O_3 and in_features = I_1 * I_2 * I_3 and tensorized_shape = (O_1, O_2, O_3, I_1, I_2, I_3) and rank R = (R_1, R_2, R_3, R_4) with R_1 = R_4 = 1. the inputs and outputs will be jointly compressed by each tt-matrix core. In other words, the k-th core will be of shape (R_k, O_k, I_k, R_{k+1}).

By contrast, tltorch.TTLinear() simply reshapes the matrix to tensorized_shape and compresses with a tensor-train decomposition.

References

[1]	Tensorizing Neural Networks, Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, Dmitry Vetrov

Attributes:	full_weight Returns the reconstruced matrix weight of the linear layer

Methods

forward(input)	Inference using the tensorized and factorized weight matrix
get_decomposition()	Returns the decomposition parametrizing the layer
init_from_decomposition(tt_matrix[, bias])	Initializes the factorization from the given decomposition
init_from_random([decompose_full_weight])	Initializes the factorization randomly
init_from_tensor(tensor[, bias])	Initializes the layer by decomposing a full tensor

forward(input)[source]

Inference using the tensorized and factorized weight matrix

init_from_random(decompose_full_weight=True)[source]

Initializes the factorization randomly

Parameters:	decompose_full_weightbool, default is True if True, a full weight is created and decomposed to initialize the factors otherwise, the factors of the decomposition are directly initialized

init_from_decomposition(tt_matrix, bias=None)[source]

Initializes the factorization from the given decomposition

Parameters:	tucker_tensor(factors) values to initialize the decomposition parametrizing the layer to biastorch.Tensor or None, default is None

init_from_tensor(tensor, bias=None)[source]

Initializes the layer by decomposing a full tensor

Parameters:	tensortorch.Tensor must be either a matrix or a tensor must verify np.prod(tensor.shape) == np.prod(self.tensorized_shape) biastorch.Tensor or None, default is None

get_decomposition()[source]

Returns the decomposition parametrizing the layer

property full_weight

Returns the reconstruced matrix weight of the linear layer