Model evaluation¶

Functions:

tlviz.model_evaluation.core_consistency(cp_tensor, dataset, normalised=False)[source]¶

Computes the core consistency [BK03]

A CP model can be interpreted as a restricted Tucker model, where the core tensor is constrained to be superdiagonal. For a third order tensor, this means that the core tensor, \(\mathcal{G}\), satisfy \(g_{ijk}\neq0\) only if \(i = j = k\). To compute the core consistency of a CP decomposition, we use this property, and calculate the optimal Tucker core tensor given the factor matrices of the CP model.

The key observation is that if the data tensor follows the assumptions of the CP model, then the optimal core tensor should be similar to that of the CP model, i. e. superdiagonal. However, if the data can be better described by allowing for interactions between the components across modes, then the core tensor will have non-zero off-diagonal. The core consistency quantifies this measure and is defined as:

\[\text{CC} = 100 - 100 \frac{\| \mathcal{G} - \mathcal{I} \|_F^2}{N}\]

where \(\mathcal{G}\) is the estimated core tensor, \(\mathcal{I}\) is a superdiagonal tensor only ones on the superdiagonal and \(N\) is a normalising factor, either equal to the number of components or the squared frobenius norm of the estimated core tensor. A core consistency score close to 100 indicates that the CP model is likely valid. If the core consistency is low, however, then the model either has components that describe noise or the data does not follow the model’s assumptions. So the core consistency can help determine if the chosen number of components is suitable.

Parameters:

cp_tensorCPTensor or tuple

TensorLy-style CPTensor object or tuple with weights as first argument and a tuple of components as second argument

datasetnp.ndarray

Data tensor that the cp_tensor is fitted against

normalisedBool (default=False)

If True, then the squared frobenius norm of the estimated core tensor is used to normalise the core consistency. Otherwise, the number of components is used.

If normalised=False, then the core consistency formula coincides with [BK03], and if normalised=True, the core consistency formula coincides with that used in the N-Way toolbox, and is unlikely to be less than 0. For core consistencies close to 100, the formulas approximately coincide.

Returns:

float

The core consistency

Examples

We can use the core consistency diagonstic to determine the correct number of components for a CP model. Here, we only fit one model, but in practice, you should fit multiple models and select the one with the lowest SSE (to account for local minima) before computing the core consistency.

>>> from tlviz.data import simulated_random_cp_tensor
>>> from tensorly.decomposition import parafac
>>> cp_tensor, dataset = simulated_random_cp_tensor((10,11,12), 3, seed=42)
>>> # Fit many CP models with different number of components
>>> for rank in range(1, 5):
...     decomposition = parafac(dataset, rank=rank, random_state=42)
...     cc = core_consistency(decomposition, dataset, normalised=True)
...     print(f"No. components: {rank} - core consistency: {cc:.0f}")
No. components: 1 - core consistency: 100
No. components: 2 - core consistency: 100
No. components: 3 - core consistency: 81
No. components: 4 - core consistency: 0

Note

This implementation uses the fast method of estimating the core tensor [BD96, PF15]

tlviz.model_evaluation.estimate_core_tensor(factors, dataset)[source]¶

Efficient estimation of the Tucker core from a factor matrices and a data tensor.

Parameters:

factorstuple

Tuple of factor matrices used to estimate the core tensor from

datasetnp.ndarray

The data tensor that the core tensor is estimated from

Notes

In the original paper, Papalexakis and Faloutsos [PF15] present an algorithm for 3-way tensors. However, it is straightforward to generalise it to N-way tensors by using the inverse tensor product formula in [BD96].

tlviz.model_evaluation.fit(cp_tensor, dataset, sum_squared_dataset=None)[source]¶

Compute the fit (1-relative sum squared error) for a given cp_tensor.

Parameters:

cp_tensorCPTensor or tuple

TensorLy-style CPTensor object or tuple with weights as first argument and a tuple of components as second argument

datasetndarray

Tensor approximated by cp_tensor

sum_squared_dataset: float (optional)

If sum(dataset**2) is already computed, you can optionally provide it using this argument to avoid unnecessary recalculation.

Returns:

float

The relative sum of squared error, sum((X_hat - dataset)**2)/sum(dataset**2), where X_hat is the dense tensor represented by cp_tensor

Examples

Below, we create a random CP tensor and a random tensor and compute the sum of squared error for these two tensors.

>>> import tensorly as tl
>>> from tensorly.random import random_cp
>>> from tlviz.model_evaluation import fit
>>> rng = tl.check_random_state(0)
>>> cp = random_cp((4, 5, 6), 3, random_state=rng)
>>> X = rng.random_sample((4, 5, 6))
>>> fit(cp, X)
0.5182592745038558

We can see that it is equal to 1 - relative SSE

>>> from tlviz.model_evaluation import relative_sse
>>> 1 - relative_sse(cp, X)
0.5182592745038558

tlviz.model_evaluation.predictive_power(cp_tensor, y, sklearn_estimator, mode=0, metric=None, axis=None)[source]¶

Use scikit-learn estimator to evaluate the predictive power of a factor matrix.

This is useful if you evaluate the components based on their predictive power with respect to some task.

Parameters:

factor_matrixndarray(ndim=2)

Factor matrix from a tensor decomposition model

yndarray(ndim=1)

Prediction target for each row of the factor matrix in the given mode. y should have same length as the first dimension of this factor matrix (i.e. the length of the tensor along the given mode).

sklearn_estimatorscikit learn estimator

Scikit learn estimator. Must have the fit and predict methods, and if metric is None, then it should also have the score method. See https://scikit-learn.org/stable/developers/develop.html.

modeint

Which mode to perform the scoring along

metricCallable

Callable (typically function) with the signature metric(y_true, y_pred), where y_true=labels and y_pred is the predicted values obtained from sklearn_estimator. See https://scikit-learn.org/stable/developers/develop.html#specific-models.

axisint (optional)

Alias for mode, if set, then mode cannot be set.

Returns:

float

Score based on the estimator’s performance.

Examples

predictive_power can be useful to evaluate the predictive power of a CP decomposition. To illustrate this, we start by creating a simulated CP tensor and a variable we want to predict that is linearly related to one of the factor matrices.

>>> from tlviz.data import simulated_random_cp_tensor
>>> import numpy as np
>>> rng = np.random.default_rng(0)
>>> cp_tensor, X = simulated_random_cp_tensor((30, 10, 10), 3, noise_level=0.1, seed=rng)
>>> weights, (A, B, C) = cp_tensor
>>> regression_coefficients = rng.standard_normal((3, 1))
>>> Y = A @ regression_coefficients

Next, we fit a PARAFAC model to this data

>>> from tensorly.decomposition import parafac
>>> est_cp_tensor = parafac(X, 3)

Finally, we see how well the estimated decomposition can describe our target variable, Y. This will use the \(R^2\)-coefficient for scoring, as that is the default scoring method for linear models.

>>> from sklearn.linear_model import LinearRegression
>>> from tlviz.model_evaluation import predictive_power
>>> linear_regression = LinearRegression()
>>> r_squared = predictive_power(cp_tensor, Y, linear_regression)
>>> print(f"The R^2 coefficient is {r_squared:.2f}")
The R^2 coefficient is 1.00

We can also specify our own scoring function

>>> from sklearn.metrics import max_error
>>> highest_error = predictive_power(cp_tensor, Y, linear_regression, metric=max_error)
>>> print(f"The maximum error is {highest_error:.2f}")
The maximum error is 0.00

tlviz.model_evaluation.relative_sse(cp_tensor, dataset, sum_squared_dataset=None)[source]¶

Compute the relative sum of squared error for a given cp_tensor.

Parameters:

cp_tensorCPTensor or tuple

TensorLy-style CPTensor object or tuple with weights as first argument and a tuple of components as second argument

datasetndarray

Tensor approximated by cp_tensor

sum_squared_dataset: float (optional)

If sum(dataset**2) is already computed, you can optionally provide it using this argument to avoid unnecessary recalculation.

Returns:

float

The relative sum of squared error, sum((X_hat - dataset)**2)/sum(dataset**2), where X_hat is the dense tensor represented by cp_tensor

Examples

Below, we create a random CP tensor and a random tensor and compute the sum of squared error for these two tensors.

>>> import tensorly as tl
>>> from tensorly.random import random_cp
>>> from tlviz.model_evaluation import relative_sse
>>> rng = tl.check_random_state(0)
>>> cp = random_cp((4, 5, 6), 3, random_state=rng)
>>> X = rng.random_sample((4, 5, 6))
>>> relative_sse(cp, X)
0.4817407254961442

tlviz.model_evaluation.sse(cp_tensor, dataset)[source]¶

Compute the sum of squared error for a given cp_tensor.

Parameters:

cp_tensorCPTensor or tuple

TensorLy-style CPTensor object or tuple with weights as first argument and a tuple of components as second argument

datasetndarray

Tensor approximated by cp_tensor

Returns:

float

The sum of squared error, sum((X_hat - dataset)**2), where X_hat is the dense tensor represented by cp_tensor

Examples

Below, we create a random CP tensor and a random tensor and compute the sum of squared error for these two tensors.

>>> import tensorly as tl
>>> from tensorly.random import random_cp
>>> from tlviz.model_evaluation import sse
>>> rng = tl.check_random_state(0)
>>> cp = random_cp((4, 5, 6), 3, random_state=rng)
>>> X = rng.random_sample((4, 5, 6))
>>> sse(cp, X)
18.948918157419186