Testing Utilities¶
Goodness of Fit Testing¶
This module implements goodness of fit tests for checking agreement between
distributions’ .sample()
and .log_prob()
methods. The main functions
return a goodness of fit pvalue gof
which for good data should be
Uniform(0,1)
distributed and for bad data should be close to zero. To use
this returned number in tests, set a global variable TEST_FAILURE_RATE
to
something smaller than 1 / number of tests in your suite, then in each test
assert gof > TEST_FAILURE_RATE
. For example:
TEST_FAILURE_RATE = 1 / 20 # For 1 in 20 chance of spurious failure.
def test_my_distribution():
d = MyDistribution()
samples = d.sample([10000])
probs = d.log_prob(samples).exp()
gof = auto_goodness_of_fit(samples, probs)
assert gof > TEST_FAILURE_RATE
This module is a port of the goftests library.

multinomial_goodness_of_fit
(probs, counts, *, total_count=None, plot=False)[source]¶ Pearson’s chi^2 test, on possibly truncated data. https://en.wikipedia.org/wiki/Pearson%27s_chisquared_test
Parameters:  probs (torch.Tensor) – Vector of probabilities.
 counts (torch.Tensor) – Vector of counts.
 total_count (int) – Optional total count in case data is truncated, otherwise None.
 plot (bool) – Whether to print a histogram. Defaults to False.
Returns: pvalue of truncated multinomial sample.
Return type:

unif01_goodness_of_fit
(samples, *, plot=False)[source]¶ Bin uniformly distributed samples and apply Pearson’s chi^2 test.
Parameters:  samples (torch.Tensor) – A vector of realvalued samples from a candidate distribution that should be Uniform(0, 1)distributed.
 plot (bool) – Whether to print a histogram. Defaults to False.
Returns: Goodness of fit, as a pvalue.
Return type:

exp_goodness_of_fit
(samples, plot=False)[source]¶ Transform exponentially distribued samples to Uniform(0,1) distribution and assess goodness of fit via binned Pearson’s chi^2 test.
Parameters:  samples (torch.Tensor) – A vector of realvalued samples from a candidate distribution that should be Exponential(1)distributed.
 plot (bool) – Whether to print a histogram. Defaults to False.
Returns: Goodness of fit, as a pvalue.
Return type:

density_goodness_of_fit
(samples, probs, plot=False)[source]¶ Transform arbitrary continuous samples to Uniform(0,1) distribution and assess goodness of fit via binned Pearson’s chi^2 test.
Parameters:  samples (torch.Tensor) – A vector list of realvalued samples from a distribution.
 probs (torch.Tensor) – A vector of probability densities evaluated at those samples.
 plot (bool) – Whether to print a histogram. Defaults to False.
Returns: Goodness of fit, as a pvalue.
Return type:

vector_density_goodness_of_fit
(samples, probs, *, dim=None, plot=False)[source]¶ Transform arbitrary multivariate continuous samples to Univariate(0,1) distribution via nearest neighbor distribution [1,2,3] and assess goodness of fit via binned Pearson’s chi^2 test.
 [1] Peter J. Bickel and Leo Breiman (1983)
 “Sums of Functions of Nearest Neighbor Distances, Moment Bounds, Limit Theorems and a Goodness of Fit Test” https://projecteuclid.org/download/pdf_1/euclid.aop/1176993668
 [2] Mike Williams (2010)
 “How good are your fits? Unbinned multivariate goodnessoffit tests in high energy physics.” https://arxiv.org/abs/1006.3019
 [3] Nearest Neighbour Distribution
 https://en.wikipedia.org/wiki/Nearest_neighbour_distribution
Parameters:  samples (torch.Tensor) – A tensor of realvectorvalued samples from a distribution.
 probs (torch.Tensor) – A vector of probability densities evaluated at those samples.
 dim (int) – Optional dimension of the submanifold on which data lie.
Defaults to
samples.shape[1]
.  plot (bool) – Whether to print a histogram. Defaults to False.
Returns: Goodness of fit, as a pvalue.
Return type:

auto_goodness_of_fit
(samples, probs, *, dim=None, plot=False)[source]¶ Dispatch on sample dimension and delegate to either
density_goodness_of_fit()
orvector_density_goodness_of_fit()
.Parameters:  samples (torch.Tensor) – A tensor of samples stacked on their leftmost dimension.
 probs (torch.Tensor) – A vector of probabilities evaluated at those samples.
 dim (int) – Optional manifold dimension, defaults to
samples.shape[1:].numel()
.  plot (bool) – Whether to print a histogram. Defaults to False.