Skip to Content

significance_analysis {bandit}

significance_analysis
Package: 
bandit
Version: 
0.4.2

Description

A convenience function to perform overall proportion comparison using prop.test, before doing pairwise comparisons, to see what outcomes seem to be better than others.

Usage

significance_analysis(x, n)

Arguments

x
as in prop.test, a vector of the number of successes
n
as in prop.test, a vector of the number of trials

Values

a data frame with the following columns:

successes
x
totals
n
estimated_proportion
x/n
lower
0.95 confidence interval on the estimated amount by which this alternative outperforms the next-lower alternative
upper
0.95 confidence interval on the estimated amount by which this alternative outperforms the next-lower alternative
significance
p-value for the test that this alternative outperforms the next-lower alternative
order
order, by highest success proportion
best
1 if it is part of the 'highest performing group' -- those groups which were not significantly different from the best group
p_best
Bayesian posterior probability that this alternative is the best binomial bandit

Note

This is intended for use in A/B split testing -- so sizes of n should be roughly equal. Also, note that alternatives which have the same rank are grouped together for analysis with the 'next-lower' alternative, so you may want to check to see if ranks are equal.

See Also

prop.test

Examples

x = c(10,20,30,50)
n = c(100,102,120,130)
sa = significance_analysis(x,n)
sa[rev(order(sa$estimated_proportion)), ]
 
x = c(37,41,30,43,39,30,31,35,50,30)
n = rep(50, length(x))
sa = significance_analysis(x,n)
sa[rev(order(sa$estimated_proportion)), ]
 
x = c(37,41,30,43,39,30,31,37,50,30)
n = rep(50, length(x))
sa = significance_analysis(x,n)
sa[rev(order(sa$estimated_proportion)), ]

Author(s)

Thomas Lotze

Documentation reproduced from package bandit, version 0.4.2. License: GPL-3