U Kv fqSã@sÂdZddlmZddlZddlmZddlmZddl m Z dd„Z dUd d„Z dd„ZeZdd„Zdd„Zdd„Zdd„Zdd„ZiZeeeeeeedœed<dd„Zdd „d!d „d"d „d#œed$<d%eied&<Gd'd(„d(ƒZdVd*d+„ZdWd,d-„ZdXd/d0„ZdYd1d2„ZGd3d4„d4ƒZed5kr¾dd6lmZej j!d7d.d8Z!e"d9ƒe"e e!d:ƒƒee!d:ƒZ#e"e# $¡ƒd%d;dd?d@gZ%e%D]Z&e"e&e# $e&d¡ƒqze"dAƒddBl'm(Z(e(e)ƒZ*d.Z+e,d)ƒD]FZ-ej. /e+¡Z!ee!d:ƒZ#e%D]$Z&e*e& 0e# $e&d¡ddC¡qÚq¼e 1dDdE„e%Dƒ¡Z2e"dFdG 3e%¡ƒe"dHe2dIk 4dC¡ƒe"dJe2dKk 4dC¡ƒe"dLe2dMk 4dC¡ƒedNd „d:d.dOd.Z+d)Z5eeƒdPe+e5ddQZ6e 7e5e 1dRdSdTg¡¡ 8e9¡Z:e"e6e:ƒdS)ZatMore Goodness of fit tests contains GOF : 1 sample gof tests based on Stephens 1970, plus AD A^2 bootstrap : vectorized bootstrap p-values for gof test with fitted parameters Created : 2011-05-21 Author : Josef Perktold parts based on ks_2samp and kstest from scipy.stats (license: Scipy BSD, but were completely rewritten by Josef Perktold) References ---------- é)ÚlmapN)Ú distributions)Úcache_readonly)Ú kolmogorovc Csîttj||fƒ\}}|jd}|jd}t|ƒ}t|ƒ}t |¡}t |¡}t ||g¡}tj||ddd|}tj||ddd|}t t  ||¡¡}t  ||t ||ƒ¡}zt |dd||ƒ} Wnd} YnX|| fS)aA Computes the Kolmogorov-Smirnof statistic on 2 samples. This is a two-sided test for the null hypothesis that 2 independent samples are drawn from the same continuous distribution. Parameters ---------- a, b : sequence of 1-D ndarrays two arrays of sample observations assumed to be drawn from a continuous distribution, sample sizes can be different Returns ------- D : float KS statistic p-value : float two-tailed p-value Notes ----- This tests whether 2 samples are drawn from the same distribution. Note that, like in the case of the one-sample K-S test, the distribution is assumed to be continuous. This is the two-sided test, one-sided tests are not implemented. The test uses the two-sided asymptotic Kolmogorov-Smirnov distribution. If the K-S statistic is small or the p-value is high, then we cannot reject the hypothesis that the distributions of the two samples are the same. Examples -------- >>> from scipy import stats >>> import numpy as np >>> from scipy.stats import ks_2samp >>> #fix random seed to get the same result >>> np.random.seed(12345678) >>> n1 = 200 # size of first sample >>> n2 = 300 # size of second sample different distribution we can reject the null hypothesis since the pvalue is below 1% >>> rvs1 = stats.norm.rvs(size=n1,loc=0.,scale=1) >>> rvs2 = stats.norm.rvs(size=n2,loc=0.5,scale=1.5) >>> ks_2samp(rvs1,rvs2) (0.20833333333333337, 4.6674975515806989e-005) slightly different distribution we cannot reject the null hypothesis at a 10% or lower alpha since the pvalue at 0.144 is higher than 10% >>> rvs3 = stats.norm.rvs(size=n2,loc=0.01,scale=1.0) >>> ks_2samp(rvs1,rvs3) (0.10333333333333333, 0.14498781825751686) identical distribution we cannot reject the null hypothesis since the pvalue is high, 41% >>> rvs4 = stats.norm.rvs(size=n2,loc=0.0,scale=1.0) >>> ks_2samp(rvs1,rvs4) (0.07999999999999996, 0.41126949729859719) rÚright)Zsideçð?縅ëQ¸¾?ç)\Âõ(¼?) rÚnpZasarrayÚshapeÚlenÚsortZ concatenateZ searchsortedÚmaxÚabsoluteÚsqrtÚfloatÚksprob) Zdata1Zdata2Zn1Zn2Zdata_allZcdf1Zcdf2ÚdÚenZprob©rúM/tmp/pip-unpacked-wheel-2v6byqio/statsmodels/sandbox/distributions/gof_new.pyÚks_2samps"H     rréÚ two_sidedÚapproxc Ks¼t|tƒr8|r||kr0tt|ƒj}tt|ƒj}ntdƒ‚t|tƒrNtt|ƒj}t|ƒrpd|i}t  |||Ž¡}nt  |¡}t |ƒ}||f|žŽ}|dkrÎt  d|d¡||  ¡} |dkrÎ| tj  | |¡fS|dkr |t  d|¡|  ¡} |d kr | tj  | |¡fS|d kr¸t  | | g¡} |d krH| tj | t |¡¡fS|d kr¸tj | t |¡¡} |d ksˆ| d|ddkr¢| tj | t |¡¡fS| tj  | |¡dfSdS)aÉ Perform the Kolmogorov-Smirnov test for goodness of fit This performs a test of the distribution G(x) of an observed random variable against a given distribution F(x). Under the null hypothesis the two distributions are identical, G(x)=F(x). The alternative hypothesis can be either 'two_sided' (default), 'less' or 'greater'. The KS test is only valid for continuous distributions. Parameters ---------- rvs : str or array or callable string: name of a distribution in scipy.stats array: 1-D observations of random variables callable: function to generate random variables, requires keyword argument `size` cdf : str or callable string: name of a distribution in scipy.stats, if rvs is a string then cdf can evaluate to `False` or be the same as rvs callable: function to evaluate cdf args : tuple, sequence distribution parameters, used if rvs or cdf are strings N : int sample size if rvs is string or callable alternative : 'two_sided' (default), 'less' or 'greater' defines the alternative hypothesis (see explanation) mode : 'approx' (default) or 'asymp' defines the distribution used for calculating p-value 'approx' : use approximation to exact distribution of test statistic 'asymp' : use asymptotic distribution of test statistic Returns ------- D : float KS test statistic, either D, D+ or D- p-value : float one-tailed or two-tailed p-value Notes ----- In the one-sided test, the alternative is that the empirical cumulative distribution function of the random variable is "less" or "greater" than the cumulative distribution function F(x) of the hypothesis, G(x)<=F(x), resp. G(x)>=F(x). Examples -------- >>> from scipy import stats >>> import numpy as np >>> from scipy.stats import kstest >>> x = np.linspace(-15,15,9) >>> kstest(x,'norm') (0.44435602715924361, 0.038850142705171065) >>> np.random.seed(987654321) # set random seed to get the same result >>> kstest('norm','',N=100) (0.058352892479417884, 0.88531190944151261) is equivalent to this >>> np.random.seed(987654321) >>> kstest(stats.norm.rvs(size=100),'norm') (0.058352892479417884, 0.88531190944151261) Test against one-sided alternative hypothesis: >>> np.random.seed(987654321) Shift distribution to larger values, so that cdf_dgp(x)< norm.cdf(x): >>> x = stats.norm.rvs(loc=0.2, size=100) >>> kstest(x,'norm', alternative = 'less') (0.12464329735846891, 0.040989164077641749) Reject equal distribution against alternative hypothesis: less >>> kstest(x,'norm', alternative = 'greater') (0.0072115233216311081, 0.98531158590396395) Do not reject equal distribution against alternative hypothesis: greater >>> kstest(x,'norm', mode='asymp') (0.12464329735846891, 0.08944488871182088) Testing t distributed random variables against normal distribution: With 100 degrees of freedom the t distribution looks close to the normal distribution, and the kstest does not reject the hypothesis that the sample came from the normal distribution >>> np.random.seed(987654321) >>> stats.kstest(stats.t.rvs(100,size=100),'norm') (0.072018929165471257, 0.67630062862479168) With 3 degrees of freedom the t distribution looks sufficiently different from the normal distribution, that we can reject the hypothesis that the sample came from the normal distribution at a alpha=10% level >>> np.random.seed(987654321) >>> stats.kstest(stats.t.rvs(3,size=100),'norm') (0.131016895759829, 0.058826222555312224) ú5if rvs is string, cdf has to be the same distributionÚsize)rÚgreaterrér)rÚlessçrrÚasympréj çš™™™™™é?ç333333Ó?ç@@éN)Ú isinstanceÚstrÚgetattrrÚcdfÚrvsÚAttributeErrorÚcallabler r r ÚarangerÚksoneÚsfÚ kstwobignr) r+r*ÚargsÚNÚ alternativeÚmodeÚkwdsÚvalsÚcdfvalsÚDplusÚDminÚDÚpval_tworrrÚkstest}s<s            r=cCsZt |¡ddt |¡}||}t d|d¡}t |t dddg¡k¡}|||fS)Nrr éþÿÿÿr&g= ×£p=ê?r©r rÚexpÚsumÚarray©ÚstatÚnobsÚ mod_factorÚ stat_modifiedÚpvalÚdigitsrrrÚdplus_st70_upps rJcCs^t |¡ddt |¡}||}dt d|d¡}t |t dddg¡k¡}|||fS)Nrr r&r>g…ëQ¸í?gHáz®Gñ?r?rCrrrÚ d_st70_upp&s rKcCsjt |¡ddt |¡}||}|d}d|dt d|¡}t |t dddg¡k¡}|||fS)Ng×£p= ×Ã?g¸…ëQ¸Î?r&ér>gö(\Âõð?g)\Âõ(ô?r?)rDrErFrGZzsqurHrIrrrÚ v_st70_upp.s rMcCsNd|}|d|d|dd|}dt dd|¡}tj}|||fS) Nrgš™™™™™Ù?g333333ã?r&rçš™™™™™©?gR¸…ëQ@é)r r@Únan©rDrEZnobsinvrGrHrIrrrÚ wsqu_st70_upp7s  rRcCspd|}|d|d|d}|dd|9}dt d|tjd¡}t |t dddg¡k¡}|||fS) Nrçš™™™™™¹?r&rr#r>gÂõ(\Ò?gÃõ(\ÂÕ?©r r@ÚpirArBrQrrrÚ usqu_st70_upp?s rVcCstd|}|d|d|d}|dd|9}dt d|d tjd¡}t |t d d d g¡k¡}|||fS) Nrgffffffæ?çÍÌÌÌÌÌì?r&rg®Gáz®ó?g‘í|?5^ô?r>ç@r g!°rh‘íÜ?rTrQrrrÚ a_st70_uppHs  rY)Úd_plusÚd_minusrÚvÚwsquÚusquÚaÚ stephens70uppcCsntj |t |¡¡}|dks2|d|ddkrP|tj |t |¡¡tjfS|tj ||¡dtjfSdS)Nr"r#r$r%r&)rr1r0r rrPr/)r;r3r<rrrÚpval_kstest_approx^sracCs|tj ||¡tjfS©N©rr/r0r rP)r9r3rrrÚfórdcCs|tj ||¡tjfSrbrc)r:r3rrrrdgrecCs|tj |t |¡¡tjfSrb)rr1r0r rrP)r;r3rrrrdhre)rZr[rÚscipyrZ scipy_approxc@s„eZdZdZddd„Zedd„ƒZedd „ƒZed d „ƒZed d „ƒZ edd„ƒZ edd„ƒZ edd„ƒZ edd„ƒZ ddd„ZdS)ÚGOFaPOne Sample Goodness of Fit tests includes Kolmogorov-Smirnov D, D+, D-, Kuiper V, Cramer-von Mises W^2, U^2 and Anderson-Darling A, A^2. The p-values for all tests except for A^2 are based on the approximatiom given in Stephens 1970. A^2 has currently no p-values. For the Kolmogorov-Smirnov test the tests as given in scipy.stats are also available as options. design: I might want to retest with different distributions, to calculate data summary statistics only once, or add separate class that holds summary statistics and data (sounds good). rrcCs¦t|tƒr8|r||kr0tt|ƒj}tt|ƒj}ntdƒ‚t|tƒrNtt|ƒj}t|ƒrpd|i}t  |||Ž¡}nt  |¡}t |ƒ}||f|žŽ}||_ ||_ ||_ dS)Nrr)r'r(r)rr*r+r,r-r r r rEZ vals_sortedr8)Úselfr+r*r2r3r6r7r8rrrÚ__init__†s       z GOF.__init__cCs(|j}|j}t d|d¡|| ¡S)Nrr©rEr8r r.r©rhrEr8rrrrZ sz GOF.d_pluscCs$|j}|j}|t d|¡| ¡S)Nr rjrkrrrr[¦sz GOF.d_minuscCst |j|jg¡Srb)r rrZr[©rhrrrr¬szGOF.dcCs |j|jS)ZKuiper)rZr[rlrrrr\°szGOF.vcCsH|j}|j}|dt d|d¡d|dd ¡d|d}|S)zCramer von MisesrXrrr&g(@)rEr8r r.rA)rhrEr8r]rrrr]µs * ÿzGOF.wsqucCs*|j}|j}|j|| ¡dd}|S)Nçà?r&)rEr8r]Úmean)rhrEr8r^rrrr^¿szGOF.usqucCsp|j}|j}d}td|ƒD]<}|||d|…}|dk}d||||<|| ¡7}q|dd||}|S)Nrrrmg@rX)rEr8ÚrangerA)rhrEr8ZmsumÚjZmjÚmaskr_rrrr_ÇszGOF.ac CsX|j}|j}dt d|d¡dt |¡t d|ddd…¡ ¡ ||}|S)z4Stephens 1974, does not have p-value formula for A^2rXrrNéÿÿÿÿ)rEr8r r.ÚlogrA)rhrEr8Úasqurrrrt×s ÿÿÿzGOF.asqurr`cCsBt||ƒ}|dkr*t||||jƒ|fSt||||jƒSdS)z r`N)r)Ú gof_pvalsrE)rhZtestidZpvalsrDrrrÚget_testãs z GOF.get_testN)rr)rr`)Ú__name__Ú __module__Ú __qualname__Ú__doc__rirrZr[rr\r]r^r_rtrvrrrrrgns&         rgédc sÄddlm}|tƒ‰tdƒD]>}||ƒ}t||ƒ}tD]"}ˆ| | |d¡dd¡q6qt  ‡fdd„tDƒ¡}t dd   t¡ƒt d |d k  d¡ƒt d |d k  d¡ƒt d|dk  d¡ƒdS)Nr©Ú defaultdictièr`rcsg|] }ˆ|‘qSrr©Ú.0Úti©ÚresultsrrÚ szgof_mc..ú ú úat 0.01:ç{®Gáz„?úat 0.05:rNúat 0.10:rS) Ú collectionsr}ÚlistrorgÚall_gofsÚappendrvr rBÚprintÚjoinrn) ZrandfnÚdistrrEr}Úir+Úgoftr€ÚresarrrrrÚgof_mcõs   "r”cCsœt|jƒ}|j|}tdƒg|}dg|}tdƒ||<tdddƒ||<dt d|d¡t|ƒdt |¡t d|t|ƒ¡| |¡ |}|S)z.vectorized Anderson Darling A^2, Stephens 1974NrrrXrr)r r Úslicer r.ÚtuplersrA)r8ÚaxisÚndimrEZ slice_reverseÚislicertrrrÚasquares     ÿÿÿþršéÈcCs0|dk r´|dkrtdƒ‚tt |t|ƒ¡ƒ}d}t|ƒD]h}|j|fd||fiŽ} |j| dd} tdd„| ƒ} tj |  | | ¡dd} t | dd} || |k  ¡7}q:|t||ƒS|j|fd||fiŽ} |j| dd} td d„| ƒ} tj |  | | ¡dd} t | dd} |dkr t  | ¡} | S| |k  ¡SdS) aMonte Carlo (or parametric bootstrap) p-values for gof currently hardcoded for A^2 only assumes vectorized fit_vec method, builds and analyses (nobs, nrep) sample in one step rename function to less generic this works also with nrep=1 Nzusing batching requires a valuerrr©r—cSs t |d¡S©Nr©r Z expand_dims©Úxrrrrd;rezbootstrap..cSs t |d¡SrržrŸrrrrdDre)Ú ValueErrorÚintr Úceilrror+Úfit_vecrr r*ršrArn)rr2rEÚnrepÚvalueZ batch_sizeZn_batchÚcountÚirepr+Úparamsr8rDZ stat_sortedrrrÚ bootstraps,     rªc Csdd}t|ƒD]J}|j|fd|iŽ}| |¡}t | ||¡¡} t| dd} || |k7}q |d|S)zþMonte Carlo (or parametric bootstrap) p-values for gof currently hardcoded for A^2 only non vectorized, loops over all parametric bootstrap replications and calculates and returns specific p-value, rename function to less generic rrrœr)ror+r¤r r r*rš) r¦rr2rEr¥r§r¨r+r©r8rDrrrÚ bootstrap2Os   r«c@s*eZdZdZd dd„Zdd„Zdd„Zd S) ÚNewNormz-just a holder for modified distributions rcCs| |¡| |¡fSrb)rnZstd)rhr r—rrrr¤pszNewNorm.fit_veccCstjj||d|ddS)Nrr)ÚlocÚscale)rÚnormr*)rhr r2rrrr*ssz NewNorm.cdfcCs&|d}|d}||tjj|dS)Nrr©r)rr¯r+)rhr2rr­r®rrrr+vsz NewNorm.rvsN)r)rwrxryrzr¤r*r+rrrrr¬ls r¬Ú__main__)Ústatsér°z scipy kstestr¯rZr[r\r]r^r_z Is it correctly sized?r|rcCsg|] }t|‘qSrrr~rrrrƒ—srƒr„r…r†r‡rˆrNr‰rScCstjjd|dS)Nr³r°)r²Útr+©rErrrrdrerµ)rr)r2rEr¥r¦g®Gáz®ï?gffffffî?rW)rrrr)r{)r)rr›r{NN)rr›r{);rzZstatsmodels.compat.pythonrZnumpyr Z scipy.statsrZstatsmodels.tools.decoratorsrZ scipy.specialrrrr=rJZdminus_st70_upprKrMrRrVrYrurargr”ršrªr«r¬rwrfr²r´r+rŽr’rvrŒr€rŠr}r‹r‚rEror‘ÚrandomZrandnrrBr“rrnr¥ZbtÚfloorZastyper¢Z quantindexrrrrÚsŒ    _    ù ý ÿ   2         &