U Qv f@sddlZddlZddlZddlZddlmZddl m Z ddl m Z m Z ddddd d gZdd d d ddZdddZd ddZddddZd!ddZd"ddZd#dd Zd$dd ZdS)%N) namedtuple) _flatten_list) remove_na_postprocess_dataframegzscore normalityhomoscedasticityandersonepsilon sphericity propagateaxisddof nan_policycCsLtdt|}t|tjjr*tjjntj}tj j |||||d}|S)aGeometric standard (Z) score. Parameters ---------- x : array_like Array of raw values. axis : int or None, optional Axis along which to operate. Default is 0. If None, compute over the whole array `x`. ddof : int, optional Degrees of freedom correction in the calculation of the standard deviation. Default is 1. nan_policy : {'propagate', 'raise', 'omit'}, optional Defines how to handle when input contains nan. 'propagate' returns nan, 'raise' throws an error, 'omit' performs the calculations ignoring nan values. Default is 'propagate'. Note that when the value is 'omit', nans in the input also propagate to the output, but they do not affect the geometric z scores computed for the non-nan values. Returns ------- gzscore : array_like Array of geometric z-scores (same shape as x). Notes ----- Geometric Z-scores are better measures of dispersion than arithmetic z-scores when the sample data come from a log-normally distributed population [1]_. Given the raw scores :math:`x`, the geometric mean :math:`\mu_g` and the geometric standard deviation :math:`\sigma_g`, the standard score is given by the formula: .. math:: z = \frac{log(x) - log(\mu_g)}{log(\sigma_g)} References ---------- .. [1] https://en.wikipedia.org/wiki/Geometric_standard_deviation Examples -------- Standardize a lognormal-distributed vector: >>> import numpy as np >>> from pingouin import gzscore >>> np.random.seed(123) >>> raw = np.random.lognormal(size=100) >>> z = gzscore(raw) >>> print(round(z.mean(), 3), round(z.std(), 3)) -0.0 0.995 z]gzscore is deprecated and will be removed in pingouin 0.7.0; use scipy.stats.gzscore instead.r) warningswarnnpZ asanyarray isinstancemaZ MaskedArraylogscipystatsZzscore)xrrrrzr9/tmp/pip-unpacked-wheel-2te3nxqf/pingouin/distribution.pyr s5 shapiro皙?c st|tjtjttjfst|dks(tt|tjr<|}ddg}t t j |t|ttjfrt |}|j dks|td|jdkstdt|}t|j}||_t|d|kdd |d <n&|d kr"|d kr"|}|jfd d dddj}||_t|d|kdd |d <ntg}||jks`_: *"Like most statistical significance tests, if the sample size is sufficiently large this test may detect even trivial departures from the null hypothesis (i.e., although there may be some statistically significant effect, it may be too small to be of any practical significance); thus, additional investigation of the effect size is typically advisable, e.g., a Q–Q plot in this case."* Note that missing values are automatically removed (casewise deletion). References ---------- * Shapiro, S. S., & Wilk, M. B. (1965). An analysis of variance test for normality (complete samples). Biometrika, 52(3/4), 591-611. * https://www.itl.nist.gov/div898/handbook/prc/section2/prc213.htm Examples -------- 1. Shapiro-Wilk test on a 1D array. >>> import numpy as np >>> import pingouin as pg >>> np.random.seed(123) >>> x = np.random.normal(size=100) >>> pg.normality(x) W pval normal 0 0.98414 0.274886 True 2. Omnibus test on a wide-format dataframe with missing values >>> data = pg.read_dataset('mediation') >>> data.loc[1, 'X'] = np.nan >>> pg.normality(data, method='normaltest').round(3) W pval normal X 1.792 0.408 True M 0.492 0.782 True Y 0.349 0.840 True Mbin 839.716 0.000 False Ybin 814.468 0.000 False W1 24.816 0.000 False W2 43.400 0.000 False 3. Pandas Series >>> pg.normality(data['X'], method='normaltest') W pval normal X 1.791839 0.408232 True 4. Long-format dataframe >>> data = pg.read_dataset('rm_anova2') >>> pg.normality(data, dv='Performance', group='Time') W pval normal Time Pre 0.967718 0.478773 True Post 0.940728 0.095157 True 5. Same but using the Jarque-Bera test >>> pg.normality(data, dv='Performance', group='Time', method="jarque_bera") W pval normal Time Pre 0.304021 0.858979 True Post 1.265656 0.531088 True )rZ normaltestZ jarque_beraWpvalr zData must be 1D.z#Data must have more than 3 samples.TFnormalNcs |SN)dropna)rfuncrrznormality..expandr)Z result_typer)observedsortzGroup z. has less than 4 valid samples. Returning NaN.)r r!r#index)methodalpha)rZ ignore_index)"rpd DataFrameZSerieslistrndarrayAssertionErrorZto_framegetattrrrZasarrayndimsizerTcolumnswhere_get_numeric_dataapplygroupbygroupskeyscountrrnanrto_numpyconcatr.namer) datadvgroupr/r0Z col_namesrnumdatagrpcolsidxtmpZst_grprr&rrLsJ       levenecKst|tjttfst|dks&tttj |}t|tjr|dkr|dkr| }|j ddksltd|| j |\}} nT||jkst||jkst|j|dd|} | jdkstd|| t|\}} nt|tr tdd|Dstt|dkstd |||\}} nDtd d|Ds>> import numpy as np >>> import pingouin as pg >>> data = pg.read_dataset('mediation') >>> pg.homoscedasticity(data[['X', 'Y', 'M']]) W pval equal_var levene 1.173518 0.310707 True 2. Same data but using a long-format dataframe >>> data_long = data[['X', 'Y', 'M']].melt() >>> pg.homoscedasticity(data_long, dv="value", group="variable") W pval equal_var levene 1.173518 0.310707 True 3. Same but using a mean center >>> pg.homoscedasticity(data_long, dv="value", group="variable", center="mean") W pval equal_var levene 1.572239 0.209303 True 4. Bartlett test using a list of iterables >>> data = [[4, 8, 9, 20, 14], np.array([5, 8, 15, 45, 12])] >>> pg.homoscedasticity(data, method="bartlett", alpha=.05) T pval equal_var bartlett 2.873569 0.090045 True )rNZbartlettNr z$Data must have at least two columns.T)r+css|]}t|ttjfVqdSr$rr3rr4.0elrrr sz#homoscedasticity..z&Data must have at least two iterables.css|]}t|ttjfVqdSr$rOrPrrrrSsFrNr r9r! equal_varr-)rr1r2r3dictr5lowerr6rrr<shaperCr9r:r>Zngroupsr=alllenvaluesr)rFrGrHr/r0kwargsr'rIZ statisticprJrTZ stat_namerrrrrs0x   Znormdistc Gst|}tj|dd}t|}t|D]R}tjj|||d\}}}||krVdnd||<|tt ||||<q(|dkr|d}|d}||fS)aKAnderson-Darling test of distribution. The Anderson-Darling test tests the null hypothesis that a sample is drawn from a population that follows a particular distribution. For the Anderson-Darling test, the critical values depend on which distribution is being tested against. This function is a wrapper around :py:func:`scipy.stats.anderson`. Parameters ---------- sample1, sample2,... : array_like Array of sample data. They may be of different lengths. dist : string The type of distribution to test against. The default is 'norm'. Must be one of 'norm', 'expon', 'logistic', 'gumbel'. Returns ------- from_dist : boolean A boolean indicating if the data comes from the tested distribution (True) or not (False). sig_level : float The significance levels for the corresponding critical values, in %. See :py:func:`scipy.stats.anderson` for more details. Examples -------- 1. Test that an array comes from a normal distribution >>> from pingouin import anderson >>> import numpy as np >>> np.random.seed(42) >>> x = np.random.normal(size=100) >>> y = np.random.normal(size=10000) >>> z = np.random.random(1000) >>> anderson(x) (True, 15.0) 2. Test that multiple arrays comes from the normal distribution >>> anderson(x, y, z) (array([ True, True, False]), array([15., 15., 1.])) 3. Test that an array comes from the exponential distribution >>> x = np.random.exponential(size=1000) >>> anderson(x, dist="expon") (True, 15.0) bool)dtyper]TFr r) rYrzerosrangerrr anyZargminabs) r^argsk from_distZ sig_leveljstZcrsigrrrr s1  cCs|jjdkr|S|jjdkr|jj}|jt|dd}t|}|ddksZtd|ddkrv|j ddd}nh|ddkr|j ddd}|j ddddd j ddj dd}|j ddd}n|d krtd ntd |Std dS)zzCheck if data has multilevel columns for wide-format repeated measures. ``func`` can be either epsilon or mauchly r )rz%Factor must have at least two levels.r)levelrTF)rlrr+Z group_keysr zEpsilon values might be innaccurate in two-way repeated measures design where each factor has more than 2 levels. Please double-check your results.zIf using two-way repeated measures design, at least one factor must have exactly two levels. More complex designs are not yet supported.z.Only one-way or two-way designs are supported.N)r:nlevelsZremove_unused_levelslevshapeZreorder_levelsrZargsortr,r5Z droplevelZ sort_indexr>Zdiffr%rr ValueError)rFr'rnrrr_check_multilevel_rms6      rpc Cs||jkstd|||jjdks2td|||jksHtd|||rdtd|t|ttfrx|g}|D]}||jks|td|q||t |||g}t j ||||dddd}|S)zConvert long-format dataframe to wide-format. This internal function is used in pingouin.epsilon and pingouin.sphericity. z%s not in dataZbfiuz%s must be numericz Cannot have missing values in %smeanT)r.rZr:Zaggfuncr%r+) r:r5r`kindZisnullrcrstrint_flr1Z pivot_table)rFrGwithinsubjectwrrr_long_to_wide_rms&ryggcCst|tjstdtdd|||fDr>> import pandas as pd >>> import pingouin as pg >>> data = pd.DataFrame({'A': [2.2, 3.1, 4.3, 4.1, 7.2], ... 'B': [1.1, 2.5, 4.1, 5.2, 6.4], ... 'C': [8.2, 4.5, 3.4, 6.2, 7.2]}) >>> gg = pg.epsilon(data, correction='gg') >>> hf = pg.epsilon(data, correction='hf') >>> lb = pg.epsilon(data, correction='lb') >>> print("%.2f %.2f %.2f" % (lb, gg, hf)) 0.50 0.56 0.62 Now using a long-format dataframe >>> data = pg.read_dataset('rm_anova2') >>> data.head() Subject Time Metric Performance 0 1 Pre Product 13 1 2 Pre Product 12 2 3 Pre Product 17 3 4 Pre Product 12 4 5 Pre Product 19 Let's first calculate the epsilon of the *Time* within-subject factor >>> pg.epsilon(data, dv='Performance', subject='Subject', ... within='Time') 1.0 Since *Time* has only two levels (Pre and Post), the sphericity assumption is necessarily met, and therefore the epsilon adjustement factor is 1. The *Metric* factor, however, has three levels: >>> round(pg.epsilon(data, dv='Performance', subject='Subject', ... within=['Metric']), 3) 0.969 The epsilon value is very close to 1, meaning that there is no major violation of sphericity. Now, let's calculate the epsilon for the interaction between the two repeated measures factor: >>> round(pg.epsilon(data, dv='Performance', subject='Subject', ... within=['Time', 'Metric']), 3) 0.727 Alternatively, we could use a wide-format dataframe with two column levels: >>> # Pivot from long-format to wide-format >>> piv = data.pivot(index='Subject', columns=['Time', 'Metric'], values='Performance') >>> piv.head() Time Pre Post Metric Product Client Action Product Client Action Subject 1 13 12 17 18 30 34 2 12 19 18 6 18 30 3 17 19 24 21 31 32 4 12 25 25 18 39 40 5 19 27 19 18 28 27 >>> round(pg.epsilon(piv), 3) 0.727 which gives the same epsilon value as the long-format dataframe. Data must be a pandas Dataframe.cSsg|] }|dk qSr$rrQvrrr szepsilon..rGrvrwr r&TZ numeric_onlyrk?r ZlbZhf)rr1r2r5rXryr%rpcovrWr:rmrnrZdiagrqsummin)rFrGrvrw correctionSnrfdofkakbZmean_varZS_meanZss_matZss_rowsnumZdenepsrrrr +s8       ( mauchlycCsxt|tjstdtdd|||fDr>> import pandas as pd >>> import pingouin as pg >>> data = pd.DataFrame({'A': [2.2, 3.1, 4.3, 4.1, 7.2], ... 'B': [1.1, 2.5, 4.1, 5.2, 6.4], ... 'C': [8.2, 4.5, 3.4, 6.2, 7.2]}) >>> spher, W, chisq, dof, pval = pg.sphericity(data) >>> print(spher, round(W, 3), round(chisq, 3), dof, round(pval, 3)) True 0.21 4.677 2 0.096 John, Nagao and Sugiura (JNS) test >>> round(pg.sphericity(data, method='jns')[-1], 3) # P-value only 0.046 Now using a long-format dataframe >>> data = pg.read_dataset('rm_anova2') >>> data.head() Subject Time Metric Performance 0 1 Pre Product 13 1 2 Pre Product 12 2 3 Pre Product 17 3 4 Pre Product 12 4 5 Pre Product 19 Let's first test sphericity for the *Time* within-subject factor >>> pg.sphericity(data, dv='Performance', subject='Subject', ... within='Time') (True, nan, nan, 1, 1.0) Since *Time* has only two levels (Pre and Post), the sphericity assumption is necessarily met. The *Metric* factor, however, has three levels: >>> round(pg.sphericity(data, dv='Performance', subject='Subject', ... within=['Metric'])[-1], 3) 0.878 The p-value value is very large, and the test therefore indicates that there is no violation of sphericity. Now, let's calculate the epsilon for the interaction between the two repeated measures factor. The current implementation in Pingouin only works if at least one of the two within-subject factors has no more than two levels. >>> spher, _, chisq, dof, pval = pg.sphericity(data, dv='Performance', ... subject='Subject', ... within=['Time', 'Metric']) >>> print(spher, round(chisq, 3), dof, round(pval, 3)) True 3.763 2 0.152 Here again, there is no violation of sphericity acccording to Mauchly's test. Alternatively, we could use a wide-format dataframe with two column levels: >>> # Pivot from long-format to wide-format >>> piv = data.pivot(index='Subject', columns=['Time', 'Metric'], values='Performance') >>> piv.head() Time Pre Post Metric Product Client Action Product Client Action Subject 1 13 12 17 18 30 34 2 12 19 18 6 18 30 3 17 19 24 21 31 32 4 12 25 25 18 39 40 5 19 27 19 18 28 27 >>> spher, _, chisq, dof, pval = pg.sphericity(piv) >>> print(spher, round(chisq, 3), dof, round(pval, 3)) True 3.763 2 0.152 which gives the same output as the long-format dataframe. r{cSsg|] }|dk qSr$rr|rrrr~szsphericity..rrr&r rkTrrrNgMbP?r"i rz)rg?F SpherResultsspherr chi2rr!)rr1r2r5rXryr%rprWrrBrVrrCrqZlinalgZeigvalshprodrrrrrZsfr rrt)rFrGrvrwr/r0rrfdrrZS_popZeigr ZlogWfZw2Zchi_sqp1p2r!rrrrrrr sRB  8  (" )NNrr)NNrNr)r )NNN)NNNrz)NNNrr)rZ scipy.statsrZnumpyrZpandasr1 collectionsrZpingouin.utilsrrurr__all__rrrr rpryr r rrrrs$  ? 3 D 7  N