U Kv f^pã @sÊdZddlmZddlZdd„Zdd„Zdd „Zd@d d „ZdAd d„Z dBdd„Z dCdd„Z dDdd„Z dd„Z dEdd„ZGdd„dƒZdd„ZGdd„dƒZGd d!„d!ƒZed"krÆeƒZe ¡e ¡e ¡ddlmZd#d$dgd%Zd&d'd(gZd)d*gZe eeƒZeeƒe  d+¡Z!e  d+¡d,Z"e  d+¡d-d-Z#d#ekrXe e"ƒ\Z$Z%e e#ƒ\Z&Z'd$ekr‚ej(e"d.dd/Z"ej(e#d.dd/Z#e"j)dZ*e e"ƒ\Z$Z%e e#ƒ\Z&Z'e e$e&dd0Z+e e$e&d1d0Z,e e$e&d2d0Z-ee .e+j/e+¡e 0e+ 1d¡¡k 2¡ƒd3d4gd%Z3d5d6gdZ4e4ej5 6e*¡Z7e3e  d%d7¡Z8e+e8 1d%¡e7Z9e,e8 1d%¡e7Z:e-e8 1d%¡e7Z;e e j@ZAe?j@ZBe ¡e \d-¡f¡Z]e?j^d?d…Z_e> WeK Ne]¡¡jXZ`ee Le Me_e`¡¡ƒee;e"e#ƒZaeea b¡jXƒeea b¡jcƒeea d¡jeƒeea d¡jcƒeea fd%¡djXƒeea fd-¡djXƒeea g¡ƒdS)Faòfunctions to work with contrasts for multiple tests contrast matrices for comparing all pairs, all levels to reference level, ... extension to 2-way groups in progress TwoWay: class for bringing two-way analysis together and try out various helper functions Idea for second part - get all transformation matrices to move in between different full rank parameterizations - standardize to one parameterization to get all interesting effects. - multivariate normal distribution - exploit or expand what we have in LikelihoodResults, cov_params, f_test, t_test, example: resols_dropf_full.cov_params(C2) - connect to new multiple comparison for contrast matrices, based on multivariate normal or t distribution (Hothorn, Bretz, Westfall) é)Ú assert_equalNcCsTg}t|ƒD]<}t|d|ƒD](}t |¡}d||<d||<| |¡qq t |¡S)zæcontrast or restriction matrix for all pairs of nm variables Parameters ---------- nm : int Returns ------- contr : ndarray, 2d, (nm*(nm-1)/2, nm) contrast matrix for all pairwise comparisons ééÿÿÿÿ)ÚrangeÚnpÚzerosÚappendÚarray)ÚnmÚcontrÚiÚjZ contr_row©rúL/tmp/pip-unpacked-wheel-2v6byqio/statsmodels/sandbox/stats/contrast_tools.pyÚcontrast_allpairss   rcCs(t t |d¡t |d¡ f¡}|S)zçcontrast or restriction matrix for all against first comparison Parameters ---------- nm : int Returns ------- contr : ndarray, 2d, (nm-1, nm) contrast matrix for all against first comparisons r)rÚ column_stackÚonesÚeye)r r rrrÚcontrast_all_one5s $rcCst |¡t ||f¡|S)zåcontrast or restriction matrix for all against mean comparison Parameters ---------- nm : int Returns ------- contr : ndarray, 2d, (nm-1, nm) contrast matrix for all against mean comparisons )rrr)r rrrÚcontrast_diff_meanEs rFcCsF|dkr:|s"t |¡dkrdSdSt |¡dkr4dSdSnt|ƒSdS)N)rrrrú+ú-Ú)rÚsignÚstr)ÚxÚnoplusrrrÚsignstrTs rcs2|rtdddƒ‰ntdƒ‰‡‡fdd„|Dƒ}|S)Nrcs*g|]"}d dd„t|ˆƒˆDƒ¡‘qS)rcSs,g|]$\}}|dkrdt|dd|f‘qS©rz%s%sT)r©r©Ú.0ÚcÚvrrrÚ csÿz.contrast_labels...©ÚjoinÚzip©r!Úrow©ÚnamesÚslrrr$csþ  ÿz#contrast_labels..)Úslice)Z contrastsr+ÚreverseÚlabelsrr*rÚcontrast_labels^s þr0cs\t|ƒ}tˆƒ}‡fdd„|Dƒ‰t d|f¡}d|d<|sRtj|t|ƒ f}ntj|t|ƒ f}t |dd…t |¡¡} t| ˆdd} ‡fdd„| Dƒ} t d|f¡} d| d<|sÐtj| t|ƒ f} ntj| t|ƒ f} t t |¡| dd…¡}‡fd d„|Dƒ}|dk rH|dk rHt |ƒ\}}t |ƒ\}}t ||ƒ}nd}ˆ| | |||fS) aˆbuild contrast matrices for products of two categorical variables this is an experimental script and should be converted to a class Parameters ---------- names1, names2 : lists of strings contains the list of level labels for each categorical variable intgroup1, intgroup2 : ndarrays TODO: this part not tested, finished yet categorical variable Notes ----- This creates a full rank matrix. It does not do all pairwise comparisons, parameterization is using contrast_all_one to get differences with first level. ? does contrast_all_pairs work as a plugin to get all pairs ? cs"g|]}ˆD]}d||f‘q qS)z%s_%sr)r!r r )Únames2rrr$sz$contrast_product..r)rrNT)r.c s0g|](}d dd„t|ˆƒddd…Dƒ¡‘qS)rcSs,g|]$\}}|dkrdt|dd|f‘qSrrr rrrr$‹sÿú/contrast_product...Nrr%r(©Ú names_prodrrr$‹sþ ÿc s0g|](}d dd„t|ˆƒddd…Dƒ¡‘qS)rcSs,g|]$\}}|dkrdt|dd|f‘qSrrr rrrr$˜sÿr2Nrr%r(r3rrr$˜sþ ÿ) ÚlenrrÚr_rrZkronrr0Údummy_1dÚ dummy_product)Znames1r1Z intgroup1Z intgroup2ÚpairsZn1Zn2Zee1ÚddZ contrast_prodZnames_contrast_prod0Znames_contrast_prodZee2Zdd2Zcontrast_prod2Znames_contrast_prod2Úd1Ú_Úd2Údummyr)r1r4rÚcontrast_producths@ þ þ   ÿr?csŒˆdkrNdd„t| ¡dƒDƒ}|dd…dft | ¡d¡k t¡|fSt |¡}‡fdd„|Dƒ}|dd…df|k t¡|fSdS)a dummy variable for id integer groups Parameters ---------- x : ndarray, 1d categorical variable, requires integers if varname is None varname : str name of the variable used in labels for category levels Returns ------- dummy : ndarray, 2d array of dummy variables, one column for each level of the category (full set) labels : list[str] labels for the columns, i.e. levels of each category Notes ----- use tools.categorical instead for more more options See Also -------- statsmodels.tools.categorical Examples -------- >>> x = np.array(['F', 'F', 'M', 'M', 'F', 'F', 'M', 'M', 'F', 'F', 'M', 'M'], dtype='|S1') >>> dummy_1d(x, varname='gender') (array([[1, 0], [1, 0], [0, 1], [0, 1], [1, 0], [1, 0], [0, 1], [0, 1], [1, 0], [1, 0], [0, 1], [0, 1]]), ['gender_F', 'gender_M']) NcSsg|] }d|‘qS)zlevel_%dr©r!r rrrr$Ùszdummy_1d..rcsg|]}ˆdt|ƒ‘qS)z_%s)rr@©Úvarnamerrr$Ýs)rÚmaxrÚarangeZastypeÚintÚunique)rrBr/Z grouplabelsrrArr7ªs ., r7ÚfullcCs:|dkrD|dd…dd…df|dd…ddd…f |jdd¡}nò|dkr¸t|dd…dd…f|dd…dd…fƒ}t t |jdt¡|dd…dd…f|dd…dd…f|f¡}n~|dkr.t|dd…dd…f|dd…dd…fƒ}t t |jdt¡|dd…dd…f|dd…dd…f|f¡}ntdƒ‚|S) aŸdummy variable from product of two dummy variables Parameters ---------- d1, d2 : ndarray two dummy variables, assumes full set for methods 'drop-last' and 'drop-first' method : {'full', 'drop-last', 'drop-first'} 'full' returns the full product, encoding of intersection of categories. The drop methods provide a difference dummy encoding: (constant, main effects, interaction effects). The first or last columns of the dummy variable (i.e. levels) are dropped to get full rank dummy matrix. Returns ------- dummy : ndarray dummy variable for product, see method rGNrrú drop-lastú drop-firstrúmethod not recognized)ÚreshapeÚshaper8rrrrEÚ ValueError)r;r=Úmethodr:Úd12rlÚd12rrrrr8ás<*B *Br8c Cs¸|j\}}t tj|dddk¡\}}t tj|dddk¡\}}t |¡}tjdg|f|k ¡r€tj||dgf|k ¡sˆtdƒ‚tjdg|df}tj|d|gf} || fS)a¬start and endpoints of groups in a sorted dummy variable array helper function for nested categories Examples -------- >>> d1 = np.array([[1, 0, 0], [1, 0, 0], [1, 0, 0], [1, 0, 0], [0, 1, 0], [0, 1, 0], [0, 1, 0], [0, 1, 0], [0, 0, 1], [0, 0, 1], [0, 0, 1], [0, 0, 1]]) >>> dummy_limits(d1) (array([0, 4, 8]), array([ 4, 8, 12])) get group slices from an array >>> [np.arange(d1.shape[0])[b:e] for b,e in zip(*dummy_limits(d1))] [array([0, 1, 2, 3]), array([4, 5, 6, 7]), array([ 8, 9, 10, 11])] >>> [np.arange(d1.shape[0])[b:e] for b,e in zip(*dummy_limits(d1))] [array([0, 1, 2, 3]), array([4, 5, 6, 7]), array([ 8, 9, 10, 11])] r©Zaxisrrzdummy variable is not sorted)rLrZnonzeroZdiffrDr6ÚallrM) ÚdÚnobsÚnvarsÚstart1Zcol1Úend1Zcol1_ÚccÚstartÚendrrrÚ dummy_limitss  ÿr[cCsL|dkr |St|ƒ\}}t|ƒ\}}t ||¡}t ||¡}||k} || } || } |dkrÊt|dd…dd…f|dd…dd…fƒ} t t |jdt¡|dd…dd…f|dd…| ff¡} nx|dkr:t|dd…dd…f|dd…dd…fƒ}t t |jdt¡|dd…dd…f|dd…| ff¡} ntdƒ‚| | | fS) aÙunfinished and incomplete mainly copy past dummy_product dummy variable from product of two dummy variables Parameters ---------- d1, d2 : ndarray two dummy variables, d2 is assumed to be nested in d1 Assumes full set for methods 'drop-last' and 'drop-first'. method : {'full', 'drop-last', 'drop-first'} 'full' returns the full product, which in this case is d2. The drop methods provide an effects encoding: (constant, main effects, subgroup effects). The first or last columns of the dummy variable (i.e. levels) are dropped to get full rank encoding. Returns ------- dummy : ndarray dummy variable for product, see method rGrHNrrrIrrJ) r[rZin1dr8rrrLrErM)r;r=rNrVrWZstart2Zend2ÚfirstÚlastÚequalZ col_dropfZ col_droplrOr:rPrrrÚ dummy_nested2s"      *< *<r_c@s8eZdZdZdd„Zdd„Zdd„Zdd „Zd d „Zd S) ÚDummyTransformaConversion between full rank dummy encodings y = X b + u b = C a a = C^{-1} b y = X C a + u define Z = X C, then y = Z a + u contrasts: R_b b = r R_a a = R_b C a = r where R_a = R_b C Here C is the transform matrix, with dot_left and dot_right as the main methods, and the same for the inverse transform matrix, C^{-1} Note: - The class was mainly written to keep left and right straight. - No checking is done. - not sure yet if method names make sense cCs4tjj||ddd|_tjj||ddd|_dS)z\C such that d1 C = d2, with d1 = X, d2 = Z should be (x, z) in arguments ? r©ZrcondrN)rÚlinalgÚlstsqÚ transf_matrixÚinvtransf_matrix)Úselfr;r=rrrÚ__init__‚szDummyTransform.__init__cCst |j|¡S)z b = C a ©rÚdotrd)rfÚarrrÚdot_leftŠszDummyTransform.dot_leftcCst ||j¡S)z z = x C rh)rfrrrrÚ dot_rightszDummyTransform.dot_rightcCst |j|¡S)z a = C^{-1} b ©rrire)rfÚbrrrÚ inv_dot_left”szDummyTransform.inv_dot_leftcCst ||j¡S)z x = z C^{-1} rm)rfÚzrrrÚ inv_dot_right™szDummyTransform.inv_dot_rightN) Ú__name__Ú __module__Ú __qualname__Ú__doc__rgrkrlrorqrrrrr`as  r`cCsZt |¡}|jd}tdƒgdg|dtdƒg}|d|| d¡d| d¡S)angroupmeans using dummy variables Parameters ---------- x : array_like, ndim data array, tested for 1,2 and 3 dimensions d : ndarray, 1d dummy variable, needs to have the same length as x in axis 0. Returns ------- groupmeans : ndarray, ndim-1 means for each group along axis 0, the levels of the groups are the last axis Notes ----- This will be memory intensive if there are many levels in the categorical variable, i.e. many columns in the dummy variable. In this case it is recommended to use a more efficient version. rNé).Nrçð?)rZasarrayÚndimr-Úsum)rrSrUZslirrrÚ groupmean_d¢s  "rzc@sBeZdZdZddd„Zdd„Zdd„Zd d „Zd d „Zd d„Z dS)ÚTwoWayaýa wrapper class for two way anova type of analysis with OLS currently mainly to bring things together Notes ----- unclear: adding multiple test might assume block design or orthogonality This estimates the full dummy version with OLS. The drop first dummy representation can be recovered through the transform method. TODO: add more methods, tests, pairwise, multiple, marginal effects try out what can be added for userfriendly access. missing: ANOVA table NcCs<|jd|_|dkrd}d}n|\}}t||ƒ\|_|_\}}t||ƒ\|_|_\} } |jd|_} | jd|_} t || ƒ} | \}}}}}}| \|_ |_ |_ |_ |_}t|| dd}t|| dd}t||ƒ|_|jd|_||_t ||¡ ¡|_|jj|_|j |j¡|_d| d| d|_|j|j|_dS)NrrjrnrrG©rNrI)rLrTr7r;Ú d1_labelsr=Ú d2_labelsÚnlevel1Únlevel2r?Z prod_labelÚC1ÚC1_labelÚC2ÚC2_labelr8r`Ú transformrUZexogÚsmÚOLSÚfitÚresolsÚparamsroZ params_dropfZstart_interactionÚ n_interaction)rfZendogZfactor1Zfactor2ÚvarnamesZvname1Zvname2r;r}r=r~rr€ÚresÚprodlabrÚC1labrƒÚC2labr<Zdp_fullZdp_dropfrrrrgÙs,     zTwoWay.__init__cCs@|j}t t ||j|f¡t |¡f¡}|j |¡}||_|S)z7contrast/restriction matrix for no interaction ) r‹rÚhstackrrUrr…rqÚR_nointer_transf)rfÚniaÚ R_nointerr’rrrÚ r_nointerûs $ zTwoWay.r_nointercCsL|j}t t ||j|f¡t |¡f¡}|j |¡}||_|j   |¡}|S©z1ttests for no-interaction terms are zero ) r‹rr‘rrUrr…rqr’r‰Út_test)rfr“r”r’Zt_resrrrÚttest_interactions $  zTwoWay.ttest_interactioncCs| ¡}|j |¡Sr–)r•r‰Zf_test)rfr’rrrÚftest_interactionszTwoWay.ftest_interactioncCs4|dkr|j |j¡|jfS|j |j¡|jfSdS)Nr)r‰r—rr‚rƒr„)rfZ factorindrrrÚttest_conditional_effectszTwoWay.ttest_conditional_effectcCsRddlm}|j |j|j¡}|j}|j}d}tdg|jd}||||||dS)Nr)Ú SimpleTablez!Estimated Coefficients by factorsz%#10.4g)Z data_fmts)ÚtitleZtxt_fmt) Zstatsmodels.iolibr›rŠrKrr€r}r~Údict)rfr›Z params_arrZstubsÚheadersrœZ table_fmtrrrÚ summary_coeffs  ÿ ÿzTwoWay.summary_coeff)N) rrrsrtrurgr•r˜r™ršrŸrrrrr{Ås "  r{c@s,eZdZdd„Zdd„Zdd„Zdd„Zd S) ÚTestContrastToolscCs†dddg|_ddg|_t dddgdddgdddgdddgdddgdddgdddgdddgdddgdddgdddgdddgg ¡|_dS)NÚa0Úa1Úa2Úb0Úb1rr)Úv1nameÚv2namerr r;)rfrrrrg.s   õzTestContrastTools.__init__cCsªtjddddddddddddg dd}t ddgddgddgddgddgddgddgddgddgddgddgddgg ¡ddg}}t|d d \}}t||ƒt||ƒdS) NÚFÚMz|S1)ZdtyperrZgender_FZgender_MZgenderrA)rr r7r)rfrrSr/Zres_dZ res_labelsrrrÚ test_dummy_1d>s*ÿ õ õ  zTestContrastTools.test_dummy_1dc Csøt|j|jƒ}dgd}ddddddg|d<t d d d d d d gd d d d d d gd d d d d d gd d d d d d gg¡|d <d dddg|d<t d d d d d d gd d d d d d gd d d d d d gg¡|d<dddg|d<tdƒD]"}tjj||||t|ƒdqÐdS)NréZa0_b0Za0_b1Za1_b0Za1_b1Za2_b0Za2_b1gð¿grwrz a1_b0-a0_b0z a1_b1-a0_b1z a2_b0-a0_b0z a2_b1-a0_b1rvéz a0_b1-a0_b0z a1_b1-a1_b0z a2_b1-a2_b0éé)Úerr_msg) r?r¦r§rr rZtestingrr)rfÚres_cpZres_tÚiirrrÚtest_contrast_productQs  ý þ  z'TestContrastTools.test_contrast_productcCs>t|jƒ\}}t|t dddg¡ƒt|t dddg¡ƒdS)Nrr­éé )r[r;rrr )rfrnÚerrrÚtest_dummy_limitsasz#TestContrastTools.test_dummy_limitsN)rrrsrtrgrªr²r¶rrrrr ,sr Ú__main__ÚsmallZlargerr¡r¢r£r¤r¥r´r­rvr®rQr|rHrIrwg{®Gáz„?gü©ñÒMbP?gš™™™™™¹?érraz" tvalues for no effect of factor 1z/each test is conditional on a level of factor 2z" tvalues for no effect of factor 2z/each test is conditional on a level of factor 1)rvr­éþÿÿÿ)F)F)NNF)N)rG)rG)hruZ numpy.testingrZnumpyrrrrrr0r?r7r8r[r_r`rzr{r rrÚttr²rªr¶Zstatsmodels.apiÚapir†Zexamplesr¦r§r°ÚprintrDÚyÚx1Zx2r;r}r=r~ÚrepeatrLrTZdd_fullZdd_droplZdd_dropfriÚTZdiagryrRZ effect_sizeZ noise_scaleÚrandomZrandnZnoiseÚbetaZ ydata_fullZ ydata_droplZ ydata_dropfr‡rˆZresols_full_fullZresols_full_dropfrŠZ params_f_fZ params_f_dfZresols_dropf_fullZresols_dropf_dropfZ params_df_fZ params_df_dfrbrcZtr_ofZtr_foZ transf_f_dfrCÚabsrqrlrorŽrrrƒrr<r—ZtvalueZ cov_paramsr‘rrZR_nointZtvaluesZ inter_directZ inter_transfZtwr˜Zpvaluer™ZfvalueršrŸrrrrÚs¼  B 7 %, /A#g=            &    ÿ  ÿ   d