U Kv f»Nã@s¢dZddlZddlmZddlZddlmZddl m Z ddl m Z dd„Z dd d „Zdd d „ZGdd„deƒZd dd„Zd!dd„Zd"dd„Zdd„Zdd„ZdS)#z= Created on Fri Sep 15 12:53:45 2017 Author: Josef Perktold éN)Ústats)Ú HolderTuple)ÚPoisson)ÚOLSc Cs²t |¡}|jdkr*d}|ddd…f}nd}g}g}tt|ƒdƒD]F}|||d…\}}| |dd…||…f d¡¡| ||¡qFt |¡}|r¤| ¡}|t |¡fS)aÅgroup columns into bins using sum This is mainly a helper function for combining probabilities into cells. It similar to `np.add.reduceat(x, edge_index, axis=-1)` except for the treatment of the last index and last cell. Parameters ---------- edge_index : array_like This defines the (zero-based) indices for the columns that are be combined. Each index in `edge_index` except the last is the starting index for a bin. The largest index in a bin is the next edge_index-1. x : 1d or 2d array array for which columns are combined. If x is 1-dimensional that it will be treated as a 2-d row vector. Returns ------- x_new : ndarray k_li : ndarray Count of columns combined in bin. Examples -------- >>> dia.combine_bins([0,1,5], np.arange(4)) (array([0, 6]), array([1, 4])) this aggregates to two bins with the sum of 1 and 4 elements >>> np.arange(4)[0].sum() 0 >>> np.arange(4)[1:5].sum() 6 If the rightmost index is smaller than len(x)+1, then the remaining columns will not be included. >>> dia.combine_bins([0,1,3], np.arange(4)) (array([0, 3]), array([1, 2])) éTNFé) ÚnpZasarrayÚndimÚrangeÚlenÚappendÚsumÚ column_stackZsqueeze) Z edge_indexÚxZis_1dZxliZkliZbin_idxÚiÚjZx_new©rúK/tmp/pip-unpacked-wheel-2v6byqio/statsmodels/discrete/_diagnostics_count.pyÚ _combine_binss)    rÚ predictedc Cs`t|tƒr|\}}n d|}}|dkr>ddlm}|jdd}| d¡}|j|d|d|j|d |d|dk r|| d|¡| ¡|  d ¡| d ¡} | jt   |¡d|d| jt   |¡d |d|dk rØ|  d|¡|  ¡|   d ¡| d ¡} |  t   |¡t   |¡d¡|  t   t |ƒ¡t |ƒt   t |ƒ¡t |ƒ¡|   d¡|  |¡|  |¡|S)a3diagnostic plots for comparing two lists of discrete probabilities Parameters ---------- freq, probs_predicted : nd_arrays two arrays of probabilities, this can be any probabilities for the same events, default is designed for comparing predicted and observed probabilities label : str or tuple If string, then it will be used as the label for probs_predicted and "freq" is used for the other probabilities. If label is a tuple of strings, then the first is they are used as label for both probabilities upp_xlim : None or int If it is not None, then the xlim of the first two plots are set to (0, upp_xlim), otherwise the matplotlib default is used fig : None or matplotlib figure instance If fig is provided, then the axes will be added to it in a (3,1) subplots, otherwise a matplotlib figure instance is created Returns ------- Figure The figure contains 3 subplot with probabilities, cumulative probabilities and a PP-plot ÚfreqNr)éé )Zfigsizei7z-o)Úlabelz-dZ probabilitiesi8zcumulative probabilitiesi9ÚozPP-plot)Ú isinstanceÚlistZmatplotlib.pyplotZpyplotÚfigureZ add_subplotZplotZset_xlimZlegendÚ set_titlerZcumsumÚaranger Z set_xlabelZ set_ylabel) rZprobs_predictedrZupp_xlimZfigZlabel0Úlabel1ZpltZax1Zax2Zax3rrrÚ plot_probsNs6            0   r!cCs4|}|j |j¡}|jjdd…dft |jd¡k t¡}|dk rlt ||ƒ\}}t ||ƒ\} }| jd}n||jd}}|} || } t  || dd…dd…ff¡} | jd} t t  | ¡| ƒ  ¡} | d| j| j}| jj|jd}||dkr ddl}| d¡|}tj ||¡}t|||| | dd}|S)u¾ chisquare test for predicted probabilities using cmt-opg Parameters ---------- results : results instance Instance of a count regression results probs : ndarray Array of predicted probabilities with observations in rows and event counts in columns bin_edges : None or array intervals to combine several counts into cells see combine_bins Returns ------- (api not stable, replace by test-results class) statistic : float chisquare statistic for tes p-value : float p-value of test df : int degrees of freedom for chisquare distribution extras : ??? currently returns a tuple with some intermediate results (diff, res_aux) Notes ----- Status : experimental, no verified unit tests, needs to be generalized currently only OPG version with auxiliary regression is implemented Assumes counts are np.arange(probs.shape[1]), i.e. consecutive integers starting at zero. Auxiliary regression drops the last column of binned probs to avoid that probabilities sum to 1. References ---------- .. [1] Andrews, Donald W. K. 1988a. “Chi-Square Diagnostic Tests for Econometric Models: Theory.†Econometrica 56 (6): 1419–53. https://doi.org/10.2307/1913105. .. [2] Andrews, Donald W. K. 1988b. “Chi-Square Diagnostic Tests for Econometric Models.†Journal of Econometrics 37 (1): 135–56. https://doi.org/10.1016/0304-4076(88)90079-6. .. [3] Manjón, M., and O. Martínez. 2014. “The Chi-Squared Goodness-of-Fit Test for Count-Data Models.†Stata Journal 14 (4): 798–816. Nréÿÿÿÿrz!auxiliary model is rank deficientÚchi2)Ú statisticÚpvalueÚdfÚdiff1Úres_auxÚ distribution)ÚmodelÚ score_obsÚparamsÚendogrrÚshapeÚastypeÚintrrrÚonesÚfitÚssrÚuncentered_tssZrankÚwarningsÚwarnrr#Úsfr)ÚresultsZprobsZ bin_edgesÚmethodÚresr+Zd_indZ d_ind_binsZk_binsZ probs_binsr'Zx_auxÚnobsr(Z chi2_statr&r5r$r%rrrÚtest_chisquare_probs:5*   úr<c@seZdZdd„ZdS)ÚDispersionResultscCs t |j|j|j|jdœ¡}|S)N)r$r%r9Ú alternative)ÚpdZ DataFramer$r%r9r>)ÚselfÚframerrrÚ summary_frameêsüzDispersionResults.summary_frameN)Ú__name__Ú __module__Ú __qualname__rBrrrrr=èsr=ÚallFcCsš|dkrtd|›dƒ‚t|dƒr(|j}|jj}|jd}| ¡}|jd}||}||}t  d|d  ¡¡} |  ¡} |  ¡| } | | } ||  ¡t  d|¡} dt j   t | ¡¡}dt j   t | ¡¡}dt j   t | ¡¡}| |g| |g| |gg}ddgd dgd d gg}||}t||ƒjd d }|jd}|jd}| ||g¡| ddg¡t||ƒjd d }|jd}|jd}| ||g¡| dd g¡||}t||ƒjdd d}|jd}|jd}| ||g¡| ddg¡t|t t|ƒ¡ƒjdd d}|jd}|jd}| ||g¡| dd g¡t |¡}|rT||fSt|dd…df|dd…dfdd„|Dƒdd„|Dƒdd}|SdS)aåScore/LM type tests for Poisson variance assumptions Null Hypothesis is H0: var(y) = E(y) and assuming E(y) is correctly specified H1: var(y) ~= E(y) The tests are based on the constrained model, i.e. the Poisson model. The tests differ in their assumed alternatives, and in their maintained assumptions. Parameters ---------- results : Poisson results instance This can be a results instance for either a discrete Poisson or a GLM with family Poisson. method : str Not used yet. Currently results for all methods are returned. _old : bool Temporary keyword for backwards compatibility, will be removed in future version of statsmodels. Returns ------- res : instance The instance of DispersionResults has the hypothesis test results, statistic, pvalue, method, alternative, as main attributes and a summary_frame method that returns the results as pandas DataFrame. )rFzunknown method "ú"Ú_resultsrrzDean Az mu (1 + a mu)zDean BzDean Cz mu (1 + a)F)Úuse_tzCT nb2zCT nb1ÚHC3)Úcov_typerIz CT nb2 HC3z CT nb1 HC3NrcSsg|] }|d‘qS)rr©Ú.0rrrrÚ Ysz+test_poisson_dispersion..cSsg|] }|d‘qS)rrrLrrrrNZszPoisson Dispersion Test)r$r%r9r>Úname)Ú ValueErrorÚhasattrrHr*r-r.ÚpredictÚresid_responserÚsqrtr rÚnormr7Úabsrr2ZtvaluesZpvaluesr r1r Úarrayr=)r8r9Z_oldr-r;ÚfittedÚresid2Zvar_resid_endogZvar_resid_fittedZstd1Zvar_resid_endog_sumZdean_aZdean_bZdean_cZ pval_dean_aZ pval_dean_bZ pval_dean_cZ results_allÚ descriptionÚendog_vZ res_ols_nb2Z stat_ols_nb2Z pval_ols_nb2Z res_ols_nb1Z stat_ols_nb1Z pval_ols_nb1Zstat_ols_hc1_nb2Zpval_ols_hc1_nb2Zstat_ols_hc1_nb1Zpval_ols_hc1_nb1r:rrrÚtest_poisson_dispersionõsz     þþ      ÿ     ûr\TrJcCs:t|dƒr|j}|jj}|jd} | ¡} |jd} |rB| |} n| | } | | } |jd}|g}|r~|j |j¡}|  |¡|dk r|  |¡t |ƒdkr¬t   |¡}d}n |d}d}t | |ƒj|||d}|r|jd}t  ||¡}| |¡}|j}|j}n0| jd} d|j|j}| |}tj ||¡}||fS) atA variable addition test for the variance function This uses an artificial regression to calculate a variant of an LM or generalized score test for the specification of the variance assumption in a Poisson model. The performed test is a Wald test on the coefficients of the `exog_new_test`. Warning: insufficiently tested, especially for options rHrrrNTF)rKÚcov_kwdsrI)rQrHr*r-r.rRrSr+r,r r rrrr2ZeyeZ wald_testr$r%r3r4rr#r7)r8Z exog_new_testZexog_new_controlZ include_scoreZ use_endogrKr]rIr-r;rXrYZ var_residr[Z k_constraintsZex_listr+ÚexZuse_waldZres_olsZk_varsÚ constraintsZhtZstat_olsZpval_olsZrsquared_noncenteredrrrÚ _test_poisson_dispersion_generic`sH         ÿ    r`cCs0t|jtƒsddl}| d¡|jjjd}|dkrBt |df¡}|jj}|jj }|  ¡}t  | ¡}|  ¡}|j |  |¡j } |j d|| |¡} ||dk||dd…df} |  d¡} | | j  |¡ | ¡} tj | ¡}|  |¡ | ¡}tj | ¡}|jd}tj ||¡}t||||dd}|S)uscore test for zero inflation or deflation in Poisson This implements Jansakul and Hinde 2009 score test for excess zeros against a zero modified Poisson alternative. They use a linear link function for the inflation model to allow for zero deflation. Parameters ---------- results_poisson: results instance The test is only valid if the results instance is a Poisson model. exog_infl : ndarray Explanatory variables for the zero inflated or zero modified alternative. I exog_infl is None, then the inflation probability is assumed to be constant. Returns ------- score test results based on chisquare distribution Notes ----- This is a score test based on the null hypothesis that the true model is Poisson. It will also reject for other deviations from a Poisson model if those affect the zero probabilities, e.g. in the direction of excess dispersion as in the Negative Binomial or Generalized Poisson model. Therefore, rejection in this test does not imply that zero-inflated Poisson is the appropriate model. Status: experimental, no verified unit tests, TODO: If the zero modification probability is assumed to be constant under the alternative, then we only have a scalar test score and we can use one-sided tests to distinguish zero inflation and deflation from the two-sided deviations. (The general one-sided case is difficult.) In this case the test specializes to the test by Broek References ---------- .. [1] Jansakul, N., and J. P. Hinde. 2002. “Score Tests for Zero-Inflated Poisson Models.†Computational Statistics & Data Analysis 40 (1): 75–96. https://doi.org/10.1016/S0167-9473(01)00104-9. rNz&Test is only valid if model is Poissonrr#)r$r%r&Z rank_scorer))rr*rr5r6r-r.rr1ÚexogrRÚexpZ cov_paramsÚTÚdotr ÚlinalgZpinvZ matrix_rankrr#r7r)Úresults_poissonZ exog_inflr5r;r-raÚmuÚ prob_zeroZcov_poiZcross_derivativeZcov_inflZscore_obs_inflZ score_inflZcov_score_inflZcov_score_infl_invr$Zdf2r&r%r:rrrÚtest_poisson_zeroinflation_jh¨s:1        ûric Cs²| ¡}t | ¡}|jj}|dk|| ¡}d|| ¡| ¡}|t |¡}dtj  t  |¡¡}tj  |¡}tj  |¡} t |||| |dtj   |dd¡ddd} | S)u`score test for zero modification in Poisson, special case This assumes that the Poisson model has a constant and that the zero modification probability is constant. This is a special case of test_poisson_zeroinflation derived by van den Broek 1995. The test reports two sided and one sided alternatives based on the normal distribution of the test statistic. References ---------- .. [1] Broek, Jan van den. 1995. “A Score Test for Zero Inflation in a Poisson Distribution.†Biometrics 51 (2): 738–43. https://doi.org/10.2307/2532959. rrrÚnormal©r$r%Zpvalue_smallerZ pvalue_largerr#Z pvalue_chi2Zdf_chi2r))rRrrbr*r-r rTrrUr7rVÚcdfrr#) rfrgrhr-ZscoreZ var_scorer$Ú pvalue_twoÚ pvalue_uppÚ pvalue_lowr:rrrÚ test_poisson_zeroinflation_broeks(   ø rpc Csî|jj}| ¡}t | ¡}|jjdk t¡}| ¡| ¡}|d|}||}tj   |j ||¡}||} | || } || } |t  | ¡} dt j t | ¡¡} t j | ¡}t j | ¡}t| | ||| dt j | dd¡ddd}|S)u´Test for excess zeros in Poisson regression model. The test is implemented following Tang and Tang [1]_ equ. (12) which is based on the test derived in He et al 2019 [2]_. References ---------- .. [1] Tang, Yi, and Wan Tang. 2018. “Testing Modified Zeros for Poisson Regression Models:†Statistical Methods in Medical Research, September. https://doi.org/10.1177/0962280218796253. .. [2] He, Hua, Hui Zhang, Peng Ye, and Wan Tang. 2019. “A Test of Inflated Zeros for Poisson Regression Models.†Statistical Methods in Medical Research 28 (4): 1157–69. https://doi.org/10.1177/0962280217749991. rrrrjrk)r*rarRrrbr-r/r0r reÚinvrcrTrrUr7rVrlrr#)r8rZmeanZprob0ÚcountsZdiffZvar1ZpmÚcZpmxZvar2Úvarr$rmrnror:rrrÚtest_poisson_zeros0s4     ø ru)rNN)NN)rFF)NFTrJNF)N)Ú__doc__ZnumpyrZscipyrZpandasr?Zstatsmodels.stats.baserZ#statsmodels.discrete.discrete_modelrZ#statsmodels.regression.linear_modelrrr!r<r=r\r`rirprurrrrÚs.    <ÿ ? [ nø H Z.