U Kv fm'ã@s,dZddlmZddlZdd„Zdd„Zd"dd „ZGd d „d ƒZe d kr(ddl m Z d Z ej e ¡ZdgZdekrüeeƒeeedƒƒeeedƒƒe e ¡e ¡¡ZeeeƒZe  ee¡e  eeeƒed¡eje e¡e eeƒ¡ddZe dd„eDƒ¡Ze  ¡e  ee¡eeƒZe  ¡e  ee¡e  ¡e   edd…e !e¡e !e¡¡e e ¡e ¡d¡Z"e  ¡e   e"dd…e !ee"ƒ¡e !e"¡¡e e¡Z#e#dde d…Z$e  ¡e   e$dd…e !ee$ƒ¡e !e$¡¡eeƒZ%ee% &¡ƒee% 'e%j(¡ƒee% )dddg¡ƒee% 'dddddg¡ƒe e ¡e ¡d¡Zeje e¡e eeƒ¡ddZeeƒZe  ¡e  ee¡ejeeddZ*e*eƒZ+e  e+e¡ej,e eeƒ¡e e¡dddZ-e-eƒZ.e  e.e¡edƒed e !e+¡ ¡ƒed!e !e¡ ¡ƒdS)#a from David Huard's scipy sandbox, also attached to a ticket and in the matplotlib-user mailinglist (links ???) Notes ===== out of bounds interpolation raises exception and would not be completely defined :: >>> scoreatpercentile(x, [0,25,50,100]) Traceback (most recent call last): ... raise ValueError("A value in x_new is below the interpolation " ValueError: A value in x_new is below the interpolation range. >>> percentileofscore(x, [-50, 50]) Traceback (most recent call last): ... raise ValueError("A value in x_new is below the interpolation " ValueError: A value in x_new is below the interpolation range. idea ==== histogram and empirical interpolated distribution ------------------------------------------------- dual constructor * empirical cdf : cdf on all observations through linear interpolation * binned cdf : based on histogram both should work essentially the same, although pdf of empirical has many spikes, fluctuates a lot - alternative: binning based on interpolated cdf : example in script * ppf: quantileatscore based on interpolated cdf * rvs : generic from ppf * stats, expectation ? how does integration wrt cdf work - theory? Problems * limits, lower and upper bound of support does not work or is undefined with empirical cdf and interpolation * extending bounds ? matlab has pareto tails for empirical distribution, breaks linearity empirical distribution with higher order interpolation ------------------------------------------------------ * should work easily enough with interpolating splines * not piecewise linear * can use pareto (or other) tails * ppf how do I get the inverse function of a higher order spline? Chuck: resample and fit spline to inverse function this will have an approximation error in the inverse function * -> does not work: higher order spline does not preserve monotonicity see mailing list for response to my question * pmf from derivative available in spline -> forget this and use kernel density estimator instead bootstrap/empirical distribution: --------------------------------- discrete distribution on real line given observations what's defined? * cdf : step function * pmf : points with equal weight 1/nobs * rvs : resampling * ppf : quantileatscore on sample? * moments : from data ? * expectation ? sum_{all observations x} [func(x) * pmf(x)] * similar for discrete distribution on real line * References : ? * what's the point? most of it is trivial, just for the record ? Created on Monday, May 03, 2010, 11:47:03 AM Author: josef-pktd, parts based on David Huard License: BSD éNcCs6t |¡}t|ƒ}t t |¡t |¡¡}||dƒS)zÄReturn the score at the given percentile of the data. Example: >>> data = randn(100) >>> scoreatpercentile(data, 50) will return the median of sample `data`. çY@)ÚnpÚarrayÚ empiricalcdfÚ interpolateÚinterp1dÚsort)ÚdataZ percentileZperÚcdfÚ interpolator©r úJ/tmp/pip-unpacked-wheel-2v6byqio/statsmodels/sandbox/stats/stats_dhuard.pyÚscoreatpercentileVs rcCs,t|ƒ}t t |¡t |¡¡}||ƒdS)aDReturn the percentile-position of score relative to data. score: Array of scores at which the percentile is computed. Return percentiles (0-100). Example r = randn(50) x = linspace(-2,2,100) percentileofscore(r,x) Raise an error if the score is outside the range of data. r)rrrrr)r Úscorer r r r r ÚpercentileofscoredsrÚHazencCsÀt t |¡¡d}t|ƒ}| ¡}|dkr:|d|}n‚|dkrP||d}nl|dkrf|d|}nV|dkr€|d|d}n<|d krš|d|d }n"|d kr´|d |d }ntdƒ‚|S)aReturn the empirical cdf. Methods available: Hazen: (i-0.5)/N Weibull: i/(N+1) Chegodayev: (i-.3)/(N+.4) Cunnane: (i-.4)/(N+.2) Gringorten: (i-.44)/(N+.12) California: (i-1)/N Where i goes from 1 to N. çð?Úhazençà?ÚweibullÚ californiaÚ chegodayevç333333Ó?çš™™™™™Ù?Úcunnaneçš™™™™™É?Ú gringortenç)\Âõ(Ü?縅ëQ¸¾?ú[Unknown method. Choose among Weibull, Hazen,Chegodayev, Cunnane, Gringorten and California.)rÚargsortÚlenÚlowerÚ ValueError)r ÚmethodÚiÚNr r r r rvs"rc@s<eZdZdZdd„Zddd„Zdd „Zd d „Zdd d„ZdS)ÚHistDistz»Distribution with piecewise linear cdf, pdf is step function can be created from empiricial distribution or from a histogram (not done yet) work in progress, not finished cCs„t |¡|_t |j ¡|j ¡g¡|_t |¡}|||_t |¡|_ |  ¡}t  |¡|_ t  |j|j ¡|_t  |j |j¡|_dS)N)rZ atleast_1dr rÚminÚmaxÚbinlimitr Z _datasortedÚrankingrrZ _empcdfsortedrrÚcdfintpÚppfintp)Úselfr Zsortindr r r r Ú__init__¤s     zHistDist.__init__NrcCsÖ|dkr|j}|j}nt t |¡¡d}t|ƒ}| ¡}|dkrP|d|}n‚|dkrf||d}nl|dkr||d|}nV|dkr–|d|d }n<|d kr°|d |d }n"|d krÊ|d |d}ntdƒ‚|S)aAReturn the empirical cdf. Methods available: Hazen: (i-0.5)/N Weibull: i/(N+1) Chegodayev: (i-.3)/(N+.4) Cunnane: (i-.4)/(N+.2) Gringorten: (i-.44)/(N+.12) California: (i-1)/N Where i goes from 1 to N. Nrrrrrrrrrrrrrr)r r+rr r!r"r#)r.r r$r%r&r r r r r°s(zHistDist.empiricalcdfcCs | |¡S©z& this is score in dh )r,)r.rr r r Úcdf_empÙszHistDist.cdf_empcCs | |¡Sr0)r-)r.Zquantiler r r Úppf_empászHistDist.ppf_empÚFreedmancCspt|jƒ}|dkr8| d¡| d¡}d||d}n |dkrXdt |j¡|d}t |j¡||_|jS)z”Find the optimal number of bins and update the bin countaccordingly. Available methods : Freedman Scott r3çè?çÐ?égUUUUUUÕ¿ZScottgìQ¸…ë @)r!r r2rZstdZptpr*Znbin)r.r$ÚnobsZIQRÚwidthr r r Úoptimize_binningës zHistDist.optimize_binning)Nr)r3) Ú__name__Ú __module__Ú __qualname__Ú__doc__r/rr1r2r9r r r r r'šs   ) r'Ú__main__édr6éré2)ÚkcCsg|]}t |¡d‘qS)r@)ÚempZ derivatives)Ú.0Úxir r r Ú srFéÿÿÿÿér5r4gà¿gпiôég¸…ëQ¸ž?)rBÚsznegative densityz(np.diff(ppfs)).min()z(np.diff(cdf_ongrid)).min())r)/r=Zscipy.interpolaterZnumpyrrrrr'r:Zmatplotlib.pyplotZpyplotZpltr7ÚrandomZrandnÚxZexamplesÚprintZlinspacer(r)ZxsuppÚposZplotZInterpolatedUnivariateSplinerrCrZpdfempÚfigureZ cdf_ongridÚstepZdiffZxsupp2ZxsoÚxsZhistdr9r1r*r2r-ZppfsZUnivariateSplineZppfempZppfer r r r ÚslR  $d         $( (    "