U Cv fq @sddlmZddlmZddlmZmZmZmZm Z m Z m Z m Z m Z mZmZddlZddlmZddlmZddlmZddlmZmZmZmZmZdd lmZm Z m!Z!m"Z"dd l#m$Z$m%Z%m&Z&erdd l'm(Z(dd l)m*Z*dd l+m,Z,ddl-m.Z.m/Z/m0Z0eee efZ1eZ2ee2ee1e2fee1e2e fee1e2e e ffZ3ee*e$e3fZ4ee,e ee4ffZ5ee4ej6ej7fZ8ee,e ee8ffZ9ej6ej7fZ:edddddddZ;GdddZZ?dwddd d!d"d#d$d%Z@d&d'ZAee$e efZBd(d)d*d+d,d-ZCdxd(d0d d1d2d3d4d5d6ZDdyd7d8d(d9d:d;ZEdd?ZFdeGd/fdcCs|d}t|dks|dkr |Sd}|dkr0d}|dkrJt|}||}|dkrVd}|dkr|ddD]"}t|||td }|d k rjqqj|dkr|}|ddD]}t|||}|sqq|std |d |r|ddD]}t|||}q|S) aReturn the unique variable from a list of variables or raise MergeError. Parameters ---------- name : hashable Name for this variable. variables : list of Variable List of Variable objects, all of which go by the same name in different inputs. compat : {"identical", "equals", "broadcast_equals", "no_conflicts", "override"}, optional Type of equality check to use. equals : None or bool, optional corresponding to result of compat test Returns ------- Variable to use in the result. Raises ------ MergeError: if any of the variables are not equal. rr&r0Nr.r-r/Zfillna)equivTz conflicting values for variable zT on objects to be combined. You can skip this check by specifying compat='override'.)lenrFZset_dimsgetattrrZcomputerG)rHr=rIr,outZcombine_methodZ dim_lengthsrCr5r5r6unique_variable_s<  rNcCs$|tkr td|dttdS)Nzcompat=z invalid: must be ) _VALID_COMPATrBsetrIr5r5r6_assert_compat_validsrRz"dict[Hashable, list[MergeElement]]zMapping[Any, MergeElement]None)grouped prioritizedr>c Cst|}tt}i}|D]<\}}|D].\}}|dk r(|t||||t|<q(q|D]r\} } t| } | |@} | rbt| t| krbddd| D} ddd| D}td| d|d|| qbdS) zbMake sure that elements given in prioritized will not corrupt any index given in grouped. Nz, css|] }|VqdSr2r5.0kr5r5r6 sz,_assert_prioritized_valid..css|] }|VqdSr2r5rVr5r5r6rYsz!cannot set or update variable(s) zA, which would corrupt the following index built from coordinates z: ) rPrlistitemsidappendrKjoinrB)rTrUZprioritized_namesZgrouped_by_indexindexesrH elements_list_indexZindex_idZindex_coord_namesZ index_namesZ common_namesZcommon_names_strZindex_names_strr5r5r6_assert_prioritized_valids" rcr.r0z!Mapping[Any, MergeElement] | Noner#zdict[Hashable, bool] | Nonez6tuple[dict[Hashable, Variable], dict[Hashable, Index]])rTrUrI combine_attrsr,r>cCs|dkr i}|dkri}t|t||i}i}i}|D]\}} ||krv||\} } | ||<| dk rt| ||<q>dd| D} | r^| d\} } | ddD]@\} }t| || | |std|d| d|d | d | d q|d kr0| ddD]4\}}t| j|jstd |d| jd|jq| ||<tdd| D|d||_| ||<q>dd| D}zt|||| |d||<Wn"tk r|dkrYnX||kr>tdd|D|d||_q>||fS)aEMerge dicts of variables, while resolving conflicts appropriately. Parameters ---------- grouped : mapping prioritized : mapping compat : str Type of equality check to use when checking for conflicts. combine_attrs : {"drop", "identical", "no_conflicts", "drop_conflicts", "override"} or callable, default: "override" A callable or a string indicating how to combine attrs of the objects being merged: - "drop": empty attrs on returned Dataset. - "identical": all attrs must be the same on every object. - "no_conflicts": attrs from all objects are combined, any that have the same name must also have the same value. - "drop_conflicts": attrs from all objects are combined, any that have the same name but different values are dropped. - "override": skip comparing and copy attrs from the first dataset to the result. If a callable, it must expect a sequence of ``attrs`` dicts and a context object as its only parameters. equals : mapping, optional corresponding to result of compat test Returns ------- Dict with keys taken by the union of keys on list_of_mappings, and Variable values corresponding to those that should be found on the merged result. NcSs g|]\}}|dk r||fqSr2r5rWvariablerbr5r5r6 sz#merge_collected..rr&zCconflicting values/indexes on objects to be combined fo coordinate z first index: z second index: z first variable: z second variable:  r+z2conflicting attribute values on combined variable z: first value: z second value: cSsg|]\}}|jqSr5attrs)rWrCrar5r5r6rg'srdcSsg|] \}}|qSr5r5)rWrfrar5r5r6rg,sr.cSsg|] }|jqSr5rirWrCr5r5r6rg9s) rRrcr[rrGrrj merge_attrsrNget)rTrUrIrdr,Z merged_varsmerged_indexesZindex_cmp_cacherHr`rfrbZindexed_elementsZ other_varZ other_indexZother_variablerar=r5r5r6merge_collectedsx(    "       rpzlist[DatasetLike]zMapping[Any, Any] | None)list_of_mappingsr_r>c s$ddlm}ddlm}|dkr$i}ttfddfdd}|D]}t||rf||j|jqH| D]\}}t||r|j }t |j} | |d| |d||| t||d }||kr܈||||qn|j|fkrt|\} || fd d | Dqn||dqnqHS) aCollect variables and indexes from list of mappings of xarray objects. Mappings must either be Dataset objects, or have values of one of the following types: - an xarray.Variable - a tuple `(dims, data[, attrs[, encoding]])` that can be converted in an xarray.Variable - or an xarray.DataArray If a mapping of indexes is given, those indexes are assigned to all variables with a matching key/name. rrr!Ncs|||fdSr2r])rHrfrb)rTr5r6r]Xsz-collect_variables_and_indexes..appendcs(|D]\}}||||qdSr2)r[rn)r=r_rHrfrrr5r6 append_all[sz1collect_variables_and_indexes..append_allrHcsi|] }|qSr5r5rVidxr5r6 rsz1collect_variables_and_indexes..)xarray.core.dataarrayr xarray.core.datasetr"rrZ isinstancer=Z_indexesr[_coordscopydictpoprr@r) rqr_r r"rsmappingrHrfZcoords_Zindexes_idx_varsr5)r]rTrvr6collect_variables_and_indexes?s4             rzlist[Coordinates])list_of_coordsr>cCsLtt}|D]:}|j}|j}|D] \}}|||||fq$q |S)zCCollect variables and indexes to be merged from Coordinate objects.)rrZr=xindexesr[r]rn)rrTcoordsr=r_rHrfr5r5r6collect_from_coordinatesysrr)objectsrU exclude_dimsrdr>c snt|}rBi}|D]&\}}fdd|D}|r|||<qn|}t|||d\} } t| t| } | | fS)zMerge variables/indexes from coordinates without automatic alignments. This function is used for merging coordinate from pre-existing xarray objects. cs$g|]\}}|jr||fqSr5) isdisjointr@rerr5r6rgs z3merge_coordinates_without_align..rk)rr[rprrP) rrUrrd collectedfilteredrHelementsZ new_elementsZ merged_coordsror5rr6merge_coordinates_without_aligns"    rzIterable[DatasetLike]z#tuple[set[Hashable], set[Hashable]])rqr>c Csddlm}ddlm}t}t}|D]b}t||rP||j||jq(| D]0\}}t||rXt|j }| |||qXq(||fS)aGiven a list of dicts with xarray object values, identify coordinates. Parameters ---------- list_of_mappings : list of dict or list of Dataset Of the same form as the arguments to expand_variable_dicts. Returns ------- coord_names : set of variable names noncoord_names : set of variable names All variable found in the input should appear in either the set of coordinate or non-coordinate names. rrr!) rxr ryr"rPrzupdater data_varsr[r{discard) rqr r" coord_namesnoncoord_namesrrHrCrr5r5r6determine_coordss       rzIterable[CoercibleMapping])rr>cCsddlm}ddlm}g}|D]d}t||r4|}nFi}t|trNt|}|D]"\}}t|trp||}|||<qV||q |S)aConvert pandas values found in a list of labeled objects. Parameters ---------- objects : list of Dataset or mapping The mappings may contain any sort of objects coercible to xarray.Variables as keys, including pandas objects. Returns ------- List of Dataset or dictionary objects. Any inputs or values in the inputs that were pandas objects have been converted into native xarray objects. rrr!) rxr ryr"rz PANDAS_TYPESr}r[r])rr r"rMobjr=rXvr5r5r6coerce_pandas_valuess        rr,z int | Nonezdict[Hashable, MergeElement])r priority_argrIr>c CsV|dkr iSt||g}t||d\}}i}|D]\}}|||f||<q6|S)ajExtract the priority variable from a list of mappings. We need this method because in some cases the priority argument itself might have conflicting values (e.g., if it is a dict with two DataArray values with conflicting coordinate values). Parameters ---------- objects : list of dict-like of Variable Dictionaries in which to find the priority variables. priority_arg : int or None Integer object whose variable should take priority. compat : {"identical", "equals", "broadcast_equals", "no_conflicts", "override"}, optional String indicating how to compare non-concatenated variables of the same name for potential conflicts. This is passed down to merge. - "broadcast_equals": all values must be equal when variables are broadcast against each other to ensure common dimensions. - "equals": all values and dimensions must be the same. - "identical": all values, dimensions and attributes must be the same. - "no_conflicts": only values which are not null in both datasets must be equal. The returned dataset then contains the combination of all non-null values. - "override": skip comparing and pick variable from first dataset Returns ------- A dictionary of variables and associated indexes (if any) to prioritize. NrQ)rrpr[rn) rrrIrr=r_rTrHrfr5r5r6_get_priority_vars_and_indexess#routerr%zMapping[Any, Index] | Noneobject)rrIr^rr_ fill_valuer>c CsVt|t|}t||d||d}t||d}t|||d} t|| |d\} } | | fS)zMerge coordinate variables. See merge_core below for argument descriptions. This works similarly to merge_core, except everything we don't worry about whether variables are coordinates or not. Fr^r|r_rr_rQ)rRrrrrrp) rrIr^rr_rcoercedalignedrrUr= out_indexesr5r5r6 merge_coords!s rcCs6t||\}}||g}|}t||||t||dS)zUsed in Dataset.__init__.)explicit_coordsr_)_create_indexes_from_coordskeys merge_corer)rrrIr^r_rrr5r5r6merge_data_and_coords:src st}|dk r||i}i}fdd|D}|D]d\}}t||d}|j|fkrt||\} |fdd| D|| || q@|||<q@||fS)zsMaybe create default indexes from a mapping of coordinates. Return those indexes and updated coordinates. Ncs*i|]"\}}|ks t|tjr||qSr5)rzpdZ MultiIndex)rWrXr)rr5r6rwWs  z/_create_indexes_from_coords..rtcsi|] }|qSr5r5rVrur5r6rwbs)r}rr[rr@r) rrZ all_variablesr_Zupdated_coordsZ index_varsrHrrfrr5)rrvr6rHs"       rcCs6|D],}||kr||j|fkrtd|dqdS)zValidate explicit coordinate names/dims. Raise a MergeError if an explicit coord shares a name with a dimension but is comprised of arbitrary dimensions. z coordinate z shares a name with a dataset dimension, but is not a 1D variable along that dimension. This is disallowed by the xarray data model.N)r@rG)r=r@rZ coord_namer5r5r6assert_valid_explicit_coordsks  rc s|sdSt|r|||dS|dkr(iS|dkr.cs,i|]$\}}|ks"t||r||qSr5)rrrir5r6rws csh|]}|kr|qSr5r5)rWr)rr5r6 szmerge_attrs..r+z6combine_attrs='identical', but attrs differ. First is z , other is .z%Unrecognised value for combine_attrs=) callabler}rrBrGstrrPrr[r)Zvariable_attrsrdrer5)rjrrr6rmzsV         rmc@s6eZdZUded<ded<ded<ded<d ed <d S) _MergeResultzdict[Hashable, Variable]r=z set[Hashable]rr<r@zdict[Hashable, Index]r_zdict[Hashable, Any]rjN)r8r9r:__annotations__r5r5r5r6rs rzSequence | None) rrIr^rdrrr_rr>csddlmddlmt|t|}t||d||d} t| |d} t| ||d} t | | ||d\} } t | }t |\}}|d k rt | ||| ||D]\}}|| kr||q||}|rtd |tfd d |D|}t| ||| |S) aCore logic for merging labeled objects. This is not public API. Parameters ---------- objects : list of mapping All values must be convertible to labeled arrays. compat : {"identical", "equals", "broadcast_equals", "no_conflicts", "override"}, optional Compatibility checks to use when merging variables. join : {"outer", "inner", "left", "right"}, optional How to combine objects with different indexes. combine_attrs : {"drop", "identical", "no_conflicts", "drop_conflicts", "override"} or callable, default: "override" How to combine attributes of objects priority_arg : int, optional Optional argument in `objects` that takes precedence over the others. explicit_coords : set, optional An explicit list of variables from `objects` that are coordinates. indexes : dict, optional Dictionary with values given by xarray.Index objects or anything that may be cast to pandas.Index objects. fill_value : scalar, optional Value to use for newly missing values Returns ------- variables : dict Dictionary of Variable objects. coord_names : set Set of coordinate names. dims : dict Dictionary mapping from dimension names to sizes. attrs : dict Dictionary of attributes Raises ------ MergeError if the merge cannot be done successfully. rrr!FrrrQ)rIrdNzZunable to determine if these variables should be coordinates or not in the merged result: cs g|]}t|fr|jqSr5)rzrjrlr r"r5r6rg szmerge_core..)rxr ryr"rRrrrrrprrrrr[add intersectionrGrmr)rrIr^rdrrr_rrrrrUr=rr@rrrDrEZambiguous_coordsrjr5rr6rsJ2         rr/z&Iterable[DataArray | CoercibleMapping]r")rrIr^rrdr>c Csddlm}ddlm}g}|D]@}t|||tfs>> x = xr.DataArray( ... [[1.0, 2.0], [3.0, 5.0]], ... dims=("lat", "lon"), ... coords={"lat": [35.0, 40.0], "lon": [100.0, 120.0]}, ... name="var1", ... ) >>> y = xr.DataArray( ... [[5.0, 6.0], [7.0, 8.0]], ... dims=("lat", "lon"), ... coords={"lat": [35.0, 42.0], "lon": [100.0, 150.0]}, ... name="var2", ... ) >>> z = xr.DataArray( ... [[0.0, 3.0], [4.0, 9.0]], ... dims=("time", "lon"), ... coords={"time": [30.0, 60.0], "lon": [100.0, 150.0]}, ... name="var3", ... ) >>> x array([[1., 2.], [3., 5.]]) Coordinates: * lat (lat) float64 35.0 40.0 * lon (lon) float64 100.0 120.0 >>> y array([[5., 6.], [7., 8.]]) Coordinates: * lat (lat) float64 35.0 42.0 * lon (lon) float64 100.0 150.0 >>> z array([[0., 3.], [4., 9.]]) Coordinates: * time (time) float64 30.0 60.0 * lon (lon) float64 100.0 150.0 >>> xr.merge([x, y, z]) Dimensions: (lat: 3, lon: 3, time: 2) Coordinates: * lat (lat) float64 35.0 40.0 42.0 * lon (lon) float64 100.0 120.0 150.0 * time (time) float64 30.0 60.0 Data variables: var1 (lat, lon) float64 1.0 2.0 nan 3.0 5.0 nan nan nan nan var2 (lat, lon) float64 5.0 nan 6.0 nan nan nan 7.0 nan 8.0 var3 (time, lon) float64 0.0 nan 3.0 4.0 nan 9.0 >>> xr.merge([x, y, z], compat="identical") Dimensions: (lat: 3, lon: 3, time: 2) Coordinates: * lat (lat) float64 35.0 40.0 42.0 * lon (lon) float64 100.0 120.0 150.0 * time (time) float64 30.0 60.0 Data variables: var1 (lat, lon) float64 1.0 2.0 nan 3.0 5.0 nan nan nan nan var2 (lat, lon) float64 5.0 nan 6.0 nan nan nan 7.0 nan 8.0 var3 (time, lon) float64 0.0 nan 3.0 4.0 nan 9.0 >>> xr.merge([x, y, z], compat="equals") Dimensions: (lat: 3, lon: 3, time: 2) Coordinates: * lat (lat) float64 35.0 40.0 42.0 * lon (lon) float64 100.0 120.0 150.0 * time (time) float64 30.0 60.0 Data variables: var1 (lat, lon) float64 1.0 2.0 nan 3.0 5.0 nan nan nan nan var2 (lat, lon) float64 5.0 nan 6.0 nan nan nan 7.0 nan 8.0 var3 (time, lon) float64 0.0 nan 3.0 4.0 nan 9.0 >>> xr.merge([x, y, z], compat="equals", fill_value=-999.0) Dimensions: (lat: 3, lon: 3, time: 2) Coordinates: * lat (lat) float64 35.0 40.0 42.0 * lon (lon) float64 100.0 120.0 150.0 * time (time) float64 30.0 60.0 Data variables: var1 (lat, lon) float64 1.0 2.0 -999.0 3.0 ... -999.0 -999.0 -999.0 var2 (lat, lon) float64 5.0 -999.0 6.0 -999.0 ... -999.0 7.0 -999.0 8.0 var3 (time, lon) float64 0.0 -999.0 3.0 4.0 -999.0 9.0 >>> xr.merge([x, y, z], join="override") Dimensions: (lat: 2, lon: 2, time: 2) Coordinates: * lat (lat) float64 35.0 40.0 * lon (lon) float64 100.0 120.0 * time (time) float64 30.0 60.0 Data variables: var1 (lat, lon) float64 1.0 2.0 3.0 5.0 var2 (lat, lon) float64 5.0 6.0 7.0 8.0 var3 (time, lon) float64 0.0 3.0 4.0 9.0 >>> xr.merge([x, y, z], join="inner") Dimensions: (lat: 1, lon: 1, time: 2) Coordinates: * lat (lat) float64 35.0 * lon (lon) float64 100.0 * time (time) float64 30.0 60.0 Data variables: var1 (lat, lon) float64 1.0 var2 (lat, lon) float64 5.0 var3 (time, lon) float64 0.0 4.0 >>> xr.merge([x, y, z], compat="identical", join="inner") Dimensions: (lat: 1, lon: 1, time: 2) Coordinates: * lat (lat) float64 35.0 * lon (lon) float64 100.0 * time (time) float64 30.0 60.0 Data variables: var1 (lat, lon) float64 1.0 var2 (lat, lon) float64 5.0 var3 (time, lon) float64 0.0 4.0 >>> xr.merge([x, y, z], compat="broadcast_equals", join="outer") Dimensions: (lat: 3, lon: 3, time: 2) Coordinates: * lat (lat) float64 35.0 40.0 42.0 * lon (lon) float64 100.0 120.0 150.0 * time (time) float64 30.0 60.0 Data variables: var1 (lat, lon) float64 1.0 2.0 nan 3.0 5.0 nan nan nan nan var2 (lat, lon) float64 5.0 nan 6.0 nan nan nan 7.0 nan 8.0 var3 (time, lon) float64 0.0 nan 3.0 4.0 nan 9.0 >>> xr.merge([x, y, z], join="exact") Traceback (most recent call last): ... ValueError: cannot align objects with join='exact' where ... Raises ------ xarray.MergeError If any variables with the same name have conflicting values. See also -------- concat combine_nested combine_by_coords rrr!zWobjects must be an iterable containing only Dataset(s), DataArray(s), and dictionaries.T)Z promote_attrs)rdr) rxr ryr"rzr} TypeErrorZ to_datasetr]rZ_construct_direct_asdict) rrIr^rrdr r"Zdict_like_objectsrZ merge_resultr5r5r6merges&c   rCoercibleMappingzHashable | Iterable[Hashable]r)datasetotheroverwrite_varsrIr^rrdr>c Cst|trt|tst|}n|h}|s6||g}d}n\|t|krP||g}d}nBi} i} |D]"\} } | |krz| | | <q`| | | <q`|| | g}d}t||||||dS)z!Guts of the Dataset.merge method.Nr&r')rrrd)rzrrrPr[r) rrrrIr^rrdobjsrZother_overwriteZother_no_overwriterXrr5r5r6dataset_merge_method s2     r)rrr>csddlm}ddlm}t||snt|}|D]:\}t|r2fddjD}|r2|||<q2t |gdj ddS) zGuts of the Dataset.update method. This drops a duplicated coordinates from `other` if `other` is not an `xarray.Dataset`, e.g., if it's a dict with DataArray values (GH2068, GH2180). rrr!cs$g|]}|jkr|jkr|qSr5)r@r)rWcrrr5r6rgHs z)dataset_update_method..r&r0)rr_rd) rxr ryr"rzr}r[rZ drop_varsrr)rrr r"rrr5rr6dataset_update_method9s"     r)r-N)Nr.r0N)N)r,)r-r)N)N)W __future__r collectionsrtypingrrrrrr r r r r rZpandasrZ xarray.corerZxarray.core.alignmentrZxarray.core.duck_array_opsrZxarray.core.indexesrrrrrZxarray.core.utilsrrrrZxarray.core.variablerrrZxarray.core.coordinatesrrxr ryr"Zxarray.core.typesr#r$r%ZDimsLikeZ ArrayLikeZ VariableLikeZ XarrayValueZ DatasetLikeZSeriesZ DataFrameZCoercibleValuerrrOr1rFrBrGrNrRZ MergeElementrcrprr frozensetrrrrZNArrrrrmrrrrrr5r5r5r6s  4           I"q:$&$0  # 6  [{.