U Cv f@s*ddlmZddlZddlZddlZddlZddlmZm Z ddl m Z m Z m Z mZmZmZddlmZddlmZddlmZddlmZmZmZmZdd lmZd Zd d ZGd dde Z ddZ!ddZ"d%ddZ#d&ddZ$ddZ%ddZ&Gddde Z'd'd d!Z(Gd"d#d#eZ)e)e d$<dS)() annotationsN)coding conventions)BACKEND_ENTRYPOINTSAbstractWritableDataStore BackendArrayBackendEntrypoint_encode_variable_name_normalize_path)StoreBackendEntrypoint)indexing) integer_types) FrozenDict HiddenKeyDictclose_on_errormodule_available)VariableZ_ARRAY_DIMENSIONScCs4t|tjr|}nt|tjr,|}n|}|S)a Encode a attribute value as something that can be serialized as json Many xarray datasets / variables have numpy arrays and values. This function handles encoding / decoding of such items. ndarray -> list scalar array -> scalar other -> other (no change) ) isinstancenpZndarraytolistZgenericitem)valueencodedr8/tmp/pip-unpacked-wheel-h316xyqg/xarray/backends/zarr.pyencode_zarr_attr_value!s    rc@s(eZdZdZddZddZddZdS) ZarrArrayWrapper) datastoredtypeshape variable_namecCs,||_||_|}|j|_|j}||_dSN)rr get_arrayrr)selfr rarrayrrrr__init__9s zZarrArrayWrapper.__init__cCs|jj|jSr!)r zarr_groupr r#rrrr"CszZarrArrayWrapper.get_arraycCs`|}t|tjr||jSt|tjr@|jt||jjSt|tj sPt |j |jSdSr!) r"rr Z BasicIndexertupleZVectorizedIndexerZvindexZ_arrayize_vectorized_indexerrZ OuterIndexerAssertionErrorZoindex)r#keyr$rrr __getitem__Fs   zZarrArrayWrapper.__getitem__N)__name__ __module__ __qualname__ __slots__r%r"r+rrrrr6s rc CsZ|s |s dS|rv|svtdd|Drksz)_determine_zarr_chunks..zIZarr requires uniform chunk sizes except for final chunk. Variable named z has incompatible dask chunks: z&. Consider rechunking using `chunk()`.css|]}|d|dkVqdS)rr0Nrr4rrrr7qszZFinal chunk of Zarr array must be the same size or smaller than the first. Variable named z has incompatible Dask chunks zb.Consider either rechunking using `chunk()` or instead deleting or modifying `encoding['chunks']`.css|]}|dVqdS)rNr)r5chunkrrrr7yszwzarr chunk sizes specified in `encoding['chunks']` must be an int or a tuple of ints. Instead found encoding['chunks']=z for variable named .r0z)Specified zarr chunks encoding['chunks']=z$ would overlap multiple dask chunks zH. Writing this array in parallel with dask could lead to corrupted data.zx Consider either rechunking using `chunk()`, deleting or modifying `encoding['chunks']`, or specify `safe_chunks=False`.z7We should never get here. Function logic must be wrong.) any ValueErrorr(rr r2_determine_zarr_chunksint TypeErrorzipNotImplementedErrorr)) Z enc_chunksZ var_chunksndimname safe_chunksZenc_chunks_tuplexZzchunkZdchunksZdchunkZ base_errorrrrr<UsL         r<c Csz|j|}Wntk r}z|s6td|d|tj|jd}t|j|}zdd|ddD}Wn4tk r}ztd|d|W5d}~XYnXW5d}~XYnXd d|jD}t|j|g|}||fS) Nz&Zarr object is missing the attribute `zA`, which is required for xarray to determine variable dimensions.z.zarraycSsg|]}tj|qSr)ospathbasenamer5dimrrr sz,_get_zarr_dims_and_attrs..Z _NCZARR_ARRAYZdimrefszZ` and the NCZarr metadata, which are required for xarray to determine variable dimensions.cSsg|]}|dr|qSZ_NC startswith)r5attrrrrrJs ) attrsKeyErrorrErFjoinjsonloadsstorer) zarr_objZ dimension_key try_nczarr dimensionseZ zarray_pathZzarrayZnc_attrs attributesrrr_get_zarr_dims_and_attrss0   &rZFTcs|j}dddddh|rBfdd|D}|r^td|nt|D]}|krJ||=qJt|d|j|j||}||d<|S) z Extract zarr encoding dictionary from xarray Variable Parameters ---------- variable : Variable raise_on_invalid : bool, optional Returns ------- encoding : dict Zarr encoding for `variable` r6 compressorfiltersZcache_metadataZwrite_empty_chunkscsg|]}|kr|qSrr)r5kZvalid_encodingsrrrJsz2extract_zarr_variable_encoding..z2unexpected encoding parameters for zarr backend: )encodingcopyr;listr<getr6rA)variableraise_on_invalidrBrCr_invalidr]r6rr^rextract_zarr_variable_encodings2  rfcCs:tj||d}tjjdd}|j||d}tj|}|S)a Converts an Variable into an Variable which follows some of the CF conventions: - Nans are masked using _FillValue (or the deprecated missing_value) - Rescaling via: scale_factor and add_offset - datetimes are converted to the CF 'units since time' format - dtype encodings are enforced. Parameters ---------- var : Variable A variable holding un-encoded data. Returns ------- out : Variable A variable which has been encoded as described above. )rBT)Zallows_unicode)rZencode_cf_variablerstringsZEncodedStringCoderencodeZensure_fixed_length_bytes)varZ needs_copyrBZcoderrrrencode_zarr_variables  rjc s|j|jkr,td|d|jd|jdi}|jD]P\}}|dk rz||krz|||\}} } | dksrt| |}|kr:|||<q:fdd|jD} || krtd|d|d| d d dS) Nz variable z/ already exists with different dimension names  != zA, but changing variable dimensions is not supported by to_zarr().r1csi|]\}}|kr||qSrr)r5rIsize append_dimrr 8sz+_validate_existing_dims..z0 already exists with different dimension sizes: z]. to_zarr() only supports changing dimension sizes when explicitly appending, but append_dim=r9)dimsr;Zsizesitemsindicesr)) var_namenew_var existing_varregionrnZexisting_sizesrIrlstartstopZstrideZ new_sizesrrmr_validate_existing_dims&s"   ryc CsBz|j|Wn,tk r<}ztd|W5d}~XYnX|S)z9Raise a more informative error message for invalid attrs.z#Invalid attribute in Dataset.attrs.N)rOputr>)rUrOrXrrr _put_attrsBs r{c @seZdZdZdZed&dd Zd'd d Zed d Z ddZ ddZ ddZ ddZ d(ddZddZddZddZeddfddZd d!Zd)d"d#Zd$d%ZdS)* ZarrStorez+Store for reading and writing data via zarr) r& _append_dim_consolidate_on_close_group_mode _read_only _synchronizer _write_region _safe_chunksrNFTc CsTddl}t|tjrt|}| dkr2t|dd} t|||d}||d<| dkr| |d<|s`|rvtd| d| d |dkrd }|r||d <|dkrd }|dkrz|j|f|}Wndt k rz"|j |f|}t j d t | d Wn*|jjk rtd|dYnXYnXn$|r4|j|f|}n|j |f|}||||| | | S)NrZ_store_versionr)mode synchronizerrFstorage_options zarr_versionz@consolidated metadata has not been implemented for zarr version z. yet. Set consolidated=False for zarr version zC. See also https://github.com/zarr-developers/zarr-specs/issues/136F chunk_storeaFailed to open Zarr store with consolidated metadata, but successfully read with non-consolidated metadata. This is typically much slower for opening a dataset. To silence this warning, consider: 1. Consolidating metadata in this existing store with zarr.consolidate_metadata(). 2. Explicitly setting consolidated=False, to avoid trying to read consolidate metadata, or 3. Explicitly setting consolidated=True, to raise an error in this case instead of falling back to try reading non-consolidated metadata.) stacklevelzNo such file or directory: '')zarrrrEPathLikefspathgetattrdictr;Zopen_consolidatedrP open_groupwarningswarnRuntimeWarningerrorsZGroupNotFoundErrorFileNotFoundError)clsrTrrgroup consolidatedconsolidate_on_closerrrn write_regionrCrrrZ open_kwargsr&rrrrZs^      zZarrStore.open_groupcCsF||_|jj|_|jj|_|jj|_||_||_||_ ||_ ||_ dSr!) r&Z read_onlyrrrrFrrr~r}rr)r#r&rrrnrrCrrrr%s    zZarrStore.__init__cCs|jSr!)r&r'rrrdssz ZarrStore.dscCstt||}|jdk}t|t|\}}t|}|dd|jtt ||j|j |j d}t |ddk rv|j |d<t||||S)Nrr\)r6Zpreferred_chunksr[r\ fill_value _FillValue)r ZLazilyIndexedArrayrrrZ DIMENSION_KEYrpopr6r?r[r\rrr)r#rB zarr_arraydatarVrWrYr_rrropen_store_variables"   zZarrStore.open_store_variablecstfddjDS)Nc3s"|]\}}|||fVqdSr!)rr5r]vr'rrr7sz*ZarrStore.get_variables..)rr&arraysr'rr'r get_variabless zZarrStore.get_variablescCsdd|jjDS)NcSs i|]\}}|ds||qSrKrLrrrrros z'ZarrStore.get_attrs..)r&rOZasdictrqr'rrr get_attrsszZarrStore.get_attrsc Cs|jdk}i}|jD]j\}}t|t|\}}t||jD]D\}}||krx|||krxtd|d|d||d|||<q.sz"ZarrStore.store..csi|]}||qSrrr)rrrro2sz#ZarrStore.store..F)deep)rzr+)r)rr3rhrZdecode_cf_variablesrrr`r_updateryrr}rrr set_variablesr~Zconsolidate_metadatar&rT)r#rrYcheck_encoding_setwriterrrZexisting_variable_namesZ new_variablesZvariables_without_encodingZvariables_encodedZ existing_varsrZvars_with_encodingrrsrtrur)r#rrrTsT      zZarrStore.storecCsdSr!rr'rrrsync[szZarrStore.synccs|D]\}}t|}||k}|j} |j} |j} |j} | dd} |jddikrf| dkrfi|_||j kr||j |}nxt ||||j d}i}| |t <| D]\}}| |||<qtj| tkrt} |j j|f| | | d|}t||}|jdk r|jnifdd| D|jdk r|j| kr| |j}|jtdksXtt|j|d|j<t|j}|||j|7<||tfdd| D}||j||qdS) a This provides a centralized method to set the variables on the data store. Parameters ---------- variables : dict-like Dictionary of key/value (variable name / xr.Variable) pairs check_encoding_set : list-like List of variables that should be checked for invalid encoding values writer unlimited_dims : list-like List of dimension names that should be treated as unlimited dimensions. rN)rdrBrC)rrrcsi|]}||tdqSr!)rbslicerHrrrrosz+ZarrStore.set_variables..c3s|]}|VqdSr!rrHrrrr7sz*ZarrStore.set_variables..)rqr rOr`rprrrr_r&rfrrrrrgZcheck_vlen_dtypestrcreater{rr}indexrr)raresizer(addr)r#rrrrrrrBcheckrOrprrrrr_Z encoded_attrsZk2Zv2Z append_axisZ new_shapervrrrr^s`         zZarrStore.set_variablescCsdSr!rr'rrrcloseszZarrStore.close) rNNFFNNNNTrN)NFNNT)N)N)r,r-r.__doc__r/ classmethodrr%propertyrrrrrrrrr frozensetrTrrrrrrrr|KsN  [    M Hr|autocKsddlm}|dkr>zddl}i}Wntk r<d}YnX|rXtdd||| | | | d|d}||||||||d || ||||d }|S) aLoad and decode a dataset from a Zarr store. The `store` object should be a valid store for a Zarr group. `store` variables must contain dimension metadata encoded in the `_ARRAY_DIMENSIONS` attribute or must have NCZarr format. Parameters ---------- store : MutableMapping or str A MutableMapping where a Zarr Group has been stored or a path to a directory in file system where a Zarr DirectoryStore has been stored. synchronizer : object, optional Array synchronizer provided to zarr group : str, optional Group path. (a.k.a. `path` in zarr terminology.) chunks : int or dict or tuple or {None, 'auto'}, optional Chunk sizes along each dimension, e.g., ``5`` or ``{'x': 5, 'y': 5}``. If `chunks='auto'`, dask chunks are created based on the variable's zarr chunks. If `chunks=None`, zarr array data will lazily convert to numpy arrays upon access. This accepts all the chunk specifications as Dask does. overwrite_encoded_chunks : bool, optional Whether to drop the zarr chunks encoded for each variable when a dataset is loaded with specified chunk sizes (default: False) decode_cf : bool, optional Whether to decode these variables, assuming they were saved according to CF conventions. mask_and_scale : bool, optional If True, replace array values equal to `_FillValue` with NA and scale values according to the formula `original_values * scale_factor + add_offset`, where `_FillValue`, `scale_factor` and `add_offset` are taken from variable attributes (if they exist). If the `_FillValue` or `missing_value` attribute contains multiple values a warning will be issued and all array values matching one of the multiple values will be replaced by NA. decode_times : bool, optional If True, decode times encoded in the standard NetCDF datetime format into datetime objects. Otherwise, leave them encoded as numbers. concat_characters : bool, optional If True, concatenate along the last dimension of character arrays to form string arrays. Dimensions will only be concatenated over (and removed) if they have no corresponding variable and if they are only used as the last dimension of character arrays. decode_coords : bool, optional If True, decode the 'coordinates' attribute to identify coordinates in the resulting dataset. drop_variables : str or iterable, optional A variable or list of variables to exclude from being parsed from the dataset. This may be useful to drop variables with problems or inconsistent values. consolidated : bool, optional Whether to open the store using zarr's consolidated metadata capability. Only works for stores that have already been consolidated. By default (`consolidate=None`), attempts to read consolidated metadata, falling back to read non-consolidated metadata if that fails. When the experimental ``zarr_version=3``, ``consolidated`` must be either be ``None`` or ``False``. chunk_store : MutableMapping, optional A separate Zarr store only for chunk data. storage_options : dict, optional Any additional parameters for the storage backend (ignored for local paths). decode_timedelta : bool, optional If True, decode variables and coordinates with time units in {'days', 'hours', 'minutes', 'seconds', 'milliseconds', 'microseconds'} into timedelta objects. If False, leave them encoded as numbers. If None (default), assume the same value of decode_time. use_cftime : bool, optional Only relevant if encoded dates come from a standard calendar (e.g. "gregorian", "proleptic_gregorian", "standard", or not specified). If None (default), attempt to decode times to ``np.datetime64[ns]`` objects; if this is not possible, decode times to ``cftime.datetime`` objects. If True, always decode times to ``cftime.datetime`` objects, regardless of whether or not they can be represented using ``np.datetime64[ns]`` objects. If False, always decode times to ``np.datetime64[ns]`` objects; if this is not possible raise an error. zarr_version : int or None, optional The desired zarr spec version to target (currently 2 or 3). The default of None will attempt to determine the zarr version from ``store`` when possible, otherwise defaulting to 2. Returns ------- dataset : Dataset The newly created dataset. See Also -------- open_dataset open_mfdataset References ---------- http://zarr.readthedocs.io/ r) open_datasetrNz-open_zarr() got unexpected keyword arguments ,)rroverwrite_encoded_chunksrrrrr)filename_or_objr decode_cfmask_and_scale decode_timesconcat_characters decode_coordsZenginer6drop_variablesbackend_kwargsdecode_timedelta use_cftimer)Zxarray.backends.apirZ dask.array ImportErrorr>rQkeys)rTrrr6rrrrrrrrrrrrrkwargsrZdaskrrrrr open_zarrsHu   rc@s2eZdZdZedZdZdZddZd d d Z dS)ZarrBackendEntrypointz Backend for ".zarr" files based on the zarr package. For more information about the underlying library, visit: https://zarr.readthedocs.io/en/stable See Also -------- backends.ZarrStore rz,Open zarr files (.zarr) using zarr in XarrayzVhttps://docs.xarray.dev/en/stable/generated/xarray.backends.ZarrBackendEntrypoint.htmlcCs4ztj|\}}Wntk r*YdSX|dkS)NF>.zarr)rErFsplitextr>)r#rrextrrrguess_can_open[s z$ZarrBackendEntrypoint.guess_can_openTNrc Csbt|}tj|| | | | d| ||d|d }t}t| |j||||||||d}W5QRX|S)NFr1) rrrrrrrrr)rrrrrrr)r r|rr rr)r#rrrrrrrrrrrrrrrrrTZstore_entrypointrrrrrbs4   z"ZarrBackendEntrypoint.open_dataset)TTTTNNNNrNNNNrN) r,r-r.rr available descriptionurlrrrrrrrKs*  rr)FNT)TN)NNrTTTTTNNFNNNNN)* __future__rrRrErZnumpyrZxarrayrrZxarray.backends.commonrrrrr r Zxarray.backends.storer Z xarray.corer Zxarray.core.pycompatr Zxarray.core.utilsrrrrZxarray.core.variablerrrrr<rZrfrjryr{r|rrrrrrs\      `# . ! c "H