U Cv f/@sUddlmZddlZddlZddlZddlZddlZddlmZm Z ddl m Z ddl m Z ddlmZddlmZe edd d d Zd ed <ejstdiZded<edZGdddZGdddeZGdddZGdddeZGdddeZdS)) annotationsN)AnyHashable)acquire)LRUCache)utils)OPTIONSZfile_cache_maxsizecCs|SN)close)kvr @/tmp/pip-unpacked-wheel-h316xyqg/xarray/backends/file_manager.pyr)maxsizeZon_evictzLRUCache[Any, io.IOBase] FILE_CACHEz$file cache must be at least size onezdict[Any, int] REF_COUNTSzc@s.eZdZdZd ddZd ddZd ddZd S) FileManageraManager for acquiring and closing a file object. Use FileManager subclasses (CachingFileManager in particular) on backend storage classes to automatically handle issues related to keeping track of many open files and transferring them between multiple processes. TcCs tdS)z*Acquire the file object from this manager.NNotImplementedErrorself needs_lockr r rr"szFileManager.acquirecCs tdS)aContext manager for acquiring a file. Yields a file object. The context manager unwinds any actions taken as part of acquisition (i.e., removes it from any cache) if an exception is raised from the context. It *does not* automatically close the file. Nrrr r racquire_context&szFileManager.acquire_contextcCs tdS)z>Close the file object associated with this manager, if needed.Nrrr r rr /szFileManager.closeN)T)T)T)__name__ __module__ __qualname____doc__rrr r r r rrs  rc@seZdZdZeddddddddddZdd Zejd d Z d d dZ ejd!ddZ d"ddZ d#ddZ ddddZddZddddZddddZdS)$CachingFileManageraWrapper for automatically opening and closing file objects. Unlike files, CachingFileManager objects can be safely pickled and passed between processes. They should be explicitly closed to release resources, but a per-process least-recently-used cache for open files ensures that you can safely create arbitrarily large numbers of FileManager objects. Don't directly close files acquired from a FileManager. Instead, call FileManager.close(), which ensures that closed files are removed from the cache as well. Example usage: manager = FileManager(open, 'example.txt', mode='w') f = manager.acquire() f.write(...) manager.close() # ensures file is closed Note that as long as previous files are still cached, acquiring a file multiple times from the same FileManager is essentially free: f1 = manager.acquire() f2 = manager.acquire() assert f1 is f2 N)modekwargslockcache manager_id ref_countszHashable | None)r$c Gs||_||_||_|dkrint||_|dkp6|dk|_|jrHtn||_|dkrZt }||_ |dkrtt t }||_||_|dkrt}t||_|j|jdS)ayInitialize a CachingFileManager. The cache, manager_id and ref_counts arguments exist solely to facilitate dependency injection, and should only be set for tests. Parameters ---------- opener : callable Function that when called like ``opener(*args, **kwargs)`` returns an open file object. The file object must implement a ``close()`` method. *args Positional arguments for opener. A ``mode`` argument should be provided as a keyword argument (see below). All arguments must be hashable. mode : optional If provided, passed as a keyword argument to ``opener`` along with ``**kwargs``. ``mode='w' `` has special treatment: after the first call it is replaced by ``mode='a'`` in all subsequent function to avoid overriding the newly created file. kwargs : dict, optional Keyword arguments for opener, excluding ``mode``. All values must be hashable. lock : duck-compatible threading.Lock, optional Lock to use when modifying the cache inside acquire() and close(). By default, uses a new threading.Lock() object. If set, this object should be pickleable. cache : MutableMapping, optional Mapping to use as a cache for open files. By default, uses xarray's global LRU file cache. Because ``cache`` typically points to a global variable and contains non-picklable file objects, an unpickled FileManager objects will be restored with the default cache. manager_id : hashable, optional Identifier for this CachingFileManager. ref_counts : dict, optional Optional dict to use for keeping track the number of references to the same file. NF)_opener_args_modedict_kwargs_use_default_lock threadingLock_lockr_cachestruuidZuuid4 _manager_id _make_key_keyr _RefCounter _ref_counter increment) ropenerr r!r"r#r$r%argsr r r__init__Ps"2   zCachingFileManager.__init__cCs:|j|j|jdkrdn|jtt|j|jf}t|S)z.Make a key for caching files in the LRU cache.wa) r&r'r(tuplesortedr*itemsr2_HashedSequencervaluer r rr3szCachingFileManager._make_keyc cs(|r|j dVW5QRXndVdS)z0Context manager for optionally acquiring a lock.N)r.rr r r_optional_locksz!CachingFileManager._optional_lockTcCs||\}}|S)aAcquire a file object from the manager. A new file is only opened if it has expired from the least-recently-used cache. This method uses a lock, which ensures that it is thread-safe. You can safely acquire a file in multiple threads at the same time, as long as the underlying file object is thread-safe. Returns ------- file-like An open file object, as returned by ``opener(*args, **kwargs)``. )_acquire_with_cache_info)rrfile_r r rrszCachingFileManager.acquireccsB||\}}z |VWn$tk r<|s6||YnXdS)z%Context manager for acquiring a file.N)rD Exceptionr )rrrEcachedr r rrs  z"CachingFileManager.acquire_contextc Cs||z|j|j}Wnttk r|j}|jtk rN|}|j|d<|j|j |}|jdkrld|_||j|j<|dfYW5QRSX|dfW5QRSW5QRXdS)z=Acquire a file, returning the file and whether it was cached.r r;r<FTN) rCr/r4KeyErrorr*r( _DEFAULT_MODEcopyr&r')rrrEr!r r rrDs     z+CachingFileManager._acquire_with_cache_infoc Cs>||*d}|j|j|}|dk r0|W5QRXdS)z;Explicitly close any associated file object (if necessary).N)rCr/popr4r )rrdefaultrEr r rr s  zCachingFileManager.closeNone)returncCsn|j|j}|sj|j|jkrjt|jddrJz|jddW5|jXtdrjt j d|dt dddS) NF)blocking)rZwarn_for_unclosed_filesz deallocating z:, but file is not already closed. This may indicate a bug.) stacklevel) r6 decrementr4r/rr.releaser rwarningswarnRuntimeWarning)rZ ref_countr r r__del__s  zCachingFileManager.__del__cCs*|jr dn|j}|j|j|j|j||jfS)zState for pickling.N)r+r.r&r'r(r*r2)rr"r r r __getstate__szCachingFileManager.__getstate__cCs0|\}}}}}}|j|f|||||ddS)zRestore from a pickle.)r r!r"r$N)r:)rstater8r9r r!r"r$r r r __setstate__szCachingFileManager.__setstate__r0c CsZdtt|j}|jtk r,|d|j7}t|jd|jd|d|j d|j d S)Nz, z, mode=(z , kwargs=z , manager_id=)) joinmapreprr'r(rJtyperr&r*r2)r args_stringr r r__repr__s  ,zCachingFileManager.__repr__)T)T)T)T)rrrrrJr:r3 contextlibcontextmanagerrCrrrDr rXrYr[rcr r r rr4s(L     rc@s(eZdZdZddZddZddZdS) r5z,Class for keeping track of reference counts.cCs||_t|_dSr )_countsr,r-r.)rcountsr r rr:&sz_RefCounter.__init__c Cs2|j"|j|dd}|j|<W5QRX|S)Nr)r.rfgetrnamecountr r rr7*s&z_RefCounter.incrementc Cs<|j,|j|d}|r&||j|<n|j|=W5QRX|S)Nrh)r.rfrjr r rrS/s  z_RefCounter.decrementN)rrrrr:r7rSr r r rr5#sr5c@s eZdZdZddZddZdS)r@zSpeedup repeated look-ups by caching hash values. Based on what Python uses internally in functools.lru_cache. Python doesn't perform this optimization automatically: https://bugs.python.org/issue1462796 cCs||dd<t||_dSr )hash hashvalue)rZ tuple_valuer r rr:Bs z_HashedSequence.__init__cCs|jSr )rn)rr r r__hash__Fsz_HashedSequence.__hash__N)rrrrr:ror r r rr@9sr@c@s<eZdZdZddZd ddZejd ddZdd d Z d S)DummyFileManagerzHFileManager that simply wraps an open file in the FileManager interface.cCs ||_dSr _valuerAr r rr:MszDummyFileManager.__init__TcCs~|jSr rqrr r rrPszDummyFileManager.acquireccs~|jVdSr rqrr r rrTsz DummyFileManager.acquire_contextcCs~|jdSr )rrr rr r rr YszDummyFileManager.closeN)T)T)T) rrrrr:rrdrerr r r r rrpJs   rp) __future__rrdior,r1rUtypingrrZxarray.backends.locksrZxarray.backends.lru_cacherZ xarray.corerZxarray.core.optionsrr__annotations__rAssertionErrorrZ ReprObjectrJrrr5listr@rpr r r rs0       p