U pu f]P@sdZddlmZddlZddlZddlZddlZddlZddlZddl m Z ddl Z ddl m Z ddl mZddlmZeded gZd d gZd aGd ddeZedAddZdBddZddZdCedddZddZdDddZdd Zd!d"ZdEd$d%Z dFd&d'Z!dGd(d)Z"d*d+Z#d,d-Z$d.d/Z%d0d1Z&d2d3Z'd4d5Z(d6d7Z)d8d9Z*d:d;Z+dd?Z-e.d@kre-dS)Hapipreqs - Generate pip requirements.txt file based on imports Usage: pipreqs [options] [] Arguments: The path to the directory containing the application files for which a requirements file should be generated (defaults to the current working directory). Options: --use-local Use ONLY local package info instead of querying PyPI. --pypi-server Use custom PyPi server. --proxy Use Proxy, parameter will be passed to requests library. You can also just set the environments parameter in your terminal: $ export HTTP_PROXY="http://10.10.1.10:3128" $ export HTTPS_PROXY="https://10.10.1.10:1080" --debug Print debug information --ignore ... Ignore extra directories, each separated by a comma --no-follow-links Do not follow symbolic links in the project --encoding Use encoding parameter for file open --savepath Save the list of requirements in the given file --print Output the list of requirements in the standard output --force Overwrite existing requirements.txt --diff Compare modules in requirements.txt to project imports --clean Clean up requirements.txt by removing modules that are not imported in project --mode Enables dynamic versioning with , or schemes. | e.g. Flask~=1.1.2 | e.g. Flask>=1.1.2 | e.g. Flask --scan-notebooks Look for imports in jupyter notebook files. )contextmanagerN)docopt) json2package) HTTPError) __version__z ^import (.+)$z!^from ((?!\.+).*?) import (?:.*)$z.pyz.pywFcs$eZdZdZeffdd ZZS)NbconvertNotInstalledzVIn order to scan jupyter notebooks, please install the nbconvert and ipython librariescst|dSN)super__init__)selfmessage __class__3/tmp/pip-unpacked-wheel-prro_duw/pipreqs/pipreqs.pyr AszNbconvertNotInstalled.__init__)__name__ __module__ __qualname__Zdefault_messager __classcell__rrr rr<srrc cst|r |dkr@|rd|kr tj}qJd|kr0tj}qJtd|n t||}z |VW5|tjtjfkrn|XdS)aOpen a file or ``sys.stdout`` depending on the provided filename. Args: filename (str): The path to the file that should be opened. If ``None`` or ``'-'``, ``sys.stdout`` or ``sys.stdin`` is returned depending on the desired mode. Defaults to ``None``. mode (str): The mode that should be used to open the file. Yields: A file handle. -rwzInvalid mode for file: {}N)sysstdinstdout ValueErrorformatopenclose)filenamemodefilerrr_openEs    r"utf-8Tc sNt}t}g}d}dddddddd g|r^g}|D]} |tjtj| q4|ttj||d } | D]P\} } } fd d | D| dd<|tj| d d | D}|dd |Dfdd | D} | D]}tj | |}t ||}zdt |}t |D]J}t |t jrF|jD]}||jq0nt |t jr||jqWqtk r}z>|rt|td|WYqntd||W5d}~XYqXqqvdd |DD] }|d\}}}||q|t||@}td|tt dd}dd|D}W5QRXt||S)NFz.hgz.svnz.gitz.tox __pycache__envvenvz.ipynb_checkpoints) followlinkscsg|]}|kr|qSrr).0d) ignore_dirsrr ~sz#get_all_imports..cSsg|]}t|tr|qSr)file_ext_is_allowedDEFAULT_EXTENSIONS)r(r!rrrr+s cSsg|]}tj|dqSrospathsplitext)r(rrrrr+scsg|]}t|r|qSr)r,)r(fn) extensionsrrr+s zFailed on file: %scSsg|] }|r|qSrr)r(nrrrr+s.zFound packages: {0}stdlibrcSsh|] }|qSrstripr(xrrr sz"get_all_imports..)setappendr0r1basenamerealpathextendget_file_extensionswalkjoinread_file_contentastparse isinstanceZImportnamesaddnameZ ImportFrommodule Exception traceback print_excloggingwarnerror partitiondebugrrlist)r1encodingextra_ignore_dirs follow_linksimportsZ raw_imports candidates ignore_errorsZignore_dirs_parsederCrootdirsfilespy_files file_namecontentstreenodeZsubnodeexcrKZ cleaned_name_packagesfdatar)r4r*rget_all_importsdsh       rjcCstrtdgStS)N.ipynb)scan_noteboooksr-rrrrrBsrB)rac CsNt|tr.t|d|d}|}W5QRXnt|dgrJtrJt||d}|S)NrrVrk)r,r-rreadrl ipynb_2_py)rarVrhrbrrrrEs   rEcCstj|d|kS)Nr/)raZ acceptablerrrr,sr,cCst}||\}}||S)z Args: file_name (str): notebook file path to parse as python script encoding (str): encoding of file Returns: str: parsed string )PythonExporter from_filenameencode)rarVZexporterbodyrfrrrros roc srt|d^}tdjt||ddd|Ddd|d|d fd d |Dd W5QRXdS) Nrz/Writing {num} requirements: {imports} to {file}, cSsg|] }|dqSrKrr:rrrr+sz.generate_requirements_file..)numr!rY{name}z {version} c3s.|]&}|drjf|n djf|VqdS)versionrxN)rr(itemfmtrr sz-generate_requirements_file..)r"rPrTrlenrDwrite)r1rYsymbolout_filerr}rgenerate_requirements_files   rcCstd||dS)Nr)r)rYrrrroutput_requirementssrhttps://pypi.python.org/pypi/c Csg}|D]}zrtd|tjd|||d}|jdkrbt|jdrVt|j }q|t|j}n|jdkr|t |j|j dWn$t k rtd|YqYnXtd ||j |j |j|||j d q|S) NzMImport named "%s" not found locally. Trying to resolve it at the PyPI server.z {0}{1}/json)proxiesdecodei,) status_codereasonz/Package "%s" does not exist or network problemszImport named "%s" was resolved to "%s:%s" package (%s). Please, verify manually the final list of requirements.txt to avoid possible dependency confusions.rKrz)rPwarningrequestsgetrrhasattrcontentrrrrrKZlatest_release_idpypi_urlr>)rY pypi_serverproxyresultr|responserirrrget_imports_infos4       rc Cs*g}dddddg}tjD] }t|D]\}}}|D]}d|kr6tj||}t|d|d}|tjd d } z| d } WnYW5QRq6YnXt } | D]"} | |kr| d |kr| | qd} t | d kr| d  dd dd} | | d | | dW5QRXq6q(q|S)NtestsZ_testsZeggZEGGinfoZ top_levelrrmrryrrpz.distz.egg)rKrzexports)rr1r0rCrDrsplitseprnr9rUr>rreplace)rVrgignorer1r]r^r_r|rhpackageZtop_level_modulesZfiltered_top_level_modulesrLrzrrrget_locally_installed_packagess6  rcsZt}g|D]0}|D]&}||dks2||dkr|qqfddtD}|S)NrrKcs(g|] \}}||ddkr|qS)rpNr)r(r5irrrr+?sz$get_import_local..)rr> enumerate)rYrVlocalr|rZ result_uniquerrrget_import_local/src Cs^t}ttdd}tdd|D}W5QRX|D]}||||q6t|dddS)zGet PyPI package names from a list of imports. Args: pkgs (List[str]): List of import names. Returns: List[str]: The corresponding PyPI package names. mappingrcss|]}|dVqdS):N)r9rr:rrrrPsz get_pkg_names..cSs|Srlower)srrrVzget_pkg_names..key)r=rrDdictrJrsorted)pkgsrrhripkgrrr get_pkg_namesDs rcCsHd|kr,td|}|r,|dd}|ddddS)Nzimport rz as r6)REGEXPmatchr9groupsrS)rKrrrrget_name_without_aliasYs rcCstjtjt|Sr)r0r1rDdirname__file__)rhrrrrDasrDc sHg}dddddg}zt|d}Wnltk rNtd|dtd Ynbtk r}z"td |d t||W5d }~XYn$Xzd d| D}W5| Xdd|D}|D]t fdd|Ds| d dD]P}||kr |}|d}|ddd} || d}||kr<| |qqq|S)u9Parse a requirements formatted file. Traverse a string until a delimiter is detected, then split at said delimiter, get module name by element index, create a dict consisting of module:version, and add dict to list of parsed modules. If file ´file_´ is not found in the system, the program will print a helpful message and end its execution immediately. Args: file_: File to parse. Raises: OSerror: If there's any issues accessing the file. Returns: list: The contents of the file, excluding comments. <>=!~rzFile z- was not found. Please, fix it and run again.rpz$There was an error opening the file z: NcSsg|]}|dkr|qS)ryr8r:rrrr+sz&parse_requirements..cSsg|]}|dr|qSr.)isalphar:rrrr+s csg|] }|kqSrr)r(yr;rrr+srrrr)rFileNotFoundErrorprintrexitOSErrorrPrRstrr readlinesanyr>rr) file_modulesdelimrhrRrirrL module_nameZmodule_versionrrrparse_requirementses6     rcsPt|fddttDfddttDtt}|S)a>Compare modules in a file to imported modules in a project. Args: file_ (str): File to parse for modules to be compared. imports (tuple): Modules being imported in the project. Returns: set: The modules not imported in the project, but do exist in the specified file. csg|]}|dqSrvrr(r)rYrrr+sz#compare_modules..csg|]}|dqSrvrr)rrrr+s)rrangerr=rrYmodules_not_importedr)rYrrcompare_moduless rc Cs0t||}td|ddd|DdS)zFDisplay the difference between modules in a file and imported modules.zBThe following modules are in {} but do not seem to be imported: {}rucss|] }|VqdSrrr:rrrrszdiff..N)rrPrrrDrrrrdiffs rcCst||}t|dkr(td|dStd|}g}zt|d}Wn&tk rpt d |Yn^XzP| D]}| |dkr|||q||d||D]}||qW5| Xtd|dS)z9Remove modules that aren't imported in project from file.rzNothing to clean in N|zr+zFailed on file: {}z(Successfully cleaned up requirements in )rrrPrrecompilerDrrrRrrrrr>seektruncater)rrYrZ re_removeZto_writerhrrrrcleans*      rcCs>|dkrdd|D}d}n|dkr*d}n |dkr6d}||fS) zDEnables dynamic versioning with , or schemes.no-pincSsg|]}|dddqS)rKrrrr{rrrr+sz&dynamic_versioning..rgtz>=compatz~=r)schemerYrrrrdynamic_versioningsrcCsBtstddSzddlmaWntk r<tYnXdS)Nz#Not scanning for jupyter notebooks.r)rq)rlrPrZ nbconvertrq ImportErrorrrrrrhandle_scan_noteboookss rc s8|d}|d}|d }|ddat|d}|dkrFd}|dkr\tjtj}|rj|d}|d rz|d n tj|d }|d s|d s|d stj |rt d dSt ||||d}t |}t dd|d}d}|dr|d}|dr|d|dd}|drr#,z --savepathzrequirements.txtz--printz--forcez..cSsg|] }|dqSrvrr:rrrr+9srr:rrrr+0szinit..)rrcSs |dS)NrKrrrrrr>rzinit..rz--diffz--cleanz--mode)rrrzFInvalid argument for mode flag, use 'compat', 'gt' or 'no-pin' insteadz==z Successfully output requirementsz(Successfully saved requirements file in )rrlrr0r1abspathcurdirrrDexistsrPrrjrrTrrrrrrrrrr) argsrVrWrXZ input_pathr1rZrrrY differencerrrrrinits                       rcCs^tttd}|drtjntj}tj|ddz t|Wntk rXt dYnXdS)N)rzz--debugz%(levelname)s: %(message)s)levelrr) r__doc__rrPDEBUGINFO basicConfigrKeyboardInterruptrr)rZ log_levelrrrmain[s  r__main__)Nr)r#NT)r#)r#)rN)r#)r#)/r contextlibrr0rrrPrFrNrrZyargrZyarg.exceptionsrZpipreqsrrrr-rlrrr"rjrBrrEr,rorrrrrrrrDrrrrrrrrrrrrrsR&        J   ! ) ;    e