Summary: | spoils buildreq results again (while reading the list of available pkgs) | ||||||||
---|---|---|---|---|---|---|---|---|---|
Product: | Sisyphus | Reporter: | Ivan Zakharyaschev <imz> | ||||||
Component: | python3-module-setuptools | Assignee: | Grigory Ustinov <grenka> | ||||||
Status: | CLOSED FIXED | QA Contact: | qa-sisyphus | ||||||
Severity: | normal | ||||||||
Priority: | P3 | CC: | antohami, cas, cow, evg, george, grenka, imz, kotopesutility, lav, nbr, qa_viy, rider, sem, shaba, sin, slev, slev, toni, vitty, viy | ||||||
Version: | unstable | ||||||||
Hardware: | all | ||||||||
OS: | Linux | ||||||||
Bug Depends on: | 34660 | ||||||||
Bug Blocks: | |||||||||
Attachments: |
|
Description
Ivan Zakharyaschev
2018-03-16 18:54:05 MSK
(In reply to comment #0) > We used to have 2 patches to fight this spoiling: > > https://packages.altlinux.org/en/p8/srpms/python-module-setuptools/patches/0001-command-test.py-skip-install_requires-and-tests_requ.patch > https://packages.altlinux.org/en/p8/srpms/python-module-setuptools/patches/0002-dist.py-skip-checking-the-existence-of-system-PKG-IN.patch > > In Sisyphus, only the second one seems to be applied. > > Some fix in the same spirit is needed. (The detected used files can be seen by > running filereq FILELIST rpm -bc *.spec.) grenka@ and me know how to re-implement the first patch. Coming soon... python-module-setuptools-1:38.4.0-alt3 -> sisyphus: Thu Mar 22 2018 Grigory Ustinov <grenka@altlinux> 1:38.4.0-alt3 - Add patch for skipping some requires from glebfm@, adapted for new code. It restores feature from commit 362ea68 in 1:18.1-alt2 release. (Closes: #34658) (In reply to comment #0) > We used to have 2 patches to fight this spoiling: > > https://packages.altlinux.org/en/p8/srpms/python-module-setuptools/patches/0001-command-test.py-skip-install_requires-and-tests_requ.patch > https://packages.altlinux.org/en/p8/srpms/python-module-setuptools/patches/0002-dist.py-skip-checking-the-existence-of-system-PKG-IN.patch slev@: патч не имеет отношения к этой проблеме. Но хорошо, что его вернули. Вспомнил, зачем он был нужен: Это было когда-то Глебом сделано вот ради чего: buildreq не записывает (и правильно делает) многие вещи, которые не используются во время прогона тестов в %%check в BuildRequires, а при попытке запуска тестов (какого0то пакета) без этих вещей в сборочной среде, setuptools проверяет их начилие (просто так -- ради проверки, а не ради использования) и может случиться ошибка (он попытается скачать). > In Sisyphus, only the second one seems to be applied. > > Some fix in the same spirit is needed. (The detected used files can be seen by > running filereq FILELIST rpm -bc *.spec.) Let me try to fix this issue. python setuptools during first import of pkg_resources builds the initial/master working set(a collection of importable distributions) from sys.path. For each egg info distribution the read of metadata is applied to read the distribution version. I guess we should ignore these readings while building filereq list. Please check the following patches: python2 http://git.altlinux.org/people/slev/packages/?p=python.git;a=commitdiff;h=f788b33a1b2eebea7fb98b80ac6eea7610235154 python3 http://git.altlinux.org/people/slev/packages/?p=python3.git;a=commitdiff;h=cac7cb9d9d3a86fe9b2e6b1acc9046452b6c67d5 Test results: 1) python3 packagereq: optimized out: python-base python-modules python3 python3-base packagereq: BuildRequires: libenchant python3-dev python3-module-setuptools 2) python packagereq: optimized out: python-base python-devel python-modules python-modules-compiler python-modules-ctypes python-modules-email python-modules-encodings packagereq: BuildRequires: libenchant python-module-setuptools python-module-sphinxcontrib Waiting for your comments... (In reply to comment #5) > python setuptools during first import of pkg_resources builds the > initial/master working set(a collection of importable distributions) from > sys.path. For each egg info distribution the read of metadata is applied to > read the distribution version. > > I guess we should ignore these readings while building filereq list. > Please check the following patches: > python2 > http://git.altlinux.org/people/slev/packages/?p=python.git;a=commitdiff;h=f788b33a1b2eebea7fb98b80ac6eea7610235154 > > python3 > http://git.altlinux.org/people/slev/packages/?p=python3.git;a=commitdiff;h=cac7cb9d9d3a86fe9b2e6b1acc9046452b6c67d5 > > Test results: > 1) python3 > packagereq: optimized out: python-base python-modules python3 python3-base > packagereq: BuildRequires: libenchant python3-dev python3-module-setuptools > > 2) python > packagereq: optimized out: python-base python-devel python-modules > python-modules-compiler python-modules-ctypes python-modules-email > python-modules-encodings > packagereq: BuildRequires: libenchant python-module-setuptools > python-module-sphinxcontrib > > Waiting for your comments... Simple comment: it's better to change the release & changelog in a separate commit (so that it is easier to cherry-pick your essential changes). Another not so important, historic comment: As for the presence of .pth in this regex in python{,3} package, it is justified by that .pth are interpreted by python itself, they affect python. As for the egg-info things such as entry_points, their presence in python{,3} packages seems not to be very justified because they are processed by setuptools/distutils, and not python{,3}. But I think that it's OK, anyway, the egg-info is tightly related with standardized Python infrastructure. As for the real reason: it interesting to know better what happens. In older versions (e.g., 18.1-alt4 932975fde88184790eaacbd7bc0a65ddd081bde7), the following patch must have been enough not to read PKG-INFO: https://packages.altlinux.org/en/p8/srpms/python-module-setuptools/patches/0002-dist.py-skip-checking-the-existence-of-system-PKG-IN.patch ? Does reading PKG-INFO happens more times in setuptools now? Is this information really needed if BuildRequires and Requires must completely describe the requirements? I don't understand yet the answer, and can't find the place in the new code where the additional reads happen. Created attachment 7483 [details]
setuptools-18.1-alt4.bt
Created attachment 7484 [details]
setuptools-39.0.1.bt
The key point here is classmethod "from_location" (pkg_resources/__init__.py). Please check an attached backtraces for details. setuptools 18.1: @classmethod def from_location(cls, location, basename, metadata=None,**kw): project_name, version, py_version, platform = [None]*4 basename, ext = os.path.splitext(basename) if ext.lower() in _distributionImpl: # .dist-info gets much metadata differently match = EGG_NAME(basename) if match: project_name, version, py_version, platform = match.group( 'name','ver','pyver','plat' ) cls = _distributionImpl[ext.lower()] return cls( location, metadata, project_name=project_name, version=version, py_version=py_version, platform=platform, **kw ) But setuptools 39: @classmethod def from_location(cls, location, basename, metadata=None, **kw): project_name, version, py_version, platform = [None] * 4 basename, ext = os.path.splitext(basename) if ext.lower() in _distributionImpl: cls = _distributionImpl[ext.lower()] match = EGG_NAME(basename) if match: project_name, version, py_version, platform = match.group( 'name', 'ver', 'pyver', 'plat' ) return cls( location, metadata, project_name=project_name, version=version, py_version=py_version, platform=platform, **kw )._reload_version() And for EggInfo Distribution: class EggInfoDistribution(Distribution): def _reload_version(self): """ Packages installed by distutils (e.g. numpy or scipy), which uses an old safe_version, and so their version numbers can get mangled when converted to filenames (e.g., 1.11.0.dev0+2329eae to 1.11.0.dev0_2329eae). These distributions will not be parsed properly downstream by Distribution and safe_version, so take an extra step and try to get the version number from the metadata file itself instead of the filename. """ md_version = _version_from_file(self._get_metadata(self.PKG_INFO)) if md_version: self._version = md_version return self So, import of pkg_resources "calls" the building of initial/master working set(a collection of importable distributions) with reading of PKG-INFO files. python3-3.6.4-alt1 -> sisyphus: Tue Apr 03 2018 Aleksei Nikiforov <darktemplar@altlinux> 3.6.4-alt1 - Updated to upstream version 3.6.4. Thu Mar 29 2018 Stanislav Levin <slev@altlinux> 3.5.4-alt7 - Add PKG-INFO files to ignore list (closes: #34658). |