ALT Linux Bugzilla – #34658

(In reply to comment #0)

> We used to have 2 patches to fight this spoiling:
> 
> https://packages.altlinux.org/en/p8/srpms/python-module-setuptools/patches/0001-command-test.py-skip-install_requires-and-tests_requ.patch
> https://packages.altlinux.org/en/p8/srpms/python-module-setuptools/patches/0002-dist.py-skip-checking-the-existence-of-system-PKG-IN.patch
> 
> In Sisyphus, only the second one seems to be applied.
> 
> Some fix in the same spirit is needed. (The detected used files can be seen by
> running filereq FILELIST rpm -bc *.spec.)

grenka@ and me know how to re-implement the first patch. Coming soon...

python-module-setuptools-1:38.4.0-alt3 -> sisyphus:

Thu Mar 22 2018 Grigory Ustinov <grenka@altlinux> 1:38.4.0-alt3
- Add patch for skipping some requires from glebfm@, adapted for new code.
  It restores feature from commit 362ea68 in 1:18.1-alt2 release. (Closes: #34658)

(In reply to comment #0)

> We used to have 2 patches to fight this spoiling:
> 
> https://packages.altlinux.org/en/p8/srpms/python-module-setuptools/patches/0001-command-test.py-skip-install_requires-and-tests_requ.patch
> https://packages.altlinux.org/en/p8/srpms/python-module-setuptools/patches/0002-dist.py-skip-checking-the-existence-of-system-PKG-IN.patch

slev@: патч не имеет отношения к этой проблеме.

Но хорошо, что его вернули.

Вспомнил, зачем он был нужен:

Это было когда-то Глебом сделано вот ради чего: buildreq не записывает (и правильно делает) многие вещи, которые не используются во время прогона тестов в %%check в BuildRequires, а при попытке запуска тестов (какого0то пакета) без этих вещей в сборочной среде,
setuptools проверяет их начилие (просто так -- ради проверки, а не ради использования) и может случиться ошибка (он попытается скачать).

 
> In Sisyphus, only the second one seems to be applied.
> 
> Some fix in the same spirit is needed. (The detected used files can be seen by
> running filereq FILELIST rpm -bc *.spec.)

Let me try to fix this issue.

python setuptools during first import of pkg_resources builds the initial/master working set(a collection of importable distributions) from sys.path. For each egg info distribution the read of metadata is applied to read the distribution version.

I guess we should ignore these readings while building filereq list.
Please check the following patches:
python2 http://git.altlinux.org/people/slev/packages/?p=python.git;a=commitdiff;h=f788b33a1b2eebea7fb98b80ac6eea7610235154

python3 http://git.altlinux.org/people/slev/packages/?p=python3.git;a=commitdiff;h=cac7cb9d9d3a86fe9b2e6b1acc9046452b6c67d5

Test results:
1) python3
packagereq: optimized out: python-base python-modules python3 python3-base
packagereq: BuildRequires: libenchant python3-dev python3-module-setuptools

2) python
packagereq: optimized out: python-base python-devel python-modules python-modules-compiler python-modules-ctypes python-modules-email python-modules-encodings
packagereq: BuildRequires: libenchant python-module-setuptools python-module-sphinxcontrib

Waiting for your comments...

(In reply to comment #5)
> python setuptools during first import of pkg_resources builds the
> initial/master working set(a collection of importable distributions) from
> sys.path. For each egg info distribution the read of metadata is applied to
> read the distribution version.
> 
> I guess we should ignore these readings while building filereq list.
> Please check the following patches:
> python2
> http://git.altlinux.org/people/slev/packages/?p=python.git;a=commitdiff;h=f788b33a1b2eebea7fb98b80ac6eea7610235154
> 
> python3
> http://git.altlinux.org/people/slev/packages/?p=python3.git;a=commitdiff;h=cac7cb9d9d3a86fe9b2e6b1acc9046452b6c67d5
> 
> Test results:
> 1) python3
> packagereq: optimized out: python-base python-modules python3 python3-base
> packagereq: BuildRequires: libenchant python3-dev python3-module-setuptools
> 
> 2) python
> packagereq: optimized out: python-base python-devel python-modules
> python-modules-compiler python-modules-ctypes python-modules-email
> python-modules-encodings
> packagereq: BuildRequires: libenchant python-module-setuptools
> python-module-sphinxcontrib
> 
> Waiting for your comments...

Simple comment: it's better to change the release & changelog in a separate commit (so that it is easier to cherry-pick your essential changes).

Another not so important, historic comment: As for the presence of .pth in this regex in python{,3} package, it is justified by that .pth are interpreted by python itself, they affect python. As for the egg-info things such as entry_points, their presence in python{,3} packages seems not to be very justified because they are processed by setuptools/distutils, and not python{,3}. But I think that it's OK, anyway, the egg-info is tightly related with standardized Python infrastructure.

As for the real reason: it interesting to know better what happens. In older versions (e.g., 18.1-alt4 932975fde88184790eaacbd7bc0a65ddd081bde7), the following patch must have been enough not to read PKG-INFO: https://packages.altlinux.org/en/p8/srpms/python-module-setuptools/patches/0002-dist.py-skip-checking-the-existence-of-system-PKG-IN.patch ? Does reading 
PKG-INFO happens more times in setuptools now? Is this information really needed if BuildRequires and Requires must completely describe the requirements?

I don't understand yet the answer, and can't find the place in the new code where the additional reads happen.

Created attachment 7483 [details]
setuptools-18.1-alt4.bt

Created attachment 7484 [details]
setuptools-39.0.1.bt

The key point here is classmethod "from_location" (pkg_resources/__init__.py).
Please check an attached backtraces for details.

setuptools 18.1: 

    @classmethod
    def from_location(cls, location, basename, metadata=None,**kw):
        project_name, version, py_version, platform = [None]*4
        basename, ext = os.path.splitext(basename)
        if ext.lower() in _distributionImpl:
            # .dist-info gets much metadata differently
            match = EGG_NAME(basename)
            if match:
                project_name, version, py_version, platform = match.group(
                    'name','ver','pyver','plat'
                )
            cls = _distributionImpl[ext.lower()]
        return cls(
            location, metadata, project_name=project_name, version=version,
            py_version=py_version, platform=platform, **kw
        )

But setuptools 39:

    @classmethod
    def from_location(cls, location, basename, metadata=None, **kw):
        project_name, version, py_version, platform = [None] * 4
        basename, ext = os.path.splitext(basename)
        if ext.lower() in _distributionImpl:
            cls = _distributionImpl[ext.lower()]

            match = EGG_NAME(basename)
            if match:
                project_name, version, py_version, platform = match.group(
                    'name', 'ver', 'pyver', 'plat'
                )
        return cls( 
            location, metadata, project_name=project_name, version=version,
            py_version=py_version, platform=platform, **kw 
        )._reload_version()

And for EggInfo Distribution:

class EggInfoDistribution(Distribution):
    def _reload_version(self):
        """
        Packages installed by distutils (e.g. numpy or scipy),
        which uses an old safe_version, and so
        their version numbers can get mangled when
        converted to filenames (e.g., 1.11.0.dev0+2329eae to
        1.11.0.dev0_2329eae). These distributions will not be
        parsed properly
        downstream by Distribution and safe_version, so
        take an extra step and try to get the version number from
        the metadata file itself instead of the filename.
        """
        md_version = _version_from_file(self._get_metadata(self.PKG_INFO))
        if md_version:
            self._version = md_version
        return self

So, import of pkg_resources "calls" the building of initial/master working set(a collection of importable distributions) with reading of PKG-INFO files.

python3-3.6.4-alt1 -> sisyphus:

Tue Apr 03 2018 Aleksei Nikiforov <darktemplar@altlinux> 3.6.4-alt1
- Updated to upstream version 3.6.4.

Thu Mar 29 2018 Stanislav Levin <slev@altlinux> 3.5.4-alt7
- Add PKG-INFO files to ignore list (closes: #34658).