Bug 32199

Summary: require the enclosing dir or corresponding python(.) module
Product: Sisyphus Reporter: Ivan Zakharyaschev <imz>
Component: rpm-build-pythonAssignee: Ivan Zakharyaschev <imz>
Status: NEW --- QA Contact: qa-sisyphus
Severity: enhancement    
Priority: P3 CC: darktemplar, george, imz, ldv, valintinr
Version: unstable   
Hardware: all   
OS: Linux   
Bug Depends on:    
Bug Blocks: 32200    

Description Ivan Zakharyaschev 2016-06-16 20:24:49 MSK
rpm-build-python-0.36.7-alt1

A note: at some places, python.prov.py works by analyzing the buildroot in its entirety, without the division between owning packages:

    if os.path.exists(d+'/__init__.py') and dirname(d) in path :
        b = basename(d)+'/'+b
        d = dirname(d)
        xprov = prov1db(d,b)
        if xprov :
            if prov :
                prov += xprov
            else :
                prov = xprov

Consider the following case, where some important deps are missing. This can be illustrated by this error message:

rpmi: opennode-tui-2.0.1-alt6.git.6af352f 1414685711 installed
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named opennode.cli.actions

$ rpm -qp /ALT/Sisyphus/noarch/RPMS.classic/opennode-tui-2.0.1-alt6.git.6af352f.noarch.rpm --scripts | head -n-1
postinstall scriptlet (through /bin/sh):
/usr/bin/python -c "from opennode.cli.actions import storage; storage.add_pool('local')" ||:
$ 

opennode.cli.actions is to be provided by this package itself:

$ rpm -qp /ALT/Sisyphus/noarch/RPMS.classic/opennode-tui-2.0.1-alt6.git.6af352f.noarch.rpm --provides | head -2
python2.7(opennode.cli)  
python2.7(opennode.cli.actions)  
$ rpm -qp /ALT/Sisyphus/noarch/RPMS.classic/opennode-tui-2.0.1-alt6.git.6af352f.noarch.rpm -l | fgrep __init
/usr/lib/python2.7/site-packages/opennode/cli/__init__.py
/usr/lib/python2.7/site-packages/opennode/cli/__init__.pyc
/usr/lib/python2.7/site-packages/opennode/cli/__init__.pyo
/usr/lib/python2.7/site-packages/opennode/cli/actions/__init__.py
/usr/lib/python2.7/site-packages/opennode/cli/actions/__init__.pyc
/usr/lib/python2.7/site-packages/opennode/cli/actions/__init__.pyo
/usr/lib/python2.7/site-packages/opennode/cli/actions/vm/__init__.py
/usr/lib/python2.7/site-packages/opennode/cli/actions/vm/__init__.pyc
/usr/lib/python2.7/site-packages/opennode/cli/actions/vm/__init__.pyo
/usr/lib/python2.7/site-packages/opennode/cli/tests/__init__.py
/usr/lib/python2.7/site-packages/opennode/cli/tests/__init__.pyc
/usr/lib/python2.7/site-packages/opennode/cli/tests/__init__.pyo
$ 

But it cannot be accessed because opennode/ contains no __init__.py.

opennode/ and opennode/__init__.py belongs to another subpackage (built from the same source package), which opennode-tui doesn't depend on:

[imz@basalt ~]$ rpm -qp /ALT/Sisyphus/noarch/RPMS.classic/python-module-opennode-2.0.1-alt6.git.6af352f.noarch.rpm -l
/usr/lib/python2.7/site-packages/opennode
/usr/lib/python2.7/site-packages/opennode/__init__.py
/usr/lib/python2.7/site-packages/opennode/__init__.pyc
/usr/lib/python2.7/site-packages/opennode/__init__.pyo
[imz@basalt ~]$ rpm -qp /ALT/Sisyphus/noarch/RPMS.classic/python-module-opennode-2.0.1-alt6.git.6af352f.noarch.rpm --provides
python2.7(opennode)  
python-module-opennode = 2.0.1-alt6.git.6af352f
[imz@basalt ~]$ rpm -qp /ALT/Sisyphus/noarch/RPMS.classic/opennode-tui-2.0.1-alt6.git.6af352f.noarch.rpm --requires 
libvirt-client  
libvirt-daemon  
screen  
/bin/sh  
/bin/bash  
/bin/sh  
/usr/bin/env  
/usr/bin/python  
/usr/lib/python2.7/site-packages  
coreutils  
gawk  
grep  
openssh-clients  
python-base  
python-modules  
python2.7(guestfs)  
python2.7(hivex)  
python2.7(libvirt)  
python2.7(logging)  
python2.7(netifaces)  
python2.7(ovf)  
python2.7(progressbar)  
python2.7(snack)  
python2.7(unittest)  
python2.7(xml)  
python2.7(yaml)  
/usr/bin/python  
rpmlib(PayloadIsLzma)  
[imz@basalt ~]$ rpm -qp /ALT/Sisyphus/noarch/RPMS.classic/opennode-tui-2.0.1-alt6.git.6af352f.noarch.rpm --requires | fgrep opennode
[imz@basalt ~]$ 

It would be nice if python code which placed into some subdirs requires these subdirs (and the packages owning them).

In a sense, just generating an autoreq like /usr/lib/python2.7/site-packages/opennode/ is not the best solution, because it may happen that maintainers are not accurate enough and package such subdirs into several packages and hence several packages would (wrongly) satisfy this requirement.

But in case of Python, such a problem seems to have a better solution: automatically require /usr/lib/python2.7/site-packages/opennode/__init__.py (which seems to be needed for this to work) or simply python2.X(opennode) (which looks a bit weird because it is associated with imports, but would imply the wanted dependency).

[do this] ...if there is some Python code inside a subdir. The first variant seems a bit dangerous, because having an __init__.py might be not the only way to make this to work. I don't see any bad consequences of the 2nd solution as for now).
Comment 1 Ivan Zakharyaschev 2016-06-17 12:34:30 MSK
I have a slight suspicion that turning off the optimization of intrapackage deps in python.req.py would mitigate the problem (at least in this case). And it must be a good change overall.

interpackage deps can be left to be treated by rpm-build itself (since 4.0.4-alt100.63), which can do better, because it should treat the division between subpkgs better.

Special attention to autoreqs with "< set-of-paths": they are not going to be optimized by rpm-build, are they? Cf. 4.0.4-alt100.64: "enhanced interdep algorithm further so that manual requirements containing "<" or ">" operators are now left intact,  while all generated requirements on subpackages are now made strict." Yes, IIUC, such autoreqs will be optimized, too, which what we actually want.
Comment 2 Ivan Zakharyaschev 2016-06-17 12:36:13 MSK
(In reply to comment #1)
> I have a slight suspicion that turning off the optimization of intrapackage
> deps in python.req.py would mitigate the problem (at least in this case). And
> it must be a good change overall.
> 
> interpackage deps can be left to be treated by rpm-build itself (since
> 4.0.4-alt100.63), which can do better, because it should treat the division
> between subpkgs better.

I'll postpone the change till the mass rebuild of packages for the substitution of x86_64 checks is over (not to break the submittability to Sisyphus because of new deps and finish this task quicker).