40249 2021-06-21 02:06:55 +0300 gb: parallelize gb-task-check-install-arch for speed 2024-04-01 17:17:35 +0300 1 1 2 Infrastructure Infrastructure girar unspecified all Linux NEW P5 enhancement --- 1 ldv placeholder glebfm iv ldv mike sin cas oldest_to_newest 199328 0 ldv 2021-06-21 02:06:55 +0300 When a task built more than just a few binary packages, it makes sense to parallelize gb-task-check-install-arch because the only thing that could be shared between install checks is the cache. 203785 1 iv 2021-10-13 21:49:04 +0300 I have a rough implementation of parallel install checks that currently works on mipsel (secondary) and riscv64 girar instances. It uses the same build node and user and runs install checks in parallel using several hasher instances (with different --number). For 4-core Loongson3A I run 5 install check threads, and it gives from 1.6 to 3 times speed up, depending on the task. Also, for trivial tasks where only one binary package is build parallel version may be slower by a second or so. I'm not posting links to code here mainly because I believe that it will be more effective to write something from scratch then to forward-port & clean up my implementation onto current girar (I'm still using an ancient pre-ga version). Still, I can write smth up on what and how I was doing, if you think it'll be useful. 205037 2 ldv 2021-11-19 03:20:56 +0300 (In reply to Dmitry V. Levin from comment #0) > When a task built more than just a few binary packages, it makes sense to > parallelize gb-task-check-install-arch because the only thing that could be > shared between install checks is the cache. Since the cache is so important for install checks, it would make sense to generate the cache once and subsequently reuse it by parallelized install checks. 243866 3 iv 2024-04-01 17:17:35 +0300 The next iteration of the parallelized install checks were deployed on loongarch64 girar instance last monday (2024-03-25). It works stably so far. We use 8 workers, as more install chroots would probably exhaust the free space at $TMPDIR. On heavy tasks with enough binary packages we see install checks became 3.5-6.5 times faster, the heavier the task is the better. On small tasks (e.g. updating one small noarch package), the install checks became slower for a few seconds, as some cache-related optimizations are now missing. Nevertheless, even now this seems like an good trade-off, and the optimization can be re-implemented in the future.