Bug 40249 - gb: parallelize gb-task-check-install-arch for speed
Summary: gb: parallelize gb-task-check-install-arch for speed
Status: NEW
Alias: None
Product: Infrastructure
Classification: Infrastructure
Component: girar (show other bugs)
Version: unspecified
Hardware: all Linux
: P5 enhancement
Assignee: placeholder@altlinux.org
QA Contact: Andrey Cherepanov
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-06-21 02:06 MSK by Dmitry V. Levin
Modified: 2024-04-01 17:17 MSK (History)
5 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Dmitry V. Levin 2021-06-21 02:06:55 MSK
When a task built more than just a few binary packages, it makes sense to parallelize gb-task-check-install-arch because the only thing that could be shared between install checks is the cache.
Comment 1 Ivan A. Melnikov 2021-10-13 21:49:04 MSK
I have a rough implementation of parallel install checks that currently works on  mipsel (secondary) and riscv64 girar instances. It uses the same build node and user and runs install checks in parallel using several hasher instances (with different --number). For 4-core Loongson3A I run 5 install check threads, and it gives from 1.6 to 3 times speed up, depending on the task. Also, for trivial tasks where only one binary package is build parallel version may be slower by a second or so.

I'm not posting links to code here mainly because I believe that it will be more effective to write something from scratch then to forward-port & clean up my implementation onto current girar (I'm still using an ancient pre-ga version). Still, I can write smth up on what and how I was doing, if you think it'll be useful.
Comment 2 Dmitry V. Levin 2021-11-19 03:20:56 MSK
(In reply to Dmitry V. Levin from comment #0)
> When a task built more than just a few binary packages, it makes sense to
> parallelize gb-task-check-install-arch because the only thing that could be
> shared between install checks is the cache.

Since the cache is so important for install checks, it would make sense to generate the cache once and subsequently reuse it by parallelized install checks.
Comment 3 Ivan A. Melnikov 2024-04-01 17:17:35 MSK
The next iteration of the parallelized install checks were deployed on loongarch64 girar instance last monday (2024-03-25). It works stably so far. We use 8 workers, as more install chroots would probably exhaust the free space at $TMPDIR. On heavy tasks with enough binary packages we see install checks became 3.5-6.5 times faster, the heavier the task is the better. On small tasks (e.g. updating one small noarch package), the install checks became slower for a few seconds, as some cache-related optimizations are now missing. Nevertheless, even now this seems like an good trade-off, and the optimization can be re-implemented in the future.