Нарвался на программную ошибку переполнения при попытке шифрования крупного файла в гибридном режиме с помощью openssl. Проверял на ПК и ноутбуке, воспроизводится.

Ниже порядок для воспроизведения ошибки на примере iso-файла ALT Server:

~/WorkDit ❯ rpm -qa | grep openssl                                                                                                         
openssl-config-3.2.0-alt1.noarch
openssl-3.3.3-alt1.x86_64
libxmlsec1-openssl-1.2.38-alt1.x86_64
openssl-gost-engine-3.0.2-alt5.x86_64
openssl-engines-3.3.3-alt1.x86_64   

~/WorkDit ❯ du -sh alt-server-11.0-x86_64.iso 
4,2G	alt-server-11.0-x86_64.iso

~/WorkDit ❯ openssl genrsa -out private.key 4096    

~/WorkDit ❯ openssl req -x509 -new -key private.key -sha512 -days 365 -out cert.crt         
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [RU]:
State or Province Name (full name) []:
Locality Name (eg, city) []:
Organization Name (eg, company) []:Test Cert
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:test@test.com

~/WorkDit ❯ openssl x509 -in cert.crt -noout -text                                                                                        
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            1b:e5:96:92:93:aa:4d:31:f2:bd:31:7a:a1:b7:96:1c:19:c4:a4:1a
        Signature Algorithm: sha512WithRSAEncryption
        Issuer: C=RU, O=Test Cert, emailAddress=test@test.com
        Validity
            Not Before: Sep 25 07:00:01 2025 GMT
            Not After : Sep 25 07:00:01 2026 GMT
        Subject: C=RU, O=Test Cert, emailAddress=test@test.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
...

~/WorkDit ❯ ls
alt-server-11.0-x86_64.iso  cert.crt  private.key

~/WorkDit ❯ openssl smime -encrypt -aes256 -in alt-server-11.0-x86_64.iso -binary -out alt-server-11.0-x86_64.iso.enc cert.crt
Error creating PKCS#7 structure
000E02689F7F0000:error:03880106:memory buffer routines:BUF_MEM_grow_clean:passed invalid argument:crypto/buffer/buffer.c:125:
000E02689F7F0000:error:03880106:memory buffer routines:BUF_MEM_grow_clean:passed invalid argument:crypto/buffer/buffer.c:125:
000E02689F7F0000:error:03880106:memory buffer routines:BUF_MEM_grow_clean:passed invalid argument:crypto/buffer/buffer.c:125:
000E02689F7F0000:error:03880106:memory buffer routines:BUF_MEM_grow_clean:passed invalid argument:crypto/buffer/buffer.c:125:
000E02689F7F0000:error:03880106:memory buffer routines:BUF_MEM_grow_clean:passed invalid argument:crypto/buffer/buffer.c:125:
000E02689F7F0000:error:03880106:memory buffer routines:BUF_MEM_grow_clean:passed invalid argument:crypto/buffer/buffer.c:125:
000E02689F7F0000:error:03880106:memory buffer routines:BUF_MEM_grow_clean:passed invalid argument:crypto/buffer/buffer.c:125:
000E02689F7F0000:error:03880106:memory buffer routines:BUF_MEM_grow_clean:passed invalid argument:crypto/buffer/buffer.c:125:
000E02689F7F0000:error:03880106:memory buffer routines:BUF_MEM_grow_clean:passed invalid argument:crypto/buffer/buffer.c:125:
000E02689F7F0000:error:03880106:memory buffer routines:BUF_MEM_grow_clean:passed invalid argument:crypto/buffer/buffer.c:125:
000E02689F7F0000:error:03880106:memory buffer routines:BUF_MEM_grow_clean:passed invalid argument:crypto/buffer/buffer.c:125:
000E02689F7F0000:error:03880106:memory buffer routines:BUF_MEM_grow_clean:passed invalid argument:crypto/buffer/buffer.c:125:
000E02689F7F0000:error:03880106:memory buffer routines:BUF_MEM_grow_clean:passed invalid argument:crypto/buffer/buffer.c:125:
000E02689F7F0000:error:03880106:memory buffer routines:BUF_MEM_grow_clean:passed invalid argument:crypto/buffer/buffer.c:125:
000E02689F7F0000:error:03880106:memory buffer routines:BUF_MEM_grow_clean:passed invalid argument:crypto/buffer/buffer.c:125: