ZSTD_insertBtAndGetAllMatches assertion failed #4610
Description
Describe the bug
I have a huge file (1300MB) and zstd hits assertion failure randomly when compress that file.
To Reproduce
Steps to reproduce the behavior:
- Downloads data https://kcwu.csie.org/~kcwu/tmp/zstd-crash.o5m.zst and decompress it. The sha256sum of the o5m file is
90746b4adaf1492cb73f4aae80917f32a98be41cb56e09e3b4f5d719d0494726. - Run 'zstd -19 -T0 -f -k -q zstd-crash.o5m' repeatedly.
- See error
zstd: ../lib/compress/zstd_opt.c:722: ZSTD_insertBtAndGetAllMatches: Assertion `memcmp(match, ip, matchLength) == 0' failed.
Caught SIGABRT signal, printing stack:
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c) [0x7e840202db2c]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e) [0x7e8401fd427e]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xdf) [0x7e8401fb78ff]
/lib/x86_64-linux-gnu/libc.so.6(+0x2881b) [0x7e8401fb781b]
/lib/x86_64-linux-gnu/libc.so.6(+0x3b517) [0x7e8401fca517]
zstd(+0x54e7b) [0x5f6d40700e7b]
zstd(+0x5f839) [0x5f6d4070b839]
zstd(+0x182a1) [0x5f6d406c42a1]
zstd(+0x1aae6) [0x5f6d406c6ae6]
zstd(+0x1bda9) [0x5f6d406c7da9]
zstd(+0x6239d) [0x5f6d4070e39d]
zstd(+0x9eb8) [0x5f6d406b5eb8]
/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4) [0x7e840202baa4]
/lib/x86_64-linux-gnu/libc.so.6(+0x129c6c) [0x7e84020b8c6c]
I tested the same command repeatedly (with cool down between runs). zstd crashed 3 out of 100 times.
Expected behavior
zstd should not crash.
Desktop (please complete the following information):
- OS: ubuntu-26.04
- Other relevant hardware specs: AMD Ryzen 7 3700X 8-Core Processor. total memory 128GB. free memory is 36GB when zstd runs.
- zstd is installed from ubuntu package. version:
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***
Additional context
- the file is a static file. No other processes touch the file when zstd compress.
- I stress tested the system and it is stable (memory is good). zstd won't crash with other similar files (repeated run two days).