module crash when pkcs#11 provider uses rsa key

[petrovr]

This is specific to interface between PKCS#11 provider and SoftHSM module.

Impacted - PKIX-SSH regression test "pkcs11_prov" when PKCS#11 provider and SoftHSM module are build from current (March 2026) code. Crash with OpenSSL either 3.5* or 3.6* releases (it seem to me is OpenSSL is not related).

Test pass with about one year old provider and module releases .
Also test pkcs11 (direct) and pkcs11_eng(engine) pass.

---

backtrace (dunno why softhsm is not dereferenced):
#0 0x00007f464e746be0 in ?? () from ..../libsofthsm2.so
#1 0x00007f464ee969de in p11prov_SessionCancel (ctx=0x608f0b0, hSession=hSession@entry=12, flags=2048) at src/interface.gen.c:1407
#2 0x00007f464eeb676f in p11prov_sig_freectx (ctx=0x61b2ef0) at src/sig/signature.c:224
#3 0x00007f464ea43fd5 in evp_pkey_ctx_free_old_ops (ctx=0x61865e0) at crypto/evp/pmeth_lib.c:362
#4 0x00007f464ea44109 in EVP_PKEY_CTX_free (ctx=0x61865e0) at crypto/evp/pmeth_lib.c:397
#5 0x00007f464ea0fe99 in evp_md_ctx_reset_ex (keep_fetched=0, ctx=ctx@entry=0x61b4ab0) at crypto/evp/digest.c:86
#6 EVP_MD_CTX_reset (ctx=ctx@entry=0x61b4ab0) at crypto/evp/digest.c:100
#7 0x00007f464ea0febe in EVP_MD_CTX_free (ctx=0x61b4ab0) at crypto/evp/digest.c:139
#8 0x000000000043d21a in ssh_pkey_sign (dgst=dgst@entry=0x7ffd4f01dd00, privkey=, sig=sig@entry=0x0, siglen=siglen@entry=0x7ffd4f01dcf8, data=data@entry=0x61af830 "", datalen=datalen@entry=6196)

src/interface.gen.c:1407
ret = intf->SessionCancel(hSession, flags);

---

PKCS#11 provider section from OpenSSL configuration:

module = ..../pkcs11.so
pkcs11-module-path = ..../libsofthsm2.so
# NOTE: Do not cache session, otherwise "digest sign update" will fail!
pkcs11-module-cache-sessions = 0
activate = 1

Remark: not tested yet if issue with cached session is resolved

[petrovr]

For protocol all SoftHSM regression tests pass.

[bukka]

This will need more info. Please can you add some more detailed steps how to recreate it?

It would be also good to add debug symbols to your SoftHSMv2 build which needs to be done using CXXFLAGS. This is how I configure it:

CFLAGS="-g -O0" CXXFLAGS="-g -O0" LDFLAGS="-L/usr/local/ssl/lib64 -Wl,-rpath,/usr/local/ssl/lib64" ./configure --with-openssl=/usr/local/ssl --disable-openssl-engines

if it doesn't crash with -O0, you can remove it and use standard optimization - what matters for backtrace is -g ...

[petrovr]

It was build with debug '-g -O2 -fno-strict-aliasing -Wall' fir C and C++ flags, but dunno why backtrace does not show symbols.

Let me try without optimisation.

[petrovr]

Compiler options are not related.

This is the pkcs#11 provider log when functionality is tested:

[<stamp>] [.../pkcs11-provider-current/src/sig/rsasig.c:635] p11prov_rsasig_digest_sign_init(): rsa digest sign init (ctx=0x55cededf2660, digest=SHA256, key=0x55cededf1930, params=(nil))
[<stamp>] [.../pkcs11-provider-current/src/obj/object.c:307] p11prov_obj_ref_no_cache(): Ref Object: 0x55cededf1930 (handle:6)
[<stamp>] [.../pkcs11-provider-current/src/provider.c:630] p11prov_ctx_cache_keys(): cache_keys = 1
[<stamp>] [.../pkcs11-provider-current/src/session.c:1198] p11prov_take_login_session(): Get login session from slot 229809896
[<stamp>] [.../pkcs11-provider-current/src/session.c:877] slot_login(): Slot login (slot=0x55cededc71a0, uri=(nil))
[<stamp>] [.../pkcs11-provider-current/src/session.c:338] session_check(): Checking session 3
[<stamp>] [.../pkcs11-provider-current/src/interface.gen.c:358] p11prov_GetSessionInfo(): Calling C_GetSessionInfo
[<stamp>] [.../pkcs11-provider-current/src/interface.gen.c:508] p11prov_CopyObject(): Calling C_CopyObject
[<stamp>] [.../pkcs11-provider-current/src/obj/object.c:115] cache_key(): Key 229809896:6 cached as 3:11

[<stamp>] [.../pkcs11-provider-current/src/sig/rsasig.c:1055] p11prov_rsasig_set_ctx_params(): rsasig set ctx params (ctx=0x55cededf2660, params=(nil))

[<stamp>] [.../pkcs11-provider-current/src/sig/rsasig.c:659] p11prov_rsasig_digest_sign_update(): rsa digest sign update (ctx=0x55cededf2660, data=0x55cededdd220, datalen=377)

[<stamp>] [.../pkcs11-provider-current/src/sig/signature.c:375] p11prov_sig_operate_init(): called (sigctx=0x55cededf2660, digest_op=true)
[<stamp>] [.../pkcs11-provider-current/src/interface.gen.c:274] p11prov_GetMechanismInfo(): Calling C_GetMechanismInfo
[<stamp>] Mechanism Info:
  name: CKM_SHA256_RSA_PKCS (64):
  min key length: 512
  max key length: 16384
  flags (0x002800):

[<stamp>]     CKF_SIGN                  (0x000800)
[<stamp>]     CKF_VERIFY                (0x002000)
[<stamp>] [.../pkcs11-provider-current/src/session.c:981] p11prov_get_session(): Get session on slot 229809896, reqlogin=true, rw=false
[<stamp>] [.../pkcs11-provider-current/src/session.c:989] p11prov_get_session(): single-shot request for slot 229809896
[<stamp>] [.../pkcs11-provider-current/src/session.c:667] check_slot(): Checking Slot id=229809896, uri=0x55cededf14e0, mechtype=40, rw=false)
[<stamp>] [.../pkcs11-provider-current/src/provider.c:618] p11prov_ctx_login_behavior(): login_behavior = 0
[<stamp>] [.../pkcs11-provider-current/src/session.c:877] slot_login(): Slot login (slot=0x55cededc71a0, uri=0x55cededf14e0)
[<stamp>] [.../pkcs11-provider-current/src/session.c:338] session_check(): Checking session 3
[<stamp>] [.../pkcs11-provider-current/src/interface.gen.c:358] p11prov_GetSessionInfo(): Calling C_GetSessionInfo
[<stamp>] [.../pkcs11-provider-current/src/session.c:333] session_check(): Checked session 0 handle invalid
[<stamp>] [.../pkcs11-provider-current/src/interface.gen.c:303] p11prov_OpenSession(): Calling C_OpenSession
[<stamp>] [.../pkcs11-provider-current/src/session.c:82] token_session_open(): C_OpenSession ret:0 (session: 12)
[<stamp>] [.../pkcs11-provider-current/src/interface.gen.c:358] p11prov_GetSessionInfo(): Calling C_GetSessionInfo
[<stamp>] [.../pkcs11-provider-current/src/interface.gen.c:1029] p11prov_SignInit(): Calling C_SignInit
[<stamp>] [.../pkcs11-provider-current/src/interface.gen.c:1086] p11prov_SignUpdate(): Calling C_SignUpdate

[<stamp>] [.../pkcs11-provider-current/src/sig/rsasig.c:690] p11prov_rsasig_digest_sign_final(): rsa digest sign final (ctx=0x55cededf2660, sig=(nil), siglen=0, sigsize=0)

????
[<stamp>] [.../pkcs11-provider-current/src/interface.gen.c:1029] p11prov_SignInit(): Calling C_SignInit
[<stamp>] [.../pkcs11-provider-current/src/interface.gen.c:1034] p11prov_SignInit(): Error 7 returned by C_SignInit
[<stamp>] [.../pkcs11-provider-current/src/interface.gen.c:1406] p11prov_SessionCancel(): Calling C_SessionCancel

Note rsa calls are separated by new line.

ssh_pkey_sign (dgst=dgst@entry=0x7ffd4f01dd00, privkey=, sig=sig@entry=0x0, siglen=siglen@entry=0x7ffd4f01dcf8, data=data@entry=0x61af830 "", datalen=datalen@entry=6196)

The call p11prov_rsasig_digest_sign_final() is with NULL argument for signature. This is expected as from "top" level call in PKIX-SSH code. The function return signature length to "top" call.

EVP_MD_CTX_free (ctx=0x61b4ab0) at crypto/evp/digest.c:139
Now the "context" is released and this causes a crash.

Remark: "direct" and "engine" test pass for RSA. No issue with EC, Ed, ML-DSA(*!) keys and certificates.

I have no idea how to debug/trace which part of a shared library is causing crash.

[bukka]

Strange it should be usual there. The system one doesn't have it btw. I can always get debug symbols only in my built though

$ readelf -S /usr/local/lib/softhsm/libsofthsm2.so | grep debug
  [29] .debug_aranges    PROGBITS         0000000000000000  0015a105
  [30] .debug_info       PROGBITS         0000000000000000  0016f875
  [31] .debug_abbrev     PROGBITS         0000000000000000  004698b2
  [32] .debug_line       PROGBITS         0000000000000000  004ad0e8
  [33] .debug_str        PROGBITS         0000000000000000  0051a991
  [34] .debug_line_str   PROGBITS         0000000000000000  005f22de
  [35] .debug_rnglists   PROGBITS         0000000000000000  005f360f
$ readelf -S /usr/lib/softhsm/libsofthsm2.so | grep debug
  [27] .gnu_debuglink    PROGBITS         0000000000000000  000e1340

I can see that it's failing during session cancel for CKM_SHA256_RSA_PKCS so I might try and see if anything is failing for me too.

[petrovr]

I guess that gdb bt does not show symbols as stack/memory is corrupted.

system)
# LANG=C nm /usr/.../libsofthsm2.so
nm: libsofthsm2.so: no symbols

custom)
# nm /opt/.../libsofthsm2.so
....
00000000000f73a8 r ZZNK7OSSLAES9getCipherEvE8__func_
00000000000f7870 r ZZNK7OSSLDES9getCipherEvE8__func_

---

Also
(gdb) info functions SoftHSM::
All functions matching regular expression "SoftHSM::":

File ..../src/lib/SoftHSM.cpp:
3212: CK_RV SoftHSM::AsymDecryptInit(unsigned long, _CK_MECHANISM*, unsigned long);
....

[petrovr]

Tests with build from "develop" branch pass.

The error appears to be related to an uninitialized pointer.