Add verify-only LMS and XMSS support

[Frauschi]

Summary

Adds PKCS#11 verify-only support for the two NIST-approved stateful hash-based signature (SHBS) families:

• LMS / HSS (RFC 8554, NIST SP 800-208)
• XMSS / XMSS^MT (RFC 8391, NIST SP 800-208)

Signature verification and raw public-key import are implemented on top of wolfSSL's wc_LmsKey_* / wc_XmssKey_* APIs, using the standard OASIS PKCS#11 v3.3 mechanism, key-type, and attribute constants. Everything is gated behind the new --enable-lms and --enable-xmss build options and is off by default.

This is step 1 (verify only). Signing, key generation, and private-key import are deliberately out of scope and are rejected at runtime; the object model and store layout are laid out so a future sign-capable step can add private keys without an interface break.

What's included

Interface (standard PKCS#11 v3.3)

Category	Constants
Key types	CKK_HSS (0x46), CKK_XMSS (0x47), CKK_XMSSMT (0x48)
Mechanisms	CKM_HSS (0x4033), CKM_XMSS (0x4036), CKM_XMSSMT (0x4037)
Attributes	CKA_HSS_LEVELS, CKA_HSS_LMS_TYPE(S), CKA_HSS_LMOTS_TYPE(S), CKA_HSS_KEYS_REMAINING, CKA_PARAMETER_SET (XMSS)

All values match the OASIS PKCS#11 v3.3 working headers exactly (not vendor-defined).

Behaviour

• Public-key import — a public key is an ordinary token or session object built from the raw RFC 8554 / RFC 8391 key in CKA_VALUE. wolfSSL derives the parameter set from the key bytes.
• Parameter validation — supplied CKA_HSS_* parameter attributes and the XMSS CKA_PARAMETER_SET OID are validated against the imported key; a mismatch, a malformed key, or a wrong-length key is rejected with CKR_ATTRIBUTE_VALUE_INVALID.
• Attribute read-back — the HSS parameter attributes and the XMSS CKA_PARAMETER_SET are exposed on read. CKA_HSS_KEYS_REMAINING is a private-key state property and is reported as unavailable in a verify-only build.
• C_Verify — validates the mechanism against the key type (CKR_KEY_TYPE_INCONSISTENT on a mismatch) and reports a wrong-length signature (both schemes) or an empty message (XMSS only, a wolfSSL backend limitation) as an invalid signature rather than a function failure.
• Persistence — public keys are stored and reload from disk via new store types WOLFPKCS11_STORE_HSSKEY_PUB (0x11) and WOLFPKCS11_STORE_XMSSKEY_PUB (0x13), with 0x10 / 0x12 reserved for future private keys.

Verify-only constraints

• Key generation (CKM_*_KEY_PAIR_GEN) and private-key import are rejected.
• Copying any HBS key is prohibited, so that a future sign-capable build can never duplicate one-time-signature state.

Build

# wolfSSL must be built with LMS and XMSS enabled:
./configure --enable-lms --enable-xmss ...
make && sudo make install

# wolfPKCS11:
./configure --enable-lms --enable-xmss
make && make check

Testing

New coverage in tests/pkcs11v3test.c and a dedicated tests/hbs_persistence_test.c:

• Known-answer verification for HSS, XMSS, and XMSS^MT (valid, tampered, and wrong-length signatures).
• Parameter-set validation (matching and mismatching CKA_HSS_* / CKA_PARAMETER_SET).
• Attribute read-back, including the per-level array attributes and the unavailable CKA_HSS_KEYS_REMAINING.
• Token-object persistence across a full library reload.
• Negative cases: private-key import rejected, key generation rejected, wrong mechanism/key-type pairings, empty templates, and malformed, wrong-length, and structurally invalid public keys.

All tests pass; the key-lifecycle paths (import, atomic re-import, free) were additionally checked clean under AddressSanitizer, including a --disable-rsa --enable-lms build.

Notes and limitations

• wolfSSL uses uniform LMS/LMOTS parameters across all HSS levels, so the per-level array attributes (CKA_HSS_LMS_TYPES / CKA_HSS_LMOTS_TYPES) report the single top-level type for every level. Heterogeneous HSS keys are not expressible today.
• The spec defines no XMSS "keys remaining" attribute; XMSS exposes only CKA_PARAMETER_SET.
• The CKL_LMS_* / CKL_LMOTS_* convenience macros are wolfPKCS11-local (the spec header defines none); their values are the RFC 8554 registry codes.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #206

Scan targets checked: wolfpkcs11-bugs, wolfpkcs11-src

Findings: 2
2 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #206

Scan targets checked: wolfpkcs11-bugs, wolfpkcs11-src

Findings: 3
3 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[Frauschi]

The failing CMake Build Test will be resolved once wolfSSL/wolfssl#10929 lands.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #206

Scan targets checked: wolfpkcs11-bugs, wolfpkcs11-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Findings are non-blocking.

[philljj]

the CMake Build Tests is failing:

In file included from /home/runner/work/wolfPKCS11/wolfPKCS11/src/internal.c:76:
/home/runner/work/wolfPKCS11/wolfPKCS11/wolfpkcs11/internal.h:133:2: error: #error Compiling with XMSS requires XMSS support in wolfSSL (--enable-xmss).
  133 | #error Compiling with XMSS requires XMSS support in wolfSSL (--enable-xmss).
      |  ^~~~~

edit: is a separate issue: #206 (comment)