Customer reported issues

[embhorn]

Fixes ZD21548

• Fix wolfSSH_ProcessBuffer to check type
• In SendKexDhReply, handle when WOLFSSH_CERTS is not defined
• Fix DoUserAuthRequestRsa() and DoUserAuthRequestRsaCert() to accept ssh-rsa, rsa-sha2-256, and rsa-sha2-512
• Add test case test_wolfSSH_CTX_UsePrivateKey_buffer_pem

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Addresses customer-reported issues by correcting PEM private-key processing, improving compatibility for RSA publickey auth signatures (including RFC6187/X.509 key types), and adding a regression test for loading PEM private keys from buffers.

Changes:

• Update PEM-to-DER conversion to use the correct wolfCrypt API for private keys in wolfSSH_ProcessBuffer.
• Adjust RSA user-auth signature type validation to accept ssh-rsa, rsa-sha2-256, and rsa-sha2-512 when the public key is x509v3-ssh-rsa.
• Add an API test covering wolfSSH_CTX_UsePrivateKey_buffer() with PEM input.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
tests/api.c	Adds a new regression test for loading RSA/ECC PEM private keys via buffer APIs.
src/internal.c	Fixes PEM private-key decoding, relaxes RSA signature type matching for X.509 key types, and tweaks KEXDH reply formatting for RFC6187 host keys.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.