tree 3bc4ff5c642e5270894c6fd0c76205fe905341dd
parent 489688c0f7eddf57334d1d7e8ad0f03ba6bf82d0
author Andre Goddard Rosa <agoddardrosa@roku.com> 1714582032 -0500
committer Andre Goddard Rosa <agoddardrosa@roku.com> 1714582934 -0500

Add invalid `padding_len` check in `get_pkcs_padding`

When trying to decrypt data with an invalid key, we found that `mbedtls`
returned `0x6200` (`-25088`), which means "_CIPHER - Input data contains
invalid padding and is rejected_" from `mbedtls_cipher_finish`, but it also
set the output len as `18446744073709551516`.

In case we detect an error with padding, we leave the output len zero'ed
and return `MBEDTLS_ERR_CIPHER_INVALID_PADDING`. I believe that the current
test cases are sufficient, as they fail if I return the alternative code
`MBEDTLS_ERR_CIPHER_BAD_INPUT_DATA`, so they do already expect a padding
failure, but now we don't change the output len in the error case.

Here's a reference for the way `openssl` checks the padding length:
  - https://github.com/openssl/openssl/blob/1848c561ec39a9ea91ff1bf740a554be274f98b0/crypto/evp/evp_enc.c#L1023
  - https://github.com/openssl/openssl/commit/b554eef43b9ac5b92f590da6a120dbfd9ca0582e

Signed-off-by: Andre Goddard Rosa <andre.goddard@gmail.com>
Signed-off-by: Andre Goddard Rosa <agoddardrosa@roku.com>