Fix undefined behavior in unsigned-to-signed conversion
The code assumed that `int x = - (unsigned) u` with 0 <= u < INT_MAX
sets `x` to the negative of u, but actually this calculates
(UINT_MAX - u) and then converts this value to int, which overflows.
Cast to int before applying the unary minus operator to guarantee the
desired behavior.
diff --git a/library/rsa.c b/library/rsa.c
index 7bcc751..4b3cc02 100644
--- a/library/rsa.c
+++ b/library/rsa.c
@@ -1564,9 +1564,9 @@
* - OUTPUT_TOO_LARGE if the padding is good but the decrypted
* plaintext does not fit in the output buffer.
* - 0 if the padding is correct. */
- ret = - if_int( bad, - MBEDTLS_ERR_RSA_INVALID_PADDING,
- if_int( output_too_large, - MBEDTLS_ERR_RSA_OUTPUT_TOO_LARGE,
- 0 ) );
+ ret = - (int) if_int( bad, - MBEDTLS_ERR_RSA_INVALID_PADDING,
+ if_int( output_too_large, - MBEDTLS_ERR_RSA_OUTPUT_TOO_LARGE,
+ 0 ) );
/* If the padding is bad or the plaintext is too large, zero the
* data that we're about to copy to the output buffer.