Fix data loss in unsigned int cast in PK

This patch introduces some additional checks in the PK module for 64-bit
systems only. The problem is that the API functions in the PK
abstraction accept a size_t value for the hashlen, while the RSA module
accepts an unsigned int for the hashlen. Instead of silently casting
size_t to unsigned int, this change checks whether the hashlen overflows
an unsigned int and returns an error.
3 files changed