CryptDecryptMessage fails, returning an CRYPT_E_ASN1_MEMORY error value

CryptDecryptMessage fails, returning an CRYPT_E_ASN1_MEMORY error value


Other Resources
Security Developer Center
Cryptography Topics on MSDN
Follow us on Twitter

Code that called the CryptDecryptMessage function successfully under previous versions of Windows may fail with a CRYPT_E_ASN1_MEMORY error under Windows 7.

Cause

There is a counter in the ASN structure that is increased every time an octet block is processed when the ASN sequence that wraps the octets is BER encoded.  It adds the number of bytes that have already been processed plus the new block size; as the number of processed bytes grows that is added to the counter and the rate at which the counter grows increases. Since the counter is represented by an integer it can overflow.  On Windows 7, the ASN code checks for overflow so the function fails.  On pre-Windows 7 platforms, the overflow still occurs but it wasn't checked.  That counter is not critical for operation and did not cause issues in the code for the previous versions of the OS.

Resolution

 Download the hotfix described in Microsoft Knowledge Base article 2480994, "ASN1 out of memory" error when the CryptMsgUpdate function decodes a message that is larger than 5 MB in Windows 7 or in Windows Server 2008 R2 (http://support.microsoft.com/kb/2480994).
You can also call the lower level message functions (CryptMsgOpenToDecode, CryptMsgUpdate, CryptMsgControl) to process BER encoded content. 


See Also

Leave a Comment
  • Please add 2 and 7 and type the answer here:
  • Post
Wiki - Revision Comment List(Revision Comment)
Sort by: Published Date | Most Recent | Most Useful
Comments
  • Carsten Siemens edited Revision 3. Comment: Added tags: en-US, has comment

  • Jewel Lambert edited Revision 2. Comment: Corrected spelling typo

  • Ed Price MSFT edited Original. Comment: Added "See Also" section.

Page 1 of 1 (3 items)
Wikis - Comment List
Sort by: Published Date | Most Recent | Most Useful
Posting comments is temporarily disabled until 10:00am PST on Saturday, December 14th. Thank you for your patience.
Comments
  • Ed Price MSFT edited Original. Comment: Added "See Also" section.

  • Hi, do you know if Microsoft will release a fix for this? I have been struggling with the error for quite some time, and I have not been able to successfully implement a solution using the lower level message functions.

  • Jewel Lambert edited Revision 2. Comment: Corrected spelling typo

  • Good Article.

  • Carsten Siemens edited Revision 3. Comment: Added tags: en-US, has comment

Page 1 of 1 (5 items)