Segmentation fault in EVP_PKEY_decrypt

Currently, C++ implements public key encryption and decryption using the OpenSSL library.
However, the second EVP_PKEY_decrypt among the following causes Segmentation fault when running the program:
The operating system is Ubuntu 14.04.3 and the compiler is ~~clang++-3.6 or~~ clang++-3.8.

bool decryptData(EVP_PKEY*key, unsigned char*data, size_t datalen, unsigned char**decdata, size_t*declen){
    intret;
    EVP_PKEY_CTX*ctx=EVP_PKEY_CTX_new(key,NULL); // Key is RSA

    if((ret=EVP_PKEY_decrypt_init(ctx)))!=1){
        std::cerr<<"init:"<<ret<<std::endl;
        return false;
    }
    if((ret=EVP_PKEY_decrypt(ctx,NULL,declen,data,dataren))!=1){ 
        std::cerr<<"dec1:"<<ret<<std::endl;
        return false;
    }

    *decdata=(unsigned char*) realoc(*decdata,sizeof(unsigned char*)**declen);
    // *decdata=(unsigned char*) malloc(sizeof(unsigned char*)**declen);
    if((ret=EVP_PKEY_decrypt(ctx,*decdata,declen,data,data)))!=1) {//<--
        std::cerr<<"dec2:"<<ret<<std::endl;
        return false;
    }
        return true;
}

If you look at the valgrind, you will see the following:

==21343==Invalid read of size 8
==21343 == at 0x4EFF421: ???(in/lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==21343 == by 0x4F047B6: ???(in/lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
==21343 == by 0x402FC9: decryptData(evp_pkey_st*, unsigned char*, unsigned long, unsigned char**, unsigned long*) (pubkeyenc.h:53)
==21343 == by 0x40323E: main(tmp.cpp:27)
==21343 == Address 0x0 is not stack'd, malloc'd or (recently) free'd

I have confirmed that all the addresses of the variables in the decryptData() argument are not zero, but what is the possible cause of this error?
I look forward to your kind cooperation.

The caller has the following code and encryptData only encrypts the decrypt in decryptData.

int main(){
    unsigned char msg [ ] = "hoge"; just
    unsigned char*enc = nullptr;
    // unsigned char*enc = NULL;
    size_t*enclen=(size_t*)malloc(sizeof(size_t)));
    EVP_PKEY*key=/*Function to place the key*/
    if(key==nullptr)cerr<<"priv key error"<<endl;
    if(!(encryptData(key, msg, strlen(reinterpret_cast<char*>(msg))), &enc,enclen))}
        cerr<<"error"<<endl;
    } else {
        cerr<<"enc success"<<endl;
    }

    unsigned char*decoded=nullptr;
    // unsigned char*decoded=NULL;
    size_t*declen=(size_t*)malloc(sizeof(size_t)));
    key=/* function to put the key*/
    if(key==nullptr)cerr<<"pub key error"<<endl;
    if(!(decryptData(key,enc,*enclen,&decoded,declen))){
        cerr<<"dec error"<<endl;
    } else {
        cerr<<"dec success"<<endl;
    }   

    free(enclen);
    free(declen);
    if(enc!=nullptr)free(enc);
    if(decoded!=nullptr)free(decoded);

    return 0;
}

The key used for encryption was PEM_read_PrivateKey, the secret key created with ssh-keygen, the public key created with ssh-keygen, and the key used for decryption was PEM_read_PUBKEY.
However, when I reversed them and encrypted them with a public key and decrypted them with a private key, the program worked without any problems.

c++ c openssl

2022-09-30 20:13

1 Answers

Address 0x0 is not stack'd, malloc'd or (recently) free'd

So I'm trying to read the value from nullptr.There is a possibility of a bug on the OpenSSL side, but it probably gave the wrong value.

I modified the code listed and tried to execute it, but it worked fine.

In addition, enclendeclen does not need to be size_t* and

*decdata=(unsigned char*) realoc(*decdata,sizeof(unsigned char*)**declen);

reserves eight times as much memory as requested.

2022-09-30 20:13

If you have any answers or tips

Popular Tags

python x 4647

android x 1593

java x 1494

javascript x 1427

c x 927

c++ x 878

ruby-on-rails x 696

php x 692

python3 x 685

html x 656