About updating the keystore after transferring the Android app

You have to use the original keystore.

The transfer of apps is a matter between Google Play accounts, so it is separate from the digital signature issue for the app package (apk).

Regardless of the app transfer, you cannot change the digital signature of the app, so it is natural that the app transfer does not make it possible to change it.

For more information, the app is identified by the package name.In other words, rather than an apk file, it is an image with a "digital signature" tied to the "package name."

Unless you change the specifications so that Google can change this link later in the future, you cannot change it now.

Digital certificates are used for this digital signature.

Also, since it's about "digital certificate," it's different from "keystore" itself.A keystore is a database that holds digital certificates.I think it is possible to export the certificate of the application from the old keystore and import it into the new keystore.See the appropriate information for details such as how to operate it.
Note: Using the keytool utility

You have already answered this question, but I have just quoted it from the official document.

From Signing Your Applications

You should sign all of your apps with the same certificate through the expected life span of your applications.

Applications must continue to use the same certificate.For example, if you lose the keystore, you will never be able to update it again.

If you sign the new version with a different certificate, you must assign a different package name to the application — in this case, the user installs the new version as a complete new application.

If you want to publish a new version signed with a different certificate, you must publish it as a completely different application with a different package name.