Support custom signer for S3Client

[marin-ma]

Describe the feature

We are currently using the hadoop-aws module with a self-implemented signer for our Spark/Hadoop applications. (reference: https://hadoop.apache.org/docs/r3.4.1/hadoop-aws/tools/hadoop-aws/index.html#Advanced_-_Custom_Signers)

Now we're moving from the java s3 client to the c++ client that provided in the aws-sdk-cpp library. The base client class AwsClient provides a constructor that accepts a signer/signer provider, which allows to use a custom signer implementation.

/**
 * Configuration will be used for http client settings, retry strategy, throttles, and signing information.
 * Pass a signer provider to determine the proper signer for a given request; AWS services will use
 * SigV4 signer. errorMarshaller tells the client how to convert error payloads into AWSError objects.
 */
AWSClient(const Aws::Client::ClientConfiguration& configuration,
          const std::shared_ptr<Aws::Auth::AWSAuthSignerProvider>& signerProvider,
          const std::shared_ptr<AWSErrorMarshaller>& errorMarshaller);

However, the S3Client currently does not expose this option.

Use Case

Currently the only way to use the custom signing process in our application with the cpp sdk is to add a patch to the AwsAuthV4Signer that modifies the signing process. This approach brings extra effort for maintenance when upgrading the aws-cpp-sdk library.

Proposed Solution

Supporting creating S3Client with the custom singer or singer provider would allow applications to integrate self-implemented signers in a consistent way across AWS SDKs, rather than being constrained to the default signer.

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

[sbiscigl]

sorry its taken a while to get back to you as far as overriding the signer -- why do you want to override the signer? what can you accomplish with overriding the signer that you cannot accomplish by overriding the credentials provider?

[marin-ma]

@sbiscigl Thanks for following this. In our use case, the signing process is actually handled by an external service. For the payload to sign, we need to send a http request to the server and extract the signature from the response. This is a custom signing process but the format of the signature strictly follows the official document for the signature calculation.

[sbiscigl]

so i guess my question is then why do you need a external service to handle the signing process? why does that need to happen instead of providing a custom way to retrieve credentials? there are loads of ways to query credentials from a external source, but why does this customization need to happen in the signer and not the credentials?

[marin-ma]

so i guess my question is then why do you need a external service to handle the signing process? why does that need to happen instead of providing a custom way to retrieve credentials? there are loads of ways to query credentials from a external source, but why does this customization need to happen in the signer and not the credentials?

@sbiscigl Apologise that I might not capable of answering this question. I'm not part of the system design. The requirements to fit our application into the system is that we need to adapt the signing process to the existing system. On java side the aws sdk for Hadoop provides a way to use the custom signer through the configuration fs.s3a.custom.signers, but on native side there's no way we can plug in.

[sbiscigl]

I think before I would want to do this I would want a answer around why this needs to be a override of the signer explicitly and not just a overridden credentials provider. I understand that you might not have the answer, but I need a more detailed explanation of why you cant use a overridden credentials provider + our normal sigv4 signer

[marin-ma]

I think before I would want to do this I would want a answer around why this needs to be a override of the signer explicitly and not just a overridden credentials provider. I understand that you might not have the answer, but I need a more detailed explanation of why you cant use a overridden credentials provider + our normal sigv4 signer

In our use case I can't not leverage the existing sigv4 signer. The whole singing process is delegated to a remove service, and the ak/sk credentials are also stored in that service which is transparent to the client side. Therefore simply overriding the credentials provider cannot fit into our use case.

[sbiscigl]

the ak/sk credentials are also stored in that service which is transparent to the client side

if it is transparent to the client side, why could you not override the credentials to use these credentials?

[marin-ma]

@sbiscigl We need to implement a custom signer, in the signer the payload hash and some other metadata are sent to an external signing service which complete the signature calculation process, and payload is signed with the signature returned from the service. We cannot achieve that only through a custom credentials provider, we also need a custom signer in this case.

[sbiscigl]

I get that you want to use a external service to sign it, that still doesn't explain why its needed. why can this only be achieved through overriding the signer outside of the fact that you want to be able to.

What customization exists in your signer that cannot be overriden at a different point in the SDK?

[marin-ma]

@sbiscigl The reason why we cannot use it is that the credentials is not provided by the client side, instead they are stored within the external service which are completely transparent to the client. The service uses the stored credentials (ak/sk) for generating the signature. There are two things in the SignRequest method of AwsAuthV4Signer that doesn't satisfy the requirement of our use case:

1. The signature is calculated in SignRequest, but we need it calculated and returned by the external service. Therefore, we need to use our custom implementation that overrides this method.
2. The signature calculation requires a valid ak/sk from the credentials provider for the signature calculation. In our use case the ak/sk are only kept in the external service for the signature calculation and are transparent to the client side, which means the client doesn't need to provide ak/sk to connect with the s3.

Currently the S3Client only accepts a custom credentials provider, but not a custom signer provider. Based on my understanding we cannot achieve the custom signing process with a credentials provider implementation.