API Gateway Webhooks Lambda HMAC Validation

In the previous post HMAC Validation we explored how to validate HMAC in an API project. The video Webhooks Processing: HTTP API Gateway + SQS +Lambda shows how to created a scalable solution for receiving webhooks and throttling them with SQS and Lambda. In this article we will look at how to use HMAC Validation with AWS API Gateway, SQS, and Lambda.

HTTP API Gateway Webhooks HMAC Validation Implementation

As a reminder, we created HTTP API Gateway to receive webhooks. We placed webhooks on an SQS. From the SQS webhooks were picked up by lambda for further processing. In case processing failed, we created a DLQ, dead letter queue for the failed messages. Optionally, we will also add an S3 bucket to save webhook’s request body.

Most importantly though, we used SQS Message Attributes to map webhook headers that are important for processing, so we can get access to them in Lambda function. Those headers were x-topc and x-hmac. Obviously those headers will be different depending on the service you are integrating with. For, example, corresponding Recharge headers are called X-Recharge-Topic and X-Recharge-Hmac-Sha256 However, for the sake of simplicity, we will just stick with x-topic and x-hmac Since we are going to be saving webhook’s request body to an S3 bucket, an additional attribute x-webhook-id will be mapped as well and it will be used as the key in s3 bucket. Corresponding header in Shopify webhook, for example, is X-Shopify-Webhook-Id

In the video Webhooks Processing: HTTP API Gateway + SQS + Lambda we showed how to create AWS resources through AWS console. This time we will be using terraform code to re-create the resources and add an S3 bucket. The terraform code is available in this GitHub Repository.

In this article we will only look at the Lambda function code for validating HMAC with HTTP API Gateway an saving webhooks to S3 bucket. Please watch the video below for more in depth explanation.

In order to validate HMAC and save the body of the webhooks in S3 bucket we put the code below in Lambda.

// modules/lambda/index.mjs
import crypto from "crypto";
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";

function createHmac(message) {
  const hmac = crypto.createHmac("sha256", process.env.API_SECRET);
  hmac.update(message);
  return hmac.digest("hex");
}

function verifyHmac(message, receivedHmac) {
  const hmacFromMessage = createHmac(message);
  return crypto.timingSafeEqual(
    Buffer.from(hmacFromMessage, "utf8"),
    Buffer.from(receivedHmac, "utf8")
  );
}

async function saveToS3(s3client, webhookBody, key) {
  const input = {
    Body: webhookBody,
    Bucket: process.env.BUCKET,
    Key: key,
  };
  const command = new PutObjectCommand(input);
  await s3client.send(command);
}

export const handler = async (event) => {
  console.log("Event: ", event);
  const response = {
    batchItemFailures: [],
  };

  for (const record of event.Records) {
    console.log("message attributes ", record.messageAttributes);
    const payload = JSON.parse(record.body);

    const bodyString = record.body;
    const hmacSignature = record.messageAttributes.hmac.stringValue;

    try {
      if (!verifyHmac(bodyString, hmacSignature)) {
        console.log("Invalid HMAC. Dropping webhook");
        continue;
      }
    } catch (error) {
      console.log("Error verifying HMAC. Dropping webhook", error);
      continue;
    }

    const client = new S3Client({
      region: process.env.REGION,
    });

    try {
      await saveToS3(
        client,
        bodyString,
        record.messageAttributes["webook-id"].stringValue
      );
      console.log("Processing payload webhook", payload);
    } catch (error) {
      console.log("Error processing webhook", error);
      // failed record will go back to queue
      response.batchItemFailures.push({ itemIdentifier: record.messageId });
    }
  }

  return response;
};

In the code above we create HMAC helper functions to validate HMAC. Next we also create a function to save webhook’s body into S3 bucket. In Lambda handler function we defined a response as an object with a property batchItemsFailure that is assigned to an empty array. When setting up SQS as Lambda trigger we selected ReportBatchItemFailures. In this case if we get let’s say a batch of 10 webhooks from SQS and 3 fails, we can update batchItemsFaillure array to contain the IDs of the events for those webhooks. SQS will know to that only those events failed and will make them visible again. The other 7 events for webhooks will be considered successfully processed an would be delete from the queue.

We wrap HMAC validation in try catch statement and simply drop the webhook if validation fails, because API Gateway already responded with OK 200 when it received the webhook and placed it on the queue.

Finally we attempt to save webhook’s body to the S3 bucket and also create a console log statement designating processing of the webhook. If an error happens the catch block will log it and update batchItemFailures array with a failed event message ID.

API Gateway Webhooks Testing

To test the setup we use the same approach with Postman’s pre-request script as we used in HMAC Validation article. Please, provide x-topic, and x-wehbook-id headers (x-hmac header will be added automatically by the pre-request script). The API Gateway will reject the requests without these heasders. Also don’t forget to add API_SECRET variable to Lambda function’s environment variables that matches the secret used in pre-request script. If everything works, you should see webhooks saved in the S3 bucket.

Conclusion

In conclusion, our exploration of AWS API Gateway, SQS, Lambda, and HMAC Validation underscores the pivotal role of Amazon Web Services in modern API development. By mastering these core components, we empower ourselves to design scalable, efficient, and secure APIs.