Scaling Out Azure Functions With Event Hubs Effectively 2

Overview

I introduced how to scale-out Azure Functions with Event Hubs effectively in the previous blog post with real world project example. In addition to that, I want to introduce best practice about how to separate Azure resources here.

Background

When you create application with Azure Functions (Consumption plan) and Event Hubs, you need to consider how to separate following resources. It can affect your app performance.

  • Function App
  • App Service Plan
  • Application Insights
  • Azure Storage Account
  • Event Hub Namespace and Event Hub

Best Practices

Case Study

Let’s assume your team develops app with following architecture. 1st Event Hub need to handle 12,000 events/sec, then 1st Function App filter out 1 % unexpected message (ex. value because of sensor accident). 2nd Function App labels message based on payload.

In this case, for achieving best performance, I recommend to compose solution with following resources.

  • 2 Function App to run specific function (ex. 1st Function App only runs filter function).
  • These functions App uses same App Service Plan and Application Insights.
  • Each Function App utilize unique Azure Storage Account.
  • 2 Event Hub Namespace which includes only 1 Event Hub. In this case study, total number of message/sec is 23,800, and the system needs 24 TU. As default, max number of TU is 20. You can ask Azure support to increase TU, but I prefer to separate namespace, because it’s easy to setup and each namespace needs 12TU now and can be scaled by 20TU in the future.

Infrastructure as a Code (IaC) on best practice

For load testing and performance tuning phase, I experienced to setup Azure resources again and again. For example, I needed to change Event Hub partition count and run test to see better result.

For automating it and provide result quickly, I recommend to utilize IaC. For implementing IaC, I prefer to use Terraform than Azure Cli and ARM template, because..

  • It takes care delta. If I updates infra code, it update infrastructure.
  • Easy to code and run in local and CD pipeline.

You can use sample code in my GitHub repo which I applied best practice above.

Software development engineer https://www.linkedin.com/in/masayuki-ota/

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store