Scaling Out Azure Functions With Event Hubs Effectively 2

Overview

I introduced how to scale-out Azure Functions with Event Hubs effectively in the previous blog post with real world project example. In addition to that, I want to introduce best practice about how to separate Azure resources here.

Background

When you create application with Azure Functions (Consumption plan) and Event Hubs, you need to consider how to separate following resources. It can affect your app performance.

• Function App
• App Service Plan
• Application Insights
• Azure Storage Account
• Event Hub Namespace and Event Hub

Best Practices

• Separate Function App if you have function which highly consumes compute resource, because all functions within a function app share resource within an instance.
• Can use a same Consumption plan by Function Apps, because function apps that share the same Consumption plan are scaled independently.
• Use single Application Insights resource for application components that are deployed together. It can help you to track end-to-end latency. You may want to see Shervyna’s blog post for more detail.
• Create Azure Storage Account per Function App, because you can separate high volume of storage transactions from Function App.
• Separate Event Hub Namespace if you have event hub which needs high Throughput Unit (TU).

Case Study

Let’s assume your team develops app with following architecture. 1st Event Hub need to handle 12,000 events/sec, then 1st Function App filter out 1 % unexpected message (ex. value because of sensor accident). 2nd Function App labels message based on payload.

In this case, for achieving best performance, I recommend to compose solution with following resources.

• 2 Function App to run specific function (ex. 1st Function App only runs filter function).
• These functions App uses same App Service Plan and Application Insights.
• Each Function App utilize unique Azure Storage Account.
• 2 Event Hub Namespace which includes only 1 Event Hub. In this case study, total number of message/sec is 23,800, and the system needs 24 TU. As default, max number of TU is 20. You can ask Azure support to increase TU, but I prefer to separate namespace, because it’s easy to setup and each namespace needs 12TU now and can be scaled by 20TU in the future.

Infrastructure as a Code (IaC) on best practice

For load testing and performance tuning phase, I experienced to setup Azure resources again and again. For example, I needed to change Event Hub partition count and run test to see better result.

For automating it and provide result quickly, I recommend to utilize IaC. For implementing IaC, I prefer to use Terraform than Azure Cli and ARM template, because..

• It takes care delta. If I updates infra code, it update infrastructure.
• Easy to code and run in local and CD pipeline.

You can use sample code in my GitHub repo which I applied best practice above.