Introduction

Performance tuning is a bit more art than science. Not every single tuning technique solves every problem, and very often techniques interact. The approach should always be methodical, including baseline and differential testing particularly when justifying capital expenditures.

Few Problems 

At some point of time, we may face issues in our BizTalk farm and then we think about ways to identify and resolve those issues and we start searching for BizTalk Performance Tuning and Optimization options. Those issues could be one of following or somewhere similar: 

• BizTalk is performing slower ,when we drop the messages or during high volume of messages.
• Large no of messages are causing memory errors in BizTalk farm.
• Prolonged BizTalk Throttling.
• BizTalk receive location is not picking up the files.
• BizTalk Database Jobs are taking too much time to complete.
• Response time of message processing is increasing. There is a high latency in environment.
• It seems BizTalk is not able to process any message, The messages or service instance are in Active/Running State, however nothing is happening.
• SOAP Adapter is taking long time call a Web Service.
• Orchestration/Receive Port is taking  long time to debatch a large message.
• BizTalk production farm has suddenly started behaving in a strange manner.
• High Memory/CPU utilization of the BizTalk servers.

Few Requirements

It is true, at some point of time, we start getting queries, thought or requirement similar to below and then we look for options to scale or optimize the BizTalk Server:

• Looking for creating scalable BizTalk solutions.
• Trying to achieve a particular latency for a BizTalk application.
• Requirement to process 1000 messages in 1 second by a BizTalk application.
• Requirement of processing large messages in BizTalk Server.
• Planning to change the performance throughput.
• Trying to determine/test maximum sustainable throughput ( MST ) of a BizTalk farm.
• Trying to figure if BizTalk farm can support next 20+ applications which are going to be deployed in future.
• How should I decide if My BizTalk farm needs to be tuned or optimized ?
• How  should I decide if BizTalk production farm needs to be scaled-up or scaled-out ?

What Should We Do

First Few Steps

If  you have anything similar in the list above (Few Problems), then we should start with doing below things:

• Check SQL Server Agent and BizTalk Server Jobs Status of the BizTalk SQL Server Instance - The agent should always be running and all the BizTalk Jobs should be enabled from the day one in your production Farm , If there are any issues refer this in-depth Wiki , "BizTalk Server:Databases: Survival Guide".
• The BizTalk Host instance that has the "allow Host tracking" option set is stopped - If the host is stopped , then it should be started without any delay. As a practice there should always one dedicated "Tracking Host" should be running in BizTalk Farm. Please refer "Configuring a Dedicated Tracking Host" MSDN page for more details.
• Record count of the Spool table- the row count in the tables should not be too much or it should not keep growing. Check the record count of the Host Instance Queue tables. Refer "Understanding Host Queue Tables" article for more details.
• Free space of the Data and log file drives on SQL Server, as there should be enough free space in the log and data files drive on SQL Server.
• Size of BizTalkDTADb database - Need to make sure the size is not too much of this database, It is a standard practice to keep an eye on the size of this database. Refer the following articles:
  • How to Configure the DTA Purge and Archive Job
  • BizTalk Server 2010: Database BizTalkDTADb
• Excessive Disk I/O Latency (on SQL Server) - If the incoming rate of data into the MessageBox database is faster than which the system can process and move the data to the BizTalkDTADb database, backlog can build up in the MessageBox database. If the backlog continues to grow unabated, this is a very serious problem and system performance will degrade over time. One way to alleviate this problem is to install faster disks and/or upgrade the hardware to help ensure that the system is capable of recovering from any message backlogs experienced over time.

Review of Solution Design

Start reviewing the application design and architecture. Sometime a design change may have huge impact over the application performance. Following should be the starting reference points while reviewing the design and code of BizTalk solutions:

• Code review of Custom Pipeline component.
• Are we using XMLDom to load a very large message in pipeline ?
• What is the method used to read the values in incoming message , Use streaming while reading the message or use XpathReader ?
• Code of Orchestration ?
• Design ( use of functoid or inline xslt) of MAP used in transforming for large messages.
• Are we using Enterprise Integration Patterns or not while designing enterprise solutions ?
• What kind of Custom Code we are using?
• Can we use inline send port instead of normal send ports to achieve low latency solutions ?
• How we are calling Business Rule engine in our BizTalk Orchestrations?
• How many scopes have been used in orchestration ?
• What kind of instrumentation or tracing is used ?
• Use of BAM API's or TPE.

References for Code Review/Solution Design

Following articles might be useful while reviewing the solution architecture and the BizTalk Code.

• Optimizing Pipeline Performance
• Optimizing Memory Usage with Streaming
• Optimizing Orchestration Performance
• FAQ for BizTalk Orchestrations
• BizTalk Orchestration Profiler
• BizTalk Orchestrations XPath: Survival Guide - Refer for the optimized ways to read values from XML
• How To Boost Message Transformations Using the XslCompiledTransform class Extended
• Thoughts on Orchestration Performance 
• BizTalk Orchestration Performance: Thoughts from BizTalk Support 
• 4 Different ways to process an XLANGMessage within an helper component invoked by an orchestration Part 1
• Best Practices for Instrumenting High Performance BizTalk Solutions
• Bad practices for Designing and Developing BizTalk/ESB Solutions - By Howard Edidin
• BizTalk Code Review checklist- By Richard Seroter
• 8 Tips And Tricks For Better BizTalk Programming
• Drawback of Inline Send Port (MSDN Social forum Thread)- refer this to get the opinion about using inline send port in BizTalk Server solutions to achieve low latency. 
• BizTalk Server and Protocol Transition- By Paolo Salvatori , refer this to explore alternative ways to integrate systems with BizTalk Server
• Project Planning Recommendations by Phase-There are a number of software development lifecycle models in existence today, each with their own approaches, benefits, and limitations. The goal of this section is to provide a set of recommendations that will help you plan appropriately for a successful BizTalk Server development project.

Review Everything Else!

It is also very crucial to review BizTalk environment including SQL and everything else which is your control. Following could be some initial points to start with:

1. Hardware configurations of BizTalk Server Box, and yes its not only about looking RAM and CPU there are other things as well like antivirus settings, hyper-threading etc. It is very important to decide these things while designing the farm. Please see below reference links for more details.
2. Hardware and SAN configuration details of SQL Server instance. Choosing right RAID level of SAN disks really matters!!
3. Network settings can be crucial. Check for instance this article from Sandro Pereira.
4. Load balancing of BizTalk Web Services really matters while designing low latency solutions . Need to check options of IIS tuning.
5. Setting BizTalk databases and log file drive size really matters need to review them. See below references for more details.

References

Below are few references , They have tons of information on performance optimization and tuning of BizTalk.

• ( Important! ) Microsoft BizTalk Server 2010 Performance Optimization Guide- Always Keep a Copy of this guide, It provides prescriptive guidance on the best practices and techniques that should be followed to optimize BizTalk Server performance. This guide should be consulted while designing the BizTalk Production or Test Farms.
• Microsoft BizTalk Server 2010 Operations Guide- Provides detailed information for planning a BizTalk Server environment, as well as recommendations and best practices for configuring, testing, maintaining, monitoring, and optimizing this environment.
• BizTalk Administrator's Checklist Compiled by Microsoft BizTalk Support
• How to troubleshoot SQL Server performance issues
• Performance Tuning Guideline for Windows Server 2008 R2
• BizTalk 2009 – End to end performance testing (1/2) ( Webcast ) - By Paolo Salvatori 
• BizTalk 2009 – End to end performance testing (2/2) ( Webcast ) - By Paolo Salvatori
• ( Important! ) How to exploit the Text In Row table option to boost BizTalk Server Performance
• ( Important! ) BizTalk Server Performance: Configuring BizTalk Server for Performance ( Webcast ) - A must watch webcast to understand the summary of performance tuning with BizTalk Server 2010
• Performance Tuning for Low Latency - A in-depth article about BizTalk Server to achieve Low Latency. If there is a requirement to host low latency messaging solutions in BizTalk farm , this article might give lot of information about different parameter changes to achieve it.
• Scaling Out the MessageBox Database - 
• Scaling Out the BizTalk Server Tier - Scaling-out is the process of adding additional computers. For example, if BizTalk Server is bottlenecked by CPU resources, adding another server provides double the CPU resources which may provide double the throughput. You scale out BizTalk Server by adding more computers to your BizTalk Server topology. The following figure shows an example of how you might scale out the BizTalk tier
• Performance Tips and Tricks - This article provides useful tips for ensuring optimal performance for your BizTalk Server system, It covers following points in-depth
  
  • Identifying Performance Bottlenecks
  • Setting the EPM Threadpool Size 
  • Troubleshooting MessageBox Latency Issues 

MSDN

• BizTalk Server 2010 Database Infrastructure Poster- Refer this to understand the architecture of BizTalk Database
• BizTalk Server 2010 Scale-out Configurations Poster - Refer this to understand the scale-out options for BizTalk Server
• BizTalk Server 2010 Runtime Architecture Poster
• How BizTalk Server Implements Host Throttling - Refer this about getting familiar with Throttling concepts and how the BizTalk Engine does it
• What is Host Throttling - Refer this about details of Host Throttling
• Microsoft BizTalk Server Automatic Throttling (Wiki) - In-depth details about throttling
• Throttling - MSDN Blog - Provides details about throttling and how to use performance counters to know more about it.
• Host Throttling Performance Counters - Provides the performance monitor counters that measure system parameters that impact host throttling
• BizTalk Server 2010 Host Throttling - Refer this TechNet Wiki to understand in-depth  of Throttling in BizTalk 2010
• Measuring Maximum Sustainable Engine Throughput-  Refer this for more details about MST and Testing approaches to determine MST of your BizTalk Engine.
• Load Pattern Tests to Determine Maximum Sustainable Throughput- Refer this about using Visual studio to perform automated testing to figure MST
• Measuring Maximum Sustainable Tracking Throughput - Refer this to know more about MST of BizTalk DTA Tracking throughput
• Processing Large Messages - The article defines large messages and provides guidelines for how to configure BizTalk Server process large messages efficiently
• Load Testing BizTalk Server Solutions with Visual Studio 2010 - Refer this TechNet Wiki article about load testing using Visual Studio and BizTalk 2010
• Evaluating BizTalk Server Performance on Hyper-V -The article describes considerations for evaluating and optimizing performance of a BizTalk Server application running on a guest operating system installed on a Hyper-V virtual machine. It also provides a description of tools that can be used to measure performance of a BizTalk Server application
• Managing a Successful Performance Lab - This paper outlines some key strategies for managing a successful performance lab with Microsoft BizTalk® Server 2006.
• How to Monitor BizTalk RFID Using Performance Counters
• Checklist for Analyzing Slow-Running Queries - Since BizTalk database is highly optimized database and there is no need to further tune the Queries, However sometime we use custom databases in our applications, and they could become bottleneck sometime. Refer this article for tuning the SQL queries.
• BizTalk Server 64-Bit Support - The article provides some frequently asked questions about 64 bit support for BizTalk Server. Refer this article if your are planning to deploy 64 bit BizTalk hosts.
• BizTalk Server 2006 Comparative Adapter Study - This white paper describes the results of a study that compares the performance of BizTalk Server 2004 and BizTalk Server 2006 adapters
• BizTalk Server 2006: Scalability Case Study Using the SOAP Adapter in BizTalk Server 2006 - This document demonstrates the potential increase in efficiency through scaled-up server and adapter combinations. Tests were performed on various combinations of BizTalk Servers and SQL Servers. Test criteria, procedures, server and adapter configurations, and results are presented. The data can help the reader determine which configuration best-suits their existing or anticipated business requirements.

MSDN Social

• MSDN Social Forum for the discussion with Community members - Please refer this forum for posting a new Question or reading about on-going discussions.
• MSDN Social Forum - All threads related to BizTalk Performance Tuning & Optimization

Tools

There are lots of tools described in "Performance optimization guide" by Microsoft and "BizTalk Server 2010:Tools" wiki, however below are few tools which are very common and it is good to be familiar with them:

• PAL - Use this tool to gather and analyze performance data from BizTalk, SQL or any other Windows server. This tool provides pre-configured templates with performance counters related with BizTalk server and SQL Server. The templates can be configured in perfmon tool to get the performance data, and further data is used by this tool to generate the performance report.
  • Using the Performance Analysis of Logs (PAL) Tool
  • How to use the PAL Tool for BizTalk Performance Analysis
• Message Box Viewer - A great tool to inspect BizTalk group. The tool generates a comprehensive report summarizing all the issues and summary of critical potential problems.
• BizTalk Terminator
  • Using BizTalk Terminator to resolve issues identified by BizTalk MsgBoxViewer and the BizTalk 2010 Monitor BizTalk Server Job
  • Clean up the MarkLog table with Terminator
• BizTalk Benchmark Wizard- Use this tool to review BizTalk farm after configuring the Farm.
• BizTalk Monitoring Tools - Refer this Wiki to know all the tools to monitor BizTalk Server Farm
• LoadGen - Refer this tool to load generation while performing load testing on BizTalk Farm
  • Random File Message Creator for LoadGen- By Randal
  • LoadGen and a Simple WCF Load test - By Thiago Almeida
  • MSDN - Using BizUnit to run LoadGen.
• Process Explorer - The Process Explorer display consists of two sub-windows. The top window always shows a list of the currently active processes, including the names of their owning accounts, whereas the information displayed in the bottom window depends on the mode that Process Explorer is in: if it is in handle mode you'll see the handles that the process selected in the top window has opened; if Process Explorer is in DLL mode you'll see the DLLs and memory-mapped files that the process has loaded. Process Explorer also has a powerful search capability that will quickly show you which processes have particular handles opened or DLLs loaded.
• Adapter Trace Utility - Sometime it is required to trace a BizTalk adapter to get the insights of message processing, refer this article for "how to use BizTalk Adapter Trace" tools
• Debugging Tools for Windows - Use Debugging Tools for Windows to debug drivers, applications, and services on Windows systems. Debugging Tools for Windows includes a core debugging engine and several tools that provide interfaces to the debugging engine. A Visual Studio extension provides a graphical user interface, as does Windows Debugger (WinDbg). Console Debugger (CDB), NT Symbolic Debugger (NTSD), and Kernel Debugger (KD) provide command line user interfaces.

Blogs

Following are few community blogs posts which might be helpful while working with tuning & optimization.

• Identify Performance Bottlenecks in your BizTalk Environment – Part I
• Identify Performance Bottlenecks in your BizTalk Environment – Part II
• Threads Database Sessions and Throttling 
• Throttling in Full Action - By Yossi Dahan
• BizTalk Orchestration Throttling Pattern- By Richard Seroter
• Configure BizTalk hosts per Application- By Thiago Almeida
• Host Tracking and the BizTalk MessageBox- By Nick Heppleston
• Orchestration Performance ( Webcast ) - By Alan Smith
• Clustering BizTalk Hosts - By Kent Weare
• BizTalk Tips to Improve IIS performance - By Nino Crudele

Hot Fixes

Following are few product hot fixes which may be helpful, But it is strongly recommended to read the KB article first and understand if they are absolute necessary for the product.

• KB2627804 - Hotfix that improves the performance of the tracking feature in BizTalk Server
• KB970070  - Support for Locked Pages on SQL Server 2008 R2 Standard Edition x64, on SQL Server 2005 Standard Edition 64-bit systems, and on SQL Server 2008 Standard Edition 64-bit systems
• KB944426  - Orphaned cache instances may be built in the Instances and Hosts Queue tables of the BizTalkMsgBoxDb database in BizTalk Server 2006 and in BizTalk Server 2006 R2
• KB943165  - A CPU usage spike occurs on all the BizTalk hosts after you install security bulletin MS07-040 on a server that is running Microsoft BizTalk Server 2006 or Microsoft BizTalk Server 2006 R2
• kb2387313 - FIX: Poor performance occurs when you use WCF-based BizTalk Adapter for Oracle Database on a dynamic port that is used by BizTalk Server

See Also

Another important place to find a huge amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is BizTalk Server Resources on the TechNet Wiki.