NetScaler commissioned The Tolly Group, the world’s leading provider of third-party validation services to IT vendors, to conduct rigorous benchmarking tests comparing NetScaler VPX and F5 BIG-IP Virtual Edition (VE) virtualized software application delivery controllers (ADCs). The results are unequivocal: NetScaler outperforms F5 in every scenario, delivering better CPU efficiency, lower latency, and higher throughput — all of which have a significant impact on application response time, application end-user experience, and total cost of ownership (TCO).
Testing conditions and scenarios
ADCs have evolved from simple load balancers to multifunctional devices handling various traffic management and security tasks. The key role of an ADC is to enhance application performance. But if it induces excessive latency and slows traffic, it becomes part of the problem it is meant to solve.
When testing ADC performance, it’s critical to conduct the tests in simulated real-world scenarios to provide a realistic reflection of the performance that customers can expect. To represent the most common internet traffic scenarios, all test traffic was encrypted with TLS 1.3. This is because analysis of data available from web crawlers across the top 1 million websites showed that 73 percent of SSL traffic uses the latest TLS 1.3 protocol.
And because many enterprises are consolidating their application security workloads such as web application firewalls (WAF), into their ADCs, tests were conducted with ADCs running various combinations of traditional load balancing (baseline test), content rewriting/manipulation (policy), and application security inspection (app sec) in increasing order of task complexity.
NetScaler outperforms F5 in all scenarios
NetScaler consistently outperforms F5 by using fewer CPU resources and introducing significantly lower latency across various scenarios and workload conditions. Overall, F5’s performance declines when carrying out multiple, common ADC tasks.
NetScaler runs as software in any environment and delivers the following advantages over F5 in a series of tests:
NetScaler achieves 3x higher throughput for similar CPU usage compared to F5
NetScaler demonstrates higher throughput in all test scenarios when compared with F5. For instance, with load balancing, content rewriting, and application security inspection combined (load balancing + policy + app sec), NetScaler throughput was 3x that of F5 at 2,461 Mbps compared with 797 Mbps:
64 percent more efficient CPU usage than F5 for a given throughput
NetScaler consistently shows more efficient CPU usage than F5 across all test scenarios. For example, with load balancing and application security enabled, NetScaler CPU usage is about half that of F5:
It is worth noting that the test scenario involving load balancing, policy, and application security inspection had to be adjusted because F5 could not handle 2.3 Gbps of throughput without maxing out its CPU resources. Consequently, the throughput was throttled down to 1.2 Gbps to prevent F5 from exhausting all CPU resources. Even then, NetScaler CPU usage was 64 percent lower than F5. As shown in Figure 2, when the number of tasks increased, F5’s CPU usage increased to almost 3x that of NetScaler, irrespective of the throughput being throttled.
F5 introduces up to 9x latency (application response delay) for similar throughput
In today’s application economy, application experience is frequently synonymous with user experience. A substantial increase in application response delay can significantly affect application performance, leading to a poor application end-user experience and reduced service quality.
53% of mobile users abandon sites that take longer than 3 seconds to load.
Google
In the testing scenario with load balancing, policy, and application security enabled (again with throughput throttled down to 1.2 Gbps because F5 could not handle 2.3 Gbps of throughput without running out of CPU resources), NetScaler application response delay per request was 89 percent less (better) than F5:
NetScaler’s one-pass architecture allows it to process security functions at the same time as the load balancing function, which results in low latency even when security functions are enabled.
To reach the same level of functionality and performance, you would need more/larger F5 ADC instances than NetScaler ADCs, which would add complexity and increase costs.
NetScaler shows 96% less latency than F5 as the number of simultaneous connections increases
One of the most important roles for ADCs is scalability, which refers to the ability to process multiple simultaneous connections without impacting application response times.
This test measured P95 latency (measurement of the 95th percentile latency to show the delay for the worst 5 percent of traffic flows) at load levels 200, 1,000, 2,000, and 4,000 simultaneous connections/users with load balancing, policies, and application security capabilities enabled. As the load levels increased from 200 to 4,000 connections, the differences in latency between NetScaler and F5 were dramatic:
At 4,000 users, NetScaler’s application response time was 96 percent less than that of F5. In the case of F5, the individual sessions experienced degraded response times. With 4,000 concurrent users, which is typical for e-commerce applications, F5’s delay in application response time (latency) was a whopping 26 times that of NetScaler (48.24 ms compared with 1.85 ms).
Impact: How ADC performance affects business outcomes
Let’s take a look at how the above test results translate to real-world applications for three industries that rely on ADCs the most:
E-commerce
High-performing e-commerce applications have one thing in common — exceptionally fast response times. Customers expect instant connection/fast load times and always-on availability from these applications. Failing to deliver leads to shopping cart abandonment and/or application deletion by users and translates to lost revenue, poor customer experience, and loss of brand reputation and loyalty for the business.
The tests show that NetScaler can handle more concurrent connections/customers per ADC without users experiencing degradation.
17% of shoppers abandoned their carts because the website crashed or had other errors.
Statista
Financial services
Financial services is another industry where application response time plays a critical role. Poor application response times can lead to critical issues such as failed transactions, unavailability of online banking services, and significant customer dissatisfaction, potentially damaging the institution’s reputation and financial performance. An ADC with efficient CPU usage, lower latency, and higher throughput enhances user experience and engagement, giving financial services companies a competitive advantage over their rivals.
Healthcare
High latency or slow application response could result in a delay in booking an appointment, which in turn could result in a delay in getting a diagnosis or medication, negatively impacting a patient’s health.
For the healthcare provider, it could result in delays in accessing electronic health records (EHR) and downtime leading to delays in patient care, which can have a long-lasting impact on the patient’s health and the healthcare provider’s reputation and revenue.
Summary of test results
The test results conclude that NetScaler can process high volumes of traffic while using fewer CPU resources and achieving higher throughput than F5. NetScaler’s ability to provide multiple services including traffic management, policy enforcement, and application security makes it an indispensable component of modern IT infrastructure.
Fewer, smaller ADCs required: NetScaler’s efficiency means that fewer and smaller NetScaler ADC instances are needed to handle the same workload compared to F5, which can significantly reduce infrastructure complexity and costs.
Latency matters: Lower latency directly translates to better application performance and application end-user experience. NetScaler’s one-pass architecture ensures less latency for faster application response times even under heavy loads.
Faster, more reliable application response: With NetScaler, applications respond faster and more reliably, enhancing overall application end-user productivity and satisfaction.
The tests conducted by The Tolly Group demonstrate that NetScaler not only outperforms F5 in CPU efficiency, latency, and throughput, but also delivers tangible benefits such as reduced infrastructure complexity and lower operational costs that can significantly impact such business outcomes as improved customer satisfaction, enhanced user experience, and increased revenue.
With NetScaler, enterprises can achieve more efficient and faster application performance with fewer and/or smaller ADCs when compared with F5, resulting in reduced operational complexity and costs and an optimal application end-user experience.
NetScaler’s superior performance in CPU efficiency, latency, and throughput makes NetScaler the clear choice for enterprises looking to optimize their application delivery and security.
Read the complete report: Performance Benchmarking NetScaler VPX vs. F5 BIG-IP Virtual Edition (no sign-up required)