Here at YOTTAA, we are always looking to make performance improvements on anything and everything. From speeding up top eCommerce websites, to providing high-powered coffee machines in our own office kitchens (caffeine is a must). We, as a company, live and breathe efficiency. 

Of course, the security we provide our customers is no exception. As a part of our security offering, we leverage some of the components of the ModSecurity WAF (Web Application Firewall). 

ModSecurity is an open-source WAF that is developed by Trustwave SpiderLabs. It has a comprehensive set of capabilities that allow rule customization to reduce false positives (traffic blocks that shouldn’t have happened) and false negatives (traffic that was incorrectly allowed to pass).  

While evaluating an updated version of ModSecurity, some of our engineers found that the behavior of the newer version was markedly different from the older version for certain types of web traffic. Initially, it wasn’t clear if this was a ModSecurity issue, an intentional change, or a problem with our integration.

Here is what our engineers learned and how they helped improve ModSecurity for all.

Want to know more about YOTTAA’s dedication to efficiency? Learn how we speed up eCommerce websites, and what we can do for yours!

A storm is coming

We rely on NGINX’s flexibility and speed for multiple services, and wanted to use it in our security offering as well.  The challenge was finding an efficient way to leverage ModSecurity with NGINX. For this, we decided to evaluate ModSecurity 3 in order to obtain better NGINX support, a fix for inconsistent audit logs, and improved performance. 

It turns out that ModSecurity 3 is a completely different implementation of a WAF, and ModSecurity 2 had been entirely rewritten to create this newer version. Since ModSecurity 3 is a re-write, some features may be removed, optimized, or modified. This was an indicator that transitioning to the new product would require significant regression testing to ensure ModSecurity 3 was back-compatible with ModSecurity 2 in areas that are important to us.

“The takeaway is that we need to go to ModSecurity 3, but it’s like changing vendors, like switching from Ford to Chevrolet” says Mike Melo, Senior Software Engineer for YOTTAA, confirming that the transition would be less than smooth sailing. 

The entrance into choppy waters

With the warning signs heeded, we kept our customers top of mind when planning out the product transition. 

Because ModSecurity 3 could start raising some false flags, i.e. customers thinking they had more malicious traffic than they actually did, we wanted to tread carefully and avoid as many disturbances as possible. 

In order to be cautious, we set up multiple test environments. During one of our deployments, we took the old ModSecurity 2, and a new ModSecurity 3, and we connected them (with no impact on customer traffic) in order to compare behavior. We ran traffic through ModSecurity 2, where it would decide to block or pass the traffic. Then, the traffic would proceed into ModSecurity 3. After running them for a couple days, we looked at the report cards. The results indicated a problem.

Our team found that there was a massive discrepancy in the amount of traffic blocked by each product. For example, ModSecurity 2 would indicate 20 percent fewer blocks than ModSecurity 3 would record in the same timeframe. So, we drilled down to figure out why ModSecurity 3 found those extra blocks.

Navigating the course

When we looked at the 20 percent of traffic blocks more closely, our team realized that the extra blocks were being suppressed in ModSecurity 2 by two rule configuration directives. 

YOTTAA had used ModSecurity configuration directives to modify existing rules for specific customer sites in order to avoid false positives. The configuration was based on ModSecurity 2. When ModSecurity 3 was written, the configuration directives were implemented differently, which caused the YOTTAA modifications to have no effect. This resulted in the additional traffic blocks. 

 After we had found the origin of the issue, we decided to contact the ModSecurity development team at Trustwave SpiderLabs.

“The reason why we were very hesitant to fix it ourselves was because unless we could get the ModSecurity team to agree that this fix was done correctly, and that they would accept the fix upstream to the GitHub repo, then suddenly we would become an island with respect to this product. Every time they update the product — and it’s a very active product — we would have to keep merging this fix in… and that turns into a bad situation” explains Melo.

We filed a bug with Trustwave SpiderLabs, submitting the complete data set of all the information we had found, as well as our belief as to the source code that needed to be changed to fix the issue. 

Improving ModSecurity 3 for all

We worked with Modsecurity’s lead developer, architect, and maintainer of ModSecurity 3, Felipe “Zimmerle” Costa, to see if they fundamentally bought into the product fix. Our team also wanted to make sure that they would send the fix upstream, so that it would be included in any future updates for all users. 

Felipe agreed that the bug we found was indeed an issue, and that the problem in the code was where we both thought it was. Then, they implemented the fix, and let us know so we could test it. 

Initially, the fix didn’t work with our customized rules, perhaps the result of a small oversight during the code refactoring. When we pointed that out, Felipe accepted our diff, and the bug was finally resolved. They even put YOTTAA in the product CHANGES file for catching the oversight, supporting the fact that YOTTAA’s dedication to efficiency bleeds into all areas of our business.

Want to know more about YOTTAA’s dedication to efficiency? Learn how we speed up eCommerce websites, and what we can do for yours!