Translated from 日本語 • View original 
Did you know that about 40% of all traffic on modern websites comes from bots?
Some of these include "good bots" like search engine crawlers, while others include "bad bots" that aim to buy up inventory or take over accounts.
The problem is that bots these days have become so sophisticated that they have evolved to a level that is indistinguishable from humans.
In this article, we'll show you a practical way to use Azure WAF's Bot Protection feature to ensure that good bots are welcome but bad bots are blocked.
By properly setting these settings, you can "operate a website that is not swayed by bots".
Why is Bot Management Important?
Poor bot management can lead to the following problems:
- Waste of resources
Malicious bots can take over server resources and affect access for legitimate users.
CDN and server costs will also increase unnecessarily. - Direct damage to the business
It can cause direct damage to your business, such as hoarding inventory, scraping price information, or brute force attacks on your account. - Data distortion
Analytics data is tainted with bot traffic, preventing accurate business decisions. - Negative impact on SEO
If your site is heavily visited by malicious bots, your search engine reputation will also drop.
Let's take a look at specific ways to use Azure WAF's Bot Protection feature.
Azure WAF Bot Protection Basic Setup
1. Enabling Bot Protection Rulesets
Azure Application Gateway WAF v2 provides a Bot Protection ruleset.
First, open the WAF policy in the Azure portal and configure the settings from the management rules.
WAF Policies > Management Rules > Bot Protection Rulesets
There are three actions that can be configured:
- Log: Detect and log bots (recommended for initial deployment)
- Block: Block malicious bots
- Redirect: Redirect to a specific page
We recommend starting in "Log" mode during initial deployment and then moving to "Block" mode after reviewing actual traffic patterns.
2. Understanding Bot Categories
Azure WAF bots are roughly categorized as follows:
- Search engine
- Social Media
- Other benign bots
- Malicious bots
- Unknown bots
Practical Anti-Bot Techniques
1. Utilizing Rate Limiting Rules
Rate limiting is very effective in anti-bot measures.
Azure WAF allows you to implement rate limiting in custom rules.
For example, if there are more than 200 accesses from a specific IP address in one minute, it is possible to block that communication.
2. Filtering Based on Geographic Location
If your business is geographically limited, geographic filtering is also effective.
If you only want to access from within Japan, you can enforce geographical restrictions with custom rules to deny access from overseas.
3. Restricting Access to Specific Paths
Additional protection can also be provided for critical endpoints (login pages, APIs, etc.).
For example, if you want to impose stricter restrictions on the /login page, you can use custom rules to control it.
Monitoring and Tuning
1. Log analysis in Azure Monitor
Regular log analysis is important to measure the effectiveness of Bot Protection.
Check logs over a period of time to check for false positives for business communications and legitimate access from customers.
2. Identifying and Addressing False Positives
To reduce false positives, check the following metrics regularly:
- Blocks from the same IP in a short period of time
- Continuous blocking of specific User-Agents
- Complaints from authorized users
3. Performance Impact Assessment
Before and after Bot Protection activation, you may want to compare the following:
- Average response time
- Server resource utilization
- Success rate for regular users
Best Practices
1. Phased Implementation Approach
If you release in a "blocked" state from the beginning, there is a high possibility that your business will be affected by unexpected false positives.
We recommend that you switch from "Detect" to "Block" by following the steps below.
- Stage 1
- All rules are operated in "Log" mode
- Traffic pattern analysis
- second stage
- Only obviously malicious bots have been changed to "Block"
- Enhanced monitoring of false positives
- Stage 3
- Full-scale operation begins
- Add custom rules
2. Continuous Improvement Cycle
WAF rules don't end there once you're in place.
Regularly review the rules as follows to ensure safer operations.
- Weekly review: Checking for new bot patterns
- Monthly Report: Overall Effectiveness Measurement
- Quarterly: Major review of ruleset
3. Aligning with Other Security Measures
Introducing a WAF can increase the level of security, but by introducing and operating other solutions together, more secure service operations can be expected.
- Combined with DDoS Protection
- Unified monitoring in Azure Sentinel
- Application-side monitoring in Application Insights
Summary: Combating Evolving Bots
Azure WAF's Bot Protection feature can be a powerful tool for protecting web applications from various bot threats when properly configured and operational.
Key takeaways are:
- Data-Driven Approach: Make decisions based on actual log data, not guesswork
- Gradual enhancement: Optimize gradually, not suddenly set strict settings
- Continuous Monitoring: Bot techniques evolve daily, making regular reviews essential
If you are a company that wants to learn more about how to implement and operate Azure WAF and how to best configure it for your environment, please contact Colorkrew.