Translated from 日本語 • View original 
"Too many logs, inflating storage costs"
"I put everything in Sentinel, but I don't refer to it very much."
"I want to focus on long-term storage and trend analysis rather than real-time detection."
An option to consider is to integrate with Azure Data Lake Storage (ADLS).
Sentinel alone is not practical
Sentinel's backend, Log Analytics , is a
It is optimized for real-time detection and hunting.
However, when the amount of logs becomes enormous, the following challenges become apparent.
- Increased costs: Costs skyrocket due to long-term retention and large log storage
- Poor performance: Queries and visualizations become heavier
- Analysis limitations: KQL-centric operation makes it difficult to integrate with BI tools and AI.
As a result, it is necessary to separate "data used for monitoring" and "data used for analysis and storage".
Large-scale logs are entering the era of "direct data lake"
In recent years, it has become common to divide the destinations according to the nature of the data.
Image of the dataflow:
- Log Generator
- Logs that require real-time analysis → Microsoft Sentinel (Analytics tier)
- Bulk, archival logs → Azure Data Lake Storage (Data Lake tier)
Sentinel only leaves logs that are "used for detection and correlation now",
Other large data (e.g., raw Firewall logs, proxies, system audit logs)
Adopt a configuration that sends directly to the Data Lake.
This separation allows you to achieve three things at the same time:
- Maintain real-time detection performance
- Low-cost long-term storage of large logs
- Expansion of the analysis platform that can be flexibly used with BI and AI at a later date
Key Benefits of Data Lake Integration
| Benefits | Contents |
|---|---|
| Cost Optimization | Lower Sentinel Retention Costs and Cheaper Storage with ADLS |
| Freedom of Analysis | Easily integrate with Power BI, Synapse, Databricks, and more |
| Long-term storage | Easy to retain history and comply with laws and regulations in multi-year increments |
| Future-proof Scalability | Available as a Data Source for Machine Learning and Anomaly Detection Models |
How Colorkrew Security Can Help
Colorkrew Security provides the following
We design and implement the integrated architecture of "Sentinel × Data Lake".
- Log collection design (which data goes to Sentinel and which data to the data lake)
- Transfer and storage configuration design with Azure Data Lake Storage
- Building a visualization platform through Synapse/Power BI linkage
- Optimize costs and develop retention policies
- Establish a log utilization model that connects SOC operations and analysis teams
From Collect All Logs,
Go to "Put data where you need it, in the way you need it."
Colorkrew Security can help you design and operate it consistently.