Microsoft Sentinel Deep Dive SEPT. 2023 Update 

Please ask your question here.

@@theacademyhub Thank you for your message. My question is as follows: TLDR: Is it possible to utilize Logstash along with the "microsoft-sentinel-log-analytics-logstash-output-plugin" output plugin to send CEF logs to a standard table like CommonSecurityLog, without the necessity to craft a specific transformation for each log source? Similar to the way CEF-formatted logs are directed to the Azure Monitor Agent (AMA), which seamlessly ingests the logs into the appropriate location. Some context: We are a reasonably sized company with around 10,000 users, heavily relying on Microsoft/Azure products. Currently transitioning from Splunk to Sentinel for our SIEM needs, we encountered an issue with the limited buffer size (10 GB) of an AMA agent, which is insufficient for our daily data volume of 1 TB. Concerned about potential data loss during network hiccups due to the quickly filling buffer, we restructured our on-premises data collection architecture. We opted for Logstash using the "microsoft-sentinel-log-analytics-logstash-output-plugin," drawing inspiration from this article (learn.microsoft.com/en-us/azure/sentinel/connect-logstash-data-connection-rules). While successfully sending syslogs to custom tables, we are facing challenges when attempting to send CEF-formatted logs to a standard table. The result is empty entries in the table with no parsing. The data we send conforms to the format outlined in the sample file at the bottom of this page: github.com/Azure/Azure-Sentinel/tree/master/DataConnectors/microsoft-sentinel-log-analytics-logstash-output-plugin. Although we could potentially resolve this by creating a transformation to precisely match each field, we believe this is not future-proof and not the ideal approach. Are we overlooking something? Despite consulting documentation and blogs, I have not found a resource addressing this specific problem. I appreciate your assistance and your valuable content. Thank you!

Thank you for your question. I do believe it’s possible. I would start here: 1. Verify that your CEF logs are compliant with the CEF standard and that they include the required fields. Use learn.microsoft.com/en-us/azure/sentinel/cef-name-mapping to validate your CEF logs. 2. Ensure Logstash has the right configuration. a. The input plugin should use the codec cef to parse the CEF logs. input { tcp { port => XYZ codec => cef } } b. Rename the CEF fields to match the CommonSecurityLog field names if you haven’t done so already. Use learn.microsoft.com/en-us/azure/azure-monitor/reference/tables/commonsecuritylog as a reference. rename => { "deviceVendor" => "DeviceVendor" "deviceProduct" => "DeviceProduct" "deviceVersion" => "DeviceVersion" "deviceEventClassId" => "DeviceEventClassID" "name" => "Name" "severity" => "Severity" …so on for other fields } c. The output plugin should use “microsoft-sentinel-log-analytics-logstash-output-plugin” to send the CEF logs to the CommonSecurityLog table. You need to specify the log_type as CommonSecurityLog and the time_generated_field as end. log_type => "CommonSecurityLog" time_generated_field => "end" d. Restart the Logstash service and check the CommonSecurityLog table, let me know if this worked.

The PPT is linked directly above the video.