Add: New azure-data-collection-rules skill for Azure Monitor DCR authoring

plugin/skills/azure-data-collection-rules/SKILL.md

@@ -0,0 +1,44 @@
+---
+name: azure-data-collection-rules
+description: "Author, edit, validate, and deploy Azure Monitor Data Collection Rules (DCRs), Log Analytics workspace tables, and KQL ingestion-time transformations. Covers single-stage and multi-stage transformation DCRs, client-side and ingestion-side processors, stream declarations, and custom table creation. Also covers direct ingestion DCRs for the Log Ingestion API. WHEN: create DCR, edit DCR, data collection rule, DCR JSON, add transformation, KQL transform, custom table, stream declaration, multi-stage transformation, processor, client-side transform, ingestion-time transform, parse JSON logs, filter syslog, aggregate events, custom log table, DCR schema, DCR authoring, rename columns, drop columns, CEF parsing, XML parsing, data collection, Log Ingestion API, direct ingestion, send custom logs, custom log ingestion, logs ingestion endpoint, DCR endpoint."
+argument-hint: "Describe the data source type, desired transformations, and destination table"
+license: MIT
+metadata:
+  author: Microsoft
+  version: "1.0.0"
+---
+
+# DCR Authoring Skill
+
+Author, validate, and deploy Azure Monitor Data Collection Rules with single-stage, multi-stage, or direct ingestion configurations.
+
+## Procedure
+
+Follow the [full procedure](./references/procedure.md):
+
+1. **Gather requirements** — ingestion method, data source, intent, destination, split/copy needs
+2. **Determine DCR kind** — per [DCR kinds guide](./references/dcr-kinds.md)
+3. **Design transformation pipeline** — native filters, processors, or `transformKql`
+4. **Author the DCR** — per [DCR schema](./references/dcr-schema.md) and [routing rules](./references/destination-routing.md)
+5. **Validate** — run [validate-dcr.ps1](./scripts/validate-dcr.ps1)
+6. **Deploy** — run [put-dcr.ps1](./scripts/put-dcr.ps1), prepare tables via [create-custom-table.ps1](./scripts/create-custom-table.ps1)
+7. **Verify** — query destination table, check `_LogOperation` for errors
+
+## References
+
+- [Procedure](./references/procedure.md) — full step-by-step workflow
+- [DCR kinds](./references/dcr-kinds.md) — kind selection, data source types, transformation sections
+- [DCR schema](./references/dcr-schema.md) — top-level structure, column constraints, dataFlows, transformations, REST API
+- [Stream declarations](./references/stream-declarations.md) — custom stream schemas (Direct + logFiles only)
+- [Processors: headers](./references/processors-headers.md) — header processor types, stage availability, output columns
+- [Processors: operations](./references/processors-operations.md) — filter, map, parse, aggregate, enrich, KQL syntax
+- [Processor heuristics: filters](./references/processor-heuristics-filters.md) — native filter check, filtering intent map
+- [Processor heuristics: transforms](./references/processor-heuristics-transforms.md) — parsing, schema, aggregation, enrichment, routing intent maps
+- [Processor heuristics: staging](./references/processor-heuristics-staging.md) — stage placement, cost optimization, multi-processor chains
+- [Destination routing](./references/destination-routing.md) — stream-to-table mapping
+- [Supported tables](./references/supported-tables.md) — standard tables accepting custom streams
+- [KQL transforms](./references/kql-transforms.md) — common KQL patterns
+- [LA tables](./references/la-tables.md) — table creation, plans
+- [Direct ingestion](./references/direct-ingestion.md) — Log Ingestion API DCRs
+- [Decision guide](./references/decision-guide.md) — scenario routing table
+- [Limits](./references/limits.md) — DCR structure limits, column constraints, API quotas

plugin/skills/azure-data-collection-rules/examples/custom-json-log.json

@@ -0,0 +1,75 @@
+{
+    "location": "eastus",
+    "properties": {
+        "streamDeclarations": {
+            "Custom-JsonAppLogs": {
+                "columns": [
+                    { "name": "TimeGenerated", "type": "datetime" },
+                    { "name": "Computer", "type": "string" },
+                    { "name": "Level", "type": "string" },
+                    { "name": "UserId", "type": "string" },
+                    { "name": "RequestId", "type": "string" },
+                    { "name": "Message", "type": "string" },
+                    { "name": "Duration", "type": "real" }
+                ]
+            }
+        },
+        "dataSources": {
+            "logFiles": [
+                {
+                    "name": "appJsonLogs",
+                    "filePatterns": ["/var/log/myapp/*.json"],
+                    "format": "json",
+                    "streams": ["Custom-JsonAppLogs"],
+                    "transform": "parse_app_json"
+                }
+            ]
+        },
+        "destinations": {
+            "logAnalytics": [
+                {
+                    "name": "myWorkspace",
+                    "workspaceResourceId": "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.OperationalInsights/workspaces/{ws}",
+                    "workspaceId": "{workspace-guid}"
+                }
+            ]
+        },
+        "dataFlows": [
+            {
+                "streams": ["Custom-JsonAppLogs"],
+                "destinations": ["myWorkspace"],
+                "outputStream": "Custom-AppLogs_CL",
+                "transformKql": "source | where Level != 'Debug' | project-away Duration"
+            }
+        ],
+        "transformations": [
+            {
+                "name": "parse_app_json",
+                "headerProcessor": {
+                    "processor": "header.TextLog"
+                },
+                "processors": [
+                    {
+                        "processor": "parse.JsonPath",
+                        "configuration": {
+                            "columnName": "RawData",
+                            "all": [
+                                { "path": "$.level", "nameAs": "Level", "typeAs": "string" },
+                                { "path": "$.userId", "nameAs": "UserId", "typeAs": "string" },
+                                { "path": "$.requestId", "nameAs": "RequestId", "typeAs": "string" },
+                                { "path": "$.message", "nameAs": "Message", "typeAs": "string" },
+                                { "path": "$.durationMs", "nameAs": "Duration", "typeAs": "real" }
+                            ]
+                        }
+                    },
+                    {
+                        "processor": "map.Drop",
+                        "configuration": {
+                            "columnNames": ["RawData", "FilePath"]
+                        }
+                    }
+                ]
+            }
+        ]
+    }
+}

plugin/skills/azure-data-collection-rules/examples/direct-ingestion-custom-table.json

@@ -0,0 +1,35 @@
+{
+    "location": "eastus",
+    "kind": "Direct",
+    "properties": {
+        "streamDeclarations": {
+            "Custom-MyAppLogs": {
+                "columns": [
+                    { "name": "Time", "type": "datetime" },
+                    { "name": "Computer", "type": "string" },
+                    { "name": "Application", "type": "string" },
+                    { "name": "Level", "type": "string" },
+                    { "name": "Message", "type": "string" },
+                    { "name": "RequestId", "type": "string" },
+                    { "name": "DurationMs", "type": "real" }
+                ]
+            }
+        },
+        "destinations": {
+            "logAnalytics": [
+                {
+                    "workspaceResourceId": "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.OperationalInsights/workspaces/{ws}",
+                    "name": "myworkspace"
+                }
+            ]
+        },
+        "dataFlows": [
+            {
+                "streams": ["Custom-MyAppLogs"],
+                "destinations": ["myworkspace"],
+                "transformKql": "source | project TimeGenerated = Time, Computer, Application, Level, Message, RequestId, DurationMs | where Level != 'Debug'",
+                "outputStream": "Custom-AppLogs_CL"
+            }
+        ]
+    }
+}

plugin/skills/azure-data-collection-rules/examples/perf-counter-aggregation.json

@@ -0,0 +1,72 @@
+{
+    "location": "eastus",
+    "properties": {
+        "streamDeclarations": {
+            "Custom-PerfSummary": {
+                "columns": [
+                    { "name": "TimeGenerated", "type": "datetime" },
+                    { "name": "Host", "type": "string" },
+                    { "name": "CounterName", "type": "string" },
+                    { "name": "AvgValue", "type": "real" },
+                    { "name": "MaxValue", "type": "real" },
+                    { "name": "RecordCount", "type": "int" }
+                ]
+            }
+        },
+        "dataSources": {
+            "performanceCounters": [
+                {
+                    "name": "perfCpu",
+                    "samplingFrequencyInSeconds": 10,
+                    "counterSpecifiers": [
+                        "\\Processor(_Total)\\% Processor Time",
+                        "\\Memory\\Available MBytes"
+                    ],
+                    "streams": ["Custom-PerfSummary"],
+                    "transform": "aggregate_perf"
+                }
+            ]
+        },
+        "destinations": {
+            "logAnalytics": [
+                {
+                    "name": "myWorkspace",
+                    "workspaceResourceId": "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.OperationalInsights/workspaces/{ws}",
+                    "workspaceId": "{workspace-guid}"
+                }
+            ]
+        },
+        "dataFlows": [
+            {
+                "streams": ["Custom-PerfSummary"],
+                "destinations": ["myWorkspace"],
+                "outputStream": "Custom-PerfSummary_CL"
+            }
+        ],
+        "transformations": [
+            {
+                "name": "aggregate_perf",
+                "headerProcessor": {
+                    "processor": "header.WindowsPerformanceCounters"
+                },
+                "processors": [
+                    {
+                        "processor": "aggregate.Basic",
+                        "configuration": {
+                            "batchingSettings": {
+                                "timeWindow": "5m",
+                                "maxBatchRows": 1000
+                            },
+                            "aggregates": [
+                                { "columnName": "CounterValue", "operator": "avg", "nameAs": "AvgValue" },
+                                { "columnName": "CounterValue", "operator": "max", "nameAs": "MaxValue" },
+                                { "operator": "count", "nameAs": "RecordCount" }
+                            ],
+                            "dimensionColumns": ["Host", "CounterName"]
+                        }
+                    }
+                ]
+            }
+        ]
+    }
+}

plugin/skills/azure-data-collection-rules/examples/syslog-filter-drop.json

@@ -0,0 +1,54 @@
+{
+    "location": "eastus",
+    "properties": {
+        "dataSources": {
+            "syslog": [
+                {
+                    "name": "syslogAuth",
+                    "facilityNames": ["auth", "authpriv"],
+                    "logLevels": ["Warning", "Error", "Critical", "Alert", "Emergency"],
+                    "streams": ["Microsoft-Syslog"],
+                    "transform": "drop_auth_columns"
+                },
+                {
+                    "name": "syslogOther",
+                    "facilityNames": ["daemon", "kern"],
+                    "logLevels": ["Error", "Critical", "Alert", "Emergency"],
+                    "streams": ["Microsoft-Syslog"]
+                }
+            ]
+        },
+        "destinations": {
+            "logAnalytics": [
+                {
+                    "name": "myWorkspace",
+                    "workspaceResourceId": "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.OperationalInsights/workspaces/{ws}",
+                    "workspaceId": "{workspace-guid}"
+                }
+            ]
+        },
+        "dataFlows": [
+            {
+                "streams": ["Microsoft-Syslog"],
+                "destinations": ["myWorkspace"],
+                "outputStream": "Microsoft-Syslog"
+            }
+        ],
+        "transformations": [
+            {
+                "name": "drop_auth_columns",
+                "headerProcessor": {
+                    "processor": "header.Syslog"
+                },
+                "processors": [
+                    {
+                        "processor": "map.Drop",
+                        "configuration": {
+                            "columnNames": ["ProcessId", "HostIP"]
+                        }
+                    }
+                ]
+            }
+        ]
+    }
+}

plugin/skills/azure-data-collection-rules/examples/windows-events-split.json

@@ -0,0 +1,83 @@
+{
+    "location": "eastus",
+    "properties": {
+        "streamDeclarations": {
+            "Custom-SecurityEvents": {
+                "columns": [
+                    { "name": "TimeGenerated", "type": "datetime" },
+                    { "name": "Computer", "type": "string" },
+                    { "name": "EventID", "type": "int" },
+                    { "name": "Channel", "type": "string" },
+                    { "name": "UserName", "type": "string" },
+                    { "name": "EventDescription", "type": "string" }
+                ]
+            }
+        },
+        "dataSources": {
+            "windowsEventLogs": [
+                {
+                    "name": "securityEvents",
+                    "xPathQueries": [
+                        "Security!*[System[(EventID=4624 or EventID=4625 or EventID=4648)]]"
+                    ],
+                    "streams": ["Custom-SecurityEvents"],
+                    "transform": "parse_security_events"
+                },
+                {
+                    "name": "systemEvents",
+                    "xPathQueries": [
+                        "System!*[System[(Level=1 or Level=2 or Level=3)]]"
+                    ],
+                    "streams": ["Microsoft-Event"]
+                }
+            ]
+        },
+        "destinations": {
+            "logAnalytics": [
+                {
+                    "name": "myWorkspace",
+                    "workspaceResourceId": "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.OperationalInsights/workspaces/{ws}",
+                    "workspaceId": "{workspace-guid}"
+                }
+            ]
+        },
+        "dataFlows": [
+            {
+                "streams": ["Custom-SecurityEvents"],
+                "destinations": ["myWorkspace"],
+                "outputStream": "Custom-SecurityEvents_CL"
+            },
+            {
+                "streams": ["Microsoft-Event"],
+                "destinations": ["myWorkspace"],
+                "outputStream": "Microsoft-Event"
+            }
+        ],
+        "transformations": [
+            {
+                "name": "parse_security_events",
+                "headerProcessor": {
+                    "processor": "header.WindowsEvents"
+                },
+                "processors": [
+                    {
+                        "processor": "parse.XmlPath",
+                        "configuration": {
+                            "columnName": "RawXml",
+                            "all": [
+                                { "path": "/Event/System/EventID", "nameAs": "EventID", "typeAs": "int" },
+                                { "path": "/Event/EventData/Data[@Name='TargetUserName']", "nameAs": "UserName", "typeAs": "string" }
+                            ]
+                        }
+                    },
+                    {
+                        "processor": "map.Drop",
+                        "configuration": {
+                            "columnNames": ["RawXml", "RenderingInfo", "EventRecordId", "PublisherId"]
+                        }
+                    }
+                ]
+            }
+        ]
+    }
+}