Connect Application Insights to Microsoft Copilot Studio for Real-Time Monitoring

Monitoring your Copilot Studio agents is essential for understanding user interactions, identifying issues, and optimizing performance. Azure Application Insights provides comprehensive real-time monitoring and analytics capabilities for your copilot conversations. This guide walks you through connecting Application Insights to your Copilot Studio agent, enabling detailed telemetry, conversation tracking, and performance monitoring.

Table of Contents

Why Application Insights for Copilot Studio?

Azure Application Insights provides powerful monitoring and analytics capabilities specifically designed for understanding how users interact with your Copilot Studio agents.

Key Benefits

  • ✅ Real-Time Monitoring: Track conversations as they happen
  • ✅ Performance Metrics: Monitor response times, success rates, and errors
  • ✅ User Analytics: Understand user behavior and conversation patterns
  • ✅ Conversation Transcripts: Capture detailed conversation logs with enhanced metadata
  • ✅ Custom Queries: Use KQL (Kusto Query Language) to analyze telemetry data
  • ✅ Alerts and Notifications: Set up alerts for specific events or thresholds
  • ✅ Integration: Connect with Azure Monitor, Power BI, and other Azure services

Use Cases

  • Performance Optimization: Identify slow responses or bottlenecks
  • User Experience Analysis: Understand where users drop off or get confused
  • Error Tracking: Quickly identify and resolve issues
  • Compliance and Auditing: Maintain detailed logs of all interactions
  • Usage Insights: Track adoption and usage patterns
  • A/B Testing: Compare performance of different copilot configurations

Prerequisites

Before connecting Application Insights to your Copilot Studio agent, ensure you have:

  • Azure Subscription: Active Azure subscription with appropriate permissions
  • Copilot Studio Access: Owner or contributor access to the copilot agent
  • Azure Permissions: Ability to create Application Insights resources
  • Copilot Studio Agent: An existing copilot agent to monitor

Required Roles:

  • Azure: Contributor or Owner role on the resource group
  • Copilot Studio: Owner or Editor role on the copilot

Step 1: Create Application Insights Resource

Application Insights is an Azure service that collects and analyzes telemetry data from your applications.

Create via Azure Portal

  1. Navigate to Azure Portal

  2. Create New Resource

    • Click Create a resource
    • Search for Application Insights
    • Click Create

  3. Configure Basic Settings

    Create Application Insights

    • Subscription: Select your Azure subscription
    • Resource Group: Choose existing or create new
    • Name: Enter a descriptive name (e.g., copilot-agent-insights)
    • Region: Select the region closest to your users
    • Resource Mode: Choose Workspace-based (recommended)
    • Log Analytics Workspace: Select existing or create new

  4. Review and Create

    • Review your configuration
    • Click Create
    • Wait for deployment to complete (usually 1-2 minutes)

Create via Azure CLI

# Variables
resourceGroup="copilot-rg"
location="eastus"
workspaceName="copilot-workspace"
appInsightsName="copilot-agent-insights"

# Create resource group (if needed)
az group create --name $resourceGroup --location $location

# Create Log Analytics workspace
az monitor log-analytics workspace create \
    --resource-group $resourceGroup \
    --workspace-name $workspaceName \
    --location $location

# Get workspace ID
workspaceId=$(az monitor log-analytics workspace show \
    --resource-group $resourceGroup \
    --workspace-name $workspaceName \
    --query id -o tsv)

# Create Application Insights
az monitor app-insights component create \
    --app $appInsightsName \
    --location $location \
    --resource-group $resourceGroup \
    --workspace $workspaceId

Output PowerShell

Create via PowerShell

# Variables
$resourceGroup = "copilot-rg"
$location = "East US"
$workspaceName = "copilot-workspace"
$appInsightsName = "copilot-agent-insights"

# Create resource group (if needed)
New-AzResourceGroup -Name $resourceGroup -Location $location

# Create Log Analytics workspace
$workspace = New-AzOperationalInsightsWorkspace `
    -ResourceGroupName $resourceGroup `
    -Name $workspaceName `
    -Location $location

# Create Application Insights
New-AzApplicationInsights `
    -ResourceGroupName $resourceGroup `
    -Name $appInsightsName `
    -Location $location `
    -WorkspaceResourceId $workspace.ResourceId

Write-Host "Application Insights created successfully!" -ForegroundColor Green

Output

Output PowerShell

Application Insights resource

Application Insights created

Step 2: Retrieve the Connection String

After creating the Application Insights resource, you need to retrieve the connection string to link it with your Copilot Studio agent.

Method 1: Azure Portal

  1. Navigate to Application Insights Resource

    • In Azure Portal, go to your Application Insights resource
    • You can find it in All resources or in your resource group

  2. Copy the Connection String

    Copy Connection String

    • In the Overview page, locate the Connection String
    • Click the Copy icon next to the connection string
    • Save it for the next step

    Note: The connection string looks like:

    InstrumentationKey=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx;IngestionEndpoint=https://region.applicationinsights.azure.com/;LiveEndpoint=https://region.livediagnostics.monitor.azure.com/

Method 2: Azure CLI

# Get connection string
az monitor app-insights component show \
    --app $appInsightsName \
    --resource-group $resourceGroup \
    --query connectionString -o tsv

Method 3: PowerShell

# Get Application Insights details
$appInsights = Get-AzApplicationInsights `
    -ResourceGroupName $resourceGroup `
    -Name $appInsightsName

    # Display connection string
    Write-Host "Connection String:" -ForegroundColor Cyan
    Write-Host $appInsights.ConnectionString
    
    # Display instrumentation key
    Write-Host "`nInstrumentation Key:" -ForegroundColor Cyan
    Write-Host $appInsights.InstrumentationKey

Now that you have the connection string, link it to your Copilot Studio agent to start collecting telemetry.

Configure in Copilot Studio

  1. Open Copilot Studio

  2. Access Settings

    • Click on Settings (gear icon) in the top right
    • Navigate to Advanced section
    • Select Application Insights

  3. Enter Connection Details

    Link Application Insights to Copilot Agent

    • Paste the Connection String you copied from Azure into the Connection String
    • Click Save

Important Configuration Notes

  • Data Retention: By default, Application Insights retains data for 90 days
  • Sampling: For high-volume copilots, consider configuring sampling to manage costs
  • Privacy: Ensure you comply with data privacy regulations when logging conversations
  • Cost: Application Insights charges based on data ingestion volume

Step 4: Verify Configuration

After linking Application Insights, verify that telemetry data is being collected correctly.

Step 5: Check Metadata in Application Insights within Copilo Studio Agent

Check Metadata of Application Insights

Note: The Log Sensitive Types toggle, contains User ID, Name and Text of the prompt.

Privacy Considerations - Review what personal data will be logged - Ensure compliance with GDPR, HIPAA, and other regulations - Configure data masking if needed - Document data retention policies

Step 6: Test

  1. Test Your Copilot
    • Go back to Copilot Studio
    • Test your copilot by having a conversation
    • Return to Application Insights after a few minutes

Telemetry captured

  1. Query Telemetry Data
    • In Application Insights, click Logs in the left menu
    • Run a simple query to verify data:
   // View recent custom events
customEvents
| where cloud_RoleInstance == "Holiday Buddy"  and customDimensions["channelId"] == "pva-studio"
| project timestamp, name, session_Id, user_Id, cloud_RoleInstance, customDimensions, customDimensions["TopicName"], customDimensions["text"], customDimensions["speak"]

KQL App Insights

Step 7: Enable Enhanced Transcripts

Enhanced transcripts provide detailed conversation logs with additional metadata for deeper analysis.

Enable Enhanced Logging

  1. Access Analytics Settings

    • In Copilot Studio, go to Settings > Advanced >Application Insights
    • Look for Enhanced transcripts option

  2. Enable Transcript Logging

    • Toggle Enable enhanced transcripts to On Enable Enhanced Transcripts

    Details about nodes, such as name, type, and start and end times, can be saved to the transcript stored in Dataverse.

References