Connect Application Insights to Microsoft Copilot Studio for Real-Time Monitoring

Monitoring your Copilot Studio agents is essential for understanding user interactions, identifying issues, and optimizing performance. Azure Application Insights provides comprehensive real-time monitoring and analytics capabilities for your copilot conversations. This guide walks you through connecting Application Insights to your Copilot Studio agent, enabling detailed telemetry, conversation tracking, and performance monitoring.

Table of Contents

Why Application Insights for Copilot Studio?

Azure Application Insights provides powerful monitoring and analytics capabilities specifically designed for understanding how users interact with your Copilot Studio agents.

Key Benefits

  • ✅ Real-Time Monitoring: Track conversations as they happen
  • ✅ Performance Metrics: Monitor response times, success rates, and errors
  • ✅ User Analytics: Understand user behavior and conversation patterns
  • ✅ Conversation Transcripts: Capture detailed conversation logs with enhanced metadata
  • ✅ Custom Queries: Use KQL (Kusto Query Language) to analyze telemetry data
  • ✅ Alerts and Notifications: Set up alerts for specific events or thresholds
  • ✅ Integration: Connect with Azure Monitor, Power BI, and other Azure services

Use Cases

  • Performance Optimization: Identify slow responses or bottlenecks
  • User Experience Analysis: Understand where users drop off or get confused
  • Error Tracking: Quickly identify and resolve issues
  • Compliance and Auditing: Maintain detailed logs of all interactions
  • Usage Insights: Track adoption and usage patterns
  • A/B Testing: Compare performance of different copilot configurations

Prerequisites

Before connecting Application Insights to your Copilot Studio agent, ensure you have:

  • Azure Subscription: Active Azure subscription with appropriate permissions
  • Copilot Studio Access: Owner or contributor access to the copilot agent
  • Azure Permissions: Ability to create Application Insights resources
  • Copilot Studio Agent: An existing copilot agent to monitor

Required Roles:

  • Azure: Contributor or Owner role on the resource group
  • Copilot Studio: Owner or Editor role on the copilot

Step 1: Create Application Insights Resource

Application Insights is an Azure service that collects and analyzes telemetry data from your applications.

Create via Azure Portal

  1. Navigate to Azure Portal

  2. Create New Resource

    • Click Create a resource
    • Search for Application Insights
    • Click Create

  3. Configure Basic Settings

    Create Application Insights

    • Subscription: Select your Azure subscription
    • Resource Group: Choose existing or create new
    • Name: Enter a descriptive name (e.g., copilot-agent-insights)
    • Region: Select the region closest to your users
    • Resource Mode: Choose Workspace-based (recommended)
    • Log Analytics Workspace: Select existing or create new

  4. Review and Create

    • Review your configuration
    • Click Create
    • Wait for deployment to complete (usually 1-2 minutes)

Create via Azure CLI

# Variables
resourceGroup="copilot-rg"
location="eastus"
workspaceName="copilot-workspace"
appInsightsName="copilot-agent-insights"

# Create resource group (if needed)
az group create --name $resourceGroup --location $location

# Create Log Analytics workspace
az monitor log-analytics workspace create \
    --resource-group $resourceGroup \
    --workspace-name $workspaceName \
    --location $location

# Get workspace ID
workspaceId=$(az monitor log-analytics workspace show \
    --resource-group $resourceGroup \
    --workspace-name $workspaceName \
    --query id -o tsv)

# Create Application Insights
az monitor app-insights component create \
    --app $appInsightsName \
    --location $location \
    --resource-group $resourceGroup \
    --workspace $workspaceId

Output PowerShell

Create via PowerShell

# Variables
$resourceGroup = "copilot-rg"
$location = "East US"
$workspaceName = "copilot-workspace"
$appInsightsName = "copilot-agent-insights"

# Create resource group (if needed)
New-AzResourceGroup -Name $resourceGroup -Location $location

# Create Log Analytics workspace
$workspace = New-AzOperationalInsightsWorkspace `
    -ResourceGroupName $resourceGroup `
    -Name $workspaceName `
    -Location $location

# Create Application Insights
New-AzApplicationInsights `
    -ResourceGroupName $resourceGroup `
    -Name $appInsightsName `
    -Location $location `
    -WorkspaceResourceId $workspace.ResourceId

Write-Host "Application Insights created successfully!" -ForegroundColor Green

Output

Output PowerShell

Application Insights resource

Application Insights created

Step 2: Retrieve the Connection String

After creating the Application Insights resource, you need to retrieve the connection string to link it with your Copilot Studio agent.

Method 1: Azure Portal

  1. Navigate to Application Insights Resource

    • In Azure Portal, go to your Application Insights resource
    • You can find it in All resources or in your resource group

  2. Copy the Connection String

    Copy Connection String

    • In the Overview page, locate the Connection String
    • Click the Copy icon next to the connection string
    • Save it for the next step

    Note: The connection string looks like:

    InstrumentationKey=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx;IngestionEndpoint=https://region.applicationinsights.azure.com/;LiveEndpoint=https://region.livediagnostics.monitor.azure.com/

Method 2: Azure CLI

# Get connection string
az monitor app-insights component show \
    --app $appInsightsName \
    --resource-group $resourceGroup \
    --query connectionString -o tsv

Method 3: PowerShell

# Get Application Insights details
$appInsights = Get-AzApplicationInsights `
    -ResourceGroupName $resourceGroup `
    -Name $appInsightsName

    # Display connection string
    Write-Host "Connection String:" -ForegroundColor Cyan
    Write-Host $appInsights.ConnectionString
    
    # Display instrumentation key
    Write-Host "`nInstrumentation Key:" -ForegroundColor Cyan
    Write-Host $appInsights.InstrumentationKey

Now that you have the connection string, link it to your Copilot Studio agent to start collecting telemetry.

Configure in Copilot Studio

  1. Open Copilot Studio

  2. Access Settings

    • Click on Settings (gear icon) in the top right
    • Navigate to Advanced section
    • Select Application Insights

  3. Enter Connection Details

    Link Application Insights to Copilot Agent

    • Paste the Connection String you copied from Azure into the Connection String
    • Click Save

Important Configuration Notes

  • Data Retention: By default, Application Insights retains data for 90 days
  • Sampling: For high-volume copilots, consider configuring sampling to manage costs
  • Privacy: Ensure you comply with data privacy regulations when logging conversations
  • Cost: Application Insights charges based on data ingestion volume

Step 4: Verify Configuration

After linking Application Insights, verify that telemetry data is being collected correctly.

Step 5: Check Metadata in Application Insights within Copilo Studio Agent

Check Metadata of Application Insights

Note: The Log Sensitive Types toggle, contains User ID, Name and Text of the prompt.

Privacy Considerations - Review what personal data will be logged - Ensure compliance with GDPR, HIPAA, and other regulations - Configure data masking if needed - Document data retention policies

Step 6: Test

  1. Test Your Copilot
    • Go back to Copilot Studio
    • Test your copilot by having a conversation
    • Return to Application Insights after a few minutes

Telemetry captured

  1. Query Telemetry Data
    • In Application Insights, click Logs in the left menu
    • Run a simple query to verify data:
   // View recent custom events
customEvents
| where cloud_RoleInstance == "Holiday Buddy"  and customDimensions["channelId"] == "pva-studio"
| project timestamp, name, session_Id, user_Id, cloud_RoleInstance, customDimensions, customDimensions["TopicName"], customDimensions["text"], customDimensions["speak"]

KQL App Insights

Step 7: Enable Enhanced Transcripts

Enhanced transcripts provide detailed conversation logs with additional metadata for deeper analysis.

Enable Enhanced Logging

  1. Access Analytics Settings

    • In Copilot Studio, go to Settings > Advanced >Application Insights
    • Look for Enhanced transcripts option

  2. Enable Transcript Logging

    • Toggle Enable enhanced transcripts to On Enable Enhanced Transcripts

    Details about nodes, such as name, type, and start and end times, can be saved to the transcript stored in Dataverse.

Step 8: Access the Copilot Studio Dashboard in Application Insights

Microsoft has introduced a built-in Copilot Studio Dashboard inside Azure Application Insights — a significant advancement for anyone building or managing custom agents. This integration eliminates the need to jump between multiple tools, providing a unified monitoring experience.

Why Use the Copilot Studio Dashboard?

With this dashboard, you get one centralized location to monitor all critical aspects of your copilot agents:

  • ✅ Agent Health: Overall system health and availability metrics
  • ✅ Telemetry: Detailed event tracking and user interactions
  • ✅ Performance Trends: Response times, throughput, and bottlenecks
  • ✅ User Experience Insights: Conversation patterns, satisfaction scores, and engagement metrics

This integration makes it incredibly easy to optimize your agents with real, actionable data — all within the Azure ecosystem you already use.

How to Access the Dashboard

Follow these steps to open the Copilot Studio Dashboard in Application Insights:

  1. Navigate to Application Insights

    • Open the Azure Portal
    • Go to your Application Insights resource that’s connected to your Copilot Studio agent
    • You can find it under All resources or in your resource group

  2. Open the Monitoring Section

    • From the left navigation pane, locate the Monitoring section
    • Click on Workbooks under the Monitoring category

  3. Select Copilot Studio Dashboard

    • In the Workbooks gallery, browse or search for Copilot Studio Dashboard
    • Click on the dashboard to open it
    • The dashboard will load with your copilot’s telemetry data

Copilot Studio Dashboard in Application Insights Workbooks

Dashboard Components

The Copilot Studio Dashboard provides comprehensive visualizations including:

Performance Metrics:

  • Request rates and response times
  • Success vs. failure rates
  • Conversation duration statistics
  • Peak usage times and patterns

User Engagement:

  • Active users and sessions
  • Conversation completion rates
  • Most frequently triggered topics
  • User drop-off points

Technical Health:

  • Error rates and exception details
  • Dependency calls and external service health
  • Resource utilization
  • Custom event tracking

Benefits of the Integrated Dashboard

BenefitDescription
Unified ExperienceNo need to switch between Copilot Studio and Azure Portal
Real-Time DataLive telemetry updates as conversations happen
Custom QueriesUse KQL to create custom visualizations
Exportable InsightsShare reports with stakeholders easily
Azure IntegrationLeverage Azure Monitor alerts and automation

Best Practices for Dashboard Usage

  1. Regular Monitoring

    • Check the dashboard daily during initial deployment
    • Establish baseline metrics for normal operation
    • Set up alerts for anomalies or threshold breaches

  2. Performance Optimization

    • Identify slow-responding topics or actions
    • Monitor external API call performance
    • Track resource consumption patterns
    • Optimize based on peak usage times

  3. User Experience Insights

    • Analyze conversation flow patterns
    • Identify topics with high abandonment rates
    • Review error messages that confuse users
    • Track sentiment trends over time

  4. Continuous Improvement

    • Use telemetry to prioritize feature development
    • A/B test conversation flows with data-driven decisions
    • Document improvements and track impact
    • Share insights with stakeholders regularly

Next Steps After Dashboard Setup

Now that you have the complete monitoring solution in place:

  • Connect custom agents to Application Insights following Steps 1-7
  • Analyze real-time telemetry using both the dashboard and custom KQL queries
  • Use the Copilot Studio dashboard effectively to drive continuous improvements
  • Set up alerts in Azure Monitor for proactive issue detection
  • Create custom workbooks for specific business requirements
  • Export and share reports with your team and stakeholders

Video how it works

[Monitor Copilot Studio Agents Like a Pro: Connect Azure Application Insights for Real-Time Analytics][https://www.youtube.com/watch?v=8EFP63vLO0M&t=239s]

References