GitHub Copilot Developer Training — Advanced Topics

Duration: ~3 hours 20 minutes (200 min)
Format: Presentation + Live Demo + Hands-On
Audience: Developers extending GitHub Copilot with integrations, evaluation frameworks, and diagnostic tools
Focus: Extensions, MCP, evaluating AI output, and troubleshooting Copilot
Repo: microsoft/GitHubCopilot_Customized (OctoCAT Supply)

Part of the Copilot Developer Training curriculum (Foundations · Agentic Patterns · Advanced Topics). This module can be delivered standalone.

Workshop Overview

This module covers the advanced topics that round out a developer's Copilot proficiency: extending Copilot with third-party integrations and MCP servers, building evaluation frameworks to measure AI output quality, and mastering the diagnostic tools needed when things go wrong.

Learning Objectives

• Understand VS Code chat participants and GitHub Copilot Extensions
• Configure and use MCP (Model Context Protocol) servers for external tool integration
• Define success criteria and quality rubrics for AI-generated code
• Build automated and human-in-the-loop evaluation workflows
• Navigate Copilot's output logs, debug mode, and agent debug traces
• Collect and export diagnostics for troubleshooting

Prerequisites / What You Should Know

If you're taking this module standalone (without Modules 1–2), you should be comfortable with these concepts:

If you attended Modules 1–2, skip this section — you've covered all of the above.

Prerequisites

Session Agenda

Total: ~200 min (~3h 20min) including breaks

6.1. Opening & AI Safety: "Third-Party Trust" (5 min)

Key Points

• Extensions and MCP servers are third-party code that runs alongside Copilot — they can access your codebase and execute actions
• Trust decisions matter: who built it? What data does it access? What actions can it take?
• Evaluate before installing: read permissions, check the publisher, review community feedback
• Organization admins can control which extensions and MCP servers are allowed

Trust Evaluation Checklist

Discussion Points

• How does your team currently vet VS Code extensions?
• What's your organization's policy on third-party AI integrations?

6.2. VS Code Chat Participants (10 min)

Key Points

• Chat participants are specialized handlers built into VS Code that route your questions to domain-specific logic
• They're the @ mentions you use in chat: @workspace, @vscode, @terminal
• Each participant has access to different data and capabilities

Built-in Participants

How Participants Work Under the Hood

1. You type @workspace How is auth implemented?
2. VS Code routes the message to the @workspace participant
3. The participant searches your project (file tree, symbols, content)
4. Relevant code snippets and file references are injected into the context
5. The augmented context is sent to the model for response

🖥️ Demo: Participant Capabilities

1. @workspace — Ask "What is the database schema for orders?" — show it finding the schema across files
2. @vscode — Ask "How do I configure format on save?" — show it referencing VS Code settings
3. @terminal — After a failed npm test, ask "Why did the test fail?" — show it reading terminal output

Discussion Points

• Which participant do you use most? Which do you underuse?
• How does @workspace decide which files are relevant to your question?

6.3. GitHub Copilot Extensions (15 min)

Key Points

• Copilot Extensions are GitHub Apps that add chat capabilities to Copilot
• They appear as @extension-name participants in chat — you interact with them through natural language
• Extensions can access external services (Docker, Azure, Sentry, etc.) and bring domain-specific knowledge into Copilot

Extension Capabilities

Marketplace Discovery

• Browse at https://github.com/marketplace?type=apps&copilot_app=true
• Filter by category: development tools, security, cloud, productivity
• Check: publisher verification, install count, recent activity

🖥️ Demo: Using a Copilot Extension

1. Show the Copilot Extensions marketplace
2. Install an extension (e.g., Docker, GitHub Models, or Azure)
3. Use it in chat: @extension-name [question]
4. Show how the response includes domain-specific knowledge beyond Copilot's base training

Discussion Points

• What external services would benefit from a Copilot Extension in your workflow?
• How do extensions differ from MCP servers (covered next)?
• What security considerations apply to extensions that access external services?

6.4. MCP Architecture (15 min)

Key Points

• MCP (Model Context Protocol) is an open standard for connecting AI models to external tools and data sources
• Think of it as "USB for AI" — a standard interface that any tool can implement
• MCP separates three roles: Host (VS Code), Client (Copilot), and Server (the tool provider)

MCP Architecture

┌─────────────────┐
│     VS Code     │  ← HOST: manages connections
│    (Host)       │
├─────────────────┤
│   Copilot       │  ← CLIENT: sends requests to servers
│   (Client)      │
└────────┬────────┘
         │
    ┌────┴────┐
    │         │
    ▼         ▼
┌────────┐ ┌────────┐
│ MCP    │ │ MCP    │  ← SERVERS: provide tools, resources, prompts
│Server A│ │Server B│
│(GitHub)│ │(File   │
│        │ │System) │
└────────┘ └────────┘

MCP Capabilities

Transport Types

MCP vs. Copilot Extensions

Discussion Points

• What internal tools or services would you expose via MCP?
• How does the host/client/server separation improve security?
• When would you choose MCP over a Copilot Extension (or vice versa)?

6.5. MCP Configuration (10 min)

Key Points

• MCP servers are configured in .vscode/mcp.json in your workspace
• Each entry specifies a server name, command/transport, and optional environment variables
• VS Code discovers the servers on startup and makes their tools available in chat

Configuration Syntax

{
  "servers": {
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "${input:github-token}"
      }
    },
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "./src"]
    }
  }
}

Security Considerations

🖥️ Demo: MCP Server Setup

1. Create .vscode/mcp.json with a filesystem server pointing to the api/ directory
2. Open chat — show the MCP tools appearing in the tool list
3. Ask Copilot to use the filesystem tool: "List all TypeScript files in the API directory"
4. Add a GitHub MCP server — show it pulling issue data into the chat context

Discussion Points

• What .vscode/mcp.json configuration would you set up first for your project?
• How would you share MCP configurations across your team?
• What's the security review process for adding a new MCP server?

6.6. Session 6 Summary & Discussion (5 min)

Key Takeaways

• VS Code chat participants (@workspace, @vscode, @terminal) route questions to specialized handlers
• Copilot Extensions bring third-party domain knowledge into chat via GitHub Apps
• MCP is the open standard for tool integration — configure in .vscode/mcp.json
• Always evaluate trust, permissions, and security before adding extensions or MCP servers

Discussion Points

• What's the first extension or MCP server you'll set up after this session?
• How does MCP change what's possible with Copilot in your workflow?
• What governance process would you recommend for your org's MCP server list?

☕ Break — 10 Minutes

7.1. Opening & AI Safety: "When to Trust, When to Verify" (5 min)

Key Points

• Not all AI output deserves the same level of scrutiny — risk should drive verification effort
• Low risk: Boilerplate code, test scaffolding, documentation → quick glance
• Medium risk: Business logic, API routes, data transformations → careful review
• High risk: Security code, authentication, financial calculations, infrastructure → thorough verification + testing
• The goal is a trust calibration framework — not "trust everything" or "verify everything"

Risk-Based Verification

Discussion Points

• Where does your team currently draw the trust/verify line?
• Has AI-generated code ever introduced a bug that made it to production?

7.2. Defining Success Criteria (15 min)

Key Points

• Before using Copilot for a task, define what "good output" looks like
• Success criteria prevent the trap of accepting the first thing that compiles
• Different task types need different criteria

Success Criteria by Task Type

Defining Criteria Before Coding

Template:

Task: [what you're asking Copilot to do]
Success looks like:
  - Functional: [what the code should do]
  - Quality: [readability, patterns, style]
  - Constraints: [performance, security, compatibility]
  - Not acceptable: [what the code should NOT do]

🖥️ Demo: Criteria-Driven Prompting

1. Define success criteria for a task: "Add pagination to the products API endpoint"
2. Include criteria in the prompt: "Must be cursor-based, max 100 items, no offset, include Link headers"
3. Evaluate the output against the criteria — does it meet each one?
4. Compare to asking without criteria: "Add pagination to products"

Discussion Points

• How often do you define success criteria before writing code (with or without AI)?
• What's the minimum criteria you'd set for every Copilot interaction?
• How do success criteria change for different team roles (junior vs. senior dev)?

7.3. Output Quality Rubrics (15 min)

Key Points

• A rubric is a structured scoring system for evaluating AI output quality
• Rubrics make evaluation consistent, repeatable, and shareable across a team
• Score each dimension independently — a fix that's correct but unreadable still needs work

Rubric Template

Applying a Rubric

Example — Agent-generated pagination code:

🖥️ Demo: Rubric Evaluation

1. Ask Agent mode to add a search feature to the products API
2. Score the output against the rubric dimensions
3. Show where it falls short — ask Copilot to fix the specific weaknesses
4. Re-score — show the improvement

Discussion Points

• What dimensions matter most for your team's code quality standards?
• How would you automate parts of the rubric (e.g., lint = style, tests = correctness)?
• Would you share rubrics across the team, or let each developer customize theirs?

7.4. Evaluation Methods (15 min)

Key Points

• Evaluation happens at two levels: automated (machines check) and human-in-the-loop (developers review)
• The best approach combines both in a verification pipeline
• Automated checks catch the obvious; human review catches the subtle

Automated Evaluation

Human-in-the-Loop Evaluation

The Verification Pipeline

┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
│  Agent   │───►│  Lint    │───►│  Type    │───►│  Test    │───►│  Human   │
│  Output  │    │  Check   │    │  Check   │    │  Suite   │    │  Review  │
└──────────┘    └──────────┘    └──────────┘    └──────────┘    └──────────┘
                   ▼ Fail          ▼ Fail          ▼ Fail          ▼ Reject
              Auto-reject     Auto-reject     Auto-reject     Send back

Batch Evaluation

For systematic quality assessment across many tasks:

1. Define a set of representative prompts (10-20 tasks)
2. Run each through Copilot
3. Score outputs against the rubric
4. Track aggregate scores over time
5. Identify patterns: which task types does Copilot handle well vs. poorly?

🖥️ Demo: Automated Evaluation Pipeline

1. Generate code with Agent mode
2. Run the lint + type check + test pipeline
3. Show a failure — observe how the automated gate catches the issue
4. Fix the issue and re-run — show the pipeline passing

Discussion Points

• What automated checks does your team already have in CI?
• How would you adapt your PR review process for AI-generated code?
• What's the right balance between automated and human evaluation?

7.5. Tracking & Improvement (15 min)

Key Points

• Evaluation without tracking is a one-time exercise — tracking creates a feedback loop
• Monitor quality trends over time: are your prompts/instructions improving?
• The feedback loop: evaluate → identify weakness → improve instructions → re-evaluate

Metrics to Track

The Feedback Loop

┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
│ Generate │───►│ Evaluate │───►│ Identify │───►│ Improve  │
│ code     │    │ with     │    │ weakness │    │ prompts/ │
│          │    │ rubric   │    │          │    │ instruct │
└──────────┘    └──────────┘    └──────────┘    └────┬─────┘
     ▲                                               │
     └───────────────────────────────────────────────┘

Trust Calibration by Task Type

🖥️ Demo: Feedback Loop in Action

1. Generate a utility function — score it (e.g., correctness: 3, style: 2)
2. Identify the style weakness — add a specific instruction to copilot-instructions.md
3. Regenerate the same function — show the improved style score
4. Discuss how this scales across a team

Discussion Points

• What metrics would be most valuable for your team to track?
• How do you currently measure developer productivity? How would AI change that?
• What feedback loops already exist in your development process?

7.6. Usage, Billing & Cost Strategies (10 min)

Key Points

• GitHub is transitioning from premium-request-based billing to usage-based billing (AI Credits) starting June 1, 2026
• GitHub AI Credits are the new billing unit: 1 AI credit = $0.01 USD
• Billing is based on actual token consumption: input tokens + output tokens + cached tokens
• Code completions and next edit suggestions remain unlimited — they are NOT billed in AI credits
• Credits are pooled at the billing entity level — power users are offset by lighter users

The Billing Transition

Model Cost Tiers — Know What You're Spending

10 Developer Cost Strategies

Admin Budget Controls

Budgets can be set at four levels — each can trigger alerts or enforce hard stops:

1. Enterprise-level — caps all orgs, repos, cost centers
2. Organization-level — caps all repos in the org
3. Cost-center-level — caps a single cost center
4. User-level — caps individual users ($0 budget = no access)

Note: There is no automatic fallback to cheaper models when a budget is exhausted. Usage is simply blocked until the next billing cycle.

Discussion Points

• Which cost tier do most of your daily tasks fall into?
• How would you implement per-user budgets in your organization?
• What's the ROI calculation for using a 7.5x model vs. a 1x model?

7.7. Session 7 Summary & Discussion (5 min)

Key Takeaways

• Define success criteria before asking Copilot to generate code
• Use rubrics to make evaluation consistent and repeatable
• Combine automated checks with human review in a verification pipeline
• Track metrics over time — use the feedback loop to continuously improve prompts and instructions

Discussion Points

• What's the first quality metric you'll start tracking?
• How would you introduce rubric-based evaluation to your team?
• What's the biggest quality challenge with AI-generated code in your experience?

☕ Break — 10 Minutes

8.1. Opening & AI Safety: "Debugging the Black Box" (5 min)

Key Points

• AI models are often treated as black boxes — input goes in, output comes out, and it's unclear why
• Copilot provides transparency tools: logs, debug mode, and traces that show what the model received and how it reasoned
• Transparency builds trust — when you can see the context, you can understand the output
• Debugging AI is different from debugging code: you're debugging the context and instructions, not an algorithm

Discussion Points

• Have you ever been confused by a Copilot response? How did you investigate?
• How does transparency change your trust in AI-generated output?

8.2. Output Log Channels (15 min)

Key Points

• VS Code has multiple output channels related to Copilot — each logs different information
• The main channels: GitHub Copilot, GitHub Copilot Chat, and language-specific log channels
• Output logs are your first diagnostic tool — they show connection status, errors, and feature availability

Output Channels

How to Access

1. Open the Output panel: Ctrl+Shift+U (Windows) / Cmd+Shift+U (Mac)
2. Select the channel from the dropdown
3. Look for errors (red), warnings (yellow), and info messages

Common Log Patterns

🖥️ Demo: Reading Output Logs

1. Open the Copilot output channel — show the log stream
2. Trigger a completion — identify the log entry for that event
3. Show an error scenario (e.g., disconnect network briefly) — identify the error in logs
4. Show the Chat output channel — trace a chat request through the logs

Discussion Points

• How often do you check output logs when Copilot behaves unexpectedly?
• What's the most common error you've seen in Copilot logs?

8.3. Chat Debug Mode (15 min)

Key Points

• Chat debug mode reveals what Copilot sees: the full context sent to the model, token counts, timing, and model selection
• Enable via VS Code setting: github.copilot.chat.debugMode
• This is the single most useful diagnostic tool for understanding "why did Copilot say that?"

What Debug Mode Reveals

How to Enable

{
  "github.copilot.chat.debugMode": true
}

After enabling, debug information appears in the chat output channel and optionally inline with responses.

Debug Output Anatomy

A typical debug output shows:

[DEBUG] Context assembly:
  System prompt: 1,247 tokens
  Repository instructions: 312 tokens
  File-targeted instructions: 0 tokens
  Attached context (#file): 3,891 tokens
  Conversation history: 2,104 tokens
  Current message: 87 tokens
  ────────────────────────
  Total input: 7,641 tokens

[DEBUG] Model: gpt-4o
[DEBUG] Response: 342 tokens (1.2s)
[DEBUG] Tool calls: workspace.search (284ms), file.read (12ms)

🖥️ Demo: Chat Debug Mode Walkthrough

1. Enable github.copilot.chat.debugMode in VS Code settings
2. Send a chat message — open the output channel to see the debug dump
3. Point out: instructions loaded, context composition, token counts
4. Send a message with a large #file reference — show the token count jump
5. Show how different models appear in the debug output

Discussion Points

• What surprised you about the context composition in debug mode?
• How would you use token count information to optimize your prompts?
• When should you enable debug mode vs. leave it off?

8.4. Agent Debug Logs (15 min)

Key Points

• Agent mode creates detailed logs of every iteration in its agentic loop
• These logs show: plan, tool calls, results, decisions, and error recovery
• Agent debug logs are essential for understanding why the agent took a particular approach

Agent Log Location

Agent debug logs are written to a session-specific log file. Access them via:

1. Command Palette → "GitHub Copilot: Open Agent Debug Log"
2. Or find them in the workspace storage directory

Understanding Agent Iteration Logs

Each iteration in the agent loop creates a log entry:

[Iteration 1] Plan: Add validation middleware to orders route
  Tool: file.read → api/routes/orders.ts (success, 2,341 tokens)
  Tool: file.read → api/routes/products.ts (success, 1,892 tokens)
  Tool: file.write → api/middleware/validate-order.ts (created)
  Tool: file.edit → api/routes/orders.ts (modified lines 12-18)
  Tool: terminal.run → npm run lint (exit code: 1)
  Result: FAIL — lint error on line 5: 'Request' is not defined

[Iteration 2] Fix: Add missing import
  Tool: file.edit → api/middleware/validate-order.ts (modified line 1)
  Tool: terminal.run → npm run lint (exit code: 0)
  Tool: terminal.run → npm test (exit code: 0)
  Result: PASS — all validation gates passed

Tool Call Traces

🖥️ Demo: Tracing an Agent Failure

1. Give Agent mode a task that will fail on first attempt (e.g., "Add a feature that requires a missing dependency")
2. Open the agent debug log — trace through the iterations
3. Show where the agent detected the failure (observe phase) and how it adjusted (reflect phase)
4. Point out the tool call trace: what was read, what was written, what commands ran

Discussion Points

• How would you use agent debug logs to improve your agent instructions?
• What patterns in the logs indicate the agent is stuck?
• How do agent debug logs compare to traditional application debugging?

8.5. Diagnostics Collection & Curriculum Wrap-Up (10 min)

Key Points

• When filing issues or requesting support, a diagnostics bundle provides all the information needed
• VS Code can export Copilot diagnostics including: extension version, log files, configuration, and system info
• Always sanitize diagnostics before sharing — remove tokens, secrets, and proprietary code

Diagnostics Collection

1. Command Palette → "GitHub Copilot: Collect Diagnostics"
2. Review the generated file — redact any sensitive content
3. Attach to GitHub issues or support requests

Diagnostic Toolkit Cheat Sheet

Curriculum Wrap-Up — All 8 Sessions Complete

What to Do Next

• Set up .github/copilot-instructions.md in your primary repository
• Create 1-2 custom agents for your team's common workflows
• Configure MCP servers for your internal tools
• Establish an evaluation rubric and start tracking quality metrics
• Share learnings with your team — the best way to learn is to teach

Further Learning

Discussion Points

• Across all 8 sessions, what's the single most impactful concept for your daily work?
• What's the first change you'll make to your team's Copilot setup?
• What topics would you want to explore in a follow-up session?

Workshop guide for GitHub Copilot Developer Training — Advanced Topics (Module 3 of 3)