Outlook Agent — Drafts, Meetings & Calendar Blocks

Outlook Agent — Draft mails, meeting requests, calendar blockers

This post describes an Outlook-focused agent built with Microsoft Semantic Kernel. The agent’s responsibilities:

• Draft and refine email drafts using semantic functions.
• Propose and schedule meeting requests (candidate times, attendees).
• Create calendar “busy” blocks (holds/tentative slots) for focused work.

High-level design principles

• Keep all side-effectful operations (sending mail, creating events) in native functions. This ensures deterministic behavior, easier retries, and safe audit/logging.
• Use semantic functions for language and reasoning tasks: composing text, extracting meeting details, proposing times, and summarizing threads.
• Require an explicit user confirmation step before performing any action that changes remote state.

Semantic Kernel mapping

• Kernel: central runtime that wires the LLM provider, memory provider (vector store), and skills.
• Semantic functions (prompts): compose_draft, summarize_thread, extract_meeting_details, propose_times.
• Native functions (code): create_calendar_event, send_mail, create_busy_block — these call Microsoft Graph (or other calendar/mail APIs).
• Memory: store drafts, user preferences (default meeting lengths, working hours), and recent context to make follow-up actions smoother.
• Planner: decomposes goals like “set up a kickoff” into ordered steps and coordinates semantic + native calls.

Typical orchestration flow

1. User: “Draft an email to Alex proposing a meeting next Monday or Tuesday morning about the project handoff.”
2. Agent runs extract_meeting_details (semantic) -> returns a structured object: {subject, body_draft, preferred_times}
3. Agent runs propose_times (semantic or hybrid) -> formats candidate time slots and optionally checks availability using calendar API.
4. Agent presents draft and proposed slots to the user for confirmation.
5. On user confirm, agent calls create_calendar_event (native) to create invites and send_mail (native) to send the final message (or saves draft depending on preference).

Conceptual C# sketch (pseudo-code)

// Build and configure kernel (pseudo)
var kernel = new KernelBuilder()
    .WithOpenAITextCompletion("oai", apiKey)
    .WithMemory(new YourVectorStore())
    .Build();

// Semantic function to compose a draft (prompt-backed)
var composeDraft = kernel.CreateSemanticFunction("compose_draft", "Given recipient, purpose and tone, write a subject and body.");

// Native function: deterministic call to Microsoft Graph (create event)
kernel.RegisterNativeFunction("create_calendar_event", async (input) => {
    var graph = GraphClientFactory.Create(msalToken);
    var evt = new Event {
        Subject = input.subject,
        Start = input.start,
        End = input.end,
        Attendees = input.attendees
    };
    // Use idempotency key if available
    return await graph.Me.Events.Request().AddAsync(evt);
});

// Orchestration: draft -> propose -> confirm -> create event
var draft = await kernel.RunAsync(composeDraft, new { recipient = "alex@contoso.com", purpose = "project handoff" });
// present draft to user...
// on confirm:
var ev = await kernel.RunAsync("create_calendar_event", new { subject = "Project Handoff", start = slot.Start, end = slot.End, attendees = attendees });

Practical recommendations

• Use explicit confirmation: always ask the user to confirm before sending mail or inviting people.
• Implement idempotency and retries in native functions (store event IDs, use idempotency keys when creating events).
• Minimize scopes: request the least Graph permissions necessary (Mail.Send, Calendars.ReadWrite).
• Offer preview/draft mode: let users edit the draft before sending. Persist drafts in memory or draft mailbox.
• Add audit logs: record who asked the agent to do what and when (useful for troubleshooting and security reviews).

Security & privacy

• Use delegated auth flows (MSAL) with incremental consent.
• Avoid embedding or storing sensitive PII in vector memory unless encrypted and explicitly permitted.
• Provide opt-outs for specific folders or calendar delegates.

Extending the Outlook Agent

• Auto-detect preferred meeting length and timezones from user profile.
• Add heuristics to choose tentative slots that avoid conflicts with protected times (e.g., focus time, travel).
• Implement a “meeting negotiation” flow: propose multiple slots and follow up if attendees decline.