Apple’s integration of generative AI into macOS 26 Shortcuts via the "Use Model" action opens a new frontier for automation. It promises to transform mundane tasks with on-device or Private Cloud Compute (PCC) intelligence. But as developers Jason Snell and Dan Moren discovered through real-world experimentation, harnessing this power is less like coding and more like wrangling an unpredictable, occasionally brilliant, but often frustratingly obtuse assistant.

The Alt-Text Conundrum: When AI Descriptions Go Rogue

Snell aimed to enhance his image upload shortcut by automatically generating descriptive alt text using AI – a seemingly perfect application. His journey quickly illuminated core challenges:

  1. Model Constraints & Workarounds: Apple's on-device model couldn't handle images, forcing a switch to PCC. Full-size images caused delays, necessitating an added resizing step. `

    alt="Article illustration 3"
    loading="lazy">
` 2. **Prompt Engineering Battles:** Initial descriptions were verbose and included double quotes, breaking HTML. Refined prompts ("Limit to two sentences! No double quotes! Use single quotes!") were inconsistently obeyed. "Computer programs behave identically every time," Snell notes, "but AI does not!" 3. **The Hybrid Solution:** Snell's final workflow became a Rube Goldberg machine of compromises: * Prompt: "Describe for alt text. Max 200 characters. Include screenshot words." * Check character count; if over limit, pass text to the *on-device* model with "Shorten to <250 chars". * Finally, a dedicated Shortcuts step replaces any remaining double quotes with single quotes. Despite this engineering, errors persisted – hashtags mysteriously appended, incorrect times added to Watch screenshots. "It’s simultaneously an amazing piece of technology and incredibly stupid," Snell concludes, emphasizing that human review remains essential.

Expense Automation: When $209.49 Becomes $20,949

Dan Moren tackled a different beast: automating expense report entries from receipts using AI text extraction. His goal was elegant – feed a PDF/email to the model and get structured data for his spreadsheet. Reality proved thornier:

  1. Formatting Wars: Initial attempts to get comma-separated values failed. Switching to tab/newline delimiters clashed with Numbers. The breakthrough came by having the AI output CSV, splitting it into a list within Shortcuts, and feeding that list directly into Numbers. `



    alt="Article illustration 4"

    loading="lazy">

2. **The Perils of Complexity:** Moren's ambitious prompt requested multiple data points (date, vendor, amount) *and* a specially formatted filename (YYYY-MM-DD-VENDOR-AMOUNT). This overload confused the model: * **Counting Failures:** Extracting the "first five alphanumeric characters of the vendor" yielded 5 characters... sometimes. Or 6. * **Format Confusion:** Requests for the dollar amount in different formats (decimal for spreadsheet, no-decimal for filename) led to catastrophic errors like interpreting$209.49as20949and inserting$20,949` into the spreadsheet.
3. Non-Deterministic Nightmare: Running the same receipt data repeatedly yielded different results. "Anathema... to the sensibilities of any programmer," Moren observes. The core reliability expected in programming is absent. "I could run it 99 times and the 100th it could spit back something different and totally incorrect."

"What I’ve ended up with is essentially a not very bright assistant whose work I need to double check. And if I can’t trust that work, does it actually save me time?" — Dan Moren

The Uncomfortable Truth: AI as an Unruly Tool, Not a Solution

These experiments underscore critical lessons for developers diving into Apple's new AI toolkit:

  • Prompt Engineering is an Art (of Diminishing Returns): Crafting the perfect prompt is elusive. Specificity helps but doesn't guarantee consistency or correctness. Expect iteration and frustration.
  • Hybrid Workflows are Mandatory: Pure AI solutions are fragile. Combining AI steps with traditional programming logic (like string replacement, list manipulation, and error checking) is essential for robustness.
  • Human-in-the-Loop is Non-Negotiable: AI hallucinations and inconsistencies mean outputs must be reviewed. Automation might streamline, not eliminate, human effort.
  • Trust is the Biggest Hurdle: As Moren's expense shock shows, errors can be severe. Deploying such automation for critical tasks without rigorous safeguards and verification is perilous.

Apple's integration of AI models into Shortcuts is undeniably powerful, offering glimpses of a more automated future. Yet, Snell and Moren's hands-on struggles reveal we're firmly in an era of experimentation. Success demands embracing AI's potential while meticulously designing around its inherent unpredictability and acknowledging that, for now, the programmer's discerning eye remains the most crucial component in the loop. The path from intriguing demo to reliable tool is proving significantly rockier than anticipated.

Source: Experimenting with Apple's AI models inside Shortcuts - Six Colors (Jason Snell & Dan Moren)