Contextual Awareness
How VoiceInk uses on-screen text to improve accuracy.
What is Contextual Awareness?
Contextual Awareness is a powerful feature that allows VoiceInk to understand the content of your active application window. By seeing the text on your screen, the AI can provide much more accurate and relevant transcriptions and enhancements.
This feature is particularly useful when your speech contains jargon, code, or specific names that appear on the screen.
How It Works (And Protects Your Privacy)
VoiceInk's Contextual Awareness is designed with privacy as the top priority. Here’s a step-by-step breakdown of the process:
-
One-Time Screenshot: When you use a feature that requires context, VoiceInk takes a temporary, one-time screenshot of your currently active window. It does not record your screen continuously.
-
On-Device OCR: The screenshot is processed directly on your Mac using Apple's native Vision framework. This technology scans the image for text and converts it into plain text. This process is often referred to as Optical Character Recognition (OCR).
-
Context for the AI: This extracted text is then used as additional context for the AI model.
Data and Privacy
-
Local AI Models: If you are using a local transcription model (like the default Whisper model) and a local enhancement provider (like Ollama), no data from the screenshot ever leaves your machine. The entire process, from screenshot to OCR to AI processing, happens offline.
-
Cloud AI Providers: If you are using a cloud-based AI provider for enhancement (like OpenAI, Groq, etc.), only the extracted text from the screenshot is sent to the provider as part of the request. The image itself is never uploaded.
How to Enable It
Contextual awareness has two separate toggles, both found in the Enhancement tab's settings panel (gear icon):
- Clipboard Context — includes your current clipboard contents as context for the AI
- Screen Context — takes a one-time screenshot, extracts on-screen text via OCR, and passes it to the AI
You can enable either or both independently. See Configuring AI Enhancements for more details on the Enhancement settings panel.