fully working transcription

This commit is contained in:
LeonardoTrapani
2025-08-18 19:43:14 +02:00
parent f03c9c6a87
commit bcaf1181a0
5 changed files with 294 additions and 38 deletions
+68 -19
View File
@@ -7,7 +7,7 @@ Press a toggle key, speak, and get instant text input. Built natively for Waylan
- **Toggle workflow**: Press once to start recording, press again to stop and inject text - **Toggle workflow**: Press once to start recording, press again to stop and inject text
- **Wayland native**: Purpose-built for Wayland compositors - no legacy X11 dependencies or hacky workarounds - **Wayland native**: Purpose-built for Wayland compositors - no legacy X11 dependencies or hacky workarounds
- **Real-time feedback**: Desktop notifications for recording states and transcription status - **Real-time feedback**: Desktop notifications for recording states and transcription status
- **Multiple transcription backends**: OpenAI Whisper (planned: whisper.cpp for local processing) - **Multiple transcription backends**: OpenAI Whisper (planned: whisper.cpp for local processing, and more)
- **Smart text injection**: Clipboard save/restore with direct typing fallback - **Smart text injection**: Clipboard save/restore with direct typing fallback
- **Daemon architecture**: Lightweight control plane with efficient pipeline management - **Daemon architecture**: Lightweight control plane with efficient pipeline management
@@ -72,7 +72,7 @@ sudo apt install pipewire-pulse pipewire-bin wl-clipboard
sudo dnf install pipewire-utils wl-clipboard sudo dnf install pipewire-utils wl-clipboard
``` ```
**For text injection (recommended):** **For text injection:**
```bash ```bash
# Arch Linux # Arch Linux
@@ -196,7 +196,7 @@ Configuration will be read from `~/.config/hyprvoice/config.toml` (planned). Cur
Hyprvoice will support multiple transcription backends: Hyprvoice will support multiple transcription backends:
#### OpenAI Whisper API (Planned) #### OpenAI Whisper API
Fast, accurate cloud-based transcription: Fast, accurate cloud-based transcription:
@@ -219,6 +219,31 @@ model_path = "~/models/ggml-base.en.bin"
threads = 4 threads = 4
``` ```
#### Text Injection (Current)
Configurable text injection with multiple modes:
```toml
[injection]
mode = "fallback" # "clipboard", "type", or "fallback"
always_copy_clipboard = true
restore_clipboard = true
wtype_timeout = "5s"
clipboard_timeout = "3s"
```
**Injection Modes:**
- **`fallback`** (default): Try direct typing first, fallback to clipboard
- **`type`**: Direct typing using wtype only
- **`clipboard`**: Copy to clipboard only
**Behavior:**
- `always_copy_clipboard = true`: Always copy text to clipboard regardless of mode
- `restore_clipboard = true`: Save and restore original clipboard content
- Smart fallback ensures text injection always succeeds when possible
### Service Configuration ### Service Configuration
#### Systemd Service #### Systemd Service
@@ -256,17 +281,17 @@ systemctl --user enable --now hyprvoice.service
## Development Status ## Development Status
| Component | Status | Notes | | Component | Status | Notes |
| --------------------- | ------ | --------------------------------- | | --------------------- | ------ | -------------------------------- |
| Core daemon & IPC | ✅ | Unix socket control plane | | Core daemon & IPC | ✅ | Unix socket control plane |
| Recording workflow | ✅ | Toggle recording via PipeWire | | Recording workflow | ✅ | Toggle recording via PipeWire |
| Audio capture | ✅ | Efficient PipeWire integration | | Audio capture | ✅ | Efficient PipeWire integration |
| Desktop notifications | ✅ | Status feedback via notify-send | | Desktop notifications | ✅ | Status feedback via notify-send |
| OpenAI transcription | ✅ | HTTP API integration | | OpenAI transcription | ✅ | HTTP API integration |
| Text injection | | Clipboard + typing implementation | | Text injection | | Clipboard + wtype with fallback |
| Configuration system | ⏳ | TOML-based user settings | | Configuration system | ⏳ | TOML-based user settings |
| Comprehensive tests | ⏳ | Pipeline and integration testing | | Comprehensive tests | ⏳ | Pipeline and integration testing |
| whisper.cpp support | ⏳ | Local model inference | | whisper.cpp support | ⏳ | Local model inference |
**Legend**: ✅ Complete · ⏳ Planned **Legend**: ✅ Complete · ⏳ Planned
@@ -290,8 +315,8 @@ flowchart LR
end end
subgraph Pipeline subgraph Pipeline
A["Audio Capture"] A["Audio Capture"]
T["Transcribing (ASR TBD)"] T["Transcribing"]
I["Injecting (stub)"] I["Injecting (wtype + clipboard)"]
end end
N["notify-send/log"] N["notify-send/log"]
@@ -404,11 +429,34 @@ sudo apt install libnotify-bin # Ubuntu/Debian
#### Text Injection Issues #### Text Injection Issues
**Text not appearing (when implemented):** **Text not appearing:**
- Ensure cursor is in a text field when toggling off recording - Ensure cursor is in a text field when toggling off recording
- Check that `wtype` or clipboard tools are installed - Check that `wtype` and `wl-clipboard` tools are installed:
- Verify window manager supports the text injection method used
```bash
# Test wtype directly
wtype "test text"
# Test clipboard tools
echo "test" | wl-copy
wl-paste
```
- Verify Wayland compositor supports text input protocols
- Check injection mode in configuration (fallback mode is most robust)
**Clipboard issues:**
```bash
# Install wl-clipboard if missing
sudo pacman -S wl-clipboard # Arch
sudo apt install wl-clipboard # Ubuntu/Debian
# Test clipboard functionality
wl-copy "test text"
wl-paste
```
### Debug Mode ### Debug Mode
@@ -453,6 +501,7 @@ hyprvoice/
├── internal/ ├── internal/
│ ├── bus/ # IPC (Unix socket) + PID management │ ├── bus/ # IPC (Unix socket) + PID management
│ ├── daemon/ # Control daemon (lifecycle management) │ ├── daemon/ # Control daemon (lifecycle management)
│ ├── injection/ # Text injection (clipboard + wtype)
│ ├── notify/ # Desktop notification integration │ ├── notify/ # Desktop notification integration
│ ├── pipeline/ # Audio processing pipeline + state machine │ ├── pipeline/ # Audio processing pipeline + state machine
│ ├── recording/ # PipeWire audio capture │ ├── recording/ # PipeWire audio capture
+53
View File
@@ -0,0 +1,53 @@
package injection
import (
"context"
"fmt"
"os/exec"
"strings"
"time"
)
// getClipboard retrieves the current clipboard content using wl-paste
func getClipboard(ctx context.Context, timeout time.Duration) (string, error) {
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
cmd := exec.CommandContext(ctx, "wl-paste", "--no-newline")
output, err := cmd.Output()
if err != nil {
// wl-paste returns non-zero exit code if clipboard is empty or unavailable
// This is normal behavior, so we return empty string instead of error
return "", nil
}
return string(output), nil
}
// setClipboard sets the clipboard content using wl-copy
func setClipboard(ctx context.Context, text string, timeout time.Duration) error {
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
cmd := exec.CommandContext(ctx, "wl-copy")
cmd.Stdin = strings.NewReader(text)
if err := cmd.Run(); err != nil {
return fmt.Errorf("wl-copy failed: %w", err)
}
return nil
}
// checkClipboardAvailable checks if wl-clipboard tools are available
func checkClipboardAvailable() error {
if _, err := exec.LookPath("wl-copy"); err != nil {
return fmt.Errorf("wl-copy not found: %w (install wl-clipboard)", err)
}
if _, err := exec.LookPath("wl-paste"); err != nil {
return fmt.Errorf("wl-paste not found: %w (install wl-clipboard)", err)
}
return nil
}
+112
View File
@@ -0,0 +1,112 @@
package injection
import (
"context"
"fmt"
"time"
)
// Injector interface for text injection
type Injector interface {
Inject(ctx context.Context, text string) error
}
// Config for text injection
type Config struct {
Mode string // "clipboard", "type", "fallback"
AlwaysCopyClipboard bool // Always copy to clipboard regardless of mode
RestoreClipboard bool // Restore original clipboard after injection
WtypeTimeout time.Duration // Timeout for wtype commands
ClipboardTimeout time.Duration // Timeout for clipboard operations
}
// DefaultConfig returns sensible defaults for injection
func DefaultConfig() Config {
return Config{
Mode: "fallback",
AlwaysCopyClipboard: true,
RestoreClipboard: true,
WtypeTimeout: 5 * time.Second,
ClipboardTimeout: 3 * time.Second,
}
}
// injector implements the Injector interface
type injector struct {
config Config
}
// NewInjector creates a new injector with the given config
func NewInjector(config Config) Injector {
return &injector{
config: config,
}
}
// NewDefaultInjector creates an injector with default configuration
func NewDefaultInjector() Injector {
return NewInjector(DefaultConfig())
}
// Inject performs text injection based on the configured mode
func (i *injector) Inject(ctx context.Context, text string) error {
if text == "" {
return fmt.Errorf("cannot inject empty text")
}
// Always copy to clipboard if configured
var originalClipboard string
var err error
if i.config.AlwaysCopyClipboard || i.config.Mode == "clipboard" || i.config.Mode == "fallback" {
if err := checkClipboardAvailable(); err != nil {
return fmt.Errorf("clipboard tools not available: %w", err)
}
if i.config.RestoreClipboard {
originalClipboard, _ = getClipboard(ctx, i.config.ClipboardTimeout)
}
if err := setClipboard(ctx, text, i.config.ClipboardTimeout); err != nil {
return fmt.Errorf("failed to copy text to clipboard: %w", err)
}
}
// Handle different injection modes
switch i.config.Mode {
case "clipboard":
// Already handled above
return nil
case "type":
err = typeText(ctx, text, i.config.WtypeTimeout)
if err != nil {
return fmt.Errorf("failed to type text: %w", err)
}
case "fallback":
// Try typing first, fallback to clipboard
err = typeText(ctx, text, i.config.WtypeTimeout)
if err != nil {
// Typing failed, but clipboard is already set from above
// Just log the typing error but don't fail the injection
return nil
}
default:
return fmt.Errorf("unsupported injection mode: %s", i.config.Mode)
}
// Restore original clipboard if configured and we have it
if i.config.RestoreClipboard && originalClipboard != "" {
// Restore after a short delay to ensure the text has been processed
go func() {
time.Sleep(100 * time.Millisecond)
restoreCtx, cancel := context.WithTimeout(context.Background(), i.config.ClipboardTimeout)
defer cancel()
setClipboard(restoreCtx, originalClipboard, i.config.ClipboardTimeout)
}()
}
return nil
}
+36
View File
@@ -0,0 +1,36 @@
package injection
import (
"context"
"fmt"
"os/exec"
"time"
)
// typeText types the given text using wtype
func typeText(ctx context.Context, text string, timeout time.Duration) error {
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
// Check if wtype is available
if err := checkWtypeAvailable(); err != nil {
return err
}
cmd := exec.CommandContext(ctx, "wtype", text)
if err := cmd.Run(); err != nil {
return fmt.Errorf("wtype failed: %w", err)
}
return nil
}
// checkWtypeAvailable checks if wtype is available on the system
func checkWtypeAvailable() error {
if _, err := exec.LookPath("wtype"); err != nil {
return fmt.Errorf("wtype not found: %w (install wtype package)", err)
}
return nil
}
+25 -19
View File
@@ -7,6 +7,7 @@ import (
"sync/atomic" "sync/atomic"
"time" "time"
"github.com/leonardotrapani/hyprvoice/internal/injection"
"github.com/leonardotrapani/hyprvoice/internal/recording" "github.com/leonardotrapani/hyprvoice/internal/recording"
"github.com/leonardotrapani/hyprvoice/internal/transcriber" "github.com/leonardotrapani/hyprvoice/internal/transcriber"
) )
@@ -116,6 +117,19 @@ func (p *pipeline) run(ctx context.Context) {
} }
}() }()
// Forward errors from component channels to unified pipeline error channel
go func() {
for err := range tErrCh {
p.sendError("Transcription Error", "Transcription processing error", err)
}
}()
go func() {
for err := range rErrCh {
p.sendError("Recording Error", "Recording stream error", err)
}
}()
for { for {
select { select {
case <-frameCh: case <-frameCh:
@@ -127,14 +141,6 @@ func (p *pipeline) run(ctx context.Context) {
return return
} }
case err := <-tErrCh:
p.handleTranscriberError(err)
return
case err := <-rErrCh:
p.handleRecordingError(err)
return
case <-ctx.Done(): case <-ctx.Done():
return return
} }
@@ -191,14 +197,6 @@ func (p *pipeline) sendError(title, message string, err error) {
} }
} }
func (p *pipeline) handleTranscriberError(err error) {
p.sendError("Transcription Error", "Transcription processing error", err)
}
func (p *pipeline) handleRecordingError(err error) {
p.sendError("Recording Error", "Recording stream error", err)
}
func (p *pipeline) handleInjectAction(ctx context.Context, recorder *recording.Recorder, t transcriber.Transcriber) { func (p *pipeline) handleInjectAction(ctx context.Context, recorder *recording.Recorder, t transcriber.Transcriber) {
status := p.Status() status := p.Status()
@@ -214,6 +212,7 @@ func (p *pipeline) handleInjectAction(ctx context.Context, recorder *recording.R
if err := t.Stop(ctx); err != nil { if err := t.Stop(ctx); err != nil {
p.sendError("Transcription Error", "Failed to stop transcriber during injection", err) p.sendError("Transcription Error", "Failed to stop transcriber during injection", err)
return
} }
transcriptionText, err := t.GetFinalTranscription() transcriptionText, err := t.GetFinalTranscription()
@@ -223,9 +222,16 @@ func (p *pipeline) handleInjectAction(ctx context.Context, recorder *recording.R
} }
log.Printf("Pipeline: Final transcription text: %s", transcriptionText) log.Printf("Pipeline: Final transcription text: %s", transcriptionText)
log.Printf("Pipeline: Simulating injection work") injector := injection.NewDefaultInjector()
time.Sleep(10 * time.Millisecond)
log.Printf("Pipeline: Injection work done, returning to idle") if err := injector.Inject(ctx, transcriptionText); err != nil {
p.sendError("Injection Error", "Failed to inject text", err)
} else {
log.Printf("Pipeline: Text injection completed successfully")
}
// Return to idle state after injection is complete
p.setStatus(Idle)
} }
func (p *pipeline) Stop() { func (p *pipeline) Stop() {