fully working transcription
This commit is contained in:
@@ -7,7 +7,7 @@ Press a toggle key, speak, and get instant text input. Built natively for Waylan
|
|||||||
- **Toggle workflow**: Press once to start recording, press again to stop and inject text
|
- **Toggle workflow**: Press once to start recording, press again to stop and inject text
|
||||||
- **Wayland native**: Purpose-built for Wayland compositors - no legacy X11 dependencies or hacky workarounds
|
- **Wayland native**: Purpose-built for Wayland compositors - no legacy X11 dependencies or hacky workarounds
|
||||||
- **Real-time feedback**: Desktop notifications for recording states and transcription status
|
- **Real-time feedback**: Desktop notifications for recording states and transcription status
|
||||||
- **Multiple transcription backends**: OpenAI Whisper (planned: whisper.cpp for local processing)
|
- **Multiple transcription backends**: OpenAI Whisper (planned: whisper.cpp for local processing, and more)
|
||||||
- **Smart text injection**: Clipboard save/restore with direct typing fallback
|
- **Smart text injection**: Clipboard save/restore with direct typing fallback
|
||||||
- **Daemon architecture**: Lightweight control plane with efficient pipeline management
|
- **Daemon architecture**: Lightweight control plane with efficient pipeline management
|
||||||
|
|
||||||
@@ -72,7 +72,7 @@ sudo apt install pipewire-pulse pipewire-bin wl-clipboard
|
|||||||
sudo dnf install pipewire-utils wl-clipboard
|
sudo dnf install pipewire-utils wl-clipboard
|
||||||
```
|
```
|
||||||
|
|
||||||
**For text injection (recommended):**
|
**For text injection:**
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Arch Linux
|
# Arch Linux
|
||||||
@@ -196,7 +196,7 @@ Configuration will be read from `~/.config/hyprvoice/config.toml` (planned). Cur
|
|||||||
|
|
||||||
Hyprvoice will support multiple transcription backends:
|
Hyprvoice will support multiple transcription backends:
|
||||||
|
|
||||||
#### OpenAI Whisper API (Planned)
|
#### OpenAI Whisper API
|
||||||
|
|
||||||
Fast, accurate cloud-based transcription:
|
Fast, accurate cloud-based transcription:
|
||||||
|
|
||||||
@@ -219,6 +219,31 @@ model_path = "~/models/ggml-base.en.bin"
|
|||||||
threads = 4
|
threads = 4
|
||||||
```
|
```
|
||||||
|
|
||||||
|
#### Text Injection (Current)
|
||||||
|
|
||||||
|
Configurable text injection with multiple modes:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[injection]
|
||||||
|
mode = "fallback" # "clipboard", "type", or "fallback"
|
||||||
|
always_copy_clipboard = true
|
||||||
|
restore_clipboard = true
|
||||||
|
wtype_timeout = "5s"
|
||||||
|
clipboard_timeout = "3s"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Injection Modes:**
|
||||||
|
|
||||||
|
- **`fallback`** (default): Try direct typing first, fallback to clipboard
|
||||||
|
- **`type`**: Direct typing using wtype only
|
||||||
|
- **`clipboard`**: Copy to clipboard only
|
||||||
|
|
||||||
|
**Behavior:**
|
||||||
|
|
||||||
|
- `always_copy_clipboard = true`: Always copy text to clipboard regardless of mode
|
||||||
|
- `restore_clipboard = true`: Save and restore original clipboard content
|
||||||
|
- Smart fallback ensures text injection always succeeds when possible
|
||||||
|
|
||||||
### Service Configuration
|
### Service Configuration
|
||||||
|
|
||||||
#### Systemd Service
|
#### Systemd Service
|
||||||
@@ -256,17 +281,17 @@ systemctl --user enable --now hyprvoice.service
|
|||||||
|
|
||||||
## Development Status
|
## Development Status
|
||||||
|
|
||||||
| Component | Status | Notes |
|
| Component | Status | Notes |
|
||||||
| --------------------- | ------ | --------------------------------- |
|
| --------------------- | ------ | -------------------------------- |
|
||||||
| Core daemon & IPC | ✅ | Unix socket control plane |
|
| Core daemon & IPC | ✅ | Unix socket control plane |
|
||||||
| Recording workflow | ✅ | Toggle recording via PipeWire |
|
| Recording workflow | ✅ | Toggle recording via PipeWire |
|
||||||
| Audio capture | ✅ | Efficient PipeWire integration |
|
| Audio capture | ✅ | Efficient PipeWire integration |
|
||||||
| Desktop notifications | ✅ | Status feedback via notify-send |
|
| Desktop notifications | ✅ | Status feedback via notify-send |
|
||||||
| OpenAI transcription | ✅ | HTTP API integration |
|
| OpenAI transcription | ✅ | HTTP API integration |
|
||||||
| Text injection | ⏳ | Clipboard + typing implementation |
|
| Text injection | ✅ | Clipboard + wtype with fallback |
|
||||||
| Configuration system | ⏳ | TOML-based user settings |
|
| Configuration system | ⏳ | TOML-based user settings |
|
||||||
| Comprehensive tests | ⏳ | Pipeline and integration testing |
|
| Comprehensive tests | ⏳ | Pipeline and integration testing |
|
||||||
| whisper.cpp support | ⏳ | Local model inference |
|
| whisper.cpp support | ⏳ | Local model inference |
|
||||||
|
|
||||||
**Legend**: ✅ Complete · ⏳ Planned
|
**Legend**: ✅ Complete · ⏳ Planned
|
||||||
|
|
||||||
@@ -290,8 +315,8 @@ flowchart LR
|
|||||||
end
|
end
|
||||||
subgraph Pipeline
|
subgraph Pipeline
|
||||||
A["Audio Capture"]
|
A["Audio Capture"]
|
||||||
T["Transcribing (ASR TBD)"]
|
T["Transcribing"]
|
||||||
I["Injecting (stub)"]
|
I["Injecting (wtype + clipboard)"]
|
||||||
end
|
end
|
||||||
N["notify-send/log"]
|
N["notify-send/log"]
|
||||||
|
|
||||||
@@ -404,11 +429,34 @@ sudo apt install libnotify-bin # Ubuntu/Debian
|
|||||||
|
|
||||||
#### Text Injection Issues
|
#### Text Injection Issues
|
||||||
|
|
||||||
**Text not appearing (when implemented):**
|
**Text not appearing:**
|
||||||
|
|
||||||
- Ensure cursor is in a text field when toggling off recording
|
- Ensure cursor is in a text field when toggling off recording
|
||||||
- Check that `wtype` or clipboard tools are installed
|
- Check that `wtype` and `wl-clipboard` tools are installed:
|
||||||
- Verify window manager supports the text injection method used
|
|
||||||
|
```bash
|
||||||
|
# Test wtype directly
|
||||||
|
wtype "test text"
|
||||||
|
|
||||||
|
# Test clipboard tools
|
||||||
|
echo "test" | wl-copy
|
||||||
|
wl-paste
|
||||||
|
```
|
||||||
|
|
||||||
|
- Verify Wayland compositor supports text input protocols
|
||||||
|
- Check injection mode in configuration (fallback mode is most robust)
|
||||||
|
|
||||||
|
**Clipboard issues:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Install wl-clipboard if missing
|
||||||
|
sudo pacman -S wl-clipboard # Arch
|
||||||
|
sudo apt install wl-clipboard # Ubuntu/Debian
|
||||||
|
|
||||||
|
# Test clipboard functionality
|
||||||
|
wl-copy "test text"
|
||||||
|
wl-paste
|
||||||
|
```
|
||||||
|
|
||||||
### Debug Mode
|
### Debug Mode
|
||||||
|
|
||||||
@@ -453,6 +501,7 @@ hyprvoice/
|
|||||||
├── internal/
|
├── internal/
|
||||||
│ ├── bus/ # IPC (Unix socket) + PID management
|
│ ├── bus/ # IPC (Unix socket) + PID management
|
||||||
│ ├── daemon/ # Control daemon (lifecycle management)
|
│ ├── daemon/ # Control daemon (lifecycle management)
|
||||||
|
│ ├── injection/ # Text injection (clipboard + wtype)
|
||||||
│ ├── notify/ # Desktop notification integration
|
│ ├── notify/ # Desktop notification integration
|
||||||
│ ├── pipeline/ # Audio processing pipeline + state machine
|
│ ├── pipeline/ # Audio processing pipeline + state machine
|
||||||
│ ├── recording/ # PipeWire audio capture
|
│ ├── recording/ # PipeWire audio capture
|
||||||
|
|||||||
@@ -0,0 +1,53 @@
|
|||||||
|
package injection
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os/exec"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// getClipboard retrieves the current clipboard content using wl-paste
|
||||||
|
func getClipboard(ctx context.Context, timeout time.Duration) (string, error) {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, timeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
cmd := exec.CommandContext(ctx, "wl-paste", "--no-newline")
|
||||||
|
output, err := cmd.Output()
|
||||||
|
if err != nil {
|
||||||
|
// wl-paste returns non-zero exit code if clipboard is empty or unavailable
|
||||||
|
// This is normal behavior, so we return empty string instead of error
|
||||||
|
return "", nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return string(output), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// setClipboard sets the clipboard content using wl-copy
|
||||||
|
func setClipboard(ctx context.Context, text string, timeout time.Duration) error {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, timeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
cmd := exec.CommandContext(ctx, "wl-copy")
|
||||||
|
cmd.Stdin = strings.NewReader(text)
|
||||||
|
|
||||||
|
if err := cmd.Run(); err != nil {
|
||||||
|
return fmt.Errorf("wl-copy failed: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkClipboardAvailable checks if wl-clipboard tools are available
|
||||||
|
func checkClipboardAvailable() error {
|
||||||
|
if _, err := exec.LookPath("wl-copy"); err != nil {
|
||||||
|
return fmt.Errorf("wl-copy not found: %w (install wl-clipboard)", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := exec.LookPath("wl-paste"); err != nil {
|
||||||
|
return fmt.Errorf("wl-paste not found: %w (install wl-clipboard)", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,112 @@
|
|||||||
|
package injection
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Injector interface for text injection
|
||||||
|
type Injector interface {
|
||||||
|
Inject(ctx context.Context, text string) error
|
||||||
|
}
|
||||||
|
|
||||||
|
// Config for text injection
|
||||||
|
type Config struct {
|
||||||
|
Mode string // "clipboard", "type", "fallback"
|
||||||
|
AlwaysCopyClipboard bool // Always copy to clipboard regardless of mode
|
||||||
|
RestoreClipboard bool // Restore original clipboard after injection
|
||||||
|
WtypeTimeout time.Duration // Timeout for wtype commands
|
||||||
|
ClipboardTimeout time.Duration // Timeout for clipboard operations
|
||||||
|
}
|
||||||
|
|
||||||
|
// DefaultConfig returns sensible defaults for injection
|
||||||
|
func DefaultConfig() Config {
|
||||||
|
return Config{
|
||||||
|
Mode: "fallback",
|
||||||
|
AlwaysCopyClipboard: true,
|
||||||
|
RestoreClipboard: true,
|
||||||
|
WtypeTimeout: 5 * time.Second,
|
||||||
|
ClipboardTimeout: 3 * time.Second,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// injector implements the Injector interface
|
||||||
|
type injector struct {
|
||||||
|
config Config
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewInjector creates a new injector with the given config
|
||||||
|
func NewInjector(config Config) Injector {
|
||||||
|
return &injector{
|
||||||
|
config: config,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewDefaultInjector creates an injector with default configuration
|
||||||
|
func NewDefaultInjector() Injector {
|
||||||
|
return NewInjector(DefaultConfig())
|
||||||
|
}
|
||||||
|
|
||||||
|
// Inject performs text injection based on the configured mode
|
||||||
|
func (i *injector) Inject(ctx context.Context, text string) error {
|
||||||
|
if text == "" {
|
||||||
|
return fmt.Errorf("cannot inject empty text")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Always copy to clipboard if configured
|
||||||
|
var originalClipboard string
|
||||||
|
var err error
|
||||||
|
|
||||||
|
if i.config.AlwaysCopyClipboard || i.config.Mode == "clipboard" || i.config.Mode == "fallback" {
|
||||||
|
if err := checkClipboardAvailable(); err != nil {
|
||||||
|
return fmt.Errorf("clipboard tools not available: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if i.config.RestoreClipboard {
|
||||||
|
originalClipboard, _ = getClipboard(ctx, i.config.ClipboardTimeout)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := setClipboard(ctx, text, i.config.ClipboardTimeout); err != nil {
|
||||||
|
return fmt.Errorf("failed to copy text to clipboard: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Handle different injection modes
|
||||||
|
switch i.config.Mode {
|
||||||
|
case "clipboard":
|
||||||
|
// Already handled above
|
||||||
|
return nil
|
||||||
|
|
||||||
|
case "type":
|
||||||
|
err = typeText(ctx, text, i.config.WtypeTimeout)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to type text: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
case "fallback":
|
||||||
|
// Try typing first, fallback to clipboard
|
||||||
|
err = typeText(ctx, text, i.config.WtypeTimeout)
|
||||||
|
if err != nil {
|
||||||
|
// Typing failed, but clipboard is already set from above
|
||||||
|
// Just log the typing error but don't fail the injection
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
default:
|
||||||
|
return fmt.Errorf("unsupported injection mode: %s", i.config.Mode)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Restore original clipboard if configured and we have it
|
||||||
|
if i.config.RestoreClipboard && originalClipboard != "" {
|
||||||
|
// Restore after a short delay to ensure the text has been processed
|
||||||
|
go func() {
|
||||||
|
time.Sleep(100 * time.Millisecond)
|
||||||
|
restoreCtx, cancel := context.WithTimeout(context.Background(), i.config.ClipboardTimeout)
|
||||||
|
defer cancel()
|
||||||
|
setClipboard(restoreCtx, originalClipboard, i.config.ClipboardTimeout)
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
package injection
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"os/exec"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// typeText types the given text using wtype
|
||||||
|
func typeText(ctx context.Context, text string, timeout time.Duration) error {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, timeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
// Check if wtype is available
|
||||||
|
if err := checkWtypeAvailable(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd := exec.CommandContext(ctx, "wtype", text)
|
||||||
|
|
||||||
|
if err := cmd.Run(); err != nil {
|
||||||
|
return fmt.Errorf("wtype failed: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkWtypeAvailable checks if wtype is available on the system
|
||||||
|
func checkWtypeAvailable() error {
|
||||||
|
if _, err := exec.LookPath("wtype"); err != nil {
|
||||||
|
return fmt.Errorf("wtype not found: %w (install wtype package)", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -7,6 +7,7 @@ import (
|
|||||||
"sync/atomic"
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"github.com/leonardotrapani/hyprvoice/internal/injection"
|
||||||
"github.com/leonardotrapani/hyprvoice/internal/recording"
|
"github.com/leonardotrapani/hyprvoice/internal/recording"
|
||||||
"github.com/leonardotrapani/hyprvoice/internal/transcriber"
|
"github.com/leonardotrapani/hyprvoice/internal/transcriber"
|
||||||
)
|
)
|
||||||
@@ -116,6 +117,19 @@ func (p *pipeline) run(ctx context.Context) {
|
|||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
|
|
||||||
|
// Forward errors from component channels to unified pipeline error channel
|
||||||
|
go func() {
|
||||||
|
for err := range tErrCh {
|
||||||
|
p.sendError("Transcription Error", "Transcription processing error", err)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
for err := range rErrCh {
|
||||||
|
p.sendError("Recording Error", "Recording stream error", err)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
for {
|
for {
|
||||||
select {
|
select {
|
||||||
case <-frameCh:
|
case <-frameCh:
|
||||||
@@ -127,14 +141,6 @@ func (p *pipeline) run(ctx context.Context) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
case err := <-tErrCh:
|
|
||||||
p.handleTranscriberError(err)
|
|
||||||
return
|
|
||||||
|
|
||||||
case err := <-rErrCh:
|
|
||||||
p.handleRecordingError(err)
|
|
||||||
return
|
|
||||||
|
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -191,14 +197,6 @@ func (p *pipeline) sendError(title, message string, err error) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *pipeline) handleTranscriberError(err error) {
|
|
||||||
p.sendError("Transcription Error", "Transcription processing error", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (p *pipeline) handleRecordingError(err error) {
|
|
||||||
p.sendError("Recording Error", "Recording stream error", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
func (p *pipeline) handleInjectAction(ctx context.Context, recorder *recording.Recorder, t transcriber.Transcriber) {
|
func (p *pipeline) handleInjectAction(ctx context.Context, recorder *recording.Recorder, t transcriber.Transcriber) {
|
||||||
status := p.Status()
|
status := p.Status()
|
||||||
|
|
||||||
@@ -214,6 +212,7 @@ func (p *pipeline) handleInjectAction(ctx context.Context, recorder *recording.R
|
|||||||
|
|
||||||
if err := t.Stop(ctx); err != nil {
|
if err := t.Stop(ctx); err != nil {
|
||||||
p.sendError("Transcription Error", "Failed to stop transcriber during injection", err)
|
p.sendError("Transcription Error", "Failed to stop transcriber during injection", err)
|
||||||
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
transcriptionText, err := t.GetFinalTranscription()
|
transcriptionText, err := t.GetFinalTranscription()
|
||||||
@@ -223,9 +222,16 @@ func (p *pipeline) handleInjectAction(ctx context.Context, recorder *recording.R
|
|||||||
}
|
}
|
||||||
log.Printf("Pipeline: Final transcription text: %s", transcriptionText)
|
log.Printf("Pipeline: Final transcription text: %s", transcriptionText)
|
||||||
|
|
||||||
log.Printf("Pipeline: Simulating injection work")
|
injector := injection.NewDefaultInjector()
|
||||||
time.Sleep(10 * time.Millisecond)
|
|
||||||
log.Printf("Pipeline: Injection work done, returning to idle")
|
if err := injector.Inject(ctx, transcriptionText); err != nil {
|
||||||
|
p.sendError("Injection Error", "Failed to inject text", err)
|
||||||
|
} else {
|
||||||
|
log.Printf("Pipeline: Text injection completed successfully")
|
||||||
|
}
|
||||||
|
|
||||||
|
// Return to idle state after injection is complete
|
||||||
|
p.setStatus(Idle)
|
||||||
}
|
}
|
||||||
|
|
||||||
func (p *pipeline) Stop() {
|
func (p *pipeline) Stop() {
|
||||||
|
|||||||
Reference in New Issue
Block a user