Initial code commit
This commit is contained in:
大森
commit
89d9f1928a
43
.gitignore
vendored
Normal file
43
.gitignore
vendored
Normal file
@ -0,0 +1,43 @@
|
||||
# OS / editors
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
.vscode/
|
||||
.idea/
|
||||
|
||||
# Python
|
||||
.venv/
|
||||
venv/
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
.pytest_cache/
|
||||
.mypy_cache/
|
||||
.ruff_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
*.egg-info/
|
||||
|
||||
# Node / frontend
|
||||
node_modules/
|
||||
dist/
|
||||
build/
|
||||
.next/
|
||||
coverage/
|
||||
*.tsbuildinfo
|
||||
|
||||
# Rust / Tauri
|
||||
/target/
|
||||
**/target/
|
||||
|
||||
# Local tooling caches
|
||||
.codegraph/
|
||||
|
||||
# Nested repositories
|
||||
test_window/
|
||||
|
||||
# Generated / local artifacts
|
||||
*.log
|
||||
*.tmp
|
||||
backup/
|
||||
**/backup/
|
||||
PixPin_*.png
|
||||
*_vis.png
|
||||
632
desktop_mark/TAURI_REWRITE_SPEC.md
Normal file
632
desktop_mark/TAURI_REWRITE_SPEC.md
Normal file
@ -0,0 +1,632 @@
|
||||
# Rust + Tauri 重构开发文档
|
||||
|
||||
## 1. 目标
|
||||
|
||||
用 Rust + Tauri 重构当前 Python + PySide6 版本，功能行为保持一致：
|
||||
|
||||
- 启动后显示 80×80 置顶悬浮窗。
|
||||
- `Ctrl+1` 进入窗口识别模式。
|
||||
- 全屏半透明遮罩覆盖虚拟屏幕。
|
||||
- 鼠标移动时识别底层应用窗口，蓝色边框高亮；窗口内部无遮罩。
|
||||
- 鼠标在桌面或无合格窗口时，默认选择整个虚拟屏幕。
|
||||
- 鼠标点击锁定当前窗口；锁定后鼠标移动不再切换目标。
|
||||
- `Esc` 或回车退出当前选择/标注流程，不删除已保存标注。
|
||||
- 锁定窗口后 `Ctrl+R` 进入矩形标注模式。
|
||||
- 可创建多个标注框；标注框可选中、移动、缩放。
|
||||
- 左侧显示 300px 标注信息面板。
|
||||
- 标注框附近显示编辑表单，可编辑控件名、描述、类型、动作、特征图占位。
|
||||
- 自动保存到用户目录 JSON 文件。
|
||||
|
||||
首版仍只支持 Windows 10/11。
|
||||
|
||||
## 2. 当前 Python 版本功能清单
|
||||
|
||||
### 2.1 应用启动
|
||||
|
||||
当前入口为 `python -m desktop_mark`。
|
||||
|
||||
启动时创建：
|
||||
|
||||
- `WindowProvider`：Win32 窗口识别。
|
||||
- `FloatingButton`：80×80 悬浮窗。
|
||||
- `OverlayWindow`：窗口选择与标注 overlay。
|
||||
- `AnnotationPanel`：左侧标注信息面板。
|
||||
- `AnnotationFormPopup`：标注编辑表单。
|
||||
- `HotkeyWindow`：全局快捷键桥接。
|
||||
|
||||
启动后只显示悬浮窗；overlay、信息面板、表单默认隐藏。
|
||||
|
||||
### 2.2 DPI 策略
|
||||
|
||||
当前 Python 版启动时调用：
|
||||
|
||||
- `SetProcessDpiAwarenessContext(DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2)`
|
||||
- Qt 高 DPI rounding policy：`PassThrough`
|
||||
|
||||
Tauri 版应在 Rust 入口尽早设置 Win32 DPI awareness，避免 Win32 坐标和 WebView 坐标偏移。
|
||||
|
||||
### 2.3 悬浮窗
|
||||
|
||||
当前行为：
|
||||
|
||||
- 固定尺寸：80×80。
|
||||
- 置顶、无边框、工具窗口。
|
||||
- 深色半透明圆角背景。
|
||||
- 主文本：`RPA`。
|
||||
- 副文本：`Ctrl+1`。
|
||||
- 点击等价于触发窗口选择。
|
||||
- 支持鼠标拖动移动。
|
||||
- 自身窗口句柄必须加入窗口识别排除列表，避免选中工具自身。
|
||||
|
||||
### 2.4 全局快捷键
|
||||
|
||||
当前快捷键：
|
||||
|
||||
- `Ctrl+1`：进入窗口选择模式。
|
||||
- `Ctrl+R`：进入标注模式；如果没有锁定窗口，则先进入窗口选择模式。
|
||||
|
||||
当前实现使用 Win32 `RegisterHotKey`，不是键盘 hook。Tauri 版也应使用 Rust 原生层注册系统热键，避免管理员权限和杀软误报。
|
||||
|
||||
### 2.5 窗口识别
|
||||
|
||||
当前算法：
|
||||
|
||||
1. 不直接使用 `WindowFromPoint`，因为全屏 overlay 会遮住底层窗口。
|
||||
2. 从 `GetTopWindow(None)` 开始，按 Z 序循环 `GetWindow(hwnd, GW_HWNDNEXT)`。
|
||||
3. 跳过：
|
||||
- 已排除窗口句柄。
|
||||
- 不可见窗口。
|
||||
- 最小化窗口。
|
||||
- `WS_EX_TOOLWINDOW` 工具窗口。
|
||||
- 宽高小于等于 0 的窗口。
|
||||
4. 用 `GetWindowRect(hwnd)` 得到窗口矩形。
|
||||
5. 返回第一个包含鼠标点的合格顶层窗口。
|
||||
6. 标题来自 `GetWindowText(hwnd)`。
|
||||
7. 进程名来自 `GetWindowThreadProcessId` + `OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION)` + 模块路径文件名。
|
||||
8. 没有命中任何窗口时，返回虚拟屏幕桌面窗口：
|
||||
- `hwnd = null`
|
||||
- `title = "Desktop"`
|
||||
- `process_name = "explorer.exe"`
|
||||
- `rect = virtual_screen_rect`
|
||||
- `is_desktop = true`
|
||||
|
||||
虚拟屏幕使用：
|
||||
|
||||
- `SM_XVIRTUALSCREEN`
|
||||
- `SM_YVIRTUALSCREEN`
|
||||
- `SM_CXVIRTUALSCREEN`
|
||||
- `SM_CYVIRTUALSCREEN`
|
||||
|
||||
### 2.6 Overlay 状态机
|
||||
|
||||
状态枚举：
|
||||
|
||||
```rust
|
||||
enum OverlayMode {
|
||||
Idle,
|
||||
PickingWindow,
|
||||
WindowLocked,
|
||||
DrawingAnnotation,
|
||||
EditingAnnotation,
|
||||
MovingAnnotation,
|
||||
ResizingAnnotation,
|
||||
}
|
||||
```
|
||||
|
||||
行为：
|
||||
|
||||
- `Idle`：overlay 隐藏。
|
||||
- `PickingWindow`：overlay 显示，每 50ms 读取鼠标位置并识别窗口。
|
||||
- `WindowLocked`：点击窗口后进入；蓝框保持锁定，不再跟随鼠标。
|
||||
- `DrawingAnnotation`：锁定窗口内拖拽新建矩形。
|
||||
- `EditingAnnotation`：显示表单，选中标注。
|
||||
- `MovingAnnotation`：拖动已有标注框。
|
||||
- `ResizingAnnotation`：拖动 8 个 resize handles。
|
||||
|
||||
### 2.7 Overlay 绘制规则
|
||||
|
||||
当前视觉规则：
|
||||
|
||||
- 全屏半透明黑色遮罩，透明度约 120/255。
|
||||
- 当前目标窗口区域清除遮罩。
|
||||
- 目标窗口边缘绘制 2px 蓝色边框。
|
||||
- 右上角显示窗口标题和 bbox。
|
||||
- 已有标注框为橙色 2px 边框。
|
||||
- 标注序号显示在框左上角。
|
||||
- 选中标注显示 8 个白底蓝边 resize handles。
|
||||
- 新建中的临时框为蓝色 2px 边框。
|
||||
|
||||
Tauri 版建议：
|
||||
|
||||
- 用独立透明 always-on-top overlay window。
|
||||
- 前端用 SVG 或 Canvas 绘制遮罩、挖空、边框、标注框。
|
||||
- 遮罩挖空可用 SVG `mask` / `clipPath`，或 Canvas `globalCompositeOperation = "destination-out"`。
|
||||
|
||||
### 2.8 标注交互
|
||||
|
||||
创建标注：
|
||||
|
||||
1. `Ctrl+R` 进入 `DrawingAnnotation`。
|
||||
2. 鼠标左键按下记录起点。
|
||||
3. 拖动时显示临时矩形。
|
||||
4. 鼠标释放：
|
||||
- 对矩形做 `normalized()`。
|
||||
- 裁剪到锁定窗口 `clamped_to(locked_window.rect)`。
|
||||
- 小于 `6×6` 像素视为误触，不创建。
|
||||
- 合格则添加到 `project.annotations`。
|
||||
- 进入 `EditingAnnotation`。
|
||||
- 在标注框右下附近显示编辑表单。
|
||||
|
||||
命中测试顺序：
|
||||
|
||||
1. resize handle。
|
||||
2. 标注框内部移动。
|
||||
3. 空白区域新建。
|
||||
|
||||
多个标注重叠时，最后创建的标注优先命中。
|
||||
|
||||
移动规则：
|
||||
|
||||
- 移动时保持标注框宽高不变。
|
||||
- 移动结果必须完整留在锁定窗口内。
|
||||
- 不允许因简单裁剪导致宽高变小。
|
||||
|
||||
缩放规则：
|
||||
|
||||
- 8 个 handles：`nw`、`n`、`ne`、`e`、`se`、`s`、`sw`、`w`。
|
||||
- 最小尺寸：`6×6`。
|
||||
- 缩放结果裁剪到锁定窗口内。
|
||||
|
||||
取消规则：
|
||||
|
||||
- `Esc` 或回车：退出当前 overlay 流程。
|
||||
- 清空当前 hover/locked/temp 状态。
|
||||
- 隐藏 overlay、左侧面板、表单。
|
||||
- 不删除内存中的项目数据和已保存数据。
|
||||
|
||||
## 3. 数据模型契约
|
||||
|
||||
Tauri 版 Rust 与前端必须使用同一份 JSON schema。建议 Rust `serde` 类型如下：
|
||||
|
||||
```rust
|
||||
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
|
||||
pub struct Rect {
|
||||
pub x: i32,
|
||||
pub y: i32,
|
||||
pub width: i32,
|
||||
pub height: i32,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
pub struct WindowInfo {
|
||||
pub hwnd: Option<isize>,
|
||||
pub title: String,
|
||||
pub process_name: String,
|
||||
pub rect: Rect,
|
||||
pub is_desktop: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum AnnotationType {
|
||||
Button,
|
||||
Input,
|
||||
Text,
|
||||
Image,
|
||||
Other,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
pub struct Annotation {
|
||||
pub id: String,
|
||||
pub rect: Rect,
|
||||
pub control_name: String,
|
||||
pub description: String,
|
||||
#[serde(rename = "type")]
|
||||
pub annotation_type: AnnotationType,
|
||||
pub actions: Vec<String>,
|
||||
pub feature_preview_path: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
pub struct AnnotationProject {
|
||||
pub window: WindowInfo,
|
||||
pub custom_window_name: String,
|
||||
pub window_description: String,
|
||||
pub annotations: Vec<Annotation>,
|
||||
}
|
||||
```
|
||||
|
||||
### 3.1 `Rect` 方法
|
||||
|
||||
必须实现并测试：
|
||||
|
||||
- `right() -> i32`：`x + width`
|
||||
- `bottom() -> i32`：`y + height`
|
||||
- `contains(x, y) -> bool`：使用 normalized 后半开区间 `[x, right)`、`[y, bottom)`。
|
||||
- `normalized() -> Rect`：支持负向拖拽。
|
||||
- `clamped_to(bounds) -> Rect`：矩形裁剪到边界内。
|
||||
- `move_clamped(dx, dy, bounds) -> Rect`：保持原宽高移动，并完整限制在边界内。
|
||||
|
||||
### 3.2 坐标约定
|
||||
|
||||
所有坐标都使用屏幕绝对物理像素：
|
||||
|
||||
- 窗口 bbox：屏幕绝对物理像素。
|
||||
- 标注框：屏幕绝对物理像素。
|
||||
- JSON 保存：屏幕绝对物理像素。
|
||||
|
||||
首版不实现窗口移动后的标注自动重定位。
|
||||
|
||||
## 4. JSON 保存格式
|
||||
|
||||
保存路径必须兼容当前版本：
|
||||
|
||||
```text
|
||||
%USERPROFILE%\.desktop_mark\last_project.json
|
||||
```
|
||||
|
||||
JSON 字段固定：
|
||||
|
||||
```json
|
||||
{
|
||||
"window": {
|
||||
"hwnd": 123456,
|
||||
"title": "Untitled - Notepad",
|
||||
"process_name": "notepad.exe",
|
||||
"rect": { "x": 100, "y": 100, "width": 800, "height": 600 },
|
||||
"is_desktop": false
|
||||
},
|
||||
"custom_window_name": "登录页",
|
||||
"window_description": "用户登录窗口",
|
||||
"annotations": [
|
||||
{
|
||||
"id": "uuid-string",
|
||||
"rect": { "x": 150, "y": 160, "width": 120, "height": 32 },
|
||||
"control_name": "用户名输入框",
|
||||
"description": "输入登录用户名",
|
||||
"type": "input",
|
||||
"actions": ["input_text"],
|
||||
"feature_preview_path": null
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
保存时机：
|
||||
|
||||
- 标注表单保存后。
|
||||
- 窗口名称或窗口简介变化后。
|
||||
|
||||
加载规则：
|
||||
|
||||
- 启动时如果文件存在则加载到内存。
|
||||
- 不自动显示 overlay。
|
||||
- 加载失败时弹出错误，继续空项目。
|
||||
- 不覆盖坏文件。
|
||||
|
||||
## 5. Tauri 推荐架构
|
||||
|
||||
### 5.1 窗口划分
|
||||
|
||||
建议使用 4 个 Tauri windows：
|
||||
|
||||
1. `floating`
|
||||
- 80×80。
|
||||
- always-on-top。
|
||||
- decorations false。
|
||||
- transparent true。
|
||||
- resizable false。
|
||||
- skip taskbar true。
|
||||
- 显示 `RPA` / `Ctrl+1`。
|
||||
|
||||
2. `overlay`
|
||||
- 覆盖虚拟屏幕。
|
||||
- always-on-top。
|
||||
- decorations false。
|
||||
- transparent true。
|
||||
- skip taskbar true。
|
||||
- 初始隐藏。
|
||||
- 用于窗口选择、遮罩、标注绘制。
|
||||
|
||||
3. `panel`
|
||||
- 宽 300。
|
||||
- 高度为虚拟屏幕高度。
|
||||
- 位于虚拟屏幕左侧。
|
||||
- always-on-top。
|
||||
- 初始隐藏。
|
||||
- 显示窗口信息和标注列表。
|
||||
|
||||
4. `annotation_form`
|
||||
- 约 320×420。
|
||||
- always-on-top。
|
||||
- 初始隐藏。
|
||||
- 位于标注框右下附近，越界则左/上偏移。
|
||||
|
||||
也可以把 `panel` 和 `annotation_form` 做成 overlay 内部 DOM，但独立窗口更接近当前 PySide6 行为。
|
||||
|
||||
### 5.2 Rust 模块建议
|
||||
|
||||
```text
|
||||
src-tauri/src/
|
||||
main.rs
|
||||
models.rs
|
||||
rect.rs
|
||||
win32_window.rs
|
||||
hotkeys.rs
|
||||
storage.rs
|
||||
window_manager.rs
|
||||
commands.rs
|
||||
```
|
||||
|
||||
职责：
|
||||
|
||||
- `models.rs`：`WindowInfo`、`Annotation`、`AnnotationProject`、`AnnotationType`。
|
||||
- `rect.rs`：`Rect` 几何方法和单元测试。
|
||||
- `win32_window.rs`：窗口枚举、进程名、虚拟屏幕。
|
||||
- `hotkeys.rs`：`RegisterHotKey` 或 Tauri global shortcut 适配。
|
||||
- `storage.rs`：JSON 保存/加载。
|
||||
- `window_manager.rs`：创建/定位/显示隐藏 Tauri windows。
|
||||
- `commands.rs`：暴露给前端的 Tauri commands。
|
||||
|
||||
### 5.3 前端模块建议
|
||||
|
||||
```text
|
||||
src/
|
||||
main.tsx
|
||||
state/projectStore.ts
|
||||
types.ts
|
||||
windows/Floating.tsx
|
||||
windows/Overlay.tsx
|
||||
windows/Panel.tsx
|
||||
windows/AnnotationForm.tsx
|
||||
geometry/rect.ts
|
||||
```
|
||||
|
||||
职责：
|
||||
|
||||
- `types.ts`：与 Rust serde 对齐的类型。
|
||||
- `projectStore.ts`：当前项目、锁定窗口、hover 窗口、选中标注、overlay mode。
|
||||
- `Overlay.tsx`：遮罩、窗口高亮、标注框、鼠标交互。
|
||||
- `Panel.tsx`：窗口信息、窗口名称/简介、标注列表。
|
||||
- `AnnotationForm.tsx`：控件名、描述、类型、动作、预览占位。
|
||||
- `rect.ts`：前端几何算法，必须与 Rust 测试一致。
|
||||
|
||||
## 6. Rust Native API / Tauri Commands
|
||||
|
||||
建议命令：
|
||||
|
||||
```rust
|
||||
#[tauri::command]
|
||||
fn get_virtual_screen_rect() -> Rect;
|
||||
|
||||
#[tauri::command]
|
||||
fn window_at(x: i32, y: i32, excluded_hwnds: Vec<isize>) -> WindowInfo;
|
||||
|
||||
#[tauri::command]
|
||||
fn foreground_window(excluded_hwnds: Vec<isize>) -> Option<WindowInfo>;
|
||||
|
||||
#[tauri::command]
|
||||
fn save_project(project: AnnotationProject) -> Result<(), String>;
|
||||
|
||||
#[tauri::command]
|
||||
fn load_project() -> Result<Option<AnnotationProject>, String>;
|
||||
|
||||
#[tauri::command]
|
||||
fn set_tool_window_exclusions(labels: Vec<String>) -> Result<Vec<isize>, String>;
|
||||
```
|
||||
|
||||
前端事件：
|
||||
|
||||
```text
|
||||
hotkey:start-window-picking
|
||||
hotkey:start-annotation
|
||||
project:changed
|
||||
annotation:edit-requested
|
||||
annotation:selected
|
||||
flow:cancelled
|
||||
```
|
||||
|
||||
## 7. Win32 实现要点
|
||||
|
||||
Rust 可使用 `windows` crate。
|
||||
|
||||
必须调用或等价实现：
|
||||
|
||||
- `SetProcessDpiAwarenessContext(DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2)`
|
||||
- `GetTopWindow(HWND(0))`
|
||||
- `GetWindow(hwnd, GW_HWNDNEXT)`
|
||||
- `IsWindowVisible(hwnd)`
|
||||
- `IsIconic(hwnd)`
|
||||
- `GetWindowLongPtrW(hwnd, GWL_EXSTYLE)`，跳过 `WS_EX_TOOLWINDOW`
|
||||
- `GetWindowRect(hwnd)`
|
||||
- `GetWindowTextW(hwnd)`
|
||||
- `GetWindowThreadProcessId(hwnd, &mut pid)`
|
||||
- `OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, false, pid)`
|
||||
- `QueryFullProcessImageNameW` 或等价 API 获取进程路径
|
||||
- `GetSystemMetrics(SM_XVIRTUALSCREEN)` 等虚拟屏幕指标
|
||||
|
||||
不要把 overlay window 作为候选窗口。所有 Tauri 工具窗口 hwnd 必须加入排除集合。
|
||||
|
||||
## 8. UI 字段与枚举
|
||||
|
||||
### 8.1 标注类型
|
||||
|
||||
固定值：
|
||||
|
||||
```text
|
||||
button
|
||||
input
|
||||
text
|
||||
image
|
||||
other
|
||||
```
|
||||
|
||||
### 8.2 动作枚举
|
||||
|
||||
固定值：
|
||||
|
||||
```text
|
||||
click
|
||||
double_click
|
||||
right_click
|
||||
input_text
|
||||
hover
|
||||
scroll
|
||||
```
|
||||
|
||||
### 8.3 左侧面板字段
|
||||
|
||||
- 窗口标题。
|
||||
- 进程名。
|
||||
- bbox：`x, y, width, height`。
|
||||
- 窗口名称：可编辑。
|
||||
- 窗口简介：可编辑。
|
||||
- 标注列表：`序号. label (type)`。
|
||||
|
||||
`label` 优先级：
|
||||
|
||||
1. `control_name`
|
||||
2. `description`
|
||||
3. `标注 {index}`
|
||||
|
||||
### 8.4 标注表单字段
|
||||
|
||||
- 控件名：文本输入。
|
||||
- 描述：多行文本。
|
||||
- 类型：下拉选择。
|
||||
- 动作：多选列表。
|
||||
- 特征图：占位，显示 `预览待生成`。
|
||||
- 保存按钮。
|
||||
- 取消按钮。
|
||||
|
||||
保存行为：
|
||||
|
||||
- 更新当前 annotation。
|
||||
- `feature_preview_path` 保持 `null`。
|
||||
- 刷新左侧列表。
|
||||
- 自动保存 JSON。
|
||||
- 隐藏表单。
|
||||
|
||||
取消行为：
|
||||
|
||||
- 只关闭表单。
|
||||
- 不删除 annotation。
|
||||
|
||||
## 9. 关键行为验收
|
||||
|
||||
### 9.1 启动
|
||||
|
||||
命令启动后：
|
||||
|
||||
- 桌面出现 80×80 `RPA` 悬浮窗。
|
||||
- overlay、panel、form 不显示。
|
||||
- 无控制台异常。
|
||||
|
||||
### 9.2 窗口选择
|
||||
|
||||
准备：打开记事本和浏览器。
|
||||
|
||||
操作：按 `Ctrl+1`。
|
||||
|
||||
预期：
|
||||
|
||||
- 全屏半透明遮罩出现。
|
||||
- 鼠标移动到记事本/浏览器时，对应窗口显示蓝色边框。
|
||||
- 被高亮窗口内部无遮罩。
|
||||
- 鼠标移动到桌面空白时，蓝框覆盖虚拟屏幕。
|
||||
|
||||
### 9.3 锁定与取消
|
||||
|
||||
操作：在浏览器窗口上点击。
|
||||
|
||||
预期：
|
||||
|
||||
- 蓝框锁定浏览器。
|
||||
- 鼠标移动到其他窗口不切换蓝框。
|
||||
- 按 `Esc` 或回车后 overlay 隐藏。
|
||||
- 悬浮窗仍存在。
|
||||
|
||||
### 9.4 标注
|
||||
|
||||
操作：重新 `Ctrl+1` 锁定记事本，按 `Ctrl+R`，拖拽两个矩形。
|
||||
|
||||
预期：
|
||||
|
||||
- 两个橙色标注框出现。
|
||||
- 标注框不能拖出记事本窗口。
|
||||
- 选中框显示 8 个调整手柄。
|
||||
- 左侧出现 300px 信息面板并列出两个标注。
|
||||
|
||||
### 9.5 表单与保存
|
||||
|
||||
操作：完成标注后，在表单输入：
|
||||
|
||||
- 控件名：`用户名输入框`
|
||||
- 描述：`输入登录用户名`
|
||||
- 类型：`input`
|
||||
- 动作：`input_text`
|
||||
|
||||
点击保存。
|
||||
|
||||
预期：
|
||||
|
||||
- 左侧列表同步显示该标注。
|
||||
- `%USERPROFILE%\.desktop_mark\last_project.json` 存在。
|
||||
- JSON 包含对应 `control_name`、`description`、`type`、`actions`。
|
||||
|
||||
### 9.6 边界
|
||||
|
||||
操作：在锁定窗口边缘向窗口外拖拽创建标注。
|
||||
|
||||
预期：
|
||||
|
||||
- 最终矩形被裁剪到锁定窗口内。
|
||||
- 小于 `6×6` 的点击/拖拽不会新增标注。
|
||||
|
||||
## 10. 与当前 Python 版本保持一致的边界决定
|
||||
|
||||
- 只支持 Windows 10/11。
|
||||
- 坐标保存为屏幕绝对物理像素。
|
||||
- 不实现窗口移动后的标注自动重定位。
|
||||
- 不实现特征图真实截图，只显示占位。
|
||||
- 快捷键冲突时显示错误，不自动改用其他快捷键。
|
||||
- 工具自身窗口必须被窗口识别排除。
|
||||
- 没有命中应用窗口时选择整个虚拟屏幕。
|
||||
|
||||
## 11. 推荐开发顺序
|
||||
|
||||
1. 建 Tauri 项目和四个窗口。
|
||||
2. 实现 Rust `Rect`、模型、JSON 保存/加载和单元测试。
|
||||
3. 实现 Win32 虚拟屏幕与窗口识别。
|
||||
4. 实现全局快捷键。
|
||||
5. 实现 `floating` 窗口。
|
||||
6. 实现 `overlay` 状态机和窗口高亮。
|
||||
7. 实现标注框创建、移动、缩放。
|
||||
8. 实现 `panel` 和 `annotation_form`。
|
||||
9. 串联自动保存和启动加载。
|
||||
10. 按第 9 节逐项手工验收。
|
||||
|
||||
## 12. 当前 Python 版本验证基线
|
||||
|
||||
当前版本已验证：
|
||||
|
||||
```text
|
||||
uv venv --python 3.11 --seed --clear
|
||||
.\.venv\Scripts\python -m pip install -e .[dev]
|
||||
.\.venv\Scripts\python -m pytest
|
||||
```
|
||||
|
||||
结果：
|
||||
|
||||
```text
|
||||
6 passed
|
||||
```
|
||||
|
||||
启动烟测：
|
||||
|
||||
```text
|
||||
.\.venv\Scripts\python -m desktop_mark
|
||||
```
|
||||
|
||||
进程保持运行 3 秒，无 stdout/stderr 异常；测试后主动终止。
|
||||
7
desktop_mark/python/REAMD.md
Normal file
7
desktop_mark/python/REAMD.md
Normal file
@ -0,0 +1,7 @@
|
||||
# 桌面应用RPA标注工具
|
||||
|
||||
|
||||
### 期望实现目标
|
||||
打开软件后，桌面出现一个悬浮窗。80x80
|
||||
按下快捷键ctrl+1 开始识别窗口，默认全屏半透明遮罩，通过识别鼠标在哪个应用，或者哪个应用获取到了鼠标焦点，那么就获取应用的x,y,w,h信息的bbox边框位置，然后该应用窗口用蓝色边框切应用内窗口去掉半透明遮罩，同时如果鼠标移动到其他应用窗口，则继续切换窗口激活状态。如果鼠标在桌面上，没有任何应用窗口上，则默认选中全屏。然后点击鼠标后，则选中当前窗口，此时移动鼠标则不会激活其他窗口，可以通过ESC回车撤销选中。选中窗口后，可以通过Ctrl+r 开启矩形标注框，可以在选中的应用窗口中开始标注，同时屏幕左侧会出现一个300x1200的标注信息框，标注的信息会在此处展示，同时在顶部显示当前选中的窗口信息，可以定义窗口名称与简介
|
||||
标注可以多次标注，当开启标注后，拖动鼠标可以在应用窗口上进行绘制，点击鼠标后结束标注，同时已标注的框可以调整大小与位置，然后在标注框附近显示一个表单弹窗，可以输入控件、描述、类型、可选动作、特征图预览等
|
||||
26
desktop_mark/python/pyproject.toml
Normal file
26
desktop_mark/python/pyproject.toml
Normal file
@ -0,0 +1,26 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=68", "wheel"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "desktop-mark"
|
||||
version = "0.1.0"
|
||||
description = "Windows desktop RPA annotation tool"
|
||||
readme = "REAMD.md"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"PySide6>=6.7,<7",
|
||||
"pywin32>=306",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=8,<9",
|
||||
]
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
pythonpath = ["src"]
|
||||
4
desktop_mark/python/src/desktop_mark/__init__.py
Normal file
4
desktop_mark/python/src/desktop_mark/__init__.py
Normal file
@ -0,0 +1,4 @@
|
||||
"""Desktop RPA annotation tool."""
|
||||
|
||||
__all__ = ["__version__"]
|
||||
__version__ = "0.1.0"
|
||||
3
desktop_mark/python/src/desktop_mark/__main__.py
Normal file
3
desktop_mark/python/src/desktop_mark/__main__.py
Normal file
@ -0,0 +1,3 @@
|
||||
from .app import main
|
||||
|
||||
main()
|
||||
204
desktop_mark/python/src/desktop_mark/annotation_panel.py
Normal file
204
desktop_mark/python/src/desktop_mark/annotation_panel.py
Normal file
@ -0,0 +1,204 @@
|
||||
"""Annotation metadata side panel and edit popup."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from PySide6.QtCore import Qt, Signal
|
||||
from PySide6.QtGui import QCloseEvent
|
||||
from PySide6.QtWidgets import (
|
||||
QComboBox,
|
||||
QFormLayout,
|
||||
QHBoxLayout,
|
||||
QLabel,
|
||||
QLineEdit,
|
||||
QListWidget,
|
||||
QListWidgetItem,
|
||||
QPushButton,
|
||||
QPlainTextEdit,
|
||||
QVBoxLayout,
|
||||
QWidget,
|
||||
)
|
||||
|
||||
from .models import Annotation, AnnotationProject, AnnotationType, Rect
|
||||
|
||||
|
||||
ACTIONS = ["click", "double_click", "right_click", "input_text", "hover", "scroll"]
|
||||
|
||||
|
||||
class AnnotationPanel(QWidget):
|
||||
window_metadata_changed = Signal(str, str)
|
||||
annotation_selected = Signal(str)
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.project: AnnotationProject | None = None
|
||||
self._refreshing = False
|
||||
self.setWindowTitle("标注信息")
|
||||
self.setFixedWidth(300)
|
||||
self.setWindowFlags(Qt.WindowType.Tool | Qt.WindowType.WindowStaysOnTopHint)
|
||||
|
||||
self.title_label = QLabel("-")
|
||||
self.process_label = QLabel("-")
|
||||
self.bbox_label = QLabel("-")
|
||||
self.name_edit = QLineEdit()
|
||||
self.description_edit = QPlainTextEdit()
|
||||
self.description_edit.setFixedHeight(90)
|
||||
self.annotation_list = QListWidget()
|
||||
|
||||
layout = QVBoxLayout(self)
|
||||
layout.addWidget(QLabel("窗口信息"))
|
||||
form = QFormLayout()
|
||||
form.addRow("标题", self.title_label)
|
||||
form.addRow("进程", self.process_label)
|
||||
form.addRow("BBox", self.bbox_label)
|
||||
form.addRow("窗口名称", self.name_edit)
|
||||
form.addRow("窗口简介", self.description_edit)
|
||||
layout.addLayout(form)
|
||||
layout.addWidget(QLabel("标注列表"))
|
||||
layout.addWidget(self.annotation_list, 1)
|
||||
|
||||
self.name_edit.textChanged.connect(self._emit_window_metadata)
|
||||
self.description_edit.textChanged.connect(self._emit_window_metadata)
|
||||
self.annotation_list.currentItemChanged.connect(self._on_current_item_changed)
|
||||
|
||||
def set_project(self, project: AnnotationProject | None) -> None:
|
||||
self.project = project
|
||||
self.refresh()
|
||||
|
||||
def refresh(self) -> None:
|
||||
self._refreshing = True
|
||||
try:
|
||||
self.annotation_list.clear()
|
||||
if self.project is None:
|
||||
self.title_label.setText("-")
|
||||
self.process_label.setText("-")
|
||||
self.bbox_label.setText("-")
|
||||
self.name_edit.clear()
|
||||
self.description_edit.clear()
|
||||
return
|
||||
window = self.project.window
|
||||
rect = window.rect
|
||||
self.title_label.setText(window.title)
|
||||
self.process_label.setText(window.process_name or "-")
|
||||
self.bbox_label.setText(f"{rect.x}, {rect.y}, {rect.width}, {rect.height}")
|
||||
self.name_edit.setText(self.project.custom_window_name)
|
||||
self.description_edit.setPlainText(self.project.window_description)
|
||||
for index, annotation in enumerate(self.project.annotations, start=1):
|
||||
label = annotation.control_name or annotation.description or f"标注 {index}"
|
||||
item = QListWidgetItem(f"{index}. {label} ({annotation.type.value})")
|
||||
item.setData(Qt.ItemDataRole.UserRole, annotation.id)
|
||||
self.annotation_list.addItem(item)
|
||||
finally:
|
||||
self._refreshing = False
|
||||
|
||||
def _emit_window_metadata(self) -> None:
|
||||
if self._refreshing or self.project is None:
|
||||
return
|
||||
self.project.custom_window_name = self.name_edit.text()
|
||||
self.project.window_description = self.description_edit.toPlainText()
|
||||
self.window_metadata_changed.emit(
|
||||
self.project.custom_window_name,
|
||||
self.project.window_description,
|
||||
)
|
||||
|
||||
def _on_current_item_changed(self, current: QListWidgetItem | None, previous: QListWidgetItem | None) -> None:
|
||||
if current is None or self._refreshing:
|
||||
return
|
||||
annotation_id = current.data(Qt.ItemDataRole.UserRole)
|
||||
if annotation_id:
|
||||
self.annotation_selected.emit(str(annotation_id))
|
||||
|
||||
|
||||
class AnnotationFormPopup(QWidget):
|
||||
saved = Signal(object)
|
||||
cancelled = Signal()
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.annotation: Annotation | None = None
|
||||
self.setWindowTitle("编辑标注")
|
||||
self.setWindowFlags(Qt.WindowType.Tool | Qt.WindowType.WindowStaysOnTopHint)
|
||||
|
||||
self.control_name_edit = QLineEdit()
|
||||
self.description_edit = QPlainTextEdit()
|
||||
self.description_edit.setFixedHeight(100)
|
||||
self.type_combo = QComboBox()
|
||||
for annotation_type in AnnotationType:
|
||||
self.type_combo.addItem(annotation_type.value, annotation_type)
|
||||
self.actions_list = QListWidget()
|
||||
self.actions_list.setSelectionMode(QListWidget.SelectionMode.MultiSelection)
|
||||
for action in ACTIONS:
|
||||
self.actions_list.addItem(action)
|
||||
self.preview_label = QLabel("预览待生成")
|
||||
self.preview_label.setAlignment(Qt.AlignmentFlag.AlignCenter)
|
||||
self.preview_label.setMinimumHeight(80)
|
||||
self.preview_label.setStyleSheet("border: 1px dashed #999; color: #666;")
|
||||
|
||||
save_button = QPushButton("保存")
|
||||
cancel_button = QPushButton("取消")
|
||||
save_button.clicked.connect(self._save)
|
||||
cancel_button.clicked.connect(self._cancel)
|
||||
|
||||
form = QFormLayout()
|
||||
form.addRow("控件名", self.control_name_edit)
|
||||
form.addRow("描述", self.description_edit)
|
||||
form.addRow("类型", self.type_combo)
|
||||
form.addRow("动作", self.actions_list)
|
||||
form.addRow("特征图", self.preview_label)
|
||||
|
||||
buttons = QHBoxLayout()
|
||||
buttons.addStretch(1)
|
||||
buttons.addWidget(cancel_button)
|
||||
buttons.addWidget(save_button)
|
||||
|
||||
layout = QVBoxLayout(self)
|
||||
layout.addLayout(form)
|
||||
layout.addLayout(buttons)
|
||||
self.resize(320, 420)
|
||||
|
||||
def edit_annotation(self, annotation: Annotation, anchor_rect: Rect) -> None:
|
||||
self.annotation = annotation
|
||||
self.control_name_edit.setText(annotation.control_name)
|
||||
self.description_edit.setPlainText(annotation.description)
|
||||
index = self.type_combo.findData(annotation.type)
|
||||
self.type_combo.setCurrentIndex(index if index >= 0 else 0)
|
||||
selected_actions = set(annotation.actions)
|
||||
for row in range(self.actions_list.count()):
|
||||
item = self.actions_list.item(row)
|
||||
item.setSelected(item.text() in selected_actions)
|
||||
self.preview_label.setText("预览待生成")
|
||||
self._move_near(anchor_rect)
|
||||
self.show()
|
||||
self.raise_()
|
||||
self.activateWindow()
|
||||
|
||||
def closeEvent(self, event: QCloseEvent) -> None: # noqa: N802
|
||||
self.cancelled.emit()
|
||||
super().closeEvent(event)
|
||||
|
||||
def _save(self) -> None:
|
||||
if self.annotation is None:
|
||||
return
|
||||
self.annotation.control_name = self.control_name_edit.text().strip()
|
||||
self.annotation.description = self.description_edit.toPlainText().strip()
|
||||
self.annotation.type = self.type_combo.currentData()
|
||||
self.annotation.actions = [item.text() for item in self.actions_list.selectedItems()]
|
||||
self.annotation.feature_preview_path = None
|
||||
self.saved.emit(self.annotation)
|
||||
self.hide()
|
||||
|
||||
def _cancel(self) -> None:
|
||||
self.hide()
|
||||
self.cancelled.emit()
|
||||
|
||||
def _move_near(self, anchor_rect: Rect) -> None:
|
||||
screen = self.screen()
|
||||
available = screen.availableGeometry() if screen is not None else self.geometry()
|
||||
x = anchor_rect.right() + 12
|
||||
y = anchor_rect.bottom() + 12
|
||||
if x + self.width() > available.right():
|
||||
x = anchor_rect.x - self.width() - 12
|
||||
if y + self.height() > available.bottom():
|
||||
y = anchor_rect.y - self.height() - 12
|
||||
x = max(available.left(), min(x, available.right() - self.width()))
|
||||
y = max(available.top(), min(y, available.bottom() - self.height()))
|
||||
self.move(x, y)
|
||||
134
desktop_mark/python/src/desktop_mark/app.py
Normal file
134
desktop_mark/python/src/desktop_mark/app.py
Normal file
@ -0,0 +1,134 @@
|
||||
"""Application entrypoint and object wiring."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ctypes
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from PySide6.QtCore import Qt
|
||||
from PySide6.QtWidgets import QApplication, QMessageBox
|
||||
|
||||
from .annotation_panel import AnnotationFormPopup, AnnotationPanel
|
||||
from .floating_button import FloatingButton
|
||||
from .hotkeys import HotkeyWindow
|
||||
from .models import AnnotationProject
|
||||
from .overlay import OverlayWindow
|
||||
from .storage import DEFAULT_PROJECT_PATH, load_project, save_project
|
||||
from .win32_window import WindowProvider
|
||||
|
||||
|
||||
DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2 = -4
|
||||
|
||||
|
||||
class DesktopMarkApp:
|
||||
def __init__(self) -> None:
|
||||
self.qt_app = QApplication.instance() or QApplication(sys.argv)
|
||||
self.qt_app.setQuitOnLastWindowClosed(False)
|
||||
self.provider = WindowProvider()
|
||||
self.floating_button = FloatingButton()
|
||||
self.overlay = OverlayWindow(self.provider)
|
||||
self.panel = AnnotationPanel()
|
||||
self.form = AnnotationFormPopup()
|
||||
self.hotkeys = HotkeyWindow()
|
||||
self.project_path = DEFAULT_PROJECT_PATH
|
||||
self.project: AnnotationProject | None = self._load_last_project(self.project_path)
|
||||
|
||||
self._configure_panel_geometry()
|
||||
self._connect_signals()
|
||||
self._exclude_tool_windows()
|
||||
if self.project is not None:
|
||||
self.panel.set_project(self.project)
|
||||
self.overlay.set_project(self.project)
|
||||
self.hotkeys.register()
|
||||
self.floating_button.show()
|
||||
|
||||
def run(self) -> int:
|
||||
return self.qt_app.exec()
|
||||
|
||||
def _connect_signals(self) -> None:
|
||||
self.floating_button.clicked.connect(self.overlay.start_window_picking)
|
||||
self.hotkeys.start_window_picking.connect(self.overlay.start_window_picking)
|
||||
self.hotkeys.start_annotation.connect(self.overlay.start_annotation)
|
||||
self.overlay.project_changed.connect(self._on_project_changed)
|
||||
self.overlay.annotation_edit_requested.connect(self.form.edit_annotation)
|
||||
self.overlay.hidden_by_user.connect(self._hide_annotation_ui)
|
||||
self.panel.annotation_selected.connect(self.overlay.select_annotation)
|
||||
self.panel.window_metadata_changed.connect(self._on_window_metadata_changed)
|
||||
self.form.saved.connect(self._on_annotation_saved)
|
||||
self.form.cancelled.connect(self.overlay.update)
|
||||
self.qt_app.aboutToQuit.connect(self.hotkeys.unregister)
|
||||
|
||||
def _exclude_tool_windows(self) -> None:
|
||||
hwnds = {
|
||||
int(self.floating_button.winId()),
|
||||
int(self.overlay.winId()),
|
||||
int(self.panel.winId()),
|
||||
int(self.form.winId()),
|
||||
int(self.hotkeys.winId()),
|
||||
}
|
||||
self.provider.set_excluded_hwnds(hwnds)
|
||||
|
||||
def _configure_panel_geometry(self) -> None:
|
||||
virtual = self.provider.virtual_screen_rect()
|
||||
self.panel.setFixedHeight(virtual.height)
|
||||
self.panel.move(virtual.x, virtual.y)
|
||||
|
||||
def _on_project_changed(self, project: AnnotationProject | None) -> None:
|
||||
self.project = project
|
||||
self.panel.set_project(project)
|
||||
if project is not None:
|
||||
self.panel.show()
|
||||
|
||||
def _on_window_metadata_changed(self, name: str, description: str) -> None:
|
||||
if self.project is None:
|
||||
return
|
||||
self.project.custom_window_name = name
|
||||
self.project.window_description = description
|
||||
self._save_current_project()
|
||||
|
||||
def _on_annotation_saved(self, annotation) -> None: # noqa: ANN001
|
||||
if self.project is None:
|
||||
return
|
||||
self.panel.refresh()
|
||||
self.overlay.update()
|
||||
self._save_current_project()
|
||||
|
||||
def _hide_annotation_ui(self) -> None:
|
||||
self.panel.hide()
|
||||
self.form.hide()
|
||||
|
||||
def _save_current_project(self) -> None:
|
||||
if self.project is None:
|
||||
return
|
||||
try:
|
||||
save_project(self.project, self.project_path)
|
||||
except Exception as exc:
|
||||
QMessageBox.warning(None, "保存失败", f"无法保存标注项目：\n{exc}")
|
||||
|
||||
def _load_last_project(self, path: Path) -> AnnotationProject | None:
|
||||
if not path.exists():
|
||||
return None
|
||||
try:
|
||||
return load_project(path)
|
||||
except Exception as exc:
|
||||
QMessageBox.warning(None, "加载失败", f"无法加载上次标注项目，将从空项目继续：\n{exc}")
|
||||
return None
|
||||
|
||||
|
||||
def _enable_dpi_awareness() -> None:
|
||||
try:
|
||||
ctypes.windll.user32.SetProcessDpiAwarenessContext(
|
||||
DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
||||
def main() -> int:
|
||||
_enable_dpi_awareness()
|
||||
QApplication.setHighDpiScaleFactorRoundingPolicy(
|
||||
Qt.HighDpiScaleFactorRoundingPolicy.PassThrough
|
||||
)
|
||||
app = DesktopMarkApp()
|
||||
return app.run()
|
||||
79
desktop_mark/python/src/desktop_mark/floating_button.py
Normal file
79
desktop_mark/python/src/desktop_mark/floating_button.py
Normal file
@ -0,0 +1,79 @@
|
||||
"""Small always-on-top launcher button."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from PySide6.QtCore import QPoint, Qt, Signal
|
||||
from PySide6.QtGui import QColor, QFont, QMouseEvent, QPainter, QPaintEvent, QPen
|
||||
from PySide6.QtWidgets import QWidget
|
||||
|
||||
|
||||
class FloatingButton(QWidget):
|
||||
clicked = Signal()
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self.setFixedSize(80, 80)
|
||||
self.setWindowTitle("Desktop Mark")
|
||||
self.setWindowFlags(
|
||||
Qt.WindowType.FramelessWindowHint
|
||||
| Qt.WindowType.WindowStaysOnTopHint
|
||||
| Qt.WindowType.Tool
|
||||
)
|
||||
self.setAttribute(Qt.WidgetAttribute.WA_TranslucentBackground, True)
|
||||
self.setMouseTracking(True)
|
||||
self._drag_start_global: QPoint | None = None
|
||||
self._drag_start_pos: QPoint | None = None
|
||||
self._dragging = False
|
||||
self.move(32, 160)
|
||||
|
||||
def paintEvent(self, event: QPaintEvent) -> None: # noqa: N802
|
||||
painter = QPainter(self)
|
||||
painter.setRenderHint(QPainter.RenderHint.Antialiasing, True)
|
||||
painter.setPen(QPen(QColor(255, 255, 255, 80), 1))
|
||||
painter.setBrush(QColor(20, 24, 32, 210))
|
||||
painter.drawRoundedRect(self.rect().adjusted(1, 1, -1, -1), 18, 18)
|
||||
|
||||
painter.setPen(QColor(255, 255, 255))
|
||||
title_font = QFont(self.font())
|
||||
title_font.setBold(True)
|
||||
title_font.setPointSize(18)
|
||||
painter.setFont(title_font)
|
||||
painter.drawText(self.rect().adjusted(0, 12, 0, -24), Qt.AlignmentFlag.AlignCenter, "RPA")
|
||||
|
||||
sub_font = QFont(self.font())
|
||||
sub_font.setPointSize(8)
|
||||
painter.setFont(sub_font)
|
||||
painter.setPen(QColor(190, 205, 255))
|
||||
painter.drawText(self.rect().adjusted(0, 50, 0, -8), Qt.AlignmentFlag.AlignCenter, "Ctrl+1")
|
||||
super().paintEvent(event)
|
||||
|
||||
def mousePressEvent(self, event: QMouseEvent) -> None: # noqa: N802
|
||||
if event.button() == Qt.MouseButton.LeftButton:
|
||||
self._drag_start_global = event.globalPosition().toPoint()
|
||||
self._drag_start_pos = self.pos()
|
||||
self._dragging = False
|
||||
event.accept()
|
||||
return
|
||||
super().mousePressEvent(event)
|
||||
|
||||
def mouseMoveEvent(self, event: QMouseEvent) -> None: # noqa: N802
|
||||
if self._drag_start_global is not None and self._drag_start_pos is not None:
|
||||
delta = event.globalPosition().toPoint() - self._drag_start_global
|
||||
if delta.manhattanLength() > 4:
|
||||
self._dragging = True
|
||||
self.move(self._drag_start_pos + delta)
|
||||
event.accept()
|
||||
return
|
||||
super().mouseMoveEvent(event)
|
||||
|
||||
def mouseReleaseEvent(self, event: QMouseEvent) -> None: # noqa: N802
|
||||
if event.button() == Qt.MouseButton.LeftButton and self._drag_start_global is not None:
|
||||
was_dragging = self._dragging
|
||||
self._drag_start_global = None
|
||||
self._drag_start_pos = None
|
||||
self._dragging = False
|
||||
if not was_dragging:
|
||||
self.clicked.emit()
|
||||
event.accept()
|
||||
return
|
||||
super().mouseReleaseEvent(event)
|
||||
72
desktop_mark/python/src/desktop_mark/hotkeys.py
Normal file
72
desktop_mark/python/src/desktop_mark/hotkeys.py
Normal file
@ -0,0 +1,72 @@
|
||||
"""Global hotkey bridge backed by Win32 RegisterHotKey."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from ctypes import wintypes
|
||||
import win32con
|
||||
import win32gui
|
||||
from PySide6.QtCore import Signal
|
||||
from PySide6.QtWidgets import QMessageBox, QWidget
|
||||
|
||||
|
||||
HOTKEY_PICK_WINDOW = 1
|
||||
HOTKEY_START_ANNOTATION = 2
|
||||
|
||||
|
||||
class HotkeyWindow(QWidget):
|
||||
start_window_picking = Signal()
|
||||
start_annotation = Signal()
|
||||
|
||||
def __init__(self) -> None:
|
||||
super().__init__()
|
||||
self._registered_ids: set[int] = set()
|
||||
self.setWindowTitle("Desktop Mark Hotkeys")
|
||||
self.hide()
|
||||
|
||||
def register(self) -> None:
|
||||
hwnd = int(self.winId())
|
||||
self._register_one(hwnd, HOTKEY_PICK_WINDOW, win32con.MOD_CONTROL, ord("1"), "Ctrl+1")
|
||||
self._register_one(hwnd, HOTKEY_START_ANNOTATION, win32con.MOD_CONTROL, ord("R"), "Ctrl+R")
|
||||
|
||||
def unregister(self) -> None:
|
||||
hwnd = int(self.winId())
|
||||
for hotkey_id in tuple(self._registered_ids):
|
||||
try:
|
||||
win32gui.UnregisterHotKey(hwnd, hotkey_id)
|
||||
except Exception:
|
||||
pass
|
||||
finally:
|
||||
self._registered_ids.discard(hotkey_id)
|
||||
|
||||
def nativeEvent(self, eventType, message): # noqa: N802, ANN001
|
||||
msg = self._message_from_native(message)
|
||||
if eventType == "windows_generic_MSG" and msg.message == win32con.WM_HOTKEY:
|
||||
hotkey_id = int(msg.wParam)
|
||||
if hotkey_id == HOTKEY_PICK_WINDOW:
|
||||
self.start_window_picking.emit()
|
||||
return True, 0
|
||||
if hotkey_id == HOTKEY_START_ANNOTATION:
|
||||
self.start_annotation.emit()
|
||||
return True, 0
|
||||
return super().nativeEvent(eventType, message)
|
||||
|
||||
def _message_from_native(self, message): # noqa: ANN001
|
||||
if hasattr(message, "message") and hasattr(message, "wParam"):
|
||||
return message
|
||||
return wintypes.MSG.from_address(int(message))
|
||||
|
||||
def closeEvent(self, event): # noqa: N802, ANN001
|
||||
self.unregister()
|
||||
super().closeEvent(event)
|
||||
|
||||
def _register_one(self, hwnd: int, hotkey_id: int, modifiers: int, key: int, label: str) -> None:
|
||||
try:
|
||||
win32gui.RegisterHotKey(hwnd, hotkey_id, modifiers, key)
|
||||
except Exception as exc:
|
||||
QMessageBox.critical(
|
||||
self,
|
||||
"快捷键注册失败",
|
||||
f"全局快捷键 {label} 注册失败，可能已被其他程序占用。\n\n{exc}",
|
||||
)
|
||||
return
|
||||
self._registered_ids.add(hotkey_id)
|
||||
87
desktop_mark/python/src/desktop_mark/models.py
Normal file
87
desktop_mark/python/src/desktop_mark/models.py
Normal file
@ -0,0 +1,87 @@
|
||||
"""Data models and coordinate helpers for annotation projects."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from enum import StrEnum
|
||||
from uuid import uuid4
|
||||
|
||||
from PySide6.QtCore import QRect
|
||||
|
||||
|
||||
class AnnotationType(StrEnum):
|
||||
BUTTON = "button"
|
||||
INPUT = "input"
|
||||
TEXT = "text"
|
||||
IMAGE = "image"
|
||||
OTHER = "other"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Rect:
|
||||
x: int
|
||||
y: int
|
||||
width: int
|
||||
height: int
|
||||
|
||||
def right(self) -> int:
|
||||
return self.x + self.width
|
||||
|
||||
def bottom(self) -> int:
|
||||
return self.y + self.height
|
||||
|
||||
def contains(self, x: int, y: int) -> bool:
|
||||
normalized = self.normalized()
|
||||
return normalized.x <= x < normalized.right() and normalized.y <= y < normalized.bottom()
|
||||
|
||||
def normalized(self) -> "Rect":
|
||||
x1 = min(self.x, self.right())
|
||||
y1 = min(self.y, self.bottom())
|
||||
x2 = max(self.x, self.right())
|
||||
y2 = max(self.y, self.bottom())
|
||||
return Rect(x1, y1, x2 - x1, y2 - y1)
|
||||
|
||||
def clamped_to(self, bounds: "Rect") -> "Rect":
|
||||
rect = self.normalized()
|
||||
limit = bounds.normalized()
|
||||
x1 = max(limit.x, min(rect.x, limit.right()))
|
||||
y1 = max(limit.y, min(rect.y, limit.bottom()))
|
||||
x2 = max(limit.x, min(rect.right(), limit.right()))
|
||||
y2 = max(limit.y, min(rect.bottom(), limit.bottom()))
|
||||
if x2 < x1:
|
||||
x2 = x1
|
||||
if y2 < y1:
|
||||
y2 = y1
|
||||
return Rect(x1, y1, x2 - x1, y2 - y1)
|
||||
|
||||
def to_qrect(self) -> QRect:
|
||||
rect = self.normalized()
|
||||
return QRect(rect.x, rect.y, rect.width, rect.height)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class WindowInfo:
|
||||
hwnd: int | None
|
||||
title: str
|
||||
process_name: str
|
||||
rect: Rect
|
||||
is_desktop: bool = False
|
||||
|
||||
|
||||
@dataclass
|
||||
class Annotation:
|
||||
id: str = field(default_factory=lambda: str(uuid4()))
|
||||
rect: Rect = field(default_factory=lambda: Rect(0, 0, 0, 0))
|
||||
control_name: str = ""
|
||||
description: str = ""
|
||||
type: AnnotationType = AnnotationType.OTHER
|
||||
actions: list[str] = field(default_factory=list)
|
||||
feature_preview_path: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class AnnotationProject:
|
||||
window: WindowInfo
|
||||
custom_window_name: str = ""
|
||||
window_description: str = ""
|
||||
annotations: list[Annotation] = field(default_factory=list)
|
||||
375
desktop_mark/python/src/desktop_mark/overlay.py
Normal file
375
desktop_mark/python/src/desktop_mark/overlay.py
Normal file
@ -0,0 +1,375 @@
|
||||
"""Selection and annotation overlay state machine."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from enum import StrEnum
|
||||
|
||||
from PySide6.QtCore import QPoint, QRect, QTimer, Qt, Signal
|
||||
from PySide6.QtGui import QColor, QCursor, QFont, QKeyEvent, QMouseEvent, QPainter, QPaintEvent, QPen
|
||||
from PySide6.QtWidgets import QWidget
|
||||
|
||||
from .models import Annotation, AnnotationProject, Rect, WindowInfo
|
||||
from .win32_window import WindowProvider
|
||||
|
||||
|
||||
MIN_ANNOTATION_SIZE = 6
|
||||
HANDLE_SIZE = 8
|
||||
|
||||
|
||||
class OverlayMode(StrEnum):
|
||||
IDLE = "idle"
|
||||
PICKING_WINDOW = "picking_window"
|
||||
WINDOW_LOCKED = "window_locked"
|
||||
DRAWING_ANNOTATION = "drawing_annotation"
|
||||
EDITING_ANNOTATION = "editing_annotation"
|
||||
MOVING_ANNOTATION = "moving_annotation"
|
||||
RESIZING_ANNOTATION = "resizing_annotation"
|
||||
|
||||
|
||||
class OverlayWindow(QWidget):
|
||||
project_changed = Signal(object)
|
||||
annotation_edit_requested = Signal(object, object)
|
||||
hidden_by_user = Signal()
|
||||
|
||||
def __init__(self, provider: WindowProvider) -> None:
|
||||
super().__init__()
|
||||
self.provider = provider
|
||||
self.mode = OverlayMode.IDLE
|
||||
self.hover_window: WindowInfo | None = None
|
||||
self.locked_window: WindowInfo | None = None
|
||||
self.project: AnnotationProject | None = None
|
||||
self.selected_annotation_id: str | None = None
|
||||
self._virtual_rect = self.provider.virtual_screen_rect()
|
||||
self._timer = QTimer(self)
|
||||
self._timer.setInterval(50)
|
||||
self._timer.timeout.connect(self._poll_hover_window)
|
||||
self._drag_start: QPoint | None = None
|
||||
self._temp_rect: Rect | None = None
|
||||
self._moving_annotation: Annotation | None = None
|
||||
self._moving_original: Rect | None = None
|
||||
self._resizing_annotation: Annotation | None = None
|
||||
self._resizing_original: Rect | None = None
|
||||
self._resize_handle: str | None = None
|
||||
|
||||
self.setWindowFlags(
|
||||
Qt.WindowType.FramelessWindowHint
|
||||
| Qt.WindowType.WindowStaysOnTopHint
|
||||
| Qt.WindowType.Tool
|
||||
)
|
||||
self.setAttribute(Qt.WidgetAttribute.WA_TranslucentBackground, True)
|
||||
self.setMouseTracking(True)
|
||||
self.hide()
|
||||
|
||||
def start_window_picking(self) -> None:
|
||||
self._prepare_screen_geometry()
|
||||
self.mode = OverlayMode.PICKING_WINDOW
|
||||
self.hover_window = self.provider.window_at(*self._cursor_xy())
|
||||
self._temp_rect = None
|
||||
self._drag_start = None
|
||||
self._timer.start()
|
||||
self.show()
|
||||
self.raise_()
|
||||
self.activateWindow()
|
||||
self.update()
|
||||
|
||||
def start_annotation(self) -> None:
|
||||
if self.locked_window is None:
|
||||
self.start_window_picking()
|
||||
return
|
||||
self._ensure_project(self.locked_window)
|
||||
self.mode = OverlayMode.DRAWING_ANNOTATION
|
||||
self._timer.stop()
|
||||
self.show()
|
||||
self.raise_()
|
||||
self.activateWindow()
|
||||
self.update()
|
||||
|
||||
def select_annotation(self, annotation_id: str) -> None:
|
||||
self.selected_annotation_id = annotation_id
|
||||
if self.mode not in {OverlayMode.IDLE, OverlayMode.PICKING_WINDOW}:
|
||||
self.mode = OverlayMode.EDITING_ANNOTATION
|
||||
self.update()
|
||||
|
||||
def set_project(self, project: AnnotationProject | None) -> None:
|
||||
self.project = project
|
||||
self.locked_window = project.window if project is not None else None
|
||||
self.selected_annotation_id = None
|
||||
self.update()
|
||||
|
||||
def cancel_flow(self) -> None:
|
||||
self._timer.stop()
|
||||
self.mode = OverlayMode.IDLE
|
||||
self.hover_window = None
|
||||
self.locked_window = None
|
||||
self._temp_rect = None
|
||||
self._drag_start = None
|
||||
self._moving_annotation = None
|
||||
self._resizing_annotation = None
|
||||
self._resize_handle = None
|
||||
self.hide()
|
||||
self.hidden_by_user.emit()
|
||||
self.update()
|
||||
|
||||
def paintEvent(self, event: QPaintEvent) -> None: # noqa: N802
|
||||
if self.mode == OverlayMode.IDLE:
|
||||
return
|
||||
painter = QPainter(self)
|
||||
painter.setRenderHint(QPainter.RenderHint.Antialiasing, False)
|
||||
painter.fillRect(self.rect(), QColor(0, 0, 0, 120))
|
||||
|
||||
target = self.locked_window or self.hover_window
|
||||
if target is not None:
|
||||
self._clear_target_rect(painter, target.rect)
|
||||
self._draw_window_highlight(painter, target)
|
||||
|
||||
self._draw_annotations(painter)
|
||||
if self._temp_rect is not None:
|
||||
self._draw_rect_outline(painter, self._temp_rect, QColor(51, 153, 255), 2)
|
||||
super().paintEvent(event)
|
||||
|
||||
def mousePressEvent(self, event: QMouseEvent) -> None: # noqa: N802
|
||||
if event.button() != Qt.MouseButton.LeftButton:
|
||||
super().mousePressEvent(event)
|
||||
return
|
||||
point = event.globalPosition().toPoint()
|
||||
if self.mode == OverlayMode.PICKING_WINDOW:
|
||||
self.locked_window = self.hover_window or self.provider.window_at(point.x(), point.y())
|
||||
self._timer.stop()
|
||||
if self.locked_window is not None:
|
||||
self._ensure_project(self.locked_window)
|
||||
self.mode = OverlayMode.WINDOW_LOCKED
|
||||
self.project_changed.emit(self.project)
|
||||
self.update()
|
||||
event.accept()
|
||||
return
|
||||
if self.locked_window is None:
|
||||
event.accept()
|
||||
return
|
||||
|
||||
hit_annotation, handle = self._hit_test(point.x(), point.y())
|
||||
if handle is not None and hit_annotation is not None:
|
||||
self.selected_annotation_id = hit_annotation.id
|
||||
self._resizing_annotation = hit_annotation
|
||||
self._resizing_original = hit_annotation.rect
|
||||
self._resize_handle = handle
|
||||
self._drag_start = point
|
||||
self.mode = OverlayMode.RESIZING_ANNOTATION
|
||||
self.update()
|
||||
event.accept()
|
||||
return
|
||||
if hit_annotation is not None:
|
||||
self.selected_annotation_id = hit_annotation.id
|
||||
self._moving_annotation = hit_annotation
|
||||
self._moving_original = hit_annotation.rect
|
||||
self._drag_start = point
|
||||
self.mode = OverlayMode.MOVING_ANNOTATION
|
||||
self.update()
|
||||
event.accept()
|
||||
return
|
||||
|
||||
if self.locked_window.rect.contains(point.x(), point.y()):
|
||||
self._drag_start = point
|
||||
self._temp_rect = Rect(point.x(), point.y(), 0, 0)
|
||||
self.mode = OverlayMode.DRAWING_ANNOTATION
|
||||
self.update()
|
||||
event.accept()
|
||||
|
||||
def mouseMoveEvent(self, event: QMouseEvent) -> None: # noqa: N802
|
||||
point = event.globalPosition().toPoint()
|
||||
if self.mode == OverlayMode.DRAWING_ANNOTATION and self._drag_start is not None:
|
||||
self._temp_rect = Rect(
|
||||
self._drag_start.x(),
|
||||
self._drag_start.y(),
|
||||
point.x() - self._drag_start.x(),
|
||||
point.y() - self._drag_start.y(),
|
||||
).normalized().clamped_to(self.locked_window.rect) # type: ignore[union-attr]
|
||||
self.update()
|
||||
event.accept()
|
||||
return
|
||||
if self.mode == OverlayMode.MOVING_ANNOTATION and self._moving_annotation is not None and self._moving_original is not None and self._drag_start is not None:
|
||||
dx = point.x() - self._drag_start.x()
|
||||
dy = point.y() - self._drag_start.y()
|
||||
moved = self._move_rect_clamped(
|
||||
self._moving_original,
|
||||
dx,
|
||||
dy,
|
||||
self.locked_window.rect, # type: ignore[union-attr]
|
||||
)
|
||||
self._moving_annotation.rect = moved
|
||||
self.project_changed.emit(self.project)
|
||||
self.update()
|
||||
event.accept()
|
||||
return
|
||||
if self.mode == OverlayMode.RESIZING_ANNOTATION and self._resizing_annotation is not None and self._resizing_original is not None and self._drag_start is not None:
|
||||
resized = self._resize_rect(self._resizing_original, self._resize_handle or "", point)
|
||||
self._resizing_annotation.rect = resized.clamped_to(self.locked_window.rect) # type: ignore[union-attr]
|
||||
self.project_changed.emit(self.project)
|
||||
self.update()
|
||||
event.accept()
|
||||
return
|
||||
super().mouseMoveEvent(event)
|
||||
|
||||
def mouseReleaseEvent(self, event: QMouseEvent) -> None: # noqa: N802
|
||||
if event.button() != Qt.MouseButton.LeftButton:
|
||||
super().mouseReleaseEvent(event)
|
||||
return
|
||||
if self.mode == OverlayMode.DRAWING_ANNOTATION and self._temp_rect is not None:
|
||||
rect = self._temp_rect.normalized().clamped_to(self.locked_window.rect) # type: ignore[union-attr]
|
||||
self._drag_start = None
|
||||
self._temp_rect = None
|
||||
if rect.width >= MIN_ANNOTATION_SIZE and rect.height >= MIN_ANNOTATION_SIZE:
|
||||
annotation = Annotation(rect=rect)
|
||||
self._ensure_project(self.locked_window) # type: ignore[arg-type]
|
||||
self.project.annotations.append(annotation) # type: ignore[union-attr]
|
||||
self.selected_annotation_id = annotation.id
|
||||
self.mode = OverlayMode.EDITING_ANNOTATION
|
||||
self.project_changed.emit(self.project)
|
||||
self.annotation_edit_requested.emit(annotation, rect)
|
||||
else:
|
||||
self.mode = OverlayMode.DRAWING_ANNOTATION
|
||||
self.update()
|
||||
event.accept()
|
||||
return
|
||||
if self.mode in {OverlayMode.MOVING_ANNOTATION, OverlayMode.RESIZING_ANNOTATION}:
|
||||
edited = self._moving_annotation or self._resizing_annotation
|
||||
self._moving_annotation = None
|
||||
self._moving_original = None
|
||||
self._resizing_annotation = None
|
||||
self._resizing_original = None
|
||||
self._resize_handle = None
|
||||
self._drag_start = None
|
||||
self.mode = OverlayMode.EDITING_ANNOTATION
|
||||
self.project_changed.emit(self.project)
|
||||
if edited is not None:
|
||||
self.annotation_edit_requested.emit(edited, edited.rect)
|
||||
self.update()
|
||||
event.accept()
|
||||
return
|
||||
super().mouseReleaseEvent(event)
|
||||
|
||||
def keyPressEvent(self, event: QKeyEvent) -> None: # noqa: N802
|
||||
if event.key() in {Qt.Key.Key_Escape, Qt.Key.Key_Return, Qt.Key.Key_Enter}:
|
||||
self.cancel_flow()
|
||||
event.accept()
|
||||
return
|
||||
super().keyPressEvent(event)
|
||||
|
||||
def _prepare_screen_geometry(self) -> None:
|
||||
self._virtual_rect = self.provider.virtual_screen_rect()
|
||||
self.setGeometry(self._virtual_rect.to_qrect())
|
||||
|
||||
def _poll_hover_window(self) -> None:
|
||||
if self.mode != OverlayMode.PICKING_WINDOW:
|
||||
return
|
||||
x, y = self._cursor_xy()
|
||||
self.hover_window = self.provider.window_at(x, y)
|
||||
self.update()
|
||||
|
||||
def _cursor_xy(self) -> tuple[int, int]:
|
||||
pos = QCursor.pos()
|
||||
return pos.x(), pos.y()
|
||||
|
||||
def _ensure_project(self, window: WindowInfo) -> None:
|
||||
if self.project is None or self.project.window != window:
|
||||
self.project = AnnotationProject(window=window)
|
||||
self.locked_window = window
|
||||
|
||||
def _clear_target_rect(self, painter: QPainter, rect: Rect) -> None:
|
||||
painter.save()
|
||||
painter.setCompositionMode(QPainter.CompositionMode.CompositionMode_Clear)
|
||||
painter.fillRect(self._to_local_qrect(rect), Qt.GlobalColor.transparent)
|
||||
painter.restore()
|
||||
|
||||
def _draw_window_highlight(self, painter: QPainter, window: WindowInfo) -> None:
|
||||
self._draw_rect_outline(painter, window.rect, QColor(0, 122, 255), 2)
|
||||
qrect = self._to_local_qrect(window.rect)
|
||||
label = f"{window.title} [{window.rect.x}, {window.rect.y}, {window.rect.width}, {window.rect.height}]"
|
||||
painter.setFont(QFont(self.font().family(), 9))
|
||||
metrics = painter.fontMetrics()
|
||||
label_rect = metrics.boundingRect(label).adjusted(-8, -4, 8, 4)
|
||||
label_rect.moveTopRight(qrect.topRight() + QPoint(-6, 6))
|
||||
painter.fillRect(label_rect, QColor(0, 122, 255, 220))
|
||||
painter.setPen(QColor(255, 255, 255))
|
||||
painter.drawText(label_rect.adjusted(8, 4, -8, -4), Qt.AlignmentFlag.AlignCenter, label)
|
||||
|
||||
def _draw_annotations(self, painter: QPainter) -> None:
|
||||
if self.project is None:
|
||||
return
|
||||
for index, annotation in enumerate(self.project.annotations, start=1):
|
||||
selected = annotation.id == self.selected_annotation_id
|
||||
self._draw_rect_outline(painter, annotation.rect, QColor(255, 145, 0), 2)
|
||||
qrect = self._to_local_qrect(annotation.rect)
|
||||
painter.fillRect(QRect(qrect.left(), qrect.top() - 18, 24, 18), QColor(255, 145, 0, 230))
|
||||
painter.setPen(QColor(255, 255, 255))
|
||||
painter.drawText(QRect(qrect.left(), qrect.top() - 18, 24, 18), Qt.AlignmentFlag.AlignCenter, str(index))
|
||||
if selected:
|
||||
for handle in self._handle_rects(annotation.rect).values():
|
||||
painter.fillRect(self._to_local_qrect(handle), QColor(255, 255, 255))
|
||||
painter.setPen(QPen(QColor(0, 122, 255), 1))
|
||||
painter.drawRect(self._to_local_qrect(handle))
|
||||
|
||||
def _draw_rect_outline(self, painter: QPainter, rect: Rect, color: QColor, width: int) -> None:
|
||||
painter.setPen(QPen(color, width))
|
||||
painter.setBrush(Qt.BrushStyle.NoBrush)
|
||||
painter.drawRect(self._to_local_qrect(rect).adjusted(0, 0, -1, -1))
|
||||
|
||||
def _to_local_qrect(self, rect: Rect) -> QRect:
|
||||
normalized = rect.normalized()
|
||||
return QRect(
|
||||
normalized.x - self._virtual_rect.x,
|
||||
normalized.y - self._virtual_rect.y,
|
||||
normalized.width,
|
||||
normalized.height,
|
||||
)
|
||||
|
||||
def _hit_test(self, x: int, y: int) -> tuple[Annotation | None, str | None]:
|
||||
if self.project is None:
|
||||
return None, None
|
||||
for annotation in reversed(self.project.annotations):
|
||||
for name, handle_rect in self._handle_rects(annotation.rect).items():
|
||||
if handle_rect.contains(x, y):
|
||||
return annotation, name
|
||||
if annotation.rect.contains(x, y):
|
||||
return annotation, None
|
||||
return None, None
|
||||
|
||||
def _handle_rects(self, rect: Rect) -> dict[str, Rect]:
|
||||
r = rect.normalized()
|
||||
half = HANDLE_SIZE // 2
|
||||
cx = r.x + r.width // 2
|
||||
cy = r.y + r.height // 2
|
||||
points = {
|
||||
"nw": (r.x, r.y),
|
||||
"n": (cx, r.y),
|
||||
"ne": (r.right(), r.y),
|
||||
"e": (r.right(), cy),
|
||||
"se": (r.right(), r.bottom()),
|
||||
"s": (cx, r.bottom()),
|
||||
"sw": (r.x, r.bottom()),
|
||||
"w": (r.x, cy),
|
||||
}
|
||||
return {name: Rect(px - half, py - half, HANDLE_SIZE, HANDLE_SIZE) for name, (px, py) in points.items()}
|
||||
|
||||
def _move_rect_clamped(self, original: Rect, dx: int, dy: int, bounds: Rect) -> Rect:
|
||||
rect = original.normalized()
|
||||
limit = bounds.normalized()
|
||||
max_x = max(limit.x, limit.right() - rect.width)
|
||||
max_y = max(limit.y, limit.bottom() - rect.height)
|
||||
x = max(limit.x, min(rect.x + dx, max_x))
|
||||
y = max(limit.y, min(rect.y + dy, max_y))
|
||||
return Rect(x, y, rect.width, rect.height)
|
||||
|
||||
def _resize_rect(self, original: Rect, handle: str, point: QPoint) -> Rect:
|
||||
left = original.x
|
||||
top = original.y
|
||||
right = original.right()
|
||||
bottom = original.bottom()
|
||||
if "w" in handle:
|
||||
left = min(point.x(), right - MIN_ANNOTATION_SIZE)
|
||||
if "e" in handle:
|
||||
right = max(point.x(), left + MIN_ANNOTATION_SIZE)
|
||||
if "n" in handle:
|
||||
top = min(point.y(), bottom - MIN_ANNOTATION_SIZE)
|
||||
if "s" in handle:
|
||||
bottom = max(point.y(), top + MIN_ANNOTATION_SIZE)
|
||||
return Rect(left, top, right - left, bottom - top)
|
||||
93
desktop_mark/python/src/desktop_mark/storage.py
Normal file
93
desktop_mark/python/src/desktop_mark/storage.py
Normal file
@ -0,0 +1,93 @@
|
||||
"""JSON persistence for annotation projects."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from .models import Annotation, AnnotationProject, AnnotationType, Rect, WindowInfo
|
||||
|
||||
DEFAULT_PROJECT_PATH = Path.home() / ".desktop_mark" / "last_project.json"
|
||||
|
||||
|
||||
def rect_to_dict(rect: Rect) -> dict[str, int]:
|
||||
return {"x": rect.x, "y": rect.y, "width": rect.width, "height": rect.height}
|
||||
|
||||
|
||||
def rect_from_dict(data: dict[str, Any]) -> Rect:
|
||||
return Rect(
|
||||
x=int(data["x"]),
|
||||
y=int(data["y"]),
|
||||
width=int(data["width"]),
|
||||
height=int(data["height"]),
|
||||
)
|
||||
|
||||
|
||||
def project_to_dict(project: AnnotationProject) -> dict[str, Any]:
|
||||
return {
|
||||
"window": {
|
||||
"hwnd": project.window.hwnd,
|
||||
"title": project.window.title,
|
||||
"process_name": project.window.process_name,
|
||||
"rect": rect_to_dict(project.window.rect),
|
||||
"is_desktop": project.window.is_desktop,
|
||||
},
|
||||
"custom_window_name": project.custom_window_name,
|
||||
"window_description": project.window_description,
|
||||
"annotations": [
|
||||
{
|
||||
"id": annotation.id,
|
||||
"rect": rect_to_dict(annotation.rect),
|
||||
"control_name": annotation.control_name,
|
||||
"description": annotation.description,
|
||||
"type": annotation.type.value,
|
||||
"actions": list(annotation.actions),
|
||||
"feature_preview_path": annotation.feature_preview_path,
|
||||
}
|
||||
for annotation in project.annotations
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def project_from_dict(data: dict[str, Any]) -> AnnotationProject:
|
||||
window_data = data["window"]
|
||||
window = WindowInfo(
|
||||
hwnd=window_data.get("hwnd"),
|
||||
title=str(window_data.get("title", "")),
|
||||
process_name=str(window_data.get("process_name", "")),
|
||||
rect=rect_from_dict(window_data["rect"]),
|
||||
is_desktop=bool(window_data.get("is_desktop", False)),
|
||||
)
|
||||
annotations = []
|
||||
for item in data.get("annotations", []):
|
||||
annotation_type = AnnotationType(item.get("type", AnnotationType.OTHER.value))
|
||||
annotations.append(
|
||||
Annotation(
|
||||
id=str(item["id"]),
|
||||
rect=rect_from_dict(item["rect"]),
|
||||
control_name=str(item.get("control_name", "")),
|
||||
description=str(item.get("description", "")),
|
||||
type=annotation_type,
|
||||
actions=[str(action) for action in item.get("actions", [])],
|
||||
feature_preview_path=item.get("feature_preview_path"),
|
||||
)
|
||||
)
|
||||
return AnnotationProject(
|
||||
window=window,
|
||||
custom_window_name=str(data.get("custom_window_name", "")),
|
||||
window_description=str(data.get("window_description", "")),
|
||||
annotations=annotations,
|
||||
)
|
||||
|
||||
|
||||
def save_project(project: AnnotationProject, path: Path) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(
|
||||
json.dumps(project_to_dict(project), ensure_ascii=False, indent=2),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def load_project(path: Path) -> AnnotationProject:
|
||||
return project_from_dict(json.loads(path.read_text(encoding="utf-8")))
|
||||
118
desktop_mark/python/src/desktop_mark/win32_window.py
Normal file
118
desktop_mark/python/src/desktop_mark/win32_window.py
Normal file
@ -0,0 +1,118 @@
|
||||
"""Win32 top-level window discovery."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import win32api
|
||||
import win32con
|
||||
import win32gui
|
||||
import win32process
|
||||
|
||||
from .models import Rect, WindowInfo
|
||||
|
||||
|
||||
PROCESS_QUERY_LIMITED_INFORMATION = getattr(
|
||||
win32con, "PROCESS_QUERY_LIMITED_INFORMATION", 0x1000
|
||||
)
|
||||
|
||||
|
||||
class WindowProvider:
|
||||
def __init__(self, excluded_hwnds: set[int] | None = None):
|
||||
self._excluded_hwnds: set[int] = set(excluded_hwnds or set())
|
||||
|
||||
def set_excluded_hwnds(self, hwnds: set[int]) -> None:
|
||||
self._excluded_hwnds = set(hwnds)
|
||||
|
||||
def window_at(self, x: int, y: int) -> WindowInfo:
|
||||
for hwnd in self._iter_top_level_windows():
|
||||
if not self._is_candidate_window(hwnd):
|
||||
continue
|
||||
rect = self._rect_for_hwnd(hwnd)
|
||||
if rect is not None and rect.contains(x, y):
|
||||
return self._window_info(hwnd, rect)
|
||||
return self._desktop_window()
|
||||
|
||||
def foreground_window(self) -> WindowInfo | None:
|
||||
hwnd = win32gui.GetForegroundWindow()
|
||||
if not hwnd or not self._is_candidate_window(hwnd):
|
||||
return None
|
||||
rect = self._rect_for_hwnd(hwnd)
|
||||
if rect is None:
|
||||
return None
|
||||
return self._window_info(hwnd, rect)
|
||||
|
||||
def virtual_screen_rect(self) -> Rect:
|
||||
return Rect(
|
||||
win32api.GetSystemMetrics(win32con.SM_XVIRTUALSCREEN),
|
||||
win32api.GetSystemMetrics(win32con.SM_YVIRTUALSCREEN),
|
||||
win32api.GetSystemMetrics(win32con.SM_CXVIRTUALSCREEN),
|
||||
win32api.GetSystemMetrics(win32con.SM_CYVIRTUALSCREEN),
|
||||
)
|
||||
|
||||
def _iter_top_level_windows(self):
|
||||
hwnd = win32gui.GetTopWindow(None)
|
||||
while hwnd:
|
||||
yield hwnd
|
||||
hwnd = win32gui.GetWindow(hwnd, win32con.GW_HWNDNEXT)
|
||||
|
||||
def _is_candidate_window(self, hwnd: int) -> bool:
|
||||
if hwnd in self._excluded_hwnds:
|
||||
return False
|
||||
if not win32gui.IsWindowVisible(hwnd):
|
||||
return False
|
||||
if win32gui.IsIconic(hwnd):
|
||||
return False
|
||||
ex_style = win32gui.GetWindowLong(hwnd, win32con.GWL_EXSTYLE)
|
||||
if ex_style & win32con.WS_EX_TOOLWINDOW:
|
||||
return False
|
||||
rect = self._rect_for_hwnd(hwnd)
|
||||
return rect is not None and rect.width > 0 and rect.height > 0
|
||||
|
||||
def _rect_for_hwnd(self, hwnd: int) -> Rect | None:
|
||||
try:
|
||||
left, top, right, bottom = win32gui.GetWindowRect(hwnd)
|
||||
except win32gui.error:
|
||||
return None
|
||||
width = right - left
|
||||
height = bottom - top
|
||||
if width <= 0 or height <= 0:
|
||||
return None
|
||||
return Rect(left, top, width, height)
|
||||
|
||||
def _window_info(self, hwnd: int, rect: Rect) -> WindowInfo:
|
||||
title = win32gui.GetWindowText(hwnd) or "Untitled"
|
||||
return WindowInfo(
|
||||
hwnd=hwnd,
|
||||
title=title,
|
||||
process_name=self._process_name(hwnd),
|
||||
rect=rect,
|
||||
is_desktop=False,
|
||||
)
|
||||
|
||||
def _desktop_window(self) -> WindowInfo:
|
||||
return WindowInfo(
|
||||
hwnd=None,
|
||||
title="Desktop",
|
||||
process_name="explorer.exe",
|
||||
rect=self.virtual_screen_rect(),
|
||||
is_desktop=True,
|
||||
)
|
||||
|
||||
def _process_name(self, hwnd: int) -> str:
|
||||
process_handle = None
|
||||
try:
|
||||
_, pid = win32process.GetWindowThreadProcessId(hwnd)
|
||||
process_handle = win32api.OpenProcess(
|
||||
PROCESS_QUERY_LIMITED_INFORMATION, False, pid
|
||||
)
|
||||
executable = win32process.GetModuleFileNameEx(process_handle, 0)
|
||||
return Path(executable).name
|
||||
except Exception:
|
||||
return ""
|
||||
finally:
|
||||
if process_handle is not None:
|
||||
try:
|
||||
win32api.CloseHandle(process_handle)
|
||||
except Exception:
|
||||
pass
|
||||
26
desktop_mark/python/tests/test_models.py
Normal file
26
desktop_mark/python/tests/test_models.py
Normal file
@ -0,0 +1,26 @@
|
||||
from desktop_mark.models import Rect
|
||||
|
||||
|
||||
def test_normalized_keeps_positive_drag() -> None:
|
||||
assert Rect(10, 20, 30, 40).normalized() == Rect(10, 20, 30, 40)
|
||||
|
||||
|
||||
def test_normalized_handles_negative_drag() -> None:
|
||||
assert Rect(50, 60, -20, -30).normalized() == Rect(30, 30, 20, 30)
|
||||
|
||||
|
||||
def test_clamped_to_bounds() -> None:
|
||||
rect = Rect(80, 90, 50, 40).clamped_to(Rect(100, 100, 100, 100))
|
||||
assert rect == Rect(100, 100, 30, 30)
|
||||
|
||||
|
||||
def test_clamped_negative_drag_to_bounds() -> None:
|
||||
rect = Rect(250, 250, -200, -200).clamped_to(Rect(100, 100, 100, 100))
|
||||
assert rect == Rect(100, 100, 100, 100)
|
||||
|
||||
|
||||
def test_contains_uses_normalized_half_open_bounds() -> None:
|
||||
rect = Rect(20, 20, -10, -10)
|
||||
assert rect.contains(10, 10)
|
||||
assert rect.contains(19, 19)
|
||||
assert not rect.contains(20, 20)
|
||||
24
desktop_mark/python/tests/test_storage.py
Normal file
24
desktop_mark/python/tests/test_storage.py
Normal file
@ -0,0 +1,24 @@
|
||||
from desktop_mark.models import Annotation, AnnotationProject, AnnotationType, Rect, WindowInfo
|
||||
from desktop_mark.storage import project_from_dict, project_to_dict
|
||||
|
||||
|
||||
def test_project_json_round_trip() -> None:
|
||||
project = AnnotationProject(
|
||||
window=WindowInfo(123, "记事本", "notepad.exe", Rect(1, 2, 300, 400)),
|
||||
custom_window_name="登录页",
|
||||
window_description="用户登录窗口",
|
||||
annotations=[
|
||||
Annotation(
|
||||
id="a1",
|
||||
rect=Rect(10, 20, 30, 40),
|
||||
control_name="用户名输入框",
|
||||
description="输入登录用户名",
|
||||
type=AnnotationType.INPUT,
|
||||
actions=["input_text"],
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
restored = project_from_dict(project_to_dict(project))
|
||||
|
||||
assert restored == project
|
||||
168
desktop_mark/python/uv.lock
generated
Normal file
168
desktop_mark/python/uv.lock
generated
Normal file
@ -0,0 +1,168 @@
|
||||
version = 1
|
||||
revision = 3
|
||||
requires-python = ">=3.11"
|
||||
|
||||
[[package]]
|
||||
name = "colorama"
|
||||
version = "0.4.6"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "desktop-mark"
|
||||
version = "0.1.0"
|
||||
source = { editable = "." }
|
||||
dependencies = [
|
||||
{ name = "pyside6" },
|
||||
{ name = "pywin32" },
|
||||
]
|
||||
|
||||
[package.optional-dependencies]
|
||||
dev = [
|
||||
{ name = "pytest" },
|
||||
]
|
||||
|
||||
[package.metadata]
|
||||
requires-dist = [
|
||||
{ name = "pyside6", specifier = ">=6.7,<7" },
|
||||
{ name = "pytest", marker = "extra == 'dev'", specifier = ">=8,<9" },
|
||||
{ name = "pywin32", specifier = ">=306" },
|
||||
]
|
||||
provides-extras = ["dev"]
|
||||
|
||||
[[package]]
|
||||
name = "iniconfig"
|
||||
version = "2.3.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/72/34/14ca021ce8e5dfedc35312d08ba8bf51fdd999c576889fc2c24cb97f4f10/iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730", size = 20503, upload-time = "2025-10-18T21:55:43.219Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "packaging"
|
||||
version = "26.2"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/d7/f1/e7a6dd94a8d4a5626c03e4e99c87f241ba9e350cd9e6d75123f992427270/packaging-26.2.tar.gz", hash = "sha256:ff452ff5a3e828ce110190feff1178bb1f2ea2281fa2075aadb987c2fb221661", size = 228134, upload-time = "2026-04-24T20:15:23.917Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl", hash = "sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e", size = 100195, upload-time = "2026-04-24T20:15:22.081Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pluggy"
|
||||
version = "1.6.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/f9/e2/3e91f31a7d2b083fe6ef3fa267035b518369d9511ffab804f839851d2779/pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", size = 69412, upload-time = "2025-05-15T12:30:07.975Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pygments"
|
||||
version = "2.20.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pyside6"
|
||||
version = "6.11.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "pyside6-addons" },
|
||||
{ name = "pyside6-essentials" },
|
||||
{ name = "shiboken6" },
|
||||
]
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/da/a6/27ba5947ed48918f7b74b7c43a1e280aac069e36f25adeb4c9adfac835c4/pyside6-6.11.1-cp310-abi3-macosx_13_0_universal2.whl", hash = "sha256:537682c3b7530817203e667c1f5a2f00486b37bf52c52eeab438544c7a0917f6", size = 571921, upload-time = "2026-05-13T09:47:36.402Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d8/de/af89d71410c83b10654d86ff9aff2a4f87c30163658f1cc145242e222526/pyside6-6.11.1-cp310-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:b1fc521ba2bb5109425ab8add06bddbdd524abcad06cfa012cc39a22a189feb2", size = 572102, upload-time = "2026-05-13T09:47:38.249Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/b6/0e/d583bd3f7bf5046a4497b36f3902cfb64aa29554489a5a25c18e6b4ac0ac/pyside6-6.11.1-cp310-abi3-manylinux_2_39_aarch64.whl", hash = "sha256:75f0005c3eb95c07cfb65522ec50d0815ac007a96482c21dc3cb4b4c04895d84", size = 572098, upload-time = "2026-05-13T09:47:39.44Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/57/f2/d9d8ce1373dabb37e5919f63cd18446556079631d3f2eea3ada03c29f6b8/pyside6-6.11.1-cp310-abi3-win_amd64.whl", hash = "sha256:0968877ab1fb4ef3587a284da6fe05e8647ada56a6a3750b6395188e01f4aba6", size = 578377, upload-time = "2026-05-13T09:47:40.76Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/96/02/a6057d8bd2bdb1940820fff2d627fdf4013148c9c57adf69fa40d3452ac3/pyside6-6.11.1-cp310-abi3-win_arm64.whl", hash = "sha256:acee467cb5f256cc47ebb9d815a054c1d8416da380c191b247a76d164aa3f805", size = 561765, upload-time = "2026-05-13T09:47:41.9Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pyside6-addons"
|
||||
version = "6.11.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "pyside6-essentials" },
|
||||
{ name = "shiboken6" },
|
||||
]
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/3f/6b/8bc94aff48b63f788f2d84e5467c12362d68906ba742c0942f46cb04c879/pyside6_addons-6.11.1-cp310-abi3-macosx_13_0_universal2.whl", hash = "sha256:54733c77f789bef5f03c6aff4ad3bec8b2eff021f0cfcbc53d5e6c250ded24f9", size = 331714589, upload-time = "2026-05-13T09:39:12.36Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/dd/62/fb1428a523b2a4541e232aab50d9e789e6b4526f37fd9593452a7ea5b6b3/pyside6_addons-6.11.1-cp310-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:8e6c65fbd73a512d6f72cda8d8277444a85a34dc99dd1dae9c21d35b8671bb1f", size = 175063224, upload-time = "2026-05-13T09:39:34.185Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ee/9b/2ccd52f66db55c06de65d0501170a1935d04d64d0a230c0d892284a02ce3/pyside6_addons-6.11.1-cp310-abi3-manylinux_2_39_aarch64.whl", hash = "sha256:bf1c6c4e954e5eba3d2a7c661ad4b9689e8f09c7f4a16bdf29713371d11af993", size = 170553429, upload-time = "2026-05-13T09:39:54.424Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/9a/bd/8adc4d350b3b363f3dfc8fccdcf5bfed25f7e36c2fff30c64e106f4f1572/pyside6_addons-6.11.1-cp310-abi3-win_amd64.whl", hash = "sha256:0d13c4dfd671b050a48e4f8d8ddc724b7248f9c0437e7fc47fdf316278572923", size = 168816308, upload-time = "2026-05-13T09:40:13.541Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/65/b7/9a840d97f0f0f04e372a87e205dd30ee285b4e3b021b188459a917c9dc76/pyside6_addons-6.11.1-cp310-abi3-win_arm64.whl", hash = "sha256:3494f480dee92f415be2f2d989c0b3f4755ac332b28045cbf4ba0f5c5a22ba37", size = 35759347, upload-time = "2026-05-13T09:40:21.199Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pyside6-essentials"
|
||||
version = "6.11.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "shiboken6" },
|
||||
]
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/b3/da/10d9197e7370eb4fed8df5fc547b7548dec88e5c5949e2d450db4ae96feb/pyside6_essentials-6.11.1-cp310-abi3-macosx_13_0_universal2.whl", hash = "sha256:228de53c2bc26b07e5021fbe3614fc44ca08e4dab9999af08c2b389d2c239957", size = 110352945, upload-time = "2026-05-13T09:43:08.006Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/5c/49/0e1237c4400bec7e335d2c4eeb49bc40d9fd88a9ac44ca9083ce1abdc308/pyside6_essentials-6.11.1-cp310-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:e3ef7027b41e4e55fadb56e3b3257dc8ee92154b639fe67fc4c8e05e9d976c60", size = 79908535, upload-time = "2026-05-13T09:43:24.836Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/4c/c5/da4c5f23c6540ac5211a1f60177c8dee84b1bf40f2719479587ab8c60731/pyside6_essentials-6.11.1-cp310-abi3-manylinux_2_39_aarch64.whl", hash = "sha256:a039b6da68a3a4b9d243217b2b98d475eed3f617159ef6be925badab53c11b0d", size = 78960051, upload-time = "2026-05-13T09:43:35.423Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/64/0e/b663ecc96ca57b5c91b83b6615d6b174380b0faf30338125c26e053d6aa7/pyside6_essentials-6.11.1-cp310-abi3-win_amd64.whl", hash = "sha256:63311bd48e32c584599ab04b9ef7c324082374cd2c9fa533f978fb893bb47e40", size = 77549267, upload-time = "2026-05-13T09:43:44.92Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/f1/12/eb6723faf5cb7fa581145da1c15f40d641b96e080f0491af2f1859fdeedb/pyside6_essentials-6.11.1-cp310-abi3-win_arm64.whl", hash = "sha256:11253ea52aabecefe9febddbbe78b43a824129e3af1cec98431028fba7fa954f", size = 57964512, upload-time = "2026-05-13T09:43:52.968Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pytest"
|
||||
version = "8.4.2"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "colorama", marker = "sys_platform == 'win32'" },
|
||||
{ name = "iniconfig" },
|
||||
{ name = "packaging" },
|
||||
{ name = "pluggy" },
|
||||
{ name = "pygments" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/a3/5c/00a0e072241553e1a7496d638deababa67c5058571567b92a7eaa258397c/pytest-8.4.2.tar.gz", hash = "sha256:86c0d0b93306b961d58d62a4db4879f27fe25513d4b969df351abdddb3c30e01", size = 1519618, upload-time = "2025-09-04T14:34:22.711Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/a8/a4/20da314d277121d6534b3a980b29035dcd51e6744bd79075a6ce8fa4eb8d/pytest-8.4.2-py3-none-any.whl", hash = "sha256:872f880de3fc3a5bdc88a11b39c9710c3497a547cfa9320bc3c5e62fbf272e79", size = 365750, upload-time = "2025-09-04T14:34:20.226Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pywin32"
|
||||
version = "312"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/1f/f5/10a6e845a00fc5e7afd0a988b744f403d4d57162a28d160a093c4d9322f0/pywin32-312-cp311-cp311-win32.whl", hash = "sha256:17948aeadbdb091f0ced6ef0841620794e68327b94ee415571c1203594b7215c", size = 6362659, upload-time = "2026-06-04T07:49:21.349Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/35/c4/dcd2d62b5944b6d5db53413a5899016ccd57ffcb7278f3f81655d25d2027/pywin32-312-cp311-cp311-win_amd64.whl", hash = "sha256:d11417d84412f859b722fad0841b3614459ed0047f7542d8362e77884f6b6e8a", size = 6928825, upload-time = "2026-06-04T07:49:23.934Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/b7/56/3cbb433fe4501cdba2eb9040f56a4e1a8243faa4186b25295564d1a7a79d/pywin32-312-cp311-cp311-win_arm64.whl", hash = "sha256:b2200a054ca6d6625c4842fc56a4976a4b47f96b73dbe5538c3f813a80359f47", size = 6721875, upload-time = "2026-06-04T07:49:26.416Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/83/ff/32aa7d2ed0ab12b323aaa64f9b75e6ad4f8fd09f9ccfc28c79414d46838d/pywin32-312-cp312-cp312-win32.whl", hash = "sha256:dab4f65ac9c4e48400a2a0530c46c3c579cd5905ecd11b80692373915269208b", size = 6371877, upload-time = "2026-06-04T07:49:28.836Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/03/d9/77040d3b43df3f3be32ea289433d660d2727f5ba327bc73be835127d9d60/pywin32-312-cp312-cp312-win_amd64.whl", hash = "sha256:b457f6d628a47e8a7346ce22acb7e1a46a4a78b52e1d17e1af56871bd19a93bc", size = 6914841, upload-time = "2026-06-04T07:49:31.85Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e3/cc/7b1ec671775756020a0ee7f4feeaf3c568f0ab86bd3900088cf986937a92/pywin32-312-cp312-cp312-win_arm64.whl", hash = "sha256:6017c58e12f6809fbb0555b75df144c2922a9ffd18e4b9b5afa863b6c1a9d950", size = 6727901, upload-time = "2026-06-04T07:49:34.244Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/2d/41/12fbfd7f36ed2146d8bc9de96c2741296bf0d490b98508496cff322e274c/pywin32-312-cp313-cp313-win32.whl", hash = "sha256:7a27df850933d16a8eabfbaeb73d52b273e2da667f80d70b01a89d1f6828d02c", size = 6370184, upload-time = "2026-06-04T07:49:36.253Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ba/db/36a78e3403099d31d9746d13fdcde5accc43c1155f375a34d15983a479a7/pywin32-312-cp313-cp313-win_amd64.whl", hash = "sha256:c53e878d15a1c44788082bfe712a905433473aa38f86375b7cf8b45e3acbaaf9", size = 6914298, upload-time = "2026-06-04T07:49:38.876Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/84/37/c1697194092b76de9ed47ca124323f02c57ffc8a45c06f88a3d5acaf01eb/pywin32-312-cp313-cp313-win_arm64.whl", hash = "sha256:59aba5d5940842075343a5ddc6b11f1cdf0d1567fe745290359dfbcc7c2eb831", size = 6727640, upload-time = "2026-06-04T07:49:41.083Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/fc/2b/1f3cded5822fd49c02f40544cbb5f58c7cfd6b1694869fd476cb6170ee97/pywin32-312-cp314-cp314-win32.whl", hash = "sha256:a77a90fbb6881238d2ca9c6fd797b25817f3768fe78d214a90137ff055a75f5b", size = 6468928, upload-time = "2026-06-04T07:49:43.188Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/21/82/3bf86d2e2808902013132e1ce905a7da0da53790f3836c64bf44d55e24f3/pywin32-312-cp314-cp314-win_amd64.whl", hash = "sha256:a4dd3a848290ef724347b19f301045831d8e802fa4464f491b98b1e0a081432e", size = 7024157, upload-time = "2026-06-04T07:49:45.34Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/a4/0e/73f6d6800b4f27655abd9e9f6aaeaefcddb2b946e4674efa2bab184a7f7b/pywin32-312-cp314-cp314-win_arm64.whl", hash = "sha256:9fce94568364e0155e6dfb781ac5d95903be8baf28670632beab1b523f300daa", size = 6839598, upload-time = "2026-06-04T07:49:47.613Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/eb/61/caa39686032d2ebdd04ff0ab5cbe163126c0066d98e00c9018646e42393b/pywin32-312-cp315-cp315-win32.whl", hash = "sha256:5c1fbe4a937a73ae9297384a3da38518cbc694c68ad8a809b2e19acd350f03ed", size = 6471159, upload-time = "2026-06-04T07:49:50.035Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/0f/cd/7e1de64a4a6f69c04214169657ccab0d93a670ea50e35eb8f489d7378249/pywin32-312-cp315-cp315-win_amd64.whl", hash = "sha256:c2f03a0f73f804a13c2735b99392b0cd426bb4f2c4d0178e5ac966a0f21618d5", size = 7025293, upload-time = "2026-06-04T07:49:54.857Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/23/ed/4532e9388e65fa16b46776ef47ad631a64eda1631884488af707666350ed/pywin32-312-cp315-cp315-win_arm64.whl", hash = "sha256:a8597d28f267b39074aef51fa593530082b39cbe5a074226096857b1fed2dfb9", size = 6840337, upload-time = "2026-06-04T07:49:57.531Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "shiboken6"
|
||||
version = "6.11.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/17/f3/f2b63df0251e7cd3172ea28e32ede52739de9566bcefcd0178681538ac81/shiboken6-6.11.1-cp310-abi3-macosx_13_0_universal2.whl", hash = "sha256:1a16867f103ef1c662a5f09dfed03273a9f81688b174555162c58e83650a3f02", size = 476874, upload-time = "2026-05-13T09:47:01.091Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/c7/9b/e0355d8897b5c150770f1d95718aad17d432fcc9c035c04f3f58427d4693/shiboken6-6.11.1-cp310-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:9a8bccfafc8805254cabcfa1edfaf55cd52889f4998c91ad0d9a4433fb1bcdbe", size = 272222, upload-time = "2026-05-13T09:47:02.653Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/57/d5/dd4f1defed400be03340f2ede34b61f846776650b4e7ed9ebaf4c71979a2/shiboken6-6.11.1-cp310-abi3-manylinux_2_39_aarch64.whl", hash = "sha256:1bd2f4314414df2d122d9f646e03b731bc6d6b5f77a5f53f99a4fe4e97d84e6f", size = 270350, upload-time = "2026-05-13T09:47:04.02Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/52/b5/3f6fb2ee65b534193fb4ef713dd619dc31dadff5d12c16979a7699ad58be/shiboken6-6.11.1-cp310-abi3-win_amd64.whl", hash = "sha256:c2c6863aa80ec18c0f82cea3417837b279cdc60024ac17123461dc9042577df7", size = 1223647, upload-time = "2026-05-13T09:47:05.924Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/98/d1/f15ca0e1666faae02c945f48e745ea35f8fcd8243b176109b4e2c4251f47/shiboken6-6.11.1-cp310-abi3-win_arm64.whl", hash = "sha256:7c8d9af17db4495d4fa5b1c393f218311c4855546b9dfa6a0bd21bcd66b55e9d", size = 1784170, upload-time = "2026-05-13T09:47:07.617Z" },
|
||||
]
|
||||
12
desktop_mark/tauri/index.html
Normal file
12
desktop_mark/tauri/index.html
Normal file
@ -0,0 +1,12 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Desktop Mark</title>
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
<script type="module" src="/src/main.tsx"></script>
|
||||
</body>
|
||||
</html>
|
||||
1197
desktop_mark/tauri/package-lock.json
generated
Normal file
1197
desktop_mark/tauri/package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
1
desktop_mark/tauri/package.json
Normal file
1
desktop_mark/tauri/package.json
Normal file
@ -0,0 +1 @@
|
||||
{"scripts":{"dev":"tauri dev","build":"tauri build","typecheck":"tsc --noEmit"},"dependencies":{"@tauri-apps/api":"^2.0.0","@vitejs/plugin-react":"latest","vite":"latest","typescript":"latest","react":"latest","react-dom":"latest"},"devDependencies":{"@tauri-apps/cli":"^2.0.0","@types/react":"latest","@types/react-dom":"latest"}}
|
||||
4568
desktop_mark/tauri/src-tauri/Cargo.lock
generated
Normal file
4568
desktop_mark/tauri/src-tauri/Cargo.lock
generated
Normal file
File diff suppressed because it is too large
Load Diff
17
desktop_mark/tauri/src-tauri/Cargo.toml
Normal file
17
desktop_mark/tauri/src-tauri/Cargo.toml
Normal file
@ -0,0 +1,17 @@
|
||||
[package]
|
||||
name = "desktop-mark-tauri"
|
||||
version = "0.1.0"
|
||||
description = "Rust Tauri rewrite of Desktop Mark"
|
||||
authors = ["Desktop Mark"]
|
||||
edition = "2021"
|
||||
|
||||
[build-dependencies]
|
||||
tauri-build = { version = "2", features = [] }
|
||||
|
||||
[dependencies]
|
||||
tauri = { version = "2", features = [] }
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
dirs = "6"
|
||||
tauri-plugin-global-shortcut = "2"
|
||||
windows = { version = "0.58", features = ["Win32_Foundation", "Win32_UI_WindowsAndMessaging", "Win32_UI_HiDpi", "Win32_System_SystemInformation", "Win32_System_Threading", "Win32_System_ProcessStatus", "Win32_System_SystemServices"] }
|
||||
3
desktop_mark/tauri/src-tauri/build.rs
Normal file
3
desktop_mark/tauri/src-tauri/build.rs
Normal file
@ -0,0 +1,3 @@
|
||||
fn main() {
|
||||
tauri_build::build()
|
||||
}
|
||||
16
desktop_mark/tauri/src-tauri/capabilities/default.json
Normal file
16
desktop_mark/tauri/src-tauri/capabilities/default.json
Normal file
@ -0,0 +1,16 @@
|
||||
{
|
||||
"$schema": "../gen/schemas/desktop-schema.json",
|
||||
"identifier": "default",
|
||||
"description": "Desktop Mark windows and IPC permissions",
|
||||
"windows": ["floating", "overlay", "panel", "annotation_form"],
|
||||
"permissions": [
|
||||
"core:event:default",
|
||||
"core:window:default",
|
||||
"core:window:allow-show",
|
||||
"core:window:allow-hide",
|
||||
"core:window:allow-set-focus",
|
||||
"core:window:allow-set-position",
|
||||
"core:window:allow-set-size",
|
||||
"core:window:allow-start-dragging"
|
||||
]
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
@ -0,0 +1 @@
|
||||
{"default":{"identifier":"default","description":"Desktop Mark windows and IPC permissions","local":true,"windows":["floating","overlay","panel","annotation_form"],"permissions":["core:event:default","core:window:default","core:window:allow-show","core:window:allow-hide","core:window:allow-set-focus","core:window:allow-set-position","core:window:allow-set-size","core:window:allow-start-dragging"]}}
|
||||
2358
desktop_mark/tauri/src-tauri/gen/schemas/desktop-schema.json
Normal file
2358
desktop_mark/tauri/src-tauri/gen/schemas/desktop-schema.json
Normal file
File diff suppressed because it is too large
Load Diff
2358
desktop_mark/tauri/src-tauri/gen/schemas/windows-schema.json
Normal file
2358
desktop_mark/tauri/src-tauri/gen/schemas/windows-schema.json
Normal file
File diff suppressed because it is too large
Load Diff
BIN
desktop_mark/tauri/src-tauri/icons/icon.ico
Normal file
BIN
desktop_mark/tauri/src-tauri/icons/icon.ico
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 78 B |
30
desktop_mark/tauri/src-tauri/src/commands.rs
Normal file
30
desktop_mark/tauri/src-tauri/src/commands.rs
Normal file
@ -0,0 +1,30 @@
|
||||
use tauri::AppHandle;
|
||||
|
||||
use crate::models::{AnnotationProject, Rect, WindowInfo};
|
||||
use crate::{storage, win32_window, window_manager};
|
||||
|
||||
#[tauri::command]
|
||||
pub fn save_project(project: AnnotationProject) -> Result<(), String> { storage::save_project(&project) }
|
||||
|
||||
#[tauri::command]
|
||||
pub fn load_project() -> Result<Option<AnnotationProject>, String> { storage::load_project() }
|
||||
|
||||
#[tauri::command]
|
||||
pub fn get_virtual_screen_rect() -> Rect { win32_window::virtual_screen_rect() }
|
||||
|
||||
#[tauri::command]
|
||||
pub fn window_at(x: i32, y: i32, excluded_hwnds: Vec<isize>) -> WindowInfo { win32_window::window_at(x, y, &excluded_hwnds) }
|
||||
|
||||
#[tauri::command]
|
||||
pub fn foreground_window(excluded_hwnds: Vec<isize>) -> Option<WindowInfo> { win32_window::foreground_window(&excluded_hwnds) }
|
||||
|
||||
#[tauri::command]
|
||||
pub fn cursor_position() -> (i32, i32) { win32_window::cursor_position() }
|
||||
|
||||
#[tauri::command]
|
||||
pub fn tool_window_hwnds(app: AppHandle) -> Result<Vec<isize>, String> { Ok(window_manager::tool_window_hwnds(&app)) }
|
||||
|
||||
#[tauri::command]
|
||||
pub fn exit_app(app: AppHandle) {
|
||||
app.exit(0);
|
||||
}
|
||||
24
desktop_mark/tauri/src-tauri/src/hotkeys.rs
Normal file
24
desktop_mark/tauri/src-tauri/src/hotkeys.rs
Normal file
@ -0,0 +1,24 @@
|
||||
use tauri::{AppHandle, Emitter};
|
||||
use tauri_plugin_global_shortcut::{Code, GlobalShortcutExt, Modifiers, Shortcut};
|
||||
|
||||
pub fn register(app: &AppHandle) -> Vec<String> {
|
||||
let mut errors = Vec::new();
|
||||
let ctrl_1 = Shortcut::new(Some(Modifiers::CONTROL), Code::Digit1);
|
||||
let ctrl_r = Shortcut::new(Some(Modifiers::CONTROL), Code::KeyR);
|
||||
|
||||
if app.global_shortcut().register(ctrl_1).is_err() {
|
||||
errors.push("全局快捷键 Ctrl+1 注册失败，可能已被其他程序占用。".to_string());
|
||||
}
|
||||
if app.global_shortcut().register(ctrl_r).is_err() {
|
||||
errors.push("全局快捷键 Ctrl+R 注册失败，可能已被其他程序占用。".to_string());
|
||||
}
|
||||
errors
|
||||
}
|
||||
|
||||
pub fn unregister(app: &AppHandle) {
|
||||
let _ = app.global_shortcut().unregister_all();
|
||||
}
|
||||
|
||||
pub fn emit_registration_errors(app: &AppHandle, errors: Vec<String>) {
|
||||
for error in errors { let _ = app.emit("hotkey:registration-error", error); }
|
||||
}
|
||||
54
desktop_mark/tauri/src-tauri/src/main.rs
Normal file
54
desktop_mark/tauri/src-tauri/src/main.rs
Normal file
@ -0,0 +1,54 @@
|
||||
mod commands;
|
||||
mod hotkeys;
|
||||
mod models;
|
||||
mod rect;
|
||||
mod storage;
|
||||
mod win32_window;
|
||||
mod window_manager;
|
||||
|
||||
use tauri::{Emitter, Manager};
|
||||
use tauri_plugin_global_shortcut::{Code, Modifiers, Shortcut, ShortcutState};
|
||||
|
||||
fn main() {
|
||||
win32_window::set_process_dpi_awareness();
|
||||
|
||||
tauri::Builder::default()
|
||||
.plugin(
|
||||
tauri_plugin_global_shortcut::Builder::new()
|
||||
.with_handler(|app, shortcut, event| {
|
||||
if event.state() != ShortcutState::Pressed { return; }
|
||||
let ctrl_1 = Shortcut::new(Some(Modifiers::CONTROL), Code::Digit1);
|
||||
let ctrl_r = Shortcut::new(Some(Modifiers::CONTROL), Code::KeyR);
|
||||
if shortcut == &ctrl_1 {
|
||||
let _ = app.emit("hotkey:start-window-picking", ());
|
||||
} else if shortcut == &ctrl_r {
|
||||
let _ = app.emit("hotkey:start-annotation", ());
|
||||
}
|
||||
})
|
||||
.build(),
|
||||
)
|
||||
.invoke_handler(tauri::generate_handler![
|
||||
commands::save_project,
|
||||
commands::load_project,
|
||||
commands::get_virtual_screen_rect,
|
||||
commands::window_at,
|
||||
commands::foreground_window,
|
||||
commands::cursor_position,
|
||||
commands::tool_window_hwnds,
|
||||
commands::exit_app,
|
||||
])
|
||||
.setup(|app| {
|
||||
let handle = app.handle().clone();
|
||||
window_manager::create_all_windows(&handle)?;
|
||||
let errors = hotkeys::register(&handle);
|
||||
hotkeys::emit_registration_errors(&handle, errors);
|
||||
Ok(())
|
||||
})
|
||||
.on_window_event(|window, event| {
|
||||
if matches!(event, tauri::WindowEvent::Destroyed) {
|
||||
if window.label() == "floating" { hotkeys::unregister(&window.app_handle()); }
|
||||
}
|
||||
})
|
||||
.run(tauri::generate_context!())
|
||||
.expect("error while running tauri application");
|
||||
}
|
||||
48
desktop_mark/tauri/src-tauri/src/models.rs
Normal file
48
desktop_mark/tauri/src-tauri/src/models.rs
Normal file
@ -0,0 +1,48 @@
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
|
||||
pub struct Rect {
|
||||
pub x: i32,
|
||||
pub y: i32,
|
||||
pub width: i32,
|
||||
pub height: i32,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
pub struct WindowInfo {
|
||||
pub hwnd: Option<isize>,
|
||||
pub title: String,
|
||||
pub process_name: String,
|
||||
pub rect: Rect,
|
||||
pub is_desktop: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum AnnotationType {
|
||||
Button,
|
||||
Input,
|
||||
Text,
|
||||
Image,
|
||||
Other,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
pub struct Annotation {
|
||||
pub id: String,
|
||||
pub rect: Rect,
|
||||
pub control_name: String,
|
||||
pub description: String,
|
||||
#[serde(rename = "type")]
|
||||
pub annotation_type: AnnotationType,
|
||||
pub actions: Vec<String>,
|
||||
pub feature_preview_path: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
pub struct AnnotationProject {
|
||||
pub window: WindowInfo,
|
||||
pub custom_window_name: String,
|
||||
pub window_description: String,
|
||||
pub annotations: Vec<Annotation>,
|
||||
}
|
||||
86
desktop_mark/tauri/src-tauri/src/rect.rs
Normal file
86
desktop_mark/tauri/src-tauri/src/rect.rs
Normal file
@ -0,0 +1,86 @@
|
||||
use crate::models::Rect;
|
||||
|
||||
impl Rect {
|
||||
pub fn right(&self) -> i32 { self.x + self.width }
|
||||
pub fn bottom(&self) -> i32 { self.y + self.height }
|
||||
|
||||
pub fn contains(&self, x: i32, y: i32) -> bool {
|
||||
let rect = self.normalized();
|
||||
rect.x <= x && x < rect.right() && rect.y <= y && y < rect.bottom()
|
||||
}
|
||||
|
||||
pub fn normalized(&self) -> Rect {
|
||||
let x1 = self.x.min(self.right());
|
||||
let y1 = self.y.min(self.bottom());
|
||||
let x2 = self.x.max(self.right());
|
||||
let y2 = self.y.max(self.bottom());
|
||||
Rect { x: x1, y: y1, width: x2 - x1, height: y2 - y1 }
|
||||
}
|
||||
|
||||
pub fn clamped_to(&self, bounds: Rect) -> Rect {
|
||||
let rect = self.normalized();
|
||||
let limit = bounds.normalized();
|
||||
let x1 = limit.x.max(rect.x.min(limit.right()));
|
||||
let y1 = limit.y.max(rect.y.min(limit.bottom()));
|
||||
let mut x2 = limit.x.max(rect.right().min(limit.right()));
|
||||
let mut y2 = limit.y.max(rect.bottom().min(limit.bottom()));
|
||||
if x2 < x1 { x2 = x1; }
|
||||
if y2 < y1 { y2 = y1; }
|
||||
Rect { x: x1, y: y1, width: x2 - x1, height: y2 - y1 }
|
||||
}
|
||||
|
||||
pub fn move_clamped(&self, dx: i32, dy: i32, bounds: Rect) -> Rect {
|
||||
let rect = self.normalized();
|
||||
let limit = bounds.normalized();
|
||||
let max_x = limit.x.max(limit.right() - rect.width);
|
||||
let max_y = limit.y.max(limit.bottom() - rect.height);
|
||||
let x = limit.x.max((rect.x + dx).min(max_x));
|
||||
let y = limit.y.max((rect.y + dy).min(max_y));
|
||||
Rect { x, y, width: rect.width, height: rect.height }
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn positive_drag_normalization() {
|
||||
assert_eq!(Rect { x: 10, y: 20, width: 30, height: 40 }.normalized(), Rect { x: 10, y: 20, width: 30, height: 40 });
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn negative_drag_normalization() {
|
||||
assert_eq!(Rect { x: 40, y: 60, width: -30, height: -40 }.normalized(), Rect { x: 10, y: 20, width: 30, height: 40 });
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn clamps_shrinking_corners() {
|
||||
let bounds = Rect { x: 100, y: 100, width: 100, height: 100 };
|
||||
let input = Rect { x: 80, y: 90, width: 50, height: 40 };
|
||||
assert_eq!(input.clamped_to(bounds), Rect { x: 100, y: 100, width: 30, height: 30 });
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn clamps_negative_drag_to_bounds() {
|
||||
let bounds = Rect { x: 100, y: 100, width: 100, height: 100 };
|
||||
let input = Rect { x: 250, y: 250, width: -200, height: -200 };
|
||||
assert_eq!(input.clamped_to(bounds), Rect { x: 100, y: 100, width: 100, height: 100 });
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn contains_is_half_open_after_normalization() {
|
||||
let rect = Rect { x: 20, y: 20, width: -10, height: -10 };
|
||||
assert!(rect.contains(10, 10));
|
||||
assert!(rect.contains(19, 19));
|
||||
assert!(!rect.contains(20, 20));
|
||||
assert!(!rect.contains(9, 10));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn move_clamped_preserves_size_at_edges() {
|
||||
let bounds = Rect { x: 100, y: 100, width: 100, height: 100 };
|
||||
let input = Rect { x: 150, y: 150, width: 40, height: 30 };
|
||||
assert_eq!(input.move_clamped(80, 80, bounds), Rect { x: 160, y: 170, width: 40, height: 30 });
|
||||
}
|
||||
}
|
||||
24
desktop_mark/tauri/src-tauri/src/storage.rs
Normal file
24
desktop_mark/tauri/src-tauri/src/storage.rs
Normal file
@ -0,0 +1,24 @@
|
||||
use std::fs;
|
||||
use std::path::PathBuf;
|
||||
|
||||
use crate::models::AnnotationProject;
|
||||
|
||||
pub fn default_project_path() -> Result<PathBuf, String> {
|
||||
let home = dirs::home_dir().ok_or_else(|| "无法定位用户目录".to_string())?;
|
||||
Ok(home.join(".desktop_mark").join("last_project.json"))
|
||||
}
|
||||
|
||||
pub fn save_project(project: &AnnotationProject) -> Result<(), String> {
|
||||
let path = default_project_path()?;
|
||||
let parent = path.parent().ok_or_else(|| "无法定位项目目录".to_string())?;
|
||||
fs::create_dir_all(parent).map_err(|err| format!("无法创建项目目录：{err}"))?;
|
||||
let json = serde_json::to_string_pretty(project).map_err(|err| format!("无法序列化标注项目：{err}"))?;
|
||||
fs::write(&path, json).map_err(|err| format!("无法保存标注项目：{err}"))
|
||||
}
|
||||
|
||||
pub fn load_project() -> Result<Option<AnnotationProject>, String> {
|
||||
let path = default_project_path()?;
|
||||
if !path.exists() { return Ok(None); }
|
||||
let text = fs::read_to_string(&path).map_err(|err| format!("无法读取标注项目：{err}"))?;
|
||||
serde_json::from_str(&text).map(Some).map_err(|err| format!("无法解析标注项目：{err}"))
|
||||
}
|
||||
121
desktop_mark/tauri/src-tauri/src/win32_window.rs
Normal file
121
desktop_mark/tauri/src-tauri/src/win32_window.rs
Normal file
@ -0,0 +1,121 @@
|
||||
use std::path::Path;
|
||||
|
||||
use windows::core::PWSTR;
|
||||
use windows::Win32::Foundation::{CloseHandle, BOOL, HWND, POINT, RECT};
|
||||
use windows::Win32::System::Threading::{OpenProcess, QueryFullProcessImageNameW, PROCESS_NAME_FORMAT, PROCESS_QUERY_LIMITED_INFORMATION};
|
||||
use windows::Win32::UI::HiDpi::{SetProcessDpiAwarenessContext, DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2};
|
||||
use windows::Win32::UI::WindowsAndMessaging::{
|
||||
GetCursorPos, GetForegroundWindow, GetSystemMetrics, GetTopWindow, GetWindow,
|
||||
GetWindowLongPtrW, GetWindowRect, GetWindowTextLengthW, GetWindowTextW,
|
||||
GetWindowThreadProcessId, IsIconic, IsWindowVisible, GW_HWNDNEXT, GWL_EXSTYLE,
|
||||
SM_CXVIRTUALSCREEN, SM_CYVIRTUALSCREEN, SM_XVIRTUALSCREEN, SM_YVIRTUALSCREEN,
|
||||
WS_EX_TOOLWINDOW,
|
||||
};
|
||||
|
||||
use crate::models::{Rect, WindowInfo};
|
||||
|
||||
pub fn set_process_dpi_awareness() {
|
||||
unsafe { let _ = SetProcessDpiAwarenessContext(DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2); }
|
||||
}
|
||||
|
||||
pub fn virtual_screen_rect() -> Rect {
|
||||
unsafe {
|
||||
Rect {
|
||||
x: GetSystemMetrics(SM_XVIRTUALSCREEN),
|
||||
y: GetSystemMetrics(SM_YVIRTUALSCREEN),
|
||||
width: GetSystemMetrics(SM_CXVIRTUALSCREEN),
|
||||
height: GetSystemMetrics(SM_CYVIRTUALSCREEN),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub fn cursor_position() -> (i32, i32) {
|
||||
let mut point = POINT::default();
|
||||
unsafe { let _ = GetCursorPos(&mut point); }
|
||||
(point.x, point.y)
|
||||
}
|
||||
|
||||
pub fn window_at(x: i32, y: i32, excluded_hwnds: &[isize]) -> WindowInfo {
|
||||
let mut hwnd = unsafe { GetTopWindow(HWND(std::ptr::null_mut())).ok() };
|
||||
while let Some(current) = hwnd {
|
||||
if is_candidate_window(current, excluded_hwnds) {
|
||||
if let Some(rect) = rect_for_hwnd(current) {
|
||||
if rect.contains(x, y) { return window_info(current, rect); }
|
||||
}
|
||||
}
|
||||
hwnd = unsafe { GetWindow(current, GW_HWNDNEXT).ok() };
|
||||
}
|
||||
desktop_window()
|
||||
}
|
||||
|
||||
pub fn foreground_window(excluded_hwnds: &[isize]) -> Option<WindowInfo> {
|
||||
let hwnd = unsafe { GetForegroundWindow() };
|
||||
if hwnd.0.is_null() || !is_candidate_window(hwnd, excluded_hwnds) { return None; }
|
||||
rect_for_hwnd(hwnd).map(|rect| window_info(hwnd, rect))
|
||||
}
|
||||
|
||||
fn is_candidate_window(hwnd: HWND, excluded_hwnds: &[isize]) -> bool {
|
||||
if excluded_hwnds.iter().any(|value| *value == hwnd.0 as isize) { return false; }
|
||||
unsafe {
|
||||
if !IsWindowVisible(hwnd).as_bool() || IsIconic(hwnd).as_bool() { return false; }
|
||||
let ex_style = GetWindowLongPtrW(hwnd, GWL_EXSTYLE) as u32;
|
||||
if ex_style & WS_EX_TOOLWINDOW.0 != 0 { return false; }
|
||||
}
|
||||
rect_for_hwnd(hwnd).is_some_and(|rect| rect.width > 0 && rect.height > 0)
|
||||
}
|
||||
|
||||
fn rect_for_hwnd(hwnd: HWND) -> Option<Rect> {
|
||||
let mut rect = RECT::default();
|
||||
if unsafe { GetWindowRect(hwnd, &mut rect) }.is_err() { return None; }
|
||||
let width = rect.right - rect.left;
|
||||
let height = rect.bottom - rect.top;
|
||||
if width <= 0 || height <= 0 { return None; }
|
||||
Some(Rect { x: rect.left, y: rect.top, width, height })
|
||||
}
|
||||
|
||||
fn window_info(hwnd: HWND, rect: Rect) -> WindowInfo {
|
||||
WindowInfo {
|
||||
hwnd: Some(hwnd.0 as isize),
|
||||
title: window_title(hwnd),
|
||||
process_name: process_name(hwnd),
|
||||
rect,
|
||||
is_desktop: false,
|
||||
}
|
||||
}
|
||||
|
||||
fn desktop_window() -> WindowInfo {
|
||||
WindowInfo {
|
||||
hwnd: None,
|
||||
title: "Desktop".into(),
|
||||
process_name: "explorer.exe".into(),
|
||||
rect: virtual_screen_rect(),
|
||||
is_desktop: true,
|
||||
}
|
||||
}
|
||||
|
||||
fn window_title(hwnd: HWND) -> String {
|
||||
let len = unsafe { GetWindowTextLengthW(hwnd) };
|
||||
if len <= 0 { return "Untitled".into(); }
|
||||
let mut buf = vec![0u16; (len + 1) as usize];
|
||||
let copied = unsafe { GetWindowTextW(hwnd, &mut buf) };
|
||||
if copied <= 0 { return "Untitled".into(); }
|
||||
let text = String::from_utf16_lossy(&buf[..copied as usize]).trim().to_string();
|
||||
if text.is_empty() { "Untitled".into() } else { text }
|
||||
}
|
||||
|
||||
fn process_name(hwnd: HWND) -> String {
|
||||
let mut pid = 0u32;
|
||||
unsafe { GetWindowThreadProcessId(hwnd, Some(&mut pid)); }
|
||||
if pid == 0 { return String::new(); }
|
||||
let handle = match unsafe { OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, BOOL(0), pid) } {
|
||||
Ok(handle) => handle,
|
||||
Err(_) => return String::new(),
|
||||
};
|
||||
let mut buf = vec![0u16; 32768];
|
||||
let mut size = buf.len() as u32;
|
||||
let result = unsafe { QueryFullProcessImageNameW(handle, PROCESS_NAME_FORMAT(0), PWSTR(buf.as_mut_ptr()), &mut size) };
|
||||
unsafe { let _ = CloseHandle(handle); }
|
||||
if result.is_err() || size == 0 { return String::new(); }
|
||||
let path = String::from_utf16_lossy(&buf[..size as usize]);
|
||||
Path::new(&path).file_name().map(|name| name.to_string_lossy().into_owned()).unwrap_or_default()
|
||||
}
|
||||
71
desktop_mark/tauri/src-tauri/src/window_manager.rs
Normal file
71
desktop_mark/tauri/src-tauri/src/window_manager.rs
Normal file
@ -0,0 +1,71 @@
|
||||
use tauri::{AppHandle, Manager, WebviewUrl, WebviewWindowBuilder};
|
||||
|
||||
use crate::win32_window;
|
||||
|
||||
const LABELS: [&str; 4] = ["floating", "overlay", "panel", "annotation_form"];
|
||||
|
||||
pub fn create_all_windows(app: &AppHandle) -> Result<(), String> {
|
||||
let screen = win32_window::virtual_screen_rect();
|
||||
WebviewWindowBuilder::new(app, "floating", WebviewUrl::App("index.html".into()))
|
||||
.title("RPA")
|
||||
.inner_size(80.0, 80.0)
|
||||
.position(32.0, 160.0)
|
||||
.decorations(false)
|
||||
.transparent(true)
|
||||
.resizable(false)
|
||||
.always_on_top(true)
|
||||
.skip_taskbar(true)
|
||||
.visible(true)
|
||||
.build()
|
||||
.map_err(|err| err.to_string())?;
|
||||
|
||||
WebviewWindowBuilder::new(app, "overlay", WebviewUrl::App("index.html".into()))
|
||||
.title("Desktop Mark Overlay")
|
||||
.inner_size(screen.width as f64, screen.height as f64)
|
||||
.position(screen.x as f64, screen.y as f64)
|
||||
.decorations(false)
|
||||
.transparent(true)
|
||||
.resizable(false)
|
||||
.always_on_top(true)
|
||||
.skip_taskbar(true)
|
||||
.visible(false)
|
||||
.build()
|
||||
.map_err(|err| err.to_string())?;
|
||||
|
||||
WebviewWindowBuilder::new(app, "panel", WebviewUrl::App("index.html".into()))
|
||||
.title("标注信息")
|
||||
.inner_size(300.0, screen.height as f64)
|
||||
.position(screen.x as f64, screen.y as f64)
|
||||
.decorations(false)
|
||||
.transparent(false)
|
||||
.resizable(false)
|
||||
.always_on_top(true)
|
||||
.skip_taskbar(true)
|
||||
.visible(false)
|
||||
.build()
|
||||
.map_err(|err| err.to_string())?;
|
||||
|
||||
WebviewWindowBuilder::new(app, "annotation_form", WebviewUrl::App("index.html".into()))
|
||||
.title("编辑标注")
|
||||
.inner_size(320.0, 420.0)
|
||||
.decorations(false)
|
||||
.transparent(false)
|
||||
.resizable(false)
|
||||
.always_on_top(true)
|
||||
.skip_taskbar(true)
|
||||
.visible(false)
|
||||
.build()
|
||||
.map_err(|err| err.to_string())?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn tool_window_hwnds(app: &AppHandle) -> Vec<isize> {
|
||||
let mut hwnds = Vec::new();
|
||||
for label in LABELS {
|
||||
if let Some(window) = app.get_webview_window(label) {
|
||||
#[cfg(target_os = "windows")]
|
||||
if let Ok(hwnd) = window.hwnd() { hwnds.push(hwnd.0 as isize); }
|
||||
}
|
||||
}
|
||||
hwnds
|
||||
}
|
||||
17
desktop_mark/tauri/src-tauri/tauri.conf.json
Normal file
17
desktop_mark/tauri/src-tauri/tauri.conf.json
Normal file
@ -0,0 +1,17 @@
|
||||
{
|
||||
"$schema": "https://schema.tauri.app/config/2",
|
||||
"productName": "Desktop Mark RPA",
|
||||
"version": "0.1.0",
|
||||
"identifier": "com.desktopmark.tauri",
|
||||
"build": {
|
||||
"beforeDevCommand": "vite --host 127.0.0.1",
|
||||
"devUrl": "http://127.0.0.1:1420",
|
||||
"beforeBuildCommand": "npm run typecheck && vite build",
|
||||
"frontendDist": "../dist"
|
||||
},
|
||||
"app": {
|
||||
"windows": [],
|
||||
"security": { "csp": null }
|
||||
},
|
||||
"bundle": { "active": true, "targets": "all", "icon": ["icons/icon.ico"] }
|
||||
}
|
||||
42
desktop_mark/tauri/src/geometry/rect.ts
Normal file
42
desktop_mark/tauri/src/geometry/rect.ts
Normal file
@ -0,0 +1,42 @@
|
||||
import type { Rect } from '../types';
|
||||
|
||||
export function right(rect: Rect): number { return rect.x + rect.width; }
|
||||
export function bottom(rect: Rect): number { return rect.y + rect.height; }
|
||||
|
||||
export function normalized(rect: Rect): Rect {
|
||||
const x1 = Math.min(rect.x, right(rect));
|
||||
const y1 = Math.min(rect.y, bottom(rect));
|
||||
const x2 = Math.max(rect.x, right(rect));
|
||||
const y2 = Math.max(rect.y, bottom(rect));
|
||||
return { x: x1, y: y1, width: x2 - x1, height: y2 - y1 };
|
||||
}
|
||||
|
||||
export function contains(rect: Rect, x: number, y: number): boolean {
|
||||
const r = normalized(rect);
|
||||
return r.x <= x && x < right(r) && r.y <= y && y < bottom(r);
|
||||
}
|
||||
|
||||
export function clampedTo(rect: Rect, bounds: Rect): Rect {
|
||||
const r = normalized(rect);
|
||||
const limit = normalized(bounds);
|
||||
const x1 = Math.max(limit.x, Math.min(r.x, right(limit)));
|
||||
const y1 = Math.max(limit.y, Math.min(r.y, bottom(limit)));
|
||||
let x2 = Math.max(limit.x, Math.min(right(r), right(limit)));
|
||||
let y2 = Math.max(limit.y, Math.min(bottom(r), bottom(limit)));
|
||||
if (x2 < x1) x2 = x1;
|
||||
if (y2 < y1) y2 = y1;
|
||||
return { x: x1, y: y1, width: x2 - x1, height: y2 - y1 };
|
||||
}
|
||||
|
||||
export function moveClamped(rect: Rect, dx: number, dy: number, bounds: Rect): Rect {
|
||||
const r = normalized(rect);
|
||||
const limit = normalized(bounds);
|
||||
const maxX = Math.max(limit.x, right(limit) - r.width);
|
||||
const maxY = Math.max(limit.y, bottom(limit) - r.height);
|
||||
return {
|
||||
x: Math.max(limit.x, Math.min(r.x + dx, maxX)),
|
||||
y: Math.max(limit.y, Math.min(r.y + dy, maxY)),
|
||||
width: r.width,
|
||||
height: r.height,
|
||||
};
|
||||
}
|
||||
46
desktop_mark/tauri/src/main.tsx
Normal file
46
desktop_mark/tauri/src/main.tsx
Normal file
@ -0,0 +1,46 @@
|
||||
import React from 'react';
|
||||
import { createRoot } from 'react-dom/client';
|
||||
import { invoke } from '@tauri-apps/api/core';
|
||||
import { listen } from '@tauri-apps/api/event';
|
||||
import { getCurrentWindow } from '@tauri-apps/api/window';
|
||||
import './styles.css';
|
||||
import { Floating } from './windows/Floating';
|
||||
import { Overlay, startAnnotation, startWindowPicking } from './windows/Overlay';
|
||||
import { Panel } from './windows/Panel';
|
||||
import { AnnotationForm } from './windows/AnnotationForm';
|
||||
import { installStateSync, setState } from './state/appStore';
|
||||
import type { AnnotationProject } from './types';
|
||||
|
||||
async function bootstrap(): Promise<void> {
|
||||
await installStateSync();
|
||||
const label = getCurrentWindow().label;
|
||||
const root = createRoot(document.getElementById('root')!);
|
||||
const componentByLabel: Record<string, React.ReactElement> = {
|
||||
floating: <Floating />,
|
||||
overlay: <Overlay />,
|
||||
panel: <Panel />,
|
||||
annotation_form: <AnnotationForm />,
|
||||
};
|
||||
root.render(componentByLabel[label] ?? <Floating />);
|
||||
|
||||
await listen('hotkey:start-window-picking', () => { void startWindowPicking(); });
|
||||
await listen('hotkey:start-annotation', () => { void startAnnotation(); });
|
||||
await listen<string>('hotkey:registration-error', (event) => setState({ alertMessage: event.payload }));
|
||||
|
||||
if (label === 'floating') {
|
||||
try {
|
||||
const project = await invoke<AnnotationProject | null>('load_project');
|
||||
if (project) setState({ project }, { broadcast: true });
|
||||
} catch (error) {
|
||||
setState({ alertMessage: `无法加载标注项目：${String(error)}`, project: null });
|
||||
}
|
||||
try {
|
||||
const hwnds = await invoke<number[]>('tool_window_hwnds');
|
||||
setState({ toolWindowHwnds: hwnds });
|
||||
} catch (error) {
|
||||
setState({ alertMessage: `无法读取工具窗口句柄：${String(error)}` });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void bootstrap();
|
||||
47
desktop_mark/tauri/src/state/appStore.ts
Normal file
47
desktop_mark/tauri/src/state/appStore.ts
Normal file
@ -0,0 +1,47 @@
|
||||
import { useSyncExternalStore } from 'react';
|
||||
import { emit, listen } from '@tauri-apps/api/event';
|
||||
import type { AppState } from '../types';
|
||||
|
||||
const initialState: AppState = {
|
||||
mode: 'idle',
|
||||
hoverWindow: null,
|
||||
lockedWindow: null,
|
||||
project: null,
|
||||
selectedAnnotationId: null,
|
||||
toolWindowHwnds: [],
|
||||
tempRect: null,
|
||||
alertMessage: null,
|
||||
};
|
||||
|
||||
let state = initialState;
|
||||
let applyingRemote = false;
|
||||
const listeners = new Set<() => void>();
|
||||
|
||||
export function getState(): AppState { return state; }
|
||||
|
||||
function notify(): void { for (const listener of listeners) listener(); }
|
||||
|
||||
export function setState(update: Partial<AppState> | ((current: AppState) => AppState), options: { broadcast?: boolean } = {}): AppState {
|
||||
state = typeof update === 'function' ? update(state) : { ...state, ...update };
|
||||
notify();
|
||||
if (options.broadcast !== false && !applyingRemote) void emit('state:sync', state);
|
||||
return state;
|
||||
}
|
||||
|
||||
export function subscribe(listener: () => void): () => void {
|
||||
listeners.add(listener);
|
||||
return () => listeners.delete(listener);
|
||||
}
|
||||
|
||||
export function useAppState(): AppState {
|
||||
return useSyncExternalStore(subscribe, getState, getState);
|
||||
}
|
||||
|
||||
export async function installStateSync(): Promise<void> {
|
||||
await listen<AppState>('state:sync', (event) => {
|
||||
applyingRemote = true;
|
||||
state = event.payload;
|
||||
notify();
|
||||
applyingRemote = false;
|
||||
});
|
||||
}
|
||||
27
desktop_mark/tauri/src/styles.css
Normal file
27
desktop_mark/tauri/src/styles.css
Normal file
@ -0,0 +1,27 @@
|
||||
html, body, #root { margin: 0; width: 100%; height: 100%; overflow: hidden; font-family: "Microsoft YaHei", "Segoe UI", sans-serif; }
|
||||
* { box-sizing: border-box; }
|
||||
button, input, textarea, select { font: inherit; }
|
||||
.floating-shell { position: relative; width: 100%; height: 100%; user-select: none; }
|
||||
.floating-canvas { width: 80px; height: 80px; display: block; cursor: grab; }
|
||||
.floating-menu { position: absolute; left: 4px; top: 84px; width: 142px; padding: 6px; border-radius: 10px; background: rgba(24, 24, 28, .96); color: #fff; border: 1px solid rgba(255,255,255,.16); box-shadow: 0 10px 28px rgba(0,0,0,.36); }
|
||||
.floating-menu button { width: 100%; min-height: 30px; border: 0; border-radius: 7px; padding: 6px 8px; color: #fff; background: transparent; text-align: left; cursor: pointer; }
|
||||
.floating-menu button:hover { background: rgba(255,255,255,.12); }
|
||||
.floating-menu button.danger { color: #ffb3b3; }
|
||||
.overlay-root { width: 100vw; height: 100vh; cursor: crosshair; }
|
||||
.panel { width: 300px; height: 100vh; padding: 14px; background: rgba(246, 248, 252, .96); color: #20242a; border-right: 1px solid #d8dde7; overflow: auto; }
|
||||
.panel h2 { font-size: 16px; margin: 8px 0 12px; }
|
||||
.panel label { display: block; font-size: 12px; color: #596273; margin-top: 10px; }
|
||||
.panel .value { padding: 6px 0; word-break: break-all; }
|
||||
.panel input, .panel textarea, .form input, .form textarea, .form select { width: 100%; border: 1px solid #c7cedb; border-radius: 6px; padding: 7px 8px; background: #fff; }
|
||||
.panel textarea { height: 90px; resize: none; }
|
||||
.annotation-list { display: flex; flex-direction: column; gap: 6px; }
|
||||
.annotation-item { border: 1px solid #d5dbea; background: #fff; border-radius: 8px; padding: 8px; text-align: left; cursor: pointer; }
|
||||
.annotation-item.selected { border-color: #007aff; box-shadow: 0 0 0 2px rgba(0,122,255,.15); }
|
||||
.form { width: 100vw; height: 100vh; padding: 14px; background: #f8f9fb; color: #20242a; border: 1px solid #d4d9e4; }
|
||||
.form label { display:block; font-size: 12px; color: #596273; margin: 8px 0 4px; }
|
||||
.form textarea { height: 82px; resize: none; }
|
||||
.actions { display: grid; grid-template-columns: 1fr 1fr; gap: 4px 8px; }
|
||||
.actions label { color: #20242a; font-size: 12px; display:flex; align-items:center; gap:4px; margin:0; }
|
||||
.preview { height: 56px; border: 1px dashed #999; color: #666; display:flex; align-items:center; justify-content:center; margin-top: 4px; }
|
||||
.form-buttons { display:flex; justify-content:flex-end; gap: 8px; margin-top: 12px; }
|
||||
.warning { background:#fff7d6; border:1px solid #f2c94c; color:#5c4700; padding:8px; border-radius:6px; margin-bottom:8px; }
|
||||
49
desktop_mark/tauri/src/types.ts
Normal file
49
desktop_mark/tauri/src/types.ts
Normal file
@ -0,0 +1,49 @@
|
||||
export interface Rect { x: number; y: number; width: number; height: number }
|
||||
|
||||
export interface WindowInfo {
|
||||
hwnd: number | null;
|
||||
title: string;
|
||||
process_name: string;
|
||||
rect: Rect;
|
||||
is_desktop: boolean;
|
||||
}
|
||||
|
||||
export type AnnotationType = 'button' | 'input' | 'text' | 'image' | 'other';
|
||||
export type AnnotationAction = 'click' | 'double_click' | 'right_click' | 'input_text' | 'hover' | 'scroll';
|
||||
|
||||
export interface Annotation {
|
||||
id: string;
|
||||
rect: Rect;
|
||||
control_name: string;
|
||||
description: string;
|
||||
type: AnnotationType;
|
||||
actions: AnnotationAction[];
|
||||
feature_preview_path: string | null;
|
||||
}
|
||||
|
||||
export interface AnnotationProject {
|
||||
window: WindowInfo;
|
||||
custom_window_name: string;
|
||||
window_description: string;
|
||||
annotations: Annotation[];
|
||||
}
|
||||
|
||||
export type OverlayMode =
|
||||
| 'idle'
|
||||
| 'picking_window'
|
||||
| 'window_locked'
|
||||
| 'drawing_annotation'
|
||||
| 'editing_annotation'
|
||||
| 'moving_annotation'
|
||||
| 'resizing_annotation';
|
||||
|
||||
export interface AppState {
|
||||
mode: OverlayMode;
|
||||
hoverWindow: WindowInfo | null;
|
||||
lockedWindow: WindowInfo | null;
|
||||
project: AnnotationProject | null;
|
||||
selectedAnnotationId: string | null;
|
||||
toolWindowHwnds: number[];
|
||||
tempRect: Rect | null;
|
||||
alertMessage: string | null;
|
||||
}
|
||||
1
desktop_mark/tauri/src/vite-env.d.ts
vendored
Normal file
1
desktop_mark/tauri/src/vite-env.d.ts
vendored
Normal file
@ -0,0 +1 @@
|
||||
declare module '*.css';
|
||||
55
desktop_mark/tauri/src/windows/AnnotationForm.tsx
Normal file
55
desktop_mark/tauri/src/windows/AnnotationForm.tsx
Normal file
@ -0,0 +1,55 @@
|
||||
import { invoke } from '@tauri-apps/api/core';
|
||||
import { getCurrentWindow } from '@tauri-apps/api/window';
|
||||
import { useEffect, useState, type ReactElement } from 'react';
|
||||
import { setState, useAppState } from '../state/appStore';
|
||||
import type { AnnotationAction, AnnotationType } from '../types';
|
||||
|
||||
const ACTIONS: AnnotationAction[] = ['click', 'double_click', 'right_click', 'input_text', 'hover', 'scroll'];
|
||||
const TYPES: AnnotationType[] = ['button', 'input', 'text', 'image', 'other'];
|
||||
|
||||
export function AnnotationForm(): ReactElement {
|
||||
const state = useAppState();
|
||||
const annotation = state.project?.annotations.find((item) => item.id === state.selectedAnnotationId) ?? null;
|
||||
const [controlName, setControlName] = useState('');
|
||||
const [description, setDescription] = useState('');
|
||||
const [type, setType] = useState<AnnotationType>('other');
|
||||
const [actions, setActions] = useState<AnnotationAction[]>([]);
|
||||
|
||||
useEffect(() => {
|
||||
setControlName(annotation?.control_name ?? '');
|
||||
setDescription(annotation?.description ?? '');
|
||||
setType(annotation?.type ?? 'other');
|
||||
setActions(annotation?.actions ?? []);
|
||||
}, [annotation?.id]);
|
||||
|
||||
const save = async () => {
|
||||
if (!state.project || !annotation) return;
|
||||
const updated = { ...annotation, control_name: controlName.trim(), description: description.trim(), type, actions, feature_preview_path: null };
|
||||
const project = { ...state.project, annotations: state.project.annotations.map((item) => item.id === updated.id ? updated : item) };
|
||||
setState({ project });
|
||||
try { await invoke('save_project', { project }); await getCurrentWindow().hide(); }
|
||||
catch (error) { setState({ alertMessage: `无法保存标注项目：${String(error)}` }); }
|
||||
};
|
||||
|
||||
return <div className="form">
|
||||
{state.alertMessage ? <div className="warning">{state.alertMessage}</div> : null}
|
||||
<label>控件名</label>
|
||||
<input value={controlName} onChange={(event) => setControlName(event.target.value)} />
|
||||
<label>描述</label>
|
||||
<textarea value={description} onChange={(event) => setDescription(event.target.value)} />
|
||||
<label>类型</label>
|
||||
<select value={type} onChange={(event) => setType(event.target.value as AnnotationType)}>
|
||||
{TYPES.map((item) => <option key={item} value={item}>{item}</option>)}
|
||||
</select>
|
||||
<label>动作</label>
|
||||
<div className="actions">{ACTIONS.map((action) => <label key={action}><input type="checkbox" checked={actions.includes(action)} onChange={(event) => {
|
||||
setActions((current) => event.target.checked ? [...current, action] : current.filter((item) => item !== action));
|
||||
}} />{action}</label>)}</div>
|
||||
<label>特征图</label>
|
||||
<div className="preview">预览待生成</div>
|
||||
<div className="form-buttons">
|
||||
<button onClick={() => { void getCurrentWindow().hide(); }}>取消</button>
|
||||
<button onClick={() => { void save(); }}>保存</button>
|
||||
</div>
|
||||
</div>;
|
||||
}
|
||||
110
desktop_mark/tauri/src/windows/Floating.tsx
Normal file
110
desktop_mark/tauri/src/windows/Floating.tsx
Normal file
@ -0,0 +1,110 @@
|
||||
import { useEffect, useRef, useState, type ReactElement } from 'react';
|
||||
import { invoke } from '@tauri-apps/api/core';
|
||||
import { PhysicalSize, getCurrentWindow } from '@tauri-apps/api/window';
|
||||
import { useAppState } from '../state/appStore';
|
||||
import { startWindowPicking } from './Overlay';
|
||||
|
||||
export function Floating(): ReactElement {
|
||||
const state = useAppState();
|
||||
const canvasRef = useRef<HTMLCanvasElement>(null);
|
||||
const [menuOpen, setMenuOpen] = useState(false);
|
||||
|
||||
useEffect(() => {
|
||||
const canvas = canvasRef.current;
|
||||
if (!canvas) return;
|
||||
const size = 80;
|
||||
const dpr = window.devicePixelRatio || 1;
|
||||
canvas.width = size * dpr;
|
||||
canvas.height = size * dpr;
|
||||
canvas.style.width = `${size}px`;
|
||||
canvas.style.height = `${size}px`;
|
||||
const ctx = canvas.getContext('2d');
|
||||
if (!ctx) return;
|
||||
ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
|
||||
ctx.clearRect(0, 0, size, size);
|
||||
|
||||
const body = ctx.createLinearGradient(10, 4, 72, 76);
|
||||
body.addColorStop(0, '#30333d');
|
||||
body.addColorStop(0.55, '#171a22');
|
||||
body.addColorStop(1, '#0b0d12');
|
||||
roundRect(ctx, 1, 1, 78, 78, 18);
|
||||
ctx.fillStyle = body;
|
||||
ctx.fill();
|
||||
ctx.strokeStyle = 'rgba(255,255,255,.18)';
|
||||
ctx.lineWidth = 1;
|
||||
ctx.stroke();
|
||||
|
||||
const glow = ctx.createRadialGradient(30, 24, 2, 30, 24, 34);
|
||||
glow.addColorStop(0, 'rgba(0,122,255,.55)');
|
||||
glow.addColorStop(1, 'rgba(0,122,255,0)');
|
||||
ctx.fillStyle = glow;
|
||||
ctx.fillRect(0, 0, size, size);
|
||||
|
||||
ctx.fillStyle = '#ffffff';
|
||||
ctx.font = '700 22px "Segoe UI", sans-serif';
|
||||
ctx.textAlign = 'center';
|
||||
ctx.textBaseline = 'middle';
|
||||
ctx.fillText('RPA', 40, 31);
|
||||
ctx.fillStyle = 'rgba(255,255,255,.72)';
|
||||
ctx.font = '500 12px "Segoe UI", sans-serif';
|
||||
ctx.fillText('Ctrl+1', 40, 55);
|
||||
|
||||
if (state.alertMessage) {
|
||||
ctx.beginPath();
|
||||
ctx.arc(63, 17, 7, 0, Math.PI * 2);
|
||||
ctx.fillStyle = '#ffd966';
|
||||
ctx.fill();
|
||||
ctx.fillStyle = '#5c4700';
|
||||
ctx.font = '700 11px "Segoe UI", sans-serif';
|
||||
ctx.fillText('!', 63, 17);
|
||||
}
|
||||
}, [state.alertMessage]);
|
||||
|
||||
useEffect(() => {
|
||||
void getCurrentWindow().setSize(new PhysicalSize(menuOpen ? 150 : 80, menuOpen ? 126 : 80));
|
||||
}, [menuOpen]);
|
||||
|
||||
return (
|
||||
<div
|
||||
className="floating-shell"
|
||||
onContextMenu={(event) => {
|
||||
event.preventDefault();
|
||||
setMenuOpen(true);
|
||||
}}
|
||||
onPointerDown={(event) => {
|
||||
if (event.button === 0 && !menuOpen) void getCurrentWindow().startDragging();
|
||||
}}
|
||||
title={state.alertMessage ?? 'Ctrl+1'}
|
||||
>
|
||||
<canvas
|
||||
ref={canvasRef}
|
||||
className="floating-canvas"
|
||||
aria-label={state.alertMessage ?? 'RPA 悬浮窗，左键开始识别窗口，右键打开菜单'}
|
||||
role="button"
|
||||
onClick={() => {
|
||||
if (menuOpen) {
|
||||
setMenuOpen(false);
|
||||
return;
|
||||
}
|
||||
void startWindowPicking();
|
||||
}}
|
||||
/>
|
||||
{menuOpen ? (
|
||||
<div className="floating-menu" role="menu">
|
||||
<button type="button" role="menuitem" onClick={() => { setMenuOpen(false); void startWindowPicking(); }}>识别窗口</button>
|
||||
<button type="button" role="menuitem" className="danger" onClick={() => { void invoke('exit_app'); }}>退出应用</button>
|
||||
</div>
|
||||
) : null}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
function roundRect(ctx: CanvasRenderingContext2D, x: number, y: number, width: number, height: number, radius: number): void {
|
||||
ctx.beginPath();
|
||||
ctx.moveTo(x + radius, y);
|
||||
ctx.arcTo(x + width, y, x + width, y + height, radius);
|
||||
ctx.arcTo(x + width, y + height, x, y + height, radius);
|
||||
ctx.arcTo(x, y + height, x, y, radius);
|
||||
ctx.arcTo(x, y, x + width, y, radius);
|
||||
ctx.closePath();
|
||||
}
|
||||
253
desktop_mark/tauri/src/windows/Overlay.tsx
Normal file
253
desktop_mark/tauri/src/windows/Overlay.tsx
Normal file
@ -0,0 +1,253 @@
|
||||
import { invoke } from '@tauri-apps/api/core';
|
||||
import { PhysicalPosition, PhysicalSize } from '@tauri-apps/api/window';
|
||||
import { WebviewWindow } from '@tauri-apps/api/webviewWindow';
|
||||
import { useEffect, type ReactElement } from 'react';
|
||||
import { getState, setState, useAppState } from '../state/appStore';
|
||||
import type { Annotation, Rect, WindowInfo } from '../types';
|
||||
import { bottom, clampedTo, contains, moveClamped, normalized, right } from '../geometry/rect';
|
||||
|
||||
const MIN_ANNOTATION_SIZE = 6;
|
||||
const HANDLE_SIZE = 8;
|
||||
type HandleName = 'nw' | 'n' | 'ne' | 'e' | 'se' | 's' | 'sw' | 'w';
|
||||
let pollId: number | null = null;
|
||||
let virtualRect: Rect = { x: 0, y: 0, width: 1, height: 1 };
|
||||
let dragStart: { x: number; y: number } | null = null;
|
||||
let moving: { id: string; original: Rect } | null = null;
|
||||
let resizing: { id: string; original: Rect; handle: HandleName } | null = null;
|
||||
|
||||
async function overlayWindow(): Promise<WebviewWindow> { return existingWindow('overlay'); }
|
||||
async function panelWindow(): Promise<WebviewWindow> { return existingWindow('panel'); }
|
||||
async function formWindow(): Promise<WebviewWindow> { return existingWindow('annotation_form'); }
|
||||
async function existingWindow(label: string): Promise<WebviewWindow> {
|
||||
const win = await WebviewWindow.getByLabel(label);
|
||||
if (!win) throw new Error(`窗口 ${label} 尚未创建`);
|
||||
return win;
|
||||
}
|
||||
function toLocal(rect: Rect): Rect { const r = normalized(rect); return { x: r.x - virtualRect.x, y: r.y - virtualRect.y, width: r.width, height: r.height }; }
|
||||
function screenPoint(event: React.PointerEvent): { x: number; y: number } { return { x: Math.round(event.clientX + virtualRect.x), y: Math.round(event.clientY + virtualRect.y) }; }
|
||||
function stopPoll(): void { if (pollId !== null) window.clearInterval(pollId); pollId = null; }
|
||||
function sameWindow(a: WindowInfo, b: WindowInfo): boolean { return a.hwnd === b.hwnd && a.title === b.title && a.process_name === b.process_name && JSON.stringify(a.rect) === JSON.stringify(b.rect); }
|
||||
|
||||
async function pollHover(): Promise<void> {
|
||||
const [x, y] = await invoke<[number, number]>('cursor_position');
|
||||
const hoverWindow = await invoke<WindowInfo>('window_at', { x, y, excludedHwnds: getState().toolWindowHwnds });
|
||||
setState({ hoverWindow });
|
||||
}
|
||||
|
||||
export async function startWindowPicking(): Promise<void> {
|
||||
virtualRect = await invoke<Rect>('get_virtual_screen_rect');
|
||||
const overlay = await overlayWindow();
|
||||
await overlay.setPosition(new PhysicalPosition(virtualRect.x, virtualRect.y));
|
||||
await overlay.setSize(new PhysicalSize(virtualRect.width, virtualRect.height));
|
||||
await overlay.show();
|
||||
await overlay.setFocus();
|
||||
stopPoll();
|
||||
setState({ mode: 'picking_window', hoverWindow: null, tempRect: null });
|
||||
await pollHover();
|
||||
pollId = window.setInterval(() => { void pollHover(); }, 50);
|
||||
}
|
||||
|
||||
export async function startAnnotation(): Promise<void> {
|
||||
const current = getState();
|
||||
if (!current.lockedWindow) { await startWindowPicking(); return; }
|
||||
virtualRect = await invoke<Rect>('get_virtual_screen_rect');
|
||||
const overlay = await overlayWindow();
|
||||
await overlay.show();
|
||||
await overlay.setFocus();
|
||||
stopPoll();
|
||||
setState({ mode: 'drawing_annotation', tempRect: null });
|
||||
}
|
||||
|
||||
export async function cancelFlow(): Promise<void> {
|
||||
stopPoll();
|
||||
dragStart = null; moving = null; resizing = null;
|
||||
setState({ mode: 'idle', hoverWindow: null, lockedWindow: null, tempRect: null });
|
||||
await (await overlayWindow()).hide();
|
||||
await (await panelWindow()).hide();
|
||||
await (await formWindow()).hide();
|
||||
}
|
||||
|
||||
async function openFormNear(rect: Rect): Promise<void> {
|
||||
const form = await formWindow();
|
||||
const width = 320;
|
||||
const height = 420;
|
||||
let x = right(rect) + 12;
|
||||
let y = bottom(rect) + 12;
|
||||
if (x + width > right(virtualRect)) x = rect.x - width - 12;
|
||||
if (y + height > bottom(virtualRect)) y = rect.y - height - 12;
|
||||
x = Math.max(virtualRect.x, Math.min(x, right(virtualRect) - width));
|
||||
y = Math.max(virtualRect.y, Math.min(y, bottom(virtualRect) - height));
|
||||
await form.setPosition(new PhysicalPosition(x, y));
|
||||
await form.show();
|
||||
await form.setFocus();
|
||||
}
|
||||
|
||||
function updateAnnotation(id: string, rect: Rect): void {
|
||||
setState((state) => {
|
||||
if (!state.project) return state;
|
||||
return { ...state, project: { ...state.project, annotations: state.project.annotations.map((item) => item.id === id ? { ...item, rect } : item) } };
|
||||
});
|
||||
}
|
||||
|
||||
function handleRects(rect: Rect): Record<HandleName, Rect> {
|
||||
const r = normalized(rect);
|
||||
const half = HANDLE_SIZE / 2;
|
||||
const cx = r.x + Math.floor(r.width / 2);
|
||||
const cy = r.y + Math.floor(r.height / 2);
|
||||
return {
|
||||
nw: { x: r.x - half, y: r.y - half, width: HANDLE_SIZE, height: HANDLE_SIZE },
|
||||
n: { x: cx - half, y: r.y - half, width: HANDLE_SIZE, height: HANDLE_SIZE },
|
||||
ne: { x: right(r) - half, y: r.y - half, width: HANDLE_SIZE, height: HANDLE_SIZE },
|
||||
e: { x: right(r) - half, y: cy - half, width: HANDLE_SIZE, height: HANDLE_SIZE },
|
||||
se: { x: right(r) - half, y: bottom(r) - half, width: HANDLE_SIZE, height: HANDLE_SIZE },
|
||||
s: { x: cx - half, y: bottom(r) - half, width: HANDLE_SIZE, height: HANDLE_SIZE },
|
||||
sw: { x: r.x - half, y: bottom(r) - half, width: HANDLE_SIZE, height: HANDLE_SIZE },
|
||||
w: { x: r.x - half, y: cy - half, width: HANDLE_SIZE, height: HANDLE_SIZE },
|
||||
};
|
||||
}
|
||||
|
||||
function hitTest(x: number, y: number): { annotation: Annotation | null; handle: HandleName | null } {
|
||||
const { project, selectedAnnotationId } = getState();
|
||||
if (!project) return { annotation: null, handle: null };
|
||||
const selected = project.annotations.find((annotation) => annotation.id === selectedAnnotationId);
|
||||
if (selected) {
|
||||
for (const [name, rect] of Object.entries(handleRects(selected.rect)) as [HandleName, Rect][]) {
|
||||
if (contains(rect, x, y)) return { annotation: selected, handle: name };
|
||||
}
|
||||
}
|
||||
for (let i = project.annotations.length - 1; i >= 0; i -= 1) {
|
||||
const annotation = project.annotations[i];
|
||||
if (contains(annotation.rect, x, y)) return { annotation, handle: null };
|
||||
}
|
||||
return { annotation: null, handle: null };
|
||||
}
|
||||
|
||||
function resizeRect(original: Rect, handle: HandleName, point: { x: number; y: number }): Rect {
|
||||
let left = original.x;
|
||||
let top = original.y;
|
||||
let rectRight = right(original);
|
||||
let rectBottom = bottom(original);
|
||||
if (handle.includes('w')) left = Math.min(point.x, rectRight - MIN_ANNOTATION_SIZE);
|
||||
if (handle.includes('e')) rectRight = Math.max(point.x, left + MIN_ANNOTATION_SIZE);
|
||||
if (handle.includes('n')) top = Math.min(point.y, rectBottom - MIN_ANNOTATION_SIZE);
|
||||
if (handle.includes('s')) rectBottom = Math.max(point.y, top + MIN_ANNOTATION_SIZE);
|
||||
return { x: left, y: top, width: rectRight - left, height: rectBottom - top };
|
||||
}
|
||||
|
||||
export function Overlay(): ReactElement {
|
||||
const state = useAppState();
|
||||
const target = state.lockedWindow ?? state.hoverWindow;
|
||||
|
||||
useEffect(() => {
|
||||
const onKey = (event: KeyboardEvent) => {
|
||||
if (['Escape', 'Enter', 'NumpadEnter'].includes(event.code)) void cancelFlow();
|
||||
};
|
||||
window.addEventListener('keydown', onKey);
|
||||
return () => window.removeEventListener('keydown', onKey);
|
||||
}, []);
|
||||
|
||||
const onPointerDown = async (event: React.PointerEvent) => {
|
||||
if (event.button !== 0) return;
|
||||
const point = screenPoint(event);
|
||||
if (state.mode === 'picking_window') {
|
||||
stopPoll();
|
||||
const lockedWindow = state.hoverWindow ?? await invoke<WindowInfo>('window_at', { x: point.x, y: point.y, excludedHwnds: state.toolWindowHwnds });
|
||||
setState((current) => {
|
||||
const keepProject = current.project && sameWindow(current.project.window, lockedWindow);
|
||||
return { ...current, lockedWindow, hoverWindow: null, mode: 'window_locked', project: keepProject ? current.project : { window: lockedWindow, custom_window_name: '', window_description: '', annotations: [] } };
|
||||
});
|
||||
await (await panelWindow()).show();
|
||||
return;
|
||||
}
|
||||
if (!state.lockedWindow) return;
|
||||
const hit = hitTest(point.x, point.y);
|
||||
if (hit.annotation && hit.handle) {
|
||||
resizing = { id: hit.annotation.id, original: hit.annotation.rect, handle: hit.handle };
|
||||
dragStart = point;
|
||||
setState({ selectedAnnotationId: hit.annotation.id, mode: 'resizing_annotation' });
|
||||
return;
|
||||
}
|
||||
if (hit.annotation) {
|
||||
moving = { id: hit.annotation.id, original: hit.annotation.rect };
|
||||
dragStart = point;
|
||||
setState({ selectedAnnotationId: hit.annotation.id, mode: 'moving_annotation' });
|
||||
return;
|
||||
}
|
||||
if (contains(state.lockedWindow.rect, point.x, point.y)) {
|
||||
dragStart = point;
|
||||
setState({ mode: 'drawing_annotation', tempRect: { x: point.x, y: point.y, width: 0, height: 0 } });
|
||||
}
|
||||
};
|
||||
|
||||
const onPointerMove = (event: React.PointerEvent) => {
|
||||
const point = screenPoint(event);
|
||||
const lockedWindow = getState().lockedWindow;
|
||||
if (!lockedWindow || !dragStart) return;
|
||||
if (getState().mode === 'drawing_annotation') {
|
||||
setState({ tempRect: clampedTo({ x: dragStart.x, y: dragStart.y, width: point.x - dragStart.x, height: point.y - dragStart.y }, lockedWindow.rect) });
|
||||
} else if (getState().mode === 'moving_annotation' && moving) {
|
||||
updateAnnotation(moving.id, moveClamped(moving.original, point.x - dragStart.x, point.y - dragStart.y, lockedWindow.rect));
|
||||
} else if (getState().mode === 'resizing_annotation' && resizing) {
|
||||
updateAnnotation(resizing.id, clampedTo(resizeRect(resizing.original, resizing.handle, point), lockedWindow.rect));
|
||||
}
|
||||
};
|
||||
|
||||
const onPointerUp = async (event: React.PointerEvent) => {
|
||||
if (event.button !== 0) return;
|
||||
const current = getState();
|
||||
if (current.mode === 'drawing_annotation' && current.tempRect && current.lockedWindow) {
|
||||
const rect = clampedTo(current.tempRect, current.lockedWindow.rect);
|
||||
dragStart = null;
|
||||
if (rect.width >= MIN_ANNOTATION_SIZE && rect.height >= MIN_ANNOTATION_SIZE) {
|
||||
const annotation: Annotation = { id: crypto.randomUUID(), rect, control_name: '', description: '', type: 'other', actions: [], feature_preview_path: null };
|
||||
setState((state) => ({ ...state, mode: 'editing_annotation', tempRect: null, selectedAnnotationId: annotation.id, project: state.project ? { ...state.project, annotations: [...state.project.annotations, annotation] } : { window: current.lockedWindow!, custom_window_name: '', window_description: '', annotations: [annotation] } }));
|
||||
await openFormNear(rect);
|
||||
} else {
|
||||
setState({ mode: 'drawing_annotation', tempRect: null });
|
||||
}
|
||||
return;
|
||||
}
|
||||
if ((current.mode === 'moving_annotation' && moving) || (current.mode === 'resizing_annotation' && resizing)) {
|
||||
const editedId = moving?.id ?? resizing?.id;
|
||||
moving = null; resizing = null; dragStart = null;
|
||||
setState({ mode: 'editing_annotation' });
|
||||
const edited = getState().project?.annotations.find((annotation) => annotation.id === editedId);
|
||||
if (edited) await openFormNear(edited.rect);
|
||||
}
|
||||
};
|
||||
|
||||
const localTarget = target ? toLocal(target.rect) : null;
|
||||
const annotations = state.project?.annotations ?? [];
|
||||
return (
|
||||
<div className="overlay-root" onPointerDown={onPointerDown} onPointerMove={onPointerMove} onPointerUp={onPointerUp}>
|
||||
{state.alertMessage ? <div className="warning" style={{ position: 'absolute', top: 12, left: 12, zIndex: 2 }}>{state.alertMessage}</div> : null}
|
||||
<svg width="100%" height="100%">
|
||||
<defs>
|
||||
<mask id="target-mask">
|
||||
<rect x="0" y="0" width="100%" height="100%" fill="white" />
|
||||
{localTarget ? <rect x={localTarget.x} y={localTarget.y} width={localTarget.width} height={localTarget.height} fill="black" /> : null}
|
||||
</mask>
|
||||
</defs>
|
||||
<rect x="0" y="0" width="100%" height="100%" fill="black" opacity="0.47" mask="url(#target-mask)" />
|
||||
{localTarget && target ? <>
|
||||
<rect x={localTarget.x} y={localTarget.y} width={localTarget.width} height={localTarget.height} fill="none" stroke="#007aff" strokeWidth="2" />
|
||||
<text x={localTarget.x + localTarget.width - 6} y={localTarget.y + 20} textAnchor="end" fill="white" fontSize="12">{target.title} [{target.rect.x}, {target.rect.y}, {target.rect.width}, {target.rect.height}]</text>
|
||||
</> : null}
|
||||
{annotations.map((annotation, index) => {
|
||||
const r = toLocal(annotation.rect);
|
||||
const selected = annotation.id === state.selectedAnnotationId;
|
||||
return <g key={annotation.id}>
|
||||
<rect x={r.x} y={r.y} width={r.width} height={r.height} fill="none" stroke="#ff9100" strokeWidth="2" />
|
||||
<rect x={r.x} y={r.y - 18} width="24" height="18" fill="#ff9100" opacity="0.9" />
|
||||
<text x={r.x + 12} y={r.y - 5} textAnchor="middle" fill="white" fontSize="12">{index + 1}</text>
|
||||
{selected ? (Object.values(handleRects(annotation.rect)).map((handle, handleIndex) => {
|
||||
const h = toLocal(handle);
|
||||
return <rect key={handleIndex} x={h.x} y={h.y} width={h.width} height={h.height} fill="white" stroke="#007aff" strokeWidth="1" />;
|
||||
})) : null}
|
||||
</g>;
|
||||
})}
|
||||
{state.tempRect ? (() => { const r = toLocal(state.tempRect); return <rect x={r.x} y={r.y} width={r.width} height={r.height} fill="none" stroke="#3399ff" strokeWidth="2" />; })() : null}
|
||||
</svg>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
54
desktop_mark/tauri/src/windows/Panel.tsx
Normal file
54
desktop_mark/tauri/src/windows/Panel.tsx
Normal file
@ -0,0 +1,54 @@
|
||||
import { invoke } from '@tauri-apps/api/core';
|
||||
import { WebviewWindow } from '@tauri-apps/api/webviewWindow';
|
||||
import type { ReactElement } from 'react';
|
||||
import { setState, useAppState } from '../state/appStore';
|
||||
import type { AnnotationProject } from '../types';
|
||||
|
||||
|
||||
async function showAnnotationForm(): Promise<void> {
|
||||
const form = await WebviewWindow.getByLabel('annotation_form');
|
||||
if (!form) throw new Error('窗口 annotation_form 尚未创建');
|
||||
await form.show();
|
||||
await form.setFocus();
|
||||
}
|
||||
async function saveProjectOrWarn(project: AnnotationProject): Promise<void> {
|
||||
try { await invoke('save_project', { project }); }
|
||||
catch (error) { setState({ alertMessage: `无法保存标注项目：${String(error)}` }); }
|
||||
}
|
||||
|
||||
export function Panel(): ReactElement {
|
||||
const state = useAppState();
|
||||
const project = state.project;
|
||||
const window = project?.window;
|
||||
return <div className="panel">
|
||||
{state.alertMessage ? <div className="warning">{state.alertMessage}</div> : null}
|
||||
<h2>窗口信息</h2>
|
||||
<label>标题</label><div className="value">{window?.title ?? '-'}</div>
|
||||
<label>进程</label><div className="value">{window?.process_name || '-'}</div>
|
||||
<label>BBox</label><div className="value">{window ? `${window.rect.x}, ${window.rect.y}, ${window.rect.width}, ${window.rect.height}` : '-'}</div>
|
||||
<label>窗口名称</label>
|
||||
<input value={project?.custom_window_name ?? ''} onChange={(event) => {
|
||||
if (!project) return;
|
||||
const next = { ...project, custom_window_name: event.target.value };
|
||||
setState({ project: next });
|
||||
void saveProjectOrWarn(next);
|
||||
}} />
|
||||
<label>窗口简介</label>
|
||||
<textarea value={project?.window_description ?? ''} onChange={(event) => {
|
||||
if (!project) return;
|
||||
const next = { ...project, window_description: event.target.value };
|
||||
setState({ project: next });
|
||||
void saveProjectOrWarn(next);
|
||||
}} />
|
||||
<h2>标注列表</h2>
|
||||
<div className="annotation-list">
|
||||
{(project?.annotations ?? []).map((annotation, index) => {
|
||||
const label = annotation.control_name || annotation.description || `标注 ${index + 1}`;
|
||||
return <button key={annotation.id} className={`annotation-item ${annotation.id === state.selectedAnnotationId ? 'selected' : ''}`} onClick={() => {
|
||||
setState({ selectedAnnotationId: annotation.id, mode: state.mode !== 'picking_window' && state.mode !== 'idle' ? 'editing_annotation' : state.mode });
|
||||
void showAnnotationForm().catch((error) => setState({ alertMessage: `无法打开标注弹窗：${String(error)}` }));
|
||||
}}>{index + 1}. {label} ({annotation.type})</button>;
|
||||
})}
|
||||
</div>
|
||||
</div>;
|
||||
}
|
||||
1
desktop_mark/tauri/tsconfig.json
Normal file
1
desktop_mark/tauri/tsconfig.json
Normal file
@ -0,0 +1 @@
|
||||
{"compilerOptions":{"target":"ES2022","useDefineForClassFields":true,"lib":["ES2022","DOM","DOM.Iterable"],"allowJs":false,"skipLibCheck":true,"esModuleInterop":true,"allowSyntheticDefaultImports":true,"strict":true,"forceConsistentCasingInFileNames":true,"module":"ESNext","moduleResolution":"Bundler","resolveJsonModule":true,"isolatedModules":true,"noEmit":true,"jsx":"react-jsx"},"include":["src"],"references":[]}
|
||||
17
desktop_mark/tauri/vite.config.ts
Normal file
17
desktop_mark/tauri/vite.config.ts
Normal file
@ -0,0 +1,17 @@
|
||||
import { defineConfig } from 'vite';
|
||||
import react from '@vitejs/plugin-react';
|
||||
|
||||
export default defineConfig({
|
||||
plugins: [react()],
|
||||
clearScreen: false,
|
||||
server: {
|
||||
strictPort: true,
|
||||
port: 1420,
|
||||
host: '127.0.0.1',
|
||||
watch: {
|
||||
ignored: ['**/src-tauri/target/**'],
|
||||
},
|
||||
},
|
||||
envPrefix: ['VITE_', 'TAURI_'],
|
||||
build: { target: 'es2022', minify: false },
|
||||
});
|
||||
7
desktop_rap/README.md
Normal file
7
desktop_rap/README.md
Normal file
@ -0,0 +1,7 @@
|
||||
# 这个目录做一些python|go|rust的rpa控制自动化实验
|
||||
|
||||
# 基于desktop_mark 标注工具已完成，并且已经有了一定的视觉节点树信息
|
||||
# 在这里测试Agent通过视觉节点树去控制电脑自动化执行
|
||||
# 包括使用豆包VLM模型进行内容识别，还有一些OCR识别，
|
||||
# 包括区域滚动、控件点击双击拖动等
|
||||
# agent 需要支持收到任务、读取记忆、理解需求、 执行、验证、总结经验教训、存储记忆、错误补标、重标。任务打断、打断恢复等
|
||||
10
test_ocr/py_ocr/app.py
Normal file
10
test_ocr/py_ocr/app.py
Normal file
@ -0,0 +1,10 @@
|
||||
from rapidocr import RapidOCR
|
||||
|
||||
engine = RapidOCR()
|
||||
|
||||
img_url = "history_wangzhen_long.png"
|
||||
result = engine(img_url)
|
||||
print(result)
|
||||
|
||||
# 可视化会触发程序自动下载所需字体文件。如果是离线环境，可注释掉该行
|
||||
# result.vis("vis_result.jpg")
|
||||
Loading…
x
Reference in New Issue
Block a user