Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5e08a2768b | ||
|
|
79a0c15214 | ||
|
|
5571cf1aea | ||
|
|
2eb33587fe |
No files matched your search
@@ -40,3 +40,10 @@ release/
|
||||
# 重装备份数据(含设备软件清单,不入库)
|
||||
../win-backup/
|
||||
win-backup/
|
||||
|
||||
# ---- 在建工程(用户 WIP,暂不入库;2026-10-07 保护性添加)----
|
||||
# 背景:发版脚本的 add -A 曾把这些未跟踪文件误提交到公开仓库,已移出跟踪。
|
||||
# 这些是皇上正在开发的 Rust 客户端与控制台方案,待其评审后再决定是否入库。
|
||||
/crates/
|
||||
/docs/console-plan.md
|
||||
/docs/ROADMAP.md
|
||||
@@ -81,7 +81,28 @@ OEM 符号键(`oem_plus`/`oem_comma`/`oem_period`… → 支持 `win+d`、`ctr
|
||||
|
||||
> MCP 侧同名工具为 `na_key`(`action: "press"`),支持同一套参数。
|
||||
|
||||
**36 项能力** · **32 个 MCP 工具** · **114 项单元测试** · **26 项端到端用例**(CI 在真实 Windows 上验证)
|
||||
## 等待条件 gui.await —— v1.6
|
||||
|
||||
GUI 操作从「按一下→睡几秒→截图碰运气」变成**可断言**流程:
|
||||
|
||||
```bash
|
||||
# 等界面元素出现(最多 30s)
|
||||
nodeagent --node win invoke gui.await --args '{"condition":"control","text":"完成","timeout_ms":30000}'
|
||||
# 等它消失
|
||||
nodeagent --node win invoke gui.await --args '{"condition":"control","text":"安装中","state":"absent"}'
|
||||
# 等窗口 / 进程 / 文件
|
||||
nodeagent --node win invoke gui.await --args '{"condition":"window","title":"安装程序","timeout_ms":10000}'
|
||||
nodeagent --node win invoke gui.await --args '{"condition":"process","process":"setup.exe"}'
|
||||
nodeagent --node win invoke gui.await --args '{"condition":"file","path":"C:\\log.txt"}'
|
||||
```
|
||||
|
||||
四类条件各自复用 `window.list` / `screen.find` / `process.list` / `fs.stat`,
|
||||
平台行为(Windows UIA/OCR、macOS Vision OCR)与那些能力完全一致,**只读低危**。
|
||||
轮询期单次异常不视为失败(启动中的程序常短暂报错),超时未命中才带回 `last_error`。
|
||||
|
||||
**典型闭环**:`await control(text:"立即安装")` → `mouse.click(x, y)` → `await control(text:"完成", timeout_ms:60000)` → `click` → `await absent(text:"安装中")`。
|
||||
|
||||
**43 项能力** · **32 个 MCP 工具** · **114 项单元测试** · **26 项端到端用例**(CI 在真实 Windows 上验证)
|
||||
|
||||
> 关键里程碑:**GUI 语义**(`window.list` + `screen.find`,UIA 找不到自动降级 OCR)
|
||||
> 让 AI 从「看得到画面但读不懂界面」变成「按名字取坐标点下去」。
|
||||
@@ -513,11 +534,12 @@ nodeagent/
|
||||
- [`docs/SECURITY.md`](docs/SECURITY.md) —— **安全模型与加固指南**(钥匙=root、证书指纹钉住、网段白名单、应急处置)
|
||||
- [`docs/VERSIONING.md`](docs/VERSIONING.md) —— **版本与兼容性契约**(升级顺序、什么算破坏性、发版流程、更新方式)
|
||||
- [`docs/REMOTE-LIMITS.md`](docs/REMOTE-LIMITS.md) —— 远程能力边界与真实故障案例(含杀软误拦)
|
||||
- [`docs/ROADMAP.md`](docs/ROADMAP.md) —— **当前进度盘点与下一步路线**(2026-10-07 基线:待办收尾 / 语义可断言 / 状态采样 / 安全残余)
|
||||
|
||||
- [产品需求文档(PRD)](./PRD.md)
|
||||
- [开发文档(DEVELOPMENT)](./docs/DEVELOPMENT.md) —— 协议细节、能力 schema、安全模型、各端实现指南
|
||||
- [**跨机接管的边界与局限**](./docs/REMOTE-LIMITS.md) —— 哪些问题能远程修、哪些必须人工,附真实故障诊断案例
|
||||
- [**实战复盘:短板与改进优先级**](./docs/RETROSPECTIVE.md) —— 基于真机使用的短板分析(GUI 语义 / 编码 / 自愈 / 可观测性)
|
||||
- [**实战复盘:短板与改进优先级**](./docs/RETROSPECTIVE.md) —— 基于真机使用的短板分析(GUI 语义 / 编码 / 自愈 / 可观测性)。**写于 2026-09-28(19 项能力时),多数 P0/P1 已解决 —— 最新状态见 [ROADMAP](./docs/ROADMAP.md) §3 对照表**
|
||||
|
||||
## 关键决策
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@nodeagent/agent",
|
||||
"version": "1.5.0",
|
||||
"version": "1.6.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
/**
|
||||
* gui.await —— 等待条件成立(v1.6)。
|
||||
*
|
||||
* ## 为什么需要它(复盘三大短板之「语义」)
|
||||
* 原来的 GUI 操作是**盲试**:按一下、截个图、看看有没有变化,不行再试。
|
||||
* 问题不是"找不到元素"(screen.find 已解决),而是**操作之后无法断言界面到了预期状态**。
|
||||
* 有了本能力,"点了安装 → 等待『完成』按钮出现" 从"睡 3 秒碰运气"变成**可验证的等待**:
|
||||
*
|
||||
* await control(text:"完成", timeout:30s) → click(found) → await absent(text:"安装中")
|
||||
*
|
||||
* ## 实现要点
|
||||
* 不写任何新的平台探测代码 —— 四个条件全部**复用既有能力**(window.list /
|
||||
* screen.find / process.list / fs.stat),因此 Windows/macOS 行为与它们完全一致。
|
||||
* 轮询期间异常一律视为"尚未命中"并记录 last_error,避免单次抖动整句失败;
|
||||
* 但超时未命中时会把它带回,方便判断是"真的没有"还是"每次都在报错"。
|
||||
*/
|
||||
|
||||
import { CapabilityError, ErrorCodes } from '@nodeagent/protocol';
|
||||
import { windowList } from './window.js';
|
||||
import { screenFind } from './window.js';
|
||||
import { processList } from './system.js';
|
||||
import { fsStat } from './fs.js';
|
||||
|
||||
type Args = Record<string, unknown>;
|
||||
|
||||
type Condition = 'window' | 'control' | 'process' | 'file';
|
||||
|
||||
const CONDITIONS: readonly Condition[] = ['window', 'control', 'process', 'file'];
|
||||
|
||||
/** 判断一次底层调用是否"命中"(兼容各能力不同的返回形态)。 */
|
||||
function isHit(result: unknown): boolean {
|
||||
if (result === null || result === undefined) return false;
|
||||
if (typeof result === 'boolean') return result;
|
||||
const r = result as Record<string, unknown>;
|
||||
if (r['found'] === true) return true;
|
||||
if (r['exists'] === true) return true;
|
||||
for (const key of ['windows', 'processes', 'matches', 'elements']) {
|
||||
const arr = r[key];
|
||||
if (Array.isArray(arr) && arr.length > 0) return true;
|
||||
}
|
||||
if (Array.isArray(result) && result.length > 0) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
/** 按条件组装底层调用的参数(只透传该条件支持的字段)。 */
|
||||
function buildInnerArgs(condition: Condition, args: Args): Args {
|
||||
const num = (k: string): number | undefined => {
|
||||
const v = args[k];
|
||||
return typeof v === 'number' && Number.isFinite(v) ? v : undefined;
|
||||
};
|
||||
switch (condition) {
|
||||
case 'window': {
|
||||
const inner: Args = {};
|
||||
const title = (args['title'] as string | undefined) ?? (args['pattern'] as string | undefined);
|
||||
if (title) inner['title_pattern'] = title;
|
||||
const limit = num('limit');
|
||||
if (limit !== undefined) inner['limit'] = limit;
|
||||
return inner;
|
||||
}
|
||||
case 'control': {
|
||||
// 透传 screen.find 的全部相关参数(method=image 时也一样走轮询)
|
||||
const inner: Args = {};
|
||||
for (const k of ['text', 'window', 'control_type', 'method', 'template', 'limit', 'timeout_ms']) {
|
||||
if (args[k] !== undefined) inner[k] = args[k];
|
||||
}
|
||||
return inner;
|
||||
}
|
||||
case 'process': {
|
||||
const inner: Args = {};
|
||||
const name = args['process'] as string | undefined;
|
||||
if (name) inner['filter'] = { name_pattern: name };
|
||||
const limit = num('limit');
|
||||
if (limit !== undefined) inner['limit'] = limit;
|
||||
return inner;
|
||||
}
|
||||
case 'file': {
|
||||
const p = args['path'] as string | undefined;
|
||||
if (!p) {
|
||||
throw new CapabilityError(ErrorCodes.PARAM_INVALID, 'condition=file 需要 path', {
|
||||
hint: '如 {"condition":"file","path":"C:\\\\Windows\\\\Temp\\\\setup.log"}',
|
||||
});
|
||||
}
|
||||
return { path: p };
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** 条件 → 底层能力(本进程内直接调用,省去 RPC 往返开销)。 */
|
||||
function innerProbe(condition: Condition): (args: Args) => Promise<unknown> {
|
||||
switch (condition) {
|
||||
case 'window':
|
||||
return (a) => windowList(a);
|
||||
case 'control':
|
||||
return (a) => screenFind(a);
|
||||
case 'process':
|
||||
return (a) => processList(a);
|
||||
case 'file':
|
||||
return (a) => fsStat(a);
|
||||
}
|
||||
}
|
||||
|
||||
export async function guiAwait(args: Args): Promise<unknown> {
|
||||
const condition = args['condition'] as Condition | undefined;
|
||||
if (!condition || !CONDITIONS.includes(condition)) {
|
||||
throw new CapabilityError(ErrorCodes.PARAM_INVALID, `不支持的 condition: ${String(condition)}`, {
|
||||
allowed: CONDITIONS,
|
||||
hint: 'window=等窗口出现/消失;control=等界面元素;process=等进程;file=等文件',
|
||||
});
|
||||
}
|
||||
const state = (args['state'] as string | undefined) ?? 'present';
|
||||
if (state !== 'present' && state !== 'absent') {
|
||||
throw new CapabilityError(ErrorCodes.PARAM_INVALID, `不支持的 state: ${state}`, {
|
||||
allowed: ['present', 'absent'],
|
||||
});
|
||||
}
|
||||
const timeoutMs = Math.max(0, Math.min(60_000, (args['timeout_ms'] as number | undefined) ?? 5000));
|
||||
const intervalMs = Math.max(50, Math.min(5000, (args['interval_ms'] as number | undefined) ?? 400));
|
||||
|
||||
const innerArgs = buildInnerArgs(condition, args);
|
||||
const probe = innerProbe(condition);
|
||||
|
||||
const startedAt = Date.now();
|
||||
let attempts = 0;
|
||||
let lastHit = false;
|
||||
let lastError: string | null = null;
|
||||
|
||||
for (;;) {
|
||||
attempts += 1;
|
||||
try {
|
||||
const result = await probe(innerArgs);
|
||||
lastHit = isHit(result);
|
||||
lastError = null;
|
||||
} catch (err) {
|
||||
// 轮询期异常不代表条件不成立(可能正忙于启动)——记下再试
|
||||
lastError = err instanceof Error ? err.message : String(err);
|
||||
lastHit = false;
|
||||
}
|
||||
const satisfied = state === 'present' ? lastHit : !lastHit;
|
||||
if (satisfied) {
|
||||
return {
|
||||
satisfied: true,
|
||||
condition,
|
||||
state,
|
||||
elapsed_ms: Date.now() - startedAt,
|
||||
attempts,
|
||||
last_seen: state,
|
||||
note:
|
||||
state === 'absent'
|
||||
? '条件已消失(连续一次未命中即视为消失;如需更稳可加大 interval_ms)'
|
||||
: undefined,
|
||||
};
|
||||
}
|
||||
if (Date.now() - startedAt >= timeoutMs) {
|
||||
return {
|
||||
satisfied: false,
|
||||
condition,
|
||||
state,
|
||||
elapsed_ms: Date.now() - startedAt,
|
||||
attempts,
|
||||
last_seen: lastHit ? 'present' : 'absent',
|
||||
last_error: lastError ?? undefined,
|
||||
note:
|
||||
state === 'present'
|
||||
? `等待 ${timeoutMs}ms 仍未出现${lastError ? `,期间持续报错:${lastError.slice(0, 160)}` : ''}`
|
||||
: `等待 ${timeoutMs}ms 仍未消失${lastError ? `,期间持续报错:${lastError.slice(0, 160)}` : ''}`,
|
||||
};
|
||||
}
|
||||
await new Promise((r) => setTimeout(r, intervalMs));
|
||||
}
|
||||
}
|
||||
@@ -13,6 +13,7 @@ import { audioGet, audioSet } from './audio.js';
|
||||
import { agentUpdate } from './selfupdate.js';
|
||||
import { mouseMove, mouseClick, mouseScroll, mouseDrag, keyType, keyPress, setInputPolicy } from './input.js';
|
||||
import { fsList, fsStat, fsRead, fsWrite, setFsRoots } from './fs.js';
|
||||
import { guiAwait } from './await.js';
|
||||
|
||||
export interface HandlerContext {
|
||||
/** 连接标识(事件订阅归属 / 断开清理) */
|
||||
@@ -52,6 +53,8 @@ export function createCapabilityRegistry(cfg: AgentConfig): Record<string, Capab
|
||||
[CapabilityNames.MouseDrag]: mouseDrag,
|
||||
[CapabilityNames.KeyType]: keyType,
|
||||
[CapabilityNames.KeyPress]: keyPress,
|
||||
// v1.6 等待条件(把盲试变为可断言)
|
||||
[CapabilityNames.GuiAwait]: guiAwait,
|
||||
// v3+ 审计
|
||||
[CapabilityNames.AuditList]: auditList,
|
||||
// v11 审计防篡改
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@nodeagent/cli",
|
||||
"version": "1.5.0",
|
||||
"version": "1.6.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@nodeagent/hub",
|
||||
"version": "1.5.0",
|
||||
"version": "1.6.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@nodeagent/mcp",
|
||||
"version": "1.5.0",
|
||||
"version": "1.6.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
# nodeagent 实战复盘:短板、瓶颈与改进优先级
|
||||
|
||||
> ⚠️ **本文写于 2026-09-28(当时 19 项能力),多数结论已过时。**
|
||||
> 2026-10-07 逐项核查:P0 全部解决、P1 仅「定时采样」未做、P2 已补大半。
|
||||
> **当前状态与下一步路线以 [`ROADMAP.md`](./ROADMAP.md) 为准**,本文保留作为「问题从哪来」的历史记录。
|
||||
|
||||
> 基于 2026-09-27 ~ 09-28 的真实使用(Windows 真机部署 + v2rayN 故障排查 + 重装前信息采集)
|
||||
> 所有问题均来自**实际踩坑**,非理论推演。
|
||||
|
||||
|
||||
+1
-1
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "nodeagent",
|
||||
"private": true,
|
||||
"version": "1.5.0",
|
||||
"version": "1.6.0",
|
||||
"description": "跨机 AI 接管框架 —— 让 Mac 上的 AI 无缝接管 Windows,装软件、查状态",
|
||||
"type": "module",
|
||||
"engines": {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@nodeagent/client",
|
||||
"version": "1.5.0",
|
||||
"version": "1.6.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@nodeagent/protocol",
|
||||
"version": "1.5.0",
|
||||
"version": "1.6.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
|
||||
@@ -488,4 +488,61 @@ export const GRAPHICS_CAPABILITIES: CapabilityDescriptor[] = [
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: CapabilityNames.GuiAwait,
|
||||
version: '1.0',
|
||||
description:
|
||||
'等待一个条件成立再返回(v1.6,**只读低危**)。四类条件:' +
|
||||
'window=等窗口出现/消失;control=等界面元素出现/消失(复用 screen.find);' +
|
||||
'process=等进程出现/消失;file=等文件出现/消失。' +
|
||||
'用途:把 GUI 操作从「按一下→睡几秒→截图碰运气」变成**可断言**的流程,例如:' +
|
||||
'await control(text:"完成", timeout_ms:30000) → mouse.click(found.x, found.y) → await state:absent(text:"安装中")。' +
|
||||
'state=present(默认)等到出现,absent 等到消失。' +
|
||||
'四个条件各自复用 window.list / screen.find / process.list / fs.stat,' +
|
||||
'平台行为(Windows UIA/OCR、macOS Vision OCR)与那些能力完全一致。' +
|
||||
'轮询期单次异常不视为失败(启动中的程序常短暂报错),只在超时未命中时带回 last_error。',
|
||||
risk: 'low',
|
||||
params_schema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
condition: {
|
||||
type: 'string',
|
||||
enum: ['window', 'control', 'process', 'file'],
|
||||
description: '等待哪类条件',
|
||||
},
|
||||
state: {
|
||||
type: 'string',
|
||||
enum: ['present', 'absent'],
|
||||
default: 'present',
|
||||
description: '等到出现(默认)还是等到消失',
|
||||
},
|
||||
timeout_ms: { type: 'integer', minimum: 0, maximum: 60000, default: 5000, description: '总超时' },
|
||||
interval_ms: { type: 'integer', minimum: 50, maximum: 5000, default: 400, description: '轮询间隔' },
|
||||
title: { type: 'string', description: 'window 条件:窗口标题(正则)' },
|
||||
pattern: { type: 'string', description: 'window 条件的 title 别名' },
|
||||
text: { type: 'string', description: 'control 条件:元素文本' },
|
||||
window: { type: 'string', description: 'control 条件:限定窗口标题(正则)' },
|
||||
control_type: { type: 'string', description: 'control 条件:限定控件类型(仅 UIA)' },
|
||||
method: { type: 'string', enum: ['auto', 'uia', 'ocr', 'image'], description: 'control 条件:查找方式' },
|
||||
template: { type: 'string', description: 'control 条件 + method=image 时的模板' },
|
||||
process: { type: 'string', description: 'process 条件:进程名(正则)' },
|
||||
path: { type: 'string', description: 'file 条件:文件路径' },
|
||||
limit: { type: 'integer', description: '底层 list 类条件的返回条数' },
|
||||
},
|
||||
additionalProperties: false,
|
||||
},
|
||||
returns_schema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
satisfied: { type: 'boolean' },
|
||||
condition: { type: 'string' },
|
||||
state: { type: 'string' },
|
||||
elapsed_ms: { type: 'integer' },
|
||||
attempts: { type: 'integer', description: '实际轮询次数' },
|
||||
last_seen: { type: 'string' },
|
||||
last_error: { type: 'string', description: '超时未命中时带回的末次异常(若有)' },
|
||||
note: { type: 'string' },
|
||||
},
|
||||
},
|
||||
},
|
||||
];
|
||||
@@ -18,6 +18,7 @@ export const CapabilityNames = {
|
||||
MouseDrag: 'input.mouse.drag',
|
||||
KeyType: 'input.key.type',
|
||||
KeyPress: 'input.key.press',
|
||||
GuiAwait: 'gui.await',
|
||||
// v3+ 审计
|
||||
AuditList: 'system.audit.list',
|
||||
// v5 文件传输
|
||||
|
||||
@@ -186,6 +186,20 @@ for (const a of assets) {
|
||||
console.log(` latest.json / SHA256SUMS`);
|
||||
|
||||
// ---------- ⑤ 提交 + 标签 + 推送 ----------
|
||||
// ⚠️ 安全闸:发版前若存在**未跟踪**文件,先停下来问,绝不盲目 add -A。
|
||||
// 2026-10-07 真机教训:工作区里有用户在建工程(crates/、docs/console-plan.md 等),
|
||||
// 发版脚本的 add -A 把它们连同版本号变更一起提交并推到公开仓库。
|
||||
const untracked = sh('git', ['ls-files', '--others', '--exclude-standard'])
|
||||
.split('\n')
|
||||
.filter((f) => f && !f.startsWith('.github/') && f !== '.github');
|
||||
if (untracked.length > 0) {
|
||||
fail(
|
||||
'发现未跟踪文件,发版已中止(避免把无关文件带入提交):\n' +
|
||||
untracked.map((f) => ` ${f}`).join('\n') +
|
||||
'\n请确认:这些文件是否应纳入版本库?若要,先 git add 并单独提交;' +
|
||||
'若不要,加入 .gitignore 后重试。',
|
||||
);
|
||||
}
|
||||
sh('git', ['add', '-A', '--', '.', ':(exclude).github']);
|
||||
sh('git', ['commit', '-m', `chore(release): v${version}`]);
|
||||
sh('git', ['tag', '-a', `v${version}`, '-m', `nodeagent v${version}`]);
|
||||
|
||||
@@ -63,7 +63,7 @@ test('清单:v7~v11 新能力均已登记', () => {
|
||||
for (const name of expected) {
|
||||
assert.ok(findCapability(name), `缺少 ${name}`);
|
||||
}
|
||||
assert.equal(CAPABILITY_MANIFEST.length, 42, '当前应有 42 项能力');
|
||||
assert.equal(CAPABILITY_MANIFEST.length, 43, '当前应有 43 项能力');
|
||||
});
|
||||
|
||||
// ---------- 参数校验(新能力) ----------
|
||||
|
||||
Reference in new issue
Block a user