AgentStudio/bench_s_50_full.log at main · derekhearst/AgentStudio · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
$ bun --tsconfig-override=scripts/bench/tsconfig.json run scripts/bench/longmemeval/smoke.ts "--dataset=s" "--limit=50" "--maxSessions=50" --useRerank
[smoke] runId=lme_20260501203758_9l21 dataset=s limit=50 maxSessions=50

=== ingest: bun --tsconfig-override=scripts/bench/tsconfig.json run scripts/bench/longmemeval/ingest.ts --dataset=s --runId=lme_20260501203758_9l21 --limit=50 --maxSessions=50 ===
[db] Database ready
[ingest] dataset=s runId=lme_20260501203758_9l21 instances=50
[ingest] (1/50) e47becba sessions=50 drawers=516 eta=14357s
[memory] extractor returned unusable JSON, using fallback
[ingest] (2/50) 118b2229 sessions=45 drawers=485 eta=13907s
[ingest] (3/50) 51a45a95 sessions=50 drawers=529 eta=13926s
[ingest] (4/50) 58bf7951 sessions=50 drawers=509 eta=13731s
[ingest] (5/50) 1e043500 sessions=50 drawers=510 eta=13363s
[ingest] (6/50) c5e8278d sessions=44 drawers=469 eta=12852s
[ingest] (7/50) 6ade9755 sessions=50 drawers=505 eta=12519s
[ingest] (8/50) 6f9b354f sessions=50 drawers=500 eta=12273s
[ingest] (9/50) 58ef2f1c sessions=50 drawers=518 eta=12047s
[ingest] (10/50) f8c5f88b sessions=46 drawers=487 eta=11620s
[ingest] (11/50) 5d3d2817 sessions=50 drawers=475 eta=11238s
[ingest] (12/50) 7527f7e2 sessions=45 drawers=490 eta=10881s
[ingest] (13/50) c960da58 sessions=46 drawers=504 eta=10630s
[ingest] FAILED 3b6f954b 72 | 	const out: number[][] = new Array(texts.length)
73 |
74 | 	for (let start = 0; start < texts.length; start += MAX_BATCH) {
75 | 		const slice = texts.slice(start, start + MAX_BATCH)
76 | 		const result = await callEmbeddings(model, slice)
77 | 		for (const item of result.data) {
                          ^
TypeError: undefined is not an object (evaluating 'result.data')
      at embed (C:\Users\derek\Documents\Projects\AgentStudio\src\lib\memory\embeddings.server.ts:77:22)
      at mineSession (C:\Users\derek\Documents\Projects\AgentStudio\src\lib\memory\mining.server.ts:209:27)
      at ingestInstance (C:\Users\derek\Documents\Projects\AgentStudio\scripts\bench\longmemeval\ingest.ts:109:24)
      at main (C:\Users\derek\Documents\Projects\AgentStudio\scripts\bench\longmemeval\ingest.ts:129:40)
      at processTicksAndRejections (native:7:39)

[ingest] (14/50) 726462e0 sessions=48 drawers=501 eta=10925s
[ingest] (15/50) 94f70d80 sessions=49 drawers=510 eta=10496s
[ingest] (16/50) 66f24dbb sessions=50 drawers=507 eta=10118s
[ingest] (17/50) ad7109d1 sessions=44 drawers=466 eta=9705s
[ingest] (18/50) af8d2e46 sessions=50 drawers=495 eta=9417s
[ingest] (19/50) dccbc061 sessions=49 drawers=505 eta=9085s
[memory] extractor returned unusable JSON, using fallback
[ingest] (20/50) c8c3f81d sessions=50 drawers=463 eta=8733s
[memory] extractor returned unusable JSON, using fallback
[ingest] (21/50) 8ebdbe50 sessions=48 drawers=487 eta=8421s
[ingest] (22/50) 6b168ec8 sessions=50 drawers=482 eta=8124s
[ingest] (23/50) 75499fd8 sessions=50 drawers=468 eta=7784s
[memory] extractor returned unusable JSON, using fallback
[ingest] (24/50) 21436231 sessions=50 drawers=512 eta=7476s
[ingest] (25/50) 95bcc1c8 sessions=46 drawers=487 eta=7332s
[ingest] (26/50) 0862e8bf sessions=50 drawers=490 eta=7067s
[ingest] (27/50) 853b0a1d sessions=50 drawers=495 eta=6844s
[ingest] (28/50) a06e4cfe sessions=50 drawers=508 eta=6633s
[ingest] (29/50) 37d43f65 sessions=49 drawers=533 eta=6413s
[ingest] FAILED b86304ba 72 | 	}
73 |
74 | 	throw lastError ?? new Error('Embeddings request exhausted retries')
75 | }
76 |
77 | export async function embed(texts: string[], options: EmbedOptions = {}): Promise<number[][]> {
                          ^
TypeError: undefined is not an object (evaluating 'result.data')
      at embed (C:\Users\derek\Documents\Projects\AgentStudio\src\lib\memory\embeddings.server.ts:77:22)
      at mineSession (C:\Users\derek\Documents\Projects\AgentStudio\src\lib\memory\mining.server.ts:209:27)
      at ingestInstance (C:\Users\derek\Documents\Projects\AgentStudio\scripts\bench\longmemeval\ingest.ts:109:24)
      at main (C:\Users\derek\Documents\Projects\AgentStudio\scripts\bench\longmemeval\ingest.ts:129:40)
      at processTicksAndRejections (native:7:39)

[ingest] (30/50) d52b4f67 sessions=50 drawers=455 eta=6368s
[ingest] (31/50) 25e5aa4f sessions=45 drawers=506 eta=6080s
[ingest] (32/50) caf9ead2 sessions=43 drawers=466 eta=5799s
[ingest] (33/50) 8550ddae sessions=42 drawers=453 eta=5515s
[ingest] (34/50) 60d45044 sessions=50 drawers=533 eta=5272s
[memory] extractor returned unusable JSON, using fallback
[memory] extractor returned unusable JSON, using fallback
[ingest] (35/50) 3f1e9474 sessions=50 drawers=500 eta=4943s
[ingest] (36/50) 86b68151 sessions=47 drawers=485 eta=4626s
[memory] extractor call failed RequestTimeoutError: Request timed out: TimeoutError: The operation timed out.
      at C:\Users\derek\Documents\Projects\AgentStudio\node_modules\@openrouter\sdk\esm\lib\sdks.js:181:36
      at processTicksAndRejections (native:7:39)

[memory] extractor returned unusable JSON, using fallback
[ingest] (37/50) 577d4d32 sessions=46 drawers=487 eta=4416s
[ingest] (38/50) ec81a493 sessions=50 drawers=493 eta=4099s
[ingest] (39/50) 15745da0 sessions=47 drawers=474 eta=3755s
[ingest] (40/50) e01b8e2f sessions=50 drawers=497 eta=3413s
[ingest] (41/50) bc8a6e93 sessions=48 drawers=537 eta=3066s
[ingest] (42/50) ccb36322 sessions=46 drawers=471 eta=2728s
[ingest] (43/50) 001be529 sessions=50 drawers=504 eta=2406s
[ingest] (44/50) b320f3f8 sessions=46 drawers=463 eta=2069s
[ingest] FAILED 19b5f2b3 53 | }
54 |
55 | /** Slug a token down to lowercase + hyphen for stable AAAK fragments. */
56 | export function slugifyToken(value: string): string {
57 | 	return value
58 | 		.trim()
        ^
TypeError: value.trim is not a function. (In 'value.trim()', 'value.trim' is undefined)
      at slugifyToken (C:\Users\derek\Documents\Projects\AgentStudio\src\lib\memory\aaak.server.ts:58:4)
      at normaliseTagValues (C:\Users\derek\Documents\Projects\AgentStudio\src\lib\memory\aaak.server.ts:71:16)
      at encodeAaak (C:\Users\derek\Documents\Projects\AgentStudio\src\lib\memory\aaak.server.ts:83:18)
      at mineSession (C:\Users\derek\Documents\Projects\AgentStudio\src\lib\memory\mining.server.ts:225:16)
      at ingestInstance (C:\Users\derek\Documents\Projects\AgentStudio\scripts\bench\longmemeval\ingest.ts:109:24)
      at main (C:\Users\derek\Documents\Projects\AgentStudio\scripts\bench\longmemeval\ingest.ts:129:40)
      at processTicksAndRejections (native:7:39)

[ingest] (45/50) 4fd1909e sessions=50 drawers=500 eta=1735s
[ingest] (46/50) 545bd2b5 sessions=44 drawers=461 eta=1384s
[ingest] (47/50) 8a137a7f sessions=50 drawers=507 eta=1038s
[ingest] done in 16267.1s
Internal error: directory mismatch for directory "C:\Users\derek\Documents\Projects\AgentStudio\scripts\bench\tsconfig.json", fd 384[handle]. You don't need to do anything, but this indicates a bug.

=== retrieve: bun --tsconfig-override=scripts/bench/tsconfig.json run scripts/bench/longmemeval/retrieve.ts --dataset=s --runId=lme_20260501203758_9l21 --limit=50 --useRerank ===
[db] Database ready
[retrieve] dataset=s runId=lme_20260501203758_9l21 ΓåÆ C:\Users\derek\Documents\Projects\AgentStudio\retrieval_logs\lme_20260501203758_9l21.jsonl
[retrieve] (10/50) eta=105s
[retrieve] (20/50) eta=95s
[retrieve] (30/50) eta=58s
[retrieve] (40/50) eta=32s
[retrieve] (50/50) eta=0s
[retrieve] done in 148.8s ΓåÆ C:\Users\derek\Documents\Projects\AgentStudio\retrieval_logs\lme_20260501203758_9l21.jsonl
Internal error: directory mismatch for directory "C:\Users\derek\Documents\Projects\AgentStudio\scripts\bench\tsconfig.json", fd 384[handle]. You don't need to do anything, but this indicates a bug.

=== score-retrieval: bun --tsconfig-override=scripts/bench/tsconfig.json run scripts/bench/longmemeval/score-retrieval.ts --runId=lme_20260501203758_9l21 ===
[score-retrieval] 50 instances (abstention excluded)
Session-level metrics:
	recall_all@5 = 0.9600 (n=50)
	ndcg_any@5 = 0.9600 (n=50)
	recall_all@10 = 0.9600 (n=50)
	ndcg_any@10 = 0.9600 (n=50)
Internal error: directory mismatch for directory "C:\Users\derek\Documents\Projects\AgentStudio\scripts\bench\tsconfig.json", fd 172[handle]. You don't need to do anything, but this indicates a bug.

[smoke] OK ΓÇö runId=lme_20260501203758_9l21
Internal error: directory mismatch for directory "C:\Users\derek\Documents\Projects\AgentStudio\scripts\bench\tsconfig.json", fd 384[handle]. You don't need to do anything, but this indicates a bug.