Repository navigation
初筛和抽取换成华东师大开放平台的模型 - #113
Merged
Merged
Conversation
维护者要换用学校开放平台的模型。平台的 System One 接口和原来的一样;它的 Responses 接口收下 JSON Schema 却不按它输出,Chat Completions 的约束解码才生效,所以抽取和读图改走 Chat Completions。 - 初筛用 ecnu-decision,在标注集上重定门槛:零漏检最高 0.754,取 0.5,无关过滤率从 74.72% 升到 97.78% - 抽取、试抓和读图用 ecnu-max(就是 DeepSeek V4.1 Flash),客户端从 responses.py 换成 chat.py - 抽取评测按平台的 credits 估算费用 - 总体设计记下初筛的结果和抽取在平台上的复验
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
关联
部署内测前的准备,见路线图 #2。维护者决定初筛和抽取都换成华东师大开放平台的模型。
改了什么
平台的 System One 接口和原来的一样。它的 Responses 接口会收下 JSON Schema,但不按 Schema 输出(实测回了一句话,不是 JSON);Chat Completions 的约束解码才生效。所以抽取、试抓和读图都改走 Chat Completions。
ecnu-decision,在标注集上重定门槛,从 0.1 改成 0.5ecnu-max,也就是 DeepSeek V4.1 Flash。客户端从extraction/responses.py换成extraction/chat.py,图片改用image_url内容块怎么验证的
crawler/的./check.sh全部通过,351 个测试初筛:在 489 条标注上调用,免费,两种输入各 483 次,没有失败
原来的 SemIf 在门槛 0.1 时,召回率 100%、无关过滤率 74.72%。新门槛 0.5 比零漏检最高门槛低 0.254,比取最高门槛只多放 5 条无关通知。门槛 0.9 起开始漏。
抽取:标注集里的 50 条通知有 1 条已被源站删掉(404,它没有标注截止),用剩下的 49 条跑,低谷时段约 64 credits
那 1 次失败是平台的约束解码没守住 Schema 的
minItems,交了空的证据列表,被校验拦下了。线上遇到这种情况,这条通知这一轮算抽取失败,下一轮重试。读图:用真实代码读了一张生成的图片,原样读出,331 个输入 token
Codex 逐文件审过,报了 2 处,都已改:
--off-peak的说明还写着「官方」影响
SYSTEM_ONE_API_KEY和LLM_API_KEY要换成平台的 key。用量里记的模型名是平台返回的deepseek-flash和vllm-sr/d3-flash