feat: support OpenAI image edits#56
Conversation
|
CodeAnt AI is reviewing your PR. Thanks for using CodeAnt! 🎉We're free for open-source projects. if you're enjoying it, help us grow by sharing. Share on X · |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request expands the capabilities of the OpenAI image provider by enabling support for image editing workflows. It introduces logic to handle reference images via multipart requests and adds sophisticated size resolution to ensure compatibility with various OpenAI models and user-defined aspect ratios. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
codeant-ai
Bot
added
the
size:XL
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 116 |
| Duplication | 14 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request enhances the OpenAI image provider by adding support for image editing (edits) and dynamic resolution scaling based on aspect ratios. It introduces multipart form handling for reference images and includes comprehensive unit tests. Key feedback points include the use of non-standard API parameters that may cause errors, incorrect multipart field naming for image masks, and the need for stricter PNG format validation to comply with OpenAI's requirements. Additionally, recommendations were made to optimize memory usage for large file uploads using streaming and to improve code documentation for complex logic as per the style guide.
| if body.Quality != "" { | ||
| fields["quality"] = body.Quality | ||
| } | ||
| if body.InputFidelity != "" { | ||
| fields["input_fidelity"] = body.InputFidelity | ||
| } | ||
| if body.Background != "" { | ||
| fields["background"] = body.Background | ||
| } | ||
| if body.OutputFormat != "" { | ||
| fields["output_format"] = body.OutputFormat | ||
| } | ||
| if body.Moderation != "" { | ||
| fields["moderation"] = body.Moderation | ||
| } |
There was a problem hiding this comment.
这些字段（quality, input_fidelity, background, output_format, moderation）并非标准 OpenAI Image Edit API 的参数。如果此 Provider 旨在兼容官方 API，发送这些非标准参数会导致 400 错误。如果是为了支持特定的代理模型，建议在代码中增加注释说明或根据模型 ID 条件性地添加这些字段。
| for _, ref := range refs { | ||
| header := make(textproto.MIMEHeader) | ||
| header.Set("Content-Disposition", fmt.Sprintf(`form-data; name="image"; filename="%s"`, escapeMultipartFilename(ref.Name))) | ||
| header.Set("Content-Type", ref.MIME) | ||
| part, err := writer.CreatePart(header) | ||
| if err != nil { | ||
| return nil, nil, fmt.Errorf("构建 OpenAI Images Edit 图片字段失败: %w", err) | ||
| } | ||
| if _, err := part.Write(ref.Content); err != nil { | ||
| return nil, nil, fmt.Errorf("写入 OpenAI Images Edit 图片字段失败: %w", err) | ||
| } | ||
| } |
There was a problem hiding this comment.
OpenAI 的 /v1/images/edits 接口要求主图片字段名为 image，可选的遮罩字段名为 mask。当前逻辑将所有参考图都命名为 image，这不符合官方规范。如果提供了多张参考图，通常应将第一张作为 image，第二张作为 mask，并忽略多余的图片。
| for _, ref := range refs { | |
| header := make(textproto.MIMEHeader) | |
| header.Set("Content-Disposition", fmt.Sprintf(`form-data; name="image"; filename="%s"`, escapeMultipartFilename(ref.Name))) | |
| header.Set("Content-Type", ref.MIME) | |
| part, err := writer.CreatePart(header) | |
| if err != nil { | |
| return nil, nil, fmt.Errorf("构建 OpenAI Images Edit 图片字段失败: %w", err) | |
| } | |
| if _, err := part.Write(ref.Content); err != nil { | |
| return nil, nil, fmt.Errorf("写入 OpenAI Images Edit 图片字段失败: %w", err) | |
| } | |
| } | |
| for i, ref := range refs { | |
| fieldName := "image" | |
| if i == 1 { | |
| fieldName = "mask" | |
| } else if i > 1 { | |
| break // OpenAI Edit 仅支持一张图片和一个遮罩 | |
| } | |
| header := make(textproto.MIMEHeader) | |
| header.Set("Content-Disposition", fmt.Sprintf(`form-data; name="%s"; filename="%s"`, fieldName, escapeMultipartFilename(ref.Name))) | |
| header.Set("Content-Type", ref.MIME) | |
| part, err := writer.CreatePart(header) | |
| if err != nil { | |
| return nil, nil, fmt.Errorf("构建 OpenAI Images Edit 图片字段失败: %w", err) | |
| } | |
| if _, err := part.Write(ref.Content); err != nil { | |
| return nil, nil, fmt.Errorf("写入 OpenAI Images Edit 图片字段失败: %w", err) | |
| } | |
| } |
| if !strings.HasPrefix(mimeType, "image/") { | ||
| return nil, fmt.Errorf("第 %d 张参考图不是有效图片", idx+1) | ||
| } |
There was a problem hiding this comment.
OpenAI 的图片编辑接口（Edits）明确要求输入图片必须是 PNG 格式。当前代码仅检查是否为 image/ 前缀，允许了 JPEG、WebP 等格式，这会导致 API 调用失败。建议在此处严格校验 MIME 类型为 image/png。
| if !strings.HasPrefix(mimeType, "image/") { | |
| return nil, fmt.Errorf("第 %d 张参考图不是有效图片", idx+1) | |
| } | |
| if mimeType != "image/png" { | |
| return nil, fmt.Errorf("第 %d 张参考图不是有效的 PNG 图片，OpenAI Edit 仅支持 PNG 格式", idx+1) | |
| } |
| var payload bytes.Buffer | ||
| writer := multipart.NewWriter(&payload) |
There was a problem hiding this comment.
根据仓库规范第 68 条，大文件操作应注意内存使用，建议使用流式处理。当前实现使用 bytes.Buffer 缓存整个 multipart payload，在处理多张大图（单张限制 20MB）时会导致内存占用激增。建议改用 io.Pipe 结合 goroutine 进行流式写入。
References
- 大文件操作需注意内存使用(流式处理) (link)
| req.Header.Set("Content-Type", writer.FormDataContentType()) | ||
| req.Header.Set("Accept", "application/json") | ||
| req.Header.Set("Authorization", "Bearer "+strings.TrimSpace(p.config.APIKey)) | ||
| req.Header.Set("Connection", "close") |
There was a problem hiding this comment.
手动设置 Connection: close 会禁用 HTTP 连接复用，增加握手开销。由于 httpClient 的 Transport 已配置了 DisableKeepAlives: true，此处显式设置 Header 是多余的，且不利于性能优化。
![codeant-ai[bot]](https://avatars.githubusercontent.com/in/646884?s=60&v=4){kind=small}
| if size, _ := params["size"].(string); strings.TrimSpace(size) != "" { | ||
| return strings.TrimSpace(strings.ToLower(size)) |
There was a problem hiding this comment.
Suggestion: Explicit size values are returned as-is without checking model compatibility, so size="auto" (or any unsupported explicit size) will be sent to models like dall-e-2/dall-e-3 that require fixed dimensions, causing avoidable OpenAI 400 failures. Validate explicit sizes against the selected model or normalize invalid values before building the request body. [logic error]
Severity Level: Major ⚠️
- ❌ OpenAI-image GenerateHandler requests can fail on unsupported sizes.
- ⚠️ Image-edit requests may 400 when size conflicts with model.Steps of Reproduction ✅
1. A client calls the JSON image-generation endpoint `GenerateHandler` in
`backend/internal/api/handlers.go:1-7`, setting `provider` to `"openai-image"` and
including `params` with `"prompt": "..."` and an explicit `"size": "auto"` (or any
arbitrary dimension) as part of the request body.
2. Inside `GenerateHandler`, the provider is resolved
(`provider.GetProvider(req.Provider)` at `handlers.go:8-13`), `req.Params` is passed into
`ResolveModelID` for image purpose (`handlers.go:29-35`), and the resolved `model_id` is
added back into `req.Params["model_id"]` if non-empty (`handlers.go:36-38`).
3. `GenerateHandler` then calls `p.ValidateParams(req.Params)` (`handlers.go:40-43`). For
`OpenAIImageProvider`, `ValidateParams` in
`backend/internal/provider/openai_image.go:56-89` loads `size, _ :=
params["size"].(string)` (`openai_image.go:70`) and accepts `"auto"` or any `宽x高` value
because `isValidOpenAIImageSize` returns true for `"auto"` and general
`^[1-9][0-9]{1,4}x[1-9][0-9]{1,4}$` (`openai_image.go:63-69`), so the request passes
validation regardless of the actual model's supported sizes.
4. When the worker later executes the task, it calls `OpenAIImageProvider.Generate`
(`backend/internal/provider/openai_image.go:92-164`), which builds the body via
`buildImagesGenerationRequestBody` (`openai_image.go:166-199`).
`buildImagesGenerationRequestBody` calls `resolveOpenAIImageSize(modelID, params)`
(`openai_image.go:176 & 71-89`), and because `params["size"]` is non-empty,
`resolveOpenAIImageSize` immediately returns the explicit size unchanged
(`openai_image.go:467-468`). This `Size` field is then sent to `/images/generations` or
`/images/edits` (`openai_image.go:201-207` and `openai_image.go:280-285`), so if the
configured OpenAI model rejects `"auto"` or the arbitrary dimensions for that endpoint,
the API responds with HTTP 400 and the user's image request fails, even though the
provider could have normalized or constrained the size for that model.Fix in Cursor | Fix in VSCode Claude
(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** backend/internal/provider/openai_image.go
**Line:** 467:468
**Comment:**
*Logic Error: Explicit `size` values are returned as-is without checking model compatibility, so `size="auto"` (or any unsupported explicit size) will be sent to models like `dall-e-2`/`dall-e-3` that require fixed dimensions, causing avoidable OpenAI 400 failures. Validate explicit sizes against the selected model or normalize invalid values before building the request body.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
|
CodeAnt AI finished reviewing your PR. |
|
Handled the review feedback in 89142ab:
Validation: go test ./... |
|
Handled the latest CodeAnt size-compatibility finding in 726309d:
Validation: go test ./... |
|
Handled the remaining active review thread by adding a function-level Chinese comment for dynamic OpenAI image size calculation. Verified with |
|
CodeAnt AI is running the review. Thanks for using CodeAnt! 🎉We're free for open-source projects. if you're enjoying it, help us grow by sharing. Share on X · |
codeant-ai
Bot
added
size:XL
Sequence DiagramThis PR updates the OpenAI image provider to resolve image size based on model and aspect ratio and to route requests with reference images through the OpenAI image edits endpoint instead of rejecting them. sequenceDiagram
participant Client
participant OpenAIImageProvider
participant OpenAIImagesAPI
Client->>OpenAIImageProvider: Generate image with prompt, model and options
OpenAIImageProvider->>OpenAIImageProvider: Resolve image size by model and aspect ratio
OpenAIImageProvider->>OpenAIImageProvider: Collect and validate reference images
alt No reference images
OpenAIImageProvider->>OpenAIImagesAPI: Send JSON generations request
else With reference images
OpenAIImageProvider->>OpenAIImagesAPI: Send multipart edits request with prompt and references
end
OpenAIImagesAPI-->>OpenAIImageProvider: Return base64 encoded images
OpenAIImageProvider-->>Client: Return generated images
Generated by CodeAnt AI |
| reader, contentType := openAIImageEditBody(fields, refs) | ||
| req, buildErr := http.NewRequestWithContext(ctx, http.MethodPost, requestURL, reader) | ||
| if buildErr != nil { | ||
| return nil, fmt.Errorf("构建 OpenAI Images Edit 请求失败: %w", buildErr) |
There was a problem hiding this comment.
Suggestion: A multipart writer goroutine is started before request construction succeeds, but if http.NewRequestWithContext fails (for example due to an invalid configured base URL), the pipe reader is never closed and the goroutine can block indefinitely while writing. Close the reader on build failure (or build the multipart body synchronously) to avoid leaking goroutines/retry-attempt resources. [resource leak]
Severity Level: Major ⚠️
- ⚠️ Misconfigured API base leaks goroutines per edit attempt.
- ⚠️ Long-running service may accumulate blocked multipart writers.
- ⚠️ Retry logic amplifies leaks on repeated build failures.Steps of Reproduction ✅
1. Misconfigure the OpenAI Images API base URL in the `openai-image` provider
configuration so that it is not a valid URL (e.g., an invalid scheme or characters that
cause `http.NewRequestWithContext` to fail). This configuration is stored in
`model.ProviderConfig.APIBase` for provider `openai-image` (backend/scripts/seed.go:42-53
and openai_image.go:40-45).
2. Trigger an image edit by calling the HTTP endpoint `POST
/api/v1/tasks/generate-with-images` with provider `openai-image` and at least one
reference image, which flows through `GenerateWithImagesHandler`
(backend/internal/api/handlers.go:520-567) into a task whose `ProviderName` is
`openai-image`.
3. The worker executes the task and calls `OpenAIImageProvider.Generate`
(backend/internal/worker/pool.go:175; openai_image.go:81-117), which collects references
and invokes `doImagesEditRequest` because `len(refImages) > 0` (openai_image.go:96-117,
252-255).
4. In `doImagesEditRequest`, the retry helper `doRequestWithRetry` calls the closure that
first executes `openAIImageEditBody(fields, refs)` (openai_image.go:268-269).
`openAIImageEditBody` creates an `io.Pipe`, starts a goroutine that writes multipart data
via `writeOpenAIImageEditMultipart`, and returns the pipe reader
(openai_image.go:342-353).
5. Still inside the closure, `http.NewRequestWithContext` is called with the malformed
`requestURL` and the pipe `reader` (openai_image.go:270-272). Because the URL is invalid,
`NewRequestWithContext` returns an error, the closure returns that error up to
`doRequestWithRetry`, and no HTTP client ever reads from the pipe reader.
6. The writer goroutine started in `openAIImageEditBody` attempts to write to the pipe,
but since there is no consumer on the reader side, its first write blocks indefinitely.
The pipe is never closed, and the goroutine is leaked for each failed attempt, consuming
resources across retries and over the life of the process.Fix in Cursor | Fix in VSCode Claude
(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** backend/internal/provider/openai_image.go
**Line:** 269:272
**Comment:**
*Resource Leak: A multipart writer goroutine is started before request construction succeeds, but if `http.NewRequestWithContext` fails (for example due to an invalid configured base URL), the pipe reader is never closed and the goroutine can block indefinitely while writing. Close the reader on build failure (or build the multipart body synchronously) to avoid leaking goroutines/retry-attempt resources.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix| fieldName := "image" | ||
| if idx == 1 { | ||
| fieldName = "mask" |
There was a problem hiding this comment.
Suggestion: The second uploaded reference image is always sent as a mask part instead of another image part. Since callers only provide generic reference images (not a dedicated mask input), normal two-image edit requests will be misinterpreted as image+mask and can fail with upstream validation errors or produce incorrect edits. Send non-mask references as image fields, and only map to mask when a true mask parameter is explicitly provided. [api mismatch]
Severity Level: Critical 🚨
- ❌ Two-image edits misinterpret second image as mask.
- ⚠️ Upstream OpenAI edits may reject invalid mask.
- ⚠️ Users see confusing failures with dual reference uploads.Steps of Reproduction ✅
1. From the frontend, configure an image generation with provider `openai-image` and at
least two reference images so that `config.refFiles` has length ≥ 2
(frontend/src/hooks/useGenerate.ts:340-352 and 496-513 append each file as `refImages`
multipart parts).
2. Submit the request; the browser sends `POST /api/v1/tasks/generate-with-images` with
multipart form fields `prompt`, `provider=openai-image`, `model_id`, and two `refImages`
files (backend/cmd/server/main.go:277 routes to `GenerateWithImagesHandler`).
3. In `GenerateWithImagesHandler` (backend/internal/api/handlers.go:520-567), the server
parses the multipart body into `MultipartRequest.RefImages` and converts each file's bytes
into a `[]interface{}` slice `refImageBytes`, which is stored as
`taskParams["reference_images"]`.
4. The worker later executes the task and calls `OpenAIImageProvider.Generate` when
`ProviderName` is `openai-image` (backend/internal/worker/pool.go:175 and
backend/internal/provider/openai_image.go:48-81), which calls
`collectOpenAIImageReferences` and then `doImagesEditRequest` because `len(refImages) > 0`
(openai_image.go:81-117).
5. `doImagesEditRequest` builds the multipart body via `openAIImageEditBody` and
`writeOpenAIImageEditMultipart` (openai_image.go:252-255 and 342-349). In
`writeOpenAIImageEditMultipart` (openai_image.go:365-371), the first reference uses form
field `image`, but the second reference is always labeled `mask` (`fieldName := "image";
if idx == 1 { fieldName = "mask" }`), even though the UI and API only expose generic
`refImages`/`reference_images` with no mask-specific parameter.
6. As a result, a normal two-image edit request is interpreted by the upstream
`/v1/images/edits` API as `image` + `mask` instead of two independent references; if the
second file is not a proper transparency mask PNG, OpenAI may reject the request or apply
edits according to unintended masked regions.Fix in Cursor | Fix in VSCode Claude
(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** backend/internal/provider/openai_image.go
**Line:** 366:368
**Comment:**
*Api Mismatch: The second uploaded reference image is always sent as a `mask` part instead of another image part. Since callers only provide generic reference images (not a dedicated mask input), normal two-image edit requests will be misinterpreted as image+mask and can fail with upstream validation errors or produce incorrect edits. Send non-mask references as image fields, and only map to `mask` when a true mask parameter is explicitly provided.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix| } else if idx > 1 { | ||
| break |
There was a problem hiding this comment.
Suggestion: Reference images beyond the first two are silently dropped, so requests with 3+ references lose user input without any error. This breaks the "one or more reference images" behavior and leads to inconsistent results. Either include all supported references in the multipart payload or return a clear validation error when the limit is exceeded. [incomplete implementation]
Severity Level: Major ⚠️
- ⚠️ Third and later reference images are silently ignored.
- ⚠️ Multi-reference edits yield incomplete, confusing results.
- ⚠️ Behavior contradicts "one or more reference images" UX.Steps of Reproduction ✅
1. In the frontend, select three or more reference images so that `config.refFiles`
contains at least three entries and each is appended to the multipart request as
`refImages` (frontend/src/hooks/useGenerate.ts:496-513).
2. The browser sends `POST /api/v1/tasks/generate-with-images` with provider
`openai-image` and three `refImages` file parts; the handler `GenerateWithImagesHandler`
parses these into `MultipartRequest.RefImages` and then into `refImageBytes`
(backend/internal/api/handlers.go:520-567).
3. The handler builds `taskParams` including `"reference_images": refImageBytes`
(handlers.go:560-587), and the worker later invokes `OpenAIImageProvider.Generate` for
this task (backend/internal/worker/pool.go:175 and openai_image.go:81-117).
4. Inside `Generate`, `collectOpenAIImageReferences` converts all three entries in
`taskParams["reference_images"]` into a `[]openAIImageReference` of length 3
(openai_image.go:388-421), and `doImagesEditRequest` is called because `len(refImages) >
0` (openai_image.go:96-117, 252-255).
5. `writeOpenAIImageEditMultipart` iterates over `refs` (openai_image.go:365-371) and
writes the first reference as form field `image` and the second as `mask`. For `idx > 1`
(the third and later images), the loop hits `else if idx > 1 { break }`
(openai_image.go:369-370), so no additional parts are written and extra user-supplied
references are silently dropped.
6. The upstream `/v1/images/edits` endpoint therefore receives only one `image` and one
`mask` file even when the user provided three or more references, so some user input is
ignored without any validation error or warning.Fix in Cursor | Fix in VSCode Claude
(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** backend/internal/provider/openai_image.go
**Line:** 369:370
**Comment:**
*Incomplete Implementation: Reference images beyond the first two are silently dropped, so requests with 3+ references lose user input without any error. This breaks the "one or more reference images" behavior and leads to inconsistent results. Either include all supported references in the multipart payload or return a clear validation error when the limit is exceeded.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
|
CodeAnt AI finished running the review. Thanks for using CodeAnt! 🎉We're free for open-source projects. if you're enjoying it, help us grow by sharing. Share on X · |
1 participant
User description
Summary
Tests
Notes
CodeAnt-AI Description
Support image edits with reference photos and smarter image sizing
What Changed
widthxheightsizes and normalize unsupported sizes to a valid OpenAI optionImpact
✅ Image edits with reference photos✅ Fewer invalid size errors✅ Correct image sizing for more model and aspect ratio combinations🔄 Retrigger CodeAnt AI Review
Details
💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.