Skip to content

[do not merge] improving + adding features to go client#19

Draft
elee1766 wants to merge 9 commits into
taskforcesh:mainfrom
elee1766:go-client
Draft

[do not merge] improving + adding features to go client#19
elee1766 wants to merge 9 commits into
taskforcesh:mainfrom
elee1766:go-client

Conversation

@elee1766

@elee1766 elee1766 commented Apr 26, 2024

Copy link
Copy Markdown

hi, im opening this pr just so you can see what im up to.

if you have any questions or wants, please bring them up

i will also ask questions to you if i have them in this PR, if that is okay? @manast

@elee1766

elee1766 commented Apr 26, 2024

Copy link
Copy Markdown
Author

still a skeleton, but i have placeholders for what i think is the http api?

some of the paths didnt work, and i had to change a ioredis setting as well for my test to pass, i'm not too sure why.

it complained that the maxretrycount for redis connection was null

since the basepath and token is already in the client, i felt it made sense to also use it to dial websocket so i added them, but i havent updated anything to use those functions yet.

also, for some reason, you were passing token with a query parameter, but i don't understand because i couldn't see anywhere in the ts where you were parsing the token as a query param (i only see the authorization header), maybe im missing something

@elee1766

elee1766 commented Apr 26, 2024

Copy link
Copy Markdown
Author

@manast ok i filled in a bunch of stuff, still not completely done, but let me know if this is on the right path to what you wanted.

i added context.Context to some of the existing wss based functions as well, a few are still missing proper ctx handling tho

@manast manast requested a review from roggervalf April 27, 2024 08:21
@manast

manast commented Apr 30, 2024

Copy link
Copy Markdown
Contributor

Sorry for the delay in reviewing this, I will do it asap-

@manast manast left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. I saw you spent some time on the WS version of the Proxy. I think it will be more important to focus on the HTTP version first. But still your changes are good to have as we will finalize the WS in the future as well.
Anyway, I wrote some comments. It would be great to have a couple more of tests for this new functionality.

Comment thread clients/golang/pkg/client/client.go Outdated
Comment thread clients/golang/pkg/client/client.go Outdated
Comment thread clients/golang/pkg/client/client.go Outdated
Comment thread clients/golang/pkg/client/client.go Outdated
_, err = c.httpClient.R().
SetBody(jobs).
SetResult(out).
ForceContentType("application/json").

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resty should be configured to use JSON by default so that it is not required to use ForceContentType here.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i wasn't sure about this because some endpoints seem to return an OK instead of "OK". since the first is not valid JSON i am worried.

Comment thread clients/golang/pkg/client/client.go Outdated
@@ -0,0 +1,26 @@
package proxyapi

type GetJobsResponse struct {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think maybe it is more standard to have one file per struct instead of having all in this generic file.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ive been splitting these out into files that make sense as ive been getting more familiar with the api.

Comment thread src/index.ts Outdated
if (config.redis.url) {
connection = new IORedis(config.redis.url, {
retryStrategy: () => 3000,
maxRetriesPerRequest: 20,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is strange because I think this is the default value (20) for maxRetriesPerRequest in IORedis.

@elee1766 elee1766 Apr 30, 2024

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes this was really odd. for some reason it crash on my computer when it is null. maybe it is my bun version?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I don't think so, what error do you get?

@elee1766

elee1766 commented Apr 30, 2024

Copy link
Copy Markdown
Author

@manast here is the error when i dont manually specify the ioredis setting. (it in fact tells me to set to not be null)

[1714502403939][INFO] BullMQ Proxy: Worker connected for queue test with concurrency 10
61 |         this.initializing.catch(err => this.emit('error', err));
62 |     }
63 |     checkBlockingOptions(msg, options, throwError = false) {
64 |         if (this.blocking && options && options.maxRetriesPerRequest) {
65 |             if (throwError) {
66 |                 throw new Error(msg);
                           ^
error: BullMQ: Your redis options maxRetriesPerRequest must be null.
      at checkBlockingOptions (/home/a/repo/github.com/taskforcesh/bullmq-proxy/node_modules/bullmq/dist/cjs/classes/redis-connection.js:66:23)
      at new RedisConnection (/home/a/repo/github.com/taskforcesh/bullmq-proxy/node_modules/bullmq/dist/cjs/classes/redis-connection.js:47:13)
      at new QueueBase (/home/a/repo/github.com/taskforcesh/bullmq-proxy/node_modules/bullmq/dist/cjs/classes/queue-base.js:35:27)
      at new Worker (/home/a/repo/github.com/taskforcesh/bullmq-proxy/node_modules/bullmq/dist/cjs/classes/worker.js:34:9)
      at /home/a/repo/github.com/taskforcesh/bullmq-proxy/src/controllers/websocket/worker.ts:27:20
      at openWorker (/home/a/repo/github.com/taskforcesh/bullmq-proxy/src/controllers/websocket/worker.ts:16:34)
      at open (/home/a/repo/github.com/taskforcesh/bullmq-proxy/src/proxy.ts:29:7)
      at /home/a/repo/github.com/taskforcesh/bullmq-proxy/src/fetch-handler.ts:73:9
      at /home/a/repo/github.com/taskforcesh/bullmq-proxy/src/fetch-handler.ts:38:49
error: script "start" exited with code 1

as for websocket - if it's not finalized yet then that's fine. I just saw some issues with the impl and so did some small fixes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants