The Big One – Logs have been saved from event

[tameTNT]

The app broke for everyone because a list was empty and we tried to access the 0th element. No idea how this happened but it was fixed simply by a restart of the web server (DB info was preserved)

[Lordfirespeed]

2025/11/02 14:20:36 [Recovery] 2025/11/02 - 14:20:36 panic recovered:
runtime error: index out of range [0] with length 0
/snap/go/10971/src/runtime/panic.go:115 (0x4427f3)
        goPanicIndex: panic(boundsError{x: int64(x), signed: true, y: y, code: boundsIndex})
/home/durhack/projects/durhack-jury/server/judging/comparisons.go:145 (0xb5157a)
        (*Comparisons).FindLeastCompared: curr += c.Arr[vId][pId]
/home/durhack/projects/durhack-jury/server/judging/judging_flow.go:124 (0xb521cf)
        PickNextProject: return comps.FindLeastCompared(items, judge.SeenProjects), nil
/home/durhack/projects/durhack-jury/server/router/judge.go:245 (0xb6013d)
        GetNextJudgeProject.func1: project, err = judging.PickNextProject(db, judge, ctx, comps)
/home/durhack/go/pkg/mod/go.mongodb.org/mongo-driver@v1.14.0/mongo/session.go:193 (0xaedbf1)
        (*sessionImpl).WithTransaction: res, err := fn(NewSessionContext(ctx, s))
/home/durhack/projects/durhack-jury/server/database/util.go:20 (0xaf9c6c)
        WithTransaction: _, err = session.WithTransaction(context.Background(), fn, txnOptions)
/home/durhack/projects/durhack-jury/server/router/judge.go:243 (0xb5f8fc)
        GetNextJudgeProject: err = database.WithTransaction(db, func(ctx mongo.SessionContext) (interface{}, error) {
/home/durhack/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0x9414ea)
        (*Context).Next: c.handlers[c.index](c)
/home/durhack/projects/durhack-jury/server/router/middleware.go:128 (0xb699c7)
        NewRouter.AuthoriseJudge.func7: ctx.Next()
/home/durhack/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0x9414ea)
        (*Context).Next: c.handlers[c.index](c)
/home/durhack/projects/durhack-jury/server/router/middleware.go:98 (0xb69644)
        NewRouter.Authenticate.func6: ctx.Next()
/home/durhack/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0x9414ea)
        (*Context).Next: c.handlers[c.index](c)
/home/durhack/go/pkg/mod/github.com/gin-contrib/sessions@v1.0.1/sessions.go:54 (0xb5dd04)
        NewRouter.Sessions.func5: c.Next()
/home/durhack/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0x9414ea)
        (*Context).Next: c.handlers[c.index](c)
/home/durhack/projects/durhack-jury/server/router/init.go:167 (0xb5de31)
        NewRouter.useVar.func4: ctx.Next()
/home/durhack/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0x9414ea)
        (*Context).Next: c.handlers[c.index](c)
/home/durhack/projects/durhack-jury/server/router/init.go:167 (0xb5de91)
        NewRouter.useVar.func3: ctx.Next()
/home/durhack/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0x9414ea)
        (*Context).Next: c.handlers[c.index](c)
/home/durhack/projects/durhack-jury/server/router/init.go:167 (0xb5def1)
        NewRouter.useVar.func2: ctx.Next()
/home/durhack/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0x94e0ee)
        (*Context).Next: c.handlers[c.index](c)
/home/durhack/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/recovery.go:102 (0x94e0db)
        CustomRecoveryWithWriter.func1: c.Next()
/home/durhack/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0x94d1dc)
        (*Context).Next: c.handlers[c.index](c)
/home/durhack/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/logger.go:240 (0x94d1c3)
        LoggerWithConfig.func1: c.Next()
/home/durhack/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/context.go:174 (0x94c76d)
        (*Context).Next: c.handlers[c.index](c)
/home/durhack/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:620 (0x94c407)
        (*Engine).handleHTTPRequest: c.Next()
/home/durhack/go/pkg/mod/github.com/gin-gonic/gin@v1.9.1/gin.go:576 (0x94bf4c)
        (*Engine).ServeHTTP: engine.handleHTTPRequest(c)
/snap/go/10971/src/net/http/server.go:3340 (0x70d1ed)
        serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)
/snap/go/10971/src/net/http/server.go:2109 (0x6ebbe4)
        (*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)
/snap/go/10971/src/runtime/asm_amd64.s:1693 (0x4837a0)
        goexit: BYTE    $0x90   // NOP

[Lordfirespeed]

File server/router/project.go, function DeleteProject, line 244:

// todo: also remove project from all judges' current lists so that they don't get an error

Also, DeleteProject does not invoke RemoveProjectFromComparison (server/judging/comparisons.go) - we need to consider whether it would be sensible to do this.

At approx. 13:10, the fatal panics started happening
At approx. 13:07 and 13:09 (resp.) projects 69074ec40011546ca227ed74 and 69074ec40011546ca227edc2 were deleted.

From the logs alone, it is impossible to tell for sure, but I think either (or both) of these projects had been judged by at least one judge.
From this point on, the functions UpdateProjectComparisonCount and FindLeastCompared always failed for those judges as the prevSeen parameter will contain a project that no longer exists.
UpdateProjectComparisonCount is a subroutine of the PUT /api/judge/score handler and thus submitting/updating project score is impossible (for those judges)
In specific scenarios, FindLeastCompared is used to select the next project for a judge to see, and thus this is also impossible (for those judges)