Dynamic and Batch Support for Object Detector#29
Conversation
| let imageArray = NDArray(descriptor: imageDescriptor) | ||
| _ = try await function.run(inputs: [imageInputName: imageArray]) | ||
| let defaults = DetectionParameters() | ||
| let warmupShape = zip(imageDescriptor.shape, [1, 3, defaults.inputHeight, defaults.inputWidth]) |
There was a problem hiding this comment.
Here we warmup with default, not the shape the caller will actually use at inference time. We should check if this is expected -- we might be getting kernels compiled for the default shapes but inferences runs at a different shape which would be a different kernel, so we'd be wasting cost here
Maybe warmup() should accept the parameters you detect (DetectionParameters, or a BatchPlan?) so we compile for the right shape
There was a problem hiding this comment.
Updated to use the same shape during warmup
|
|
||
| let plan = try Self.planBatch( | ||
| expectedShape: expectedShape, | ||
| imageSizes: images.map { CGSize(width: $0.width, height: $0.height) }, |
There was a problem hiding this comment.
I am wondering if we use the width and height here? We seem to use .count for the batching, but H and W always come from expectedShape in static or parameters.inputHeight/inputWidth in dynamic
If you think this is expected, we should change function signature :)
If not, maybe we should use them?
There was a problem hiding this comment.
This is expected, so I changed the function signature to show we're just using count
kevchengcodes
force-pushed
the
obj_detr_dynamic
branch
from
27eb514 to
c87c099
Compare
2 participants
Summary
Enables the
ObjectDetectorto run dynamic models as well as static models withbatch_size > 1.Changes include resolving dynamic dimensions similar to how
llm-runnerdoes it, constructing a batched input tensor based on the list of input images, and post-processing each output.Models can have dynamic batch size, image input height/width. Note that batched images will be resized to a common size when constructing the (B,C,H,W) input tensor. Users can optionally input a custom input height/width for their dynamic batch.
Verification
Tested with YoloS:
New unit tests to validate batch support is working.