Skip to content

Commit 283f2d9

Browse files
ivonnyssenclaudeRReverser
authored
fix(client): handle unaligned ImageBytes response buffers (#19)
* fix(client): handle unaligned ImageBytes response buffers `ASCOMResult<ImageArray>::from_reqwest` reads the 44-byte `ImageBytesMetadata` header via `bytemuck::try_from_bytes` and the pixel payload via `bytemuck::try_cast_slice::<u8, T>`. Both APIs require the source `&[u8]` to be aligned to the target type's alignment (4 for `i32`, 2 for `i16`/`u16`). The `&[u8]` here is a slice of the HTTP response body — typically a `bytes::Bytes` chunk that reqwest assembled from one or more chunked reads — and its start pointer is not guaranteed to satisfy that alignment. When it doesn't, both calls return `PodCastError::TargetAlignmentGreaterAndInputNotAligned` and the client panics with `ASCOM error UNSPECIFIED: TargetAlignmentGreaterAndInputNotAligned` — intermittently, depending on the host allocator's runtime behaviour. Fix: use the unaligned-friendly equivalents. - Metadata: `bytemuck::pod_read_unaligned` reads the full struct via `ptr::read_unaligned`, no source-alignment requirement. - Pixel data: `bytemuck::pod_collect_to_vec` allocates a fresh `Vec<T>` (which IS aligned) and `memcpy`s the payload in. Adds an upfront length-modulo check to keep the existing slop-error semantics. Adds a regression test that constructs an `imagebytes` payload at each of the four leading-pad offsets (0..4) within a `Vec<u8>`, guaranteeing at least three of them place the body slice on a non-4-aligned start. The test fails on the previous code path (`try_from_bytes` rejects the unaligned metadata immediately) and passes after the fix. Closes #18. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: address clippy + Copilot feedback - cast_raw_data: keep the zero-copy fast path. Try `bytemuck::try_cast_slice::<u8, T>` first; only fall back to the copying `pod_collect_to_vec` when it returns `TargetAlignmentGreaterAndInputNotAligned`. Slop / length-mismatch errors propagate verbatim. Common (aligned) case keeps the pre-PR memory and copy profile; unaligned case still works. - Drop redundant `if !data.len().is_multiple_of(elem_size)` length check now that the slop error path is reached via try_cast_slice directly. - Metadata fix doc comment: refer to `align_of::<ImageBytesMetadata>()` instead of the hard-coded `align_of::<i32>() == 4` so it stays correct if the struct's layout grows a wider field. - Test helper doc comment: clarify the alignment guarantee — the `Vec<u8>` base allocation is high-aligned by the system allocator, so looping leading_pad over 0..4 covers all four mod-4 rotations and guarantees at least three unaligned cases. - Test fn signature: split into a `Result<()>`-returning helper (`check_one_offset`) using `eyre::ensure!` for validation and a panicking `#[test]` shell that surfaces the failing leading_pad in the panic message. Avoids clippy::panic_in_result_fn while keeping diagnostic context. - Replace `data.len() % elem_size != 0` with the `is_multiple_of` form (clippy::manual_is_multiple_of), superseded by the fast-path refactor above. - Clean up `as i32` casts in the test helper to `i32::from(...)` for the enum reprs and `i32::try_from(...)?` for the size_of/len conversions (clippy::cast_lossless / cast_possible_truncation). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(client): fuse cast_raw_data slow path into single-pass widen Replaces the slow path's two-memcpy bytes→Vec<T>→Vec<i32> staging with a single chunked pass that decodes each `size_of::<T>()`-byte chunk via a new `widen_from_le_chunk` trait method on `AsTransmissionElementType` and writes straight into the output `Vec<i32>`. Savings: - One full pixel-buffer memcpy on unaligned input. - Memory ceiling on the slow path drops from `data.len() + sizeof_output` to just `sizeof_output`. For a 32 MP `i32` image that's ~128 MB peak instead of ~256 MB. - Drops the `bytemuck::NoUninit` bound on the local function (was only required by `pod_collect_to_vec`). The fast (aligned) path is unchanged from the previous commit — `try_cast_slice` reinterprets in place, and we iterate-and-widen into the output. So the common case keeps the pre-PR memory and copy profile; the slow path is now strictly an improvement. `from_le_bytes` on `i32` lowers to the same instruction as a normal aligned `i32` load on little-endian targets, so the per-element loop in the slow path is essentially identical in cost to a bulk memcpy + widening loop. For the smaller types (`i16` / `u16` / `u8`) the upconversion was element-wise anyway, so the staging buffer was pure overhead. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: use as_chunks with a fixed-size &[u8; N] widening trait Address RReverser's review on the cast_raw_data unaligned fallback: swap chunks_exact for <[u8]>::as_chunks::<N>() and move the per-element little-endian widening onto a companion WidenLeChunk<const N> trait whose method takes a fixed-size &[u8; N]. This ties the byte-chunk width to the element type at the type level (i16/u16 => 2, i32 => 4, u8 => 1) and drops the per-pixel bounds checks; as_chunks' remainder replaces the manual is_multiple_of slop check while preserving the OutputSliceWouldHaveSlop guarantee. The aligned try_cast_slice fast path is unchanged. WidenLeChunk is gated behind #[cfg(feature = "client")] since it is only used by the client-gated cast_raw_data. Also reword a stale test doc comment that referenced pod_collect_to_vec. Verified: from_reqwest_handles_unaligned_imagebytes_slice passes; clippy -D warnings clean on --all-features and on the client,camera / server,camera feature combos. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * review: fold unaligned widening onto AsTransmissionElementType Per maintainer feedback, replace the separate `WidenLeChunk<const N>` trait with a single client-gated `widen_unaligned_le` method on the existing `AsTransmissionElementType` trait, and drop the const generic from `cast_raw_data` (call sites are now `cast_raw_data::<i16>` etc.). Each impl hard-codes `as_chunks::<N>()` to its own byte width, so the slow path still decodes fixed-size `&[u8; N]` chunks with no per-pixel bounds checks. The "obvious" form (an associated `const WIDTH` driving `&[u8; Self::WIDTH]` / `as_chunks::<{ T::WIDTH }>()`) needs the unstable `generic_const_exprs` feature, so the explicit-width method is the stable-toolchain realization of the same intent. No behavioural change: the aligned fast path is byte-for-byte identical, and the slow path preserves the exact error semantics (a non-empty `as_chunks` remainder -> `OutputSliceWouldHaveSlop`; only the alignment failure falls back; all other `PodCastError`s propagate verbatim). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * review: collapse unaligned widening into one default method Fold the four near-identical `widen_unaligned_le` impls into a single default method on `AsTransmissionElementType`, reading each pixel with `bytemuck::pod_read_unaligned` over `chunks_exact(size_of::<Self>())` instead of per-type `as_chunks::<N>` + `from_le_bytes`. The four impls shrink to just `const TYPE`. Drop the redundant `'static` supertrait (already implied by `AnyBitPattern`) and rename the method to `widen_unaligned`: the read is host-endian, and little-endianness is guaranteed by the crate's little-endian-only `compile_error!` guard rather than by this function. The one-time `chunks_exact` remainder check preserves the `OutputSliceWouldHaveSlop` guarantee and keeps the per-element read's length-mismatch panic unreachable, so the output `Vec<i32>` is still built in a single exact-capacity pass. Add a test covering sign/zero extension across all four element widths, which the i32-only regression test did not exercise. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Associate types with fixed-size arrays --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: Ingvar Stepanyan <me@rreverser.com>
1 parent e4a828b commit 283f2d9

3 files changed

Lines changed: 136 additions & 11 deletions

File tree

Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ axum = { version = "0.8.8", optional = true }
3636
bytemuck = { version = "1.24.0", features = [
3737
"derive",
3838
"extern_crate_std",
39+
"must_cast",
3940
], optional = true }
4041
bytes = { version = "1.10.1", optional = true }
4142
criterion = { version = "0.8.1", optional = true }

src/api/camera/image_array/client.rs

Lines changed: 128 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,15 @@
11
use super::{
2-
AsTransmissionElementType, ImageArray, ImageArrayRank, ImageBytesMetadata, ImageElementType,
3-
TransmissionElementType, COLOUR_AXIS, IMAGE_BYTES_TYPE,
2+
AsTransmissionElementType, COLOUR_AXIS, IMAGE_BYTES_TYPE, ImageArray, ImageArrayRank,
3+
ImageBytesMetadata, ImageElementType, TransmissionElementType,
44
};
55
use crate::client::{Response, ResponseTransaction, ResponseWithTransaction};
66
use crate::{ASCOMError, ASCOMErrorCode, ASCOMResult};
77
use bytemuck::PodCastError;
88
use mime::Mime;
99
use ndarray::{Array2, Array3};
1010
use num_enum::TryFromPrimitive;
11-
use serde::de::{DeserializeOwned, IgnoredAny, MapAccess, Visitor};
1211
use serde::Deserialize;
12+
use serde::de::{DeserializeOwned, IgnoredAny, MapAccess, Visitor};
1313
use serde_ndim::de::MakeNDim;
1414
use std::fmt::{self, Formatter};
1515

@@ -83,11 +83,39 @@ impl<'de> Deserialize<'de> for JsonImageArray {
8383
}
8484

8585
fn cast_raw_data<T: AsTransmissionElementType>(data: &[u8]) -> Result<Vec<i32>, PodCastError> {
86-
Ok(bytemuck::try_cast_slice::<u8, T>(data)?
87-
.iter()
88-
.copied()
89-
.map(T::into)
90-
.collect())
86+
// Fast path: if `data` is already aligned to `align_of::<T>()`,
87+
// `try_cast_slice` reinterprets in place and we iterate-and-widen
88+
// into the output `Vec<i32>`. This is the common case — most host
89+
// allocators hand out 16-aligned buffers so the body slice that
90+
// reqwest hands us satisfies a 4-byte alignment requirement.
91+
//
92+
// Slow path: the HTTP response body is a `bytes::Bytes` slice
93+
// from reqwest's chunked read, and its start pointer is not
94+
// *guaranteed* to be `align_of::<T>()`-aligned (4 bytes for
95+
// `i32`, 2 for `i16` / `u16`). When the fast cast fails with
96+
// `TargetAlignmentGreaterAndInputNotAligned`, hand the bytes to
97+
// `T::widen_unaligned`, which splits `data` into
98+
// `size_of::<T>()`-wide chunks and reads each one with
99+
// `bytemuck::pod_read_unaligned` straight into the output
100+
// `Vec<i32>` — one pass, no intermediate aligned `Vec<T>`. A
101+
// non-empty `chunks_exact` remainder reproduces the
102+
// `OutputSliceWouldHaveSlop` guarantee. Other cast errors
103+
// propagate verbatim — only the alignment failure falls back.
104+
match bytemuck::try_cast_slice::<u8, T>(data) {
105+
Ok(aligned) => Ok(aligned.iter().copied().map(T::into).collect()),
106+
Err(PodCastError::TargetAlignmentGreaterAndInputNotAligned) => {
107+
// Read as unaligned byte blobs of the same size as `T`.
108+
Ok(bytemuck::try_cast_slice::<u8, T::Bytes>(data)?
109+
.iter()
110+
.copied()
111+
// Then reinterpret each byte blob as `T`.
112+
.map(bytemuck::must_cast)
113+
// And convert to the target `i32`.
114+
.map(T::into)
115+
.collect())
116+
}
117+
Err(other) => Err(other),
118+
}
91119
}
92120

93121
impl Response for ASCOMResult<ImageArray> {
@@ -108,10 +136,17 @@ impl Response for ASCOMResult<ImageArray> {
108136
},
109137
});
110138
}
111-
let metadata = bytes
139+
let metadata_bytes = bytes
112140
.get(..size_of::<ImageBytesMetadata>())
113141
.ok_or_else(|| eyre::eyre!("not enough bytes to read image metadata"))?;
114-
let metadata = bytemuck::try_from_bytes::<ImageBytesMetadata>(metadata)?;
142+
// `bytemuck::try_from_bytes::<ImageBytesMetadata>` requires
143+
// the source slice to be aligned to
144+
// `align_of::<ImageBytesMetadata>()`. The body buffer's start
145+
// pointer is not guaranteed to satisfy that (see
146+
// `cast_raw_data` for the parallel issue);
147+
// `pod_read_unaligned` reads via `ptr::read_unaligned` and
148+
// works for any pointer.
149+
let metadata: ImageBytesMetadata = bytemuck::pod_read_unaligned(metadata_bytes);
115150
eyre::ensure!(
116151
metadata.metadata_version == 1_i32,
117152
"unsupported metadata version {}",
@@ -168,3 +203,86 @@ impl Response for ASCOMResult<ImageArray> {
168203
})
169204
}
170205
}
206+
207+
#[cfg(test)]
208+
mod tests {
209+
use super::*;
210+
211+
/// Build an `application/imagebytes` payload at the given byte
212+
/// offset within a `Vec<u8>`. The `Vec`'s base allocation is
213+
/// usually a high-alignment chunk from the system allocator, so
214+
/// looping `leading_pad` over `0..align_of::<i32>()` is sufficient
215+
/// to cover all four `mod 4` cases — at least three of those
216+
/// rotations land the body slice on a non-4-aligned start, which
217+
/// is what reproduces the upstream-pre-fix
218+
/// `TargetAlignmentGreaterAndInputNotAligned` path.
219+
fn imagebytes_payload_at_offset(
220+
leading_pad: usize,
221+
pixels: &[i32],
222+
) -> eyre::Result<(Vec<u8>, std::ops::Range<usize>)> {
223+
let metadata = ImageBytesMetadata {
224+
metadata_version: 1_i32,
225+
error_number: 0_i32,
226+
client_transaction_id: None,
227+
server_transaction_id: None,
228+
data_start: i32::try_from(size_of::<ImageBytesMetadata>())?,
229+
image_element_type: i32::from(TransmissionElementType::I32),
230+
transmission_element_type: i32::from(TransmissionElementType::I32),
231+
rank: i32::from(ImageArrayRank::Rank2),
232+
dimension_1: i32::try_from(pixels.len())?,
233+
dimension_2: 1_i32,
234+
dimension_3: 0_i32,
235+
};
236+
let mut buf = vec![0u8; leading_pad];
237+
let payload_start = buf.len();
238+
buf.extend_from_slice(bytemuck::bytes_of(&metadata));
239+
buf.extend_from_slice(bytemuck::cast_slice(pixels));
240+
let payload_end = buf.len();
241+
Ok((buf, payload_start..payload_end))
242+
}
243+
244+
/// Regression test for issue #18: parsing an `imagebytes` body
245+
/// whose slice start is not 4-byte aligned must succeed.
246+
/// Pre-fix this path failed with
247+
/// `bytemuck::PodCastError::TargetAlignmentGreaterAndInputNotAligned`
248+
/// because both the metadata `try_from_bytes` and the pixel
249+
/// `try_cast_slice` require source alignment. After the fix —
250+
/// metadata via `pod_read_unaligned`, and pixels via an aligned
251+
/// `try_cast_slice` fast path with a `pod_read_unaligned`
252+
/// fallback — the parse is alignment-independent.
253+
fn check_one_offset(leading_pad: usize, pixels: &[i32]) -> eyre::Result<()> {
254+
let (buf, range) = imagebytes_payload_at_offset(leading_pad, pixels)?;
255+
let slice = &buf[range];
256+
let mime: Mime = IMAGE_BYTES_TYPE.parse()?;
257+
let parsed = <ASCOMResult<ImageArray> as Response>::from_reqwest(mime, slice)?;
258+
let array = parsed
259+
.response
260+
.map_err(|e| eyre::eyre!("ascom error: {e:?}"))?;
261+
eyre::ensure!(
262+
array.shape() == [pixels.len(), 1_usize, 1_usize],
263+
"shape mismatch: {:?}",
264+
array.shape()
265+
);
266+
for (i, &p) in pixels.iter().enumerate() {
267+
let actual = array[[i, 0_usize, 0_usize]];
268+
eyre::ensure!(
269+
actual == p,
270+
"pixel {i} mismatch: got {actual}, expected {p}"
271+
);
272+
}
273+
Ok(())
274+
}
275+
276+
#[test]
277+
fn from_reqwest_handles_unaligned_imagebytes_slice() {
278+
let pixels: Vec<i32> = (0_i32..16_i32).collect();
279+
// Try every offset 0..4 — at least three of these rotations
280+
// land the body slice on a non-4-aligned start (regardless of
281+
// the `Vec<u8>` base alignment), which is the case that fails
282+
// pre-fix.
283+
for leading_pad in 0_usize..4_usize {
284+
check_one_offset(leading_pad, &pixels)
285+
.unwrap_or_else(|e| panic!("leading_pad={leading_pad}: {e:?}"));
286+
}
287+
}
288+
}

src/api/camera/image_array/mod.rs

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,24 +62,30 @@ pub(crate) enum ImageElementType {
6262
I32 = 2,
6363
}
6464

65-
trait AsTransmissionElementType: 'static + Into<i32> + AnyBitPattern {
65+
trait AsTransmissionElementType: Into<i32> + AnyBitPattern {
6666
const TYPE: TransmissionElementType;
67+
type Bytes: Pod;
6768
}
6869

6970
impl AsTransmissionElementType for i16 {
7071
const TYPE: TransmissionElementType = TransmissionElementType::I16;
72+
type Bytes = [u8; size_of::<Self>()];
7173
}
7274

7375
impl AsTransmissionElementType for i32 {
7476
const TYPE: TransmissionElementType = TransmissionElementType::I32;
77+
type Bytes = [u8; size_of::<Self>()];
7578
}
7679

7780
impl AsTransmissionElementType for u16 {
7881
const TYPE: TransmissionElementType = TransmissionElementType::U16;
82+
type Bytes = [u8; size_of::<Self>()];
7983
}
8084

8185
impl AsTransmissionElementType for u8 {
8286
const TYPE: TransmissionElementType = TransmissionElementType::U8;
87+
#[expect(clippy::use_self)]
88+
type Bytes = [u8; size_of::<Self>()];
8389
}
8490

8591
/// Image array.

0 commit comments

Comments
 (0)