Conversation
Disk-saving and maintenance tooling for v0.3.2. vacuum/analyze: - `vacuum pipes` (pipe.vacuum(full=)) reclaims dead-tuple disk space: VACUUM/VACUUM FULL (Postgres family, autocommit), VACUUM on TimescaleDB hypertables (recurses chunks), OPTIMIZE TABLE (MySQL/MariaDB), ALTER TABLE REBUILD (MSSQL), whole-db VACUUM (SQLite). - `analyze pipes` (pipe.analyze()) refreshes planner statistics. - Mirrors the compress feature end-to-end: SQLConnector methods, Pipe methods, instance-connector defaults, API client + routes (protected table guard), and the `--full` CLI flag. get_pipe_size reports reclaimed bytes in a stats table. Native range partitioning (PostgreSQL/postgis): - Opt-in via parameters['hypertable']=True on a non-TimescaleDB pipe with a datetime column; partition width reuses verify.chunk_minutes. - Parent table created PARTITION BY RANGE(dt) with the partition column folded into a composite primary key; child partitions are pre-created in sync_pipe before insert, epoch-aligned for stable boundaries. - get_pipe_size sums the partition tree (parent holds no rows). - MySQL/MariaDB, MSSQL, and CockroachDB partitioning are staged next; dispatch is generic (PARTITIONABLE_FLAVORS, partition_by_column). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Builds on the PostgreSQL range partitioning: opt-in via parameters['hypertable']=True on a non-TimescaleDB pipe with a datetime column, width from verify.chunk_minutes. MySQL/MariaDB use PARTITION BY RANGE COLUMNS(dt) on the DATETIME column (timezone-naive literals). Since an empty RANGE-partitioned table is invalid, the initial partitions are declared inline at CREATE TABLE (computed from the creation dataframe); later syncs append partitions via ALTER TABLE ADD PARTITION, walking the interval grid from the highest existing boundary (read from information_schema.PARTITIONS) up to the new data's max. Values below the highest boundary already land in an existing partition. The partition column is folded into the primary key (primary-key-first ordering so AUTO_INCREMENT stays the first key column). get_create_table_queries gains a partition_bounds parameter; the _partition module dispatches per family (PG_PARTITION_FLAVORS vs MYSQL_PARTITION_FLAVORS). Verified end-to-end on mysql and mariadb (create, upward extension, upsert, size, non-partitioned regression); test_sync.py passes on mariadb. MSSQL and CockroachDB partitioning remain staged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes native range partitioning across the non-TimescaleDB SQL families. Opt-in via parameters['hypertable']=True on an MSSQL pipe with a datetime column. MSSQL partitions via database-level objects: a partition function (RANGE RIGHT on DATETIMEOFFSET/DATETIME2) and a partition scheme are created (idempotently) before the table, and the table's clustered primary key is placed on the scheme with the partition column as the leading key (this is what partitions the storage). Later syncs extend the function with ALTER PARTITION SCHEME ... NEXT USED + ALTER PARTITION FUNCTION ... SPLIT RANGE, walking the epoch-aligned grid from the highest existing boundary (read from sys.partition_range_values) to the new data's max. drop_pipe drops the scheme then the function so the objects don't outlive the table and block re-creation. get_create_table_queries gains partition_scheme_name; the _partition module dispatches per family. Verified end-to-end on mssql (create, SPLIT-based extension, upsert, size via dm_db_partition_stats, drop cleanup, non-partitioned regression); test_sync.py passes on mssql. CockroachDB partitioning (enterprise-gated, distinct syntax) remains the only staged flavor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
get_chunk_bounds() gains an `align` parameter. When True, interior chunk boundaries snap to the same fixed Unix-epoch grid used by native range partitioning (SQLConnector._partition_bounds) instead of being anchored to `begin`; the first/last chunk edges are still clamped to begin/end. This makes verify chunks coincide with a pipe's partitions and stay deterministic across re-syncs regardless of the requested begin. Default is False (preserves the begin-anchored behavior for data iteration and everything else); Pipe.verify() opts in with align=True. Verified the aligned interior bounds match the partition grid; test_sync.py passes on sqlite and timescaledb. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`get_parameters()` resolves reference pipes and `{{ Pipe() }}` / `MRSM{}`
symlinks on every call. It is a hot path (hit by `.dtypes`, `.columns`,
`.precision`, etc.) and the reference walk can build connectors and incur
cold-connection latency, e.g. minutes of stalls while building compress
queries on a pipe with references.
Memoize the resolved result in memory, keyed on the identity of the raw
`_attributes['parameters']` dict. Every mutation path (`update_parameters`,
the setter, `edit`) reassigns that dict to a new object, so identity change
is a reliable invalidation signal. `refresh=True` drops the memo explicitly.
Only the top-level, symlink-resolving, non-refresh call is memoized; the
result is deep-copied on return so callers that mutate it (e.g.
`infer_dtypes(persist=True)`) cannot corrupt the cache, and `_symlinks` is
stashed and restored on hits.
Schema freshness is unaffected: dynamic-schema dtypes are handled separately
by `get_columns_types()`' TTL cache, so dynamic pipes stay correct.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The dbc_dark.css form-control, dash-dropdown, and dash-options-list rule blocks were unscoped, so their `background-color: var(--bs-dark) !important` declarations leaked onto every Dash app served by the API — forcing any plugin (e.g. the Swamp Rabbit Analytics portal) to fight each one with a higher-specificity `!important` override. Scope those blocks under `.dbc_dark` so the dark theme is opt-in, and set `class="dbc_dark"` on <body> by default so the console — including its body-portaled dropdown menus, which can't be reached by a descendant `.dbc_dark` selector otherwise — keeps its current look. A plugin page can now remove the class from <body> to render cleanly without `!important`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dash console quick wins, no restructuring: - Fix duplicate invalidate_token_click name (delete-modal opener); drop dead table_header line and unused get_web_connector import in tokens.py - Add title tooltips to icon-only buttons and alt text to logo/banner images - Wrap jobs/tokens/dashboard content in dcc.Loading spinners - Gate job deletion behind a confirmation modal (start/stop/pause unchanged) - Surface real errors: CSV-download failures logged via warn; accordion columns/recent-data handlers include the exception text Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Remove the content-div-right dcc.Loading: update_content writes both the webterm and content divs in one callback, so the spinner fired during the slow webterm startup. Jobs/tokens loaders kept. - Tokens table: responsive=True for horizontal scroll on small screens. - Add a mobile-only Termux-style extra-keys row above the webterm (ESC/CTRL/SHIFT/TAB/arrows). Arrows/ESC/TAB post raw escape sequences to the terminal iframe; CTRL/SHIFT are sticky modifiers applied to the next key via a capture-phase keydown handler in termpage.html, with button highlight cleared on use. Hidden on md+ (physical keyboard). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Daemon.__getstate__ pickled self.target (always `entry` for jobs) with dill
by value, serializing the target's entire global graph. That graph can reach
live, unpicklable state — e.g. a connector caching an `_asyncio.Task` — raising
"TypeError: cannot pickle '_asyncio.Task' object" when starting a job.
dill only pickles a function by reference when it can match it by identity;
with two importable copies of meerschaum (a stale install shadowing a venv) the
identity check fails and dill falls back to by-value, so byref=True alone does
not help.
Store importable, top-level targets by {module, qualname} reference and
re-import them in the daemon process instead. Closures/lambdas (run_daemon)
still pickle via dill. Backward-compatible: pickles without target_ref use the
existing dill path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
update_content unpacked exactly two values from each trigger handler, but get_plugins_cards returns four (cards, alerts, total_pages, total_count) for pagination, raising "ValueError: too many values to unpack (expected 2, got 4)" when clicking Plugins. Take the first two values so trailing extras are ignored. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Higher contrast: light outline buttons instead of dark secondary text. - Smaller font (0.7rem) so labels fit. - Lay the keys out as two rows of four on the top-left (4-col grid), balancing the terminal controls on the top-right; controls pinned right via margin-left:auto. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Keep the soft keyboard open when tapping a key button: preventDefault on pointerdown stops the button from stealing focus from the terminal textarea. - Move ESC/CTRL/SHIFT/TAB to a row along the bottom, right above the webterm, with a buffer above; controls stay top-right. - Move the arrow keys to the right, stacked like a real keyboard (↑ on top, ← ↓ → below) via a CSS grid, filling the space up to the controls. CTRL/SHIFT already toggle off when clicked again. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
They were full-width and stacked in a column. Use a 2-column grid (3.6em columns, a touch wider than the arrow keys) so they sit two-up, two-down. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ESC/CTRL/SHIFT/TAB grid and the arrow keys sit on the left (small gap between them); the refresh/fullscreen/new-tab controls move onto the same row, pinned right via margin-left:auto. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fullscreen toggle replaced content-col-right's className with a bare col-12/col-6, clobbering its responsive classes (col-md-12 col-lg-6) and inferring state from leftCol's display, which could leave the column at the wrong width. The embedded terminal also never refit, keeping its old narrow column count. Track fullscreen state on a data attribute, drive width via inline flex/width styles (preserving the responsive classes), and dispatch a resize on the iframe so the terminal refits to the new width. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Explain the two dark layers (global Darkly base + class-scoped dbc_dark) under the @web_page decorator section, and show how a plugin opts out of dbc_dark. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Plugin pages inherit the console's dbc_dark component theme by default. A page can now opt out declaratively with @web_page(dark_theme=False) instead of hand-injecting JS to toggle the body class. The Web Console tracks opt-out endpoints and toggles the `dbc_dark` class on <body> per route (removing it on opted-out pages, restoring it elsewhere, including on in-app navigation) via a dbc-dark-store + clientside callback. Opt-out is per-page because the body class is route-scoped; there is no plugin-wide switch. Docs: document the two theme layers and the kwarg under @web_page. Changelog updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Previously dark_theme=False only dropped the dbc_dark overrides, leaving the global Darkly base (dark background) in place — the name over-promised. Now both the dark (Bootswatch Darkly) and light (Flatly, v5.1.0 to match) Bootstrap themes are loaded, with exactly one enabled per route. An opted-out page enables Flatly and drops the dbc_dark class; the console and all other pages stay on Darkly. The light sheet is disabled before first paint to avoid a flash. Vendors bootstrap_light.min.css (Bootswatch Flatly 5.1.0, MIT). Docs + changelog updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The offcanvas nav buttons forced background-color: var(--bs-dark), which showed a jarring dark box on the light theme and a tight teal-gray box (smaller than the row) on hover in dark. Make the buttons transparent with theme-aware text (inherit), and move the hover highlight to the full list-group-item row using a subtle neutral color that reads well in both themes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- The previous sidebar restyle zeroed list-group-item padding/borders globally, shrinking rows and dropping borders. Restore the original structure (keep Bootstrap padding/borders, re-add the accordion body padding rule) while keeping the color changes: transparent buttons, theme-aware text, subtle full-row hover. - Give each plugin AccordionItem an explicit item_id. Without it, dbc's accordion read item_id off an undefined item (TypeError: can't access property "item_id"), tripping React's error boundary and remounting the offcanvas — which closed it and required a second click to open. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- The always_open Accordion had no active_item, so dbc called .join on an undefined value and read item_id off nothing (TypeError on render, seen when navigating). Pass a list active_item (the group holding the current page, else empty) so the value is always defined. - Style the accordion group headers (e.g. "Settings") like the other nav links — transparent and theme-aware — instead of a dark box. - Highlight the current page with a slightly darker "selected" shade and expand its accordion group. build_pages_offcanvas_children now takes the active path (passed from the toggle callback via mrsm-location.pathname). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The page-nav accordion still crashed on navigation (TypeError: item_id of undefined). always_open=True is the trigger — dbc reads item_id/.join off an undefined active_item during the mount/unmount that navigation causes. Mirror the pipe accordion that never crashes: drop always_open/active_item, give the Accordion an id, keep item_id on items. Also make the accordion container/item/button transparent so the Settings group isn't darker than the other items on the dark theme. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pages offcanvas lived inside pages_navbar, which is part of every page's layout in page-layout-div — so navigating destroyed and recreated the offcanvas (and its dbc accordion) each time, triggering "TypeError: can't access property item_id" during the unmount/remount. Hoist the single offcanvas into the persistent top-level app layout so it is created once and survives navigation; remove it from pages_navbar. The logo in the navbars still toggles it via the existing callback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add mrsm-location.pathname as a trigger to the offcanvas callback: navigation (selecting a page) closes the sidebar, while the logo still toggles it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hoisting the offcanvas to the top-level layout left the old per-page copies in the web-console layout (pages/dashboard.py) and the pipes navbar (pipes.py), so those pages rendered two components with id "pages-offcanvas". The duplicate id left the top-level offcanvas dead: pages with a local copy (web console, pipes) still toggled theirs, but pages_navbar pages (jobs, tokens, plugins, password-reset), which rely on the top-level one, stopped responding to the logo. Keep only the single top-level offcanvas. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The combined callback fired on mrsm-location.pathname (initial load + every navigation) while also declaring Input logo-img.n_clicks. On the initial load the routed page isn't rendered yet, so logo-img doesn't exist — Dash raised "A nonexistent object was used in an Input ... logo-img". Move close-on-navigation into its own callback that only depends on the pathname; the logo toggle keeps its own callback. MultiplexerTransform allows both to output is_open. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The page navbar (pages_navbar) and its logo/sign-out were module-level singletons reused across every page that includes them. Reusing one component object in multiple layouts is a Dash anti-pattern: React keeps the shared subtree mounted and mis-reconciles its siblings on navigation, so a previous page's components linger in the DOM. A plugin page whose URL callback lacks a pathname guard then repaints into its lingering div — e.g. the test plugin's H1 showing on the password-reset page. Convert the navbar pieces to factories (build_pages_navbar / build_logo_row / build_sign_out_button) and call them fresh at every page/use site, so navigation fully remounts page subtrees and stale components are torn down. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…itch Building fresh navbar objects wasn't enough: React reconciles by type + position (not Dash id), so navigating between two pages with the same top-level shape ([Div, Container]) reused the existing DOM nodes and only patched props — leaving the previous page's subtree (e.g. the test plugin's test-location/test-output-div) mounted, so its unguarded URL callback kept repainting its header onto the next page. Wrap the routed layout in a path-keyed container so React fully unmounts the old page and mounts the new one. Because the navbar (and its logo) now remount on navigation, the logo's n_clicks resets and re-fires the offcanvas toggle; return no_update for is_open there so it doesn't fight the close-on-navigation callback and re-open the sidebar (the glitch seen navigating to Jobs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove delay_hide=1000 on the jobs page dcc.Loading so the spinner no longer lingers a full second after content is ready, and bump the refresh-jobs-interval from 1s to 3s to cut background churn. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove delay_hide=1000 on the pipes and tokens page dcc.Loading components, matching the jobs page fix, so the spinner no longer lingers a full second after content is ready. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the dcc.Loading wrapper entirely on the jobs and tokens pages (and the now-unused dcc import in tokens.py) so content paints as soon as the page-local render callback returns. Add api/dash/README.md documenting the hand-rolled routing model (_paths router, per-page lazy Location rendering, the page-content key), refresh intervals, and the delay_hide spinner pitfall for future agents. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v3.4.1
Scope the
dbc_darktheme so it no longer leaks onto every Dash page.Several rule blocks in
dbc_dark.css(.form-control, the.dash-dropdown-*family, and.dash-options-list) were unscoped, so theirbackground-color: var(--bs-dark) !importantdeclarations applied to every Dash app served by the API — forcing plugins to fight each one with higher-specificity!importantoverrides. These blocks are now scoped under.dbc_dark, making the dark theme opt-in.To keep the web console (including its body-portaled dropdown menus) unchanged, the Dash app now sets
class="dbc_dark"on<body>by default. A plugin page can remove the class from<body>to render cleanly without the theme's!importantoverrides.Fix starting jobs whose target reaches unpicklable state (e.g. an
asyncio.Task).Starting a job (e.g.
sync pipes ... --loop --name foo) could fail withTypeError: cannot pickle '_asyncio.Task' objectwhen a connector cached a live async task. The daemon pickled its target function by value, dragging in the target's global graph; that walk reached the unpicklable state. Importable, top-level targets are now pickled by reference (re-imported in the daemon process), so the global graph is never serialized. Closures still pickle by value, unchanged.Add a Termux-style on-screen key row to the web console terminal (mobile).
On small screens the Webterm now shows a row of keys above the terminal —
ESC,CTRL,SHIFT,TAB, and arrow keys laid out like a physical keyboard.CTRL/SHIFTare sticky modifiers that apply to the next keypress (tap again to toggle off), and tapping a key keeps the soft keyboard open. The row is hidden on larger screens with a physical keyboard.Web console UX and accessibility improvements.
alttext on the logo and banner images.Fix the web console fullscreen terminal rendering at half width.
Toggling the Webterm's fullscreen button clobbered the column's responsive classes and left the terminal narrow without refitting. Fullscreen now preserves the responsive layout and the terminal refits to fill the width.
Fix the web console "Plugins" button raising a
ValueError.Clicking Plugins in the dashboard raised
too many values to unpackafter the plugins view started returning pagination metadata. The dashboard now ignores the extra trailing values.