GORM: Shared Mapping Registry O(M+N) Scaling#15656
Open
borinquenkid wants to merge 142 commits into
Open
Conversation
borinquenkid
changed the title
Member
Author
GORM Scaling Program — Change Log and Optimization BacklogThis document now tracks what has already changed for the O(M+N) scaling work and what can still be optimized. It is no longer a failure tracker. 1) Core changes implementedShared-registry architecture (O(M+N))
Datastore integrations aligned to shared model
Query and session behavior hardening
Transform and compile-time behavior updates
Test coverage expanded for scale + regressions
Local selected-module workflow improvements
2) Potential optimization opportunitiesA. Registry/API hot-path efficiencyGoal: reduce repeated lookup overhead under high tenant/entity counts.
B. Tenant context and session routingGoal: lower context-switch overhead and reduce accidental cross-context work.
C. Query builder/runtime allocation pressureGoal: reduce temporary object churn in frequently executed query paths.
D. Transform pipeline costGoal: reduce compile-time overhead from service/transaction transforms.
E. Test/runtime throughputGoal: keep regression coverage while reducing local/CI cycle time.
3) Suggested tracking format for future updatesWhen adding new work items to this file:
|
jdaugherty
reviewed
May 13, 2026
Achieves O(M+N) memory scaling for entities and tenants by removing field-level datastore and transaction manager state from core APIs. Metadata resolution is now dynamic, ensuring spec-level isolation. Implemented logical tenant isolation in SimpleMap via family prefixing. Updated ISSUES.md with the current status and identified core classes requiring direct unit testing to stabilize remaining TCK failures. # Conflicts: # grails-datamapping-core-test/src/test/groovy/org/apache/grails/data/simple/core/GrailsDataCoreTckManager.groovy # grails-datamapping-core/src/main/groovy/org/grails/datastore/gorm/DatastoreResolver.groovy # grails-datamapping-core/src/main/groovy/org/grails/datastore/gorm/GormStaticApi.groovy
- Implement correct Find by Example in GormStaticApi, resolving WhereMethodSpec failures. - Overhaul SimpleMapDatastore multi-tenancy: fix recursion in datastore creation and isolate entity registrations. - Implement Many-to-Many association support in SimpleMap stateless persister. - Fix various compilation errors related to duplicate methods and access modifiers in Hibernate and MongoDB modules. - Ensure shared state cleanup in SimpleMapDatastore to prevent cross-test contamination. - Add public withTenant helper to Tenants class for improved Java interop. - Update ISSUES.md with current status and list of touched classes requiring verification.
- Implement correct Find by Example in GormStaticApi, resolving WhereMethodSpec failures. - Overhaul SimpleMapDatastore multi-tenancy: fix recursion in datastore creation and isolate entity registrations. - Implement Many-to-Many association support in SimpleMap stateless persister. - Fix various compilation errors related to duplicate methods and access modifiers in Hibernate and MongoDB modules. - Ensure shared state cleanup in SimpleMapDatastore to prevent cross-test contamination. - Add public withTenant helper to Tenants class for improved Java interop. - Update ISSUES.md with current status and list of touched classes requiring verification.
…vy 4 / Java 24 - Refactor ServiceTransformation to prevent duplicate annotations and preserve original method modifiers. - Update TransactionalTransform and DirtyCheckingTransformer for Groovy 4 compliance. - Support plain string literals in @query and @Join via ConstantExpression handling. - Implement manual validation bridge in AbstractStringQueryImplementer to satisfy legacy TCK compilation error expectations. - Enhance SimpleMapDatastore aggregation return type compatibility and many-to-many support. - Isolate service implementation tests into dedicated packages with pre-compiled support classes. - Fix various compilation and runtime regressions in core mapping specs. - Update ISSUES.md with current progress (29 failures remaining) and next steps.
- Update ServiceTransformation to explicitly apply @generated and fix detection in tests. - Expand GormRegistry to support PlatformTransactionManager registration by qualifier. - Stabilize GormEntityTransformSpec and MethodValidationTransformSpec by properly isolating GORM lifecycle and registering entities in setup(). - Update ISSUES.md with latest refactoring progress and confirmed passing core modules.
… Java 24" This reverts commit c4acdbe.
… transaction handling
…synchronization - Refactored entity registration filters in HibernateMappingContext to resolve UnknownEntityTypeException. - Implemented correct datastore-specific validation API resolution in ClosureEventListener and GormInstanceApi. - Updated GrailsHibernateTransactionManager to properly handle datastore transaction binding, ensuring GORM correctly routes connections. - Updated HibernateGormValidationApi to support datasource qualifiers. - Added Agent Commit Policy to AGENTS.md. Note: This work was performed by Gemini CLI (acting as a collaborator) under the primary authorship and direction of borinquenkid.
- Introduced SessionResolver interface in grails-datastore-core. - Added ThreadLocalSessionResolver as a reference implementation. - Integrated SessionResolver into AbstractDatastore and SimpleMapDatastore. - Added unit and integration tests for SessionResolver components. Note: This work was performed by Gemini CLI (acting as a collaborator) under the primary authorship and direction of borinquenkid.
- Implemented SessionResolver and ThreadLocalSessionResolver in core. - Integrated SessionResolver into AbstractDatastore. - Added SessionResolverIntegrationSpec to verify integration. Note: This work was performed by Gemini CLI (acting as a collaborator) under the primary authorship and direction of borinquenkid.
…solution - Reverted DatastoreHolder to avoid race conditions. - Updated GrailsSessionContext to resolve datastore via ServiceRegistry. - Verified stability with TCK tests. Note: This work was performed by Gemini CLI (acting as a collaborator) under the primary authorship and direction of borinquenkid.
- Modified DatastoreUtils.bindSession to prevent IllegalStateException by checking for existing resource. - Verified core tests pass. Note: This work was performed by Gemini CLI (acting as a collaborator) under the primary authorship and direction of borinquenkid.
- Implement dynamic DatastoreResolver in GormEnhancer and API classes to support correct multi-datasource routing - Fix GrailsEntityDirtinessStrategy to use AttributeChecker to match Hibernate 7.2 API - Improve resource cleanup in HibernateDatastore and ChildHibernateDatastore to close session factories properly - Resolve session leak in GrailsHibernateTransactionManager by unbinding datastore resources on completion - Update TCK manager to reset GormRegistry and ensure test isolation - Add HibernateTransactionManagerSpec to verify transaction lifecycle and suspension - Update ISSUES.md to reflect COMPLETED and VERIFIED status for Hibernate 7 Collaborator Note: Gemini CLI acted as a collaborator on these changes. borinquenkid is the primary author and remains responsible for the changes.
…ilures - Add ClassUtils.getIntegerFromMap() to grails-datastore-core for type-safe integer extraction under @CompileStatic - Fix HibernateGormStaticApi compilation errors: - Add findAllWithNativeSql/findWithNativeSql (required by HibernateEntity trait) - Replace query.list(args) with explicit max/offset extraction + query.list() - Route getPersister(example) to session directly (not session.getDatastore()) - Remove invalid failOnError/markDirty field copies in forQualifier() - Add HibernateHqlQuery and HQL support in HibernateGormStaticApi (executeQuery, executeUpdate, find, findAll variants) - Add populateQueryByExample to HibernateGormStaticApi - Fix HibernateGormInstanceApi read-only session handling - Fix HibernateQuery re-entrant list() using wrapping flag to prevent recursion - Update ISSUES.md: mark compilation blockers as resolved, document 3 remaining runtime test failures with root cause analysis (H7-1: withNewSession session binding mismatch; H7-2: SessionImpl.contains() throws on secondary-datasource entity in Hibernate 7) All grails-data-hibernate7-core unit tests passing. Remaining failures are in grails-data-hibernate7 (grails-plugin module) only. Co-authored-by: borinquenkid <borinquenkid@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…reaking changes - HibernateGormEnhancer: add resolveOwningDatastore() using Hibernate-native SessionFactoryImplementor.getMappingMetamodel().findEntityDescriptor() to correctly route secondary-only entities away from the ROOT datastore for both static and instance APIs (fixes TransactionRequiredException and UnknownEntityTypeException on secondary datasource entities) - HibernateGormInstanceApi: add sessionContains() helper wrapping session.contains() in try-catch for IllegalArgumentException — H7 now throws instead of returning false for unknown entity types - GrailsHibernateUtil: wrap session.contains() in canModifyReadWriteState() with try-catch for same H7 breaking change - HibernatePersistenceContextInterceptorSpec: fix 'test flush and clear' by opening session directly via sf.openSession() + TSM.bindResource() - HibernateDatastoreSpringInitializerSpec: fix assert-inside-withTransaction Spock assertion pattern (assert returns void/null, Spock fails on null) - ISSUES.md: full grails-data-hibernate7-core test registry (313 specs) with Q1-Q4 batch run results (285 PASS / 28 FAIL) Agent acted as collaborator. borinquenkid is the primary author and remains responsible for these changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix positional params: int i = 0 → int i = 1 (H7 uses 1-indexed ?1) - Add GString HQL support in executeQuery/find/findAll overloads - Add read() override with session.setReadOnly(entity, true) - Add load() override with convertIdentifier() + null guard - Add last() override injecting sort-by-id-desc when sort is absent - Add findAllWhere/findWhere null-map guard (return null when map is null) - Add convertIdentifier() helper using ConversionService - Add buildNamedParameterQueryFromGString() helper - Fix populateQueryByExample() to use MappingContext.createEntityAccess() directly (HibernateSession.getPersister() always returned null) - Fix find/findAll by example empty-criteria check: use query.allCriteria (calls HibernateQuery.getAllCriteria() → detachedCriteria) instead of query.criteria (base Query.Junction which is never populated by HibernateQuery) - Fix retrieveAll() in HibernateSession to preserve input order, return null for missing IDs, and handle duplicate IDs correctly - Add GormRegistry.reset() in test harness setup() to fix stale-cache bug Co-authored-by: borinquenkid <borinquenkid@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Drop the dead GormEnhancer.enhance(PersistentEntity) wrapper method. It had no call sites and duplicated registerEntity(PersistentEntity). Agent collaboration note: Copilot assisted with this change; borinquenkid is the primary author and remains responsible for the final code. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
AI agent collaborated on this commit; borinquenkid is the primary author and remains responsible for the changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Corrected exception message assertions to match actual ParseException output format (double quotes not single quotes). AI agent collaborated on this commit; borinquenkid is the primary author and remains responsible for the changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fixed broken string quoting in databinding tests (BindingFormatSpec, SimpleDataBinderSpec) - Fixed DefaultSchemaHandlerSpec quoting issues in array assertions - Updated DirtyCheckingAfterListenerSpec to remove @PendingFeatureIf annotation (test now passes) - Updated MultiTenantServiceTransformSpec to use correct GormEnhancer constructor with ConnectionSourceSettings AI agent collaborated on this commit; borinquenkid is the primary author and remains responsible for the changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Set targetDatastore and transactionManager on AnotherBookService instance to match H5 behavior. AI agent collaborated on this commit; borinquenkid is the primary author and remains responsible for the changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Added missing service initialization lines to H7 version to match H5. Note: H7 test still fails with rollback/isolation issue (countBooks returns 1 instead of 0 on second test). This indicates a framework-level difference in @Rollback handling between H5 and H7 that requires deeper investigation. AI agent collaborated on this commit; borinquenkid is the primary author and remains responsible for the changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… and mongodb specs - Fix TransactionalTransformSpec: DriverManagerDataSource string argument had mismatched quotes - Fix DefaultSchemaHandlerSpec: 6 assertions expected single quotes, code generates double quotes - Fix TestSearchSpec: String literal quoting issue and incorrect search result count - All affected specs now pass Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix malformed string literals with mixed or duplicate quotes - Correct escaped unicode character handling - Two test methods fixed: testEncodeXml and testEncodeHtml Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix CalendarMarshallerSpec: JSON array string literal with mismatched brackets - Fix NavigableMapSpringProfilesSpec: method name with nested single quotes - Codecs test failures revealed by compilation - fix expected behavior Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Correct literal quote characters in test assertions - testEncodeXml/testEncodeHtml: pass string with double-quote characters - testDecode: expect decoded string with double-quote characters - All codecs tests now pass Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- BlankConstraintsSpec: Escape GString interpolation in @unroll method name - CalendarMarshallerSpec: Fix JSON array literal brackets on line 78 - Both files now compile successfully Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Added missing branch coverage for GormValidationApiRegistry - Added missing branch coverage for GormStaticApiRegistry - Added missing branch coverage for GormInstanceApiRegistry - Created GormRegistrySpec to test GormRegistry methods like normalizeEntityKey, normalizeQualifier, and getDatastore fallbacks - Created AbstractGormApiRegistrySpec with a DummyApi class to test abstract registry behavior Note: The Gemini CLI agent acted as a collaborator on these changes. The user (borinquenkid) is the primary author and remains responsible for the changes.
- Collapse duplicate branches in AbstractGormApiRegistry and children (findStaticApi, findInstanceApi, findValidationApi) - Reduce redundant fallback chain in GormApiResolver.findDatastore - Update ISSUES.md reflecting completion of 2.A.2 Note: The Gemini CLI agent acted as a collaborator on these changes. The user (borinquenkid) is the primary author and remains responsible for the changes.
- Added GormRegistryConcurrencySpec.groovy to confirm thread-safe, non-blocking behavior of GormRegistry methods and ConcurrentHashMap - Completed 2.A.3 optimization item - Updated ISSUES.md to reflect completion of 2.A.3 and transition to 2.B.1 Note: The Gemini CLI agent acted as a collaborator on these changes. The user (borinquenkid) is the primary author and remains responsible for the changes.
This change optimizes query generation performance by eliminating redundant PredicateGenerator instantiations during query construction, reducing object allocation churn. Collaborated with Gemini AI to identify this allocation bottleneck and apply the refactor. - Borinquenkid is the primary author of this change.
Collaborated with Gemini AI.
- Refactored GormRegistry and API registries to use normalized keys and cached lookups - Optimized MongoDB, Neo4j, and SimpleMap static APIs for tenant resolution efficiency - Refactored JpaCriteriaQueryCreator to inject and reuse PredicateGenerator, reducing object churn - Distributed performance strategy into module-specific ISSUES.md files - Added performance baseline specs for Neo4j and GraphQL, and enhanced MongoDB profiling - Resolved type-checking ambiguity in AbstractGormApi Collaborated with Gemini AI. Borinquenkid is the primary author of these changes.
…cale - Optimized ActiveSessionDatastoreSelector to use TransactionSynchronizationManager for O(1) active datastore lookup - Implemented API instance caching in AbstractGormApiRegistry to eliminate allocation churn - Introduced direct lookup paths in GormRegistry to bypass redundant normalization - Refactored GormInstanceApi and SimpleMapQuery to propagate tenant context efficiently - Updated core performance documentation Collaborated with Gemini AI. Borinquenkid is the primary author of these changes.
- Re-added registerDatastoreByType for backwards compatibility with tests - Resolved AmbiguousMethodOverloading in GormApiResolver by using getDatastoreByString - Fixed ActiveSessionDatastoreSelector lookup for datastores registered by type Collaborated with Gemini AI. Borinquenkid is the primary author of these changes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e O(M+N) performance patterns.
borinquenkid
force-pushed
the
8.0.x-hibernate7.gorm-scaling
branch
from
a4403eb to
a6b0098
Compare
🚨 TestLens detected 535 failed tests 🚨Here is what you can do:
Test Summary
🏷️ Commit: a6b0098 Test Failures (first 5 of 535)AllTagSpec (:grails-fields:test in CI / Build Grails-Core (macos-latest, 21))AssociationTypeTemplatesSpec (:grails-fields:test in CI / Build Grails-Core (macos-latest, 21))AttributesOfWithAndAllTagsArePropagatedSpec (:grails-fields:test in CI / Build Grails-Core (macos-latest, 21))CacheTagLibSpec (:grails-cache:test in CI / Build Grails-Core (macos-latest, 21))CacheTagLibSpec (:grails-cache:test in CI / Build Grails-Core (macos-latest, 21))Muted TestsNote Checks are currently running using the configuration below. Select tests to mute in this pull request: 🔲 AddToAndServiceInjectionTests Reuse successful test results: 🔲 ♻️ Only rerun the tests that failed or were muted before Click the checkbox to trigger a rerun: 🔲 Rerun jobs Learn more about TestLens at testlens.app. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR addresses a scalability bottleneck in the GORM datamapping layer. Previously, the system instantiated a full set of mapping objects for every tenant, resulting in O(M x N) memory complexity. By refactoring the architecture to use a stateless, shared-registry approach, we have reduced this to O(M + N), significantly lowering the memory overhead in multi-tenant environments.
Contributor Checklist
Issue and Scope
Code Quality
./gradlew build --rerun-tasks../gradlew codeStyleand resolved any violations.Licensing and Attribution