v 0.2 refactor (breaking changes)

[monkeypants]

What started as "C4 architecture domain: sphinx extensions, API + MCP service" has grown into a rather large, breaking set of changes.

The purpose of this branch was originally to:

• add Architecture domain (to compliment HCD domain) to the sphinx extensions
• add API (REST style, noun oriented) and MCP (json-rpc style, verb oriented) interfaces

But things changed in interesting ways. When it was apparent that some breaking changes were necessary, I went ahead and made all the breaking changes, since this is now a hairy breaking change, we might as well get them all over and done with.

Rater than try to follow the ~230 commits and slightly absurd number of new lines of code, let me give you an overview of the changes, then propose the things to review. I'll also make a bit of a simplified retrospective narrative that hopefully serves as some kind of justification for the pain.

There are a few things going on here:

Doctrine

• code layout changes that simplify/correct for some initial missteps (my bad, failure to see ahead)
• a set of unittests that enforce "architecture doctrine". This is a wrapper of src/julee/core/doctrine/ tests, which can be run as a test suite runner and also via the julee-admin doctrine verify command. These tests scan the codebase introspectively.
• this was iterated - I would note a couple of inconsistent things, decide the rule, enforce the doctrine, fix everywhere to make consistent. This process drove the architectural refactor.

The tests ARE the doctrine" - docstrings state rules, test assertions enforce them.

File	Doctrine Area
test_entity.py	Entity naming, documentation, type annotations, BaseModel usage
test_use_case.py	UseCase naming, structure, matching Request/Response
test_request.py	Request naming/documentation
test_response.py	Response naming/documentation
test_repository_protocol.py	Repository patterns
test_service_protocol.py	Service patterns
test_bounded_context.py	BC structure rules
test_dependency_rule.py	Import/dependency constraints
test_pipeline.py	Pipeline patterns
test_application.py	Application structure
test_deployment.py	Deployment patterns
test_solution.py	Solution structure
test_documentation.py	Documentation requirements
test_documentation_links.py	Link validity (slow)
test_doctrine_coverage.py	Ensures doctrine tests exist
test_semantic_relation.py	Semantic relation patterns
test_sphinx_extension.py	Sphinx integration

Semantic Relations

Note test_semantic_relation.py above. You can run julee-admin doctrine show --area semantic -v to see those tests.

This is all about the behavior of the @semantic_relation decorator, defined in src/julee/core/decorators.py:545-614. This is a way of asserting a semantic relationship between one domain model entity and another (from a different bounded context). For example, see how the hcd/entities/story.py

@semantic_relation("julee.hcd.entities.app.App", RelationType.PART_OF)
@semantic_relation("julee.hcd.entities.persona.Persona", RelationType.REFERENCES)
@semantic_relation("julee.core.entities.use_case.UseCase", RelationType.PROJECTS)
class Story(BaseModel):

What this means is that:

• Apps, Personas, and UseCases are also entities defined in the current Julee solution.
• a Story entity is PART_OF an App. Imagine a user story "as a foo wrangler, I need to fire the laser cannon on my amphibious jeep, so that I can impress the ducks". The amphibious jeep is the app, one way to think about the the app is as a container for the stories about how it is used to meet the needs of its users.
• A story REFERENCES a persona. I.e. the foo wrangler. If I am thinking about the persona, it might be nice to know what stories reference them. If I am thinking about the story, it might be nice to know more about this foo wrangler character that the story mentions by name.
• A story PROJECTS the usecase. Usecase is our core implementation concept. It's not metadata, it's an actual clean-architecture code artefact that forms part of our domain model (and our doctrine enforces how). Projection is a nuanced concept here. The HCD domain projects onto the implementation, it is a viewpoint or perspective on the implementation. As a developer, it tells us that when the designers are talking about the system from an HCD perspective (in terms of Stories), we can map this to implementation through the usecase artefacts in code. The same usecase code might be reused in multiple apps (an API, a worker, a cli, etc) but HCD people don't need to know about that, they just know about the stories. We coders know the story is a manifestation of the usecase code in some application, through this semantic relationship.

This goes to the bigger picture of the framework. The framework has a "core" bounded context that relates to implementation of every other thing in a julee solution. The code is bound to core through doctrine. But the domain of the framework is software engineering, so we have multiple "partial ontologies" for different "aspects" of software engineering. Currently that's only HCD and Architecture, but in the future it could be extended to Security, Maintenance, Support, Packaging, Deployment, etc. Those "framework bounded contexts" all have their own semantics which projects onto core, which means they can be projected onto implementation of other julee solutions. So, for a julee solution that is not the framework itself (e.g. Evil World Domination Corporation), it might have it's own bounded contexts (Extortion, Revenge, Warfare and Politics, Very Large Kites, etc). If those are compliant with doctrine, then they can utilise the various framework-provided software engineering views over their implementation.

The semantic relations are defined by an enum:

Type	Meaning	Example
IS_A	Specialization	CustomerSegment is_a Persona
PROJECTS	View/projection	Accelerator projects BoundedContext
IMPLEMENTS	Protocol impl	SqlAlchemyRepo implements Repository
ENABLES	Supports	AuthUseCase enables Story
PART_OF	Contained in	Story part_of App
CONTAINS	Aggregates	Epic contains Story
REFERENCES	Non-owning ref	Story references Persona
BROADER	More general (SKOS)	Vehicle broader TransportMode
NARROWER	More specific (SKOS)	TransportMode narrower Vehicle
RELATED	Associative	Generic relationship

The implementation of these semantic relation decorators are currently used for

1. Navigation - Build hierarchies (Epic → Story → Scenario) as part of document generation / the sphinx app uses relations to link entity pages
2. Architectural validation - Verify entities properly declare dependencies (noting semantic dependency rules apply as well as runtime/compile time dependency rules - the code and semantics both need to be "clean")
3. Introspection - Discover how bounded contexts relate

Sphinx App

The key concept here is that rst documentation can be considered "a database that renders to documents".

• custom "describe-entity" type directives can be considered like database tables, schema definitions
• custom "list-entity" type directives can be considered like database queries
• :foo: properties on directives are like columns in a database, and they may also be foreign key relationships to other domain objects

So, if our rst documentation is strongly typed with lots of custom directives, then we can treat our documentation like a normalised database. This means we can read and write to it with repository methods, so it can become part of our clean architecture. The whole "rendering to html/latexpdf/etc" bag of tricks that comes with sphinx is still available, but that's a side effect in the outside world, internally they are just documented domain objects persisted to disk.

We can also use the AST to parse the code, understand inheritance, parameter types, etc - that's also "just infrastructure" that implements a repository protocol that happens to knows about the code artefacts. So we can have a directive saying "let there be an application, because I have documented it" and we can have an actual application implementation that is also documented because it exists. The sphinx app can use both repositories to build a picture of everything that is built and has been conceived/documented (but not yet built) and render them both together in the one solution documentation.

This is a work in progress, but it's not-blocking (if the docs are good enough, ship them then refine and ship again).

generic_crud usecase generators

Clean Architecture has pros and cons. The pros include clarity - single responsibility, interfaces that protect from variation, etc. The cons include verbosity - more files/classes than would otherwise be used. The worst part was all the boilerplate CRUD usecases - {GET/UPDATE/CREATE/DELETE/LIST}{the entity}. So I cheated and made a factory method for generic crud usecases (src/julee/core/use_cases/generic_crud.py).

  generic_crud.generate(
      Story,
      StoryRepository,
      filters=["app_slug", "persona"],
  )

  # Generates and injects into module namespace:
  # - GetStoryRequest, GetStoryResponse, GetStoryUseCase
  # - ListStoriesRequest, ListStoriesResponse, ListStoriesUseCase
  # - CreateStoryRequest, CreateStoryResponse, CreateStoryUseCase
  # - UpdateStoryRequest, UpdateStoryResponse, UpdateStoryUseCase
  # - DeleteStoryRequest, DeleteStoryResponse, DeleteStoryUseCase

This has a couple of tricks. First, it uses the Repository protocol specification with a template to infer the properties of the Request and Response classes. Second, the (optional) "filters" property determines what filters can be applied to the list method. It has a few other tricks (like janky pluralisation guessing code that could probably be improved) but the upshot is that we can make a {bounded-context}.usecases.crud module cheaply that gives us the CRUD methods without having to think about it, and wecan wire them up to a REST API using similar patterns, and that side of things can go very fast.

Bounded contexts and reserved words.

The idea here is that every top level directory is a bounded context of the solution, except for a handful of magic names that have other meanings, unless it is a "nested solution" (a sub solution containing bounded contexts, and reserved words). The contrib/ module is one of those nested solutions.

They are documented in src/julee/core/doctrine_constants.py, currently as

Reserved Word	Constant	Purpose
apps	APPS_ROOT	Application layer with its own discovery
deployments	DEPLOYMENTS_ROOT	Deployment configurations
docs	DOCS_ROOT	Documentation (required for every solution)

apps/ contains apps. An app has dependencies on bounded contexts, but the BC has no idea what apps depend on it.. Apps are deployable/usable things.

deploy/ depends on apps. And so on - there is potential set of more onion rings of dependencies that have the software at the core, and supporting apparatus around it (maintenance, support, governance, etc). This is what the reserved words are really about.

There is some wooly thinking about if these things are framework bounded contexts, or if they are "outer architectural layers". Basically, apps need to be well organised code that imports and uses bounded contexts, and because one app may involve multiple bounded contexts, it has to be outside them for practical reasons. Similarly, Deployment may involve multiple apps, so they have to sit outside them too. It's a pragmatic distinction rather than theoretical one.

In summary, don't try to review the changes in this PR, it's too much. Look at the PR as a new, slightly radical change. Most of the disruptive changes relate to the application of doctrine. The other changes (sphinx docs, semantic relations) are the (hopefully) useful work that I was doing as I dogfooded the doctrine. The other thing to lok at, if you have access to it (sorry it's not a public repo) is the modernise-solution-layout branch of pyx-labs. This is a multi-bounded context julee solutuion that has had the doctrine applied to it too. Iterating through the doctrine changes while using that migration target was very informative. I'm sure that we will continue to refine the doctrine as we migrate other solutions, but the current doctrine version has been shaken out already.

Final Note - the differences between Repositories and Services

Doctrine development forced me to clarify this.

1. A repository interface is bound to the semantics of a single entity. A service interface is bound to the semantics of more than one entity. That's it. Services are typically transformational - give me an apple and I will give you an apple pie (that's two different entities), and they may be some kind of value-adding process that may be performed by a 3rd party as part of a digital supply chain, and they may involve AI or other funky non-determinisims, but none of that is definitional. They are like a repository in that they are a protocol have one or more implementations (which are bound to tech).

repositories and services both furnish interfaces (protocols) that deal in domain objects and/or primitive types.

usecases furnish interfaces that are request and response objects. They consume domain language interfaces of services and repositories that are injected at construction time.

A lot of the mind-bending changes during this big hairy refactor was clarifying this - redistributing responsibility between services, usecases and repositories so that the above was always true (doctrinal compliance). These might have been the most disruptive changes, but the end result is simpler and a better basis for going forward with more services and usecases.

Merge suggestion

This is a big breaking change. I propose we create a legacy branch for 0.1.x (old master) and maintain that as long as necessary, merge this into master, and increment the Major number on the release (0.2.0). Since we are not yet in 1.x.x I think we are allowed to make non-backwards compatible changes without incrementing the architecture number.

Timing of the merge should be after the other work has landed and been deployed, then the actual merge will be a bit hairy (bringing that new work into this branch, testing and fixing as we go, then flopping this on top in a big squash).

[monkeypants]

totes squasher :(

[monkeypants]

At this point, what I am doing in the PR is to try to get the julee framework to be self-documenting using the architecture and hcd modules. This may potentially involve some violent refactors, for that I apologise in advance. If that proves necessary then the sooner they are done the better.

What I mean by self-documenting is that the framework (and solutions based on the framework) will have architecture documentation and HCD documentation that is coherent and sensible, and is mostly levered-out of the implementation. I want the docs and code to embody a doctrine that is easy for agents to follow.

[monkeypants]

Recent Changes

Supply Chain Bounded Context

• Moved Accelerator entity from HCD to new supply_chain bounded context
• Added Party entity for UNTP (UN Transparency Protocol) support
• Updated admin CLI imports to use new location

UNTP Projection Layer

• Added contrib/untp projection layer for UNTP vocabulary mapping
• Implements semantic mapping between julee entities and UNTP concepts
• Supports digital supply chain transparency standards

Semantic Relations Infrastructure

• Added entity-graph directive for visualizing semantic relations
• Added unified-links directive for automatic bidirectional documentation
• Grouped semantic links by bounded context in documentation
• Documented Entity as ontological root for all domain entities
• Added RelationTraversal service for semantic relation infrastructure

Doctrine Improvements

• Fixed BC doctrine tests to use local constants for synthetic tests
• Added handling for skipped tests in doctrine verification
• Use dynamic search_root fixture for doctrine tests

Tests

All unit tests passing (1862 passed, 22 skipped). CEAP integration tests require MinIO infrastructure.