Skip to content

Commit ebd7514

Browse files
pwnage101pwnage101
committed
chore: Eliminate devstack integration
Transition away from the "remote devstack" model where IDAs implemented their own devstack integration. This is no longer necessary because all IDAs should now be directly integrated with devstack by adding make targets, provision scripts, and docker-compose services DIRECTLY within edx/devstack. - Overhaul docker-compose.yml to completely change the purpose of the `app` container to be exclusively for running tests. - Update a bunch of other files to reflect the new purpose of the testing-only `app` container. Doing this first in enterprise-access to serve as a model we can test and iron out before exporting to the rest of enterprise IDAs.
1 parent 2efc827 commit ebd7514

11 files changed

Lines changed: 112 additions & 313 deletions

File tree

‎.claude/skills/quality-tests/SKILL.md‎

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,17 @@
11
---
22
name: quality-tests
33
description: Run code quality checks (linting, style, PII annotations) in the enterprise-access Docker container. Use when the user wants to run quality checks, lint code, or verify code style compliance.
4-
allowed-tools: Bash(docker *), Bash(make *)
4+
allowed-tools: Bash(docker *), Bash(make *), Bash(colima *)
55
---
66

77
## Steps
88

99
### 1. Make sure the app container is running
1010

11-
Determine if the enterprise-access.app container is running:
11+
Determine if the app container is running:
1212

1313
```bash
14-
docker ps | grep enterprise-access.app
14+
docker compose ps
1515
```
1616

1717
Start the app container if not running:
@@ -23,5 +23,25 @@ make dev.up
2323
### 2. Run quality checks
2424

2525
```bash
26-
docker exec enterprise-access.app bash -c "make quality"
26+
docker compose exec app bash -c "make quality"
27+
```
28+
29+
## Troubleshooting
30+
31+
### ModuleNotFoundError
32+
33+
If quality checks fail due to missing imports, first try to install requirements:
34+
35+
```bash
36+
docker compose exec app make ci_requirements
37+
```
38+
39+
This is necessary at least when adding new requirements which have not yet been built into the image.
40+
41+
### Failed to connect to the docker API on MacOS
42+
43+
This likely just means colima needs to be started:
44+
45+
```bash
46+
colima start
2747
```

‎.claude/skills/unit-tests/SKILL.md‎

Lines changed: 27 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
name: unit-tests
33
description: Run Django unit tests in the enterprise-access Docker container. Use when the user wants to run tests, check if tests pass, or verify test coverage.
44
argument-hint: "<TEST_FILES>"
5-
allowed-tools: Bash(docker *), Bash(make *)
5+
allowed-tools: Bash(docker *), Bash(make *), Bash(colima *)
66
---
77

88
## Arguments
@@ -26,10 +26,10 @@ Examples:
2626

2727
### 1. Make sure the app container is running
2828

29-
Determine if the enterprise-access.app container is running:
29+
Determine if the app container is running:
3030

3131
```bash
32-
docker ps | grep enterprise-access.app
32+
docker compose ps
3333
```
3434

3535
Start the app container if not running:
@@ -43,7 +43,7 @@ make dev.up
4343
If `<TEST_FILES>` specify .py files or functions/classes within, run only those tests using `pytest.local.ini` to disable coverage and warnings:
4444

4545
```bash
46-
docker exec enterprise-access.app bash -c "pytest -c pytest.local.ini <TEST_FILES>"
46+
docker compose exec app bash -c "pytest -c pytest.local.ini <TEST_FILES>"
4747
```
4848

4949
Example `<TEST_FILES>` which match this case:
@@ -60,10 +60,10 @@ If `<TEST_FILES>` specifies a single directory which represents a domain, such a
6060
Convert the directory path to a Python module path for `--cov`: replace `/` with `.` and strip any trailing slash. Example: `enterprise_access/apps/bffs/` → `enterprise_access.apps.bffs`
6161

6262
```bash
63-
docker exec enterprise-access.app bash -c "pytest -c pytest.local.ini <TEST_DOMAIN> --cov=<MODULE_PATH_OF_TEST_DOMAIN>"
63+
docker compose exec app bash -c "pytest -c pytest.local.ini <TEST_DOMAIN> --cov=<MODULE_PATH_OF_TEST_DOMAIN>"
6464

6565
# Example: <TEST_FILES> is "enterprise_access/apps/bffs"
66-
docker exec enterprise-access.app bash -c "pytest -c pytest.local.ini enterprise_access/apps/bffs --cov=enterprise_access.apps.bffs"
66+
docker compose exec app bash -c "pytest -c pytest.local.ini enterprise_access/apps/bffs --cov=enterprise_access.apps.bffs"
6767
```
6868

6969
Example `<TEST_FILES>` which match this case:
@@ -80,7 +80,27 @@ Whole-domain coverage will be reported in the console output. Specific line numb
8080
If no arguments are given, assume the user wants to run the full test suite for the entire project:
8181

8282
```bash
83-
docker exec enterprise-access.app bash -c "make test"
83+
docker compose exec app bash -c "make test"
8484
```
8585

8686
Whole-project coverage will be reported in the console output. Specific line numbers with missing coverage will be reported.
87+
88+
## Troubleshooting
89+
90+
### ModuleNotFoundError
91+
92+
If tests fail due to missing imports, first try to install requirements:
93+
94+
```bash
95+
docker compose exec app make ci_requirements
96+
```
97+
98+
This is necessary at least when adding new requirements which have not yet been built into the image.
99+
100+
### Failed to connect to the docker API on MacOS
101+
102+
This likely just means colima needs to be started:
103+
104+
```bash
105+
colima start
106+
```

‎.ralph/PROMPT.md‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -122,9 +122,8 @@ Check if a reusable code pattern exists in `.ralph/specs/stdlib` before executin
122122

123123
### Docker Development
124124

125-
- Server runs on `localhost:18270`
126-
- Uses MySQL 8.0, Memcache, Redis, and Celery worker
127-
- Event bus via Kafka (Confluent Control Center at localhost:9021)
125+
- Full devstack (app, worker, DB, Kafka, etc.) is managed in the devstack repository
126+
- The `docker-compose.yml` in this repo provides a lightweight container for running tests and quality checks only
128127

129128
## Testing Notes
130129

‎CLAUDE.md‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -62,9 +62,8 @@ directory does not exist).
6262

6363
### Docker Development
6464

65-
- Server runs on `localhost:18270`
66-
- Uses MySQL 8.0, Memcache, Redis, and Celery worker
67-
- Event bus via Kafka (Confluent Control Center at localhost:9021)
65+
- Full devstack (app, worker, DB, Kafka, etc.) is managed in the devstack repository
66+
- The `docker-compose.yml` in this repo provides a lightweight container for running tests and quality checks only
6867

6968
## Testing Notes
7069

‎Makefile‎

Lines changed: 38 additions & 89 deletions
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,12 @@
55
test coverage isort_check isort style lint quality pii_check validate \
66
migrate html_coverage upgrade extract_translation dummy_translations \
77
compile_translations fake_translations pull_translations \
8-
push_translations start-devstack open-devstack pkg-devstack \
9-
detect_changed_source_translations validate_translations check_keywords
10-
11-
COMMON_CONSTRAINTS_TXT=requirements/common_constraints.txt
12-
.PHONY: $(COMMON_CONSTRAINTS_TXT)
13-
$(COMMON_CONSTRAINTS_TXT):
14-
wget -O "$(@)" https://raw.githubusercontent.com/edx/edx-lint/master/edx_lint/files/common_constraints.txt || touch "$(@)"
8+
push_translations \
9+
detect_changed_source_translations validate_translations check_keywords \
10+
dev.pull dev.up dev.down dev.stop dev.makemigrations dev.shell dev.logs \
11+
dev.restart-container dev.attach \
12+
app-up app-down app-stop app-makemigrations app-shell app-logs \
13+
app-restart-container app-attach
1514

1615
define BROWSER_PYSCRIPT
1716
import os, webbrowser, sys
@@ -103,9 +102,6 @@ migrate: ## apply database migrations
103102
html_coverage: ## generate and view HTML coverage report
104103
coverage html && open htmlcov/index.html
105104

106-
subsidy_client_local: # re-install edx-enterprise-subsidy-client from local code
107-
pip uninstall -y edx-enterprise-subsidy-client && pip install -e /edx/src/edx-enterprise-subsidy-client/ && pip freeze | grep subsidy-client
108-
109105
COMMON_CONSTRAINTS_TXT=requirements/common_constraints.txt
110106
.PHONY: $(COMMON_CONSTRAINTS_TXT)
111107
$(COMMON_CONSTRAINTS_TXT):
@@ -152,12 +148,6 @@ pull_translations: ## pull translations from Transifex
152148
push_translations: ## push source translation files (.po) from Transifex
153149
tx push -s
154150

155-
start-devstack: ## run a local development copy of the server
156-
docker-compose --x-networking up
157-
158-
open-devstack: ## open a shell on the server started by start-devstack
159-
docker exec -it enterprise-access /edx/app/enterprise-access/devstack.sh open
160-
161151
detect_changed_source_translations: ## check if translation files are up-to-date
162152
cd enterprise_access && i18n_tool changed
163153

@@ -176,90 +166,49 @@ travis_docker_push: travis_docker_tag travis_docker_auth ## push to docker hub
176166
docker push 'openedx/enterprise-access:latest-newrelic'
177167
docker push "openedx/enterprise-access:$$TRAVIS_COMMIT-newrelic"
178168

179-
dev.pull:
180-
docker image pull edxops/enterprise-access-dev
181-
182-
dev.provision:
183-
bash ./provision-enterprise-access.sh
184-
185-
# devstack-themed shortcuts
186-
# Starts all containers
187-
dev.up: dev.pull dev.up.redis
188-
docker-compose up -d
189-
190-
dev.up.with-events: dev.up.kafka-control-center dev.up
191-
192-
# Start redis via the devstack docker-compose.yml
193-
dev.up.redis:
194-
docker-compose -f $(DEVSTACK_WORKSPACE)/devstack/docker-compose.yml up -d redis
195-
196-
# Start kafka via the devstack docker-compose.yml
197-
# https://github.com/openedx-unsupported/devstack/blob/323b475b885a2704489566b262e2895a4dca62b6/docker-compose.yml#L140
198-
dev.up.kafka-control-center:
199-
docker-compose -f $(DEVSTACK_WORKSPACE)/devstack/docker-compose.yml up -d kafka-control-center
200-
201-
# Useful for just restarting everything related to the event broker
202-
dev.down.kafka-control-center:
203-
docker-compose -f $(DEVSTACK_WORKSPACE)/devstack/docker-compose.yml down kafka zookeeper schema-registry kafka-control-center
204-
205-
# Kills containers and all of their data that isn't in volumes
206-
dev.down:
207-
docker-compose down
208-
209-
# Stops containers so they can be restarted
210-
dev.stop:
211-
docker-compose stop
212-
213-
dev.backup:
214-
docker-compose stop app worker
215-
docker-compose up -d mysql80
216-
sleep 10 # let mysql process get fully warmed up
217-
docker compose exec mysql80 mysqldump --all-databases > .dev/enterprise_access_all.sql
218-
219-
dev.restore:
220-
docker-compose stop app worker
221-
docker-compose up -d mysql80
222-
sleep 10 # let mysql process get fully warmed up
223-
docker compose exec -T mysql80 mysql < .dev/enterprise_access_all.sql
224-
225-
app-shell: # Run the app shell as root
226-
docker exec -u 0 -it enterprise-access.app bash
227169

228-
db-shell-57: # Run the mysql 5.7 shell as root, enter the app's database
229-
docker exec -u 0 -it enterprise_access.db mysql -u root enterprise_access
170+
####################################################################
171+
# Docker shortcuts for managing a local test/quality container. #
172+
# For full devstack (app, worker, DB, etc.), see the devstack #
173+
# repository which now manages enterprise-access as a first-class #
174+
# service. #
175+
####################################################################
230176

231-
db-shell-8: # Run the mysql 8 shell as root, enter the app's database
232-
docker exec -u 0 -it enterprise_access.mysql80 mysql -u root enterprise_access
177+
dev.pull: ## Pulls the docker image used by the app container
178+
docker compose pull
233179

234-
dev.dbcopy8: ## Copy data from old mysql 5.7 container into a new 8 db
235-
mkdir -p .dev/
236-
docker-compose exec db bash -c "mysqldump --databases enterprise_access" > .dev/enterprise_access.sql
237-
docker-compose exec -T mysql80 bash -c "mysql" < .dev/enterprise_access.sql
238-
rm .dev/enterprise_access.sql
180+
dev.up: dev.pull ## Starts the app container
181+
docker compose up --remove-orphans -d
239182

240-
%-logs: # View the logs of the specified service container
241-
docker-compose logs -f --tail=500 $*
183+
dev.down: ## Kills the app container and all its data that isn't in volumes
184+
docker compose down
242185

243-
%-restart: # Restart the specified service container
244-
docker-compose restart $*
186+
dev.stop: ## Stops the app container so it can be restarted
187+
docker compose stop
245188

246-
app-restart-devserver: # restart just the app Django dev server
247-
docker-compose exec app bash -c 'kill $$(ps aux | egrep "manage.py ?\w* runserver" | egrep -v "while|grep" | awk "{print \$$2}")'
189+
dev.makemigrations: ## Create migrations via the app container
190+
docker compose exec app python manage.py makemigrations
248191

249-
app-attach:
250-
docker attach enterprise-access.app
192+
dev.shell: ## Launch a shell in the app container
193+
docker compose exec app bash
251194

252-
worker-attach:
253-
docker attach enterprise_access.worker
195+
dev.logs: ## View the logs of the app container
196+
docker compose logs -f --tail=500 app
254197

255-
%-shell: # Run a shell, as root, on the specified service container
256-
docker exec -u 0 -it enterprise_access.$* bash
198+
dev.restart-container: ## Restart the app container
199+
docker compose restart
257200

258-
dev.static:
259-
docker-compose exec -u 0 app python3 manage.py collectstatic --noinput
201+
dev.attach: ## Attach to the app container
202+
docker compose attach app
260203

261-
dev.migrate:
262-
docker-compose exec -u 0 app python manage.py migrate
204+
app-up: dev.up
205+
app-down: dev.down
206+
app-stop: dev.stop
207+
app-makemigrations: dev.makemigrations
208+
app-shell: dev.shell
209+
app-logs: dev.logs
210+
app-restart-container: dev.restart-container
211+
app-attach: dev.attach
263212

264213
github_docker_auth:
265214
echo "$$DOCKERHUB_PASSWORD" | docker login -u "$$DOCKERHUB_USERNAME" --password-stdin

‎README.rst‎

Lines changed: 12 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -3,74 +3,29 @@ Service to manage access to content for enterprise users.
33
Setting up enterprise-access
44
--------------------------
55

6-
Prerequisites
7-
^^^^^^^^^^^^^
8-
- Set the ``DEVSTACK_WORKSPACE`` env variable (either locally or in your shell config file: ``.bash_rc``, ``.zshrc``, or equivalent) to the folder which contains this repo and the `devstack` repo.
9-
e.g ``export DEVSTACK_WORKSPACE=/home/<your_user>/edx``
10-
- Set up `devstack <https://github.com/edx/devstack>`_
6+
Full devstack setup
7+
^^^^^^^^^^^^^^^^^^^
8+
For running the full enterprise-access application (app, worker, database, etc.), see the
9+
`devstack <https://github.com/edx/devstack>`_ repository, which manages enterprise-access
10+
as a first-class service.
1111

12-
Quick Setup
13-
^^^^^^^^^^^
12+
Running tests and quality checks locally
13+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
14+
The ``docker-compose.yml`` in this repository provides a lightweight container for running
15+
tests and quality checks without the full devstack infrastructure.
1416

1517
::
1618

17-
$ make docker_build
18-
$ make dev.provision
1919
$ make dev.up
20-
$ make app-shell
21-
# make requirements
22-
# make validate # to run full test suite
23-
24-
The server will run on ``localhost:18270``
25-
26-
Running migrations
27-
^^^^^^^^^^^^^^^^^^
28-
29-
::
30-
31-
$ make app-shell
32-
# python ./manage.py migrate
33-
34-
Setting up openedx-events
35-
^^^^^^^^^^^^^^^^^^^^^^^^^
36-
Ensure you have installed the ``edx_event_bus_kafka`` and ``openedx_events`` requirements. Entering
37-
a shell with ``make app-shell`` and then running ``make requirements`` should install these for you.
38-
39-
From your host, run ``make dev.up.with-events``, which will start a local kafka container for you.
40-
Visit http://localhost:9021/clusters to access the local "Confluent Control Center".
41-
Confluent is like a cloud wrapper around "vanilla" Kafka.
42-
43-
Your ``devstack.py`` settings should already be configured to point at this event broker,
44-
and to configure enterprise-access as an openedx event consumer and produer.
45-
46-
We have a specific enterprise "ping" event and management command defined to test
47-
that your local event bus is well-configured. Open a shell with ``make app-shell`` and run::
48-
49-
./manage.py consume_enterprise_ping_events
50-
51-
This will consume ping events from the ``dev-enterprise-core`` topic.
52-
You may see a ``Broker: Unknown topic`` error the first time you run it. When you run your
53-
test event production below, that error will resolve (producing the event creates the topic
54-
if it does not exist). **Leave the consumer running.** You should see the ``enterprise-access-service``
55-
as a registered consumer in your local confluent control center.
56-
57-
Now, go over to your **enterprise-subsidy** directory. Make sure requirements are installed,
58-
specifically the ``edx_event_bus_kafka`` and ``openedx_events`` packages. Use ``make app-shell``
59-
in this repo and we'll *produce* a ping event::
60-
61-
./manage.py produce_enterprise_ping_event
62-
63-
If this event was successfully produced, you'll see a log message that says
64-
``Message delivered to Kafka event bus: topic=dev-events-testing``.
65-
You should also now see the ``dev-events-testing`` topic available in your local confluent
66-
control center, and even the test events that are being published to the topic.
20+
$ make dev.shell
21+
# make validate # run the full test and quality suite
6722

6823
A note on creating SubsidyRequestCustomerConfiguration Objects locally
6924
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
7025

7126
*Important note*
7227

73-
In a devstack enviroment, login to the LMS and navigate to any
28+
In a devstack environment, login to the LMS and navigate to any
7429
MFE before creating SubsidyRequestCustomerConfiguration objects in the
7530
enterprise-access Django admin.
7631

0 commit comments

Comments
 (0)