docs: remove log_cli and log_file ini options; keep them as cli arguments for integration tests by tromai · Pull Request #2654 · canonical/operator

tromai · 2026-07-20T22:56:33Z

In #2619, I added the Configure Jubilant logs howto section.

In this guide, I set log_cli and log_file in the [tool.pytest.ini_options] table in pyproject.toml.

log_cli = true enables live logging for all pytest invocation.
log_file = <path> enables file logging for all pytest invocation.

This works well for tox -e integration. However, as a side effect, tox -e unit emitted live logs from the charm's code. This was mentioned by @dwilding.

tests/unit/test_charm.py::test_one PASSED
tests/unit/test_charm.py::test_two 
------------------------- live log call -------------------------
ERROR <error_level_message>
PASSED

We don't want this behaviour in unit tests.

This PR fixes the howto guide:

Remove log_cli = true, log_cli_level = <level> and log_file = <path> from all suggested configs.
Add instructions to enable them from the CLI:
- tox -e integration provides --log-cli-level=INFO to the pytest command, which enable live logging.
- tox -e integration -- --log-file logs/verbose.log will emit the verbose logs to logs/verbose.log. This requires changes to the GitHub Action workflow example.
- For juju debug-log from pytest-jubilant the command will be tox -e integration -- --log-file logs/verbose.log --juju-dump-logs logs. See a test run here

Preview

…ents for integration tests

tromai · 2026-07-21T02:34:15Z

 ```

-You can also configure pytest logging options on the command line.
+You can also configure these options on the command line. If you don't see the live Jubilant logs, make sure the pytest command (for example, in `tox.ini`) has the `--log-cli-level=INFO` argument.


I changed pytest logging to these to keep it consistent with the same sentence in Log to a file section.

I think "pytest logging options" works better in this case because there's a sample output directly above, and it might be a little unclear what "these" refers to.

For the second sentence, would you mind explaining this to me again? If log_cli_level = "DEBUG" is already set in pyproject.toml, why wouldn't I see the live logs? And why are we suggesting to set it to INFO on the CLI instead of DEBUG?

I think "pytest logging options" works better in this case because there's a sample output directly above, and it might be a little unclear what "these" refers to.

Oh, sorry. Somehow I thought this was a typo made by me in the last PR. I will revert it. I agree that pytest logging works better.

log-cli-level = "DEBUG" is already set in pyproject.toml, why wouldn't I see the live logs?

If you set log_cli_level = <level> and not set log_cli = true in pyproject.toml, live logging is disabled. To enable live logging, without using log_cli = true, we need an equivalent CLI argument. However, there is no --log-cli argument in pytest 😞 (here). The only way we could enable live logging from CLI is to use --log-cli-level=<level>, which implicitly enable live logging.

And why are we suggesting to set it to INFO on the CLI instead of DEBUG?

Ah thanks for pointing this out. In this section, we have 1 example for brief terminal logging and 1 for verbose terminal logging. Because we need --log-cli-level=<level> to enable live logging from CLI, level should be the same level defined by log_cli_level in pyproject.toml.

The sentence I have here is confusing. How about:

If you don't see the verbose Jubilant logs, make sure the pytest command (for example, in `tox.ini`) has the `--log-cli-level=DEBUG` argument.

We don't need to care about brief logs here because the default tox.ini already set that up.

[testenv:integration] ... commands = pytest \ -v \ -s \ --tb native \ --log-cli-level=INFO \ {[vars]tests_path}/integration \ {posargs}

As an additional improvement, we can even remove log_cli_level from pyproject.toml completely, and rely on its CLI version --log-cli-level to prevent confusion.

OK, I think I understand, thanks!

So this is how I understand it:

pyproject.toml:

log_cli_format = "%(levelname)s %(name)s %(message)s"

tox.ini:

[testenv:unit] - No logging options on the pytest command. No live logging. In effect, log_cli_format in pyproject.toml is ignored.

[testenv:integration] - --log-cli-level=INFO on the pytest command. Produces brief logs.

Then for verbose logs during integration tests, we recommend the developer changes two places: Switch to --log-cli-level=DEBUG in tox.ini, and adjust pyproject.toml to have:

log_cli_format = "%(asctime)s %(levelname)s %(name)s %(message)s" log_cli_date_format = "%Y-%m-%dT%H:%M:%SZ"

Did I get that right?

Yes that is correct!

Great 🙂 I would suggest a couple of changes to the doc in that case. Instead of having an "If you don't see..." sentence here, I'd tell people how we expect them to use the pytest command. Something like this (which includes a few other adjustments):

Example of brief logging

In pyproject.toml:

[tool.pytest.ini_options] ... log_cli_format = "%(levelname)s %(name)s %(message)s"

Also check the [testenv:integration] environment in tox.ini to make sure the pytest command has --log-cli-level=INFO.

Sample output of tox -e integration:

[...]

Example of verbose logging

In pyproject.toml:

[tool.pytest.ini_options] ... log_cli_format = "%(asctime)s %(levelname)s %(name)s %(message)s" log_cli_date_format = "%Y-%m-%dT%H:%M:%SZ"

The timestamp format follows ISO 8601.

See more: {external+python:ref}strftime() and strptime() behavior <strftime-strptime-behavior>

Also modify the [testenv:integration] environment in tox.ini so that the pytest command has --log-cli-level=DEBUG.

Sample output of tox -e integration:

[...]

This is great. Thank you so much.

I updated the doc to use this version.

tromai · 2026-07-21T02:36:35Z

+Use the `--log-file` option from pytest to enable file logging:
+
+```text
+tox -e integration -- --log-file logs/verbose.log


From our daily, my impression is that we are okay with leaving --log-file logs/verbose.log here.

When we update the Charmcraft profiles with GitHub Action configuration, --log-file logs/verbose.log can go into tox.ini or Makefile. We want the behaviour to be consistent running the integration tests command both in CI and local.

What would be the downside of adding --log-file logs/verbose.log to tox.ini now rather than later?

Hmmm, there is no practical downside to usability as far as I know of. I think the decision was more about: implementing log to file together with the GitHub Action workflow config and maintaining consistency.

tonyandrewmeyer

Looks good to me, thanks! I like David's suggestion for the configuration, no other comments from me.

dwilding · 2026-07-22T02:39:24Z

@@ -569,16 +573,19 @@ This is an ideal configuration for long-running integration tests (for example,
 # Otherwise, that section will have DEBUG logs (coming from log_file_level).
 log_level = "INFO"


I just had one thought. Does log_level = "INFO" mean it doesn't matter what --log-cli-level is set to in tox.ini? If tox.ini has --log-cli-level=DEBUG, will pytest output brief or verbose logs to the console? Will log_level = "INFO" take precedence?

If log_level = "INFO" doesn't take precedence, I think we need to include the same sentence as earlier:

Also check the [testenv:integration] environment in tox.ini to make sure the pytest command has --log-cli-level=INFO

We could put this immediately after the pyproject.toml sample.

Oh good point. Thank you. --log-cli-level will take precedence.

If tox.ini has --log-cli-level=DEBUG, pytest will output verbose logs to the console.

I will add the sentence in.

Looks good, thanks!

docs: remove log_cli and log_file ini options; keep them as cli argum…

dcab02a

…ents for integration tests

tromai commented Jul 21, 2026

View reviewed changes

Merge branch 'main' into remove-live-logging-from-pytest-config

3d64692

tromai marked this pull request as ready for review July 21, 2026 02:37

tromai requested review from dwilding and tonyandrewmeyer July 21, 2026 02:37

dwilding reviewed Jul 22, 2026

View reviewed changes

Comment thread docs/howto/write-integration-tests-for-a-charm.md Outdated

tonyandrewmeyer approved these changes Jul 22, 2026

View reviewed changes

remove log_cli_level and address pr feedback

c3fc51b

tromai requested a review from dwilding July 22, 2026 02:16

dwilding approved these changes Jul 22, 2026

View reviewed changes

chore: add a note to check --log-cli-level in log to a file mode

21b6819

tromai merged commit f49f4c1 into canonical:main Jul 22, 2026
61 checks passed

tromai deleted the remove-live-logging-from-pytest-config branch July 22, 2026 04:16

tromai mentioned this pull request Jul 22, 2026

Update the pytest logging options in example charm code #2659

Open

		@@ -569,16 +573,19 @@ This is an ideal configuration for long-running integration tests (for example,
		# Otherwise, that section will have DEBUG logs (coming from log_file_level).
		log_level = "INFO"

Uh oh!

Conversation

tromai commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tromai Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Example of brief logging

Example of verbose logging

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tromai Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tonyandrewmeyer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tromai commented Jul 20, 2026 •

edited

Loading

tromai Jul 21, 2026 •

edited

Loading

tromai Jul 21, 2026 •

edited

Loading