[Logging] Streamline logging methodologies

[Olshansk]

Objective

Document/consolidate/fix all the different ways of labelling.

Origin Document

While debugging, I found myself looking at this screen and am unsure why things are different as a core contributor to the project:

Example:

This logline: p.logger.Info().Int64("height", p.Height).Msg("About to release context")
Has this output: {"level":"info","module":"persistence","height":0,"time":"2023-03-22T16:14:03Z","message":"About to release context"}

This logline: m.logger.Warn().Fields(m.sharedLoggingFields()).Msgf("⏰ Interrupt ⏰ due to: %s", reason)
Has this output: level=WARN ⏰ Interrupt ⏰ due to: pacemaker timeout height=0 module=pacemaker round=7 step=1

Goals

• Investigate all the different ways things are being logged in the system
• Consolidate all the different logging mechanism and/or fix them and/or document
• Understanding what is the reason for the information in the screenshot above

Deliverable

• Updated documentation on events vs logs and where/when each one should be used
• Consolidation of attributes & labels
• A PR that prevents json objects from being logged to stdout
• Make sure all the documentation and details from [Telemetry] Dashboard for LocalNet logs viewing & filtering #189 are readily available and accessible

Non-goals / Non-deliverables

• Adding/removing existing logs
• Making sure all logs use attributes appropriately if it's embedded in the message right now

General issue deliverables

• Update the appropriate CHANGELOG(s)
• Update any relevant local/global README(s)

Testing Methodology

• All tests: make test_all
• LocalNet: verify a LocalNet is still functioning correctly by following the instructions at docs/development/README.md

---

Creator: @Olshansk
Co-Owners: @okdas

[jessicadaugherty]

Hey! This is currently in up next but hasn't been discussed in planning for a while. Do we want to resurface the conversation or move this to the backlog?

[Olshansk]

Hey! This is currently in up next but hasn't been discussed in planning for a while. Do we want to resurface the conversation or move this to the backlog?

Moved to backlog

[okdas]

I moved this to the Devnet of Servicers milestone as we're focusing on observability at the moment.