[Bug] Console disconnects shortly after connecting

[awangatboo]

Frist off Love PegaProx you all are doing a great job.

I have a new cluster of 3 proxmox nodes. All the proxmox nodes are new and have self signed certificates. PegaProx is running as a VM inside the environment. I have updated Proxmox and PegaProx to the latest version as a few minutes ago. System using CEPH for storage.

PegaProx Beta 0.9.6
Build: 2026.04.12

The issue is that I can start a console and within a few minutes at most the connection errors.

within a few minutes I get this error.

What I see in the logs makes me think PegaProx kicks a job off to one of the proxmox nodes to start the KVM session. The console connects and a little bit later the Error.

When I look at the proxmox logs I see info such is this.

Apr 14 14:33:12 jmcpve1 pvedaemon[122862]: starting vnc proxy UPID:jmcpve1:0001DFEE:00116FF8:69DE9678:vncproxy:100:root@pam!pegaprox_167366:
Apr 14 14:33:12 jmcpve1 pvedaemon[112786]: <root@pam!pegaprox_167366> starting task UPID:jmcpve1:0001DFEE:00116FF8:69DE9678:vncproxy:100:root@pam!pegaprox_167366:
Apr 14 14:33:12 jmcpve1 pvedaemon[112786]: root@pam successful auth for user 'root@pam'
Apr 14 14:33:12 jmcpve1 pvedaemon[122868]: starting vnc proxy UPID:jmcpve1:0001DFF4:0011700F:69DE9678:vncproxy:100:root@pam:
Apr 14 14:33:12 jmcpve1 pvedaemon[2390]: root@pam starting task UPID:jmcpve1:0001DFF4:0011700F:69DE9678:vncproxy:100:root@pam:
Apr 14 14:33:15 jmcpve1 pveproxy[120095]: worker exit
Apr 14 14:33:15 jmcpve1 pveproxy[3543]: worker 120095 finished
Apr 14 14:33:15 jmcpve1 pveproxy[3543]: starting 1 worker(s)
Apr 14 14:33:15 jmcpve1 pveproxy[3543]: worker 122890 started
Apr 14 14:33:20 jmcpve1 pveproxy[122890]: Clearing outdated entries from certificate cache
Apr 14 14:33:22 jmcpve1 pvedaemon[122862]: connection timed out
Apr 14 14:33:22 jmcpve1 pvedaemon[112786]: <root@pam!pegaprox_167366> end task UPID:jmcpve1:0001DFEE:00116FF8:69DE9678:vncproxy:100:root@pam!pegaprox_167366: connection timed out
Apr 14 14:33:34 jmcpve1 pvedaemon[2390]: root@pam end task UPID:jmcpve1:0001DFF4:0011700F:69DE9678:vncproxy:100:root@pam: OK
Apr 14 14:34:03 jmcpve1 pveproxy[119306]: worker exit
Apr 14 14:34:03 jmcpve1 pveproxy[3543]: worker 119306 finished
Apr 14 14:34:03 jmcpve1 pveproxy[3543]: starting 1 worker(s)
Apr 14 14:34:03 jmcpve1 pveproxy[3543]: worker 123407 started
Apr 14 14:34:03 jmcpve1 pveproxy[123407]: Clearing outdated entries from certificate cache

I have noticed that the error is really quick when the VM is not on the node that PegaProx sends the request to. In my environment if the request for proxy is sent to node1 and the VM that is on node3 then it typically takes 10-15 seconds before I get the error.

Being a Unix guy myself it is almost like a process ends and because the parent process stops the child process stops causing the timeout.

I noticed that pveproxy and pvedaemon were the ones complaining and decided to increase the MAX_WORKERS to see if that helped and it did not.

I also have the issue when I try to connect to "external" it never does. Here is the error I get when I do that.

Here is the Logs when I click that.

Apr 14 16:23:53 jmcpve2 sshd-session[15375]: Accepted publickey for root from 10.49.32.157 port 40854 ssh2: RSA SHA256:eIJyCBMtjwGXnIEqavY7nWGZWw8l5Um523zc91o1iWs
Apr 14 16:23:53 jmcpve2 sshd-session[15375]: pam_unix(sshd:session): session opened for user root(uid=0) by root(uid=0)
Apr 14 16:23:53 jmcpve2 systemd[1]: Created slice user-0.slice - User Slice of UID 0.
Apr 14 16:23:53 jmcpve2 systemd[1]: Starting user-runtime-dir@0.service - User Runtime Directory /run/user/0...
Apr 14 16:23:53 jmcpve2 systemd-logind[1620]: New session 22 of user root.
Apr 14 16:23:53 jmcpve2 systemd[1]: Finished user-runtime-dir@0.service - User Runtime Directory /run/user/0.
Apr 14 16:23:53 jmcpve2 systemd[1]: Starting user@0.service - User Manager for UID 0...
Apr 14 16:23:53 jmcpve2 (systemd)[15382]: pam_unix(systemd-user:session): session opened for user root(uid=0) by root(uid=0)
Apr 14 16:23:53 jmcpve2 systemd-logind[1620]: New session 23 of user root.
Apr 14 16:23:53 jmcpve2 systemd[15382]: Queued start job for default target default.target.
Apr 14 16:23:53 jmcpve2 systemd[15382]: Created slice app.slice - User Application Slice.
Apr 14 16:23:53 jmcpve2 systemd[15382]: Reached target paths.target - Paths.
Apr 14 16:23:53 jmcpve2 systemd[15382]: Reached target timers.target - Timers.
Apr 14 16:23:53 jmcpve2 systemd[15382]: Listening on dirmngr.socket - GnuPG network certificate management daemon.
Apr 14 16:23:53 jmcpve2 systemd[15382]: Listening on gpg-agent-browser.socket - GnuPG cryptographic agent and passphrase cache (access for web browsers).
Apr 14 16:23:53 jmcpve2 systemd[15382]: Listening on gpg-agent-extra.socket - GnuPG cryptographic agent and passphrase cache (restricted).
Apr 14 16:23:53 jmcpve2 systemd[15382]: Starting gpg-agent-ssh.socket - GnuPG cryptographic agent (ssh-agent emulation)...
Apr 14 16:23:53 jmcpve2 systemd[15382]: Starting gpg-agent.socket - GnuPG cryptographic agent and passphrase cache...
Apr 14 16:23:53 jmcpve2 systemd[15382]: Listening on keyboxd.socket - GnuPG public key management service.
Apr 14 16:23:53 jmcpve2 systemd[15382]: Starting ssh-agent.socket - OpenSSH Agent socket...
Apr 14 16:23:53 jmcpve2 systemd[15382]: Listening on gpg-agent-ssh.socket - GnuPG cryptographic agent (ssh-agent emulation).
Apr 14 16:23:53 jmcpve2 systemd[15382]: Listening on gpg-agent.socket - GnuPG cryptographic agent and passphrase cache.
Apr 14 16:23:53 jmcpve2 systemd[15382]: Listening on ssh-agent.socket - OpenSSH Agent socket.
Apr 14 16:23:53 jmcpve2 systemd[15382]: Reached target sockets.target - Sockets.
Apr 14 16:23:53 jmcpve2 systemd[15382]: Reached target basic.target - Basic System.
Apr 14 16:23:53 jmcpve2 systemd[15382]: Reached target default.target - Main User Target.
Apr 14 16:23:53 jmcpve2 systemd[15382]: Startup finished in 127ms.
Apr 14 16:23:53 jmcpve2 systemd[1]: Started user@0.service - User Manager for UID 0.
Apr 14 16:23:53 jmcpve2 systemd[1]: Started session-22.scope - Session 22 of User root.

This could be two different bugs but I think they might be one in the same just showing up differently.

What I expect is once the console starts I do not have any issue with it. I know if I login to Proxmox I can start a Console and leave it over night and it is still running the next day.

This project plus ProxMox in a cluster gives competition to the big players and I would love to help debug in any way I can. Also as a note I have ~6 VM's running on my cluster and it is currently in a test phase so I can change anything.

Environment
PegaProx Version: 0.9.6
Installation Method: VM restore from PegaProx website.
Client OS: Microslop Windows 11.
Browser: 145.0.7632.117 (Official Build) (64-bit)
Behind Reverse Proxy? No
UI? Happens in both Corporate and Modern Layout

• I have searched existing issues to make sure this is not a duplicate ( this is somewhat a duplicate but the other have been closed and the bug still exists. )
• I am using the latest version of PegaProx

[mkellermann97]

Hey @awangatboo, thanks for the detailed logs — that actually helps a lot.

The connection timed out after ~10s on pvedaemon is the key clue here. What's happening is that PegaProx requests a VNC proxy ticket from the Proxmox API, but the WebSocket connection from PegaProx to the node doesn't get established fast enough (or at all), so pvedaemon kills the VNC session.

Two things to check:

1. Since you're running PegaProx as a VM inside the cluster — are you connecting to the nodes by hostname or IP? If by hostname, make sure PegaProx can resolve all node hostnames correctly (ping jmcpve1 etc from inside the PegaProx VM). DNS resolution delays are a common cause for this.

2. The self-signed certs — PegaProx needs to reach the Proxmox API on port 8006 via HTTPS. Did you add the cluster with SSL verification disabled? If not, the TLS handshake might be failing silently.

The cross-node issue you noticed (VM on node3, request goes to node1) is expected — Proxmox handles the proxy forwarding internally, but that adds another hop which makes the timeout window tighter.

For the "external" console (SSH) — that's a separate codepath. The SSH session connects fine (your logs show successful pubkey auth) but the terminal WebSocket probably can't bind. Can you check if port 3001 is reachable from your browser to the PegaProx VM?

We're looking into improving the VNC proxy timeout handling in the next patch. If you can share the PegaProx server logs (logs/clusterid.log) when the disconnect happens that would narrow it down further.

Btw if you want to support the project — we're volunteer-driven and every bit helps: https://opencollective.com/pegaprox

– Marcus

[awangatboo]

I am not doing this in order but I started with the "external" part first.

I ran a powershell to test to see if I could reach port 3001 on pegaprox.

PS C:\windows\system32> Test-NetConnection -ComputerName pegaprox -port 3001 WARNING: TCP connect to (10.49.34.77 : 3001) failed
ComputerName : pegaprox
RemoteAddress : 10.49.34.77
RemotePort : 3001
InterfaceAlias : Ethernet
SourceAddress : 10.133.128.27
PingSucceeded : True
PingReplyDetails (RTT) : 0 ms
TcpTestSucceeded : False

Then I logged into the PegaProx VM and checked to see if anything was listening.

root@pegaprox-prod:/home/pegaprox_admin# netstat -tpnl
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:5001 0.0.0.0:* LISTEN 7037/python
tcp 0 0 0.0.0.0:5002 0.0.0.0:* LISTEN 7045/python
tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 8678/sshd-session:
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 655/sshd: /usr/sbin
tcp 0 0 0.0.0.0:1514 0.0.0.0:* LISTEN 7037/python
tcp6 0 0 :::5000 :::* LISTEN 7037/python
tcp6 0 0 :::5001 :::* LISTEN 7037/python
tcp6 0 0 :::5002 :::* LISTEN 7045/python
tcp6 0 0 ::1:6010 :::* LISTEN 8678/sshd-session:
tcp6 0 0 :::22 :::* LISTEN 655/sshd: /usr/sbin
root@pegaprox-prod:/home/pegaprox_admin#

I do not see anything listening on port 3001. Now that might be something that gets spun up when starting the console but nothing it listening by default.

[ccesario]

Hi @mkellermann97,

Im getting the same error in my envinroment - 0.9.6.1.

Regards

[awangatboo]

Working on the part that is just the internal console that bugs out.

I logged into my PregaProx VM and tried to ping the name listed as the node. The node is listed as "jmcpve1". I used 'getent hosts' with the short name and nothing was returned. I then put in the fully qualified domain and it responded. I updated my /etc/resolve.conf with a search line and rebooted the node. Sadly this did not fix the issue. I will do some more troubleshooting and give more logs. in another comment.

@ccesario Are you running PegaProx in a container or a VM? Also you say the same issue is that the issue of console dropping, issue of external console, or both?

[ccesario]

Hi @awangatboo,
Im running PegaProx in a VM.

I have another bug case with more details. #92

Regards
Carlos

[awangatboo]

Sorry but I have not gotten to getting log data today. I will do more test and post more tomorrow.

[awangatboo]

I have been working on getting this working all day but I wanted to start out by showing my cluster info. I am not using the SSL verification.

So that should not matter to the "time out". I saw somewhere else that SSL/TLS filtering can cause issues like this for proxmox. I am going to check with my network team to see if that have anything like this turned on between subnets. Pegaprox and the ProxMox nodes are on separate VLANs and the traffic is routed.

[awangatboo]

OK so I am trying to understand this so I can get to the bottom of the issue. I realized the difference between kicking the console off from Proxmox and kicking it off from PegaProx.

When I kick off the console from ProxMox I get this bar at the top.

When I kick it off from PegaProx I never see that label at the top. So how does this work? Does Pegaprox send the keys back and forth between the client and the Proxmox host? Does Pegaprox send the keys back and for between Proxmox and the PegaProx server and then just show the results? I would love to get to the bottom of this. Are there any logs on the PegaProx side I can look at? Some of these questions are me just typing them out and If I get an answer then great but I am going to try to figure out my answer.

[MrMasterbay]

Hi @awangatboo ,

please note as stated also on our bug report that we act on "best effort" behavior.
Which means sometimes we simple just don't have the time to answer because we are all volunteers and currently most of team has other priorities(busy month). Due our completely free and open-source approach.

PegaProx acts here as an proxy provider that is put in place. You simple don't see the message because pegaprox doesn't send it with it. It runs on a WS Ticket exchange that goes forth and back. We would also need the f12 console output when a connection drops as we then simple can analyze this further.

Regards

Want to help PegaProx out? Feel free to leave us a Donation on Open Collective!

[awangatboo]

@MrMasterbay

I am fine with the effort you have put into pegaprox and this ticket. Sometimes I just type these things out so anyone looking at the ticket can follow my logic and hopefully see that I am trying to learn enough to be helpful in my troubleshooting. Thank you for all you do on this Project.

If I understand this correctly I need to get the log info from the proxmox system that dropped as well as anything that would be on the F12 copy of the console for the pegaprox server? tty12? If this is the case I am going to have issue as there only seams to be tty1-6 on the pegaprox VM.

[awangatboo]

Looking it up F12 looks like something from my browser. I will see what I can figure out with this to get you the log info you want.

[awangatboo]

Here is what I see on the F12 for the bouser. I hope this is the log that was wanted. Also this is F12 for when the console errors out. Not the F12 for when I click the "external" button.