Problems when trying generate data

[Hu-Yuch]

Your program get stuck in this step when run "python scalingup/inference.py evaluation.num_episodes=50000 policy=scalingup evaluation=drawer",how can I deal with it

?

[clorislili]

True. The speed of data generation process is quite slow. Any ways to enhance its speed? Thanks

[Louis-ZhangLe]

Your program get stuck in this step when run "python scalingup/inference.py evaluation.num_episodes=50000 policy=scalingup evaluation=drawer",how can I deal with it ?

do you use the CACHE?

[huy-ha]

Hey @h129344010 and @clorislili ,

As @Louis-ZhangLe mentioned, if it's getting stuck without running, it's probably trying to make lots of OpenAI API calls before running the simulation. You can follow instructions here if you don't have the cache setup. The latest version of the codebase is not working with the cache (#22 ) but the fix is to just checkout the earliest version of this codebase.

If you are already using cache, then rendering could help speed up simulation significantly. This is because rendering is used to generate pointclouds, which is used by the grasp and placement samplers.

From your logger output @h129344010 , it looks like you're ssh-ing into the node you're using to run data generation with X11 forwarding. I might be wrong, but this could slow down rendering significantly. Further, glfw doesn't work well with X11 forwarding (it failed to open the display in the log, so I suspect your pointclouds are actually coming out blank).

In our headless servers, we installed VirtualGL, which allows us to run headless rendering on the GPU. This is my recommendation. If you don't want to go through the hassle of installing VirtualGL, you could rely on a different OpenGL backend other than glfw. For instance, running the data generation with MUJOCO_GL=osmesa or MUJOCO_GL=egl in the environment variables should work well with CPU only rendering.

Hope this helps!