NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.

[hanhanhanhanhan929]

你好，我想问在分布式微调的时候出现了这个错误
Traceback (most recent call last):
[rank0]: File "/data/hanrui/dz/GeoX-main/main/train_gsformer.py", line 103, in
[rank0]: main()
[rank0]: File "/data/hanrui/dz/GeoX-main/main/train_gsformer.py", line 99, in main
[rank0]: runner.train()
[rank0]: File "/data/hanrui/dz/GeoX-main/lavis/runners/runner_base.py", line 386, in train
[rank0]: train_stats = self.train_epoch(cur_epoch)
[rank0]: File "/data/hanrui/dz/GeoX-main/lavis/runners/runner_base.py", line 453, in train_epoch
[rank0]: self.model.train()
[rank0]: File "/data/hanrui/dz/GeoX-main/lavis/runners/runner_base.py", line 86, in model
[rank0]: self._model = self._model.to(self.device)
[rank0]: File "/data/hanrui/conda/glm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1355, in to
[rank0]: return self._apply(convert)
[rank0]: File "/data/hanrui/conda/glm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 915, in _apply
[rank0]: module._apply(fn)
[rank0]: File "/data/hanrui/conda/glm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 915, in _apply
[rank0]: module._apply(fn)
[rank0]: File "/data/hanrui/conda/glm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 915, in _apply
[rank0]: module._apply(fn)
[rank0]: [Previous line repeated 2 more times]
[rank0]: File "/data/hanrui/conda/glm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 942, in _apply
[rank0]: param_applied = fn(param)
[rank0]: File "/data/hanrui/conda/glm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1348, in convert
[rank0]: raise NotImplementedError(
[rank0]: NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.