PicklingError: Can't pickle <function DataManagerOpenCell.<lambda>

[gitclipp]

Dear Author,

I tried reproducing the steps from the readme file, but I received an error.
at step 2 the line
trainer.fit(datamanager, tensorboard_path='tb_logs')
throws the following error:

PicklingError: Can't pickle <function DataManagerOpenCell. at 0x000001070AC03550>: attribute lookup DataManagerOpenCell. on cytoself.datamanager.opencell failed.

I get the following Traceback:

PicklingError Traceback (most recent call last)
Cell In[16], line 20
12 train_args = {
13 'lr': 1e-3,
14 'max_epoch': 1,
(...)
17 'earlystop_patience': 6,
18 }
19 trainer = CytoselfFullTrainer(train_args, homepath='demo_output', model_args=model_args)
---> 20 trainer.fit(datamanager, tensorboard_path='tb_logs')

File D:\MA\CytoSelf\cytoself-main\cytoself-main\cytoself\trainer\basetrainer.py:427, in BaseTrainer.fit(self, datamanager, initial_epoch, tensorboard_path, **kwargs)
425 # Train the model
426 self.model.train(True)
--> 427 train_metrics = self.run_one_epoch(datamanager, 'train', **kwargs)
428 self.model.train(False)
430 # Validate the model

File D:\MA\CytoSelf\cytoself-main\cytoself-main\cytoself\trainer\vqvae_trainer.py:192, in VQVAETrainer.run_one_epoch(self, datamanager, phase, **kwargs)
190 raise ValueError('phase only accepts train, val or test.')
191 _metrics = []
--> 192 for _batch in tqdm(data_loader, desc=f'{phase.capitalize():>5}'):
193 loss = self.run_one_batch(
194 _batch, var, zero_grad=is_train, backward=is_train, optimize=is_train, **kwargs
195 )
196 _metrics.append(loss)

File D:\Programme\Miniconda\envs\cytoself\lib\site-packages\tqdm\std.py:1182, in tqdm.iter(self)
1179 time = self._time
1181 try:
-> 1182 for obj in iterable:
1183 yield obj
1184 # Update and possibly print the progressbar.
1185 # Note: does not call self.update(1) for speed optimisation.

File D:\Programme\Miniconda\envs\cytoself\lib\site-packages\torch\utils\data\dataloader.py:438, in DataLoader.iter(self)
436 return self._iterator
437 else:
--> 438 return self._get_iterator()

File D:\Programme\Miniconda\envs\cytoself\lib\site-packages\torch\utils\data\dataloader.py:386, in DataLoader._get_iterator(self)
384 else:
385 self.check_worker_number_rationality()
--> 386 return _MultiProcessingDataLoaderIter(self)

File D:\Programme\Miniconda\envs\cytoself\lib\site-packages\torch\utils\data\dataloader.py:1039, in _MultiProcessingDataLoaderIter.init(self, loader)
1032 w.daemon = True
1033 # NB: Process.start() actually take some time as it needs to
1034 # start a process and pass the arguments over via a pipe.
1035 # Therefore, we only add a worker to self._workers list after
1036 # it started, so that we do not call .join() if program dies
1037 # before it starts, and del tries to join but will get:
1038 # AssertionError: can only join a started process.
-> 1039 w.start()
1040 self._index_queues.append(index_queue)
1041 self._workers.append(w)

File D:\Programme\Miniconda\envs\cytoself\lib\multiprocessing\process.py:121, in BaseProcess.start(self)
118 assert not _current_process._config.get('daemon'),
119 'daemonic processes are not allowed to have children'
120 _cleanup()
--> 121 self._popen = self._Popen(self)
122 self._sentinel = self._popen.sentinel
123 # Avoid a refcycle if the target function holds an indirect
124 # reference to the process object (see bpo-30775)

File D:\Programme\Miniconda\envs\cytoself\lib\multiprocessing\context.py:224, in Process._Popen(process_obj)
222 @staticmethod
223 def _Popen(process_obj):
--> 224 return _default_context.get_context().Process._Popen(process_obj)

File D:\Programme\Miniconda\envs\cytoself\lib\multiprocessing\context.py:327, in SpawnProcess._Popen(process_obj)
324 @staticmethod
325 def _Popen(process_obj):
326 from .popen_spawn_win32 import Popen
--> 327 return Popen(process_obj)

File D:\Programme\Miniconda\envs\cytoself\lib\multiprocessing\popen_spawn_win32.py:93, in Popen.init(self, process_obj)
91 try:
92 reduction.dump(prep_data, to_child)
---> 93 reduction.dump(process_obj, to_child)
94 finally:
95 set_spawning_popen(None)

File D:\Programme\Miniconda\envs\cytoself\lib\multiprocessing\reduction.py:60, in dump(obj, file, protocol)
58 def dump(obj, file, protocol=None):
59 '''Replacement for pickle.dump() using ForkingPickler.'''
---> 60 ForkingPickler(file, protocol).dump(obj)

PicklingError: Can't pickle <function DataManagerOpenCell. at 0x000001070AC03550>: attribute lookup DataManagerOpenCell. on cytoself.datamanager.opencell failed

Thank You for Your time,

[gitclipp]

The error can be avoided by removing parallel processing from the dataloader like shown in the example below:

from cytoself.datamanager.opencell import DataManagerOpenCell

data_ch = ['pro', 'nuc']
datapath = 'Testdata' # path to download sample data
DataManagerOpenCell.download_sample_data(datapath) # donwload data
datamanager = DataManagerOpenCell(datapath, data_ch, fov_col=None)
datamanager.const_dataloader(batch_size=32, num_workers=1, label_name_position=1)
datamanager.train_loader.num_workers=0
datamanager.val_loader.num_workers=0
datamanager.test_loader.num_workers=0

[li-li-github]

Thank you @gitclipp for finding a solution. This seems a specific problem on windows. I have no problem running it on Ubuntu, and I do have problem of Can't pickle on windows. I've also seen issues on multiprocessing when running pytorch on windows. There might be some compatibility issues between recent python's multiprocessing and windows.

[li-li-github]

I mentioned this issue in the readme and linked your solution as a temporal workaround. @gitclipp