mercredi 15 mai 2019

how to protect imports from multiprocessing child processes

How to protect imports from child processes when using multiprocessing without having to put them in if __name__ == '__main__'.

If I am running 8 processes, imports that weight 2MB in total, jam memory for additional 16MB.

I tried multiprocessing.set_start_method('spawn',force=True) and placing imports in if __name__ == '__main__'. This works well but there are some problems with importing methods somewhere else or using imported decorators.

Aucun commentaire:

Enregistrer un commentaire