I had a similar issue with a different problem see here. By monitoring the dask dashboard I could see that workers were consuming more memory than I stated for the memory limit.
As a test you could try reducing the memory limit and monitor the dashboard to see how much memory is actually being consumed, as Guy suggested.