Since you all process large amounts of data: I’ve been working on a new memory profiler that gives you peak memory usage, and where all the allocations came from. So you can figure out exactly what you need to change to reduce memory usage. I just released it as open source, so would love feedback about ways I can make more useful for you.
You can learn more about it here: https://pythonspeed.com/products/filmemoryprofiler/