Concurrent versus Parallel

[Pasted here by request when talking about kerchunk…]

I’d like to take this moment to explain the difference between “parallelism” and “concurrency” and why we are doing both. This comes up a lot and is lost on most, but very cool for those that get it.

“Parallelism” (preemptive multitasking, usually means dask) is when multiple CPU cores are doing work. It can loosely also refer to multiple threads on a single CPU, but in that case, only one is really running at a time. If you have 8 cores, you might get 8x the work IF CPU cycles are your limit - which is not the case for IO. However, if you have 8 whole machines, you might well get 8x the network throughput and so 8x the performance for data loading. Even on one machine and one network interface, IO will presumably be followed by processing, so parallelism is still critical.

“Concurrency” (cooperative multitasking, usually means asyncio) is when you have some tasks that are waiting on external events most of the time. This is waiting on the server to send stuff. You can have a huge number of tasks waiting. Since requesting data from remote has a high latency (time until first byte), you can wait on all the requests at once, so you pay the latency cost once, however many requests are in flight. This is citical for many small reads, where for each request, the data transfer time is a small fraction of the request wall time.

So kerchunk/zarr in dask allows for independent, parallel, tasks (perhaps distributed in a cluster) loading many chunks each concurrently.

7 Likes

Great post Martin.

It would be awesome to turn this explanation into a graphic.

I’ll think abut that, but I don’t have an immediate idea of what such a graphic would look like.

Neither do I! That’s why we need to get some talented graphics people into this community. :wink:

Great post Martin,

Maybe this definition I found in “An introduction to Parallel Programming - Peter Pacheco - Pag 9” may also help. Sorry for this indiscriminated copy - paste of the whole section, but it is necessary so that everyone can draw his own interpretation.

"If you look at some other books on parallel computing or you search the Web
for information on parallel computing, you’re likely to also run across the terms
concurrent computing and distributed computing. Although there isn’t complete
agreement on the distinction between the terms parallel, distributed, and concurrent,
many authors make the following distinctions:
. In concurrent computing, a program is one in which multiple tasks can be in progress
at any instant [4].
. In parallel computing, a program is one in which multiple tasks cooperate closely
to solve a problem.
. In distributed computing, a program may need to cooperate with other programs
to solve a problem.

So parallel and distributed programs are concurrent, but a program such as a mul-
titasking operating system is also concurrent, even when it is run on a machine with

only one core, since multiple tasks can be in progress at any instant. There isn’t a
clear-cut distinction between parallel and distributed programs"

I’m not sure I agree with those statements! I know, I know, semantics is hard. “concurrent” and “parallel” obviously both mean “at the same time”, so it’s about the definitions you choose to live with. Perhaps it only works to assign the execution model directly with the implementation, to make it concrete.

concurrent computing, a program is one in which multiple tasks can be in progress at any instant

If this is asyncio, only one task is running at a time, but many tasks might be ready and waiting to run, perhaps partially executed. asyncio is a single-threaded thing, so you don’t pay the cost of thread switching, context switching and the GIL.

parallel computing, a program is one in which multiple tasks cooperate closely

Definitely not, the tasks executed by something like dask execute independently; whereas coordination/scheduling is something that all of the systems need to do. Perhaps the author is thinking explicitly of MPI, where coordination is part of the program.

distributed computing, a program may need to cooperate with other programs

I’m not sure what this means. I would say more simply, that distributed computing is in which we control make things run on several machines. That, to me, implies parallelism.

I agree that a modern OS is both concurrent and parallel. It runs by preemptive multitasking, threads get time-slices and then are suspended, but they can also yield control and set themselves to wait on IO channels. The OS does not enable in-thread management of IO-bound tasks, beyond allowing a wait (poll) on many channels at once.

1 Like

hello everyone,

I’ll leave this link here, which gives a graphical introduction to the terms in this discussion with a little bit more emphasis on what async is.

2 Likes

That’s a nicely described article

1 Like