climateprediction.net (CPDN) home page
Thread 'Almost 2025. Why doesn’t this project support multithreading?'

Thread 'Almost 2025. Why doesn’t this project support multithreading?'

Message boards : Number crunching : Almost 2025. Why doesn’t this project support multithreading?
Message board moderation

To post messages, you must log in.

AuthorMessage
Luca

Send message
Joined: 20 Jan 23
Posts: 3
Credit: 338,203
RAC: 1,606
Message 71804 - Posted: 3 Nov 2024, 14:18:55 UTC

As title.

It's almost 2025. Why doesn’t this project support multithreading?

The WUs are extremely long.
ID: 71804 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4540
Credit: 19,039,635
RAC: 18,944
Message 71805 - Posted: 3 Nov 2024, 15:33:35 UTC - in response to Message 71804.  
Last modified: 3 Nov 2024, 16:31:33 UTC

There are multi-threaded tasks on the way. They will be the OIFS code from ECMWF. Glenn posted a link to the program for Linux that you can run in a terminal on Linux recently and play around with the file that determines how many cores it uses. Most of the work however is still the Met Office code which is all 32bit which says something about its age in itself. Initially at least the multithreaded apps will like the other OIFS work all be Linux only.

The answer to your query however is that the project hasn't had a scientist with the required programming skills to write multi-threaded apps till Glenn came on board as a volunteer scientist. I don't know whether Andy who does the sysadmin work has the skills but his other work for the project is full time and would not allow him the space to develop the multithreaded apps.

This is the link to the thread with discussion about the multi-core app and the link to try it if you have access to a Linux box.

Interestingly, I would say that less than a quarter of the projects I have looked at have multithreaded apps and even those that do, don't allow it on all task types.

Edit: If I have missed anything important I am sure another moderator or Glenn will add to this but I can't imagine it is straightforward to modify the Met Office code to make it multithreaded even if that is possible and that is the code most of the scientists around the world are still using for their tasks.
ID: 71805 · Report as offensive     Reply Quote
Glenn Carver

Send message
Joined: 29 Oct 17
Posts: 1049
Credit: 16,575,067
RAC: 15,735
Message 71818 - Posted: 4 Nov 2024, 14:50:55 UTC - in response to Message 71804.  

As title.
It's almost 2025. Why doesn’t this project support multithreading?
The WUs are extremely long.
Depends what you mean by 'extremely long'. With a recent processor and keeping the machine on 24/7, most workunits finish in about 2 days. I switch mine off overnight so it'll take 10 days or so.

But if you prefer seeing tasks finish after a hour or two, CPDN might not be the project for you. When we introduce multithreaded apps, it will be to allow for more complex configurations to ensure they finish in a reasonable time not make the existing configurations complete quicker. Running single threaded is still the most efficient way of running the models.

All the meteorological codes that CPDN run are capable of multiprocessing, even the older ones. We disable it because it reduces the memory footprint of the models and makes their implementation in boinc simpler.
---
CPDN Visiting Scientist
ID: 71818 · Report as offensive     Reply Quote
wujj123456

Send message
Joined: 14 Sep 08
Posts: 127
Credit: 42,506,888
RAC: 76,047
Message 71822 - Posted: 4 Nov 2024, 18:16:19 UTC - in response to Message 71818.  
Last modified: 4 Nov 2024, 18:16:27 UTC

All the meteorological codes that CPDN run are capable of multiprocessing, even the older ones. We disable it because it reduces the memory footprint of the models and makes their implementation in boinc simpler.

Interesting. I always thought it's the additional work in code base being the problem. If multi-thread mode is validated anyway, I'd love to have the option through cc_config.xml to pass in the right arguments. (Or is it possible already, just not advertised so that we don't know?) That way, the project doesn't need to onboard any new complexity from server side, but the volunteers who have the need or preference can customize on their end. It can be helpful for shortening the runtime, or better utilize the cores when the host is memory bound but not running other projects.
ID: 71822 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4540
Credit: 19,039,635
RAC: 18,944
Message 71823 - Posted: 4 Nov 2024, 18:43:21 UTC

All the meteorological codes that CPDN run are capable of multiprocessing, even the older ones.
Of course they are capable of multiprocessing. They were written to run on supercomputers. I blame my last post on having a head stuffed full of cold at the moment!

I can see that total throughput of tasks might be more without multithreading. However when there are relatively small batches and the first few hundred computers grab them all, then multithreading would spread the tasks out between more computers and would get them returned more quickly.
ID: 71823 · Report as offensive     Reply Quote
Glenn Carver

Send message
Joined: 29 Oct 17
Posts: 1049
Credit: 16,575,067
RAC: 15,735
Message 71824 - Posted: 4 Nov 2024, 19:10:59 UTC - in response to Message 71823.  
Last modified: 4 Nov 2024, 19:11:17 UTC

Sorry, enabling & testing the multiprocessing in the older models is not something I'm going to spend time doing. They work fine as they are. I have more pressing things to do.
---
CPDN Visiting Scientist
ID: 71824 · Report as offensive     Reply Quote
ProfileDave Jackson
Volunteer moderator

Send message
Joined: 15 May 09
Posts: 4540
Credit: 19,039,635
RAC: 18,944
Message 71825 - Posted: 4 Nov 2024, 20:30:04 UTC - in response to Message 71824.  

Sorry, enabling & testing the multiprocessing in the older models is not something I'm going to spend time doing. They work fine as they are. I have more pressing things to do.

Fair enough. There are always going to be priorities that we won't know anything about. If climate science had the funding and resources it deserves....
ID: 71825 · Report as offensive     Reply Quote

Message boards : Number crunching : Almost 2025. Why doesn’t this project support multithreading?

©2024 cpdn.org