Dear WCG developers,

forgive me if this has been asked and discussed before. I tried a quick search but found nothing of value.

The reason why I'm posting is that, as you probably know, WCG Clean Energy Project Phase 2 WUs perform lot's of disk operations. And I mean really a massive amount of them: I measured the amount of transferred data via SSDReady (free version) and running 1 of those tasks continously would yield several 100 GB/day! If I used all 8 logical cores of my hyperthreaded 4.0 GHz i7 those numbers might easily pass 1 TB/day (if the SSD could keep up).

Generally write endurance is no big ussue for SSDs. But consumer drives are usually rated in the double digit TBs written. They'll very likely take more, but that's not guaranteed. And wearing out an SSD in less than a quarter of a year would be very shocking for your users.

I seriously doubt those disk transfers are neccessary in this way. Could you sheed some light on the issue? A few more thoughts from my point:

- using simple zip compression I reduced the data in a CEP2 slot by a factor of 2.1. If you perform this operation in the app (.zip librariers should be readily available) you'd save your users half of the file writes and might even see faster processing, because writing less data physically is always faster.

- I observed almost constant write access via SSDReady, irregardless of my "preferred check point interval" setting. Can't you simply cache those file changes in main memory and only push them out when the user preferenced checkpoint interval has passed? This way users could directly control how much safety against data loss they want to trade off against increased SSD wear. And again you'd probably see a significant speedup, if your app is single threaded and has to wait for those frequent disk transfers to finish before it can continue processing.

- keeping the calculation results / write queue in memory would of course increase memory usage, which can be a factor in some systems. However, in my case I usually have 4 of 8 GB free, including my CPUs & GPUs being fully loaded by BOINC. I could easily afford to spend a few 100 MBs more, even GBs. And you could easily check this occasionally: if the machine is running out of memory, flush the results to disk earlier than planned.

- I suppose memory caching could greatly reduce the number of file accesses, because in the slot directory I just found ~700 MB, whereas we're talking about many GBs transferred per WU. This suggests that lot's of results being written now are being overwritten later on - i.e. the disk writes could be avoided completely if you waited long enough.

- "simply dedicate an old HDD for BOINC and those WUs" - one could do that, at the expense of additional power, noise and HDD wear. However, if there's a rather simple software solution, which would benefit the project and all users, I'd much prefer that.

Best regards,
MrS