Index  | Recent Threads  | Unanswered Threads  | Who's Active  | Guidelines  | Search
 

Quick Go ยป
No member browsing this thread
Thread Status: Active
Total posts in this thread: 4
[ Jump to Last Post ]
Post new Thread
Author
Previous Thread This topic has been viewed 1344 times and has 3 replies Next Thread
jay_Orlando
Senior Cruncher
USA
Joined: Jan 4, 2006
Post Count: 181
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Boinc Manager using 98 GB of virtual memory - and more

Please see https://www.worldcommunitygrid.org/forums/wcg...ead,42178_offset,0#623068

I originally posted in Mapping Cancer Markers (above) .


Has anyone else seen this??


ONLY when I run Mapping Cancer Markers..
(So I posted here, instaed of BOINC forum. Should I pst there?)

This is strange. It is Boinc Manager - not boinc-client or the WU tasks themselves.
It is repeatable on different machines - different versions of Ubuntu.
Details are in the link above.
No errors or aborts.
Thanks!!
Jay
----------------------------------------

[Apr 3, 2020 12:09:15 AM]   Link   Report threatening or abusive post: please login first  Go to top 
adriverhoef
Master Cruncher
The Netherlands
Joined: Apr 3, 2009
Post Count: 2174
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Boinc Manager using 98 GB of virtual memory - and more

Jay,
You said you used htop to watch this particular behaviour. I'm seeing that, too, when using htop. However, when using ps or top I'm not seeing that.
$ ps l | sed -n 's/ .....//; 1p; /Kit/p'
F PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
0 8339 18112 20 0 216088 960 - S+ pts/4 0:00 sed -n s/ .....//; 1p; /Kit/p
0 28461 28448 20 0 102455064 10868 - SLl+ pts/7 0:00 /usr/libexec/webkit2gtk-4.0/WebKitWebProcess 7 23
0 28462 28448 20 0 102540892 10648 - SLl+ pts/7 0:00 /usr/libexec/webkit2gtk-4.0/WebKitNetworkProcess 8 23

Using htop, besides seeing 97G being used by 6 instances of WebKitNetworkProcess and 10 instances of WebKitWebProcess each, I'm seeing 10 instances of atril (a PDF viewer) using even more VIRT memory: 98G each. Using ps, I'm seeing nothing out of the ordinary (in column VSZ):
$ ps alx | sed -n 's/ .....//; 1p; /atril/p; /boinc/p; /Kit/p'
F PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
4 1287 1 30 10 1817564 86148 - SNsl ? 321:24 /usr/bin/boinc
4 8725 18906 20 0 103023340 117160 - Sl ? 0:08 boincmgr
0 28448 18222 20 0 103083628 124388 - Sl+ pts/7 0:28 atril eppo_4-free(6).pdf
0 28452 17650 20 0 155844 2672 - Ssl ? 0:00 /usr/libexec/atrild
0 12355 12342 20 0 102455052 40456 - SLl+ pts/7 0:00 /usr/libexec/webkit2gtk-4.0/WebKitWebProcess 7 23
0 12356 12342 20 0 85761508 40064 - SLl+ pts/7 0:00 /usr/libexec/webkit2gtk-4.0/WebKitNetworkProcess 8 23
0 12381 18112 20 0 216220 2764 - S+ pts/4 0:00 sed -n s/ .....//; 1p; /atril/p; /boinc/p; /Kit/p
0 12381 18112 20 0 216220 2764 - S+ pts/4 0:00 sed -n s/ .....//; 1p; /atril/p; /boinc/p; /Kit/p
0 12381 18112 20 0 216220 2764 - S+ pts/4 0:00 sed -n s/ .....//; 1p; /atril/p; /boinc/p; /Kit/p
nerd Adri
UPDATE:
Closing atril and boincmgr removes all (gigantic) instances of WebKitNetworkProcess and WebKitWebProcess in htop (not running any MCM1 now).
Opening atril again creates enormous VIRT instances of atril, WebKitNetworkProcess and WebKitWebProcess in htop.
UPDATE 2:
(Closing atril for a while now.)
Opening and closing boincmgr has the same effect (when not running MCM1) in htop:
- opening boincmgr creates huge VIRT instances of boincmgr, WebKitNetworkProcess and WebKitWebProcess;
- closing boincmgr removes all instances of WebKitNetworkProcess and WebKitWebProcess (and boincmgr, of course).
----------------------------------------
[Edit 6 times, last edit by adriverhoef at Apr 3, 2020 12:38:27 PM]
[Apr 3, 2020 9:16:22 AM]   Link   Report threatening or abusive post: please login first  Go to top 
jay_Orlando
Senior Cruncher
USA
Joined: Jan 4, 2006
Post Count: 181
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Boinc Manager using 98 GB of virtual memory - and more

Hello Adri, greetings!!

First, what caught my attention was the time the WU processes spent in 'sytem' time -
about 20% to 30%.(This showed up with gkrellm)

The virtual size was so large, it distracted me.
I was using top, htop, ps, gkrellm, and the Mate system monitor individually to
guess what was going on.
The Virtual mem size may be a false trail. But it was so large,
I jumped to a conclusion that it was important,

Then, I started reading.... (uh-oh.)
The 'man' page for top had a good explanation....
Linux Memory Types
For our purposes there are three types of memory, and one is optional. First is
physical memory, a limited resource where code and data must reside when executed
or referenced. Next is the optional swap file, where modified (dirty) memory can
be saved and later retrieved if too many demands are made on physical memory.
Lastly we have virtual memory, a nearly unlimited resource serving the following
goals:

1. abstraction, free from physical memory addresses/limits
2. isolation, every process in a separate address space
3. sharing, a single mapping can serve multiple needs
4. flexibility, assign a virtual address to a file

Regardless of which of these forms memory may take, all are managed as
pages (typically 4096 bytes) but expressed by default in top as KiB (kibibyte).
The memory discussed under topic `2c. MEMORY Usage' deals with
physical memory and the swap
file for the system as a whole. The memory reviewed in topic `3. FIELDS / Columns
Display' embraces all three memory types, but for individual processes.

For each such process, every memory page is restricted to a single quadrant from
the table below. Both physical memory and virtual memory can include any of the
four, while the swap file only includes #1 through #3. The memory in quadrant #4,
when modified, acts as its own dedicated swap file.

Private | Shared
1 | 2
Anonymous . stack |
. malloc() |
. brk()/sbrk() | . POSIX shm*
. mmap(PRIVATE, ANON) | . mmap(SHARED, ANON)
-----------------------+----------------------
. mmap(PRIVATE, fd) | . mmap(SHARED, fd)
File-backed . pgms/shared libs |
3 | 4

The following may help in interpreting process level memory values displayed as
scalable columns and discussed under topic `3a. DESCRIPTIONS of Fields'.

%MEM - simply RES divided by total physical memory
CODE - the `pgms' portion of quadrant 3
DATA - the entire quadrant 1 portion of VIRT plus all
explicit mmap file-backed pages of quadrant 3
RES - anything occupying physical memory which, beginning with
Linux-4.5, is the sum of the following three fields:
RSan - quadrant 1 pages, which include any
former quadrant 3 pages if modified
RSfd - quadrant 3 and quadrant 4 pages
RSsh - quadrant 2 pages
RSlk - subset of RES which cannot be swapped out (any quadrant)
SHR - subset of RES (excludes 1, includes all 2 & 4, some 3)
SWAP - potentially any quadrant except 4
USED - simply the sum of RES and SWAP
VIRT - everything in-use and/or reserved (all quadrants)

Note: Even though program images and shared libraries are considered private to a
process, they will be accounted for as shared (SHR) by the kernel.

... later ...
45. VIRT -- Virtual Memory Size (KiB)
The total amount of virtual memory used by the task. It includes all code,
data and shared libraries plus pages that have been swapped out and pages that
have been mapped but not used.


without any MCM---
top -o VIRT
shows:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2060 jay 20 0 82.5g 109008 76396 S 0.3 0.9 4:32.17 boincmgr
2097 jay 20 0 82.0g 40064 33848 S 0.0 0.3 0:00.04 WebKitNetworkPr
2096 jay 20 0 81.9g 73772 60600 S 1.0 0.6 5:59.09 WebKitWebProces

top -o USER
shows:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2254 boinc 39 19 835312 732208 27972 R 100.0 6.0 813:08.94 wcgrid_arp1_wrf
13526 boinc 39 19 147148 82020 6816 R 100.0 0.7 486:52.02 wcgrid_hst1_gro
13527 boinc 39 19 147252 82032 6816 R 100.3 0.7 486:52.95 wcgrid_hst1_gro
13530 boinc 39 19 835332 732336 28052 R 99.7 6.0 486:53.96 wcgrid_arp1_wrf
19718 boinc 39 19 385752 318476 48420 R 98.7 2.6 253:02.61 wcgrid_mip1_ros
20993 boinc 39 19 369700 302260 48256 R 100.3 2.5 116:18.61 wcgrid_mip1_ros
23830 boinc 39 19 364512 296860 48044 R 99.7 2.4 49:09.86 wcgrid_mip1_ros
23940 boinc 30 10 2300288 183680 61364 S 3.7 1.5 1:43.39 hsgamma_FGRPB1G


I could see the Virtual memory size with ps
as
ps -eo pid,drs,rss,vsz,user,fname --headers

----
SO. I have ARP, MIP, HST WU running with an Einstein WU on the GPU - with an
assigned CPU. - without any system time showing up in gkrellm.

When these complete, I will try MCM again, and tinker about some more.

Thank you for verifying the LARGE VMem sizes used!!

Jay
----------------------------------------

[Apr 4, 2020 1:38:25 AM]   Link   Report threatening or abusive post: please login first  Go to top 
jay_Orlando
Senior Cruncher
USA
Joined: Jan 4, 2006
Post Count: 181
Status: Offline
Project Badges:
Reply to this Post  Reply with Quote 
Re: Boinc Manager using 98 GB of virtual memory - and more

In case you have not used gkrellm, see
http://gkrellm.srcbox.net
----------------------------------------

[Apr 4, 2020 3:23:36 PM]   Link   Report threatening or abusive post: please login first  Go to top 
[ Jump to Last Post ]
Post new Thread