Domains and host files are using too much memory

[ndfred]

When trying to run the configuration commands the system runs out of memory:

# commit; save; exit
[ service dns forwarding blacklist ]
NOTI[001]14:58:42.963: Starting blacklist update...
INFO[002]14:58:42.965: Removing stale blacklists...
INFO[003]14:58:43.043: Downloading domains source unifi-pi-hole
INFO[004]15:00:02.339: unifi-pi-hole: downloaded: 360565
INFO[005]15:00:02.340: unifi-pi-hole: extracted: 360563
INFO[006]15:00:02.341: unifi-pi-hole: dropped: 2
INFO[007]15:00:14.945: Downloading hosts source unifi-pi-hole
INFO[008]15:01:42.968: unifi-pi-hole: downloaded: 375021
INFO[009]15:01:42.970: unifi-pi-hole: extracted: 109722
INFO[00a]15:01:42.971: unifi-pi-hole: dropped: 265299
NOTI[00b]15:01:46.341: Total entries found: 735586
NOTI[00c]15:01:46.343: Total entries extracted 470285
NOTI[00d]15:01:46.345: Total entries dropped 265301
ERRO[00e]15:01:46.349: ReloadDNS(): 
 error: fork/exec /bin/bash: cannot allocate memory


Commit failed

It looks like a lot of hosts entries get dropped, maybe bad formatting or duplicates with the domains, worth filtering out when we produce the files.

[ndfred]

This is actually the configuration process running out of memory, not the system or dnsmasq. I'll have to figure out a way around this anyway (see #12) so this may not be an issue after all.

[ndfred]

Without any filters applied we get 0.4% of system memory (4th argument), 6M virtual size (5th) and 2M resident set size (6th):

$ ps aux | grep dnsmasq.d
dnsmasq   2433  0.0  0.4   6000  2056 ?        S    16:14   0:02 /usr/sbin/dnsmasq -x /run/dnsmasq/dnsmasq.pid -u dnsmasq -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new --local-service

Digging into /proc/2433/smaps I can see 132k + 128k of heap size and 132k of stack size. Querying /proc/meminfo the system has 512M of memory and only 44% of memory is being used.

[ndfred]

After following the vanilla ad blocking set script we see 0.9% of system memory, 5.6M of self memory usage so 3.5M of additional memory usage:

$ ps aux | grep dnsmasq.d
dnsmasq  11559  0.9  1.1   9588  5596 ?        S    17:48   0:00 /usr/sbin/dnsmasq -x /run/dnsmasq/dnsmasq.pid -u dnsmasq -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new --local-service

The blacklist in /etc/dnsmasq.d/blacklist.conf is 672k so that is quite the memory hike, though not significant system-wide. I'll test with larger blacklists to see how big a difference that makes.

[ndfred]

The original file was 672k, I just tried with the new 1.9M file from v0.2.32 we see 2.6% of system memory, 11.7M of self memory usage so 9.7M of additional memory usage:

$ ps aux | grep dnsmasq.d
dnsmasq  11773  2.6  2.3  15652 11684 ?        S    23:54   0:02 /usr/sbin/dnsmasq -x /run/dnsmasq/dnsmasq.pid -u dnsmasq -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new --local-service

This is starting to get really significant. We see 48% of system memory used, up 4%.