Memory Leak since 11.0.0 Core + Appcd 3.0.0?

grradmin

Not having that issue on our servers.

xyzulu

What process is using the memory?

ivansalloum

Do you have any security solution installed?

CaveJohnson

struggling to actually get on to a box via ssh when its at 100% ram utilisation and htop fails to load.

have setup a script to run regularly to log the mem usage so i can hopefully get some more useable results.

No specific security solution in use - SElinux is currently disabled

rdbf

CaveJohnson

Not having that issue either. Do you have swap disabled?

CaveJohnson

rdbf swap is enabled but only at 1G size. Hasn't been an issue before now

slimx

I actually noticed LESS usage to be honest. I think your issue is isolated friend.

Zoinkies

No issues, less memory and CPU usage since update - I have swap disabled.

CaveJohnson

here you can see the issue - update was done on 3rd july

https://ibb.co/3rvmxvW

8Dweb

CaveJohnson - Oddly, after updating our web servers this morning to the latest appcd - 3.1.1,

We have had all of them lock with memory issues, and two of the larger ones have locked several times today. This did not happen with any other appcd updates over the past 2 months.

I am attempting to gather data, and will open a ticket when I am able to process what is taking place.

No other Ubuntu updates were handled at the same time, so... not sure.

JohnB

To troubleshoot, set swap to 5gb and use netdata or something that will warn you via an email (default) when swap usage is high. Then check what processes are consuming so much memory. Or monitor frequently manually. Point is, increase swap so that you will have time to troubleshoot.

CaveJohnson

JohnB i have set the following script on a cronjob to log memory consumption every 15 mins. hopefully get some better insight this way

#!/bin/bash
LOGFILE="/var/log/memory.log"
{
    echo "Timestamp: $(date)"
    free -h
    echo -e "PID\tPPID\tMEM_MB\t%MEM\t%CPU"
    ps --no-headers -eo pid,ppid,%mem,%cpu --sort=-%mem | head -n 20 | awk '{ printf "%-8s %-8s %-10.2f %-8s %-8s\n", $1, $2, ($3/100)*total_mem, $3, $4 }' total_mem=$(free -m | >
    echo "----------------------------------------"
} >> $LOGFILE

twest

8Dweb yeesh, that's scary!

Rich

One of my servers has been having memory issues as well. I removed a website that took up a large amount of memory: Mautic. But today I had issues with the server again and needed to reboot it, before I could even SSH into it.

DracoBlue

Check journalctl -k | grep -i memory or grep -i memory /var/log/syslog

charlesalmeida

Yes, there was a problem with that, we had to disable swap for some reason, it was simply filling up.
After disabling swap, usage stability improved. something else changing the swap to something larger stopped using the swap.
Very confusing but reduced memory usage in this version. only strange behavior like this happened.

Adam

3.1.1 had only minor changes to the srs_milter service so shouldn't have made any difference to memory usage. If anyone manages to catch the excessive usage whilst it's happening, run top -b -n 1 and send the output in a support ticket or post it here if you are comfortable doing so.

CaveJohnson

Have found a server its starting to creep up again on. Looks to be the mariadb container that is causing it?

top 3 memory consuming processes

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                          
1785591 lxd       20   0 1957864 348684  26852 S   0.6   4.3   0:03.49 mariadbd                                                                                                                                         
    392 root      19  -1  441832 162588 159352 S   5.4   2.0 111:00.33 systemd-journal                                                                                                                                  
   7762 912       20   0 2384796 116980  18536 S   0.0   1.4  10:21.58 orchd

Running mariadb version 11

DracoBlue

@CaveJohnson check docker logs mariadb

CaveJohnson

here's the log since the latest container restart

2024-07-11 16:29:56+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.3.2+maria~ubu2204 started.
2024-07-11 16:29:56+00:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB
2024-07-11 16:29:56+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-07-11 16:29:56+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.3.2+maria~ubu2204 started.
2024-07-11 16:29:56+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
2024-07-11 16:29:56 0 [Note] Starting MariaDB 11.3.2-MariaDB-1:11.3.2+maria~ubu2204 source revision 068a6819eb63bcb01fdfa037c9bf3bf63c33ee42 as process 1
2024-07-11 16:29:56 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2024-07-11 16:29:56 0 [Note] InnoDB: Number of transaction pools: 1
2024-07-11 16:29:56 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2024-07-11 16:29:56 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2024-07-11 16:29:56 0 [Note] InnoDB: Initializing buffer pool, total size = 512.000MiB, chunk size = 8.000MiB
2024-07-11 16:29:56 0 [Note] InnoDB: Completed initialization of buffer pool
2024-07-11 16:29:56 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
2024-07-11 16:29:56 0 [Warning] InnoDB: Retry attempts for reading partial data failed.
2024-07-11 16:29:56 0 [ERROR] InnoDB: Failed to read log at 75408384: I/O error
2024-07-11 16:29:56 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2024-07-11 16:29:56 0 [Note] InnoDB: Starting shutdown...
2024-07-11 16:29:56 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2024-07-11 16:29:56 0 [Note] Plugin 'FEEDBACK' is disabled.
2024-07-11 16:29:56 0 [Note] Plugin 'wsrep-provider' is disabled.
2024-07-11 16:29:56 0 [Warning] 'default-authentication-plugin' is MySQL 5.6 / 5.7 compatible option. To be implemented in later versions.
2024-07-11 16:29:56 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2024-07-11 16:29:56 0 [ERROR] Aborting
2024-07-11 16:29:57+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.3.2+maria~ubu2204 started.
2024-07-11 16:29:57+00:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB
2024-07-11 16:29:57+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-07-11 16:29:57+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.3.2+maria~ubu2204 started.
2024-07-11 16:29:57+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
2024-07-11 16:29:57 0 [Note] Starting MariaDB 11.3.2-MariaDB-1:11.3.2+maria~ubu2204 source revision 068a6819eb63bcb01fdfa037c9bf3bf63c33ee42 as process 1
2024-07-11 16:29:57 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2024-07-11 16:29:57 0 [Note] InnoDB: Number of transaction pools: 1
2024-07-11 16:29:57 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2024-07-11 16:29:57 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2024-07-11 16:29:57 0 [Note] InnoDB: Initializing buffer pool, total size = 512.000MiB, chunk size = 8.000MiB
2024-07-11 16:29:57 0 [Note] InnoDB: Completed initialization of buffer pool
2024-07-11 16:29:57 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
2024-07-11 16:29:57 0 [Note] InnoDB: End of log at LSN=27049878587
2024-07-11 16:29:57 0 [Note] InnoDB: Opened 3 undo tablespaces
2024-07-11 16:29:57 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
2024-07-11 16:29:57 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2024-07-11 16:29:57 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2024-07-11 16:29:57 0 [Note] InnoDB: log sequence number 27049878587; transaction id 10467363
2024-07-11 16:29:57 0 [Note] Plugin 'FEEDBACK' is disabled.
2024-07-11 16:29:57 0 [Note] Plugin 'wsrep-provider' is disabled.
2024-07-11 16:29:57 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2024-07-11 16:29:57 0 [Warning] 'default-authentication-plugin' is MySQL 5.6 / 5.7 compatible option. To be implemented in later versions.
2024-07-11 16:29:57 0 [Note] Server socket created on IP: '0.0.0.0'.
2024-07-11 16:29:57 0 [Note] Server socket created on IP: '::'.
2024-07-11 16:29:57 0 [Note] mariadbd: Event Scheduler: Loaded 0 events
2024-07-11 16:29:57 0 [Note] mariadbd: ready for connections.
Version: '11.3.2-MariaDB-1:11.3.2+maria~ubu2204'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2024-07-11 16:29:58 0 [Note] InnoDB: Buffer pool(s) load completed at 240711 16:29:58
2024-07-11 16:37:41 94 [Warning] Aborted connection 94 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
2024-07-11 16:52:41 235 [Warning] Aborted connection 235 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
2024-07-11 17:07:41 389 [Warning] Aborted connection 389 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
2024-07-11 17:22:41 539 [Warning] Aborted connection 539 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
2024-07-11 17:37:41 1131 [Warning] Aborted connection 1131 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
2024-07-11 17:52:41 1338 [Warning] Aborted connection 1338 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)