Troubleshooting Tomcat in a memory-lean environment

onemusicapi.com runs on AWS Elastic Beanstalk with a Tomcat 8.5 platform, using t3.micro instances. t3.micro instances have 1GB of RAM, and Elastic Beanstalk's monitoring puts instances in a Warning state if memory use exceeds 90% of available memory. We found that about a day or two after an EC2 instance was brought online, it would go into Warning state, due to memory use, even if the service wasn't actually affected.

TL;DR: We found Tomcat was using more memory than expected, and the following helped:

• Reducing -Xmx
• Limiting the code cache size
• Reducing the number of threads

Step one: measuring heap

The first step was to reduce the heap size (-Xmx) and monitor memory use. During this step, we realized that we should be able to reproduce the problem by setting -Xms & -Xmx to the same value, so that the heap allocation wouldn't change. However, the memory issue continued the same way it had - a day or two after being created, the instance would go into Warning state.

Step two: beyond the heap

The next step was to check what was using memory on the system. Using the command:

ps -eo size,pid,user,command --sort -size | head

... we noticed that Java process was using several hundred megabytes of memory over what -Xmx had been configured to. We were able to figure out how much of the memory was going to:

• Thread stacks.
• JVM overhead (by comparing it to the memory of a freshly started jshell).

However, this still left about 150MB of memory unexplained.

We assumed that this might be due to Tomcat using native memory for its internal buffers, so we tried changing every setting in that might seem like it might affect memory use.

Step three: measuring native memory

The next step was to enable Native Memory Tracking, and get "before & after" snapshots of memory use. This immediately showed us that the code cache was using over 64MB of memory. We then set the initial & maximum code cache size to get more predictable memory use. Here is the Native Memory Tracking output:

Total: reserved=1716451KB, committed=378495KB                                  |  Total: reserved=1753206KB, committed=441570KB
-                 Java Heap (reserved=235520KB, committed=235520KB)               -                 Java Heap (reserved=235520KB, committed=235520KB)
                            (mmap: reserved=235520KB, committed=235520KB)                                     (mmap: reserved=235520KB, committed=235520KB) 
                                                                               |   
-                     Class (reserved=1120844KB, committed=82468KB)            |  -                     Class (reserved=1136217KB, committed=101681KB)
                            (classes# 11852)                                   |                              (classes# 13397)
                            (  instance classes# 11125, array classes# 727)    |                              (  instance classes# 12600, array classes# 797)
                            (malloc=2636KB# 35841)                             |                              (malloc=3673KB# 55415) 
                            (mmap: reserved=1118208KB, committed=79832KB)      |                              (mmap: reserved=1132544KB, committed=98008KB) 
                            (  Metadata:   )                                                                  (  Metadata:   )
                            (    reserved=69632KB, committed=67840KB)          |                              (    reserved=83968KB, committed=83200KB)
                            (    used=66088KB)                                 |                              (    used=80220KB)
                            (    free=1752KB)                                  |                              (    free=2980KB)
                            (    waste=0KB =0.00%)                                                            (    waste=0KB =0.00%)
                            (  Class space:)                                                                  (  Class space:)
                            (    reserved=1048576KB, committed=11992KB)        |                              (    reserved=1048576KB, committed=14808KB)
                            (    used=10314KB)                                 |                              (    used=12109KB)
                            (    free=1678KB)                                  |                              (    free=2699KB)
                            (    waste=0KB =0.00%)                                                            (    waste=0KB =0.00%)
                                                                               |   
-                    Thread (reserved=82615KB, committed=9175KB)               |  -                    Thread (reserved=102238KB, committed=12062KB)
                            (thread# 80)                                       |                              (thread# 99)
                            (stack: reserved=82232KB, committed=8792KB)        |                              (stack: reserved=101764KB, committed=11588KB)
                            (malloc=289KB# 482)                                |                              (malloc=358KB# 596) 
                            (arena=94KB# 158)                                  |                              (arena=116KB# 196)
                                                                               |   
-                      Code (reserved=249146KB, committed=23006KB)             |  -                      Code (reserved=251533KB, committed=64609KB)
                            (malloc=1458KB# 8342)                              |                              (malloc=3845KB# 17739) 
                            (mmap: reserved=247688KB, committed=21548KB)       |                              (mmap: reserved=247688KB, committed=60764KB) 
                                                                               |   
-                        GC (reserved=1211KB, committed=1211KB)                |  -                        GC (reserved=1323KB, committed=1323KB)
                            (malloc=439KB# 3369)                               |                              (malloc=551KB# 4476) 
                            (mmap: reserved=772KB, committed=772KB)                                           (mmap: reserved=772KB, committed=772KB) 
                                                                               |   
-                  Compiler (reserved=424KB, committed=424KB)                  |  -                  Compiler (reserved=991KB, committed=991KB)
                            (malloc=294KB# 1015)                               |                              (malloc=861KB# 2304) 
                            (arena=131KB# 5)                                                                  (arena=131KB# 5)
                                                                               |   
-                  Internal (reserved=938KB, committed=938KB)                  |  -                  Internal (reserved=1295KB, committed=1295KB)
                            (malloc=906KB# 1817)                               |                              (malloc=1263KB# 2206) 
                            (mmap: reserved=32KB, committed=32KB)                                             (mmap: reserved=32KB, committed=32KB) 
                                                                               |   
-                     Other (reserved=134KB, committed=134KB)                  |  -                     Other (reserved=170KB, committed=170KB)
                            (malloc=134KB# 22)                                 |                              (malloc=170KB# 40) 
                                                                               |   
-                    Symbol (reserved=17441KB, committed=17441KB)              |  -                    Symbol (reserved=19006KB, committed=19006KB)
                            (malloc=14684KB# 157347)                           |                              (malloc=16153KB# 175495) 
                            (arena=2757KB# 1)                                  |                              (arena=2853KB# 1)
                                                                               |   
-    Native Memory Tracking (reserved=3356KB, committed=3356KB)                |  -    Native Memory Tracking (reserved=4162KB, committed=4162KB)
                            (malloc=19KB# 243)                                 |                              (malloc=28KB# 372) 
                            (tracking overhead=3338KB)                         |                              (tracking overhead=4135KB)
                                                                               |   
-               Arena Chunk (reserved=4368KB, committed=4368KB)                |  -               Arena Chunk (reserved=180KB, committed=180KB)
                            (malloc=4368KB)                                    |                              (malloc=180KB) 
                                                                               |   
-                   Logging (reserved=4KB, committed=4KB)                         -                   Logging (reserved=4KB, committed=4KB)
                            (malloc=4KB# 191)                                                                 (malloc=4KB# 191) 
                                                                               |   
-                 Arguments (reserved=19KB, committed=19KB)                       -                 Arguments (reserved=19KB, committed=19KB)
                            (malloc=19KB# 504)                                                                (malloc=19KB# 504) 
                                                                               |   
-                    Module (reserved=132KB, committed=132KB)                  |  -                    Module (reserved=146KB, committed=146KB)
                            (malloc=132KB# 1597)                               |                              (malloc=146KB# 1668) 
                                                                               |   
-              Synchronizer (reserved=290KB, committed=290KB)                  |  -              Synchronizer (reserved=393KB, committed=393KB)
                            (malloc=290KB# 2444)                               |                              (malloc=393KB# 3310) 
                                                                               |   
-                 Safepoint (reserved=8KB, committed=8KB)                         -                 Safepoint (reserved=8KB, committed=8KB)
                            (mmap: reserved=8KB, committed=8KB)                |                              (mmap: reserved=8KB, committed=8KB)

Step four: threads cost memory

During the above investigation, we noticed that we had 146 threads running, which was excessive for our use-case. So we removed & combined thread pools where we could, to reduce the number of threads. We also found that our web framework was creating large thread pools, but we did not have control over them. This was one of the learning outcomes of this exercise ?? limiting the number of threads to conserve memory.

One usually doesn't find articles on the Web about running Tomcat in memory-constrained environments - our searches all turned up results about multi-gigabyte heaps, where Tomcat's default settings are a much smaller fraction of the memory use. So it was an interesting experience to troubleshoot a situation where we didn't have any search results to use as a starting point.