This is going to be one of those weird blog posts, that basically convey just a single piece of information, but since the new SAP Community platform doesn’t really care for WIKI/document style knowledge storage, I had to come up with a click-bait title to make it a blog post.
I’ve written before on installing HANA cockpit and yesterday I fired it up once again.
Snooping around a bit in the tracefiles (running two HANA 2 instances + XSA stacks on my one 32GB NUC always leaves room for questions like “how much actual processing memory do we have left now?“) I found this jewel of an “error curtain”:
[26144]{-1}[-1/-1] 2017-09-12 13:30:34.208039 w Basis ELF.cpp(00770) :
[26144]{-1}[-1/-1] 2017-09-12 13:30:34.208039 w Basis ELF.cpp(00771) :
Using only dynamic symbol table for symbols of ELF file, no string section
/usr/sap/H4C/HDB96/exe/libhdbpythonbase.so
Using only dynamic symbol table for symbols of ELF file, no string section
/usr/sap/H4C/HDB96/exe/libxsengine.so
Using only dynamic symbol table for symbols of ELF file, no string section
/usr/sap/H4C/HDB96/exe/libstatisticsserver2.so
Using only dynamic symbol table for symbols of ELF file, no string section
/usr/sap/H4C/HDB96/exe/libstatisticscommon.so
Using only dynamic symbol table for symbols of ELF file, no string section
/usr/sap/H4C/HDB96/exe/libhdblivecache.so
Using only dynamic symbol table for symbols of ELF file, no string section
/usr/sap/H4C/HDB96/exe/libhdbpackman.so
Using only dynamic symbol table for symbols of ELF file, no string section
/usr/sap/H4C/HDB96/exe/libhdbpackmansqldriver.so
Using only dynamic symbol table for symbols of ELF file, no string section
/usr/sap/H4C/HDB96/exe/libhdbxsdbcserver.so
[... Buzz-Lightyear continuation mode ... ]
[... TO INFINITY ... AND BEYOND ....]Strip it!
To the average advanced computer user like myself, this looks like error messages.
Yes, these messages are marked as ‘w’ like wwwwwarning and the whole system still seems to be doing its thing, but personally, I don’t really like when I see tons of such messages and find precisely zero documents, SAP notes, KBAs or other information sources that can tell me what they mean.
I have the same issue with looking into the CONSOLE app of my MacBook by the way… #who_thinks_of_the_poor_end_user_nerd?
After some HANA source code reading (that really helps every now and then, especially when the trace file message neatly points to C++ file ELF.cpp, lines 770 and 771) I figured out two things:
- this is really just an info-warning written out when shared libraries are loaded by a HANA process. That usually happens once, at process startup, e.g. when the nameserver starts up.
and - this specific code has been around for several years now. Yet, I have not seen these messages before, which means that something else (aka not this piece of code) must have changed.
The next best thing from here was to look into the trace files of my other HANA 2 installation.
Guess what! No such messages in any of the trace files.
Ok, I thought, so maybe there is something different with the HANA installation that comes with HANA cockpit.
Fighting with ELFes
(sadly, she has nothing to do with this…) (image http://lotr.wikia.com/wiki/Arwen)
Let’s have a look into that anyway.
The warning message listed the shared objects (.so) files that it seems to have issues with, one of them was libhdbpythonbase.so.
Checking this in my HANA Cockpit installation with the Linux tool <em>file</em> shows:
file /usr/sap/H4C/HDB96/exe/libhdbpythonbase.so /usr/sap/H4C/HDB96/exe/libhdbpythonbase.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV) , dynamically linked , BuildID[sha1]=cafedde1f0db3527e5a8f3f69fc75019287f4196 , stripped <<< !
Compare this with my other HANA 2 installation:
file /usr/sap/S20/HDB20/exe/libhdbpythonbase.so /usr/sap/S20/HDB20/exe/libhdbpythonbase.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV) , dynamically linked , BuildID[sha1]=1089da0d45a4fc59a631f0877b0f2786a5282937 , not stripped <<< !
Aha! So, the files actually have been “stripped”.
What that means is that human readable information, so called symbols, about the compiled program in the .so files have been removed.
This information is not required to run the program but is extremely useful for debugging and problem analysis.
Without the information, a crash dump would only contain memory addresses for functions and since dynamic loading is used, these addresses change every time the program is run.
On the other hand, with symbols in the files, a stack trace or a crash dump literally print out the names of the involved functions and methods.
This is what makes the problem analysis actually possible in most cases.
Huh? I mean why?
“Peachy“, you think, “why on earth would then somebody strip this important info from the files“?
Great question!
The reason is the file size for the .so files.
Symbols can take quite some space in them. This alone might not be a problem since disk space is cheap, but the symbols also get loaded/mapped into memory when the shared object file is used.
And if there is a single precious resource on HANA systems it is the main memory.
How much can we save with this stripping?
First the unstripped version:
ls -lh /usr/sap/S20/HDB20/exe/libhdbpythonbase.so -r-xr-xr-x 1 s20adm sapsys 8.0M Jul 25 05:11 /usr/sap/S20/HDB20/exe/libhdbpythonbase.so
Now the stripped version:
ls -lh /usr/sap/H4C/HDB96/exe/libhdbpythonbase.so -r-xr-xr-x 1 h4cadm sapsys 2.2M Jun 26 19:42 /usr/sap/H4C/HDB96/exe/libhdbpythonbase.so
Boom!
Down from 8MB to 2.2MB!
And that is just one of 418 shared objects
(counted on my HANA 2 system via “ls -l *.so | wc -l“).
In total that amounts to quite a difference between the binary sizes for the normal HANA installation and the HANA Cockpit one:
“Normal HANA 2”
ls -lh /usr/sap/S20/HDB20/exe/ | grep total total 5.2G
“HANA Cockpit”
ls -lh /usr/sap/H4C/HDB96/exe/ | grep total total 1.7G
Obviously, it has been a goal for the HANA cockpit team to make the additional overhead of having its own HANA instance as small as possible.
There is actually another case where memory usage of HANA needed to be minimized: the HANA Express edition.
After quickly firing up my HXE virtual machine (gee, I really love my NUC system for making this so easy) I checked whether it also uses symbol stripping and lo and behold, it does:
“HANA Express edition”
ls -lh /usr/sap/HXE/HDB90/exe/libhdbpythonbase.so -r-xr-xr-x 1 hxeadm sapsys 2.2M Jul 24 19:16 /usr/sap/HXE/HDB90/exe/libhdbpythonbase.so file /usr/sap/HXE/HDB90/exe/libhdbpythonbase.so /usr/sap/HXE/HDB90/exe/libhdbpythonbase.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV) , dynamically linked , BuildID[sha1]=1089da0d45a4fc59a631f0877b0f2786a5282937 , stripped <<< !
For those interested to learn more about symbols and shared libraries here are some pointers:
https://www.akkadia.org/drepper/dsohowto.pdf
https://unix.stackexchange.com/questions/116327/loading-of-shared-libraries-and-ram-usage
https://stackoverflow.com/questions/7683630/library-symbols-and-user-symbols-in-elf
Finally, to go really down into the deep end of this rabbit hole, can we actually see that HANA uses less memory now?
Yes, we can!
The first option, on OS level:
“HANA Express”
hxeadm@hxehost:/usr/sap/HXE/HDB90> HDB info USER PID PPID %CPU VSZ RSS COMMAND [...] hxeadm 1707 1 0.0 21628 2792 sapstart pf=/usr/sap/HXE/SYS/profile/HXE_HDB90_hxehost hxeadm 1716 1707 0.0 217104 60060 \_ /usr/sap/HXE/HDB90/hxehost/trace/hdb.sapHXE_HDB90 ... hxeadm 1739 1716 9.5 6880612 5089404 \_ hdbnameserver [...]
We take the process ID of the nameserver and use the pmap tool with it.
As the man page explains, pmap displays information about process memory mappings.
hxeadm@hxehost:/usr/sap/HXE/HDB90> pmap -x 1739 1739: hdbnameserver START SIZE RSS PSS DIRTY SWAP SWAPPSS PERM MAPPING 000055b587d67000 1932K 1928K 1928K 12K 0K 0K r-xp .../hdbnameserver 000055b587f4b000 248K 248K 248K 248K 0K 0K r--p .../hdbnameserver 000055b587f89000 4K 4K 4K 4K 0K 0K rw-p .../hdbnameserver [...] 00007f0be3758000 712K 520K 288K 8K 0K 0K r-xp .../libhdbaflsdk.so 00007f0be380a000 16K 16K 16K 16K 0K 0K r--p .../libhdbaflsdk.so 00007f0be380e000 4K 4K 4K 4K 0K 0K rw-p .../libhdbaflsdk.so [...] 00007f0c54af6000 4K 4K 4K 4K 0K 0K rw-p [anon] 00007ffccc580000 136K 68K 68K 68K 0K 0K rw-p [stack] 00007ffccc5bc000 12K 0K 0K 0K 0K 0K r--p [vvar] 00007ffccc5bf000 8K 8K 0K 0K 0K 0K r-xp [vdso] ffffffffff600000 4K 0K 0K 0K 0K 0K r-xp [vsyscall] Total: 6880616K 5092916K 4825683K 4633372K 0K 0K 5870672K writable-private, 992656K readonly-private, 17288K shared, and 4621300K referenced
(full path to objects /hana/shared/HXE/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeac)
The output from pmap is usually very long and wide, so I cut out a lot here and shortened the file path.
The important bit is 17288K (~16 MB) shared in the last line.
s20adm@skullbox:/usr/sap/S20/SYS/exe/hdb> HDB info USER PID PPID %CPU VSZ RSS COMMAND [...] s20adm 20398 1 0.0 21600 228 sapstart pf=/usr/sap/S20/SYS/profile/S20_HDB20_skullbox.lab.cat5 s20adm 20407 20398 0.0 322428 580 \_ /usr/sap/S20/HDB20/skullbox.lab.cat5/trace/hdb.sapS20_HDB20 ... s20adm 20429 20407 11.8 9131476 2749264 \_ hdbnameserver [...] s20adm@skullbox:/usr/sap/S20/SYS/exe/hdb> pmap -x 20429 20429: hdbnameserver START SIZE RSS PSS DIRTY SWAP SWAPPSS PERM MAPPING 000000679a5a6000 1932K 240K 240K 8K 4K 4K r-xp .../hdbnameserver 000000679a78a000 248K 24K 24K 24K 224K 224K r--p .../hdbnameserver 000000679a7c8000 4K 4K 4K 4K 0K 0K rw-p .../hdbnameserver [...] 00007fd36a355000 76K 0K 0K 0K 0K 0K r--s .../libhdbaflsdk.so 00007fd36a368000 32K 0K 0K 0K 0K 0K r--s .../libhdbaflsdk.so 00007fd36a370000 648K 0K 0K 0K 0K 0K r--s .../libhdbaflsdk.so 00007fd36a412000 1368K 0K 0K 0K 0K 0K r--s .../libhdbaflsdk.so 00007fd36a568000 116K 0K 0K 0K 0K 0K r--s .../libhdbaflsdk.so 00007fd36a585000 156K 0K 0K 0K 0K 0K r--s .../libhdbaflsdk.so 00007fd36a5ac000 80K 0K 0K 0K 0K 0K r--s .../libhdbaflsdk.so [...] 00007fd46f0d7000 4K 4K 4K 4K 0K 0K rw-p [anon] 00007ffdd3c51000 136K 28K 28K 28K 40K 40K rw-p [stack] 00007ffdd3dd6000 12K 0K 0K 0K 0K 0K r--p [vvar] 00007ffdd3dd9000 8K 8K 0K 0K 0K 0K r-xp [vdso] ffffffffff600000 4K 0K 0K 0K 0K 0K r-xp [vsyscall] Total: 8925232K 2753308K 2717475K 2491696K 3128688K 3023716K 5720584K writable-private, 994368K readonly-private, 2210280K shared, and 2569084K referenced
(full path to objects /hana/shared/S20/exe/linuxx86_64/HDB_2.00.020.00.1500920972_7f63b0aa11dca2ea54d450aa302319302c2eeac)
Same story here, but here we have 2210280K (2.1 GB) shared data.
The HANA way
Another approach to compare the memory usage is of course to query the HANA monitoring views:
select
name,
round (value/1024/1024, 2) as value_MB
from "PUBLIC"."M_MEMORY";
HANA 2 HANA Express
NAME VALUE_MB VALUE_MB
SYSTEM_MEMORY_SIZE 32063.550 23546.860
SYSTEM_MEMORY_FREE_SIZE 272.140 246.680
PROCESS_MEMORY_SIZE 8652.050 6717.480
PROCESS_RESIDENT_SIZE 2110.720 4960.260 <<< ~ Linux RSS
PROCESS_CODE_SIZE 2017.950 3.580 <<< ~ Linux shared
PROCESS_STACK_SIZE 146.50 99.690
PROCESS_ALLOCATION_LIMIT 28857.190 21192.170
GLOBAL_ALLOCATION_LIMIT 28857.190 21192.170
[...]More about HANA memory can be found in SAP Note 1999997 – FAQ: SAP HANA Memory
If you see warning about the “using only dynamic symbol table” now, you can put this into perspective.
There you go, now you know.
Cheers,
Lars