K-TAP installation failure on Linux is not a problem longer

One of the most important value of Guardium system is its enterprise architecture. Whether that installed to monitor one or one hundred databases we can manage the environment from one place, reconfigure it with appropriate segregation of duties and role base access control.

Monitoring of database on system to cover Database Activity Monitoring (DAM) expectations requires visibility of the all sessions (local and remote) and support all database protocols (TCP, shared memory, pipes, etc.). That is why the Guardium monitoring agent (STAP) is deeply integrated with operating system kernel (KTAP module). However the Linux distribution diversity leads to necessity to support every existing kernel version on customer sites. Before version 9.1 of Guardium this process required time (2-3 weeks) for module development and tests. Now the redeveloped KTAP can be easily recompiled and the support of the particular kernel version is not problem longer.

When should I worry about KTAP?

KTAP compilation is the installation process task and usually we do not need to pay attention on it. However, sometimes the system environment prevents the proper compilation and it is necessary to analyze the situation and take the appropriate steps.

How to check whether KTAP is switched on?

Review the “STAP Status” report and notice the value in the “KTAP Installed“. Value No means that kernel driver is not installed and activated.

STAP status

STAP status

Also the “GIM Event List” report points more detailed information.

GIM Event List

GIM Event List

We have here information that STAP does not contain module for kernel on this machine (3.10.0-229.11.1).
Then the explanation the reason of the failure – the development tools has not been installed.
The last marked message points that KTAP_ALLOW_MODULE is not set to Y. It means that STAP does not try to load other modules which are nearing to target kernel. It is accurate configuration for production environments where any errors on the kernel level are unacceptable.

KTAP compilation process reinitialization

The KTAP compilation process requires on the Linux box the cc compiler (gcc), make tool and kernel development files. Check out their existence on your machine – for RedHat use these commands:

yum list installed gcc
yum list installed make
yum list installed kernel-devel

Install missing packages

Package installation

Package installation

Now we need to reinitialize KTAP compilation process. The simplest method uses GIM to reconfigure the KTAP module. Open module selection screen and unselect “Display Only Bundles” option. Then select KTAP module and go forward – Next button

KTAP module selection

KTAP module selection

Set the KTAP_ENABLED field to 1 and Apply to Client. Execute update using Install/Update

KTAP update

KTAP update

Review update status. After a while you should receive information that KTAP is installed. If you will receive status – FAILED restart analysis again from “GIM Events List” report or analyse the log files (described later).

Update status

Update status

Now we can review “GIM Events List” report again

GIM events

GIM events

and “STAP Status

STAP Status

STAP Status

The STAP has been installed and we can start data monitoring. A careful observer will notice the appearance of an additional entryGuardium-FSM related with FAM functionality.

Important: FAM agent works on kernel level. This functionality requires the KTAP installation.

More detailed information about KTAP status we can find out in the <GIM_HOME/module/KTAP/current/KTAP.log file
This sequence points STAP installation and lack of development tools

[Fri Sep 11 20:42:21 2015] -I- Installing KTAP 10.0.0_r79963_1
[Fri Sep 11 20:42:22 2015] -I- Starting KTAP 10.0.0_r79963_1
[Fri Sep 11 20:42:23 2015] -I- Informing GIM on an event : *** KTAP MODULE WARNING MESSAGE ***
Searching for modules in /opt/guardium/GIM/modules/KTAP/10.0.0_r79963_1-1441996935/modules-*.tgz
guard_ktap_loader: File /lib/modules/3.10.0-229.11.1.el7.x86_64/build/.config not found.  Local build of KTAP will not
guard_ktap_loader: be attempted.  Please install kernel development packages for 3.10.0-229.11.1.el7.x86_64 if you wish
guard_ktap_loader: to build KTAP locally.
guard_ktap_loader: ===================================================================
guard_ktap_loader: You have elected not to load close fitting module combinations.
guard_ktap_loader: To enable close fitting combinations, reinstall bundle STAP while setting the
guard_ktap_loader: KTAP_ALLOW_MODULE_COMBOS to 'Y'
guard_ktap_loader: The in-kernel functionality will now be disabled.

and here is the fragment after compilation reinitialization

[Fri Sep 11 21:49:06 2015] -I- KTAP_ENABLED changed its value to 1 ... updating guard_tap.ini)
[Fri Sep 11 21:49:06 2015] -I- checking is ktap 79963 is loaded as part of update()
[Fri Sep 11 21:49:06 2015] -I- Starting KTAP ... for the first time
[Fri Sep 11 21:49:06 2015] -I- Informing GIM on an event : *** KTAP MODULE INSTALLER PLATFORM CHECKS MESSAGE ***

[Fri Sep 11 21:49:06 2015] -I- SEOS check - ok !
[Fri Sep 11 21:49:06 2015] -I- Trying to load KTAP as part of a start request (invoker=)
[Fri Sep 11 21:49:14 2015] Searching for modules in /opt/guardium/GIM/modules/KTAP/10.0.0_r79963_1-1441996935/modules-*.tgz
Attempting to build KTAP module using dir /lib/modules/3.10.0-229.11.1.el7.x86_64/build
guard_ktap_loader: Custom module ktap-79963-rhel-7-linux-x86_64-xCUSTOMxdblin-3.10.0-229.11.1.el7.x86_64-x86_64-SMP.ko built for kernel 3.10.0-229.11.1.el7.x86_64.

In this same directory the ktap_install.log notices additional remarks

=== Fri Sep 11 21:49:07 CEST 2015 ===
Attempting to build KTAP module using dir /lib/modules/3.10.0-229.11.1.el7.x86_64/build
Custom module ktap-79963-rhel-7-linux-x86_64-xCUSTOMxdblin-3.10.0-229.11.1.el7.x86_64-x86_64-SMP.ko built for kernel 3.10.0-229.11.1.el7.x86_64.
/sbin/modprobe  ktap ktap_build_number=79963 sys_call_table_addr=ffffffff8161c3c0 kernel_toc_addr= kernel_gp_addr=   
Install OK
Load OK

What if I cannot install development packages on system?

This situation is related with production environments but we can create package on other system (test environment) with this same kernel and later install in on the target.

Method 1 – manual INSTALLATION ON TARGET SYSTEM

The list of embeded KTAP modules in the STAP release we can review in module-<STAP-release>.tgz file:

[root@dblin current]# tar tvf modules-10.0.0_r79963_trunk_1.tgz | grep .ko | awk '{print $6}'
dummy.ko
ktap-10.0.0_r79963_trunk_1-rh7x64m-3.10.0-123.9.2.el7.x86_64-x86_64-SMP.ko
ktap-10.0.0_r79963_trunk_1-rh7x64m-3.10.0-123.el7.x86_64-x86_64-SMP.ko

After recompilation the new KTAP module is located in this same KTAP directory:

[root@dblin current]# ls *.ko
ktap-79963-rhel-7-linux-x86_64-xCUSTOMxdblin-3.10.0-229.11.1.el7.x86_64-x86_64-SMP.ko

Now we can create custom module archive – guard_ktap_append_modules command

[root@dblin current]# ./guard_ktap_append_modules 
Original MD5SUM: c467e40397957a81916e0b4f6bfb2864  ./modules-10.0.0_r79963_trunk_1.tgz

The following modules will be added to ./modules-10.0.0_r79963_trunk_1.tgz
     ./ktap-79963-rhel-7-linux-x86_64-xCUSTOMxdblin-3.10.0-229.11.1.el7.x86_64-x86_64-SMP.ko

New MD5SUM: 3718924d80ee6dbbea81594521f7fc1a  ./modules-10.0.0_r79963_trunk_1.tgz

This command adds the compiled module to modules archive. Then we can manually upload modules-<STAP_release>.tgz file on the target machine to the temporary directory and execute

guard_ktap_loader retry <tmp_dir>/modules-<STAP-release>.tgz

Then restart STAP and the new KTAP module should be recognized and installed.

METHOD 2 – KTAP MODULE TRANSFER OVER GIM

Important: If STAP_UPLOAD_FEATURE parameter is set to 1 the module recompilation process creates custom STAP GIM file and transfers it to the collector which manages this STAP

The KTAP compilation process automatically creates the STAP bundle on the appliance which manages this STAP (not to GIM server). This module can be downloaded from appliance over fileserver command from /log/gim-dist-packages directory

Fileserver

Fileserver

Tip: In the version 10 the fileserver has an additional parameter and current syntax is:
fileserver ip_address_fileserver_client duration

Then you can upload this module to GIM server and install on the target machine.

Summary:
Guardium KTAP driver can be easily created for the kernel resided on the target system. Module creation process assumes existence of Quality/Assurance procedure.

Advertisements

3 thoughts on “K-TAP installation failure on Linux is not a problem longer

  1. sudou says:

    A similar phenomenon occurred in environment of Centos6.6.
    I dealt in the same way, but, as a result, was NG.
    In the environment of Centos6.6, is this problem not settled?

    Like

  2. Franco says:

    In order to use already compiled CUSTOM BUNDLE on any server you need to turn on GIM_ALLOW_CUSTOM_BUNDLES indicator to 1 (for security reasons this have to be done manually on each DB server). Turning GIM_ALLOW_CUSTOM_BUNDLES indicator back off could be done from appliance.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s