General

At the end of 2020 an odd issue in QRadar deployments occurred. The result of this defect is, that flow or/and event processing stops.
The details about this are described here: https://www.ibm.com/support/pages/node/6395080

To check, if you are affected by this issue, you can run for example the following command:

grep -i valid /var/log/qradar.log

The good news

This can be fixed by applying the following fix:

/opt/qradar/support/all_servers.sh -Ck 'if [ -f /opt/qradar/ecs/license.txt ] ; then echo -n "QRadar:Q1 Labs Inc.:0007634bda1e2:WnT9X7BDFOgB1WaXwokODc:12/31/20" > /opt/qradar/ecs/license.txt ; fi ; if [ -f /opt/ibm/si/services/ecs-ec-ingress/current/eventgnosis/license.txt ] ; then echo -n "QRadar:Q1 Labs Inc.:0007634bda1e2:WnT9X7BDFOgB1WaXwokODc:12/31/20" > /opt/ibm/si/services/ecs-ec-ingress/current/eventgnosis/license.txt ; fi ; if [ -f /opt/ibm/si/services/ecs-ep/current/eventgnosis/license.txt ] ; then echo -n "QRadar:Q1 Labs Inc.:0007634bda1e2:WnT9X7BDFOgB1WaXwokODc:12/31/20" > /opt/ibm/si/services/ecs-ep/current/eventgnosis/license.txt ; fi ; if [ -f /opt/ibm/si/services/ecs-ec/current/eventgnosis/license.txt ] ; then echo -n "QRadar:Q1 Labs Inc.:0007634bda1e2:WnT9X7BDFOgB1WaXwokODc:12/31/20" > /opt/ibm/si/services/ecs-ec/current/eventgnosis/license.txt ; fi ; if [ -f /usr/eventgnosis/ecs/license.txt ] ; then echo -n "QRadar:Q1 Labs Inc.:0007634bda1e2:WnT9X7BDFOgB1WaXwokODc:12/31/20" > /usr/eventgnosis/ecs/license.txt ; fi ; if [ -f /opt/qradar/conf/templates/ecs_license.txt ] ; then echo -n "QRadar:Q1 Labs Inc.:0007634bda1e2:WnT9X7BDFOgB1WaXwokODc:12/31/20" > /opt/qradar/conf/templates/ecs_license.txt ; fi'

Auto Updates

At the 2nd of January 2021 a fix was applied automatically with QRadar Auto Updates, hopefully already configured and enabled to receive daily updates. This is one example, why it's important to be up to date!

Important News

This defect will occur each time again, if you are adding or re-adding a host to deployment, which uses the ecs-ec/ecs-ep services! If you are not running at least QRadar 7.4.2FP1!

The Last two weeks this sophisticated issue hit me twice! In two different environments and circumstances:

1. Focus was the App-Framework of QRadar. Starting-point was  a health-check of the Apps (using an apphost) and just restarting services (vault-qrd, docker, conman, trafic, hostcontext and finaly tomcat) while investigating the app framework!
   Clue: The last restart of the micro-services regarding the apps was at October 2020! And by the way: auto updates was not configured and enabled at this time!
   
   After successfully maintained the apps, a few days later we've identified in addition to a side note of our customer, that logs seems not to be processed as expected. After applying the fix, according a longer investigation, what could be the root cause, as expected the log processing was up and running again :)
   
   
2. Exchange of the FP appliance with a current Type of 1729 M6 running 7.4.1FP1. Re-Adding the FP-Host into the deployment also hit this issue! I was wondering, why no flow processing had started immediately after the deploy.

What does this mean?

Keep in mind to re-run the fix-command mentioned above each time you add or re-add a EP or FP host not running QRadar 7.4.2FP1 at least!

By the way: this week 7.4.2FP2 was released to upgrade a QRadar deployment using a SFS File, if you are running for example QRadar 7.4.1FP2!

Only starting with QRadar Release 7.4.2FP1 this defect is solved! And adding a host should be "straight forward" again, as expected.
These details were not clearly pointed out to me! This week i've had a steep learning curve. And as an old saying mentioned: we learn not for school, but for live :)

So just an info to keep in mind :)

#qradar-highlights #qradar #ibmsecurityqradar #p4bbootcamps