Critical Vulnerability in Apache Log4j (For VCF cloud builder)

Over a period of time, softwares and its updates are boom to technologies but at the same time, they come with security threats too. Log4j security vulnerabilty is one of the latest ones.

Lets undertand what is log4j first

The software is published under terms of Apache software license and a copy of which has been included in the license file shipped with log4j distribution. Log4j is an open source project based work accumulated by many authors that allows developers to control which log statements are output with arbitrar granualirity. It is configurable at runtime using external configuration files. Apache log4j is also part of project which is known as Apache logging

Now the question arises, why logging and why log4j in specific? So the answer is simple, inserting log statements into your code is a low-tech method for debugging. It may be the only way because debuggers are not always available or applicable. On the contarary, some people think that log statement pollute source code and decrease legability. With Log4j, it is possible to enable logging at runtime without modifying the app library. The log4j package is designed so that these statements can remain in shipped code without incurring a performance cost.

CVE-2021-44228

Disclaimer : Critical vulnerabilities in apache log4j identified by VE-2021-44228 has been publicly disclosed which impact VMware products.

This is an ongoing event, please check this advisory for frequent updates as they develop.

As mentioned in the advisory article, there are numerous products which got affected. VMware has published several articles to perform temporary solutions only as workaround to address this issue. I have got these workaround instructions from few KBs issued by VMware engineers

I used the workaround for VCF cloud builder (applicable to 3.x and 4.x)

Steps for Cloud Builder

• Login to Cloud Builder VM using root credentials

• Change the user to root account using 'su' command

• Stop host imaging service using

systemctl stop imaging

• Run below command to backup original file

cp /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh.orig

• Run below command to modify /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh to add -Dlog4j2.formatMsgNoLookups=true as an extra argument for running the parent imaging service in "nohup /etc/alternatives/jre/bin/java -jar". Make sure this argument is added after -Dserver.port=$VIA_SERVICE_PORT

BEFORE:

nohup /etc/alternatives/jre/bin/java -jar -Dserver.port=$VIA_SERVICE_PORT -Dspring.config.additional-location=$VIA_EXTERNAL_PROPERTIES_PATH,$VIA_DB_PROPERTIES_FILE -Dserver.servlet.context-path=$VIA_CONTEXT_PATH $VIA_SERVICE_PATH < /dev/null >>$LOGFILE 2>&1 &

AFTER:

                                                                                                                                    
nohup /etc/alternatives/jre/bin/java -jar -Dserver.port=$VIA_SERVICE_PORT -Dlog4j2.formatMsgNoLookups=true -Dspring.config.additional-location=$VIA_EXTERNAL_PROPERTIES_PATH,$VIA_DB_PROPERTIES_FILE -Dserver.servlet.context-path=$VIA_CONTEXT_PATH $VIA_SERVICE_PATH < /dev/null >>$LOGFILE 2>&1 &

• Run below command again to take backup of original bfore editing

cp /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh.orig

• Run below command to modify /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh to add -Dlog4j2.formatMsgNoLookups=true as an extra argument for running the imaging service in "nohup /etc/alternatives/jre/bin/java -jar". Make sure this argument is added after -Dserver.port=$SECOND. The following is given as an example. Please make sure the equivalent command in the file is modified accordingly

nohup /etc/alternatives/jre/bin/java -jar -Dserver.port=$SECOND -Dlog4j2.formatMsgNoLookups=true -Dspring.config.additional-location=$VIA_DB_PROPERTIES_FILE $name < /dev/null >>$LOGFILE 2>&1 &

• Run below command to start host imaging service and verify

systemctl start imaging
systemctl status imaging

Run below command in order to confirm host imaging service is running with Dlog4j2 "true" option set

ps -ef|grep jar

and the output should be like this

root 6177 1 1 00:36 ? 00:00:26 /etc/alternatives/jre/bin/java -jar -Dserver.port=8445 -Dlog4j2.formatMsgNoLookups=true -Dspring.config.additional-location=/opt/vmware/evorack-imaging/config/via.properties,/opt/vmware/evorack-imaging/config/via-db-ext.properties -Dserver.servlet.context-path=/via /opt/vmware/evorack-imaging/services/evorack-imaging-services/via.jar
root 6647 5750 1 00:36 ? 00:00:21 /etc/alternatives/jre/bin/java -jar -Dserver.port=8081 -Dlog4j2.formatMsgNoLookups=true -Dspring.config.additional-location=/opt/vmware/evorack-imaging/config/via-db-ext.properties /opt/vmware/evorack-imaging/services/evorack-imaging-services/evorack-imaging-esxi-service-0.0.1-SNAPSHOT.jar
root 10247 3857 0 01:01 pts/0 00:00:00 grep --color=auto jar

Note: In case of event of service start failed. You should revert the config files by follwong below steps

systemctl stop imaging

• Copy the Original files back :

cp /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh.orig /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh

cp /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh.orig /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh

• Change the file permissions

chmod 755 /opt/vmware/evorack-imaging/imaging-util-scripts/start-parent-imaging-service.sh

chmod 755 /opt/vmware/evorack-imaging/imaging-util-scripts/start-imaging-services.sh

• Restart the imaging service

systemctl start imaging

The last but not the least, this information is based on the KB article provided by VMware but in case this workaround doesnt work, open up a incidence with VMware global support to mitigate vulnerability.