[German]There is a serious vulnerability (backdoor) in the firmware of power delivery units from the US manufacturer Networx. In older versions (prior to version 2.0.2a ), the firmware does not perform authentication, i.e. it is possible to switch off the power delivery unit (PDU) via the network. This is deadly if these units are used for power supply in data centers and are accessible via internet. After a reader's tip, I contacted the manufacturer and GErman CERT Alliance about this. After much back and forth, there is now a working update. Below I disclose the details. Operators should urgently isolate the PDUs and install the firmware update promptly.
Advertising
Networx Power Delivery Unit
US manufacturer TechLogix Networx offers so-called Power Delivery Units, e.g. the TL-RKPS-01, for the power supply of components – especially in data centers.
TechLogix Networx Power Delivery Unit TL-RKPS-01
This unit replaces up to twelve 5V, 12V and/or 24 Volt power supplies with its own rack. Each channel is selectable, according to the manufacturer, making it suitable for different makes and models of equipment (extenders to splitters to switchers). In addition, individual channels can be switched remotely from any location.
Networx Power Delivery Unit TL-RKPS-01 GUI, source: TechLogix
The 100 ms start sequencing ensures a smooth restart of the power supply. These power delivery units are therefore popular in data centers for power supply.
Advertising
A backdoor and its tenacious fix
A backdoor exists in the firmware of this power delivery unit in older versions that would allow third parties to turn on or off devices connected to the unit without further authentication. I've been after this issue since June 2022, a few days ago the manufacturer TechLogix Networx has now provided an update that removes this vulnerability. Since September 11, 2022 is also the end of the 90 days I set as the deadline for a responsible disclosure, I would have disclosed the details even without a patch now.
A blog reader reports a backdoor
On June 11, 2022, a blog reader commented that he had encountered a backdoor in a power delivery unit used in data centers. The manufacturer does not want to fix this backdoor, although the devices connected to the unit can be switched on and off remotely via http without access data. The reader, who wishes to remain anonymous, asked who he could contact. I then contacted the reader by email, asking for details and offering to contact the manufacturer and, if necessary, the BSI. Here are the reader's responses, which I have slightly edited:
It was simple as can be: the device has no CLI, but should be able to be turned on and off from an application interface.
Since I had been using Python scripts to address other devices via librequests anyway, I just did the same here. The usual procedure: Trace the json requests sent back and forth in Firefox and transfer them to Python. Important parameters then via CLI parameters (IP, user, password, PDU output port, on/off/cycle) passed and sanitize.
When I then test a wrong password to check whether the password from the CLI works, I found that the devices can still be switched on and off.
At first I thought there was something in the cache, but when I realized that a finished Python script does not use data from a cache somewhere when I restart it, it was clear where I was going. I was able to simply remove the login procedure.
The manufacturer (US) as well as dealer was contacted by my boss, but the manufacturer says: parts shortage, we can't deliver new devices, so we also fired all programmers, ergo: no fix possible.
The reader then sent me the following code snippet, that he used to turn the power delivery unit on or off.
#!/usr/bin/env python3
import requests
session = requests.session()
response = session.post('http://192.168.1.2:80/cgi-bin/MMX32_Keyvalue.cgi', data='{Input01CMD=12$!')The above example code switches output port 12 to Power off, where the 12 in the data field can be a number from 1 – 12, without leading 0. The reader wrote about this:
That the string starts with a curly bracket that is not closed looks like obfuscation to me.
At this point, the reader wrote me that it must be the manufacturer's turn to finally look at the bug. In the blog reader's case, the power delivery units are used in a data center, but shielded from the Internet. It can be assumed that not all data center operators handle it this way; in other words, an attacker could then switch off the power supply for any components connected to the power delivery unit via the network, Internet, etc. This would be an almighty nightmare come true. This would be a nightmare come true for operators.
Contact with the manufacturer, the BSI and CERT-Bund
At this point I tried step by step to contact the manufacturer TechLogix Networx and later to get the BSI and German CERT-Bund on board to send a warning to data center operators if necessary.
- June 14, 2022: TechLogix Networx was contacted by mail and asked what they intended to do about the vulnerability.
- June 15, 2022: The response was that the firmware development team should provide a firmware update for testing in a few weeks. They said the firmware update would fix the vulnerability mentioned and allow for renaming of inputs to provide more clarity in the use of the device.
That was when I put the case on resubmission and publication after 90 days.
- July 22, 2022: The blog reader received a firmware update for testing. The reader wrote: This has a lot of "drawbacks" because according to the manufacturer everything is set to factory defaults, including IP and voltage, which of course is a no-go for data center operation. I have done this once as a test: Conclusion: Nothing is set to factory defaults, the update runs through (new firmware visible in the web interface), but … the bug is not gone.
- July 23, 2022: Inquiry at TechLogix Networx about the status.
- July 27, 2022: Reply from TechLogix Networx that the blog reader received the updated firmware for testing [but it does not work, as the reader detailed above].
At this point, I contacted the blog reader again and advised on how to proceed. My idea was to contact the BSI and ask whether there could be a warning to data center operators. At the time, the reader replied as follows:
Normally, a good operator should not hang these power delivery units (PDUs) directly on the Internet, but I cannot judge how small providers handle this. I think I lack a sense of the state at the lower end of the IT security scale.
I received a firmware update about 1.5 weeks ago (v. 2.0.1) as well as another (v. 2.0.2). I just installed the latter on Friday. The thing was a cheek:
According to the manufacturer, the device is set to factory defaults (IP address, voltage), so that it is of course forbidden to install this during operation, i.e.: coordination with the data center operator is necessary, of course outside of regular business hours, because customer equipment is also unavailable during that time.
We therefore tested a test device beforehand and it turned out that everything was wrong, the device was not set to factory settings, but it hung up during the reboot.
All devices continue to run, but control is only possible again once you pull the power, which is of course again only possible outside the regular availability times.
New firmware had been installed, the version display showed that. So it was not the case that the firmware was simply not uploaded.
The blog reader then went on vacation and left further testing to his supervisor, so the case remained open for now. I then contacted the BSI and they told me to report it to CERT-Bund.
- August 03, 2022: 2nd report in rough form (without details) to German BSI with request for procedure.
- 08 August 2022: Acknowledgement of receiving the reported vulnerability from German CERT-Bund with a request for more details.
- August 18, 2022: Feedback from CERT-Bund that the vulnerability report is rejected – was my mistake, I did not have the time to report the details. I reported the details to CERT-Bund by reply mail on August 18th.
- August 26, 2022: Reply from CERT-Bund with the following text: Thank you very much for your feedback and the further information. If we understand this correctly, the manufacturer has provided a patch, which, however, still seems to be error-prone ("device hangs", etc.). Since the manufacturer has already been informed and the reporter has reported the problem again (error-prone application of the patch), we do not currently see any added value in a takeover of the vulnerability report by CERT-Bund.
At that point German CERT-Bund was "out of the game" and I started to prepare an article with the facts to be published in my German blog.
A firmware update is coming
Then, shortly before this blog post was published, there was new information and a firmware update that fixed the issue.
- September 06, 2022: Feedback from the reader (in response to my query) that there was no further feedback from the manufacturer, only another firmware, but it did not fix the bug. In another mail the reader wrote: According to the manual of the manufacturer (page 14 ff), which I found, it seems as if this is intended. There are all the parameters in there that you can send and what the box then does.ggf. So "works as intended". From my point of view, it doesn't change the fact that without a password it is simply a security bug. The instruction refers to serial Con, but it also works via http post command in the form I sent with the "curly braces" starting.
- September 13, 2022: The blog reader informed me: I received a new firmware today. The bug is fixed. You have to send a password now, otherwise you can't send commands.
When I asked, he added that the new firmware version 2.0.2a had been received via Google Drive. Furthermore, the reader wrote:
With this [firmware], an unsalted md5 hash of the password is required with every command sent. Communication still takes place over http. This is of course vulnerable to replay attacks, but at least better with respect to openly accessible devices on the Internet.
When updating, the box hangs and has to be powered down itself, but devices continue to run except for the short power down.
At this point, it should be noted that the manufacturer released an improved firmware that requires authentication shortly before the deadline for a responsible disclosure. It's unpleasant that everything is still transmitted via http (they probably shy away from the certificates required for https).
Anyway – was a tough process all in all – my thanks to the blog reader for the information. Data center operators using these PDUs now know what to do: make sure the PDUs are isolated from the general network/internet and then install the updated firmware from the manufacturer (note that the PDU must be powered down when updating the firmware).
