FireEye’s Mandiant Red Team recently discovered vulnerabilities present on the Logitech Harmony Hub Internet of Things (IoT) device that could potentially be exploited, resulting in root access to the device via SSH. The Harmony Hub is a home control system designed to connect to and control a variety of devices in the user’s home. Exploitation of these vulnerabilities from the local network could allow an attacker to control the devices linked to the Hub as well as use the Hub as an execution space to attack other devices on the local network. As the Harmony Hub device list includes support for devices such as smart locks, smart thermostats as well as other smart home devices, these vulnerabilities present a very high risk to the users.
FireEye disclosed these vulnerabilities to Logitech in January 2018. Logitech was receptive and has coordinated with FireEye to release this blog post in conjunction with a firmware update (4.15.96) to address these findings.
The Red Team discovered the following vulnerabilities:
- Improper certificate validation
- Insecure update process
- Developer debugging symbols left in the production firmware image
- Blank root user password
The Red Team used a combination of the vulnerabilities to gain administrative access to the Harmony Hub. This blog post outlines the discovery and analysis process, and demonstrates the necessity of rigorous security testing of consumer devices – particularly as the public places an increasing amount of trust in devices that are not just connected to home networks, but also give access to many details about the daily lives of their users.
Publicly available research indicated the presence of a universal asynchronous receiver/transmitter (UART) interface on some of the test points on the Harmony Hub. We soldered jumper wires to the test pads, which allowed us to connect to the Harmony Hub using a TTL to USB serial cable. Initial analysis of the boot process showed that the Harmony Hub booted via U-Boot 1.1.4 and ran a Linux kernel (Figure 1).
Figure 1: Initial boot log output from UART interface
After this point in the boot process, the console stopped returning output because the kernel was not configured with any console interfaces. We reconfigured the kernel boot parameters in U-Boot to inspect the full boot process, but no useful information was recovered. Furthermore, because the UART interface was configured to only transmit, no further interaction could be performed with the Harmony Hub on this interface. Therefore, we shifted our focus to gaining a better understanding of the Linux operating system and associated software running on the Harmony Hub.
Firmware Recovery and Extraction
The Harmony Hub is designed to pair with a companion Android or iOS application over Bluetooth for its initial configuration. We created a wireless network with hostapd and installed a Burp Suite Pro CA certificate on a test Android device to intercept traffic sent by the Harmony mobile application to the Internet and to the Harmony Hub. Once initial pairing is complete, the Harmony application searches for Harmony Hubs on the local network and communicates with the Harmony Hub over an HTTP-based API.
Once connected, the Harmony application sends two different requests to Harmony Hub’s API, which cause the Harmony Hub to check for updates (Figure 2).
Figure 2: A query to force the Harmony Hub to check for updates
The Harmony Hub sends its current firmware version to a Logitech server to determine if an update is available (Figure 3). If an update is available, the Logitech server sends a response containing a URL for the new firmware version (Figure 4). Despite using a self-signed certificate to intercept the HTTPS traffic sent by the Harmony Hub, we were able to observe this process – demonstrating that the Harmony Hub ignores invalid SSL certificates.
Figure 3: The Harmony Hub checks for updates to its firmware
Figure 4: The server sends a response with a URL for the updated firmware
We retrieved this firmware and examined the file. After extracting a few layers of archives, the firmware can be found in the harmony-image.squashfs file. This filesystem image is a SquashFS filesystem compressed with lzma, a common format for embedded devices. However, vendors often use old versions of squashfstools that are incompatible with more recent squashfstools builds. We used the unsqashfs_all.sh script included in firmware-mod-kit to automate the process of finding the correct version of unsquashfs to extract the filesystem image (Figure 5).
Figure 5: Using firmware-mod-kit to extract the filesystem
With the filesystem contents extracted, we investigated some of the configuration details of the Harmony Hub’s operating system. Inspection revealed that various debug details were available in the production image, such as kernel modules that were not stripped (Figure 6).
Figure 6: Unstripped Linux kernel objects on the filesystem
Investigation of /etc/passwd showed that the root user had no password configured (Figure 7). Therefore, if we can enable the dropbear SSH server, we can gain root access to the Harmony Hub through SSH without a password.
Figure 7: /etc/passwd shows no password is configured for the root user
We observed that an instance of a dropbear SSH server will be enabled during initialization if the file /etc/tdeenable is present in the filesystem (Figure 8).
Figure 8: A dropbear SSH server is enabled by /etc/init.d/rcS script if /etc/tdeenable is present
Hijacking Update Process
During the initialization process, the Harmony Hub queries the GetJson2Uris endpoint on the Logitech API to obtain a list of URLs to use for various processes (Figure 9), such as the URL to use when checking for updated firmware or a URL to obtain information about updates’ additional software packages.
Figure 9: The request to obtain a list of URL endpoints for various processes
We intercepted and modified the JSON object in the response from the server to point the GetUpdates member to our own IP address, as shown in Figure 10.
Figure 10: The modified JSON object member
Similar to the firmware update process, the Harmony Hub sends a POST request to the endpoint specified by GetUpdates containing the current versions of its internal software packages. The request shown in Figure 11 contains a sample request for the HEOS package.
Figure 11: The JSON request object containing the current version of the “HEOS” package
If the sysBuild parameter in the POST request body does not match the current version known by the server, the server responds with an initial response containing information about the new package version. For an undetermined reason, the Harmony Hub ignores this initial response and sends a second request. The second response contains multiple URLs pointing to the updated package, as shown in Figure 12.
Figure 12: The JSON response containing URLs for the software update
We downloaded and inspected the .pkg files listed in the response object, which are actually just ZIP archives. The archives contain a simple file hierarchy, as shown in Figure 13.
Figure 13: The .pkg archive file hierarchy
The manifest.json file contains information used to instruct the Harmony Hub’s update process on how to handle the archive’s contents (Figure 14).
Figure 14: The contents of the manifest.json file
The Harmony Hub’s update process executes the script provided by the installer parameter of the manifest if it is present within the archive. We modified this script, as shown in Figure 15, to create the /etc/tdeenable file, which causes the boot process to enable the SSH interface as previously described.
Figure 15: The modified update.sh file
We created a new malicious archive with the appropriate .pkg extension, which was hosted on a local web server. The next time the Harmony Hub checked for updates against the URL supplied in the modified GetJson2URIs response, we sent a modified response to point to this update. The Harmony Hub retrieved our malicious update package, and after rebooting the Harmony Hub, the SSH interface was enabled. This allowed us to access the device with the username root and a blank password, as shown in Figure 16.
Figure 16: The SSH interface was enabled after a reboot
As technology becomes further embedded into our daily lives, the trust we place in various devices unknowingly increases exponentially. Due to the fact that the Harmony Hub, like many IoT devcies, uses a common processor architecture, malicious tools could easily be added to a compromised Harmony Hub, increasing the overall impact of a targeted attack. However, Logitech worked with our team to quickly address the vulnerabilities with their current firmware, 4.15.96. Developers of the devices we place our trust should be vigilant when removing potential attack vectors that could expose end users to security risks. We also want to share Logitech’s statement on the research and work by the Red Team:
"At Logitech, we take our customers’ security and privacy very seriously. In late January 2018, security research firm FireEye pointed out vulnerabilities that could impact Logitech Harmony Hub-based products*.
If a malicious hacker had already gained access to a Hub-users network, these vulnerabilities could be exploited. We appreciate the work that professional security research firms like FireEye provide when identifying these types of vulnerabilities on IoT devices.
As soon as FireEye shared their research findings with us, we reviewed internally and immediately started to develop firmware to address it. As of April 10, we have released firmware that addresses all of the vulnerabilities that were identified. For any customers who haven’t yet updated to firmware version 4.15.96, we recommend you check the MyHarmony software and sync your Hub-based remote and receive it. Complete directions on updating your firmware can be found here.
*Hub-based products include: Harmony Elite, Harmony Home Hub, Harmony Ultimate Hub, harmony Hub, Harmony Home Control, Harmony Pro, Harmony Smart Control, Harmony Companion, Harmony Smart Keyboard, Harmony Ultimate and Ultimate Home."
A month and a half ago I posted an article in which I uncovered a series of Twitter accounts advertising adult dating (read: scam) websites. If you haven’t read it yet, I recommend taking a look at it before reading this article, since I’ll refer back to it occasionally.
To start with, let’s recap. In my previous research, I used a script to recursively query Twitter accounts for specific patterns, and found just over 22,000 Twitter bots using this process. This figure was based on the fact that I concluded my research (stopped my script) after querying only 3000 of the 22,000 discovered accounts. I have a suspicion that my script would have uncovered a lot more accounts, had I let it run longer.
This week, I decided to re-query all the Twitter IDs I found in March, to see if anything had changed. To my surprise, I was only able to query 2895 of the original 21964 accounts, indicating that Twitter has taken action on most of those accounts.
In order to find out whether the culled accounts were deleted or suspended, I wrote a small python script that utilized the requests module to directly query each account’s URL. If the script encountered a 404 error, it indicated that the account was removed or renamed. A reply indicated that the account was suspended. Of the 19069 culled accounts checked, 18932 were suspended, and 137 were deleted/renamed.
I also checked the surviving accounts in a similar manner, using requests to identify which ones were “restricted” (by checking for specific strings in the html returned from the query). Of the 2895 surviving accounts, 47 were set to restricted and the other 2848 were not.
As noted in my previous article, the accounts identified during my research had creation dates ranging from a few days old to over a decade in age. I checked the creation dates of both the culled set and the survivor’s set (using my previously recorded data) for patterns, but I couldn’t find any. Here they are, for reference:
Based on the connectivity I recorded between the original bot accounts, I’ve created a new graph visualization depicting the surviving communities. Of the 2895 survivors, only 402 presumably still belong to the communities I observed back then. The rest of the accounts were likely orphaned. Here’s a representation of what the surviving communities might look like, if the entity controlling these accounts didn’t make any changes in the meantime.
By the way, I’m using Gephi to create these graph visualizations, in case you were wondering.
Erik Ellason (@slickrockweb) contacted me recently with some evidence that the bots I’d discovered might be re-tooling. He pointed me to a handful of accounts that contained the shortened URL in a pinned tweet (instead of in the account’s description). Here’s an example profile:
Fetching a user object using the Twitter API will also return the last tweet that account published, but I’m not sure it would necessarily return the pinned Tweet. In fact, I don’t think there’s a way of identifying a pinned Tweet using the standard API. Hence, searching for these accounts by their promotional URL would be time consuming and problematic (you’d have to iterate through their tweets).
Fortunately, automating discovery of Twitter profiles similar to those Eric showed me was fairly straightforward. Like the previous botnet, the accounts could be crawled due to the fact that they follow each other. Also, all of these new accounts had text in their descriptions that followed a predictable pattern. Here’s an example of a few of those sentences:
look url in last post
go on link in top tweet
go at site in last post
It was trivial to construct a simple regular expression to find all such sentences:
desc_regex = "(look|go on|go at|see|check|click) (url|link|site) in (top|last) (tweet|post)"
I modified my previous script to include the above regular expression, seeded it with the handful of accounts that Eric had provided me, and let it run. After 24 hours, my new script had identified just over 20000 accounts. Mapping the follower/following relationships between these accounts gave me the following graph:
As we zoom in, you’ll notice that these accounts are way more connected than the older botnet. The 20,000 or so accounts identified at this point map to just over 100 separate communities. With roughly the same amount of accounts, the previous botnet contained over 1000 communities.
Zooming in further shows the presence of “hubs” in each community, similar to in our previous botnet.
Given that this botnet showed a greater degree of connectivity than the previous one studied, I decided to continue my discovery script and collect more data. The discovery rate of new accounts slowed slightly after the first 24 hours, but remained steady for the rest of the time it was running. After 4 days, my script had found close to 44,000 accounts.
And eight days later, the total was just over 80,000.
Here’s another way of visualizing that data:
Here’s the size distribution of communities detected for the 80,000 node graph. Smaller community sizes may indicate places where my discovery script didn’t yet look. The largest communities contained over 1000 accounts. There may be a way of searching more efficiently for these accounts by prioritizing crawling within smaller communities, but this is something I’ve yet to explore.
I shut down my discovery script at this point, having queried just over 30,000 accounts. I’m fairly confident this rabbit hole goes a lot deeper, but it would have taken weeks to query the next 50,000 accounts, not to mention the countless more that would have been added to the list during that time.
As with the previous botnet, the creation dates of these accounts spanned over a decade.
Here’s the oldest account I found.
Using the same methodology I used to analyze the survivor accounts from the old botnet, I checked which of these new accounts were restricted by Twitter. There was an almost exactly even split between restricted and non-restricted accounts in this new set.
Given that these new bots show many similarities to the previously discovered botnet (similar avatar pictures, same URL shortening services, similar usage of the English language) we might speculate that this new set of accounts is being managed by the same entity as those older ones. If this is the case, a further hypothesis is that said entity is re-tooling based on Twitter’s action against their previous botnet (for instance, to evade automation).
Because these new accounts use a pinned Tweet to advertise their services, we can test this hypothesis by examining the creation dates of the most recent Tweet from each account. If the entity is indeed re-tooling, all of the accounts should have Tweeted fairly recently. However, a brief examination of last tweet dates for these accounts revealed a rather large distribution, tracing back as far as 2012. The distribution had a long tail, with a majority of the most recent Tweets having been published within the last year. Here’s the last year’s worth of data graphed.
Here’s the oldest Tweet I found:
This data, on it’s own, would refute the theory that the owner of this botnet has been recently retooling. However, a closer look at some of the discovered accounts reveals an interesting story. Here are a few examples.
This account took a 6 year break from Twitter, and switched language to English.
This account mentions a “url in last post” in its bio, but there isn’t one.
This account went from posting in Korean to posting in English, with a 3 year break in between. However, the newer Tweet mentions “url in bio”. Sounds vaguely familiar.
Examining the text contained in the last Tweets from these discovered accounts revealed around 76,000 unique Tweets. Searching these Tweets for links containing the URL shortening services used by the previous botnet revealed 8,200 unique Tweets. Here’s a graph of the creation dates of those particular Tweets.
As we can see, the Tweets containing shortened URLs date back only 21 days. Here’s a distribution of domains seen in those Tweets.
My current hypothesis is that the owner of the previous botnet has purchased a batch of Twitter accounts (of varying ages) and has been, at least for the last 21 days, repurposing those accounts to advertise adult dating sites using the new pinned-Tweet approach.
One final thing – I checked the 2895 survivor accounts from the previously discovered botnet to see if any had been reconfigured to use a pinned Tweet. At the time of checking, only one of those accounts had been changed.
If you’re interested in looking at the data I collected, I’ve uploaded names/ids of all discovered accounts, the follower/following mappings found between these accounts, the gephi save file for the 80,000 node graph, and a list of accounts queried by my script (in case someone would like to continue iterating through the unqueried accounts.) You can find all of that data in this github repo.