Tag: SSH

Cloudways reuses SSH host keys on Vultr, Linode, AWS, and GCP

I was surprised to discover that Cloudways reuses the same SSH host key when provisioning Vultr, Linode, AWS, and GCP servers. Apparently SSH host key reuse isn’t a new problem and some fingerprints have been seen on as many as 250,000 devices.

I currently have two Vultr servers on Cloudways:

Vultr Server 1:

Vultr Server 2:

How do I get the server SSH fingerprint?

The easiest way is the run this at a command prompt (replacing hostname with the domain or IP address of the server):

ssh-keygen -E md5 -lf <( ssh-keyscan hostname 2>/dev/null )

Can I reproduce this on Vultr servers through Cloudways?

I created one in Dallas and was assigned the IP 144.202.70.5 and got the fingerprint b7:73:78:ac:f4:9f:01:ad:b5:7e:e2:e6:a5:93:1c:a2. I created a second one in Dallas and was assigned the IP 45.32.202.130 with the same fingerprint of b7:73:78:ac:f4:9f:01:ad:b5:7e:e2:e6:a5:93:1c:a2.

That fingerprint is in use on 269 servers in at least 5 countries.

What about Linode on Cloudways?

I created a Linode server in Dallas and was assigned the IP 45.79.35.122 with the fingerprint 55:95:ea:0d:aa:37:0a:96:c6:ee:12:4f:50:9e:ab:9a. The second server in Dallas got the IP 45.56.127.216 and the same fingerprint of 55:95:ea:0d:aa:37:0a:96:c6:ee:12:4f:50:9e:ab:9a.

That fingerprint is in use on 120 servers in at least 5 countries.

What about Amazon Web Services on Cloudways?

I created two servers and got the same fingerprint. It’s in use on 70 servers.

  • Location: Virginia, IP: 3.231.5.23, Fingerprint: da:ca:e9:fb:10:0b:61:19:5b:23:ca:39:36:60:ff:af
  • Location Virginia, IP: 3.209.101.72, Fingerprint: da:ca:e9:fb:10:0b:61:19:5b:23:ca:39:36:60:ff:af

What about Google Cloud Platform on Cloudways?

I created two servers and also got the same fingerprint. It’s in use on 47 servers.

Location: Iowa, IP: 104.197.12.88, Fingerprint: b2:d0:23:6d:90:50:27:e6:92:53:b6:98:0f:18:52:f8
Location: Iowa, IP: 35.188.175.177, Fingerprint: b2:d0:23:6d:90:50:27:e6:92:53:b6:98:0f:18:52:f8

What about Digital Ocean on Cloudways?

All three of the Digital Ocean servers I created had unique fingerprints and didn’t show up on Shodan:

  • Location: New York, IP: 67.207.95.17, Fingerprint: d1:76:7d:b0:2d:fc:9e:bb:f3:40:f7:53:b7:87:fe:ca
  • Location: San Francisco, IP: 157.230.134.54, Fingerprint: e8:3f:72:17:e3:ba:02:e6:e7:10:a9:4a:3d:d0:83:24
  • Location: San Francisco, IP: 157.230.159.241, Fingerprint: 2b:5d:55:7c:36:e7:04:86:17:66:ad:40:77:7e:cd:36

How bad is this?

It doesn’t appear this directly compromises my servers. However, it does allow an attacker to impersonate my server. They could achieve this by either stealing the private key off of another compromised Cloudways server or by provisioning a new Cloudways server and then finding a way to redirect SSH traffic to the impersonated server. Reusing an SSH key across multiple servers might make sense if all of them belonged to me. (It’s probably what Github and Bitbucket do so that their customers can use Git over SSH.) However, since these Cloudways servers are assigned to many customers, it’s a risk, and it’s definitely not good.

TL;DR: There’s a possibility of an impersonation/MitM attack but intercepting and successfully reading the encrypted SSH traffic shouldn’t be possible.

It also makes it easy to identify a large list of servers provisioned by Cloudways. If someone found a vulnerability on a Cloudways server, this would make it easy to target the vulnerable servers.

Is Cloudways going to fix this?

I reported this via live chat on October 3, 2019 and then to their privacy email address on October 3, 2019, and heard back on October 9, 2019. They don’t seem to consider this a major issue, but they did say they are working on fixing it.

Privileged SSH Port Forwarding with Sudo

There are many articles about privileged SSH port forwarding but not much about properly using SSH keys and config files.

The problem is that upon typing:

ssh dakara -L 80:localhost:80

the response is:

Privileged ports can only be forwarded by root.

The obvious solution is to just use sudo:

sudo ssh dakara -L 80:localhost:80

but this produces:

ssh: Could not resolve hostname dakara: Name or service not known

Unfortunately, “dakara” is a host configured in ~/.ssh/config and not available to root. This can be fixed with:

sudo ssh -F ~/.ssh/config dakara -L 80:localhost:80

but this tries to connect as root and prompts for a password. Adding “-l $USER” sets the user name to my user name (This could also be set in the SSH config file.), and adding “-E” to sudo preserves the environment allowing my SSH agent to be used.

sudo -E ssh -F ~/.ssh/config -l $USER dakara -L 80:localhost:80

Now everything connects, and I am not prompted for a password.

Bad Bash Sourcing

A while back I switched such that my .bash_profile sources my .bashrc instead of the reverse. Based on a few sources, this seems to be the preferred approach. Today I decided to push those files to a few servers that haven’t been updated lately. I transferred my .bash_profile and then my .bashrc:

zac@dakara:~$ scp ~/.environment/bash/bash_profile lansky:~/.bash_profile
bash_profile                                  100%  120     0.1KB/s   00:00    
zac@dakara:~$ scp ~/.environment/bash/bashrc lansky:~/.bashrc
lost connection

What happened? My .bash_profile sources my .bashrc, but since I hadn’t yet replaced the .bashrc, it still sources my .bash_profile. That means infinite loop and that I am an idiot. SSH, rsync, and scp are all broken. There is no way to remove or replace either of those files without another account. I guess it’s time to open a support ticket at DreamHost.

The moral of the story is to always be careful how you source those files. I usually try to keep an SSH session open on the remote machine when I modify those files in case I break login, but this time I didn’t. A few searches didn’t reveal any solutions. I’d love to know if someone has a way of resolving this without access to another account on the remote machine.

The Typical Vim Reaction

Every now and then I feel permitted to go on a rant. It’s unfortunate because this isn’t even a particularly good rant. Why do so many of my (instant messaging) conversations with others about Vim look like this?

Chris: do you use an IDE?
Chris: and if so which one?
Zachary: um, I generally use vim:-P
Chris: gah

Vim is a great text editor. It’s a step up from Ed which is the standard text editor. I started using Vim five years ago. That was about the time that I discovered secure shell, and I started administering servers and other computers remotely. It turns out that Vim was the best text editor over a secure shell session, and since most of my machines ran Mac OS X at the time, Emacs was a terrible option. (I’m not really sure what you call Emacs on Mac OS X. You probably shouldnt’ call it Emacs.) I limped along using very basic Vim functionality over secure shell for a while. Then I discovered a useful graphical tutorial for Vim, and it became considerably more useful.

Now I use Vim because every other text editor or word processor is slower and requires the use of a mouse. (Gah!) I write most of my documents in LaTeX using Vim. I read my email in Mutt and compose emails in Vim. I use Vim almost exclusively to edit code and configuration files on my workstations and servers which works well because it does a good job of syntax highlighting and smart indentation.

A few years ago, I wouldn’t have thought I’d be using Vim exclusively, and it was somewhat by accident that I switched, but now that I am using it, I would be unable to go back.

Install Flash on Mac OS X from the command line

My mother needed the latest version of Flash to view a web page. I decided this was a good opportunity to install the package from the command line over secure shell. She has a PowerMac G4 running Mac OS X 10.4. It turns out that the update didn’t work, but I relearned a bit about some useful command line tools for Mac OS X. I always start in the temp directory. Then I downloaded the installer and unzipped it. That’s all pretty straightforward.

cd /tmp/
curl -O http://fpdownload.macromedia.com/get/flashplayer/current/install_flash_player_osx_ub.dmg.zip
unzip install_flash_player_osx_ub.dmg.zip

Next, I used “hdiutil” to mount the disk image and then changed to that directory.

hdiutil attach Install\ Flash\ Player\ 10\ UB.dmg
cd /Volumes/Install\ Flash\ Player\ 10\ UB/

Then I used “installer” to install the package specifying the package with “-pkg Adobe\ Flash\ Player.pkg” and the target volume with “-target /”.

installer -verbose -pkg Adobe\ Flash\ Player.pkg -target /

It will run through the installation and output some information. After that, I changed back to the temp directory. I initially tried using “umount” to unmount the disk image, but that is a bad idea because it doesn’t fully unmount the image. The better approach is to use the “hdiutil” again. I find the volume to unmount by using the “df” command. Once it is unmounted, I deleted the zip file and the disk image.

cd /tmp/
hdiutil detach $(df | grep Flash | awk '{print $1}')
rm install_flash_player_osx_ub.dmg.zip
rm Install\ Flash\ Player\ 10\ UB.dmg

Most applications for Mac OS X come in disk images, and many of those have package installers. This is a quick way to install software on a remote Macintosh or a way to automate installations with scripts.

Add an SSH Key to an SSH Agent on a Different Machine

I recently discovered that I can add an SSH key to an SSH agent on a different machine. I find this useful because I am very protective of my SSH keys and prefer to keep them on my desktop (Dakara) and not on my laptop (Adria).

On Adria, you can see that I have no keys added:

zac@adria:~$ ssh-add -l
The agent has no identities.

I then secure shell to Dakara and forward my agent (-A) from Adria. I still have no keys added:

zac@adria:~$ ssh dakara -A
zac@dakara's password:
No mail.
Last login: Mon Feb  9 17:30:49 2009 from adria.lund
zac@dakara:~$ ssh-add -l
The agent has no identities.

Next, I add a key from Dakara to my agent on Adria:

zac@dakara:~$ ssh-add
Enter passphrase for /home/zac/.ssh/id_dsa:
Identity added: /home/zac/.ssh/id_dsa (/home/zac/.ssh/id_dsa)
zac@dakara:~$ ssh-add -l
2048 27:81:f8:7f:38:75:6b:ce:95:e4:46:62:02:9c:84:bd /home/zac/.ssh/id_dsa (DSA)

When I log out of Dakara, the key is still available on Adria:

zac@dakara:~$ logout
Connection to dakara closed.
zac@adria:~$ ssh-add -l
2048 27:81:f8:7f:38:75:6b:ce:95:e4:46:62:02:9c:84:bd /home/zac/.ssh/id_dsa (DSA)

Now my key is available on Adria without ever being on Adria. Even with physical access to my laptop, it would take a sophisticated hacker to steal my SSH key.