Remote login is not the same as local login

It's now been just over a month since I installed my PANOPTES exoplanet survey telescope in dome 7 of the Mars Center for Science and Technology at Wheaton College. I had hoped it would be operational by now, but sadly we're not there yet. This is my tale of woe what I've learned along the way. My apologies for the length of this post; I needed to get it off my chest! There is a brief summary at the end if you want to cut to the chase.

For context, I signed on in April, 2017 as a "beta test builder", meaning that I was testing the instructions for building a PANOPTES baseline unit, an automated telescope, including building 3 electronics boards, fabricating mounting plates for a dual camera telescope, water-proofing an equatorial mount, creating a pier to which the mount is secured, etc.

The design includes a small computer (Intel NUC) running Ubuntu Linux as its operating system; the telescope is managed by software (POCS, PANOPTES Observatory Control System) running on the NUC. The instructions written to date don't go into much detail on how to setup the NUC or run the software, so I've been blazing a trail, especially w.r.t. Linux configuration, which is the focus of this post.

Planning Ahead

I knew that once the scope was installed at Wheaton, it would be behind a firewall, preventing me from direct accessing it via SSH. In time I would be able to get VPN access, but until then it would be vital to have some way to access the NUC, so I setup Chrome Remote Desktop (CRD) on the NUC. This would enable me to login to the NUC from home, so long as both the NUC and my laptop each had access to the internet. I tested it at home, where the NUC was connected to my home Wi-Fi system.

You can't get there from here

When I planned to go to Stellafane with my PANOPTES scope, I thought that I might want to login to its computer from my laptop, but knew that there wasn't much Wi-Fi at Stellafane, so I modified the wired Ethernet settings both my laptop and the NUC so that they had fixed addresses 10.0.0.1 and 10.0.0.2, with the intent of being able to run an Ethernet cable directly between the two.

(Note: I've since learned that Ubuntu, or Linux in general, has support for networks without routers or gateways, where each computer picks its own address and can discover others on the network.)

By the time I went to install the scope at Wheaton on October 7th, I'd forgotten about the fixed address. When the NUC was connected to the college's wired network, it didn't appear on the network. This also meant that it didn't have access to the internet, so CRD wouldn't work either.

We (Sean & Joe, Wheaton sophmores, and I) tried a number of things to figure out why it wasn't working, including monitoring the traffic on the local Ethernet segment to see if we could identify the address it was using. I eventually remembered the fixed address, and needed to find a way to login into the NUC to remove that. Unfortunately, I'd left the laptop that has an Ethernet port at home, so I setup my phone with a Wi-Fi hotspot with the same SSID and password as my home Wi-Fi, thus providing a path to the Internet. CRD was finally working.

You can't change that from here

I logged into the NUC, and opened the Network Settings, with the intent of removing the fixed address. To my surprise, the dialog buttons for editing the IPv4 settings were disabled. I thought perhaps I'd use some command-line or config file to "hard-code" the address of the Ethernet connection, but I could find no evidence of that. We poked and prodded for a quite a while before I gave up, asking Joe to disconnect the NUC from all of its cables and take it out of the control box, with the plan of setting it up at home and debugging the problem "at leisure"... if you could call it that.

You can't connect this to that

As I was driving the 45 miles from Wheaton to home, it suddenly occurred to me that I'd asked Joe to disconnect one too many cables: the NUC has a mini-HDMI jack, while my monitors use the full-size HDMI connectors, so I had an adapter cable plugged into the NUC. This would allow for dragging a monitor, keyboard and mouse out to the dome if absolutely necessary. So, I ordered a pair of adapter cables from Amazon (more time efficient than driving 90).

Once home I setup the NUC in the loft above my garage, where it connected to the Google Wi-Fi point in the same room. Once I had some free time, I sat in the family room and started up CRD on my laptop, and selected the NUC as the computer to connect to. The wait cursor spun for a while, then I got an error message saying that it couldn't connect because of some problem with the network I was using. WTF. I'd tested this previously, why is it not working now! I was so frustrated. I figured that I'd have to wait for those adapters to arrive in a few days.

When I was at work that week, I took the problem of removing the fixed address to the folks at my local Tech Stop (Google's internal help desk, among other responsibilities), to see if they had any ideas. I was asked if I could login to it remotely; well, that seemed unlikely, but to my surprise, I could login to the NUC using CRD. Wow, how was that working! Well, let's hold that thought and deal with the fixed address...

It still wasn't at all clear why the Network Settings buttons for changing things were greyed out, but Frank at Tech Stop had the idea of launching the dialog from the command-line, where we could use sudo to ensure it was running with elevated permissions. Thankfully, this worked, and he was able to remove the problematic link, leaving the NUC with a wired internet connection that would use DHCP to get an address. Phew. We still didn't know why the buttons had been disabled, but at least the NUC would be able to connect to the Wheaton network.

You can't connect to that from here

Around that same time I bought a pair of Chromecast devices (I had the first generation, and want to upgrade). Like the rest of the computing devices in the house, they get internet access via our three Google Wi-Fi Points (what they call the Wi-Fi access points/extenders). There is a single primary point in the basement, providing Wi-Fi and wired Ethernet to the rest of house, and two other points, one upstairs in a bedroom, the other in the loft near the NUC. I setup the first Chromecast in the family room, which went smoothly (though I do wish they would install software updates in the middle of the night, not immediately after network setup, just when I want to try using my new toy).

My experience with the second Chromecast wasn't so smooth. It showed a code on the screen, the Google Home app had no trouble finding the Chromecast to be setup, showed me the correct code (the same one displayed on the TV), but then the process stalled. Somehow, the app wasn't able to talk over the home network to the Chromecast. WTF! The other setup was so smooth. I tried many times, but it just couldn't connect over Wi-Fi to the device right in front of me, with the Google Wi-Fi right next to the TV. I hate these sorts of random failures, so I left it for a while.

Eventually, the fact that I could connect to the NUC from work but not from my family room led me to wonder about network topology and connectivity. Could it be that the Google Wi-Fi points weren't providing complete connectivity within the house?

I decided to move the working Chromecast to the loft. It worked there. I moved the non-working Chromecast to the family room, and again tried to set it up. This worked. And continued working when I swapped the location of the two Chromecasts.

I may have isolated the problem. I tried wandering around the house, disabling and re-enabling Wi-Fi on the device with me, so that I would connect to the nearby Google Wi-Fi point with strong signal. If found that if the initiating device is connected to the primary access point (as my phone was when I tried to setup the Chromecast), and the receiving device is connected to a mesh point (i.e. a Wi-Fi extender), then there isn't a path between the two devices. This may not even be a general situation; it may only apply to trying to convert a device->Google->device connection into a device->device connection (something to do with the STUN protocol, at a guess).

You Can't Do That From There

Once the NUC was reinstalled at Wheaton on October 14th, it joined the campus network and CRD was working. I was really relieved.

Now we needed to prepare for polar alignment, for which we're going to need the cameras working and in focus. So, let's take a picture using the command-line. The first thing I did was to simply try to enumerate the cameras (gphoto2 --auto-detect). Nada. The cameras were missing. OK, a trip out to the dome to make sure they have power, that the USB cables are plugged in. Everything looked fine.

lsusb showed that the cameras were in fact visible to the NUC, but somehow we couldn't interact with them. This worked fine at home, why not now? Various prodding showed no sign of the problem.  After a while, we called it a night, and I went home to stew study the problem.

Again using CRD, I enabled debug logging in gphoto2, which showed it considering all sorts of USB devices as possible cameras, including the Arduinos, but not the actual cameras. I could see them with lsusb, get there properties, but they were invisible to gphoto2. But why?

At some point I checked the permissions on the USB devices for the cameras (e.g. /dev/bus/usb/002/007), and discovered that they were owned by group plugdev, while all the other USB devices were owned by root. Why would that be? Why would they not all be the same? Why had I been able to take pictures previously but not now?

You need permission to do that from there

I started using Unix BSD in the mid-1980s, at a time when user permissions were pretty simple. A user was represented by a line in the passwd file and adding a user to a group simply meant adding the user name to the list of members of that group, part of a single line of text in the groups file. Things have evolved considerably since then. While working on this plugdev question, I learned that Linux has a feature called PAM, Pluggable Authentication Modules, whereby system administrators can configure the behavior of APIs used by login, su and sudo, among others. This includes modifying the set of groups of which a logged in user is a member (see pam_groups for more info).

It appears that on single-user Linux installations (e.g. for a workstation or laptop, as opposed to a server), the default PAM configuration grants membership in the plugdev group (and some other groups) to locally logged in user (really, the session). The idea is that the physically present user may temporarily connect devices such as a camera, and should of course be able to access them, while by default preventing a remotely logged-in user from messing with such devices.

I later realized this same thinking applied to the problem with the Network Settings dialog: when I logged in locally, my session was added to an admin group, thus enabling the me to create the connection object with the fixed address. But later, when logged in with CRD, my session wasn't a member of this group, and thus by default it wasn't allowed to edit those settings; we started the Network control program with sudo nm-connection-editor.

Go on, give yourself permission

While we installed the desktop version of Ubuntu (The PANOPTES situation is different, as is the case of using CRD to log into your personal machine from a remote location. So, what to do? I needed to permanently add the my account to the plugdev group, but how? A search soon turned up a page with lots of examples of how to modify a user's account using the usermod command, including modifying the groups of a user:

sudo usermod -G # DON'T COPY THIS

So I made the change. Since one needs to log out and back in again for this change to take effect, and the NUC is single user, I just rebooted the system. After waiting a minute I tried to log back in again with CRD. No response. Whoops! What happened? That required a bit of study, study I clearly should have done before I ran that command.

It turns out that usermod -G replaces the list of groups that the user is a member of. usermod takes -a as an option to indicate that the command should append to the list. Argh. That command was too powerful. But why couldn't I login?

Since I didn't yet have VPN access, I asked Sean and Joe to help out. Sean logged in via ssh and reported back on the set of groups that the user panoptes was in. As expected, it was only in the groups panoptes and plugdev (note: by default, when a user is created, a group of the same name is created and is the primary group of the user is that group.) And we no longer had the ability to run sudo, so couldn't fix that user.

Fortunately we had another user on the system, james, that was the first created on the system. But Sean couldn't log in as user james. It took days for me to remember that I'd changed the initial weak password to something a bit stronger; thank goodness I'd recorded it in LastPass.

(Note: by default, the first user created on Ubuntu has greater privileges, i.e. is a member of additional groups, on the assumption that this is the "owner" of the machine.)

Sean was able to add user panoptes to the groups that user james was a member of, after which user panoptes was able to sudo. Phew. But after a reboot CRD was still not working. Sigh.

You should dig a tunnel from here to there

With Chrome Remote Desktop not working, I finally asked Prof. Maitra at Wheaton about VPN access so that I could use ssh to login from home. To the surprise of both of us, the procedure was quick and easy. I soon had the VPN software running on a Chromebook and was able to login to the NUC.

Poking around, it wasn't immediately obvious why CRD wasn't responding when I tried to connect. I could see that the service had started, but what ever needed to happen to enable user panoptes wasn't happening. Fortunately CRD isn't entirely opaque, but instead uses shell scripts to configure the service. Reading those scripts I discovered that it was looking for users that were members of the group chrome-remote-desktop. So, another group to which I needed to add the user. After doing that, and restarting the service, I was finally able to use CRD to login to the NUC.

Summary

• A local login session may have different privileges than a remote session, such as being added to user plugdev. Whether it does depends on PAM.
• When a USB device is connected, the virtual device representing it may have its configuration, including ownership, changed by rules in /etc/udev. For example, installing gphoto2 adds rules that set the group for camera devices to be plugdev.
• sudo usermod -G is dangerous! It removes the user from its existing secondary groups. Use option -a to instead add the user to additional secondary groups.
• Make sure you have more than one privileged user on your machine, and keep track of the passwords, else you may find yourself locked out. (It is possible to boot into a root shell, from where you can do anything, but that requires you be present, and have a keyboard and monitor hooked up. Not so convenient for embedded or remote systems.)
• Chrome Remote Desktop works great, except when you kill it.