Raspberry Pi fails silently to get IP address if it’s already in use

 An interesting experience while working on some raspberry pi’s around here the other day. After a power failure the network was strangely and randomly hosed up due to what turned out to be a corrupt ARP table in the main switch that had a stroke when the power came back on. After I replaced the switch everything came back to life except a single Raspberry Pi which remained offline. 

I could still reach it via ssh and it seemed to be happy that way. This was using the local network name. Trying to ping it via it’s local network name failed, but succeeded when using just the static IP address.

I then ssh’ed into it via it’s IP address and found myself connected to a totally different Raspberry Pi that is sitting on my workbench getting reloaded as it’s CF card died and needed to be rebuilt. I had obviously confused the machines when looking up what their static IP addresses should be and had assigned this new rebuilt one the wrong IP. 

I returned to the ssh session with the device having a problem and ran ifconfig to see what it thought it’s IP address was. It had a full set of all the IPv6 information necessary, and that is what ssh was using, but it had no IPv4 address assigned at all. 

There was no error message I could find, nothing in dmesg, nothing in the systemctrl status for the dhcpcd service. Just no IPv4 address. 

I fixed the new one to have the proper IP address and rebooted them both and they both came back just fine. I suppose it I had been running the GUI version of the OS on either one of them I’d have gotten a “someone else is using your IP address” type error, but without I did not see why the machine seemed only partially available over the network.

It took far too long to figure out what was going on because no errors seemed to be generated. If I had not figured it out when I did I suppose it would eventually have occurred to me to check the dhcpcd logs, if there are some somewhere. 

Airplay (Shairport-sync) on the Raspberry Pi and Disabling Wifi Power Save

As of this writing on Jul 19th 2021 I am having trouble with the packaged version of shairport-sync. This page outlines how to fairly simply compile it from a different repository and also to disable the problematic wifi power save mode which can cause skipping.

 

I am now running 4 different raspberry pi’s with the Shairport-sync software to receive airplay streams from our phones and computers and they work great. The ones connected via WiFi tend to have dropouts while the ones with physical connections seem to work perfectly all the time and almost never skip. This seems to be in spite of the power or memory of the machines in question which range from pi zero’s through Pi3 B+ and a couple of Pi4’s now.

I recently upgraded the one we use the most to a Pi4 trying to see if that would solve the problem, and I wanted it to run some other software at the same time and thought it would be worth doing a reload on that older Pi3. 

I initially tried the standard distribution of the shairport-sync installed via:

sudo apt install shairport-sync

That worked great except that I could not seem to make any changes to the configuration file. If you just want to use your hostname as the share name and do not need to change any other configuration options then this will work just fine and you don’t really need to do anything else other than enable it via the systemctl enable shairport-sync and then reboot. I didn’t want my hostname to be the share name and whenever I made any changes to the configuration file, which the standard package installs in /etc/shairport-sync.conf instead of the other standard location where it has been as long as I can remember in /usr/local/etc/shairport-sync.conf, it would just crash upon starting up.

I uninstalled it and decided to scrape around and do an actual build of the software from the same repository that I used originally for the others. There are several forks available the one I used successfully is this one by github user Mike Brady

First install some necessary packages to be able to build it, this should be done after you’ve updated the software on your pi via the normal methods.

sudo apt install autoconf libtool libdaemon-dev libasound2-dev libpopt-dev libconfig-dev
sudo apt install avahi-daemon libavahi-client-dev
sudo apt install libssl-dev
sudo apt install git

Now we need to clone the git repository and to the configure and make steps:

cd ~
git clone https://github.com/mikebrady/shairport-sync.git
cd shairport-sync
autoreconf -i -f
./configure --with-alsa --with-avahi --with-ssl=openssl --with-systemd --with-metadata

If there are other options you wish to compile in like the mqtt server or others that configure line is the place to add them.

Once those are done you can do the make and install:

make
sudo make install

And enable the app to run at startup like this:

sudo systemctl enable shairport-sync

If you wish to change the name of the share as it appears to iOS devices or increase the timeouts for resyncs which I also did for a little more leeway with a bad wifi connection you can edit the configuration file which should be placed in /usr/local/etc/shairport-sync.conf via the command:

sudo nano /usr/local/etc/shairport-sync.conf

You’ll find the name entry at the top of the first section, you can uncomment theirs and change it or just add a line like:

name="Living Room";

you can also scroll down a bit to find the drift and resync tolerances. If you really want to have several servers all receiving the stream at the same time and playing the same music in sync then you should not alter these, but if you are mostly using one at a time as I do for whatever room I’m in and not the entire system you can increase their default times by altering the default lines which look like:

// drift_tolerance_in_seconds = 0.002; // allow a timing error of this number of seconds of drift away from exact synchronis$
// resync_threshold_in_seconds = 0.050; // a synchronisation error greater than this number of seconds will cause resynchron$

uncomment those and experiment with larger values. For myself I increase the drift tolerance to 0.05 and the resync threshold to 0.1

There are SO many suggestions out there on how to force the wifi power save mode to stay off after a restart but only this one worked for me. You can check to see if the wifi power save is on by running the command:

iw dev wlan0 get power_save

and change it via the command:

sudo iw dev wlan0 set power_save off

It seems that despite all protestations by many people that this is now off by default on their pi’s it is most definitely on when mine reboot. We need to create a systemctl command to run at startup that will execute the command above. If you have more than one wifi interface or are using the descriptive names option in the newer Pi OS’s you may have to use a different name than wlan0 but that is the default if you haven’t changed anything as of this writing.

Create the file with the nano editor like this:

sudo nano /etc/systemd/system/wifi_power_save_off.service

Paste the following into the file and save with ctrl-o and then ctrl-x to exit the editor.

[Unit]
Description=Set WiFipower save off so no skipping of audio
After=sys-subsystem-net-devices-wlan0.device

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/iw dev wlan0 set power_save off

[Install]
WantedBy=sys-subsystem-net-devices-wlan0.device

In order for the systemctl system to see this additional file you have to run the reload daemon command and finally enable it with these 2 commands:

sudo systemctl daemon-reload
sudo systemctl enable wifi_power_save_off

Now reboot your pi and check to see that your shairport-sync share shows up in your iOS devices properly and then run the "iw dev wlan0 get power_save" command again to make sure that it has run properly.

Dead Instrument Cluster, part: The Second

 See my previous article and then know that 4 days later it died again. 

I was lucky in that I never cancelled the appointment to get it fixed so when that day rolled around I drive it in without any instrument display. The only danger was that I didn’t know how fast I was going which was easily solved by running Wayz on my iPhone and that I didn’t know how much gas it had in it but my wife assured me it was nearly full when it went dark again ;) 

They took over 2 hours to decide that it was actually broken again and that they would need to order a new instrument cluster. I told them that in moment one, but OK whatever. Since the last one failed only 4 months ago they are going to replace it under warranty. 

They really didn’t want me to take it back home without a speedometer, so I had to sign a hand written phrase on the receipt to the effect that I had been counseled about the dangers of driving it without a speedometer. I signed it, I have one. At least approximate on my phone. I tried to explain that too.

Sadly the part is back ordered so they don’t know how long it will take to get a replacement.

Luckily I decided to take it apart yet again and do some lower level debugging of the connections to the board. This time I was going to take it apart, actually de-solder a component, and then re-solder it with leaded solder and put it together enough to do a quick test. That way I can be sure what I touched and didn’t unlike the first time when I just randomly reflowed anything that was large enough for me to safely handle without a lot of prep work.

Step one was just the connector to the board itself. I used a combination of a solder sucker and solder wick to remove maybe 80% of the solder holding the connector in place from the back. And then generously fluxed it and resoldered it with standard leaded solder. Put the device together enough to test it with both displays connected again. and the screws holding them in installed so that the very weak little connectors for them wouldn’t get broken or torn and it lit right up.

So the problem is definitely in the connection of the connector and not in the other power supply components that I resoldered in the first entry of this saga.

If you think you can do this I would encourage you not to. See my first entry that has more info on the very delicate connectors for the screens and other parts. But since it’s dead anyway and they were going to replace it anyway I decided it couldn’t get any more dead than it was so I took a chance.

If it is still working when I finally get the call from them in the future that the real part is back in stock and to bring it in for replacement it’s going to be a tough decision what to do. Do I keep the one I’ve been fixing and hope it keeps working? Do I let them replace it with another one that is definitely going to have the same problem but now I know how to fix it? Do I let them even notice that I fixed it? Or just claim that it spontaneously started working again and has been on and off for all that time? I like that last one but I don’t want to lie to them. I’d really like to keep this one as a spare and let them replace it with a new one. Thats the offer I’ll make them I think. I’ll be sure to update with more information if the backordered replacement ever arrives ;) When we got this first replacement we had to tell them what the odometer reading was so that they could get it burned into the new one. They didn’t ask me this time so either they will call back when it is ready to go and then spend more time getting that burned in or it will just come with it set to 0. Since the car only has 40k miles on it it would be hilarious to reset it to 0. I’m pretty sure that they will call to ask me. The temptation is to tell them less than the last time just for fun ;) We keep our cars until they are ready to be junked so I won’t be using that to sell it to someone else fraudulently, but just curious what they would say...

Instrument Cluster Board Level Repairs And The Evils of Lead Free Solder

 My wife drives a Ford C-Max “energie” plugin hybrid. It is an awesome little car that Ford no longer bothers to sell in the US because they failed to market it well enough and it became unprofitable for them to keep them around. Or it is possible that Americans really are only interested in double sized trucks and SUV’s and simply don’t want smaller cars at all. I don’t know but I suspect a combination of both problems exist. In any case I was in Chicago 4 months ago caring for my mom and I got a call from home saying that the dashboard on the car was completely blank. Had to manage remotely getting her a rental and getting the car fixed at the dealership. A couple of thousand dollars later we had a new instrument cluster as the old one was dead. It took weeks to get fixed as a new one had to be ordered with the proper odometer reading burned into it. 

This morning it did it again! Not even a full 4 months later! Car starts but the instrument cluster stays dark and eventually the touch screen/ radio comes up with a message that says “Vehicle Network Communication Error” and will not do anything else. Other interesting side effects are that the climate controls defaulted back to Celsius without being able to talk to the instrument cluster. 

Since there is no way to control most of the car and no way to know easily how fast you’re going the car is simply not safe to drive that way beyond just not having to have it towed to the dealership. 

The local Ford dealership said that since this was an electrical problem it required special attention from special people and they wouldn’t be able to even begin to work on it for 5 days! Don’t bother bringing it in before this coming Wednesday! And no willingness to tell me if this would be a warranted repair since they just did it. 

As I sat here fuming about how long we’d have to rent a car for as they ordered another new one for us I decided that since it was dead anyway I’d take it out and have a look at it. I’ve rarely regretted doing that and have often found that I could make a repair to something that would have cost a lot of money to replace unnecessarily. 

It turns out that it’s very easy to remove the instrument cluster from most Ford cars. There are lots of youtube videos about it but basically you pull the cowling that connects the rubbery sheet that covers the top of the adjustable steering column to the plastic under the cluster straight towards you and then that reveals the 2 7mm bolts that hold in the cluster. Make sure to adjust the steering wheel all the way down and forward or you can’t get it out. On this C-Max there are 2 clips that you need to force to let go directly behind it, but I have no idea if just yanking it straight out is the right thing to do for your model or not. It was the right thing for mine. Grab the cluster and pull it straight towards you until they let go. If you do it wrong or yours is held on differently though you might break something expensive so find a take apart video for your specific model before you start applying excessive force to anything.

There is a single connector on the back that you can reach around and find the pinch spot to easily remove it. You may want to put a towel over the cowling part or any other exposed plastics to keep from scratching them as you figure out how to get the cluster out from between the steering wheel and the dashboard. It is possible, but only just.

 

It actually look me longer to clear enough room on the desk to work on it than to get it out of the car ;) After carefully removing all the other plastic coverings you get to the point of having to remove the speedometer meter pointer. There is no D shaped or otherwise locking connection point for this. It is strictly a friction fit piece of plastic pushed down over the pin that comes up from the stepper motor that drives it. At this point it is recommended that you take a piece of tape, I used electrical that didn't seem to damage the plastics or paint or printed parts underneath it but go easy on it, and make that line up with exactly where the pointer is at it's sopping point. You'll notice it is a little below zero but if you get this wrong when you put it back together you'll be driving at a speed other than what you think you are. "I took the dash apart Officer and evidently did not get the pointer back on right" is not going to get you out of a speeding ticket. Put some tape on the underside lined up with it's stop point so that you get it back together right.

I used a "spudger" and very gently pried it up on both sides. The plastic these parts are made of feels like the lowest quality styrofoam and I believe they would break if you so much as looked at them wrong. If you don't have a proper spudger put some cloth or something to protect the black plastic background as you pry upwards or you will damage it and have to look at the light soaking through your scratches forever.

The next thing you have to remove to get to the board are the connections to the 2 screens on either side. I didn't capture a picture of this but I HATE these kind of board connectors. I've ruined many similar ones on Raspberry Pi camera connectors or others. In this specific one the connectors had a black plastic bar across the ribbon connector that you VERY GENTLY pried upwards to release the ribbon cable and then could slide the ribbon cable out. For me they worked very well but if you have any qualms about this sort of thing don't do it. You must remove them to safely work on the board though because once you take off the front plastics nothing else is holding them in place and any force or movement will be transferred to the connector which will either rip the cables or tear the connectors right off the board. It does not take much force to do that at all so really be seriously careful at this point. I would also make sure you don't confuse the left and right screens. I couldn't see any difference in looking at them but it seemed like a good idea not to put it back together backwards. They would certainly fit just fine backwards but who knows if they have different firmware or something.

And finally you'll have the nice board you can have a good look at!

It is, of course, almost entirely surface mount and there just isn't much you can do about that unless you're much more of a professional soldering god than I am. I put it under the inspection microscope and can see a lot of connections that looked a bit iffy to me. The first thing I did was to put some flux over the through hole pins that go to the connector on the opposite side of the board. You can see them as two rows of parallel pins in the picture aboe right in the middle of the board towards the top. I then carefully touched each one up with some nice antique leaded solder making sure not to over do it. I then searched the board for other such connections and found lots that looked like they were not particularly well done. I also touched up the larger connections in the several different power regulators on both sides of the board though I don't know if that or just the connector itself was what finally brought it back to life.

Many of the CPU connections looked quite badly done to me as well but I did not try to reflow any of them in the initial attempt to get this working as I was almost certain that if I touched them I'd make things worse and not better.

I re-attached the screens and the plastic case parts necessary to hold them and provide for the cable attachment point to the car and took it back to the garage and low and behold it lit up!

Took it back upstairs and re-affixed the speedometer indicator and the rest of the plastics and it kept working! 

This is almost certainly a failure to make proper connections while using lead free solder. The connections seem to become brittle and just break. I have assembled only 1 device with lead free solder, a wonderful clock kit and every 2 years I have to take it apart and resolder one of the LED displays with good old fashioned lead free solder as the original connections fail one after another. 

It should be possible to do such things properly with lead free solder, but it is my opinion that it requires slightly different design and probably different or better flux and other such things. The solder joints themselves look so horrible with lead free solder that I think it would be impossible to validate them with visual inspection the way you can with leaded solder. A cold leaded solder joint is obvious. The ones that failed with lead free solder look identically awful to me to the ones that are still working. 

In any case, if you have the skills you should take things apart and look at them before you pay someone many thousands of dollars to replace it. We may be putting less lead into the land fills but we're putting a lot more entire assemblies into them because of it. It seems there should be a solution that properly recycles leaded boards while still letting you use what is so obviously a superior way of making things that should last for more than a few years.

I'm off to cancel the appointment at the Ford dealership for next week and to complain to someone who has no idea what the parts are made of and who won't know who to contact to complain at corporate that their instrument cluster boards are patently defective because they have not carried lead free solder construction techniques over into their new lead free board designs. I know it's pointless but I'm still going to complain to them...

Power Supplies for ESP32 Arduino Based Projects

You’ll be surprised to learn that not all power supplies are created equal ;) Or if you’ve bought more than 1 or 2 online you’ll not be surprised by that at all. There are some that are so wonderfully cheap and seductive that you purchase them only to have them fail on day 3 or do something even worse. I have fallen prey to that in the past, but not anymore. This post is nothing about any of that, but just the well documented differences between name brand (not rip off!) imported power supplies. There are still issues you need to be aware of when selecting a power supply for your Arduino or IOT based project and evidently, specifically for your ESP32 based projects. 

Here you’ll see 2 name brand, Mean Well Din rail mounted power supplies of similar ratings.

The one on the left is a MeanWell model: HDR-15-12 with a slightly blurry sticker on the top which I apologize for. It accepts line voltage input and outputs 12v at 1.25 amps. It is also very nicely tiny and uses only a very small amount of your din rail space. It’s also cheaper than any other Mean Well supply I’ve seen of similar output. It also did not work reliably with an ESP32 based IOT device. The one on the right is also a Mean Well supply model DR-15-12. It also outputs 12v at 1.25 amps but it has the “DC OK” indicator on it.

The cheaper one on the left caused my ESP32 based device to reliably startup in firmware upload mode if you waited long enough between re-powering for the internal capacitors to be discharged. If the power was disconnected for more than a minute or so the device would not properly restart when power was applied again! This is obviously not acceptable as you need to be able to plug in a device and have it actually start up. The DR-15 model on the right starts up the device every single time without any care for how long it’s been disconnected and is only slightly more expensive and only slightly wider than the cheaper one.

So whats the difference? The only thing I can find is that the “Rise Time” on the spec sheet for the 2 is significantly different. It is listed in the “output” section of the sheet and along with the “setup” time. The setup time is the amount of time between when power is applied before the output actually turns on. If you’ve used many switching power supplies you’ll have noticed that there is a delay between applying power and any voltage coming out of them. This is not generally an issue, just the housekeeping and charging up of everything that the power supply must do before it is ready to supply regulated output. It’s the “Rise Time” that seems to be important. 

In the case of the cheaper HDR-15 model the rise time is 80ms. Which doesn’t sound like a lot. The slightly more expensive one is only 30ms. I have measured these on the scope and it does indeed seem like they are the problem. In my case they are running a secondary 3.3v switching supply on the board, so your exact requirements and the point at which is causes you trouble will be different based on that.  

It does seem that a longer Rise Time can cause the ESP32 (and possibly ESP8266) based designs to startup in firmware upload mode, as if you were holding gpio0 to ground even when you are most definitely not, rather than starting up normally.  There is some allusion to this in the ESP32 documentation about making sure that if you’re running from batteries that you have a power controller that will not apply power to the ESP32 until it is above a certain level. Or as I’ve learned if it is going to within 30 or 40ms but not as long as 80ms. Keep in mind while testing that if I unplugged and replugged the supply within say 10 or 20 seconds it would come up fine. It was only with power outages that lasted long enough for it to discharge all it’s internal caps that it would fail.

While the exact timing that causes your design problems will be different based on the 3.3v regulator design if you have this specific issue then I would first recommend testing with a power supply with a faster rise time. In my case the chip just did nothing when powering up. Re-applying the programming cable revealed the “waiting for download” message from the ESP32 making it clear that it thought I was grounding the gpio0 pin even though I wasn’t. With a faster rise time it starts up reliably every single time. 

1-wire Temperature Sensors and Lightning (and other sources of interference)

I use a lot of 1-wire temp sensors around the house for various purposes. One of which was a sort of hack to know when the hot water heater was in use. I placed one on the inlet side and one on the output side of the tank and when the 2 temperatures diverge by enough I know that the hot water is actually flowing. Within a few minutes of someone turning the tap off the temperature has equalized enough that it can tell the hot water is no longer in use. 

The main reason I set this up initially is that I wanted to have my bathroom lights go off after there was no motion in the bathroom for a while, but I didn’t want them to go off while I was in the shower nor did I want to set the timeout to be so long that they would stay on unnecessarily at other times. So now if the hot water is in use the logic of the master bathroom will not turn off the lights on you, but will keep checking back to see if the water is now off or if there has been more motion.

My day job is home automation software so all the logic for that is running in XTension.

Here is the graph of the output from last night:

The blue and red lines are the inlet and outlet temps of the water heater. Physically they are just a standard 1-wire temp sensor taped to the pipes a few inches up from the water heater. The yellow highlight is when the hot water is considered to be in use. The light green line is another 1-wire sensor just hanging in the attic showing the ambient temp in the attic. You can see 2 spikes to 185 degrees between 5:30 and 6pm last evening and one identical spike on the inlet sensor. 

Those were all caused by very sharp close by lightning hits. These particular sensors are connected to a Barix Barionet device which is a fantastic remote host for 1-wire sensors as well as digital inputs and relay outputs. Very handy for home automation and fully supported by my plugin for it in XTension. I have another device that is just basically an arduino that has a similar sensor that is glued to the side of a power supply so that I can make sure it is not overheating as it is in an electrical box with no ventilation. When those lights are on that sensor reads the same 185f or 85c repeatedly until the load decreases. These 1-wire sensors are just prone to interference and when that happens they return a reading of 85c. 

The reading of 185c is within the range of temperatures the device can actually read so it isn’t good to just filter that out in the driver or plugin or hardware. I am considering for a future firmware build of the 1-wire to wifi kit that if I get an 85c reading I will attempt another reading a few seconds later and only report it if the reading is the same twice. For the moment though the solution to filter them is easy enough inside the XTension ON script for the Unit. In the case of the above graph I am not going to filter it as it is interesting and happens rarely and does not throw off any other calculations or anything. For the sensor on the power supply it happens so frequently that it is frustrating to look at the graph. In that case I use the Change Future Value verb to change the value back to the previous reading if it is exactly equal to 85c. That does potentially ignore some valid measurements, but it does let anything through that is 84.9 or below or 85.1 and above and so I’m not terribly worried about that. The script is very simple and just looks like this:

if (future value) is equal to 185 then

change future value to (value of (thisUnit))

write log "ignored bogus 185 value for porch temp sensors"

end if

 

futureValue tells you the value that the unit will be set to when the script is finished running. The (value of (thisUnit)) portion returns the current value of the unit. This is sometimes initially confusing to people new to working with XTension but it allows you to do things like this or anything where you need to compare the current value with the new value and potentially override or change it. It also lets you do very specific smoothing of readings or throwing out ones that are out of range such as this. The call to ThisUnit just returns the name of the unit that you would normally have to pass to the value of verb so that if you renamed the unit you wouldn’t have to remember to edit the script and if you had many of the devices you want to do this to you could either cut and paste the code, or create a single script that you could call for all of them.

This 185/85 reading is actually a known issue with some 1-wire temp sensors. I don’t see a difference between powered or parasitically powered networks. The sensors in the above graph are parasitically powered but the one that is on the power supply is powered. The parasitically powered ones are on an absolutely horrible network of wires flying through the attic and to say that it is not a single bus or a standard star configuration would be a dramatic understatement. There is no shielding though they are on a single twisted pair of bell wire. Anyplace I wanted to add more sensors I just spliced into it and extended it. That attic network is my test of the absolute worst case I can imagine for testing 1-wire reading devices. The barix does exceptionally well under almost all circumstances except for these lighting storms.

That network now has 10 sensors on it all over the attic and it does sometimes drop a reading for one of them entirely. It also reads the intake and output temperatures from 3 different AC units so that I can verify the temp differential and know if the units are leaking gas and will need a recharge (or in one of the older ones a replacement is coming up I’m afraid) before they actually stop working and we have to make an emergency call to the HVAC folks. That reading also lets me run the fan after an AC or Heat cycle the best length of time to get all the cold or hot out of the unit. I turn the fan to On when a cycle completes and then start watching the temp sensors. As soon as the 2 temperatures are within a few degrees of each other I turn the fan back to Auto. 

I am also going to update the hardware in the next revision of the wifi temp sensor to use the better interface that is outlined in the 1-wire best practices documentation to see if that will help with bad values or interference.  In any case, overall 1-wire sensors work great, are inexpensive and there are a lot of options for getting them into your home automation system. 

Python Motion Detection on a Raspberry Pi Zero

I have been working non-stop the last few months to make my main project software compatible with Apple’s latest OS version. I finally realized how badly this was depressing me and how much I didn’t want to be doing this and how much it was making me hate Apple. I was spending literally man months of time to rebuild from scratch things that weren’t broken before Catalina and it was making me miserable. My rants about the absolute stupidity of everything that Apple chose to do to this alleged OS update will have to wait as I can’t afford the dental bills to fix the damage that gritting my teeth that hard will cause.

So in the interest of sanity and preserving the small quantity of it with which I have been granted  I took a short vacation from everything MacOS. I have been fighting with the horrible firmware of even the best available IP security cams for some time as I struggle to write software that can connect to them and insert their video streams into the web and other interfaces of XTension. I believed that there should be software for the Raspberry Pi that could leverage that camera for a much more reasonable price that would be terrific. Sadly even the best software out there for it just wasn’t doing the things that I wanted or at least with the quality of image and recording that I wanted.

The idea of controlling everything about a security camera is compelling! Not having to setup a firewall to keep it from contacting it’s own “cloud” and making sure it isn’t setting up a UPnP passthrough even when you told it not to have revealed to me several times that even commercial products don’t really do what you think they are. I wanted one based on software that I have written myself and so knew exactly what it’s doing.

It should also run well on a Pi Zero W so that you can make something small, though if it runs better on the latest multi-core Pi4’s thats OK too. Those little machines are worth every penny and very very nice!

The built in “motion” system is really interesting, but it’s remarkably difficult to get it to run at a high framerate and a larger frame size. Who wants a non-HD security cam now days anyway? The separate OS versions available may be spectacular but I want a software package for the stock Pi. After fighting with trying to get openCV to compile on a Pi Zero for the last week I wanted something that didn’t need that either. (Which I suspect is why they went with a totally separate image for their security cam OS as getting that stuff to install properly is almost hilarious)

The python camera support for the Pi is absolutely awesome. I can stream 1080p video and I can record it to disk and then post process it to standard MP4 files in almost real time. Those were not insignificant accomplishments ;) But the real sticking point was going to be the video motion detection. Using the pycamera code as a starting point and the hardware H264 encoders ability to output it’s changes from one frame to another makes it absolutely possible to do this at least as well as any commercial product. I’ll eventually open source this project but for the moment I’m working on it just myself. Here’s the 720p stream, part of the web interface and also a stream of just the motion sensing. This works as high as 1080p but at lower framerates though I’m sure I can optimize the calculations a bit more for that.

The image on the right is a live stream of the motion content and I’m just really enjoying playing with this at the moment.

I am currently building the web interface to the ability to mask out portions of the video for motion detection. The math for that is working great and adds very little if any overhead. Lots of work yet to do of course before something that is download/install ready but it will come before long. This is making me happy! Apple is making me mad.

Lest anyone think I’ll abandon developing software for Apple products over how much they drive me crazy sometimes really needs to spend some time using any of the other truly commercial desktop OS’s available out there. Don’t worry. XTension is undergoing a lot of changes because of this, but it’s not going away and unless I sell enough of it to hire a support team in another time zone I wont even consider doing a windows version. Sorry but if Apple frustrates me then Microsoft would kill me outright.

RGB Fairy Lights with only 2 wires (part the first)

I am a huge fan of all things blinky and light up. I’m less interested in things I cannot control remotely by writing a plugin for XTension or by building a new controller for it. While looking for RGB “fairy” lights on Amazon (“fairy” lights here being defined as some strands of what looks like magnet wire with surface mount LED’s soldered between then and then sealed into a little blob of epoxy) I came across these RGB ones and was intrigued since they appeared to have only 2 wires. There was no control wire or separate PWM lines for the RGB channels. (Amazon Link) These are by “Yihong” but there may be others that are the same.

They came with an RF remote which is nicer than the IR ones that some similar things come with. Since you don’t have to point it at the controller box. Interestingly enough the remote also has an IR led and so could be used by either type of system. I have not tested to see if it is running on the standard 433mhz band or does something special. Given that it appears the same as the thousand other such remotes I’ve gotten over the years for other LED displays or just for remote arduino projects I would guess that it does.
In spite of the picture on the box showing different colors on the strand at the same time, you cannot control the LED’s individually, only as a whole strand.

This is the very tiny control box. The strand is powered from a USB plug and this box does the work of receiving the remote signals and adding whatever data is necessary to the power lines going down the strand. I have not broken the case open yet to see whats in there.

While there is tons of information out there about the “neopixel” type LED’s and the newer “dot star” that Adafruit sells and are amazingly wonderful there is very little information about anything similar to this. If anybody knows the type of chip or LED that they are made from I’d be interested to hear about it.

This you tube episode from Julian Ilett shows him reverse engineering a similar protocol. The system used by my lights looks to be very different however. So the first step to seeing what it’s doing is to get it on the scope. Since the wires are just enameled I scraped some of the insulation off them at the end of the strip. It occurs to me that I might have gotten a cleaner signal by tapping it at the front of the string instead. I’ll try that in step 2 and see if it makes it easier for the scope to trigger on it.

These LED’s are different than Julians as they don’t pulse all the way to off. The data pulses drop the voltage to the lights down to 1.5 volts, but not all the way to off. This potentially makes sense as it would let the little processor in the LED chip continue to operate as it’s being updated.

 

Here you can see the amazingly noisy USB power supply, either from the hub I was using to power this or from the controller device itself I don’t know, but it didn’t seem to bother them. You can also see the tiny blips of the control pulses coming in every 100ms or so. 

 
Zoomed in on those a bit you can start to see the data.

There are 2 lengths of pulses. Since the on portion of the pulse is always the same length the entire duration of the data packet changes based on how many 1’s there are in it.

The short pulses, which I will call a binary 0, are approximately 5us with the bus at 1.5v and then 10us back at 5v. The actual measurements I got from the scope was 6us and 12us but they varied enough that I assume they meant to use 5 and 10us. That may have to be adjusted later if the device is very picky about the timing. The 1 binary pulse is 10us at 1.5v and 10us at 5v.

Every data packet starts with 8 “0” or short timing bits. These do not appear to change no matter what the string is displaying so I will assume they are a preamble that helps the receiver lock into the data rate. In the screen shot above you can also see the end pulse which is twice the length of the “long” or binary 1 pulse, 20us at 1.5v. 

There are 32 bits in total, one byte for the preamble and 3 more bytes which obviously encode the RGB values that the LED should display. I did several captures from the various solid colors the three different intensity levels, leaving out the preamble byte: I’ve added the decimal value of the binary next to them:

RED:   FULL:  0010 0110(38)  0000 0000     0000 0000

       MED:   0100 1100(76)  0000 0000     0000 0000

       LOW:   1001 1000(152) 0000 0000     0000 0000

GREEN: FULL:  0000 0000      0010 0110(38)  0000 0000

       MED:   0000 0000      0100 1100(76)  0000 0000

       LOW:   0000 0000      1001 1000(152) 0000 0000

BLUE:  FULL:  0000 0000      0000 0000      0010 0110(38)

       MED:   0000 0000      0000 0000      0100 1100(76)

       LOW:   0000 0000      0000 0000      1001 1000(152)

It’s quite obvious already that the 3 bytes are RGB values. You can also see that even when on at full they don’t send just an FF. Perhaps the LED’s would overheat at full on. There is some visible PWM even when a solid color is set to full on. Or there is a limit in the controller device itself that is not capable of 255 individual levels. They are rotating that bit pattern across for dimming the LED. At this point I’m thinking that the value is actually how long to wait into the PWM cycle before turning on that LED. That would make sense, it would let a larger number actually be a dimmer display. Just for fun sampled some of the other colors as well:

ORANGE: FULL:  0011 0001(49)  0101 0000(80)  0000 0000

        MED:   0110 0010(98)  1010 0000(160) 0000 0000

        LOW:   1100 0100(196) 0100 0100(68)  0000 0000

YELLOW: FULL:  0010 0110(38)  0100 1100(76)  0000 0000

        MED:   0100 1100(76)  1001 1000(152) 0000 0000

        LOW:   1001 1000(152) 0011 0000(48)  0000 0000

CYAN:   FULL:  0000 0000      1101 0010(210) 1101 0010(210)

        MED:   0000 0000      1010 0100(164) 1010 0100(164)

        LOW:   0000 0000      0100 1000(72)  0100 1000(72)

and now I’m wishing I hadn’t sampled those extra colors as they don’t seem to follow the same start point of PWM. Or perhaps they are just compensating for irregularities in how much the LED’s output at any given level. They won’t be the same certainly, generally red needs a much higher value than green and blue to look the same. 

In any case there is enough information here to get started trying to build an Arduino controller for the thing. I’m not entirely sure how to provide the 1.5v pulse level in hardware. Will it be enough to just run the stream power supply through a resistor? Will have to calculate the power draw of the LED’s and do the math to drop the voltage to that level. Or are the LED’s in sync with this pulse and so will be off when the data starts and therefore I’ll need a different value since the load would be less. Or will it be necessary to provide a 1.5v regulator and use a transistor or mosfet to bypass it when the strip should be given the full voltage. Some experimentation will be necessary. When I’ve had a chance to play with that I’ll post again.

WiFi Signal Strength as presence indicator, part 2

As a followup to yesterdays post about trying to filter the wifi RSSI of an ESP8266 board in order to sense movement I added a standard variance calculation on the last 2 minutes or samples or so. I added a threshold of 1 just for testing and have let it sit here for a while. The graph of that output looks like this:

When the variance goes above 1, or whatever preset threshold, I turn on the presence register which shows in green. You can see that it definitely does work to know when I’m coming and going, but at this sample rate, buffer size and threshold it also comes on randomly with non-people moving through the room wifi interference.

In order to register anything reasonable at all you have to have the device very close to the person you want to know if they are there. This may be of use for this particular application as I plan to have the device sitting on my desk to provide some display of info for me while sitting here. It might even be able to blank and wake up it’s own display based on that. That would be useful and fun. Unfortunately if I sit very still it eventually thinks I’ve gone away. It’s decided that while I’ve been typing this paragraph actually. Lost of tweaking of settings would be necessary to make it at all useful.

I’m not sure that it will ever be useful as a generic motion sensor. You just have to be too close to the device to get it to see enough variance in the signal. Still an interesting experiment and I will keep fiddling with it to see if something useful can be done with it. It’s also possible that with the newer ESP32’s or other devices you could get better resolution or even access to things like the path being used to send the info or other similar data that would let you do a much better job. But for something that wakes up when you walk up to it and then goes to sleep shortly after nobody is standing next to it this could potentially be used.

Calculating a variance is a standard statistical technique so that you can gauge if you’ve got enough data points to realistically represent what you’re looking for among other thing. The formula is:

S² = ∑(x_i - x̄)² / (n-1)

Which translates to the Variance squared being equal to the sum of each value in the dataset minus the average of the dataset squared and then divided by the number of points in the dataset minus one. So the first thing you need is an average of the dataset.  It’s slightly complicated by the fact that my buffer array of read RSSI values may contain 0’s or values that are not a valid RSSI. As I place the values into the buffer I am making them all positive as I am not worrying about the actual db value, only the variance of the data. So the first thing you need to do is to call the scanRSSI function every second. To do this you need to compare millis() with the calculated next time you want to do something inside the loop. You can’t just compare simple unsigned longs as are returned from millis() as they will overflow back to 0 at some point which will break everything. So you can do the clever trick of casting it to a long and then subtracting it later which will yield the proper result of timing even during an overflow condition.

#define MAX_RSSI_ARRAY 128
#define RSSI_PRESENCE_THRESHOLD 1
bool currentPresence = false;
byte rssiArray[MAX_RSSI_ARRAY];
byte rssiIndex = 0; // the index into the array where we are now
unsigned long nextPresenceScan = ((long)(millis() + 5000)); // 5 second start

the code above sets up the buffer array as bytes, since the numbers will never be more than 255 there is no reason to waste memory by using 2 byte ints or anything else. We also need an index into the array so we know where to save the next value and the nextPresenceScan value which is where the trick comes in. Casting the value of millis() + 5000 as a long integer and then putting it into an unsigned long variable we get around the overflow problem as far as the comparisons later are concerned. Very clever and described in great detail on the Arduino website if you’re interested.

Next we need some code to run inside the loop() to check the time and call us when it’s time to run. Something like:

void loop( void){
 if (((long)(millis() - nextPresenceScan)) >= 0){
  nextPresenceScan = ((long)(millis() + 1000));
  scanRSSI();
 }
}

Same trick as before, sets up the nextPresenceScan and then calls the function to do the work.

void scanRSSI(){


 float currentRSSI = WiFi.RSSI();
 
 if (currentRSSI > 0){
  // sometimes the WiFi.RSSI() call can return a number greater than 0 to indicate
  // an error or that it simply couldn't get the info at this time. If that happens
  // just ignore it.
  return;
 }

 // add the new value into our buffer at the current rssi buffer index
 // multiply by -1 to turn it into a positive number
 rssiArray[ rssiIndex++] = currentRSSI * -1;
 
 // if the index is now beyond the size of our buffer reset it back to
 // the beginning of the buffer for the next read
 if (rssiIndex >= MAX_RSSI_ARRAY){ rssiIndex = 0; }
 
 // first value we need for the variance calculation is the average
 float accumulator = 0;
 
 // since the buffer may not be filled with valid data yet 
 // we need to count how many non-zero values there are for the
 // average calculation rather than just divide it by the 
 // MAX_RSSI_ARRAY
 int averageCounter = 0;
 
 float average = 0;
 
 for (int i=0; i<MAX_RSSI_ARRAY; i++){
  
  // ignore values of 0
  if (rssiArray[ i] > 0){
   if (rssiArray[ i] < mn){ mn = rssiArray[i];}
   averageCounter++;
   accumulator += rssiArray[ i];
  }
  
 }
 
 // if we don't have at least 10 valid values in the array then just drop out
 // there is no point in calculating the rest unless we have enough values to 
 // look at
 
 if (averageCounter < 10){ return; }
 
 
 average = accumulator / averageCounter;
 

 
 // now loop again to calculate the SUM portion of the variance
 // reuse the accumulator 
 accumulator = 0;
 
 // loop through the buffer array again, this time subtracting the average and then
 // squaring the result adding it to the accumulator
 
 for (int i=0; i<MAX_RSSI_ARRAY; i++){
  if (rssiArray[ i] > 0){
   // dont want to waste the memory to include the floating point or C math
   // libraries, so just put the calculated value into a local variable and
   // multiply that by itself
   float work = (rssiArray[i] - average);
   accumulator += work * work;
  }
 }



 // here we divide by n-1 
 float variance = (accumulator / (averageCounter - 1));
 
 // the variance may be less than 0? if so take absolute value
 if (variance < 0.0){ variance *= -1; }
 
 // now do something clever with the variance data.

}

That is actually pretty horrible code. I also do realize that I’m never taking the square root of the variance as calculated. That doesn’t seem necessary at this point and the poor arduino is already working hard with all those floats. If I make the value even smaller it will be harder to watch whats happening. It’s also not memory compact as it needs a full buffer. I’m sure some other kind of rolling average could be used. To get anything useful you’ll have to adjust the threshold and the scanning frequency and probably a lot of other stuff too. An interesting starting point to something potentially useful.

The device in question here is sending it’s values back to XTension for graphing and potential use. The graph above was generated by that program. XTension: Home Automation for the Mac, is my day job.

Using wifi signal strength as a motion or presence indication

It seems like it should be possible to use wifi RSSI as a presence or motion detector. I have seen some very fancy devices that made low level use of the reflection information to target motion or people in a room, but the graph below is from an ESP8266 just reporting it’s RSSI every 5 seconds.

The device is sitting on my desk just a couple of feet away, so the range might not be very good, but still there might be instances where this is useful. You can quite plainly see in the above graph that I was in the office on and off until just after 7, and then it is fairly smooth while I was attending the local Philharmonic concert to hear my wife sing Mahler’s 2nd, and then picks up in variability as soon as I started going in and out of here this morning.

A lot of hysteresis would be necessary since there are many things that can affect the signal strength. Since it would take some time to know that you were there for sure it might not be useful to turn on lights or make adjustments when you arrive, but it could definitely be used to figure out when you’ve gone. Adding a Wifi calculated presence detection to such IOT devices would be very helpful. It would not work very well for battery operated devices however since you would need to wake up the wifi radios far more often than for the regular functions of the device.

Part two of this thought has code examples and is here.

Fancy Crown Molding for New Bookshelves...

The living room in this house has been something we planned to remodel completely since before the day we moved in. All the standard white enamel cabinets and shelves that come with these houses were going to get thrown out and replaced with wonderful floor to ceiling book shelves and a library ladder and all sorts of wonderful things.

I haven’t written about this yet because even so many years later it’s still not quite finished. We’re getting closer though. I just wanted to share this piece of test molding I made, oh, a little more than a year ago now, that serves as the template for the rest of it as I slowly completed the work. More info will follow as it’s even closer to being done now.

I’ve always been a bit intimidated by fancy crown molding. I didn’t know how to design it or make it work or make it go around the things that were there. This project has forced me to come to terms with some of that.

Here is my test piece for the multi-part crown molding that will go around the top of the new bookshelves. I’m so happy with the way it turned out I can’t even tell you. It even looks great in the finished product that I’ll upload another article about shortly :)

It’s all commercially available molding from the local Home Depot or Lowes. I tend to buy molding from Lowes because they sell it in 6 and 8 foot sections where at Home Depot I need to know how much I need ahead of time and cut it from a 10 foot segment that will by no means in the known universe fit into my car. I could probably save a few bucks by figuring that all out. But it makes too much work ahead of time. I can just calculate how many 8 foot sections I need and add one more to fill in the gaps :)

Arris model DG3270A SNMP and backup decoding

I have a remote install of XTension setup at my mother in laws condo. The internet and tv company there started with DSL modems in the building, but it’s an older building and the wires in the walls never resulted in a very good connection. They finally got around to putting in a cable head end in the basement and switched everyone to cable modems. They did this without calling me first or otherwise letting me know that they were going to show up and drop in a new modem and make everything I have installed there home automation wise to drop off the network and kill all my incoming passthroughs and such. I got there last night to have a look at the new modem and reset everything up and I discovered some interesting things.

The modem is an Arris DG3270A and actually works pretty nicely. The wifi is significantly better than the old modem actually supporting a 5ghz wifi network as well as 2.4ghz AND it will allow you to set both the 2.4ghz and 5ghz to the same network name. This is the preferrable way to do this as your devices will migrate between the 2 as needed and you won’t have to be using the faster 5ghz one and then walk upstairs or into bedroom and have it drop out and have to manually switch to the other. Your device will just always connect to the strongest one.

They have the interface to it setup so that you can do all the configuration of almost everything you’d want to yourself. The default password on the box was not changed which in this case was just admin/password. There are many different models and many providers lock you out of these settings. With the old modem I had to actually call the company to get the password to it and they told me that they would give it to me but would charge me for the service call if I screwed it up ;) I didn’t screw it up ;)

I was able to change the DHCP range to carve out the same block of static IP’s that I give to her XTension machine and the Vera there as well as setup the couple of pass throughs so that I can ssh into the machine and the webcam if necessary. It even has dynamic dns support so I was able to add the modem to my account there and have it update that info too. I also have the client running on the Mac in her closet but it never hurts to have a backup to such things, without that updated I can’t talk back to her machine very easily.

With the previous modem I had written a little script that polled the status page of the modem periodically and dumped the values of the noise and signal level on the DSL line into XTdb for graphing. It was interesting to watch how bad the wires were in the building and how they got slowly worse over time. It really was time to either pull more wires or switch to the cable system as they had been promising to do for so long. So I went looking for the same status into for the cable mode to see if there was anything interesting there to parse out and track just for fun.

Even though the modem is on the standard router address of 192.168.0.1 there is another page that gives all the status information on address 192.168.100.1 which seems strange to me, why run it on a separate IP like that and not just on a different link on the main router address? I suppose it’s possible that is bound to that network and ip address regardless of what NAT network you have the rest of your network set to? So perhaps that will still work even if your network is set to use the 10.0.0.xxx range? I don’t know but it seems needlessly complicated.

There is a lot of fun info there that I’ll probably parse out in the future though I hope the cable modem will be operating well enough that I won’t really care. At the very end of the tab bar across the top though is a tab called “Advanced” and so I had to have a look at that only to be provided with a password field. Obviously setup with a daily password system that takes a known salt and does some kind of hash or encryption against the date to come up with a daily changing password. This is really a good idea as even if the user is looking over your shoulder and sees the password, or you have to send a password to one in an email or something after the day you did it it’s no longer valid and they can’t publish it and let the entire internet into the machine to screw everything up. Indeed, searching on the internet led to several sites with information about this and even daily password calculators like this one arris password of the day generator that one didn’t work as it seems each company changes the salt value for it. I never did get into that screen but I did learn some interesting things in trying to find out what it was. On that site above he talks about decoding the router.data file that you can download from the regular interface as a backup which I had already done. On his page he talks about having to untar it and then decrypt it. None of that was necessary for my file from this machine. It turned out that each line was just base64 encoded. Each line has to be decoded separately you can’t do the entire file. There was no other encryption.

The file that resulted had some initial setup stuff that starts with MAC address like data and long hex strings. None of which were the right length to have been the salt for the daily password unfortunately. And then a user section with a long list of SNMP looking addresses and all the configuration data that I had entered into the machine. Again nothing that looked like the salt but if the device supports SNMP then it may be possible to scan more values in there and find that as well as lots of other information. Of course there are no comments or anything else in the file that would help you figure it out I present mine here with any real data or password information changed. If you would benefit from seeing the actual data in these fields I’ll send you the real file, I doubt anyone really cares about getting on the wifi at my mother in laws house but I’m still not posting the passwords to the internet.


<POSTDATAVER>
version 4
</POSTDATAVER>
<CLIENTDB>
version 1
8c:ec:4b:e8:be:0f 0 4c2d31303639 7e6e6f636f6d6d656e747e 00000000
00:0e:8f:92:e4:f1 0 756e6b6e6f776e 7e6e6f636f6d6d656e747e 00000000
48:bf:6b:71:89:b3 0 4d696e64794e6168736950686f6e65 7e6e6f636f6d6d656e747e 00000000
40:4d:7f:1c:2a:a6 0 4d696e64797341706c655761746368 7e6e6f636f6d6d656e747e 00000000
dc:a9:04:8c:f2:c8 0 4a616d65732d4d6163626f6f6b 7e6e6f636f6d6d656e747e 00000000
48:d7:05:d9:14:bf 0 4d696e64794e616841697232303134 7e6e6f636f6d6d656e747e 00000000
</CLIENTDB>
<USR>
1.3.6.1.4.1.4115.1.3.7.1.1.10.0=admin;4;&_=@<20180419 233953>TR69 (likely the admin password user name, though I do not see the actual
admin password here. Likely it will only be present if you set it to something
other than the default which was just "password")
1.3.6.1.4.1.4115.1.3.7.1.1.6.0=10800;66;&_=@<20180606 120200>TR69
1.3.6.1.4.1.4115.1.20.1.1.3.22.1.3.10001=1;2;&_=@<20180727 085921>USER
1.3.6.1.4.1.4115.1.20.1.1.3.22.1.3.10101=1;2;&_=@<20180727 085923>USER
1.3.6.1.4.1.4115.1.20.1.1.2.2.1.11.200=%24C5A81164;4;&_=@<20180728 182904>USER (5 hex values, altered from what they really were)
1.3.6.1.4.1.4115.1.20.1.1.2.2.1.19.200=1;2;&_=@<20180728 182904>USER
1.3.6.1.4.1.4115.1.20.1.1.3.26.1.2.10001=wifipassword;4;&_=@<20180728 183201>USER (one of these is the password for the 2.4 ghz and one for the 5ghz
1.3.6.1.4.1.4115.1.20.1.1.3.26.1.2.10101=wifipassword;4;&_=@<20180728 183446>USER since I have them set to the same I don't know which is which)
1.3.6.1.4.1.4115.1.20.1.1.4.21.0=1;2;&_=@<20180728 183615>USER
1.3.6.1.4.1.4115.1.20.1.1.3.22.1.2.10001=wifi SSID;4;&_=@<20180728 184019>USER (the SSID of the 2.4 and 5ghz wifi networks, again I have them set
1.3.6.1.4.1.4115.1.20.1.1.3.22.1.2.10101=wifi SSID;4;&_=@<20180728 184104>USER to the same value so I don't know which is which)
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.2.1=name of a NAT map;4;&_=@<20180728 184933>USER (this is a NAT port mapping
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.3.1=start port;66;&_=@<20180728 184934>USER there are 4 port setting associated with each mapping, the external
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.4.1=end port;66;&_=@<20180728 184934>USER start and end port and the internal start and end port. I don't know
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.5.1=1;2;&_=@<20180728 184935>USER which of these 4 values here is which as I am passing through only one port
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.7.1=%24C1A81042;4;&_=@<20180728 184936>USER so all 4 values were the same.)
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.9.1=start port;66;&_=@<20180728 184937>USER
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.10.1=end port;66;&_=@<20180728 184938>USER
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.11.1=1;2;&_=@<20180728 184939>USER
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.2.2=another port map;4;&_=@<20180728 192237>USER (another port mapping here)
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.3.2=start port;66;&_=@<20180728 192237>USER
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.4.2=end port;66;&_=@<20180728 192238>USER
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.5.2=1;2;&_=@<20180728 192238>USER
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.7.2=%24C1A84012;4;&_=@<20180728 192239>USER
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.9.2=start port;66;&_=@<20180728 192240>USER
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.10.2=end port;66;&_=@<20180728 192240>USER
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.11.2=1;2;&_=@<20180728 192240>USER
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.2.3=another port map;4;&_=@<20180728 192319>USER (yet another NAT mapping)
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.3.3=ext port start;66;&_=@<20180728 192319>USER (in this case I can see this is the external port)
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.4.3=ext port end;66;&_=@<20180728 192320>USER
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.5.3=1;2;&_=@<20180728 192320>USER
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.7.3=%24C5A80BC4;4;&_=@<20180728 192321>USER
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.9.3=int port start;66;&_=@<20180728 192322>USER
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.10.3=int port end;66;&_=@<20180728 192322>USER
1.3.6.1.4.1.4115.1.20.1.1.4.12.1.11.3=1;2;&_=@<20180728 192323>USER
1.3.6.1.4.1.4115.1.20.1.1.4.18.1.0=1;2;&_=@<20180728 202234>USER
1.3.6.1.4.1.4115.1.20.1.1.4.18.2.0=1;2;&_=@<20180728 202235>USER
1.3.6.1.4.1.4115.1.20.1.1.4.18.3.0=dyndns username;4;&_=@<20180728 202235>USER
1.3.6.1.4.1.4115.1.20.1.1.4.18.4.0=dyndns password;4;&_=@<20180728 202236>USER
1.3.6.1.4.1.4115.1.20.1.1.4.18.5.0=dyndns hostname;4;&_=@<20180728 202236>USER
</USR>

Note that I’ve munged the data above including the hex values just so I don’t expose something important. Not that I think there is anything anyone could do with this info who wasn’t already standing in her living room but still. Since I don’t know what it is I altered it but kept the layout and length and format the same. My comments in parentheses are also obviously not part of the actual file either.

So it may be possible to setup these devices via SNMP including all the port mappings you wish as well as any other user data. Thats potentially useful if you wish to dynamically turn on and off port mappings or something like that. I’m sure that much more data is available to a real SNMP dump of the thing assuming it doesn’t require other passwords or encryption to set the data. In future visits I will play with the SNMP properties of the modem but I didn’t have a chance to do that this trip. I might be able to get the connection quality and error rate that way which would be interesting to graph.

If someone can see the salt for the advanced daily password in that I’d sure love to know what it is. It may be discoverable via a different SNMP address and I’ll experiment with that in the future.

Restarting an AirPort via AppleScript

Apples airports are great devices, but like so many “appliances” they need a restart once in a while. Apple has the newer models pretty well locked down. There is no web interface, nothing you can hit with a curl command and no SNMP or anything else to talk to to tell it to restart itself.

After I lost my entire wifi network the other day and I had to restart it manually I decided it was time to tackle the AirPort Utility via User Interface Scripting. It turns out you can do this, sort of... There are either bugs in the AirPort utility and it’s implementation of user interface scripting, even though you’re not supposed to have to implement it, it’s just there because of the OS, or there are some limitations to it that aren’t clearly documented.

The applescript to restart an airport is pretty simple actually, this handler will do it, sometimes:

on restartAirportByName(theName)

tell application "System Events"

tell process "AirPort Utility"

click (first image of group 1 of scroll area 1 of window 1 where its name starts with theName)

click menu item "Restart…" of menu "Base Station" of menu bar 1

end tell

end tell

end restartAirportByName

Note that handler doesn’t actually launch the AirPort Utility first, You should do something like:

launch application “AirPort Utility”
delay( 5)

Before calling that.

There are some gotchas right away, the ellipsis in the Restart menu item name is a UTF8 ellipsis character and not just three periods. If you use three periods it won’t be able to find the menu to click.

It won’t work if the application is hidden or the main window minimized. It does not have to be frontmost, but it cannot be hidden in any way.

The name of the “process” is case sensitive. It must have the same case as the actual application in order to match, even though much else of AppleScript is not case sensitive this is. It must be “AirPort Utility” or it won’t work.

Next the script or the host of the script must have assistive access turned on in order to use the UI scripting at all. You have to do that in the System Preferences, Privacy & Security pane under Assistive Access. This will also break if you are saving the script as an application and make any edits to it. You must go in and uncheck and recheck the checkbox next to the application you created or it will know that you changed it and won’t let it access the UI. Again, you don’t get the helpful error message that you’d get if you hadn’t added it to the list at all telling you that assistive access is not enabled for this program, it just fails to be able to do anything.

Beyond that there is something weird about how AirPort Utility processes the event, or there is a limitation for processes not specifically started by a user which may be the case. I would expect those to generate an error specifically telling you that you don’t have privs. Sometimes, or all the time depending on the OS version you get an error about a bad index when trying to get a reference to window 1. I’ve tried accessing the window by name and several other hacky work arounds but it doesn’t seem to matter. If you run the script from a program that has assistive access turned on, but wasn’t launched by the user you’ll definitely get this error at least once, and then it might start working for the second time. Or it might not.

In my testing the best solution is to embed that bit of script into an applescript saved as an application. Then you can give it privs to use assistive access and just launch it, or use some other system to launch the app. That seems to work. At least on OSX 10.13. There still seem to be problems doing these things on 10.12.

I’ve wrapped that and a whole lot more scanning and checking code including the ability to monitor your internet connection and get your external IP address into a larger script that talks to my home automation software of choice XTension (well it should be my choice as I’m one of the developers) which will definitely be useful for other XTension users as long as they are running an OS version that includes a less buggy AirPort Utility. That code is free and open and if you are interested in how to get the rest of that info out of it it’s also fully documented with comments in the code here.