Tuesday, May 7, 2019

WiFi Signal Strength as presence indicator, part 2

As a followup to yesterdays post about trying to filter the wifi RSSI of an ESP8266 board in order to sense movement I added a standard variance calculation on the last 2 minutes or samples or so. I added a threshold of 1 just for testing and have let it sit here for a while. The graph of that output looks like this:

When the variance goes above 1, or whatever preset threshold, I turn on the presence register which shows in green. You can see that it definitely does work to know when I’m coming and going, but at this sample rate, buffer size and threshold it also comes on randomly with non-people moving through the room wifi interference.

In order to register anything reasonable at all you have to have the device very close to the person you want to know if they are there. This may be of use for this particular application as I plan to have the device sitting on my desk to provide some display of info for me while sitting here. It might even be able to blank and wake up it’s own display based on that. That would be useful and fun. Unfortunately if I sit very still it eventually thinks I’ve gone away. It’s decided that while I’ve been typing this paragraph actually. Lost of tweaking of settings would be necessary to make it at all useful.

I’m not sure that it will ever be useful as a generic motion sensor. You just have to be too close to the device to get it to see enough variance in the signal. Still an interesting experiment and I will keep fiddling with it to see if something useful can be done with it. It’s also possible that with the newer ESP32’s or other devices you could get better resolution or even access to things like the path being used to send the info or other similar data that would let you do a much better job. But for something that wakes up when you walk up to it and then goes to sleep shortly after nobody is standing next to it this could potentially be used.

Calculating a variance is a standard statistical technique so that you can gauge if you’ve got enough data points to realistically represent what you’re looking for among other thing. The formula is:

S2 = ∑(xi - x̄)2 / (n-1)

Which translates to the Variance squared being equal to the sum of each value in the dataset minus the average of the dataset squared and then divided by the number of points in the dataset minus one. So the first thing you need is an average of the dataset.  It’s slightly complicated by the fact that my buffer array of read RSSI values may contain 0’s or values that are not a valid RSSI. As I place the values into the buffer I am making them all positive as I am not worrying about the actual db value, only the variance of the data. So the first thing you need to do is to call the scanRSSI function every second. To do this you need to compare millis() with the calculated next time you want to do something inside the loop. You can’t just compare simple unsigned longs as are returned from millis() as they will overflow back to 0 at some point which will break everything. So you can do the clever trick of casting it to a long and then subtracting it later which will yield the proper result of timing even during an overflow condition.

#define MAX_RSSI_ARRAY 128
bool currentPresence = false;
byte rssiArray[MAX_RSSI_ARRAY];
byte rssiIndex = 0; // the index into the array where we are now
unsigned long nextPresenceScan = ((long)(millis() + 5000)); // 5 second start

the code above sets up the buffer array as bytes, since the numbers will never be more than 255 there is no reason to waste memory by using 2 byte ints or anything else. We also need an index into the array so we know where to save the next value and the nextPresenceScan value which is where the trick comes in. Casting the value of millis() + 5000 as a long integer and then putting it into an unsigned long variable we get around the overflow problem as far as the comparisons later are concerned. Very clever and described in great detail on the Arduino website if you’re interested.

Next we need some code to run inside the loop() to check the time and call us when it’s time to run. Something like:

void loop( void){
 if (((long)(millis() - nextPresenceScan)) >= 0){
  nextPresenceScan = ((long)(millis() + 1000));

Same trick as before, sets up the nextPresenceScan and then calls the function to do the work.

void scanRSSI(){

 float currentRSSI = WiFi.RSSI();
 if (currentRSSI > 0){
  // sometimes the WiFi.RSSI() call can return a number greater than 0 to indicate
  // an error or that it simply couldn't get the info at this time. If that happens
  // just ignore it.

 // add the new value into our buffer at the current rssi buffer index
 // multiply by -1 to turn it into a positive number
 rssiArray[ rssiIndex++] = currentRSSI * -1;
 // if the index is now beyond the size of our buffer reset it back to
 // the beginning of the buffer for the next read
 if (rssiIndex >= MAX_RSSI_ARRAY){ rssiIndex = 0; }
 // first value we need for the variance calculation is the average
 float accumulator = 0;
 // since the buffer may not be filled with valid data yet 
 // we need to count how many non-zero values there are for the
 // average calculation rather than just divide it by the 
 int averageCounter = 0;
 float average = 0;
 for (int i=0; i<MAX_RSSI_ARRAY; i++){
  // ignore values of 0
  if (rssiArray[ i] > 0){
   if (rssiArray[ i] < mn){ mn = rssiArray[i];}
   accumulator += rssiArray[ i];
 // if we don't have at least 10 valid values in the array then just drop out
 // there is no point in calculating the rest unless we have enough values to 
 // look at
 if (averageCounter < 10){ return; }
 average = accumulator / averageCounter;

 // now loop again to calculate the SUM portion of the variance
 // reuse the accumulator 
 accumulator = 0;
 // loop through the buffer array again, this time subtracting the average and then
 // squaring the result adding it to the accumulator
 for (int i=0; i<MAX_RSSI_ARRAY; i++){
  if (rssiArray[ i] > 0){
   // dont want to waste the memory to include the floating point or C math
   // libraries, so just put the calculated value into a local variable and
   // multiply that by itself
   float work = (rssiArray[i] - average);
   accumulator += work * work;

 // here we divide by n-1 
 float variance = (accumulator / (averageCounter - 1));
 // the variance may be less than 0? if so take absolute value
 if (variance < 0.0){ variance *= -1; }
 // now do something clever with the variance data.


That is actually pretty horrible code. I also do realize that I’m never taking the square root of the variance as calculated. That doesn’t seem necessary at this point and the poor arduino is already working hard with all those floats. If I make the value even smaller it will be harder to watch whats happening. It’s also not memory compact as it needs a full buffer. I’m sure some other kind of rolling average could be used. To get anything useful you’ll have to adjust the threshold and the scanning frequency and probably a lot of other stuff too. An interesting starting point to something potentially useful.

The device in question here is sending it’s values back to XTension for graphing and potential use. The graph above was generated by that program. XTension: Home Automation for the Mac, is my day job.

Sunday, May 5, 2019

Using wifi signal strength as a motion or presence indication

It seems like it should be possible to use wifi RSSI as a presence or motion detector. I have seen some very fancy devices that made low level use of the reflection information to target motion or people in a room, but the graph below is from an ESP8266 just reporting it’s RSSI every 5 seconds.

The device is sitting on my desk just a couple of feet away, so the range might not be very good, but still there might be instances where this is useful. You can quite plainly see in the above graph that I was in the office on and off until just after 7, and then it is fairly smooth while I was attending the local Philharmonic concert to hear my wife sing Mahler’s 2nd, and then picks up in variability as soon as I started going in and out of here this morning.

A lot of hysteresis would be necessary since there are many things that can affect the signal strength. Since it would take some time to know that you were there for sure it might not be useful to turn on lights or make adjustments when you arrive, but it could definitely be used to figure out when you’ve gone. Adding a Wifi calculated presence detection to such IOT devices would be very helpful. It would not work very well for battery operated devices however since you would need to wake up the wifi radios far more often than for the regular functions of the device.

Part two of this thought has code examples and is here.

Wednesday, February 6, 2019

Fancy Crown Molding for New Bookshelves...

The living room in this house has been something we planned to remodel completely since before the day we moved in. All the standard white enamel cabinets and shelves that come with these houses were going to get thrown out and replaced with wonderful floor to ceiling book shelves and a library ladder and all sorts of wonderful things.

I haven’t written about this yet because even so many years later it’s still not quite finished. We’re getting closer though. I just wanted to share this piece of test molding I made, oh, a little more than a year ago now, that serves as the template for the rest of it as I slowly completed the work. More info will follow as it’s even closer to being done now.

I’ve always been a bit intimidated by fancy crown molding. I didn’t know how to design it or make it work or make it go around the things that were there. This project has forced me to come to terms with some of that.

Here is my test piece for the multi-part crown molding that will go around the top of the new bookshelves. I’m so happy with the way it turned out I can’t even tell you. It even looks great in the finished product that I’ll upload another article about shortly :)

It’s all commercially available molding from the local Home Depot or Lowes. I tend to buy molding from Lowes because they sell it in 6 and 8 foot sections where at Home Depot I need to know how much I need ahead of time and cut it from a 10 foot segment that will by no means in the known universe fit into my car. I could probably save a few bucks by figuring that all out. But it makes too much work ahead of time. I can just calculate how many 8 foot sections I need and add one more to fill in the gaps :)

Sunday, July 29, 2018

Arris model DG3270A SNMP and backup decoding

I have a remote install of XTension setup at my mother in laws condo. The internet and tv company there started with DSL modems in the building, but it’s an older building and the wires in the walls never resulted in a very good connection. They finally got around to putting in a cable head end in the basement and switched everyone to cable modems. They did this without calling me first or otherwise letting me know that they were going to show up and drop in a new modem and make everything I have installed there home automation wise to drop off the network and kill all my incoming passthroughs and such. I got there last night to have a look at the new modem and reset everything up and I discovered some interesting things.

The modem is an Arris DG3270A and actually works pretty nicely. The wifi is significantly better than the old modem actually supporting a 5ghz wifi network as well as 2.4ghz AND it will allow you to set both the 2.4ghz and 5ghz to the same network name. This is the preferrable way to do this as your devices will migrate between the 2 as needed and you won’t have to be using the faster 5ghz one and then walk upstairs or into bedroom and have it drop out and have to manually switch to the other. Your device will just always connect to the strongest one.

They have the interface to it setup so that you can do all the configuration of almost everything you’d want to yourself. The default password on the box was not changed which in this case was just admin/password. There are many different models and many providers lock you out of these settings. With the old modem I had to actually call the company to get the password to it and they told me that they would give it to me but would charge me for the service call if I screwed it up ;) I didn’t screw it up ;)

I was able to change the DHCP range to carve out the same block of static IP’s that I give to her XTension machine and the Vera there as well as setup the couple of pass throughs so that I can ssh into the machine and the webcam if necessary. It even has dynamic dns support so I was able to add the modem to my account there and have it update that info too. I also have the client running on the Mac in her closet but it never hurts to have a backup to such things, without that updated I can’t talk back to her machine very easily.

With the previous modem I had written a little script that polled the status page of the modem periodically and dumped the values of the noise and signal level on the DSL line into XTdb for graphing. It was interesting to watch how bad the wires were in the building and how they got slowly worse over time. It really was time to either pull more wires or switch to the cable system as they had been promising to do for so long. So I went looking for the same status into for the cable mode to see if there was anything interesting there to parse out and track just for fun.

Even though the modem is on the standard router address of there is another page that gives all the status information on address which seems strange to me, why run it on a separate IP like that and not just on a different link on the main router address? I suppose it’s possible that is bound to that network and ip address regardless of what NAT network you have the rest of your network set to? So perhaps that will still work even if your network is set to use the 10.0.0.xxx range? I don’t know but it seems needlessly complicated.

There is a lot of fun info there that I’ll probably parse out in the future though I hope the cable modem will be operating well enough that I won’t really care. At the very end of the tab bar across the top though is a tab called “Advanced” and so I had to have a look at that only to be provided with a password field. Obviously setup with a daily password system that takes a known salt and does some kind of hash or encryption against the date to come up with a daily changing password. This is really a good idea as even if the user is looking over your shoulder and sees the password, or you have to send a password to one in an email or something after the day you did it it’s no longer valid and they can’t publish it and let the entire internet into the machine to screw everything up. Indeed, searching on the internet led to several sites with information about this and even daily password calculators like this one arris password of the day generator that one didn’t work as it seems each company changes the salt value for it. I never did get into that screen but I did learn some interesting things in trying to find out what it was. On that site above he talks about decoding the router.data file that you can download from the regular interface as a backup which I had already done. On his page he talks about having to untar it and then decrypt it. None of that was necessary for my file from this machine. It turned out that each line was just base64 encoded. Each line has to be decoded separately you can’t do the entire file. There was no other encryption.

The file that resulted had some initial setup stuff that starts with MAC address like data and long hex strings. None of which were the right length to have been the salt for the daily password unfortunately. And then a user section with a long list of SNMP looking addresses and all the configuration data that I had entered into the machine. Again nothing that looked like the salt but if the device supports SNMP then it may be possible to scan more values in there and find that as well as lots of other information. Of course there are no comments or anything else in the file that would help you figure it out I present mine here with any real data or password information changed. If you would benefit from seeing the actual data in these fields I’ll send you the real file, I doubt anyone really cares about getting on the wifi at my mother in laws house but I’m still not posting the passwords to the internet.

version 4
version 1
8c:ec:4b:e8:be:0f 0 4c2d31303639 7e6e6f636f6d6d656e747e 00000000
00:0e:8f:92:e4:f1 0 756e6b6e6f776e 7e6e6f636f6d6d656e747e 00000000
48:bf:6b:71:89:b3 0 4d696e64794e6168736950686f6e65 7e6e6f636f6d6d656e747e 00000000
40:4d:7f:1c:2a:a6 0 4d696e64797341706c655761746368 7e6e6f636f6d6d656e747e 00000000
dc:a9:04:8c:f2:c8 0 4a616d65732d4d6163626f6f6b 7e6e6f636f6d6d656e747e 00000000
48:d7:05:d9:14:bf 0 4d696e64794e616841697232303134 7e6e6f636f6d6d656e747e 00000000
<USR>;4;&_=@<20180419 233953>TR69 (likely the admin password user name, though I do not see the actual
admin password here. Likely it will only be present if you set it to something
other than the default which was just "password");66;&_=@<20180606 120200>TR69;2;&_=@<20180727 085921>USER;2;&_=@<20180727 085923>USER;4;&_=@<20180728 182904>USER (5 hex values, altered from what they really were);2;&_=@<20180728 182904>USER;4;&_=@<20180728 183201>USER (one of these is the password for the 2.4 ghz and one for the 5ghz;4;&_=@<20180728 183446>USER since I have them set to the same I don't know which is which);2;&_=@<20180728 183615>USER SSID;4;&_=@<20180728 184019>USER (the SSID of the 2.4 and 5ghz wifi networks, again I have them set SSID;4;&_=@<20180728 184104>USER to the same value so I don't know which is which) of a NAT map;4;&_=@<20180728 184933>USER (this is a NAT port mapping port;66;&_=@<20180728 184934>USER there are 4 port setting associated with each mapping, the external port;66;&_=@<20180728 184934>USER start and end port and the internal start and end port. I don't know;2;&_=@<20180728 184935>USER which of these 4 values here is which as I am passing through only one port;4;&_=@<20180728 184936>USER so all 4 values were the same.) port;66;&_=@<20180728 184937>USER port;66;&_=@<20180728 184938>USER;2;&_=@<20180728 184939>USER port map;4;&_=@<20180728 192237>USER (another port mapping here) port;66;&_=@<20180728 192237>USER port;66;&_=@<20180728 192238>USER;2;&_=@<20180728 192238>USER;4;&_=@<20180728 192239>USER port;66;&_=@<20180728 192240>USER port;66;&_=@<20180728 192240>USER;2;&_=@<20180728 192240>USER port map;4;&_=@<20180728 192319>USER (yet another NAT mapping) port start;66;&_=@<20180728 192319>USER (in this case I can see this is the external port) port end;66;&_=@<20180728 192320>USER;2;&_=@<20180728 192320>USER;4;&_=@<20180728 192321>USER port start;66;&_=@<20180728 192322>USER port end;66;&_=@<20180728 192322>USER;2;&_=@<20180728 192323>USER;2;&_=@<20180728 202234>USER;2;&_=@<20180728 202235>USER username;4;&_=@<20180728 202235>USER password;4;&_=@<20180728 202236>USER hostname;4;&_=@<20180728 202236>USER

Note that Ive munged the data above including the hex values just so I dont expose something important. Not that I think there is anything anyone could do with this info who wasnt already standing in her living room but still. Since I dont know what it is I altered it but kept the layout and length and format the same. My comments in parentheses are also obviously not part of the actual file either.

So it may be possible to setup these devices via SNMP including all the port mappings you wish as well as any other user data. Thats potentially useful if you wish to dynamically turn on and off port mappings or something like that. Im sure that much more data is available to a real SNMP dump of the thing assuming it doesnt require other passwords or encryption to set the data. In future visits I will play with the SNMP properties of the modem but I didnt have a chance to do that this trip. I might be able to get the connection quality and error rate that way which would be interesting to graph.

If someone can see the salt for the advanced daily password in that Id sure love to know what it is. It may be discoverable via a different SNMP address and Ill experiment with that in the future.

Wednesday, May 30, 2018

Restarting an AirPort via AppleScript

Apples airports are great devices, but like so many “appliances” they need a restart once in a while. Apple has the newer models pretty well locked down. There is no web interface, nothing you can hit with a curl command and no SNMP or anything else to talk to to tell it to restart itself.

After I lost my entire wifi network the other day and I had to restart it manually I decided it was time to tackle the AirPort Utility via User Interface Scripting. It turns out you can do this, sort of... There are either bugs in the AirPort utility and it’s implementation of user interface scripting, even though you’re not supposed to have to implement it, it’s just there because of the OS, or there are some limitations to it that aren’t clearly documented.

The applescript to restart an airport is pretty simple actually, this handler will do it, sometimes:

on restartAirportByName(theName)
tell application "System Events"
tell process "AirPort Utility"
click (first image of group 1 of scroll area 1 of window 1 where its name starts with theName)
click menu item "Restart…" of menu "Base Station" of menu bar 1
end tell
end tell

end restartAirportByName

Note that handler doesn’t actually launch the AirPort Utility first, You should do something like:

launch application “AirPort Utility”
delay( 5)

Before calling that.

There are some gotchas right away, the ellipsis in the Restart menu item name is a UTF8 ellipsis character and not just three periods. If you use three periods it won’t be able to find the menu to click.

It won’t work if the application is hidden or the main window minimized. It does not have to be frontmost, but it cannot be hidden in any way.

The name of the “process” is case sensitive. It must have the same case as the actual application in order to match, even though much else of AppleScript is not case sensitive this is. It must be “AirPort Utility” or it won’t work.

Next the script or the host of the script must have assistive access turned on in order to use the UI scripting at all. You have to do that in the System Preferences, Privacy & Security pane under Assistive Access. This will also break if you are saving the script as an application and make any edits to it. You must go in and uncheck and recheck the checkbox next to the application you created or it will know that you changed it and won’t let it access the UI. Again, you don’t get the helpful error message that you’d get if you hadn’t added it to the list at all telling you that assistive access is not enabled for this program, it just fails to be able to do anything.

Beyond that there is something weird about how AirPort Utility processes the event, or there is a limitation for processes not specifically started by a user which may be the case. I would expect those to generate an error specifically telling you that you don’t have privs. Sometimes, or all the time depending on the OS version you get an error about a bad index when trying to get a reference to window 1. I’ve tried accessing the window by name and several other hacky work arounds but it doesn’t seem to matter. If you run the script from a program that has assistive access turned on, but wasn’t launched by the user you’ll definitely get this error at least once, and then it might start working for the second time. Or it might not.

In my testing the best solution is to embed that bit of script into an applescript saved as an application. Then you can give it privs to use assistive access and just launch it, or use some other system to launch the app. That seems to work. At least on OSX 10.13. There still seem to be problems doing these things on 10.12.

I’ve wrapped that and a whole lot more scanning and checking code including the ability to monitor your internet connection and get your external IP address into a larger script that talks to my home automation software of choice XTension (well it should be my choice as I’m one of the developers) which will definitely be useful for other XTension users as long as they are running an OS version that includes a less buggy AirPort Utility. That code is free and open and if you are interested in how to get the rest of that info out of it it’s also fully documented with comments in the code here.

Tuesday, May 22, 2018

A fix for Stupidly Slow VNC performance on the Mac

On and off I’ve had a problem with my house server where the remote desktop or VNC connection was just so slow that it was almost unusable. I normally load test versions of XTension by dragging them and dropping them onto the VNC window. This was also excruciatingly slow. Like 200k a second slow rather than the 60 or 70 megs a second I would expect.

I’ve spent months fighting with it, restarting seems to help for a few minutes but it comes right back. I’ve run new ethernet cables, I’ve tried plugging directly into the switch. Of course it never occurred to me to test other throughput to the server. Connecting over regular SMB file sharing to it results in more than 70 meg a second throughput. So it had to be something with VNC.

Just gazing at my system logs the other day I watched 3 or 4 zombied windows machines trying to log into my VNC server as it’s exposed to the internet. I have the ports for VNC passed through my router so I can log in remotely if I ever needed to. They are just running through the list of most common passwords and dictionary words hoping that my password is just “secret” and I’ve just ignored them up until now.

Just as a test I turned the VNC passthrough off on my router and guess what? I can now VNC like I could before and I can get 70meg a second throughput when dragging large files onto the VNC window! It was truly awful to try to type in an edit script window on the machine remotely, but now it’s wonderful again.

It seems that a failed password attempt causes the VNC server to hang up for a fraction of a second. Multiply that by 2 or 3 attempts a second from different infected windows machines and it comes to a screeching halt.

For myself I’ll just leave those passthroughs off unless I specifically think I’m going to need them. Moving your VNC install to a higher random port would solve the problem from just the hacked windows machines as they won’t bother to do a full port scan on your machine, but it won’t protect you from someone targeting you specifically as they will do a port scan and find it no matter where you put it. It would definitely help though.

OSX is supposed to have an adaptive firewall that you can turn on that watches the log files for those failed password attempts and if someone has more than a certain number it will temporarily block them. I can’t get this to actually work on my machine though. Much info out there suggests that this is broken on OSX but I will experiment further with it.

Saturday, January 6, 2018

Copper Veneer Is Beautiful

I am in the middle of a project building floor to ceiling bookshelves across one wall of our living room. It’s a rather larger project than I normally take on and has provided many unique and stimulating challenges both in design and fabrication... We decided on a gentle arch at the top of each book case and in order for them not to look ordinary and drab I’m experimenting with some antiqued copper veneers that I got from Veneer Supplies.com The same place I got the wonderful dark burl I used to make our Veneered Mantle. I chickened out and didn’t buy any of the really weird and wonderful copper patinas available, but opted for a sheet of just the antiqued stuff to experiment with.  this stuff is beautiful.

This is just a test run to hang one up so that She Who Must Be Obeyed can get a feel for it and make sure that it’s what she wants before I commit to making a dozen of these things. There are only 6 bookshelves, but there will be an arch at the top section, and an arch below the library ladder rail 8 feet up. With each part of this project I keep thinking that won’t be a lot of extra work, and then I remember that oh, there are six bookshelves so I have to do it six times. And then with these I have to make twice that number. But it will be worth it, these look fantastic.

Now if only I could make the stain on the routed edge look a bit more even. I will have to sand that a lot more aggressively to smooth out the wood before applying the stain on the real ones.

Dont worry about the slightly funky edges. I made it an inch or so wider than it needs to be and will trim those off to perfection on the table saw before putting it into place.