I’ve spend the last six months getting deeply into programming again and over and over I find that I learn differently than the way most tutorials take you. So here’s a quick list of new things I’ve learned and the fundamental idea that (at least of me cracked the nut).
Docker. This is the big one, the tutorial is at once simple and confusing because it is so unclear how it all works. Yes there are containers, but the biggest thing that they do not explain is the docker daemon. if you understand this one idea, then figuring out docker is pretty easy (at least for me). The concept is buried in a docker-in-docker article but the simple point is that on every machine there is a single docker daemon that controls all docker containers. Previously they had a concurrency model, but this is more maintainable. So this explains what is happening with a Dockerfile, when the daemon (a super user kind of thing) gets control, it get a zipped image of the entire directory where the Dockerfile lives and it processes from there as it’s world. It also explains how docker machine works. You can access the daemon by taking to it on a well know named socket if you have the right credentials. When you do a docker pull, it is this one daemon across all the users on the machine that does the caching. And you can connect to that daemon so that a docker container can create containers across the internet, or side-by-side.
Bash. This is one of those programming languages that no one likes but you have to use. It is natural because it is the simplest way to glue together commands. There is one key idea in bash. That is the idea that every command has a return code and this is always what’s driving what happens next. If you have set -e on it will even stop a script is you have even one bad exit codes. Even things like ((i++)) has a value and also a return code. When does this bite you, well in the middle of a long pipe, you can have a bad exit code and the thing crashes, so watch your error codes!
Objective C. The main thing is that everything is an interface, so you have to be careful about declaring and them and making sure you use the right one. Not complicated in theory, but man complicated in practice.
Python. You can think of it as a simple procedural programming language which is how most uses of it go. Sort of like bash but longer, but the real power is in its string and array functions.
Wow, it took me a long time to find these bugs, but I’ve been working on various bash scripts and I finally figure something out that is hard to know:
There are a bunch of of bash safety checks that you can turn on. Most see great, but there are real gotchas. Here are two difficult ones. I hope you never encounter them:
set -e and the && construct
- We use a restrictive set of checks here
set -ueo pipefail which means the scripts that the scripts stop is any command returns a non-zero exit code, or if anything in a pipeline fails or if you use an unset variable.
I have been using a common construct to print out errors,
$VERBOSE && echo start edits
But this has a series of consequences.
- What is the return value of the
$VERBOSE && echo foo if VERBOSE if set to false. Well, it is not the same as
if $VERBOSE; then echo foo; fi as I thought. The former returns 1 while the later returns 0.
So you can’t use the pipe on success as a synonym for the if-then. In particular if you use
set -e, the script will mysteriously fail in normal mode but run find in debug or verbose mode!
In fact any multiple pipe like
ls -l && ls -l ../lib will fail
- There is another common thing
test || echo false && echo true but be careful this is not equivalent to an if-then-else becaue if the second statement fails, then you will never get to the final thing.
The effect is more subtle actually, as long as you use just the || operator you won’t have a problem since it only tests for false, so
ssh email@example.com || true is a simple way to run something if something fails. This is ok, but the other version
ssh firstname.lastname@example.org && echo works" does not because you get an error code as the return value, you can tryssh email@example.com && echo works || false` but only if you can guarantee that the “echo works” will never fail which is of course pretty impossible :-0
set -o pipefail causes ssh to hang up and behavies differently in single step
The next one is even more insidious, one good suggestion is to turn on -o pipefail. This makes sure that in a long pipeline, you don’t mask errors, so for instance
false | true will be ok with pipefail, but will fail with it. this helps you check long pipelines. There are two things that make it unusable at least for me on Ubuntu 14.04:
- If you grep for something and don’t find it, there is no way to have the thing not fail. You can’t
grep home /etc/bar | cut -f 1 for instance, you have to cut it into pieces because of the grep fails if it doesn’t find things, so you have to have temporary variables everywhere
x=$(grep home /etc/bar) if echo $x | cut -f 1…which is really a mess.
Things get very strange with ssh and interactive traps. But if you have pipefail on and are in an interactive ssh session, it will close the ssh session if you encounter an intermediate pipeline failure. There isn’t any recoverly. Also if you use trap DEBUG to catch errors, then it will work fine because the pipefail is cause by the trap. So it is very hard to debug.
Net, net -o pipefail is optional and I’m glad it is. I’m taking it out of my stuff now.
Net, net the lesson is ignore the various google queries that are clever about && and || for pipes, use if-then-else. Sigh, I have a lot of code to fix!
OK, here are some quick notes about things you should do to get a basic level of privacy:
- Signal. This is a smart phone application that let’s you encrypt your calls and text messages. Its on IOS and Android and super easy. If you are a geek, then you can also try ChatSecure, the same idea, but uses an open source library called OTR and routes through the TOR network. Whew. Of course I don’t have anyone who I can actually chat with.
Hard disk encryption. Turn on FileVault on your Mac, BitKeeper (I think on Windows) or Ubuntu disk protection (you need to reinstall though for that).
Password Manager. KeePassX is free open source, but I like 1Password even though it is closed source because it works well with browsers.
Two Factor authentication. This is a pain to setup, but it makes it very hard to crack since the attacker needs your cell phone. Google, Amazon and Dropbox all have it.
Tor Browser. A complete pain to use and slow, but it is safer.
Ad Blockers. Mainly so folks can’t track you.
VPNs. PrivateInternetAccess works well for me.
Then there are some experimental ideas:
- Qubes. It uses Xen to be a security oriented hypervisor
Well if you missed the notification, there is something called COPPA where Amazon has to certify it doesn’t advertise to sites that market to kids under 13. If you didn’t fill out the form by October 31, they close your affiliate account and invalidate all your tags. How nice legal things are.
Oh well, so now you have to create your affiliates account and then there is the handy Better Search and Replace plugin for WordPress that let’s you change all your tags. Good luck!
Well these last three days feels like a hard slog. Here are some of the tips for Linux geek, so hopefully you can avoid problems when you are just trying to get a regular job that will pull from a git repo and run an automated build.
There are many systems available to do this, so this more educational than anything else, but it helped me learn quite a few things about ubuntu 14.04:
Crontab is a really nice utility for running user jobs at regular intervals. Like editing of system files where you want some protection it is a little strange. You run
crontab -l to see your system jobs. If you don’t have anything listed, it returns none found and it isn’t clear how to do the equivalent of a touch. The answer is
echo "" | crontab -` which says put a newline into the crontab
Crontab is also strange in that it needs a blank line at the end otherwise it will not process things!
Crontab has an interesting syntax. Basically an asterisk means run it all the time. If you put a number in a column, it means run it when that column is equal to the number. So if the minute = 0 for instance, it means run the line everytime you are at 0 minutes (that is on the hour). The syntax “*/30” means that you should run it every 30 units (for instance 30 minutes if it is the first).
Finally if you are trying to run ssh in a cron job all kinds of strange things happen. You can’t put in a password for your ssh key because it is in the background, but you can use
keychain to add it and have it remember your password (so you only have to type your password in every time the machine reboots. But in order for this to work, you have to rememeber to source a magic file in
.keychain which is named $hostname-sh. This just sets the SSH_AUTH_SOCK variable correctly for you.
You can actually manually figure out where the ssh-agent is by looking through /tmp and seeing if the a magic file has your ssh key in it using a find command and then grepping the name of the pipe.
Ok, for the truly nerdy, if you just want to change your pass phrase on a ssh key, here is how to do it:
ssh-keygen -p -f _your private key file_
When you are doing this, you have a chance with the switch from an MD5 (not very safe) to a bcrypt with lots of rounds as well.
Well, a Honda Fit is an incredibly practical car. You can buy expensive accessories, but the after market is terrific:
Floor mats. Rather than the factory ones, from Auto Anything, you can get custom fit nearly impervious ones for $99 less 6% off in a cilck through rebate and Retailmenot has a 15% off coupon. Not bad. Hexomat works well, but we are trying Lloyd this time.
Pleather seat covers. Ok, this sounds terrible, but they actually work well. Easy to wipe off and they don’t wear like cloth seats. And if you have them getting wet all the time, there is nothing like it. Coverking used to make Honda Fit covers, but no more, now we are trying. Now we are trying the 2015 covers (the 2016 Honda Fit is the same except it is made in Japan and not Mexico so hopefully more reliable). CalTrend “I Can’t Believe it is not Leather”
Yakima whispbar roof racks. You truly can’t even hear them. For the 2012 Honda Fit, you need the S24 and K446 fit kit. The newer 2015-16 need the Whispbar Flush S33 and and K853W. You can also get a long rack with the HD Bar T15 or the through bar S15 all using the same foot. The different is that the flush is very finished and the rack doesn’t extend past the foot. The through bar goes past it while the HD is designed for work applications. The flush sees the right choice. You can get from REI (remember there is a 20% off with a Discover card payment through Apple Pay plus 10% off as an REI member) if you buy at the store before DEcember 31st with a new Discoer car.
Hitch. You can actually tow up to 2500 pounds and have a tongue weight of 150 pounds!
Uh, why did they do this, you cannot run Unifi.app in /Applications, it needs a writeable directory. So it needs to live in a writeable place like /Applications/Unifi/Unifi.app and make /Applications/Unifi writeable, but this didn’t seem to help.
What did was to delete and reinstall and then it seemed to work after a reboot. I don’t know why.
The Unifi Discover application seems to take a long time.
This thing remembers old installations and removing the application doesn’t help, so if you have ever installed 3.x or 4.x, you will need to recover your password
If you have lost your password, you have to download Mongo, then run the bin/mongo, use ace and run db.admin.find() and you can see the password in clear text. Yikes, this is not the most secure system.
Wow, I don’t know why Amazon doesn’t advertise these things more, but I’m kind of stunned at the things you can get if you are Prime member now. You should sign up! In order. Well it isn’t 42, but it sure feels like it:
- Of course the obvious one is free two day shipping.
Free Amazon Now deliver if you live in the right cities.
And with Amazon Prime submemberships. You can invite four other people to get free shipping.
Amazon Prime Store Card. If you get approved for this, it is a free charge card with an amazing 5% rebate on things that are from Amazon. I’m not super clear on this, but I think it doesn’t include things that go through third party sellers. So you want to use it for things that are fulfilled by Amazon. Right now you get a $20 gift card and $50 off the Echo for signingup.
This quarter if you are a Discover It or Chase Freedom (both free cards), then for the first $1500, you get 5% off of Amazon purchases. I think this does cover merchats we sell through Amazon, but we will see. If you bought Discover It in the first year btw, this is doubled, so it is 10% off!
Amazon Prime Video. I actualy like this service quite a bit because on the iPhone it let’s you download videos (which Netflix doesn’t).
Then there are things that are cool extras:
Amazon First. You get a free Kindle book. Wow free books got to love it.
Amazon Household. Similar to Apple family plan, you can share your digital content with one other Amazon account. There is some sort of kids thing that I don’t totally undersand
Amazon Prime Music. You can listen to a million tracks for free.
Amazon Photos. I do not use it because I have high resolution photos, but for other folks, it sure is nice to have unlmited storage for photos as an archive.
Amazon Gold Box. You get 30 minutes advanced notice. Personally I’m more of a research first than impulse shopper.
Amazon Mom. You can share you diaper discount with others.
Gosh what a pain, last year we were building very fast Haswell systems in micro-ITX. We could get two dual slot graphics cards in there and it sounded like a freight train, but could get overclocking to 4.5GHz or so with the Noctua NH-12L which is low profile with great clearance underneath. One fascinating thing though is that by the numbers, the NH-12L with both fans installed is very efficient. Although low profile it seems to work as well as the big ones. It needs modules to be 43mm or less so it is a tight fit with the ADATA V3 at the 92mm fan spot. I normally use slot 2 and 4 and this is fine.
But this was not the most efficient setup as there is plenty of top clearance in the case for a full height cooler. We got the Prolima Genesis which is basically the best cooler around according to SilentPCReview. Unfortunately, it didnt’ quite fit with the Adata XPG V2 memory. You really have to know the heights of various systems. The cooler has very low clearance at the 1st and second slot. I could actually get it to fit if memory were just in slot 3 and 4, but unfortunately on the Asrock Z97M OC, for interleaving, it needs slots 1 and 3 or 2 and 4, so we are stuck.
In looking at the specifications of the Genesis it shows that a nice chart where the maximum height is 32mm at slot 1 (won’t fit) and 44 mm at slot 2 then 55mm which actually does work. Just move things to slot 3 and 4. But the deeper problem is that it is too long for a small case. It is 216mm long and over hangs into the space for the power supply in the tiny Silverstone SG10-B case which the specs show can support 165mm high coolers (The Genesis is 159mm so tight but that’s the point). The Genesis actually protrudes over a micro-ATX motherboard rear edge and hits the Power supply. If you want to use the genesis you have to use a larger case in summary.
Next up was to try the Scythe Big Shuriken Rev B. This is also a low profile unit, but with a 120mm downfacing fan, it has a ram height clearance of you can’t tell from the product page, but certainly doesn’t fit the 44mm highs, it looks like you need very low profile particularly in slot 1. This is a model that works well if you have low profile Ram.
The next one on the list is the Thermalright Silver Arrow. This one looks like a decent fit. The product spec shows it is narrower so will fit withint a micro-ATX footprint (unlike the Prolima Genesis), and in it has a 160mm height. The Ram heights are 45mm so it will fit at the furthest, but slot 1 may not work as those clearances look like 30mm and then 38mm.
One simple solution to these problems is to get lower profile memory. The gamers stuff adds kind of ridiculous coolers that add 13mm to the height, but the new ADATA V3 let’s you remove those fins. These things are usually glued on so you can remove them. But memory is still expensive at $110 per 8GB, so it is cheaper to get a better cooler or figure out a way to take these things off.