TL;dr

For web demos, use docker and docker-compose to bring up your entire system on your local Macbook.

For hardware demos, use a big beefy wireless router and your phone as a hotspot to blow through and create your own network.

Web Demos

These days basically everyone has to do demos on the road. Some of us are lucky because they are basically just web demos. In the old days, you would have to make sure the host was running and pray the VCs internet connection is good. But today, I think the best method is to run the whole demo including all the web servers and database servers on a local machine using docker and some sort of orchestration (ansible as an oldie but goodie, bash scripts if you don’t have time to mess around or docker-compose for most uses and kubernetes if you already use that).

The advantage is that it is totally self contained and as a side effect, it can be your personal development environment. Most folks are using MacBooks, so the new Docker for Mac even provides a new virtual machine environment although I’ve found virtualbox/boot2docker work pretty well

Gadget Demos

If however you have real hardware, then it is tricker, you need to bring your own network with you. Don’t rely on what is there because most folks are in very noisy environments. What you want is a super powerful 5GHz router with beam forming to cut through the noise. The Netgear AC1900 Blackhawk is a good one. Expensive but worth the money at $180. The AC1750 is nearly as good at $120.

If you also need internet access, make sure you have a good data plan and bring your cell phone with you. I particularly like the combination of an iPhone on Verizon plus a Google Nexus on GoogleFi. This gives you in effect three of the four big networks. The truly paranoid would also have an AT&T phone as well.

Well after you read the tutorial on docker, you find that there are least 1M ways that people build docker images. In trolling through the docker.com and github.com, here are the best practices:

  • Believe in micro services. If you are putting more than one major function into a container, you probably want two of them. The isolation benefits are really large when you do this and the performance hit minimal. The biggest gain is no more conflicts between libraries.
  • For each service, create a github repo or at least a subdirectory in your main project. This is where the Dockerfile will live and all the files that you need. There are some folks (oh WordPress just autocorrected that to fools, LOL!) who will either have one monster CMake or Scons script or bash script “to rule them all”, but I have found it to be way more readable to use a small Make for each service, this makes it easy to move.
  • The folks at hypriot have a really nice scheme. The directory has a Makefile and a build.sh for thing that happen at runtime. When you build, you just pick a target like build image and it happens. Put all the local configuration files in here and use Dockerfile to COPY them into your image. It is really nice to have a local Dockerfile and a local Makefiles. Note that the build.sh can usually be subsumed by the Dockerfile which is easier to read. 
  • This scheme doesn’t help much with two problems. First what if you have different flavors like Raspberry Pi, i.MX6 and Intel that use the same files, in that case, I use different Dockerfile.{rpi,intel,imx6} and then have make targets for them.
  • The other problem is that you do not want all the build libraries in the runtime image, so you need to have one build image and one runtime image. The trick here is to build the first image, then copy all the build artifacts into a docker data container. Then when you use the runtime image, you can use the volume-from to get just those. Then copy them back into the runtime image It is tricky to figure out what the build artifacts, but that makes it much easier. You can also use the host environment if you don’t like containers, then you create a /var space and use docker cp to get them out and then COPY to put them in. But this means you need to know about the host file system. Another approach is to use the VOLUME command on the Dockerfile and then use a volumes-from to copy into a runtime container. 
  • Finally when you are done, you can push these into the docker hub. Make sure to also push up a readme.MD and I haven’t figured it out yet, but you want to also publish the location of the github repo that has the Dockerfile and all the configurations.

Ok there is a place that is actually beyond the bleeding edge and I guess I’m right there. Fortunately there are amazing folks like Luxas, Rhess and the folks at Hypriot that are right there with me, but here is what I’ve learned about docker and its internals and how it handles clustering. But

TL;dr

Right now for production despite what it says the best choice is to use docker swarm with their hosted token discovery. This seems pretty stable and safe as you get a nice long 32-bit GUID. For development, I’m working on figuring out which of the many Kubernetes installations will actually work.

A tutorial

  1. docker by default is not network aware. There is a magic file called /etc/default/docker (at least when you use HypriotOS which is a customization of Raspbian) which has the docker daemon settings. Normally this is not set to listen to any external ports. It needs a variable DOCKER_OPTS to be set
  2. The simplest thing to make it aware is to use docker-machine. This will connect to any docker using the -d generic driver and it will automatically set up a secure port note that this is insecure, so docker-machine does a opens socket 2376 and adds the TLS certificates so you can access it safely. If you do not care about security that you can set any docker client to look at a machine by setting export DOCKER_HOST=tcp://machine.local:2375 note that you can use dns names. If you use docker-machine, then it manages the certificates for you and uses the secure 2376
  3. If you are using docker swarm, then you need to access the cluster on a different secure port -H tcp://0.0.0.0:3376 while the 2376 ports remain available. This has to be on the swarm master. This is all set by the --swarm to for each swarm member and --swarm-master for the main one. The whole thing works because you have a swarm docker image which manages everything.

So this will give you basic access to docker over the network. But it doesn’t let you access multiple machines as one. This is what clustering all the about and where all the fun begins.

Here are the current traps with Hypriot

Since raspbian does not have docker, the good folks at http://hypriot.com were good enough to port it to the Pi. They also have a version of raspbian called hypriotOS which basically adds their repo and then installs their packages docker-hypriot.

As with all things there are some gotchas and here the strange ones:

  1. If you install cluster-lab, then be careful about you deinstall. You *must* first systemctl stop cluster-lab which puts back the /etc/default/docker file correctly. And then do a systemctl disable cluster-lab.If you do it in the wrong order you will get a back file there and you need to fix it by removing the cluster-lab port -H in that file. The error is pretty obscure, you get a cannot start docker message, no docker host.
  2. As an aside, if the docker daemon doesn’t start, you only get a very short message, you need to run systemctl status --all docker to get the complete list, in this case, the offending failure is just beyond the last message so it is easy to miss.
  3. You cannot apt-get remove hypriot-docker because it does not correctly add back the /etc/systemd/system/docker.service  because it uses the default installation which uses the aufs driver, you have to manually change that file to use --driver=overlay.

A primer on clustering layers

  1. To make this work you basically need three services and there is lots of competition for them. It’s nice to stay with docker as much as possible but even more important to stay on the main line.
  1. Networking. With a cluster, it might be on different machines and the IP addresses are changing. So folks typically put in an overlay network so that the cluster gets its own virtual network to play in. The approaches are to use the Linux vlan to which seems to underly the other mechanisms. Docker now has overlay networking built in or to use flannel. Docker seems simplest but flannel because of kubernetes support seems most popular with the big boys. So right now hat tip to flannel.
  2. Discovery. If you have a virtual network, you need a way to figure out where the different hosts are. With docker networking you get this for free. But with others you build your own dns provider.
  3. Key/Value Store. This is where you can put parameters in so you can figure out what is running. Docker hosted is a test option but  Consul and etcd seem like the two main ones so consul being easier to use but having less support and at least for is a bit buggy. There are a few approaches like consul (cluster-lab) and running etcd in either in a container (kubernetes-on-arm) or natively (ansible-kubernetes-openshift-pi3). Right now hat tip to etcd.
  4. Orchestration. This is basically how you start a bunch of machines and connect them together. Docker-compose and ansible seem to be the two choices. Docker compose is elegant but not super powerful. You can’t specify which node for instance. Right now hat tip to ansible.

Raspberry Pi Clustering

For the Raspberry Pi, there are at least four different methods for getting clustering working. Tl;dr we are using docker swarm right now with their hosted service and hoping either luxas or rhess get it right.

  1. Docker-machine without swarms (Working rpi1 and rpi3). Currently this works fine with hypriot-docker on the rpi on their 2016 image for both rpi1 as well as their test 0.4.9 image on rpi3.
  2. Docker swarms using a hosted service (Working on rpi3) This is not supposed to be used in deployment but uses docker’s own system as the host for the cluster. You need a host so you can store central configuration information about the system. You basically generate a random token and then use it with docker-machine to create a swarm. This is pretty easy and you get it working. My main problem right now is that docker-machine doesn’t seem to be working correctly against the hypriot and is hanging. I think this might be related to using the pi account instead of root, but am not sure. The main weakness that it uses dockers hosted. Also it doesn’t handle orchestration at all. So have to either write bash scripts or use ansible. You can of course run individual jobs on specific nodes by using docker-machine env to connect to specific members and that basically works or course and is a good workaround.
  3. Hypriot Cluster-lab (not working reliably and corrupts install for docker swarm and docker machine). They bundled it all together into something that uses consul for the control. This worked amazingly well for a single cluster on a single network switch but we had lots of trouble across switches. The vlan support seemed to work but consul discovery didn’t happen reliably. And I’m not are investing more time makes sense given kubernetes seems more popular.
  4. Kubernetes on arm (does not run if cluster-lab ran before it). There are at least three flavors of kubernetes for rpi. Luxas has a nice project kubernetes-on-arm that uses a docker image prebuilt. The main problem is that I can’t get workers to connect with rpi1s. The 8080 API server is not coming up. Also there are conflicts with Hypriot prebuilt for rpi3 that causes Hans.
  5. Kubernetes/ansible. Rhess has a similar project but he adds ansible for orchestration. Haven’t tried it yet.
  6. Kubernetes hand rolled. There is a guide that shows you how to do this but I haven’t tried. Will do if the first two don’t work.

Well if you somehow can’t afford the $128K Pascal server that nVidia just announced (I can’t imagine why), what should someone do if they want to do some machine learning?

Well, unfortunately, there are no cheap good alternatives right now as we are in the middle of the Maxwell to Pascal transition with nVidia and they are only shipping out super high end systems, so the main thing to do is to build a chassis and know that you will likely get rid of the current graphics cards. In other words, don’t future proof the cards, get just what you need.

Here is the recommendations on PC Parts Picker.

Graphics Cards

This is a $5-6K machine, but most of the cost is in the graphics cards. I’d say get one or two cards to start and see how your load is. The most important variable is whether you need 6GB VRAM or 12GB VRAM. The GTX 980Ti is the bargain unit here at half the price of the Titan X, but if you are doing deep neural networks, you may have to use the Titan X.

In the actual build, While you can do heterogenous deployments, I think most folks would just get either one or the other and double it up if they need more but the choices are:

  • ASUS Strix 980 Ti GTX. These are quiet cards and run well. Hopefully your load will work in a 6GB VRAM if so go for these.
  • The Titan X are hard to come by the ASUS Titan X is slightly factory overclocked, but should reach 1.2GHz if you need it. The prices are roughly $1K each.

While the graphics cards will change, the system is designed so that you can use the next generation easily enough.

Motherboard

This is the part that lasts the longest and the ASRock X99 OC Formula has all the latest features including USB 3.1, NVMe and two M.2 slots

Power Supply

The machine has enough headroom thanks to a gigantic power supply to run 4 of these cards plus the 140 watt cpu. Although it doesn’t have the normal 20% extra capacity that you want

Processor

The Xeon E5-1620 V3 overclocks and can get to 4GHz easily so it is a bit of a ways from the 4.4-4.5GHz I can get on Skylake or even 4.8 GHz if you go oil cooled on the older Sandy Bridge, but still pretty good and it gives you 40 lanes of PCI Express which you can’t get with Skylake yet.

An alternative build is to use the Core i7 Haswell-E. This is nearly an identical build but you can’t use ECC memory which matters for reliability for $10 more. Not a bad deal.

It doesn’t look like the Skylake Xeon parts will be that much faster as they are really power optimized and in a system where 80% of the power is drawn by the GPUs that really doesn’t matter as much.

The one thing that you are giving up

Memory

Well life was easy with desktop systems. Just tell me how much memory you need. The maximum is 64GB though and for big jobs you pay for more. Plus you want ECC so that you get protection from bit flips.

There are no less than three different kinds of ECC ram.

  • Unbuffered. UDIMM. These are very fast but have a maximum memory limit
  • Registered. RDIMM. These are slower by about 12% but support double the memory of UDIMMs.
  • Low load registered. LRDIMM are slower, but have double the density of RDIMM but this is not compatible with the X99.

The main reason for this is that LRDIMM is higher density but is slower and more expensive. DDR3 maxed out at 8GB in UDIMM per memory module, but DDR4 are double that density at 16GB/slot but this is very pricey. The most economical for a fully populated system would have 128GB across 8 slots with 16GB/slot in DDR4.

The ASRock OC Formula/3.1 has 8 slots for DDR4 and supports x8 with a maximum memory of 128GB, so each slot is limited to 16GB memory (16Gb chips in x8 configuration).

SSDs

This is an all SSD build (I’ve been all SSD for the last two years). This is because it is quiet and reliable. Mechanical drives definitely seem like the first thing that fail.

Make sure you update the X99 firmware so that you get full NVMe support. But this thing has two m.2 slots. One that is NVMe 4x and the other 2x PCI Express (if you have an old Plextor you can get over SATA speeds with a full 2 lane implementation). Johnnylucky.com gives you a list by interface so the Plextor M6e is $160 so only $20 cheaper than the very fast Samsung 950 Pro. I’d recommend just getting the 950 Pro and using it in the slot. You only get half the performance, but when you want to move on you get a super capable system.

For the SATA SSD storage, the SanDisk Extreme Pro remains super fast, although if you need lots of bulk storage the 1TB or if you want a little cheaper you can get $200 1TB storage as well.

Well if you have a corrupted JPEG, how do you get it back?

  1. http://thelawlers.com/Blognosticator/?p=795 points out that Adobe Photoshop has a built in recovery application. So if you have something that doesn’t open in Mac Preview, try to load it in Adobe Photoshop. I tried this and although the image as distorted (it was shifted over, it did the recovery!). Then a little copy and paste to make it look OK and voila it returns.
  2. There are for charge programs that try to do the same thing.

OK, with the new M.2 SSDs and also lots of SSDs, you do need to be careful when building machines. Here are some notes:

  1. The new M.2 slots on motherboards share PCI Express slots with SATA. So when you install your SSDs, make sure you do not use the SATAs that are dedicated to the M.2 slots.
  2. There are certain power cables that are *not* compatible with each other. Use the power cables that come with your modular supply. I just had a friend who fried two SSDs by use different cables on a supplies. Some cables have their power and grounds inverted as these cables are *not* standard. Beware!

It is incredible what folks charge for things that should be free. They seem to rely on people forgetting what is going on and automatic charging. My buddy Holden had a web site package for $140 a year that was essentially just forwarding email. If you have the same problem here is how to get out of the mess:

  1. Figure out who is hosting your vanity site. For instance, Network Solutions (man are they expensive!) looks you into multiple year contracts. In this case, he had one going to 2018, so at least he doesn’t have to worry until then.
  2. If you don’t need web hosting, then what’s the best way to get a free vanity domain with mail. Well, Google Apps charges $5/month. You do get everything from storage to calendar, but https://simplyian.com/2015/01/07/Hacking-GMail-to-use-custom-domains-for-free/ points out that you can get free vanity domain hosting by using mailgun.com

It is for power users, but here is what you do:

  1. Get a free gmail.com account. Make sure to turn off all the pesky tracking in the settings menu.
  2. Sign up for mailgun.com and use that gmail.com
  3. Now go to your registrar and suck back your DNS provider and then you have to set up a huge set of domain name records.

Here are the records:

  1. TXT records. You need this for spam prevention with SPF. If you want it to run against the core domain, then the subdomain is (bizarrely named an @ sign). * means all other domains.
  2. TXT record with the id_rsa public key for mailgun.com
  3. MX records for mxa.mailgun.com and mxb.mailgun.com
  4. CNAME so email. points to mailgun.org

Then on mailgun.com add:

  1. Add a route that matches the forwarded address to the new gmail address
  2. In the Domain section create a new SMTP credential with a new login
  3. In gmail.com choose add a new SMTP account and add it.

Unfortunately when it starts.

  1. Mailgun.com doesn’t seem to like the SPF record that Network Solutions needs and does not match the SPF record.
  2. Gmail has bizarre problems (I’ve encountered these before), where you try to login into one account and sadly, it logs you into an entirely different account based on password. It does look as if Google accounts are password sensitive rather than account sensitive. Basically, make sure that every password on every google account is different, otherwise, you will try to login with account A and if  you type the password for account B, it will log you into accountB?!!#$#@
  3. And anyway I could not email directly into the new gmail account but it appears to take a while.
  4. Another strange gmail bug is that you can have two emails pointed to the same account and gmail doesn’t appear to know about them. So we have one name with a ‘.’ in it and another without it and it delivers to the same mailbox but the user interface doesn’t show two gmail names. Confused yet.

Finally there is a delay when you setup your new DNS records, so if you point away from your old mail server it will take time for the new one to show up. And you get after about 15 minutes:

  1. Delivery of your vanity mail via mailgun.com. As long as you get less than 10,000 mail/month this is free.
  2. Next up is testing the smtp side of it.

Well trying to get the Raspberry Pi back up and running and so many good things have happened in the last three years. Here’s the quickest way:

  1. Install latest hypriot from https://blog.hypriot.com/downloads. The latest is 0.7. The main thing I’m not sure of is how to update the latest version of hypriot. This version has all the Raspbian bits plus it knows how to speak docker! They have a very nice flash utility you can get with `git clone https://github.com/hypriot/flash` and you just point it with `flash –hostname rpi-0 https://blog.hypriot.com/downloads https://downloads.hypriot.com/hypriot-rpi-20160306-192317.img.zip` it works fine for rpi 1 and 2 but not for the new rpi 3
  2. Note that you can also build your own Hypriot which is really just raspbian with occidentalis for configuration of hostname and wifi plus the two packages below. You need to do this for rpi3 as Hypriot will not run. You should use raspbian lite for this. 
  3. Then make sure you have the latest versions of hypriot software with `sudo apt-get update && sudo apt get install hypriot-cluster-lab docker-hypriot docker-compose`
  4. To use hypriot in with docker-machine, change /etc/os-release from id=raspbian` to id =debian` so the standard docker-machine can use it with the generic driver. You can use a random Token, but much better is to use https://github.com/hypriot/cluster-lab so that with a single `cluster-lab start` you get consul running and can use port 8500 on any Raspberry pi to see what is running on the cluster.
  5. Also if you are using old Raspberry Pi 1, then you definitely want to modify /boot/config.txt to overclock to 1GHz as 700MHz is just on the edge of too slow. The new format of /boot/config.txt allows sections so you can have one config.txt that works for rpi1 and 2. 

In terms of usage what do you get?

  1. Docker at its latest version compiled for armhf in the hypriot repos
  2. Docker-compose the same way
  3. Consul so you can easily create swarms. If you then set DOCKER_HOST=rpi-0.local:2378 then all your docker commands get routed to the swarm. Then you can use constraints to run containers on different nodes.