Well it certainly isn’t easy to upgrade one. But here are your options:

  1. The Mac Pro six core uses the Xeon E3-1650 Westmere running at 3.5ghz. So it is a very decent processor
  2. Unfortunately by default it ships with all memory slots full 4x4GB but at least they are using unbuffered ECC RAM DDR-1866. So to upgrade, you essentially have to start all over. There is a 4x32GB option that costs a whopping $1400. The reasonable OWS option 4×16 for $500. But note that this again fills all your slots. This is just standard RAM, so at Newegg, it is a little less. $112 for 2×8 Crucial, or $220 for 2×16 rDIMM (registered so about $450. Nice thing is that you can reuse the memory in say an 8-slot X99 board.
  3. The disk was an anemic 256GB and it is very expensive to replace as it is a custom 4x PCI Express 2.0 capable of 2GBps. OWC sells a 1TB replacement for $900. That compares with using Thunderbolt 2 which is running at 20Gbps on each connection and there are six of them on the back (although some are doubled). What that means is that you theoretically could put something running at 2GBps outside of the chassis which is much cheaper. For instance a Thunderbolt 2 adapter is $350 for the enclosure and that means you can swap your SSDs as needed. This is a good solution for storing data for sure.

Well I’m sure happy to have gotten rid of the $20/month charge for the rented Comcast router. We had it for a while because we had all this trouble with comcast, but thanks to the right email to the right person, they did a great job fixing it.

Now onto making things work better. It is an Arris TM822g and it isn’t clear how to work with it. Here are the steps:

  1.  dslreports.com explains that this modem lives on and you can access the page there.
  2. This will not work if you have a vpn configured as the access request goes out there. You can diagnose this by using the Mac tool traceroute

When you do get there, you get a pretty technical diagnostic page. The main thing to see are that you have 8 downstream channels. They should S/N of at least 36dB. And four upstream channels. Still looking for the rest of the meanings

I never thought I would see the day when 1gb Ethernet would be slow. But moving around 160GB training data sets isn’t fun.

So looks like it’s time to look at 1000baseT networking. Also we have quite a few Poe devices now and our old linksys sge2000 literally started crashing so time for a new switch set. Here are the considerations:

  1. I’ve been running at home an over provisioned netgear 48 port 1gb Poe switch and it has been great. It was expensive seven years ago at $1000 or so but definitely reliable. It’s worked well with link aggregation too.
  2. We have been using unmanaged d-link 24 ports, 8 port d-link and 8- Port net gear Poe abs these have been fine.
  3. Cisco bought linksys and they gave a small business line.
  4. There are not a huge number of reviews of this stuff.
  5. The product lines are huge and confusing with Netgear for instance having 8+ models of 48 port
  6. Stacking and SFp are inter switch communication schemes but not super relevant.
  7. Cat 6a works to 500 MHz and up to 100m at 10Gb although power consumption is high.
  8. Poe has been extended to Poe+ with up to 25 watts over the wire.
  9. 10Gbps adapters are about $2-300 and need typically 4x of pci lanes. And switches are $100 a port vs $20 for managed 1gb or $5-10 for Unmanaged.
  10. We just had reliability problems and lost two switches. I’m again reminded that the network is one thing that you don’t want to have unreliable. So that’s a key thing.

So here are recommendations:

Main SMB switch

There are so many different switches the main features are a) managed, b) link aggregation, c) POE. Netgear and Cisco seem like the two reasonable vendors with for me a hat tip to staying in the mid range of Cisco. The 48 port POE seems like a sweet spot in terms of density and cost is not too high to have two of them. In our shop, we could also get away with a pair of 24 ports, but 48 ports is very convenient and dense.

  • Cisco. They have a host of similar models, but the $650 SG200-P has 24 ports with POE and24 ports of non POE so the SG-200FP is more convenient with all ports POE and also POE+ but it is $900 (aka SLM2048-PTNAM). Naming-wise, Cisco Small Business is really Linksys vs the real Cisco enterprise gear. The SG models are all old Linksys. The SG-200 is fanless and level 2 only while SG-300 can be level 3. They also have SG-300 and SG-500 lines as well. I don’t think we will ever to large enough to need a level 3 switch (eg router between different LAN segments 🙂 Or need stacked switches with LAG groups for failover, but that is pretty cool stuff! It is stackable
  • Netgear makes my head spin, they have Fully Managed, Smart Managed and Web Managed huh? Web managed are the smaller switches with a web interface and no POE. The Smart Managed has a few subfamilies so there are non stackable the GS748Tv5 which is no POE and the GS752TP which has POE+. Then the stackable and that corresponds to the GS752TSB and GS752TPSB so this means S for stackable and P for POE and there is -100NAS which indicates for north america. Finally B means revision B (not compatible with the earlier non-B models so you can’t stack them). Finally there is 10Gbps uplink models called the GS752TXS but doesn’t have POE.

After looking at all of this, the two models that make the most sense on the short list are so now let’s look at reliability and here is the data in order of best reviews and most quantity. So the hard part here is that there isn’t much user data except that the very inexpensive Netgear GS752TP is probably not super reliable. Sigh.

  • Cisco SG-200-P. $614 but only half the ports are POE. Main thing is that the Amazon reviews are pretty incredible but it is hard to say because most are reviews of other products with fewer ports. Newegg ($650) is 3/5 with the big complaints being the 24/24 split and there was one DOA. Not too many reviews.
  • Netgear GS752TP. $670 POE+, stackable and cheap. But Newegg reviews are 3/5 (only 11 reviews though) but complaints about web user interface bugginess and not working on a Mac plus power and hardware failures. On Amazon, it is 4.5/5 but Amazon aggregates everything together so hard to tell for just this switch. Main complaints are DOA units, bad POE, poor customer support. It theoretically has a lifetime warranty, but good luck getting that honored. It has 4 uplink 1Gb SFP ports but isn’t stackable so you can’t make fault tolerant.
  • Netgear GS752TPSB. $880 POE+ and stackable. Only two Newegg reviews, but no problems. Seven Amazon reviews which isn’t a lot. But at least no big problems except that it isn’t compatible with its older the GS752TPS it replaced.
  • Cisco SG-200-FP. $900 and all the ports are POE but only one review across Amazon and Newegg 🙁

It also turns out we have a GS516TP running POE and a GS748TP (the four year old sister of the GS752TP) that has been running everything else. I didn’t even realize it. And I won’t tell you the last time I upgraded the firmware (otoh our problems with the Linksys SGE2000 and SGE2000p seemed to start with the firmware upgrade).

10Gbps Ethernet

This is a toe dip as they are expensive. Netgear has a non-link aggregation model at $100 a port so not a bad place to start. Aggregating multiple links doesn’t see likely needed anytime soon.

  1. Netgear 8 port 10Gbps hub. Again Netgear seems to be the leader with a $800 8 port.The main limitation is not stackable, POE nor does it have LACP.
  2. Startech 10Gbps nic. Either get board with 10Gbps as the $600 ASUS has or get a $240 Startech adapter.

I’ve always used core i5 k processors for my builds. With most gaming an overclocked core i5 at 4.5ghz for $220 is all you need. And the other features don’t matter. 

The lone exception is a cheap Xeon file server where based on vlads advice we went with a six core slow Xeon for ecc and lots of lanes. 

But now that we are working in high scale Gou farms, things have changed. We got two core i7. One a $350 skylake k and the other a $600 haswell e and they were both mistakes. Here’s some better advice:

  1. The x99 motherboards can take ecc ram. They don’t say it and they max out with eight slots (16GB X 8 = 128GB) but they will work with ecc. At least the Asus board that Pugetsystems uses. So I’m buying ecc for our x99 2011-3 we have. 
  2. Xeons have the most confusing product names. Ok the i5 etc are as bad (atom, celeron, i3, i5, i7). Most are locked but one flavor. The older haswell-ws 16xx v3 is multipier unlocked so you can get to 4.5 pretty easily. Uses the x99. These are hot chips running at 140 watts. It is sad that workstations servers are two generations behind with no 2011 Broadwell or Skylake parts. Less competition and power is less of an issue.  Note that the other xeons are locked and you have to use a desktop chipset x99. But for lots of pci lanes, ecc it’s hard to beat. The three flavors are 1650 v3 with six cores. 1630 v3 and 1620 v3 with four. 
  3. X99. You want an eatx board so you can get four Gpus The asrock x99 formula 3.1 only has 4 double wide  but the USB 3.1 takes a slot. 
  4. The asrock does have two m.2 slots. It’s confusing but m.2 has lots of flavors. The ultra m.2 is 4x pcie so can be up to 4GBps potential and 2.2GBps actual and is m-keyed and with NVMe with Samsung 950 or the sm951 , then the next best is 4x pcie with the Samsung xp941 ( but is only $10 cheaper than the sm951,so get that instead), then for 2x pcie 2.0 the plextor m6e at 1GBps or 600MBps effective (again this drive is the same price as the 950 pro) and finally the lowly Sata 6G at 600MBps max or 400MBps typical with the Sandisk extreme pro being the fastest

Given this what would a dream build be for this ultimate workstation:
PCPartPicker part list / Price breakdown by merchant

Type Item Price
CPU Intel Xeon E5-1620 V3 3.5GHz Quad-Core Processor $288.99 @ SuperBiiz
CPU Cooler Thermalright SILVER ARROW IB-E 73.6 CFM CPU Cooler $79.95 @ Amazon
Motherboard ASRock X99 OC Formula/3.1 EATX LGA2011-3 Motherboard $303.98 @ Newegg
Memory Kingston 64GB (4 x 16GB) Registered DDR4-2133 Memory $359.99 @ SuperBiiz
Storage Samsung 950 PRO 512GB M.2-2280 Solid State Drive $325.99 @ SuperBiiz
Storage Mushkin Reactor 1TB 2.5″ Solid State Drive $229.89 @ OutletPC
Storage Mushkin Reactor 1TB 2.5″ Solid State Drive $229.89 @ OutletPC
Video Card Asus GeForce GTX 980 Ti 6GB STRIX Video Card (4-Way SLI) $666.99 @ SuperBiiz
Video Card Asus GeForce GTX 980 Ti 6GB STRIX Video Card (4-Way SLI) $666.99 @ SuperBiiz
Video Card Asus GeForce GTX 980 Ti 6GB STRIX Video Card (4-Way SLI) $666.99 @ SuperBiiz
Video Card Asus GeForce GTX 980 Ti 6GB STRIX Video Card (4-Way SLI) $666.99 @ SuperBiiz
Case Phanteks Enthoo Pro ATX Full Tower Case $99.99 @ Amazon
Power Supply Corsair AX1500i 1500W 80+ Titanium Certified Fully-Modular ATX Power Supply $399.05 @ Amazon
Prices include shipping, taxes, rebates, and discounts
Total (before mail-in rebates) $5005.68
Mail-in rebates -$20.00
Total $4985.68
Generated by PCPartPicker 2016-03-06 11:59 EST-0500

Well this is supposed to be the most powerful setup box made, but argh, is it hard to setup. 

It requires that you logon to Google and it is really pleasant, asks you to use your phone or your laptop. Here is what I found:

  1. The http://g.co/AndroidTV is not compatible with either iPhone or Safari. It displays a partial screen. I get it that Google doesn’t like Apple, but really. So I fired up Firefox and that seemed to work. However, it tries to figure out what network address the nVidia device lives on and at least on my laptop, it decided on some very strange network (192.168.x.x) when we are using a different network. You get an error message saying, please connect to network Xo whatever that means.
  2. If you look at the url it is using, you can then see it is trying to find a port on that random network. if you hand change it to the actual ip address of the ShieldTV then it says OK please continue. However, the next screen also has the incorrect embedded URL and so that fails.

  3. Now trying to get the Shield TV to enter from the keyboard. I gave up, it is so hard to type on that controller (and my password is kind of long :-). There are no instructions, but you can plug in a USB keyboard and type it in. However if you have a two factor authentication, then you get to a screen that is googles, but there is no way to scroll down other than to use the controller. Man, this feels so unpolished.

  4. Now you get to wait for a 850MB software download. Wow a brand new device and you need that much? Then you get another update for the Shield Experience.

  5. Finally it has agreement screens for both Google and for nVidia. And no Google, I’m not going to allow my location to be used no matter how many times you ask (they ask twice).

  6. You then get to a screen that shows the controller and there doesn’t seem to be anyway to hit the next button. It doesn’t tell you that the A button means next. It should be obvious to gamers, but not super clear and no audio feedback either.

  7. With the controller there are no instructions on how to do menu and forward and back, you just get dumped into the main screen. After some button mashing by the way, A means select and B is back. 

  8. They have Kodi for reading from your disk (yeah, I can get my Mac Mini back), Netflix, but no Amazon. So there is still no way to get it all until I try the Amazon Fire anyway. I don’t have the latest Apple TV either. 

I tried to google “nVidia Shield TV setup” but didn’t get much, so I guess I’m an early adopter. The net is that it is a hard device to setup,but once you do u get a pretty standard interface.

  1. There is kind of a merge of android games with a controller. I’ve never liked fps games in touchscreens. Too hard to control but they appear to allow controllers and maybe even a mouse. 

  2. Kodi works exactly like the Mac version so we can use our file server. A big advantage over the AppleTV. 

  3. No Anazon video I could find. 🙁 but of course Netflix. 

  4. They have streaming real games from Nvidia. I thought they allowed streaming from PCs. 

  5. 4K support so I need a 4K monitor. 

  6. There is a project to reboot it under Debian and use docker. 🙂

I think I’m a little scared I have over the years I’ve accumulated all this gear and it is the same as the wirecutters recomendations 

  1. Lowepro 350 aw. Nearly the perfect stealth bag for camera equipment. I’m on my third one. Great protection and even carries a 15 laptop. The latest is all weather with a built in rain cover
  2. Top peak messenger bag. Retro cool and I backed on Kickstarter too!
  3. Optech sling. Incredibly durable
  4. Skooba gear holder. Nice but doesn’t easily fit in 350

Ok so now that we are getting to just internet here are the tips tricks and traps

  1. Wow Comcast customer service has really improved. Actual smiling people. And retail locations that’s great.
  2. The main optimizations are getting rid of television. If you love sports this is hard but otherwise it is quite survivable. You can use Kodi on a computer (even works on a raspberry Pi or a set top box) to view your old movies stored on disk. Then Netflix and Amazon are basically everywhere. The most universal box seems like it is the nVidia shield. 4K and $200. Of course the AppleTV is the simplest but Amazon us a fight with them. 
  3. Then for phone we’ve had good luck with Obihai. It is compatible with old alarm systems which ooma is not. And it routs through google voice. (So watch what you are saying!). But it has no recurring charge. 
  4. The cable modem is really important now so get a docsis 3.0 with at least 8 streams (400Mbps). The expensive docsis 3.1 are coming out that go to 1.6Gbps which will cost $$$s but cool to have. Motorola surfboard is an incredibly cheap $50 and saves $10 a month

So how to make it all work. Rather than wait for my new hardware, I thought I might as well start with the hundred or so Raspberry Pi Model Bs we already have. Here’s a recipe for how to get it all running:

Hypriot: The Operating system

Hypriot are a great bunch and they have integrated Raspbian with docker installation. Their latest working system is something called Hector using Docker 0.9. It works great on our old Raspberry Pi Model Bs.

Note. is that docker 1.1 does not run on ARMv6, so you need to stay on docker 0.9 for now.

They have in development their next release. A big change from the previous Hypriot Cluster/Hector I used. You can download it yourself and then run a dd if=foo.img of=/dev/rdisk bs=1M Make sure to edit the occidentalist.txt file to make a different hostname for each raspberry pi sd card. Or you can use their flash utility. Right now that thing see to have a bug. Mac’s want bs=1M and not a lower case m.

The future. There you had to carefully name all the SDs differently. It is super easy to download, just get their flash utility and point it at a url which today is flash https://github.com/hypriot/image-builder-rpi/releases/download/v0.2.5/sd-card-rpi-v0.2.5.img.zip but of course that has lots of issues, right now they recommend staying on docker 1.9 and their older release of cluster lab.


This isn’t so much of an issue for the newer Raspberry Pi 2, but the Model B only runs at 700Mhz, so an overclock to 1GHz is super helpful and a real speed difference. The parameters are pretty easy in the config


Camera support

One details is that hypriot does not by default enable the camera or ship with raspi-config, so you have to manually change the /boot/config.txt image. Some important options are gpu_mem if you are running gstreamer and start_x=1 means turn the camera on.

There doesn’t seem much interest, but there are three possible containers I found that seem to try to get the raspberry pi camera working in a container:


He has the best documentation but didn’t publish the docker file however it does seem to work! We start it on a particular node. At first it didn’t like the node constraints, but this seems like some sort of glitch with consul.

Now to test run:

# assuming rpi-0 is the master and the cameras are rpi-{1-4}
docker run -p 8080:8080 --privileged --env="constraint:node=rpi-1" --name=webcam-1 nieleyde/rpi-webcam

Now try the to open http://rpi-1.local:8080 and you should see a basic status json. and http://localhost:8080/interface will give you the front screen and it will start to stream video. The light should go on after about 20 seconds and you should see the image. It will update about once every few seconds.

One interesting bug is that if you try to run raspistill while another camera app is running  it will hang the GPU. Even if you try a reboot it doesn’t seem to help so you should instead power cycle. This still hasn’t worked, so it could be a hardware problem. Still it is working on 3 of 4 nodes.

Here are the other things to try.

  • arijusg/rpi-camera. This has a Dockerfile so we can at least see how he compiles and gets raspistill to work and there is a comment on the github which shows what else needs to get added to the container to get raspistill up and it looks like the main problem is that /opt/vc/bin isn’t in the path or linked into /usr/bin.
  • monsendag/rpi-cameraserver. Haven’t tried this one yet.
  • beck/rpi-webcam. I can’t tell but could be a pull from neileyde

Building from source

Well the basic conclusion is that these libraries basically do not work. Not enough effort on them. So I guess I’m back to figuring out how to build it from source. There are lots of guides that cover this but thinkrpi seems the most complete. I’m going to just compile on the Raspberry Pi for now as cross compiling is definitely complicated and I never really understood QEMU etc.

  1. You need to git clone raspberry-pi/userland and compile it
  2. This leaves raspistill and raspivid in /opt/vc/bin which needs to go into your path. Raspbian by the way has a symbolic link from /usr/bin for raspistill,vid,iyuv, etc.

Consul vs Kubernetes: Orchestration

You need this layer to make it easy to start and stop jobs on the system. Specifically it would be nice to have something that runs a job per actual physical node and tells you want is up or down.

Consul via Hypriot Cluster Lab. Their cluster lab is a bit old, but integrated consul with docker swarm so you can do cluster assignment with a simple constraint. Note that when you do a docker pull, it will actually download the same image across the entire cluster. So longer term, you probably want the things to nfs boot so you just have one image for all of them. You have two graphical views as well. Consul shows what nodes are running on port 8500. You can also run hypriot/rpi-dockerui to see graphically what containers you have up on port 9000.

docker pull arijusg/rpi-camera
docker run --env="constraint=node==rpi-0" hypriot/armhf-run-hello-world

Wow what a great day for low cost processors. The conversion to 64 bit is on and that means much simpler software development. We are moving away from the strange hybrid architecture of the original Raspberry Pi (ARMv6 with hardware floating point) to 64-bit and fast GPUs. Also the busses are getting better so some of these boards have real SATA getting around the reliability problems of SDs and also fast gigabit ethernet that doesn’t share bandwidth with the USB. So here are requirements for building a dockerized gigapixel camera.

We have come a long way in the last four years with the latest ARM machines 10x faster than the original Raspberry Pi.

  • Wide support so I’m not in the woods but need a stack with modern docker-able system like debian jessie or Android.
  • 5MP camera. Ideally accelerated on bus. Note I can’t find any USB cameras with this specification. Ideally 8MP or more (4K implies 8MP)
  • 1080p video minimum. Although 4K video would be better in some ways this is very stressful to the processor.
  • Gigabit ethernet. As getting photos off the camera are not easy and with 4K/8MP forget about it.
  • eMMC or SATA since SDs are not super reliable for writes.
  • 64-bit for performance and easier software
  • 1GB ram. But 2GB would be better
  • Less than $100 for per board so getting to a gigapixel isn’t ridiculous (5MP x 200 = 1GPixel or 8MP x 125)
  • GPU acceleration of traditional software like gstreamer.

Finally the folks hypriot are making it easier than ever to develop for this with their baked in docker images. And with Kubernetes and other systems there is a way to control taking photos and videos.

And as yet another dark horse, as the prices for Intel based processor fall, it is always tempting just to use a low cost Intel board given the ease of software development and availability of precompiled binaries.

Here are some great choices now as explained by cmx-software here are the differences for ARM. The move to Arm v8 64-bit is a big deal because these processors are 20-30% faster than similarly clocked Arm v7 32-bit ones.

  1. Raspberry Pi 3 goes to 64 bits but only 32 bit images for now. They have the same camera system and thus raspistill will work. They have stuck with Mali saying that it has open source drivers. The big negatives are that they do not have gigabit ethernet and the fast ethernet is still stuck on the USB bridge. Although they do have onboard Wifi and BLE (so better for remote applications). Also it only have 1GB  of ram (wait did I just say just?). Also no Android support.
  2. Odroid-XU4. This is 32 bit but it has all the other features including Hypriot support a 5Mp camera that runs on USB 3.0 plus an Octacore and eMMC support. So we really need an XU4 ($75) with the C2 ($40) core. Or perhaps use this until that happens.
  3. Odroid-C2. It has real gigabit ethernet, is 64 bit with Android and Debian. However, it does not have a camera on it. So it is best for compute intensive tasks or for media output. It also runs at 2GHz and not 1GHz like the Pi. It also does 4K with HDMI 2.0 so a nice display machine. It also uses eMMC which should be more reliable than SD but not as fast as SSDs. It has an active development community
  4. Pine A64+. This has not yet shipped but uses a 1.2GHz quad core and has up to 2GB of memory. It does have a camera interface and gigabit ethernet, but no SSD or eMMC.  And it also has a wifi/bluetooth module available. All told except for lack of reliable storage, it has pretty good specs for a high performance camera. It does run Android, but not clear if it can run gstreamer or how it handles OpenGL/opencl processing.
  5. Congatec qa4. This is the most interesting new board. it is supposed to be much lower cost using Intel Braswell. It is full 64-bit quad core and can support up to 8GB memory and SATA drives plus it has a CSI camera interface supporting up to 2 cameras! And it is Intel so cross compilation  problems are smaller and it might even accelerate in the Intel graphics. Main problem is how to get it and I’m sure it has a tiny development community.

Then there are some other choices that seem to have less support but interesting options

  1. Banana BP-M2+. It also uses the Allwin quad-core with 1GB ram and 8GB eMMC and has gigabit ethernet. And a CSI camera interface so the specs are attractive but the development community is small. However except for the memory, it’s a very attractive board particularly if armbian can support the camera although it is not 64-bit.

Finally, with Intel pushing down with Atom and Celeron, another option is to just stay all Intel. Linuxium keeps track. All of these will be USB cameras to work however and they are around $150-200 per machine. So that’s an expensive megapixel machine because webcams are about $50 and are 2MP typically in real resolution so you’d need 500 of these things!

  1. Winter Pro CX-W8. At the other end of the spectrum, Intel has kind of come down to the right price, so you could also stay all Intel. It is 64-bit and runs Android or Windows with 32GB eMMC but only 100Mbps Ethernet. For a camera, you’d have to add a USB web cam, but you have a good shot at running full gstreamer.
  2. Onda M2. If you go all the way up, a true Pentium with 4GB ram 128MB SSD is now just $189.

OK more than a day, but there was a bad update where Apple accidentally blacklisted their own driver (ugh, I’m sure someone is getting yelled at right now). It should be fixed automatically if you are using the automatic update feature. Mine was broken at on Friday and fixed Saturday at 530PM, so software update is good. 

You can kick off manual install but I had not realized you can check things by holding down the option key and clicking on the Apple menu and go to Software and look at the versions installed. Should be higher than 3.28.1 for things to work again. 

Of course murphies law had me checking wired ethernet last night right in the window, mystery solved!