Cries of the vanquished

The post at roguelazer's is so juicy from every side that I'd need to quote it whole to give it justice (h/t ~avg). But its ostensible meat is etcd.[1] In that, he's building a narative of the package being elegant at first, and bloating later.

This tool was originally written in 2013 for a ... project called CoreOS. ... etcd was greater than its original use-case. Etcd provided a convenient and simple set of primitives (set a key, get a key, set-only-if-unchanged, watch-for-changes) with a drop-dead simple HTTP API on top of them.

Kubernetes was quickly changed to use etcd as its state store. Thus began the rapid decline of etcd.

... a large number of Xooglers who decided to infect etcd with Google technologies .... Etcd's simple HTTP API was replaced by a "gRPC" version; the simple internal data model was replaced by a dense and non-orthogonal data model with different types for leases, locks, transactions, and plain-old-keys.

Completely omitted from this tale is that etcd was created as a clone of Google Chumby, which did not use HTTP. The HTTP interface was implemented in etcd for expediency. So, the nostalgic image of early etcd he's projecting is in fact a primitive early draft.

It's interesting that he only mentions leases and locks in passing, painting them as a late addition, whereas the concept of coarse locking was more important for Chumby than the registry.

[1] Other matters are taken upon in the footnotes, at length. You'd think that it would be a simple matter to create a seaprate post to decry the evils of HTTP/2, but not for this guy! I may write another entry on the evils of bloat and how sympathetic I am to his cause later.

Recruiter spam

Recruitment spam, like conference spam, is a boring part of life. However, it raises an eyebrow sometimes.

A few days ago, a Facebook recruiter, JP Fenn, sent me a form e-mail to an address that I do not give to anyone. It is only visible as a contact for one of my domains, because the registrar does not believe in privacy. I was pondering if I should propose to give him a consideration in exchange for the explanation of just where he obtained the address. Purely out of curiosity.

Today, an Amazon recruiter, Jonte, sent a message to an appropriate address. But he did it with addresses in the To: header, not just the envelope. He used a hosted Exchange of all things, and there were 294 addresses in total. That should give you an idea just how hard these people work to spam and at what level of being disposable I am in their eyes.

It really is pure spam. I think it's likely that JP bought or stole a spam database. He didn't write a Python script that scraped whois information.

I remember a viral story a few years ago how one guy got a message from Google recruiter that combined his LinkedIn interests in amusing ways. It went like "we like people whose strength is Talking Like A Pirate. As for Telling Strangers On The Internet They Were Wrong, that's one of my favorite pastimes as well." You know you made it when you receive that kind of attention. Maybe one day!

Seagate and SMR in 2020

Back in 2015, I wrote about Seagate Kinetic and its relation to shingles in Seagate product. Unfortunately, even if Kinetic were a success, it would only support a fraction of workloads. But the rest of Seagate customers demanded density increases. So, to nobody's surprise, Seagate started including shingles into their general purpose disk drives, perhaps only for a part of the surface, or coupled with a flash cache. The company was an enthusiastic early adopter of hybrid drives, as a vendor. Journalists are trying to make a story out of it, because caches are only caches, and once you started spilling, the drive slows down to the shingle speed. But naturally, Seagate neglected to mention in their documentation just how exactly their drive worked. Sacre bleu!

Another perspective on Swift versus Ceph today

Seen in e-mail today:

From: Mark Kirkwood

There are a number of considerations (disclaimer we run Ceph block and Swift object storage):

Purely on a level of simplicity, Swift is easier to set up.

However, if you are already using Ceph for block storage then it makes sense to keep using it for object too (since you are likely to be expert at Ceph at this point).

On the other hand, if you have multiple Ceph clusters and want a geo replicated object storage solution, then doing this with Swift is much easier than with Ceph (geo replicated RGW still looks to be real complex to set up - a long page of archane commands).

Finally (this is my 'big deal point'). I'd like my block and object storage to be completely independent - suppose a situation nukes my block storage (Ceph) - if my object storage is Swift then people's backups etc are still viable and when the Ceph cluster is rebuilt we can restore and continue. On the other hand If your object storage is Ceph too then....

regards

Mark

Mark's perspective is largely founded in the fault tolerance and administrative overhead. However, let's a look at "keep using [Ceph] for object too".

Indeed the integration of block, POSIX, and object storage is Ceph's strength, although I should note for the record that Ceph has a large gap: all 3 APIs live in separate namespaces. So, do not expect to be able to copy a disk snapshot through CephFS or RGW. Objects in each namespace are completely invisible to two others, and the only uniform access layer is RADOS. This is why, for instance, RGW-over-NFS exists. That's right, not CephFS, but NFS. You can mount RGW.

All attempts at this sort of integration that I know in Swift always start with a uniform access first. It the opposite of Ceph in a way. Because of that, these integrations typically access from the edge inside, like making a pool that a daemon fills/spills with Swift, and mounting that. SwiftStacks's ProxyFS is a little more native to Swift, but it starts off with a shared namespace too.

Previously: Swift is faster than any competitor, says an emploee of SwiftStack.

Nvidia acquires SwiftStack

In the words of Joe Arnold:

Last year, when we announced SwiftStack 7, we unveiled our focus on the SwiftStack Data Platform for AI, HPC, and accelerated computing. This included SwiftStack 1space as a valuable piece of the puzzle, enabling data acceleration in the core, at the edge, and in the cloud.

To our existing customers — we will continue to maintain, enhance, and support 1space, ProxyFS, Swift, and the Controller. SwiftStack’s technology is already a key part of NVIDIA’s GPU-powered AI infrastructure, and this acquisition will strengthen what we do for you.

Building AI supercomputers is exciting to the entire SwiftStack team. We couldn’t be more thrilled [...]

Highlighting 1space as the centerpiece of the acquisition seems strange. All I knew about it was a cloud-to-cloud data pumping service. Hardly any HPC stuff. I could see how Nvidia might want ProxyFS to replace Hadoop, but not this.

The core Swift continues unchanged for now.

Too Real

From CKS:

The first useful property Python has is that you can't misplace the source code for your deployed Python programs.

Nobody is Google

A little while ago I been to a talk by Michael Verhulst of Terminal Labs. He made a career of descending into Devops disaster zones and righting things up, and shared some observations. One of his aphorisms was:

Nobody Is Google — Not Even Google

If I understood him right, he meant to say that scalability costs money and businesses flounder on buying more scalability than they need. Or, people think they are Google, but they are not. Even inside Google, only a small fraction of services operate at Google scale (aka planetary scale).

Apparently this sort of thing happens quite often.

Fedora versus Lulzbot

I selected Lulzbot Mini as my 3D printer in large part because of the strong connection between the true open source and the company. It came with some baggage: not the cheapest, stuck in the world of 3mm filament, Cura generates mediocre support. But I could tolerate all that.

However, some things happened.

One, the maker of the printer, Alef Objects collapsed and sold itself to FAME 3D.

Two, Fedora shipped a completely, utterly busted Cura twice: both Fedora 30 and Fedora 31 came out with the package that just cannot be used.

I am getting a tiny bit concerned for the support for my Lulzbot going forward. Since printers expend things like cleaning strips, availability of parts is a big problem. And I cannot live with software being unusable. Last time it happened, Spot fixed the bug pretty quickly, in a matter of days. So, only a tiny bit concerned.

But that bit is enough to consider alternatives. Again, FLOSS-friendly is paramount. The rest is not that important, but I need it to work.

Update 2020/05/10: Fedora 32 ships with a Cura that passes a smoke test but is again unusable.

ARM Again in 2019 or 2020

I was at this topic since NetWinder was a thing, but yet again I started looking at having an ARM box in the house. Arnd provided me with the following tips, when I asked about CuBox:

2GB of RAM [in the best CuBox] is not great, but doesn't stop you from building kerrnels. In this case, the CPU is going to be the real bottleneck.

Of course it's not SBSA, since this is an embedded 32 bit SoC. Almost no 64 bit machines are SBSA either (most big server chips claim compliance but are lacking somewhat), but generally nobody actually cares.

HoneyComb LX2K is probably what you want if you can spare the change and wait for the last missing bug fixes.

The RK3399 chip is probably a good compromise between performance and cost, there are various boards with 4GB of RAM and an m.2 slot for storage, around $100.

Why does have to be like this? All I wanted was to make usbmon work on ARM.

The assertion that nobody cares about SBSA is rather interesting. Obviously, nobody in the embedded area does. They just fork Linux, clone a bootloader, flash it and ship, and then your refrigerator sends spam and your TV is used to attack your printer, while they move on the next IoT product. But I do care. I want to download Fedora and run it, like I can on x86. Is that too much to ask?

The HoneyComb LX2K is made by the same company as CuBox, but the naked board costs $750.