Found an odd job posting at the website of Rancher:

What you will be doing

• Design and implement a block storage solution for Docker containers
• Working on development of various aspects of the storage stack: consistency, reliability, replication and performance
• Using Go for product development

Okay. Since they talk about consistency and replication together, this thing probably provides actual service, in addition to the necessary orchestration. Kind of the ill-fated Sheepdog. They may under-estimate the amount of work necesary, sure. Look no further than Ceph RBD. Remember how much work it took for a genius like Sage? But a certain arrogance is essential in a start-up, and Rancher only employs 150 people.

Also, nobody is dumb enough to write orchestration in Go, right? So this probably is not just a layer on top of Ceph or whatever.

Well, it's still possible that it's merely an in-house equivalent of OpenStack Cinder, and they want it in Go because they are a Go house and if you have a hammer everything looks like a nail.

Either way, here's the main question: what does block storage have to do with Docker?

Docker, as far as I know, is a container runtime. And containers do not consume block storage. They plug into a Linux kernel that presents POSIX to them, only namespaced. Granted, certain applications consume block storage through Linux, that is why we have O_DIRECT. But to roll out a whole service like this just for those rare appliations... I don't think so.

Why would anyone want block storage for (Docker) containers? Isn't it absurd? What am I missing and what is Rancher up to?

UPDATE:

The key to remember here is that while running containers aren't using block storage, Docker containers are distributed as disk images, and they get a their own root filesystem by default. Therefore, any time anyone adds a Docker container, they have to allocate a block device and dump the application image into it. So, yes, it is some kind of Docker Cinder they are trying to build.

See Red Hat docs about managing Docker block storage in Atomic Host (h/t penguin42).

Anyone who ever worked with object storage knows that PUT creates, GET reads, POST updates, and DELETE deletes. Naturally, right? POST is such a strange verb with oddball encodings that it's perfect to update, while GET and PUT are matching twins like read(2) and write(2). Imagine my surprise, then, when I found that the official definition of RESTful makes POST create objects and PUT update them. There is even a FAQ, which uses sophistry and appeals to the authority of RFCs in order to justify this.

So, in the world of RESTful solipcism, you would upload an object foo into a bucket buk by issuing "POST /buk?obj=foo" [1], while "PUT /buk/foo" applies to pre-existing resources. Although, they had to admit that RFC-2616 assumes that PUT creates.

All this goes to show, too much dogma is not good for you.

[1] It's worse, actually. They want you to do "POST /buk", and receive a resource ID, generated by the server, and use that ID to refer to the resource.

View the entire thread this comment is a part of

Or so they say, at least for certain workloads.

In January of 2015 I led a project to evaluate and select a next-generation storage platform that would serve as the central storage (sometimes referred to as an active archive or tier 2) for all workflows. We identified the following features as being key to the success of the platform:

• Utilization of erasure coding for data/failure protection (no RAID!)
• Open hardware and the ability to mix and match hardware (a.k.a. support heterogeneous environments)
• Open source core (preferred, but not required)
• Self-healing in response to failures (no manual processes required, like replacing a drive)
• Expandable online to exabyte-scale (no downtime for expansions or upgrades)
• High availability / fault tolerance (no single point of failure)
• Enterprise-grade support (24/7/365)
• Visibility (dashboards to depict load, errors, etc.)
• RESTful API access (S3/Swift)
• SMB/NFS access to the same data (preferred, but not required)

In hindsight, I wish we would have included two additional requirements:

• Transparently tier and migrate data to and from public cloud storage
• Span multiple geographic regions while maintaining a single global namespace

We spent the next ~1.5 years evaluating the following systems:

• SwiftStack
• Ceph (InkTank/RedHat/Canonical)
• Scality
• Cloudian
• Caringo
• Dell/EMC ECS
• Cleversafe / IBM COS
• HGST/WD ActiveScale
• NetApp StorageGRID
• Nexenta
• Qumulo
• Quantum Lattus
• Quobyte
• Hedvig
• QFS (Quantcast File System)
• AWS S3
• Sohonet FileStore

SwiftStack was the only solution that literally checked every box on our list of desired features, but that’s not the only reason we selected it over the competition.

The top three reasons behind our selection of SwiftStack were as follows:

• Speed – SwiftStack was—by far—the highest-performing object storage platform — capable of line speed and 2-4x faster than competitors. The ability to move assets between our “tier 1 NAS” and “tier 2 object” with extremely high throughput was paramount to the success of the architecture.
• [...]

Note: While SwiftStack 1space was not a part of the SwiftStack platform at the time of our evaluation and purchase, it would have been an additional deciding factor in favor of SwiftStack if it had been.

Interesting. It should be noted that performance of Swift is a great match for some workloads, but not for others. In particluar, Swift is weak on small-file workloads, such as Gnocchi, which writes a ton of 16-byte objects again and again. The overhead is a killer there, and not just on the wire: Swift has to update its accounting databases each and every time a write is done, so that "swift stat" shows things like quotas. Swift is also not particularly good at HPC-style workloads, which benefit from a great bisectional bandwidth, because we transfer all user data through so-called "proxy" servers. Unlike e.g. Ceph, Swift keeps the cluster topology hidden from the client, while a Ceph client actually tracks the ring changes, placement groups and their leaders, etc.. But as we can see, once the object sizes start climbing and the number of clients increases, Swift rapidly approaches the wire speed.

I cannot help noticing that the architecture in question has a front-facing cache of pool (tier 1), which is what the ultimate clients see instead of Swift. Most of the time, Swift is selected for its ability to serve tens of thousands of clients simultaneously, but not in this case. Apparently, the end-user invented ProxyFS independently.

There's no mention of Red Hat selling Swift in the post. Either it was not part of the evaluation at all, or the author forgot about it for the passing of time. He did list a bunch of rather weird and obscure storage solutions though.

As mentioned previously, I run a personal Fediverse instance with Pleroma, which uses Postgres. On Fedora, of course. So, a week ago, I went to do the usual "dnf distro-sync --releasever=30". And then, Postgres fails to start, because the database uses the previous format, 10, and the packages in F30 require format 11. Apparently, I was supposed to dump the database with pg_dumpall, upgrade, then restore. But now that I have binaries that refuse to read the old format, dumping is impossible. Wow.

A little web searching found an upgrader that works across formats (dnf install postgresql-upgrade; postgresql-setup --upgrade). But that one also copies the database, like a dump-restore procedure would. What if the database is too large for this? Am I the only one who finds these practices unacceptable?

Postgres was supposed to be a solid big brother to a boisterous but unreliable upstart MySQL, kind of like Postfix and Exim. But this is just such an absurd fault, it makes me think that I'm missing something essential.

UPDATE: Kaz commented that a form of -compat is conventional:

When I've upgraded in the past, Ubuntu has always just installed the new version of postgres alongside the old one, to allow me to manually export and reimport at my leisure, then remove the old version afterward. Because both are installed, you could pipe the output of one dumpall to the psql command on the other database and the size doesn't matter. The apps continue to point at their old version until I redirect them.

Yeah, as much as I can tell, Fedora does not package anything like that.

With the recent move by Google to disable the ad-blockers in Chrome (except for Enterprise level customers[1]), the interest is sure to increase for methods of protection against the ad-delivered malware, other than browser plug-ins. I'm sure Barracuda will make some coin if it's still around. And on the free software side, someone is making an all-in-one package for Raspberry Pi, called "Pi-hole". It works by screwing with DNS, which is actually an impressive demonstration of what an attack on DNS can do.

An obvious problem with Pi-hole is what happens to laptops when they are outside of the home site protection. I suppose one could devise a clone of Pi-hole that plugs into the dnsmasq. Every Fedora system runs one, because NM needs it in order to support the correct lookup on VPNs {Update: see below}. The most valuable part of Pi-hole is the blocklist, the rest is just scripting.

[1] "Google’s Enterprise ad-blocking exception doesn’t seem to include G Suite’s low and mid-tier subscribers. G Suite Basic is $6 per user per month and G Suite Business is $12 per user month."

UPDATE: Ouch. A link by Roy Schestovitz made me remember how it actually worked, and I was wrong above: NM does not run dnsmasq by default. It only has a capability to do so, if you want DNS lookup on VPNs work correctly. So, every user of VPN enables "dns=dnsmasq" in NM. But it is not the default.

UPDATE: A reader mentions that he was rooted by ads served by Space.com. Only 1 degree of separation (beyond Windows in my family).

Seen an amusing blog post today on the topic of the hideous debacle that is Google Fi (on top of being a virtual network). Here's the best part though:

About a year ago I tried to get my parents to switch from AT&T to Google Fi. I even made a spreadsheet for my dad (who likes those sorts of things) about how much money he could save. He wasn’t interested. His one point was that at anytime he can go in and get help from an AT&T rep. I kept asking “Who cares? Why would you ever need that?”. Now I know. He was paying almost $60 a month premium for the opportunity to able to talk to a real person, face-to-face! I would gladly pay that now.

Respect your elders!

Seen in a blog entry by Martin Tournoij (via):

I’ve been happily programming Python for over a decade, so I’m used to significant whitespace, but sometimes I’m still struggling with YAML. In Python the drawbacks and loss of clarity are contained by not having functions that are several pages long, but data or configuration files have no such natural limits to their length.

[...]

YAML may seem ‘simple’ and ‘obvious’ when glancing at a basic example, but turns out it’s not. The YAML spec is 23,449 words; for comparison, TOML is 3,339 words, JSON is 1,969 words, and XML is 20,603 words.

There's more where the above came from. In particular, the portability issues are rather surprising.

Unfortunately for me, OpenStack TripleO is based on YAML.

Wow, they better not be building Boeings from this crap:

NASA Launch Services Program (LSP) investigators have determined the technical root cause for the Taurus XL launch failures of NASA’s Orbiting Carbon Observatory (OCO) and Glory missions in 2009 and 2011, respectively: faulty materials provided by aluminum manufacturer, Sapa Profiles (SPI). LSP’s technical investigation led to the involvement of NASA’s Office of the Inspector General and the U.S. Department of Justice (DOJ). DOJ’s efforts, recently made public, resulted in the resolution of criminal charges and alleged civil claims against SPI, and its agreement to pay $46 million to the U.S. government and other commercial customers. This relates to a 19-year scheme that included falsifying thousands of certifications for aluminum extrusions to hundreds of customers.

BTW, those costly failures probably hastened the sale of Orbital to ATK in 2015. There were repercussions for the personnell running the Taurus program as well.

On their official blog:

Over the last several months, SwiftStack has been busy helping two large autonomous vehicle customers. These data pipelines are distributed across edge (vehicle sensors) to core (data center) to cloud/multi-cloud locations, and are challenged with ingest, labeling, training, inferencing, and retaining data at scale. [...] one deployment is handling more than a petabyte of data per week, with four thousand GPU cores from NVIDIA DGX-1 servers fed with 100 GB/s of throughput from SwiftStack cluster.

I suspect the task-queue expirer could be helpful at this. Although, if you're uploading 1 PB per week, it takes about a year to fill out a cluster as big as Turkcell's.

Apparently the actual storage is provided by Cisco UCS S3260. Some of our customers use Cisco UCS to run Swift too. I always thought about Cisco as a networking company, but it's different nowadays.

I knew about that thing because Rich Jones was a fan. Man, that guy is always ahead of the curve.

Coincidentially, a couple of days ago Amazon announced support for RISC-V in FreeRTOS (I have no idea how free that thig is. It's MIT license, but with Amazon, it might be patented up the gills.).

Okay, here's the breakdown:

@pro: Programming, computers, networking, maybe some technical fields. It's basically migrated from SeaLion and is the main account of interest for the readers of this journal.

@stuff: Pictures of butterflies, gardening, and general banality.

@gat: Boomsticks.

@avia: Flying.

@union: Politics.

@anime: Anime, manga, and weaboo. Note that Ani-nouto is still officially at Smug.

Thinking about adding @cars and @space, if needed.

You can subscribe from any Fediverse instance, just hit the "Remote follow" button.

In a Swift numbers post in 2017, I mentioned that the largest known cluster had about 20 PB. It is 2019 now and I just got a word that TurkCell is operating a cluster with 36 PB, and they are looking at growing it up to 50 PB by the end of the year. The information about its make-up is proprietary, unfortunately. The cluster was started in Icehouse release, so I'm sure there was a lot of churn and legacy, like 250 GB drives and RHEL 6.

TFW you consider an O'Reily animal-cover book and the blurb says:

Authors Simon St. Laurent and J. David Eisenberg show you how Elixir combines the robust functional programming of Erlang with an approach that looks more like Ruby, and includes powerful macro features for metaprogramming.

In the past several days, I innaugurated a private Fediverse instance, "Mu", running Pleroma for now. Although Mastodon is the dominant implementation, Pleroma is far easier to install, and uses less memory on small, private instances. By doing this, I'm bucking the trend of people hating to run their own infrastructure. Well, I do run my own e-mail service, so, what the heck, might as well join the Fediverse.

So far, it was pretty fun, but Pleroma has problem spots. For example, Pleroma has a concept of "local accounts" and "remote accounts": local ones are normal, into which users log in at the instance, and remote ones mirror accounts on other instances. This way, if users Alice@Mu and Bob@Mu follow user zaitcev@SLC, Mu creates a "remote" account UnIqUeStRiNg@Mu, which tracks zaitcev@SLC, so Alice and Bob subscribe to it locally. This permits to send zaitcev's updates over the network only once. Makes sense, right? Well... I have a "stuck" remote account now at Mu, let's call it Xprime@Mu and posit that it follows X@SPC. Updates posted by X@SPC are reflected in Xprime@Mu, but if Alice@Mu tries to follow X@SPC, she does not see updates that Xprime@Mu receives (the updates are not reflected in Alice's friends/main timeline) [1]. I asked at #pleroma about it, but all they could suggest was to try and resubscribe. I think I need to unsubscribe and purge Xprime@Mu somehow. Then, when Alice resubscribes, Pleroma will re-create a remote, say Xbis@Mu, and things hopefully ought to work. Well, maybe. I need to examine the source to be sure.

Unfortnately, aside from being somewhat complex by its nature, Pleroma is written in Elixir, which is to Erlang what Kotlin is to Java, I gather. Lain explains it thus:

As I had written a social network in Ruby for my work at around that time, I wanted to apply my [negative] experience to a new project. [...] This was also to get some experience with Elixir and the Erlang ecosystem, which seemed like a great fit for a fediverse server — and I think it is.

and to re-iterate:

When I started writing Pleroma I was already writing a social network in Ruby for my day job. Because of that, I knew a lot about the pain points of doing it with Ruby, mostly the bad performance for anything involving concurrency. I had written a Bittorrent DHT client in Elixir, so I knew that it would work well for this kind of software. I was also happy to work with functional programming again, which I like very much.

Anyway, it's all water under the bridge, and if I want to understand why Xprime@Mu is stuck, I need to learn Elixir. Early signs are not that good. Right away, it uses its own control entity that replaces make(1), packaging, and a few other things, called "mix". Sasuga desu, as they say in my weeb neighbourhood. Every goddamn language does that nowadays.

[1] It's trickier, actually. For an inexplicable reason, Alice sees some updates by X: for example, re-posts.

One of the first experiences I had in this project at Princeton was meeting great men. I had never met very many great men before. But there was an evaluation committee that had to try to help us along, and help us ultimately decide which way we were going to separate the uranium. This committee had men like Compton and Tolman and Smyth and Urey and Rabi and Oppenheimer on it. I would sit in because I understood the theory of how our process of separating isotopes worked, and so they'd ask me questions and talk about it. In these discussions one man would make a point. Then Compton, for example, would explain a different point of view. He would say it should be this way, and was perfectly right. Another guy would say, well, maybe, but there's this other possibility we have to consider against it.

So everybody is disagreeing, all around the table. I am surprised and disturbed that Compton doesn't repeat and emphasize his point. Finally, at the end, Tolman, who's the chairman, would say, ``Well, having heard all these arguments, I guess it's true that Compton's argument is the best of all, and now we have to go ahead.''

It was such a shock to me to see that a committee of men could present a whole lot of ideas, each one thinking of a new facet, while remembering what the other fella said, so that, at the end, the discussion is made as to which idea was the best — summing it all up — without having to say it three times. These were very great men indeed.

Life on l-k before CoC.

I pay a special attention to my hometown rocket enterprise, Firefly. So, it didn't escape my notice when Dr. Tom Markusic mentioned SunBelt in the SatMagazine as a potential user of launch services:

Cloud Constellation Corporation capped off 2018 funding announcements with a $100 million capital raise for space-based data centers [...]

Not a large amount of funding, but nonetheless, what are they trying to do? The official answer is provided in the whitepaper on their website.

The orbiting belt provides a greater level of security, independence from jurisdictional control, and eliminating the need for terrestrial hops for a truly worldwide network. Access to the global network is via direct satellite links, providing for a level of flexibility and security unheard of in terrestrial networks.

SpaceBelt provides a solution – a space-based storage layer for highly sensitive data providing isolation from conventional internet networks, extreme physical isolation, and sovereign independent data storage resources.

Although not pictured in the illustrations, text permits users direct access, which will become important later:

Clients can purchase or lease specialized very-small-aperture terminals (VSATs) which have customized SpaceBelt transceivers allowing highly-secure access to the network.

Interesting. But a few thoughts spring to mind.

Isolation from the Internet is vulnerable to the usual gateway problem, unintentional or malicious. If only application-level access is provided, a compromised gateway only accesses its own account. So that's fine. However, if state security services were able to insert malware into Iran's nuclear facilities, I think that the isolation may not be as impregnable as purported.

Consider also that system control has to be provided somehow, so they must have a control facility. In terms of vulnerabilities to governments and physical attacks, it is an equivalent of a datacenter hosting the intercontinental cluster's control plane, located at the point where master ground station is. In case of SpaceBelt, it is identified as "Network Management Center".

In addition, the space location invites a new spectrum of physical attacks: now the adversary can cook your data with microwaves or lasers, instead of launching ICBMs. It's a significantly lower barrier to the entry.

Turning around, it might be cheaper to store the data where the NMC is, since the physical security measures are the same, but vulnerabilities are smaller.

Of course the physical security includes a legal aspect. The whitepaper nods to "jurisdictional independence" several times. They don't explain what they mean, but they may be trying to imply that the data sent from the ground to the SpaceBelt does not traverse the ground infrastructure where NMC is located, and therefore is not a subject to any legal restrictions there, such as GDPR.

Very nice, and IANAL, but doesn't Outer Space Treaty establishes a regime of the absolute responsibility of signatory nations? I only know that OST is quite unlike the Law of The Sea: because of the absolute responsibility there is no salvage. Therefore, a case can be made, if the responsible nation is under GDPR, the whole SunBelt is too.

The above considerations apply to the "sovereign" or national data, but the international business faces more. The whitepaper implies that accessing data may be a simple matter of "leasing VSATs", but the governments still have the powers to deny this access. Usually the radio frequency licensing is involved, such as the case of OneWeb in Russia. The whitepaper mentions using traditional GSO comsats as relays, thus shifting the radio spectrum licensing hurdles onto the comsat operators. But there may be outright bans as well. I'm sure the Communist government of mainland China will not be happy if SunBelt users start downloading Falun Gong literature from space.

One other thing. If frying SpaceBelt with lasers might be too hard, there are other ways. Russia, for example, is experimenting with a rogue satellite that approaches comsats. It's not doing anything too bad to them at present, but so much for the "extreme physical isolation". If you thought that using SunBelt VSAT will isolate you from the risk of Russian submarines tapping undersea cables, then you might want to reconsider.

Overall, it's not like I would not love to work at Cloud Constellation Corporation, implementing the basic technologies their project needs. Sooner or later, humanity will have computing in space, might as well do it now. But their pitch needs work.

Finally, for your amusement:

In the future, the SpaceBelt system will be enabled to host docker containers allowing for on-orbit data processing in-situ with data storage.

Congratulations, Docker. You've became the xerox of cloud. (In the U.S., Xerox was ultimately successful is fighting the dillution: everyone now uses the word "photocopy". Not that litigation helped them to remain relevant.)

I tinker with software radio as a hobby and I am stuck solving a very basic problem. But first, a background exposition.

Bdale, what have you done to me

Many years ago, I attended an introductory lecture on software radio at a Linux conference we used to have - maybe OLS, maybe LCA, maybe ALS/Usenix even. Bdale Garbee was presenting, who I mostly knew as a Debian guy. He outlined a vision of Software Defined Radio: take what used to be a hardware problem, re-frame it as a software problem, let hackers hack on it.

Back then, people literally had sound cards as receiver back-ends, so all Bdale and his cohorts could do was HF, narrow band signals. Still, the idea seemed very powerful to me and caught my imagination.

A few years ago, the RTL-SDR appeared. I wanted to play with it, but nothing worthy came to mind, until I started flying and thus looking into various aviation data link signals, in particular ADS-B and its relatives TIS and FIS.

Silly government, were feet and miles not enough for you

At the time FAA became serious about ADS-B, two data link standards were available: Extended Squitter aka 1090ES at 1090 MHz and Universal Access Transciever aka UAT at 978 MHz. The rest of the world was converging quickly onto 1090ES, while UAT had a much higher data rate, so permitted e.g. transmission of weather information. FAA sat like a Buridan's ass in front of two heaps of hay, and decided to adopt both 1090ES and UAT.

Now, if airplane A is equipped with 1090ES and airplane B is equipped with UAT, they can't communicate. No problem, said FAA, we'll install thousands of ground stations that re-transmit the signals between bands. Also, we'll transmit weather images and data on UAT. Result is, UAT has a lot of signals all the time, which I can receive.

Before I invent a wheel, I invent an airplane

Well, I could, if I had a receiver that could decode a 1 megabit/second signal. Unfortunately, RTL-SDR could only snap 2.8 million I/Q samples/second in theory. In practice, even less. So, I ordered an expensive receiver called AirSpy, which was told to capture 20 million samples/second.

But, I was too impatient to wait for my AirSpy, so I started thinking if I could somehow receive UAT with RTL-SDR, and I came up with a solution. I let it clock at twice of the exact speed of UAT, a little more than 1 mbit/s. Then, since UAT used PSK2 encoding, I would compare phase angles between samples. Now, you cannot know for sure where the bits fall over your samples. But you can look at decoded bits and see if it's garbage or a packet. Voila, making impossible possible, at Shannon's boundary.

When I posted my code to github, it turned out that a British gentleman by the handle of mutability was thinking about the same thing. He contributed a patch or two, but he also had his own codebase, at which I hacked a bit too. His code was performing better, and it found a wide adoption under the name dump978.

Meanwhile, the AirSpy problem

AirSpy ended collecting dust, until now. I started playing with it recently, and used the 1090ES signal for tests. It was supposed to be easy... Unlike the phase shift of UAT, 1090ES is much simpler signal: raising front is 1, falling front is 0, stable is invalid and is used in the preamble. How hard can it be, right? Even when I found that AirSpy only receives the real component, it seemed immaterial: 1090ES is not phase-encoded.

But boy, was I wrong. To begin with, I need to hunt a preamble, which synchronizes the clocks for the remainder of the packet. Here's what it looks like:

The fat green square line on the top is a sample that I stole from our German friends. The thin green line is a 3-sample average of abs(sample). And the purple is raw samples off the AirSpy, real-only.

My first idea was to compute a "discriminant" function, or a kind of an integrated difference between the ideal function (in fat green) and the actual signal. If the discriminant is smaller than a threshold, we have our preamble. The idea was a miserable failure. The problem is, the signal is noisy. So, even when the signal is normalized, the noise in more powerful signal inflates the discriminant enough that it becomes larger than the discriminant of background noise.

Mind, this is a long-solved problem. Software receiver for 1090ES with AirSpy exists. I'm just playing here. Still... How do real engineers do it?

well I had to write a sysv init script today and I wished it was systemd

— moonman, 21 December 2018

To quoth:

Why not walk down the wider path, using GNU/Linux as DOM0? Well, if you like the kernel Linux, by all means, do that! I prefer an well-engineered kernel, so I choose NetBSD. [...]

Unfortunately, NetBSD's installer now fails on many PCs from 2010 and later. [...]

Update 2018-03-11: I have given up on NetBSD/Xen and now use Gentoo GNU/Linux/Xen instead. The reason is that I ran into stability problems which survived many NetBSD updates.

You have to have a heart of stone not to laugh out loud.

P.S. Use KVM already, sheesh.

P.P.S. This fate also awaits people who don't like SystemD.