Pete Zaitcev

Joe Arnold on Software-defined Storage

Thu, 16 May 2013 16:19:46 GMT

At Havana summit they were giving away a paper version of Joe Arnold's "Software Defined Storage with OpenStack Swift". Very useful book for anyone dealing with Swift, I would be glad to pay the cover price of $25. But even more interestingly than tips on care and feeding of Swift, Joe opens the whole book thus:

[...] a de-coupled management system so customers could achieve (1) amazing flexibility in terms of how (and where) they deployed their storage, (2) control of their data without being locked-in to a vendor and (3) private storage at public cloud prices.

These features are the essence of Software Defined Storage (SDS), a new term the meaning of which is being defined. [...] Key aspects of SDS are scalability, adaptability, and the ability to use most any hardware. Through this de-coupling, operators can now make choices on how their storage is scaled and managed and how users can store and access data — all driven programmatically for the entire storage tier, regardless of where the storage resources are deployed.

Parts of the above prompt questions. Firstly, what good is de-coupling in respect to lock-in? SwiftStack effectively locks in by owning the de-coupled management. Sure, you own your data and could, in theory, manage your Swift with another management plane... I do not expect anyone crazy enough to try switching by anything less than standing up a new cluster. In any case, that part is not important, IMHO. The important part is programmatic control.

The phrase "SDS" jumps off "Software-Defined Networking". When SDN came into OpenStack, I was quite skeptical about it. It seemed too much like vendor-driven marketing bullshit. However, as users deployed the Project Formerly Known as OpenStack Quantum, it became clear that SDN answers their needs. The chief need was the ability to shape networks programmatically, overlaid on top of the physical networking plant, in service of the VMs.

Before SDN, when all this cloud thing came about, practitioners also struggled with the definition of it, and in particular the difference from the plain old datacenter virtualization. The difference is the programmatic control throughout. RHEV (now oVirt) eventually grew an API, which blurred the lines. But in OpenStack it was the main feature from the start. So you can manage everything and anything programmatically, including, for example, running on bare hardware. One can say that cloud is "Software-Defined Computing".

So, how does this programmatic thing apply to Swift? Joe had interesting insights cunningly hidden in the book, like these:

In an SDS system, reliability is the responsibility of the software, not the hardware. Replication and data integrity tactics are used to ensure that data does not become corrupt and that lost data is recovered.

[...]

A crucial function of an SDS system is to orchestrate capacity — storage, networking, routing & services — for entire cluster.

Swift covers the first part well already. The second is missing, or "de-coupled".

For galactic fairness, he also wrote things that seem wrong-headed to me:

There is no application sharding or managing volumes which can drive operational knowledge and complexity into applications because the SDS system is one cohesive system. Users do not need to ask for or know 'which storage pool' should be used because there is only one namespace.

The problem with hiding the pools outside of namespace is that they become invisible to the programmatic control as well, and such control is essential to the very definition of SDS. Someone at Amazon made a brilliant decision to make buckets a unit of replication in S3, so they can be linked to a region. In effect this hides the complexity but exposes knowledge that an application needs. Thus, any S3 client can do what Joe coniders SDS, but without any de-coupling, through the namespace and inside the API (or it can chose not to do it and just use a default region, for simplicity).

Joe's employees are hard at work implementing the vision as he outlined it, using the concept of regions that are internal to Swift cluster. The problem for everyone else, however, is how the programmatic control of that stack is exclusive to SwiftStack (with some useful things leaking into Swift, such as changeable replica count).

So, in the end, today Swift offers a solid foundation and parts of an SDS system, but the orchestration is "de-coupled" away elsewhere. Seems like a clear challenge to OpenStack to (re-)create the missing pieces.

P.S. I'd love to see the missing parts inside the Swift API and even namespace, although we have a problem here. Our Accounts and Containers are not guaranteed to live anywhere specifically or even on the same nodes. Changing that would be a step that I prefer. But Joe prefers to give up on plugging programmatic orchestration into the Swift API and just "de-couple" the heck of it. John, our benevolent PTL, seems to toe that line. Maybe they are right.

P.P.S. The deal with the programmatic orchestration is something that "unified" storage projects have to address too. E.g. in GlusterFS a program can issue mkdir(2). Is this programmatic control? No, not enough. Okay, they have glusterfsd nowadays, I can create volfiles in there, is that SDS? That is getting closer!

Viva la testing revolution

Thu, 09 May 2013 20:12:25 GMT

This is not something to brag about, but apparently I managed to program computers for about 30 years without writing unit tests. Today it's recitified by adding a test to one of my projects voluntarily. I encountered the goodness of build-time testing when working on Jeff Garzik's Project Hail. And of course, OpenStack, including Swift, had them since forever. Those weren't my projects, however.

I have your posix_ipc right here

Tue, 23 Apr 2013 18:28:24 GMT

Greg Holt posted a change to Swift's object server and replicator, which adds a counting semaphore to the server. As best I can tell, the intent is to provide a form of load control, so that no more than N replicators talk to a given server. So:

    global_conf['replication_semaphore'] = [posix_ipc.Semaphore(
        None, flags=posix_ipc.O_CREAT | posix_ipc.O_EXCL,
        initial_value=int(preloaded_app_conf['replication_concurrency']))]
    .......
    try:
        self.replication_semaphore.acquire(0)
    except posix_ipc.BusyError:
        return HTTPServiceUnavailable()
    try:
        ....... replication service goes here
    finally:
        self.replication_semaphore.release()

This appears to be doing the job, but it adds a dependency on posix_ipc. Question arises, are we missing a traditional way to do this?

Since the semaphores are not taken and checked often in this application, once per replication request, something like creating a directory in /var/run, populating it with one file per server instance, and counting them with listdir would be acceptable (even with obvious inexact count), but we need something that cleans up in case of crashes.

Another trick I know is to create a pipe and prime it with N 1-byte tokens, but then again, crash cleanup is not bulletproof.

The best would be to identify an entity in Linux that counts opens. Something in /proc, perhaps? Open it, read it to see the open count, if it is too high, abort. Close when done. Kernel closes it in case of a crash, perfect. Well, I guess I just described POSIX semaphore, again. If only it was a part of Python...

UPDATE: Someone was urging Greg to use multiprocessing.Semaphore. Unfortunately, the docs for that contain omnious warnings of poor crash resistance. Cannot say more without looking at the implementation.

Amazon S3 breaks 2T objects

Thu, 18 Apr 2013 17:39:13 GMT

Techcrunch said it all:

At at AWS Summit in New York today, Amazon announced that its S3 storage service now holds more than 2 trillion objects. That’s up from 1 trillion last June and 1.3 trillion in November [...]. As Amazon’s Chief Evangelist for AWS Jeff Barr notes in a blog post today, it took Amazon six years to grow to get to 1 trillion stored objects, “and less than a year to double that number.” S3, Barr also writes, now regularly sees peaks of more than 1.1 million requests per second.

I don't think all Swift istallations combined come anywhere close, although potentially they should be able to. Pretty easy, actually. Meanwhile, we hit local problems. At the Summit two days ago, a presenter from Korea Telecom said that some of their customers put 50 million (5e7) objects into a Swift container. Naturally, container listings and even lookups become challenging.

Facebook Open Hardware

Tue, 16 Apr 2013 20:45:12 GMT

Continuing the Hardware at Havana theme, Facebook showed their "Open Hardware" platform. Like Google, they found it advantageous to design systems for themselves, but unlike Google, they opened the design. Anyone can download the detailed docs and anyone can contribute.

I was sceptical about open source for hardware. The material nature of hardware places a high barrier, so participation tends to be low. Nonetheless, the gentleman at the booth said that someone contributed an ARM server board. It slides into the slot under the drive tray where SAS extender goes in the reference design.

It shows how long I was out of the hardware, but apparently it's completely acceptable nowadays to place drives along the airflow. The Facebook tray is very much like Dell design that way, only using 3.5" drives. Fans are removable on drive trays but not on server bricks.

In a strange twist, Facebook trays take 12V DC power. Not telco-style 48V, nor contentional AC.

Dell C5220/C5125

Tue, 16 Apr 2013 16:12:51 GMT

Saw a cute thing at the "Havana" OpenStack Summit: a Dell ARM prototype. It is based on C5125 module for C5220 chassis. If you go to Dell website and check their gallery, it's readily apparent how busy the design of the x86-based module is when compared with the ARM (the x86 is AMD; Dell have an even worse module based on Intel -- 50% thicker).

Another thing, I love the way drives are bunched on trays. People keep chasing the individually accessible drives with the corresponding hit to density. If one abandons the hot-swap mania, he can do these nice trays. Also! This is the first time I see someone intelligently using the 2.5" drives. Only took 15 years. About the only critique I would level on the thing is that the air goes over the drives in sequence. There's a bit of spare width for staggering, so a few cheap baffles and/or raisers could even it out, but someone at Dell decided not to bother.

BTW, Dell people told me that the prototype is not slated for production. They are waiting for the 64-bit ARM. Typical. ARM servers are always one year away, for many years now. In the time we heard promises of ARM servers SpaceX built whole rocket and started launching it, but Netwinder seems harder to reproduce than Saturn V.

Yahoo versus Linode in datacenter networking

Sat, 09 Mar 2013 00:24:14 GMT

I just saw a blog post today that trumpets improvements that Linode made to their datacenter networks. It looks pretty good, bandwidth is improved, etc. I would never know any better, except that a month ago I was at a presentation by Yihua He of Yahoo. And, for a layman like me, it seemed that he was quite a bit ahead.

Link-wise, the He-style Yahoo networking is very similar to Linode's Nextgen. It uses a so-called "leaf-spine" architecture with Top-of-Rack (ToR) switches being the leaves. Linode's post did not articulate clearly the advantages that He architure brings: increased bisection that accomodates growing east-west traffic and reduction in oversubscription, but I presume Linode customers will reap improvements in that area too. However, there's one curious difference: Linode allowed a vendor to provide the balancing that leaf-spine requires, whereas He went with a COTS gear.

He accomplishes this by using routers as spines where Linode uses Cisco Nexus 5000. As it turns out, using iBGP works in place of whatever proprietary magic Cisco employs. He simply gives each ToR a private ASN.

Once you go with COTS gear, you are free from the yoke of Cisco. You can mix and match as you with, build datacenters incrementally, and run any kind of monitoring apps you like. And the whole thing is much, much cheaper.

To be sure, there are issues as well. For example, depending on how magical Cisco is, you might be able to migrate VMs at will throughout the datacenter. In case of He, you can't. Although, using OpenStack with floating IPs probably accomplishes a useful result in such a case anyway.

But in the end, if you can use an open solution, is it ever worth using the vendor solution? And I definitely didn't expect Linode and Yahoo feature in this ironic role reversal. Which one of them is the scrappy startup and which one is a bureaucratic monster courted by lock-in vendors?

Working Remotely

Wed, 27 Feb 2013 17:44:10 GMT

One thing I found remarkable about the recent Yahoo flap (memo) is how Marissa Mayer was supposed to be one of the bright lights of Google, who was recruited back when Google was much smaller and presumably was wetted by founders, and yet she has no clue. No disrespect to rlove, AKPM, Raph, and tytso intended, but how many supposedly ultra-smart Google people are not really? Is this legend only a myth?

By the way, Red Hat is quite good at this remote thing. At some point in 2003 or so Paul Cormier V.P. of Eng. attempted to consolidate all of engineering in Boston (Matthew was the CEO at the time). They asked me to relocate, too. But when I used my best judgement and declined, there were no sanctions. Red Hat did not threaten to fire me over it. That, I think, is the biggest difference between Paul's and Marissa's approaches. And of course he pushed for it back when it wasn't quite as obvious that remoting within reason is the way to go for programmers, and she is doing it in 2013.

P.S. Not sure what to make of Elon Musk in this light.

Hard numbers from Wikimedia

Thu, 25 Oct 2012 04:31:43 GMT

John threw a link to IRC that indicates that Wiki people hit 173,000,000 objects. This is 1e4 times fewer than Amazon S3 has, but 10 times more than Sina has. I heard more about Sina's performance issues and apparently they had issues with lots of updates to those objects; there was talk about sharding. Wiki guys probably do not update so much... Those objects are jpegs or something like that, so they only need to scale the auditors and replicators. Still, that's a heck of a pile of objects, even if so.

OpenStack Swift drops WebOb

Sun, 21 Oct 2012 00:52:26 GMT

This actually happened back in September, but it didn't click as blogworthy back then. So: we dropped WebOb for an in-tree module for 1.7.x.

The main reason was the trouble with WebOb being gratiously incompatible with itself, that came to the head with the WebOb 1.2. I am afraid myself screwing the pooch on that was a factor. Since Fedora Rawhide F18 came out with WebOb 1.2 and Swift blew up on it, I saw an opportunity to be productive and patch it over. Later, a guy from Korea, Iryong, joined in and did half of the coding. I thought I tested the patch well, but when it was committed, regressions started happening. Greg Holt came very close to requesting the whole thing being rescinded. The story dragged on long enough for Mike Barton just cut the whole gordian knot by throwing out not just my patch, but the whole WebOb. Good riddance, I say. I wish I was daring enough to do it myself.

The funniest part is that OpenStack's reliance on WebOb 1.0.8 was the reason why we had to package it in EPEL. If only I knew how this was going to turn out, we could possibly do something like that across OpenStack and avoid having two WebObs in RHEL 6. But nobody had Mike's vision.

I would not say that WebOb has poor governance or stewardship or whatever. They easily accepted my patch for BYTES= in HTTP Ranges:, so they are user-facing. But their policy of gratious incompatibility was detrimental for us in Swift, and the value we extracted from WebOb was small (look at size of Mike's replacement).

Now John started talking about getting rid of Eventlet. Woo hoo.

Slasti 2.0

Thu, 18 Oct 2012 14:42:17 GMT

With little fanfare, Slasti saw a release 2.0 (uploaded). This closes the chapter on our template nightmare. Unfortunately for that little project, I have too many exciting things going on with OpenStack, Swift, et.al, so it's going to languish, even though I use it every day (many times a day actually, via a browser homepage, per CKS).

Swift LFS and Gluster UFO

Fri, 12 Oct 2012 02:52:50 GMT

DanPB poked a few people to blog more about all the awesome stuff we have going in open cloud storage. Honestly, I find it a bit difficult of late. Well, I pulled Peter's branch of Gluster into a Swift tree today. Okay, is anyone interested? Didn't think so.

The only guy who I know as consistently interesting in the subject is JDarcy. But he's not prolific.

I sort of understand where Dan comes from. There's so much stuff going on, most of it unnecessarily silent. It is frustrating. I didn't know Portante worked on UFO (Unified Filesystem Objects) until MarkMc tracked him down by the way of taking hot iron to Kaleb. Is more blogging the answer? Dan feels it could be. I suppose I could make an effort.

Darcy on CDMI

Mon, 24 Sep 2012 17:46:51 GMT

Apropos Jeff Darcy mentioning CDMI and Swift in one paragraph, I looked briefly at the CDMI patch for Swift.

It exists
+2995 by diffstat, but well-confined in WSGI middleware pipeline
Only implements a small subset of the spec, enough to attend plugfests

The basic idea is that CDMI is the same front-end technology to Swift back-end as Jeff's model of SMB vs. Gluster. It aims to unsettle the S3 API, and you know there's a reason why Google Storage is S3 with very minor changes. Unfortunately, proponents of CDMI trade the strangehold by Amazon for a strangehold by a standards committee, and those who remember X.25 and OSI have a pretty good idea how that works out. After I poked at CDMI, I thought it wise to back away slowly.

Everyone who uses Swift these days just use Swift's native Cloud Files API. That includes Jeff's own Image Warehouse, which has two almost identical back-ends for "s3" and "cf" in about 50 lines of C code each. Try that with CDMI. Although, knowing Jeff, he probably can code a CDMI client in that much, using nothing but libcurl and libc :-)

Seeking Google Authenticator for Linux

Mon, 27 Aug 2012 15:29:20 GMT

I have enabled a 2-step authentication for Google, but I do not have a smartphone. What to do?

The official answer #0 is to buy the goddamn smartphone. But that means paying a cellular carrier for the rest of your life.

The official answer #1 is to have Google call you. This is good, but what if I travel, say, to Ireland?

The official answer #2 is to print a set of challenge-response pairs, sort of like we did in OPIE days. The only small problem is that I do not see them offering to switch from voice verification to challenge-response. Also, it's a hassle even if it works.

In any case, it would be ideal if an open source program existed that substituted for Google Authenticator. Surprisingly, I am unable to find such a thing, and I don't know why. There should be nothing too secret about the workings of the code generator, even the parameters of the PRNG (the seed is the secret).

The article at LWN says:

Traditionally, hardware authentication tokens must be physically connected to the computer to authenticate a user, though some one-time password (OTP) generators are standalone. Unlike the Android app, however, those devices are meant to make it difficult to extract the key without destroying them. Accessing the key from a phone, then running the app elsewhere (e.g. an Android emulator) would circumvent the "things you have" requirement.

OK, fine, a computing device is less secure than the RSA token [generator] that I have on my keychain. But phone and laptop are no diffrent. In fact, my laptop is far more secure than average malware-infested smartphone. Anyway, I don't want to hear the excuses, I just want this to work.

The same article suggests:

You can perform a passcode-generation hash by running:
oathtool --totp --now="the_current_time" your_secret_key
The passcodes matched, once I figured out how to correctly convert the Base32 encoding produced by Google Authenticator into the hexadecimal required by oathtool — namely, that the Base32 encoding scheme defined by RFC 4648 is not the same as base-32 mathematical notation (because the encoding avoids easy-to-confuse characters like I and O).

Ouch. Is there no better way? Maybe I should write a wrapper.

Github

Fri, 06 Jul 2012 16:48:14 GMT

I remember whailing about Sourceforge being our Open Source SPOF because everyone hosted on it. Seems like the problem solved itself now. Hah! What's interesting, I find it convenient to throw even 20-line script at Github. It's already in git, so almost zero cost to push too. With SF I would need to think: is this project important enough? Not anymore.

All this makes me wish for some kind of semantic indexing. But I know that prompts madness.

Webcam and kernel nostalgia

Wed, 04 Jul 2012 21:42:23 GMT

I don't know what made me do it, but I ripped an OV511 decompressor from v4l-utils grafted it onto fswebcam. The last time I wanted to take a picture, I found that nothing seemed to support OV511, so I gave up on it and used ibmcam. That one, however, requires a kernel patch, because it runs DMA from vmalloc-ed area, or some such. I figured it out then. Now? Meh, don't wanna reboot the laptop to test. I guess I'm not a kernel programmer anymore for real.

BTW, the changelog for fswebcam claims that I contributed something to it back in 2010. Can't remember what exactly. Something to do with ibmcam being V4L1-only.

Hard numbers from China

Fri, 13 Apr 2012 16:40:01 GMT

It looks like almost every cloud provider hide their numbers, which makes guidance and education unnecesserily difficult. To be fair, I think I saw AWS posting a few items for S3, but forgot to save it. So, I'm going to preserve what one Chinese gentleman posted to OpenStack list, in the context of Swift performance issues:

Our practice of  Sina Web Service Team https://launchpad.net/~sws:

total accounts:          121,961;
total containers:        160,703;
total objects:        14,291,519;
total storage usage:   1.3T

account replication time:      10 hours;
container replication time:    10 hours;
object replication time:       48 hours;
account audit time:             2 hours;
container audit time:           9 hours;
container update time:         19 hours;

Unfortunately he omitted the requests per second and gigabytes per second that the cluster is sustaining from the users, but it's very interesting anyway.

QUICKIE: Apparently I meant the official post about Amazon S3:

total objects:   905,000,000,000
total bytes:     ?
requests/s:           650,000

UPDATE 2012-06-12: Amazon announced that they reached a trillion:

total objects: 1,000,000,000,000
total bytes:     ?
requests/s:           650,000

The growth rate, according to my calculations, is 18,325 objects/s. The blog article claims 40,000 objects/s.

Confounding

Thu, 22 Mar 2012 03:37:14 GMT

16.1.9 JSON Format for ACLs

ACE flags and masks are members of a 32-bit quantity that is widely understood in its hexadecimal representations. The JSON data format does not support hexadecimal integers, however. For this reason, all hexadecimal integers in CDMI ACLs shall be represented as quoted strings containing a leading "0x".

"cdmi_acl" : [ { "acetype" : "0xnn", .....

If readability by humans is paramount, then why not use a bit string, like in ls(1)? If readability is not an issue, just transmit decimal.

They also have hexadecimals without "0x" prefix elsewhere in the spec.

Bugception

Mon, 19 Mar 2012 20:00:24 GMT

[root@kvm-rei zaitcev]# rpm --rebuilddb error: db5 error(-30969) from dbenv->open: BDB0091 DB_VERSION_MISMATCH: Database environment version mismatch error: cannot open Packages index using db5 - (-30969)

I have a vague feeling that I encountered this before, but I do not remember how I dealt with it.

UPDATE: Needs rm /var/lib/rpm/__db* .

Twitterfail 2

Fri, 02 Mar 2012 01:07:16 GMT

Just ran into a weird case on Twitter: If I tweet the following link to Failblog, the tweet disappears:

http://failblog.org/2012/02/23/epic-win-photos-win-wwf-win/

It disappears even if shortened with something like TinyURL. I noticed because a couple of people favourited my tweet before it disappeared, and then when it disappeared, Twitter's own "interactions" page became corrupted (one of their Javascript functions returned "undefined").

The other reST

Sat, 25 Feb 2012 00:21:04 GMT

I was moonlighting a bit as a technical writer, and run into an odd issue. In Swift SAIO doc we have a piece that comes out like this:

[swift-hash]
# random unique string that can never change (DO NOT LOSE)
swift_hash_path_suffix = changeme

Why is "hash" red?

My reading of the documentation for reST suggests nothing. If I try to escape the dash with a backslash, color disappears, but the backslash leaks into the HTML.

To make matters worse, we cannot just spit on it all and re-code everything in MD, because Sphinx is quite well entrenched.

UPDATE: trying bug 797425. As it turned out, the problem is that Sphinx treats the document as Python code and highlights accordingly.

Eucalyptus and openness

Tue, 17 Jan 2012 23:55:23 GMT

Reading what Spot blogged about Eucalyptus, them poaching Greg de K. off us, and how this indicates change, and cannot help thinking he's way too optimistic. They are unlikely to go fully open, and the proprietary disease only dies when eradicated from the whole company. Remember Cygnus? It was supposed to be the torchlight of commerical GNU. When we bought them, old hands shared stories how their proprietary offerings failed again and again, while the support for gcc kept the lights on. Nonetheless, their executives did not have the will to go full open. I'm saying it with all due respect to Michael Tiemann, of course! But I doubt the Eucalyptus' management is much more sincere or unified when hiring Greg than Sun's was when they hired Ian Murdock. Or was that Simon Phipps?

Blogs

Mon, 09 Jan 2012 03:10:53 GMT

Unwittingly, I sat myself up for a yearly excersize, but let's look for a silver lining, shall we?

Blog	Posts	Delta
Ani-nouto	149	-220
Meenuvia	93	-28
LiveJournal	27	-5
Tracy Today	1	-3
Mixi	0
Duke City Fix	0
Advogato	0

Looks like I'm taking my life back from the Internet.

SElinux in Fedora 16

Fri, 06 Jan 2012 05:11:45 GMT

Perhaps even a flatworm can learn to turn left in T-maze when he's hit with an electric shock, but every time I install new Fedora, I try to leave SElinux enabled. In Fedora 15 it lasted for almost a week. This time, things went sour in 10 minutes.

Thought 1: The troubleshooter is quite nice, they really are getting civilized these days. Poor Polkit is doing isatty(), let's see if we can help.

Thought 2: Well, this is a bit ugly, but hey, it's copy-pasteable! Hurray for supporting common UNIX tools...

Thought 3: I am dumber than a flatworm. How could I believe that this time they get it right.

Actually, I lied. I disabled the sorry thing the moment I saw this:

The reason I continue to suffer through this excercise twice a year is that the security issue is important, and the shifing focus of attacks at non-daemon, non-system applications (read: browsers) is a matter of great concern for me. Every time Dan Walsh posts something nice to his LJ it looks like progress is being made, but then I install new Fedora and feel despair. It is as if some fundamental architecture is broken or something.

Modern Linux forming new social mores

Fri, 16 Dec 2011 05:48:48 GMT

Was at a meeting today and at some point a whole side of the table was lined with half-closed laptops - except one that was closed fully. You could play "spot the GNOME 2.x user" (hello, Richard). That is because GNOME 3 on Fedora 16 has a reliable suspend, so everyone has it enabled by default, because that's what one normally wants. At a meeting, people used to close the tops to direct their attention to the presentation, or to indicate that they do. But that was then. Now, doing so triggers suspend, and that bumps you off VPN or weird WiFi providers like BitBuzz. Result is funny-looking and awkward. My neighbour eventually resorted to locking the screen (which blanks) instead of closing the top. Although blocking the distraction effectively, this has the disadvantage of not sending the right signal to the presenter, but tough cookies.

I think ideally I might like some kind of hotkey-suspend or whatnot, and disable suspend-on-close, but I dunno. Seems kind of bothersome to RTFM for GNOME. Worse, getting everyone in a meeting to agree on upon a non-default configuration seems like a non-starter.