Pete Zaitcev's Journal [entries|friends|calendar] Pete Zaitcev [ userinfo | livejournal userinfo ] [ calendar | livejournal calendar ]

Hard numbers from China [13 Apr 2012|10:39am] It looks like almost every cloud provider hide their numbers, which makes guidance and education unnecesserily difficult. To be fair, I think I saw AWS posting a few items for S3, but forgot to save it. So, I'm going to preserve what one Chinese gentleman posted to OpenStack list, in the context of Swift performance issues: Our practice of Sina Web Service Team https://launchpad.net/~sws: total accounts: 121,961; total containers: 160,703; total objects: 14,291,519; total storage usage: 1.3T account replication time: 10 hours; container replication time: 10 hours; object replication time: 48 hours; account audit time: 2 hours; container audit time: 9 hours; container update time: 19 hours; Unfortunately he omitted the requests per second and gigabytes per second that the cluster is sustaining from the users, but it's very interesting anyway. QUICKIE: Apparently I meant the official post about Amazon S3: total objects: 905,000,000,000 total bytes: ? requests/s: 650,000 [link] post comment

Confounding [21 Mar 2012|09:37pm] 16.1.9 JSON Format for ACLs ACE flags and masks are members of a 32-bit quantity that is widely understood in its hexadecimal representations. The JSON data format does not support hexadecimal integers, however. For this reason, all hexadecimal integers in CDMI ACLs shall be represented as quoted strings containing a leading "0x". "cdmi_acl" : [ { "acetype" : "0xnn", ..... If readability by humans is paramount, then why not use a bit string, like in ls(1)? If readability is not an issue, just transmit decimal. They also have hexadecimals without "0x" prefix elsewhere in the spec. [link] post comment

Bugception [19 Mar 2012|02:00pm] [root@kvm-rei zaitcev]# rpm --rebuilddb error: db5 error(-30969) from dbenv->open: BDB0091 DB_VERSION_MISMATCH: Database environment version mismatch error: cannot open Packages index using db5 - (-30969) I have a vague feeling that I encountered this before, but I do not remember how I dealt with it. UPDATE: Needs rm /var/lib/rpm/__db* . [link] 1 comment|post comment

Twitterfail 2 [01 Mar 2012|06:07pm] Just ran into a weird case on Twitter: If I tweet the following link to Failblog, the tweet disappears: http://failblog.org/2012/02/23/epic-win-photos-win-wwf-win/ It disappears even if shortened with something like TinyURL. I noticed because a couple of people favourited my tweet before it disappeared, and then when it disappeared, Twitter's own "interactions" page became corrupted (one of their Javascript functions returned "undefined"). [link] post comment

The other reST [24 Feb 2012|05:20pm] I was moonlighting a bit as a technical writer, and run into an odd issue. In Swift SAIO doc we have a piece that comes out like this: [swift-hash] # random unique string that can never change (DO NOT LOSE) swift_hash_path_suffix = changeme Why is "hash" red? My reading of the documentation for reST suggests nothing. If I try to escape the dash with a backslash, color disappears, but the backslash leaks into the HTML. To make matters worse, we cannot just spit on it all and re-code everything in MD, because Sphinx is quite well entrenched. UPDATE: trying bug 797425. As it turned out, the problem is that Sphinx treats the document as Python code and highlights accordingly. [link] 2 comments|post comment

Eucalyptus and openness [17 Jan 2012|04:55pm] Reading what Spot blogged about Eucalyptus, them poaching Greg de K. off us, and how this indicates change, and cannot help thinking he's way too optimistic. They are unlikely to go fully open, and the proprietary disease only dies when eradicated from the whole company. Remember Cygnus? It was supposed to be the torchlight of commerical GNU. When we bought them, old hands shared stories how their proprietary offerings failed again and again, while the support for gcc kept the lights on. Nonetheless, their executives did not have the will to go full open. I'm saying it with all due respect to Michael Tiemann, of course! But I doubt the Eucalyptus' management is much more sincere or unified when hiring Greg than Sun's was when they hired Ian Murdock. Or was that Simon Phipps? [link] post comment

Blogs [08 Jan 2012|08:11pm] Unwittingly, I sat myself up for a yearly excersize, but let's look for a silver lining, shall we? Blog Posts Delta Ani-nouto 149 -220 Meenuvia 93 -28 LiveJournal 27 -5 Tracy Today 1 -3 Mixi 0 Duke City Fix 0 Advogato 0 Looks like I'm taking my life back from the Internet. [link] post comment

SElinux in Fedora 16 [05 Jan 2012|10:11pm] Perhaps even a flatworm can learn to turn left in T-maze when he's hit with an electric shock, but every time I install new Fedora, I try to leave SElinux enabled. In Fedora 15 it lasted for almost a week. This time, things went sour in 10 minutes. Thought 1: The troubleshooter is quite nice, they really are getting civilized these days. Poor Polkit is doing isatty(), let's see if we can help. Thought 2: Well, this is a bit ugly, but hey, it's copy-pasteable! Hurray for supporting common UNIX tools... Thought 3: I am dumber than a flatworm. How could I believe that this time they get it right. Actually, I lied. I disabled the sorry thing the moment I saw this: The reason I continue to suffer through this excercise twice a year is that the security issue is important, and the shifing focus of attacks at non-daemon, non-system applications (read: browsers) is a matter of great concern for me. Every time Dan Walsh posts something nice to his LJ it looks like progress is being made, but then I install new Fedora and feel despair. It is as if some fundamental architecture is broken or something. [link] 4 comments|post comment

Modern Linux forming new social mores [15 Dec 2011|10:48pm] Was at a meeting today and at some point a whole side of the table was lined with half-closed laptops - except one that was closed fully. You could play "spot the GNOME 2.x user" (hello, Richard). That is because GNOME 3 on Fedora 16 has a reliable suspend, so everyone has it enabled by default, because that's what one normally wants. At a meeting, people used to close the tops to direct their attention to the presentation, or to indicate that they do. But that was then. Now, doing so triggers suspend, and that bumps you off VPN or weird WiFi providers like BitBuzz. Result is funny-looking and awkward. My neighbour eventually resorted to locking the screen (which blanks) instead of closing the top. Although blocking the distraction effectively, this has the disadvantage of not sending the right signal to the presenter, but tough cookies. I think ideally I might like some kind of hotkey-suspend or whatnot, and disable suspend-on-close, but I dunno. Seems kind of bothersome to RTFM for GNOME. Worse, getting everyone in a meeting to agree on upon a non-default configuration seems like a non-starter. [link] post comment

Playing the GNOME 3 Desktop Quest [10 Nov 2011|12:16pm] Chris Siebenmann mentioned in e-mail to me that a new terminal window opens if one Ctrl-clicks on it in Activities in GNOME 3 (there are more tips in comments of his blog post). Previously, I found that right-click there brings up a menu that can open new windows, but this takes it to the new level. Owen Taylor, you are so devious! Back in 1985 or so, a company where I worked obtained a very expensive and unique piece of equipment: an IBM PC with a hard drive. Someone installed a game on it, called "Leisure Suit Larry" one or the other, and the office took turns playing it. The game was a quest with "quasi-natural" command input. At some point in the game, the player has a dull knife that does not cut. Someone heard from someone that it needs to get sharpened, but nothing we tried worked. Nobody in the firm knew English, so I remember trying "make knife sharp", "knife sharp", "sharp knife", and any other permutation including "knife sharp do". The right answer was "sharpen knife". GNOME 3 reminds me of that quest. Random inputs that make complete sense to Owen, same as the conjugation into imperative made complete sense to an English speaker, but entirely opaque and mysterious to people who are not in on the secret. [link] 2 comments|post comment

OSPF in the home network [27 Oct 2011|04:18pm] Having VLAN at home is quite reasonable, having DNS may be excusable, but to run OSPF one must be a real nerd. Normally, I am not, but at a certain point I made a decision to use routed VM networking, rather than bridged, and things pretty much snowballed from there. The last straw was a weird regression in Fedora 15, which does not accept IPv6 statics which Fedora 14 did. KVM host, zebra.conf: ! zebra.conf hostname simbelmyne.zaitcev.lan password grumble interface eth0 interface virbr0 ! we have to run radvd because of virsh, so don't interfere into virbr0 ipv6 nd suppress-ra log file /var/log/quagga/zebra.log KVM host, ospfd.conf: password grumble ! router ospf router-id 192.168.128.10 network 192.168.0.0/16 area 0 redistribute connected ! log file /var/log/quagga/ospfd.log KVM host, ospf6d.conf: password grumble router ospf6 redistribute connected route-map TBD interface eth0 area 0.0.0.0 interface virbr0 area 0.0.0.0 !! network fd2d:acfb:74cc::/48 area 0 !ipv6 prefix-list test-prefix seq 1000 deny any ipv6 prefix-list TBD-prefix permit fd2d:acfb:74cc:3::/64 ipv6 prefix-list TBD-prefix permit 2001:470:86b3:3::/64 route-map TBD permit 10 match ipv6 address prefix-list TBD-prefix log file /var/log/quagga/ospf6d.log Router, zebra.conf: ! zebra.conf hostname elanor.zaitcev.lan password grumble !ipv6 route ::/0 sit1 interface ethmain.5 ipv6 nd suppress-ra interface wlanhome ipv6 nd suppress-ra interface ethmain.3 ipv6 nd suppress-ra log file /var/log/quagga/zebra.log Router, ospfd.conf: password grumble router ospf router-id 192.168.128.1 network 192.168.0.0/16 area 0 redistribute connected default-information originate log file /var/log/quagga/ospfd.log Router, ospf6d.conf: ! ospf6d.conf password grumble router ospf6 router-id 192.168.128.1 !! redistribute kernel because it's the only way to fetch the default !! neither static nor connected work on it for some reason redistribute kernel route-map TBD ! redistribute static redistribute connected interface ethmain.5 area 0.0.0.0 interface wlanhome area 0.0.0.0 interface ethmain.3 area 0.0.0.0 ipv6 prefix-list TBD-prefix permit ::/0 !ipv6 prefix-list TBD-prefix permit fd2d:acfb:74cc::/48 !ipv6 prefix-list TBD-prefix permit 2001:470:1f05:51::/64 !ipv6 prefix-list TBD-prefix permit 2001:470:86b3::/48 !! route-map TBD permit 10 match ipv6 address prefix-list TBD-prefix log file /var/log/quagga/ospf6d.log The biggest sticking point was the v6 default route. Defining it in zebra.conf and "redistribute static" plainly had no effect. The v4 side has "default-information originate" for that, but it's not documented for v6. Therefore, the weird workarounds with route map. [link] post comment

VLAN in a home network [24 Oct 2011|10:40pm] It took me 2.5 years since the arrival of home router with 1 Ethernet onboard to figure out that VLAN is the way to go [1]. Until today, I got by with USB Ethernet, which actually worked extremely well. However, VLAN offers some advantages, chief of which is the capability to have several networks, including a separate less trusty network for TiVo and Wii. The switch was $114. The idea of VLAN would probably never occur to me, but I read about it in a book about WRT54, which employs this architecture in order to save on Ethernet ports. And unlike WRT54, I have enough capacity in the single 1GE chip. The biggest issue thus far was that while it would be very attractive to have the main VLAN untagged in the trunk, it appears impossible. For some weird reason, if you have, say, VLAN 5 untagged (eth0), and VLAN 3 tagged (eth0.3), dhcpd running on eth0 sees traffic arriving to eth0.3 too. Moreover, it manages to reply to it and NAKs requests. For that reason, I had to make all VLANs in the trunk tagged. This precludes emergency replacement of the switch. UPDATE: The post became a little bit out of date. After sitting on all-tagged setup for a few days, it occured to me that I lose access to the router if switch goes kaputt, and vice versa (due to letting switch acquire its management address with DHCP on a now-tagged LAN). Therefore, now we are back to tagged+untagged, but it's different from it was before. Before, the untagged was the main Ethernet, legacy. Basically I introduced the switch as a normal switch and then added VLANs on top of it. Perfect if you need constant network access throughout the process. Now, the untagged VLAN is a special segment with 2 fixed addresses. There is no traffic on it and no DHCP, except the management. It is to be used in case of emergencies. Either router or switch can be easily replaced with a working piece of equipment. Everything else is on tagged VLANs in the trunk. The dhcpd listens on tagged interfaces and avoids the bug. Finally, for the record, the VLANs are configured by the good old: "cp ifcfg-eth0 ifcfg-eth0.5 && echo VLAN=yes >> ifcfg-eth0.5". It seems strange that the VLAN ID is only configured in the filename, but that appears to be the right way to do it in Fedora 15. There is no VLAN_ID=5 anywhere. [1] Actually, the box has an internal Mini-PCIe connector, but it is occupied by the WiFi, a b43 scavenged from an old laptop. [link] 4 comments|post comment

DNS in a home network [13 Oct 2011|11:54am] The zone file for zaitcev.lan contains 55 A records and 19 AAAA records. This got about primarily because I register every network device. This way traffic analysis work better, so I can see what TiVo or Wii or PSPs are doing. I think normal people get by with fully automatic configuration, but it never worked well for me. [link] 1 comment|post comment

So it was Lennart after all [04 Aug 2011|10:07am] The all-too-familiar story: user runs "yum update", laptop becomes unbootable... with a message that implicates SElinux. I posted a screencap of "Failed to load SELinux policy" and a caption, which implied that SElinux people were worse than Lennart. Well, they weren't... A desperate run of grep across the whole root shows that that message comes from systemd. So, the apparent scenario is: the system boots with selinux=0, Lennart ignores it, tries to load SElinux policy anyway and fails, then aborts the boot. What a bummer; recovery is non-obvious. I am sorry to blame Morris, Walsh, and Co. for this debacle. UPDATE: I fixed it by rebuilding systemd with --disable-selinux. Now as a conscious hacker I should create a nice patch for selinux-setup.c that picks upon selinux=0 and submit it to Lennart's judgement. UPDATE: There was a bit of discussion in comments, which warrants some elaboration. First of all, Lennart is right not to bake kernel arguments into code and use selinux library. Instead, the failure of his logic happens in the following way. He invokes selinux_init_load_policy() and expects that it either works, or fails. If the library detects SELinux disabled, it returns success. However, the way in which it detects SElinux disabled is excessively roundabout, requiring magic mounts to happen, which can itself fail. When that happens, the whole system blows up, even though the breakage in SELinux components is entirely immaterial, due to it being disabled. UPDATE 20110922: An update to the bug 727068 today: If selinux=0 is passed to the kernel, then the mount point directory /sys/fs/selinux will not exist. selinux_init_load_policy() tries to mount selinuxfs on that directory, which will hence fail with ENOENT due to the missing mount point directory. In earlier versions when the file system was still mounted to /selinux the mount point dir always existed (since it was on the root disk, not in sysfs) and hence on selinux=0 ENODEV was returned when the mount was attempted. The function does check for ENODEV and handles things properly, but it doesn't do this for ENOENT. The system continues to be unbootable (I use a local-built systemd with workarounds). [link] 9 comments|post comment

The challenge of Python's open() [04 Jul 2011|07:08pm] When I migrated slast.zaitcev.us from F14 to F15, I stepped on a small IED. Slasti used UTF-8 as the internal representation of the data, stored in Pyton str type. It worked fine, but as it turned out, by accident. In F15 systemd does not set LANG anymore, and suddenly everything in Slasty blew up. In the interests of educating myself about Pythonic ways, I switched the internal representation from UTF-8 to unicode type. This mostly worked, but Python defeated me in one place: it appears impossible to use UTF-8 filenames if LANG is not set. If a UTF-8 string is passed as argument of open(), the following happens: UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-15: ordinal not in range(128) This is trivial to reproduce in any shell by "unset LANG". The official answer from Python community is set environment properly. It may even be possible in systemd, or in Apache, but it's ming-boggling that they left no way to open a file with a binary name. The first thing I tried was os.fdopen(os.open()), but that blows up in the same way. There is no sys.open(). Next on the agenda: call a C method. With all that, I am a little surprised that webapps deployed on Fedora 15 do not blow up left and right. I may be missing something obvious. [Like that perhaps they all store their data in databases, not filesystems. — Ed.] [link] 1 comment|post comment

What did Linus mean? [25 Jun 2011|10:55am] I trolled Jeff Darcy into railing for FUSE by pointing out a quote from Linus: People who think that userspace filesystems are realistic for anything but toys are just misguided. The pronouncement was made in context of overlayfs. And it looks like everyone who chirped in is concerned with the speed, rather than nasty implementation details. I do suspect though that Linus meant something else than the syscall overhead, even if the magic mlock() takes care of the most obvious deadlocks. Maybe we should ask Al Viro if FUSE has any deep-seaing issues with directory renames or whatnot. Darcy, BTW, hasn't learned Linus' stealthy ways and falls for his "I'm just a community manager anymore" line. He asked "since when has [Linus] understood anything about storage" and called Linus an "ideologue". But I remember well a couple of cases when best and brightest in kernel were stumped, until Linus gave up on waiting and fixed the problem. One such had something to do with some ptrace thing or other and it was a pretty bad firedrill. He demonstrated a sharp understanding of VM issues, too. I'm pretty sure that if Linus says so, there were issues with userland filesystems that their partisans gloss over or even fail to grasp altogether. Oh, and finally... I'm pretty sure that userland filesystems work just fine in network environments or when parallel access is arbitrated outside of VFS. I mean, just look at GoogleFS. Heck, you probably can implement a great Gluster node in userland of Windows or FreeBSD. Who cares what's in the black box? UPDATE: The "official response" from Anand Babu of Gluster is LOL: "GlusterFS got its inspiration from the GNU Hurd kernel." [link] 1 comment|post comment

Too much free time or what? [16 Jun 2011|08:52pm] I'm thinking about writing a GNOME-VFS module for Darcy's CloudFS. Actually, what I should be thinking is pulling it all in kernel, but that's not deviant enough. Also, I heard something about GNOME programmers writing code while laying in a pool of puke, which sounds more exciting that my old highly professional kernel environment -- I mean, for a hobby project. [link] 2 comments|post comment

Berrange on KVM with routed IPv6 [16 Jun 2011|08:40pm] Saw a blog post by Dan that explains how to set up IPv6 on KVM hosts with routed virtual networks. I pretty much did everything like he suggests except I used the "Rich Jones" prefix fd2d:acfb:74cc::/48 instead of Dan's 2000:dead:beef::/48. One thing that bothers me with all this is the need to have statics at the upstream router and the ICMP redirects. It would be totally kickass if Dan came up with some kind of "standard" Zebra config or found some other way to make it all automatic. The sticking point is that in case of OSPF you still need to configure the upstream router, and in a non-obvious way, too. Static routes are well understood. Back in the 90s, we just ran routed everywhere, which required zero administration, but that's not IPv6 compatible. Honestly, the more I look at challenges that virtual hosts introduce, the more I think that bridged networking should be the default in libvirt. The only reason I personally do not do it is that historically Linux bridging was a mess full of bugs, plus it changed in incompatible ways every 7 to 10 kernel releases. This may be not the case anymore, but I am too set in my ways. [link] 10 comments|post comment

Costs of systemd [15 Jun 2011|08:39pm] I am in process of searching how to configure system logging in systemd. Back when Lennart wrote his blog articles, he even included running rsyslog into examples of socket activation. But some time around April 2011, systemd subsumed yet another function. Whoever instigated the takeover of logging thought it would be fun to stop updating /var/log/messages altogether and instead pollute the dmesg with usermode messages. He also neglected to document this in any way. If Lennart (or his overly smart disciple) were made to compensate frustrated sysadmins for hours spent banging their heads on desks, he might have thought otherwise. But the beauty of the scheme is that the companies employing said sysadmins get to pay... In fact, Lennart even listed "availability of specialized consulting" for systemd as a plus. He is my hero, really. If I were 1/10 as evil as that man, my life would've been an amazing success. Aside from the training costs, there's also an insignificant resource cost. My side by side comparisons show RSS jumping from 1.3MB to 22MB (on top of total mapped size going from 19MB to 54MB). Basically nothing to talk about. I was never hit by the "systemd ate my CPU" syndrome. UPDATE: A message by Kay suggests that rsyslog needs to be run. It simply was not configured right in F16 Rawhide, a small bug somewhere. Just needs "systemctl enable rsyslog.service". A day was wasted, however. UPDATE M0AR: Andrey's message already has all the info, but I'm going to restate how systemd actually improves upon what we had previously. The confusing part is that systemd does not merely do socket activation for syslog, but acts as a syslogd. Once rsyslogd starts, it takes over. The dmesg then is saved in the usual way. This way, syslog service is available before rsyslogd is up, e.g. for initrd time. Why is this not documented? My guess is, as long as everything works smoothly, it is a transparent improvement, so needs no explanation. [link] 3 comments|post comment

Slasti and security amateurism [13 Jun 2011|01:49pm] I am going to commit a key sin of security: amateurism. Also, file this into the folder "The Crazy Things People Do Instead Of Just Using SSL". But it's not like I am hurting anyone but myself, so here's the problem: I'm going to improve the security in Slasti. Currently, the authentication is done by accepting a cookie, which is a hash of a random string and shared secret. The server rehashes the string with its copy of the secret and matches against the hash that client presented. Details are in main.py:login_verify(). One problem with this is that anyone with tcpdump on the same hotspot can capture the cookie and then present it unchanged to edit bookmarks. Basically, the whole value of this "authentication scheme" is to prevent rogue bots from re-dowinloading export.xml all the time. I would like the next authentication scheme to withstand snooping and replay. The MITM problem is too hard, so I ignore it. At first, I was going to support RFC-2617 Digest Authentication. Experts thought it through, it seems to cover exactly the kind of attacks that I expect, and its client is built into the every browser. A win all around. Unfortunately, I cannot find a way to implement it in a WSGI application, because the API does not pass the necessary headers. After determining that, I started tinkering with cookies, but anything that I can come up with seems too vulnerable. So, what I am thinking about now is a challenge-response scheme, with client part in Javascript. At least it gives me an excuse to play with JS. P.S. After thinking about the problem in general, I am starting to get curious just what exactly all the web forums, LJ and WP use. It should be trivial to crack any account as long as you capture the login cookie. Are there any tricks? [link] 2 comments|post comment

navigation [ viewing | most recent entries ] [ go | earlier ]