60 Hours a week? How about 4 uninterrupted hours a day?
..or...the AGI meetings will continue until morale improves.
Again, a silly news story leads to me yammering about some of my time in the Silicon Valley. If you want to get to the meat of my response to Sergey’s memo exhorting employees to work 60 hours a week to usher in the next great age of AGI, scroll down a page or two….
As you might know I used to work at Google1. For 14 years of my tenure there, in addition to my other tasks, I ran the git2 team. We provided developer infrastructure for Android and, later, for a number of other teams like Chromium.
In 2007 well before the first release of Android to open source, the Android team came to me and we discussed what version control system we should standardize on for the release and, indeed, for internal development going forward.
It was an idyllic time to work at Google: I was sitting on the second floor of Androids’ building 44 with a lovely window letting light in, just so. My office featured a blue comfy couch that would attract certain frequent visitors. The first notable visitor was Ficus’ sweet, enthusiastic dog Ollie3. , but Ollie would pop in and do a few laps as Ficus walked him past after a walk.
The second frequent visitor was board-bring-up engineer Brian. I would even invest upon Brian a title appropriate to our relationship.
I briefly helped with what seems to be a now-retired or never launched Diné (Navajo) language interface to Google’s search engine. I know that Craig C nailed Cherokee down but I now find myself wondering if Dine’ ever made it.
Regardless, during this period, I figured out the Navajo translation for “Warrior who sits on the couch and talks” but I find it difficult to find now. I thought it was something like ‘Hashké dah sidá yááłti ’ though ‘’naabaahii dah asdáhí’ seems more correct in retrospect. Other translation sites says its ’naaldlooshii hólǫ́ holne'
Anyhow, I had a really comfy couch, is what I’m saying. And it would sometimes be occupied by Ollie, and sometimes by Brian .
What does this have to do with Version Control? Originally, the Android kernel/systems team, people like Brian, were on Git. Why Git? Well, flippantly, who the heck wants to develop a Linux kernel on some non-git vc?
Everyone above the kernel and systems software level were largely using Google’s massive perforce installation4. So the various surface flingers, apps, launchers, dialers, the calculator, that sort of thing, we’re all on perforce5.
As we approached the first customer shipping phones through HTC/T-Mobile in 2008, we wanted to release the Android Open Source Project to the various stakeholders so they’d see we were serious about releasing the project as open source6. So we had a decision to make , Git or Perforce?
At first I told them Git, of course, though I did engage in a bit of hedging. They should tell the teams that we could support both version control systems through the Repo/Build tools we maintained but that wasn’t good enough really. In the end, the team needed direction, so I suggested they should go ahead and blame me for the decision to settle on Git as the only VC we would use for Android.
Git was the right choice for a lot of reasons. Android had a *lot* of combative stakeholders, both inside and outside of google. You’d have:
Handset makers like HTC, Motorola and Samsung.
Chipset makers like TI, Qualcomm, Mediatek and Samsung
Support chip makers making gps chips, inertial units, compasses, sensors, etc…
Carriers like t-mobile. Orange, AT&T and Verizon
Each of these wanted to have their software flow into the other tiers. But those patches could not ever, never, not ever-never be exposed to their competitors at their same tier.
So a chipset maker like TI didn’t want someone like Qualcomm to see their drivers as they would ‘find out’ about features ahead of launch. Navigating these competitive concerns while keeping android compliant with the GPL and Apache licenses was often difficult, to say the least.
Now, all these vendors wanted attention from Google and the android team, but they also wanted Google to tell their competitors to pound sand. And which team was the rope in this game of tech tug-of-war? The Android open source project, of course. How could we balance the review and acceptance of patches while not halting development in some weird computer industry marketing/competitive race condition?
This led to the team writing a tool (repo, later Gerrit) to balance all these needs, and git was really the only version control system that with it’s distributed nature that could come close to satisfying these needs.
The other thing about Git: Kernel developers *need* to be on Git. It’s actually that simple. Application developers could be anywhere, but putting a kernel/systems/board bring up person on perforce is productivity *suicide*. I’m not saying that kernel developers' crankiness was way way worse than app devs…but… well I kinda am as that was part of it, but …. No kernel team wants to be on not-git.
To wit: By the time we moved google’s kernel team to Git, they were at least 4 minor versions (2.2) behind the modern kernel (2.6) . It was debilitating and demoralizing, it hurt security, it hurt recruiting to the platforms teams we were building out and it was a hassle to backport 2.6 features to 2.2. It was honestly madness.
Back to Android: There was some grumbling from the developers, to be sure. This was nothing like the grumbling we’d get from the kernel folks if we had made perforce a dependency for android development. Over time, Android’s use of Git would lead to us buying hardware7 for the mainline kernel team and distributing android on kernel.org.
Anyhow, time passed, Android was *very* successful8. We were approaching the release of Ice Cream Sandwich9 and we knew that if we released android 4.0 on kernel.org , it would probably mean weeks if not months of the kernel.org going down as the demand on that server was going to be off the hook.
It would not be responsible for taking down Linux development! Linus, HPA and Jim Zemlin would probably be pissed at me/us, for one. For another, we were going to release Ice Cream Sandwich10 a scant few months months after the Kernel.org team had their entire infra replaced after being confiscated by the FBI Cyber team following a cyber attack. And while Bdale (HP) and I (Google) ponied up for new machines, at the time they were still smarting from that particular attack.
By this point at Google, I had pulled together a smallish Git team under Shawn Pearce (and I think git maintainer Junio Hamano had joined us at this point) to work on git infra. So in anticipation of the ICS demand, the team worked their tails off to adapt jgit to Google's backend file serving systems. As a result of this remarkable effort, Git-on-borg was ready for the ICS 4.0 release and the kernel.org team didn’t want to smash us with a hammer.
So, victory laps all around ,right? Well, yes. It was kinda great. Other teams across Google would use GoB, and it was all rainbow sprinkles and pegacorn11 dust.
Well, maybe not that victorious, but it was definitely good. A few years later, Chromium had adopted GoB. Gerrit had replaced (sorta) Repo and we were very happy and the GoB team was one of the strongest dev infra teams at Google.
And now, dear reader, this is when I started getting in an entertaining kind of trouble that you came to this substack for: Shawn came into my office one day and said that he was getting stressed out by all the meetings he was being asked to attend in the middle of his programming time. So I told him to turn them down and send them to me.
So at the time at Google, engineers would have no end of distractions killing their flow. I’d end up setting all team meetings, 1:1s with me and even lunch-and-learns ahead of the afternoon. Many of my employees seemed to find the time after lunch to be , well, sacred for getting work done. So we’d sometimes meet at breakfast, sometimes elevenses, sometimes lunch, but after lunch…we’d work.
Shawn would start to turn down folks and some, I’m afraid to say, would *not* take it all that well. I remember not-a-few times other teams telling me I was being disrespectful, or I was being a tyrant by controlling access to my team by having this rule in place. I didn’t honestly care, I just redirected them to the morning timeslots and that was usually the end of it.
But , seriously, some people absolutely hated me for this. They complained to my manager (who thought this was hilarious), some would even complain in our perf system that I was a jerk or something.
Let’s put a fine point on it: there is a real cost to interruption. Do you want productive developers? Don’t. Interrupt. Them. Leave them be.
So, Sergey, when you say12 ‘Why don’t people show the same verve they showed in the early days?” I’ll tell you why.
First off, I get it. It’s frustrating. But, you know there’s a lot of answers to this question Some are pithy. Some are cute. Others are clever, true, wrong, and correct.
I’m not going to waste readers' time13 talking about the economics of google equity grants in the 2004s vs 2024s. Nor how Google’s workforce isn’t a bunch of single folks in their early 20s with no kids. Nor how most Googler’s don’t have staff to handle laundry, or car appointments, or doctors appointments. I won’t talk about how it’s interesting to see those benefits that made a lot of sense when Google was hundreds of people, you know, the concierges, the onsite doctor, the laundry service and machines and the creches and the actually good food that was on offer, would be phased out or diluted to the point of uselessness when Google was 1000x larger. Nope, I won’t go into all that.
Now, before I continue: I want people to know that I have very warm feelings for Sergey. Call it corny, or stupid, but I feel that his stepping back in 2014 was a sad moment for Google. I saw Google through an almost Metropolisic lens. Sergey was Freder: Google’s nerdy, super sharp heart to Larry’s brain and Eric's hand. When Sergey receded from the fore back then, and worse still when Larry backed off in 2019, it was the end of Google as something exceptional in computer science. It sucked.
Anyhow, back to the post:
As Google grew, The amount of productive time that one could enjoy a proper flow, a proper length of uninterrupted development time in , was under attacked from all sides.
Some of these attacks were well meaning. Some were pointless and silly. Some were legally mandated. Regardless, these attacks seemed destined to consume all available time.
Say you were an individual contributor and not managing anyone: You could figure that perf14 would consume at least a week a quarter, hiring committees for senior employees could be a half day a week of reading and feedback, interviewing for even junior employees could take another half day during the hypergrowth years. TGIF15 would commonly take a 3 hour block on friday afternoons. Team meetings would be a few hours a week. 1:1 with your manager might be a once every two weeks thing. Breakfast and Lunch, and hour or so? Skip meetings would be another hour or two a quarter. Company/Department wide meetings and summits…. Goodness knows. It was time consuming and your brain would sometimes resemble tapioca.
For managers, promotion committees would be another week a quarter, and figure a day or more a quarter for the recurring training on topics like sexual harassment and other legal trainings.
Lets not forget there were always training/coordination meetings on various pan-google internal efforts for your group like the utilization code yellow, latency code whatever, Google+, the 2010s would bring AI First, Mobile First, Steven Furst and I’m sure Gemini now…..
In 2010 or so I did the math and if you did all the mandatory training work, the hiring and interviewing and indulged in a talks @ google once a month or so, then, well, you’d maybe have 2 or 3 hours a working day of productive time left.
So Sergey, bubby, you want to recapture the dedication to work when we’d all work till 8 or midnight or whatever to get stuff done? I have a few tips: If you really want to get people productive, maybe let them actually work. Get them up to speed on the platforms, give them good guidance and then let them work. Those aren't small goals, by the way. Google has been struggling with engineering intake for at least a decade. Maybe it’s gotten better since I left? I hope so.
It’s funny, my group maintained the third party system, we made it easy to bring in, safely, the world's open source code so Googlers didn't have to re-write and we could start, as Switzer would say, on third base
These AI tools are actually a lot like the great wealth, the great resource, that open source code represents. You can literally start your unique interesting code on top of the work of giants, as processed by an LLM, sure, but that’s an article I already wrote!
Also, working for Google today is vastly different from working for Google in 2004, or even 2014. The singular vision of the 2000s isn’t there anymore. I hate to say it, but the best thing might be for google to break itself up so the Googlets can concentrate on what is important to them and not on the dependencies of the conglomerate. Some companies do well in complexity of mission. Even with Sergey returning to AI tools, I don’t think Google has the leadership or vision to do so. When everyone is looking out for their own skin, to the exclusion of the organization, there will be no forward movement. And Morale? Please. It’s gone. Sadly, you can’t repeat meetings about AGI with Gemini until Morale improves.
The other thing: The best developers know that once they’ve been working singularly on a problem for 4, 5, 6, 8 hours in a day, their accuracy and productivity drop into the toilet and they’ll spend more time fixing their stress induced screwups than creating good code.
So Sergey, if you want to rebuild an esprit de corps around a shared mission it will take years of consistent guidance and leadership, not drive by micromanagement and plaintive memos that only make people roll their eyes. Google needs to utterly change how it runs engineering teams.
I might point out that Google laid off, didn’t replace, alienated or absolutely failed to develop internally the kinds of people who were really good at this kind of engineering leadership. And even great sci-fi AGI LLMs aren’t going to be able to replace what Google has lost.
So if Google really wants to return to the sense of shared purpose and dedication we enjoyed 20 years ago, Google will have to make fundamental changes to how it runs programs and engineers. That’s probably not going to happen short of a complete rework of management, a breakup, or a HP style dissolution over time.
Well, this was more of a downer than I expected. I still have pals all over Google, so, uh… Buck up, little campers! Maybe they’ll pull it off! Yeah, positive positive positive! The futures bright, I can’t imagine Google descending into the madness that gripped the HP during the Fiorina days.
I mean, there’s no way they’re start piping mandatory weekly company meetings over ceiling mounted speakers, into every location, including the restrooms, updating you on Gemini’s latest thing?16
Right?
Right?
You know, the one we all switched to when Alta-vista wasn’t providing nice results anymore. They used to do search.
I didn’t know Ficus as well back then. We would end up skiing together after Google, and so now I can heartily endorse Ficus as being pretty neat.
This was before google rewrote perforce to run on top of google infrastructural code and well before we moved Git on top of some of the same code, which we called Git-on-Borg, or GoB for short.
Please note, I’m not coming down on perforce, it just wasn’t git.
Under a patent granting license, too. This was during a time when … other os vendors were … well, patent shenanigans were wild back then.
With Bdale at HP. Credit where credit is due!
At the time there were about twice as many androids, (200m), in use than iphones (90m), with Nokia coming in a not so distant 3rd (70m).
Can I take a moment to mention that I was really into the name Frangipane for the F release. But they went with FroYo as , well: “Chris, no one knows what Frangipane is, so leave it be” So, whatever, fine, but it was a better name than Froyo. I know that Yogurtlands were packed throughout the Silicon Valley and they were tasty, but jeez! I’m not bitter, who said I was bitter.
No, I didn’t lobby for a better name. I didn’t have the spirit for it after the loss of Frangipane.
I’m paraphrasing from the article here…
okay, maybe a paragraph….
Google’s quarterly 360 review process. It would go to twice a year around 2010 or so, I don’t remember when.
A company wide meeting that would review the week's accomplishments, with good food!
Seriously, this happened at HP, even in the bathrooms you could not escape it. They also didn’t feature Frangipane in their cafeterias.