Saturday, July 13, 2013

Lock Free - Part Deux

It's been a while since my last post and I've spent the last year working on lots of cool stuff at Valve. I worked on the item importer for Team Fortress 2, helped with the Linux Steam launch, and am currently working on very cool unannounced tech, which I'm looking forward to sharing with people soon.

In the meantime, I'm lucky enough to be sitting across from Bruce Dawson, who is one of the most amazing people I know when it comes to crash analysis, performance analysis, and "what the heck is my system doing?"  His blog is full of all sorts of interesting things, and you can check it out at It also turns out that he was the author of the GDC talk that I saw a few years ago which got me interested (and afraid of) lockless programming. It's a small world! :)

So occasionally I like to break out of what I'm doing day to day and catch up on SDL work, which always has different and interesting challenges.

This week I went and fixed a bug in SDL regarding OpenGL context caching. An OpenGL context is made "current" on a given thread, and each thread can have a different context active. SDL was caching the current context in a global variable, which is completely wrong.  What I really needed here was thread-local storage.

No problem, I'll just add it to SDL!

It turns out that each supported OS has the concept of thread-local storage, and all I need is a single slot of OS thread-local storage to implement a full set of SDL based thread-local storage. However, by its very nature, multiple threads may allocate from it simultaneously and I needed an efficient way to make sure only a single thread allocates the first OS slot.

No problem, SDL already has spinlocks which are a good candidate for quick initialization like this:
static DWORD thread_local_storage = TLS_OUT_OF_INDEXES;

    if (thread_local_storage == TLS_OUT_OF_INDEXES) {
        static SDL_SpinLock lock;
        if (thread_local_storage == TLS_OUT_OF_INDEXES) {
            thread_local_storage = TlsAlloc();
    return (SDL_TLSData *)TlsGetValue(thread_local_storage);

I had Bruce come over and look at it since I knew he was well versed in this sort of thing, and he pointed out that even though I'm using a lock, it's really a form of lockless programming since I don't unconditionally acquire a lock before testing state.

I thought I was pretty safe, since I was double checking the condition after acquiring a lock, so if multiple threads tried to acquire the lock simultaneously only one would actually do the initialization. The assignment is an atomic operation, so if another thread passed the if conditional it would have a valid value in thread_local_storage and would be fine.

Bruce pointed out that at the machine level it's a giant soup of memory and instructions, and memory accesses inside TlsAlloc() may be reordered relative to the assignment of the return value.  As compilers get better at inlining during link time optimization, there may not even be a function call and return value. This is a common problem with the double-checked locking that I'm doing here.

What I really want to do is guarantee that all data initialization involving TlsAlloc() are written to memory before any the write to thread_local_storage ‘publishes’ them. And conversely, I want any other thread to be sure it can see all the data associated with thread_local_storage before it starts using it.

The right tool here is memory barriers. On x86 and x64 architectures memory barriers are simply instructions that prevent the compiler from reordering CPU instructions, since the hardware is strongly ordered (other CPUs see writes in the order they occur.) On ARM and PowerPC CPUs memory barriers are hardware instructions that force synchronization. Jeff Preshing has a great blog that explains how to think about them, along with lots of other great information about lockless programming:

Essentially, the initializing thread performs a bunch of work, then sets a flag variable and issues a write-release. All other threads check the flag variable, perform a read-acquire, and then continue using the data signaled ready by the flag.

Applying this here, we end up with:
static DWORD thread_local_storage = TLS_OUT_OF_INDEXES;

    if (thread_local_storage == TLS_OUT_OF_INDEXES) {
        static SDL_SpinLock lock;
        if (thread_local_storage == TLS_OUT_OF_INDEXES) {
            DWORD storage = TlsAlloc();
            thread_local_storage = storage;
    return (SDL_TLSData *)TlsGetValue(thread_local_storage);

It's tempting to simply add the word "volatile" to the declaration of thread_local_storage, but that's not exactly what we want. Visual Studio has implemented volatile as having the correct memory barrier semantics when using /volatile:ms, but other compilers simply guarantee that the they won't reorder reads and writes around it, and the CPU is completely free to ignore that. In fact, it may be safer to define a new keyword that indicates that multiple threads access a variable but doesn't have volatile semantics, and always use memory barriers when appropriate.

The final code in SDL has error handling and a fallback to a slower generic implementation in case the process is out of thread-local storage slots, but I believe this is a correct and complete example of thread-safe initialization.


Saturday, June 2, 2012

Work/Life Balance in the Games Industry

I've been thinking a lot about what's most important to me in new opportunities...

I want to learn new skills, be a part of a talented and healthy team, and work on great things in the games industry. But as I realized a year ago, a huge priority for me is also to have a happy home life along with my professional life.

Work Life Balance

These words are tossed about to the point of buzzword cliché, but what they really boil down to is finding ways to have a happy, healthy, sustainable, efficient work and home life.

There are horror stories in the industry about people working themselves to the bone, crunching for months and sometimes towards an ever receding horizon of a project ship date, destroying their families in the process.  This was viewed as acceptable by big publishers because the title would ship, everybody would crash, take a few months to recover, and sometimes be laid off or move on to the next project and start the cycle all over again.

I think everyone agrees at this point that it's a terrible model for development.  As companies are starting to switch to a service model, they can't release a product and then have the development staff crash.  You can't have people burn out and leave the project, because it's when the title ships that the work really begins.

So what continues to drive the desire for overtime?  From a high level management perspective, the hours an employee works are the only knob they have to turn to increase production output.  And for short periods, this really works.  Need to get 6 weeks of work done to meet a deadline in a month?  Have everybody work 12-14 hours a day.  Production is increased, the deadline is met, success!

The problem of course is that this is not sustainable, and there is always a drop off in efficiency over time and a recovery period necessary.  Different people have different levels of productivity increase from overtime, and the productivity drop and recovery time vary significantly based on age, stress and other life factors, but the general pattern is the same:

In the world of game development engineering, the productivity drop off can be much steeper, because tired people tend to take longer to solve problems, solve them in non-optimal ways, and often introduce bugs that cause wasted time in the Q/A and bug fixing process.

Quality of Work

The first part of good work life balance is having a healthy work environment without extended overtime. But in today's world of rapid iteration and/or constantly increasing game features and scope, how do we achieve this?

  • Good planning and scoping
  • Good time estimations
  • Good communication
  • Improving efficiency
  • Stop working
Good planning and scoping is the first step, and something all teams try to improve.  As a team gets experienced, it gets better at defining goals and scoping a project.  Many teams plan rapid iteration and gradual expansion of scope after core elements are in place.

Good time estimation and frequent review of progress are crucial to good planning and scoping.  It's important to factor in time for iteration on customer experience and bug fixing into the overall schedule. When I'm estimating time, I usually estimate how long I think a feature or task will take to implement, then I triple it.  This usually gets pretty close to the actual total time cost, including handling unforeseen issues, making required improvements and bug fixing.

Good communication with management leads to trust on both sides (hopefully!) and helps reduce stress in the workplace. Good communication with production staff helps tighten the planning feedback loop and makes sure that re-scoping happens early enough so that teams can account for it and minimize wasted work.  Good communication among engineering staff is really helpful to share key information or smart ideas to accelerate work, or even make it unnecessary.

For example, when I was leading the social systems team, I noticed that some people new to the team (including me) were spending a lot of time digging through code, or struggling finding the right way to solve problems in the codebase. So one of the things I implemented was a rule that before starting a task or trying to fix a bug, the person should stand up in our area and mention they were working on it and ask if anyone had any advice.  This didn't affect the old hands much, since they held most of the knowledge already, but new people achieved about a threefold improvement in efficiency and very quickly ramped up on the systems and history of design decisions.  We also caught some cases where new designers were making requests that conflicted with previous design decisions and saved ourselves work that would have been thrown away.

Improving efficiency so a team can get more work done more effectively is always a good idea. Work done improving Q/A processes means better morale, better bug flow, and better interaction between Q/A and engineering. Work done to reduce iteration times, build times, export times, game startup time, etc. repays itself in spades. Experienced engineers multi-task so while waiting for things they're thinking about the next problem, or reviewing the code for correctness, etc., but there's always some time wasted in context switching, so reducing waiting time is almost always good.

The flip side of course, is that if everything is too efficient, it's easy to forget to take breaks and slowly become fatigued, which leads me to my next point...

Stop working!

If I am sitting there trying to figure out a problem and it's taking a while, the best thing for me to do is to get up, walk around and do something else. I'll often come up with the solution while away, or when I come back to the code I'll have a fresh perspective and get the solution quickly.

Solving more challenging problems are always easier for me if I go home, spend time with my family, and have a good long shower in the morning.

Both at Blizzard and 38 Studios I was a part of a hack circle that met mid afternoon, and invariably I would be much more productive after I got up and got the blood flowing, and the adrenaline pumping.

A friend of mine who led a team at Sony once told me that when his engineers came to him with a problem, he would tell them to go talk to a tree.  He wasn't being flippant - getting out and talking through the problem in the sunshine near a convenient source of elevated oxygen actually improves brain activity and helps solve problems.

Quality of Life

The second part of good work life balance is of course, life.  We are social animals and we need friends and good relationships, good food, fun and exercise to be at our best.

For me, having a happy home life is extremely important to how well I do at work. If my wife isn't happy, ain't nobody happy, and I can't really concentrate and be effective on the job. If I get smiles and hugs from my kids in the morning I have smiles for the rest of the day.

Coping with Crunch

So the planning failed, you have to work overtime to get the product ready for a milestone and production and management are willing to accept the costs. What do you do to stay healthy, happy, and efficient?

The first thing to do is know your priorities.  Make sure that you're reaching for specific goals instead of just adding more time to the stack.  Find ways to work smarter so that you're doing exactly what needs to be done to achieve those goals.  Don't need that afternoon meeting? Ditch it! Wasting time context switching with various tasks? Mark off blocks of time that are devoted to certain types of work. Some of these kinds of things are good to do anyway to improve your work processes, and some are temporary measures to let you stay super focused.  You might find that getting hyper-focused is all you need to get the work done without any overtime at all.

The second thing is to stay flexible and know your limits. Don't just work exactly 12 hours because someone mandated it. Some days you'll be more tired and need to stop early. Other days you'll be on a roll and want to finish something up for another few hours.  Remember to take your breaks to stay working efficiently and able to focus. If you need to stop, be honest with yourself and your team. It doesn't do anyone any good to make tired decisions and write poor code that will cost everyone later.

Be creative and find ways to make things work best for you. Don't neglect the important things in life even if it feels like work is temporarily more urgent. If you only have one evening available, maybe make it a date night. If you're missing meals with the family, maybe they can join you for lunch, or maybe you can swing your hours early or late so you get breakfast or dinner with them. You might want to get out of the office on breaks for a change of scenery and a chance to decompress. Take some friends and go running to blow off steam...

Be patient. Extended overtime is hard on you, your family and your team, and a good attitude goes a long way to making it easier for everyone.


Having a healthy balanced life is a big priority for me, and luckily a lot of companies know the importance of work life balance and are working to create a good sustainable environment for their employees. I'm excited to meet more of them over the next few weeks.

38 Studios Coda

So I haven't blogged in the past year, since I was busy at 38 Studios and much of what I was working on was confidential.  Suffice it to say that it was indeed my dream job. Working with good friends on an amazing project with the full support of the company and a great work/life balance.

Lots of people have talked about what happened at the company, so I'll just give the short and sweet version...
The company ran out of money, and instead of spending the last of it gracefully shutting down, they prayed for a miracle that never happened. Yes, I believe the governor's press statements didn't help, but the company was so close to zero that an extra month of operation may not have been enough. The story is tragic, and gets more so each week as more details come to light.

But, I am thankful for one of the best game development years of my life, and I am amazed and happy at how we, our friends, and the industry as a whole have been so supportive.  We intended to change the world with our game, but we are changing the world through our shared bond and the knowledge of what we had, what we did, and what can be.  I feel like I have a family that will last for the rest of our lives, spreading throughout the industry and beyond.

Thank you!

Sunday, May 15, 2011

38 Studios

So once I decided not to start my own company, I started looking around for a company in the games industry that really had employees, and more importantly, employees' families at the forefront of their priorities. I wanted a place I could raise kids, have a life, and make great games with my friends.

38 Studios stands head and shoulders above every other company in the industry in this respect. 38 Studios was founded by Curt Schilling, former Red Sox pitcher, and his passion is building teams. He knows that to create a great game you have to have great people, and treat them and their families well.

Curt will move heaven and earth for his family, and if you're on his team, you're family. Rarely have I met anyone as genuinely passionate and good hearted as he is. He has strong opinions, but he is also wise enough to trust his team and value their opinions and expertise.

When I came to visit I was very impressed by everyone I met. They are smart, nice, and honest with themselves and each other. They realize they have a tremendous task in front of them, creating a company, creating a great game, creating a world. But each time I've seen them approach a problem, I've seen them solve it in innovative and great ways. Each time I've seen them choose between the expedient path and what's right for the team, they've done what's right for the team. Not only does that hold true at the top levels, but that attitude, flexibility and self-empowerment is there in everyone I talk to.

38 Studios faces challenges, for sure. They are trying to do something that hasn't been done before. They are a privately owned company making an MMO, and trying to do it in a healthy sustainable way. They are growing quickly and trying to prepare themselves for the transition to a service organization while at the same time creating a quality content game. They are trying to find the right formula and the right people and they're trying to do it in a realistic time frame.

Can they do it? I don't know. But I am incredibly motivated to help them try, because if they can pull this off, this is my dream come true. A great company, great people, and a quality life for my family and me.

My first day starts tomorrow. Wish me luck! :)

Wednesday, April 6, 2011

Exploring the Galaxy

In 2008 I created Galaxy Gameworks as a small LLC to handle commercial licensing of Simple DirectMedia Layer on the iPhone. Since then I have been maintaining it at a minimal level in my spare time to make it possible for people to use SDL on iOS and other embedded platforms.

As soon as I made the concrete decision to leave Blizzard, my brain started exploding with ideas on how to expand Galaxy Gameworks into a full blown games middleware company. My vision was to build the company into three pillars each designed to help people make games:
  1. Software: Starting with SDL and expanding to meet important needs for small developers, based on my experience leading a development team at Blizzard Entertainment.
  2. Education: I have always been a big fan of student and indie development, and I know an awesome woman, Brandii Grace, who used to teach at Digipen and is on the board of the LA Independent Game Developer Association who could help start this.
  3. Consulting: This is a less developed idea, but there are lots of companies that need help at strategic points to help get their games shipped, and this would be a great service to complement software and education.
As soon as I left Blizzard, I blasted into overtime starting development on SDL and learning what I needed to build Galaxy Gameworks into a real business. I had lots of connections and people interested in what I was doing, and was really excited about the possibilities. I gave myself until the end of March to explore what is involved with creating a company and making my dream a reality, without committing myself.

It was a really amazing experience. I brought all the pieces together to start the company:

Company Vision
I have seen quite a few companies in my day, and I wanted to bring a company vision that incorporated the best of what I have seen, with an eye towards practicality and treating employees really well. I also wanted to have clear goals so that employees who are brought on and potentially given ownership have a good idea of the kind of company we are trying to make.

Virtual Office
I decided early on that I wanted to create a virtual office. This gives flexibility in hiring and reduces overhead, but means you have to work harder to find people who can be self-disciplined, stay motivated, and communicate well remotely. I put together a virtual office around Google services, and set up business phone and FAX. I also worked out the advantages and risks associated with distributed development, and talked to some companies that actually operate that way to find out how it works in practice.

Good branding and easy to use website are really important. Some of the important elements are a showcase for your products, good documentation, clean presentation, and easy navigation.  I hired someone to create a logo (the process is described in another blog post) and create a new website for a commercial presentation.  The result is at and while it's not complete, it's a good start.

Customer Service
I hired a friend of mine, Jen, to handle customer communication and start setting up hiring processes. She is is amazing at organization and has customer service and HR experience. She is handling the mailing list and forum, organizing bug reports, and handling customer inquiries and licensing. She also did research on the tax implications of hiring contractors and full time employees in other states and potentially other countries.

I have been working very closely with my friend Sheena, who has great technical writing skills and is learning programming, to put together high quality API documentation (work in progress at We also put together plans for tutorials and use cases with examples.

Business Plan
I worked with my wife Lauren and an old friend Jeremy on the draft of a business plan. Lauren is awesome because she is a technical writer by trade and can take large amounts of technical information and organize it in an easy to understand manner. Jeremy is awesome because he's spent the last 15 years or so in software sales and business development, and is extremely innovative and works well with customers.

Between the three of us, we came up with a great business plan centered around three products. The business plan really helped us focus on our goals and business strategy. We thought about our strengths and weaknesses (SWOT analysis), our competitors and potential partners, the trends in the games industry, and our costs and potential sources of income.

We got great feedback on our plan, and a few themes came up repeatedly:
  • What problems are you solving?
  • Keep it short - what's your elevator pitch?
  • Can you show a fast return on investment?

In response to the feedback, we cut one of the products to stay focused and reduce costs and ended up with a revenue and expense chart that looks something like this:

Investors like to see an exponential chart, and a return in the 6-18 month timeframe. :)

I'm pretty confident I could find some investors at the $300,000 price point and create a small sustainable company creating middleware in the games industry.

At the end of the day though, I realized that even though I have all the pieces I need and lots of support from friends and customers, it was taking all my waking hours and would continue to do so. In a few years when the girls are older and becoming independent that might be okay, but my time with them is precious and I want to have as much of it as possible now.

So, I'm closing Galaxy Gameworks for now, and the quest continues...

Monday, April 4, 2011

How I left Blizzard

Blizzard doesn't tend to be very open about how and why people move around at the company, and I still have people asking me what happened, so for those interested I thought I'd share my story.

As far as I know, nothing I'm writing here is confidential, so please tell me if that's not the case.

To understand something of the story, I really should start at the beginning...

After the Wrath of the Lich King launch, I was looking to move on to something new. I loved Blizzard and I didn't want to leave the company, but I wanted to work on a smaller team, a smaller project, with more responsibility. Unexpectedly, a job opened up for technical lead on a small unannounced project that was getting underway. It was a small project, with a short timeframe, at least for Blizzard, and it was a project that I personally was very interested in. I applied, and at the beginning of 2009 I was the engineering lead on a new unannounced Blizzard game!

I was very excited, and spent the first few months researching engine technology, getting some of the  infrastructure set up, and worked with the producer on a detailed technical plan for the project.  After that, since the designers had a solid idea for the game and were making great progress in the prototype, I started work on proving out the gameplay systems and hiring the most urgent engineering positions.

Come July 2009, the company sounded a call to arms to help get ready for the Starcraft II launch. There was much to do and each team loaned a person or two to help out. Since our team was so small, most of us went to help out, including our producer and UI artist. We kept a small crew of designers, an artist and our newly hired graphics engineer to keep the project alive during our tour of duty on

We kicked butt, took names, and Starcraft II launched smoothly a year later with 2.0.

During the course of that year, our producer was promoted to lead producer, and the designers made great strides in evolving the game design and the gameplay model. The graphics engineer had completed the core of the graphics engine and was roughing out the tools pipeline and content creation system.

When we returned from, I jumped right on getting our AI/gameplay engineer up to speed on the gameplay systems and started the herculean task of migrating code I had already finished over to the new gameplay model and tools systems that were developed while we were gone.

Three months later that work was being wrapped up, and I was called in to a meeting and told that there were issues on the team.  I had just worked out some issues with my gameplay engineer, but there were issues beyond that, which they needed to follow up on to give me concrete feedback.

Over the next week I implemented a great suggestion from them which improved my communication, our part-time producer led technology goal and milestone meetings which I had requested and were very helpful, I worked on a clear division of labor and responsibility among the team, and I had one-on-one meetings with all of my engineers to see how things were going.  As far as I could tell morale was improved, things were going great with my gameplay engineer, and we had a clear plan for the future.

I was feeling great, and was told that even though things had improved, there were still unspecified issues that were being looked into.  The meeting was very uncomfortable as though something were being danced around, so I mentioned that if the leadership of the team thought it was best, I was willing to step down.  The meeting suddenly relaxed and I got nervous.

A few days later, I was asked to interview someone for technical leadership of the team!  I asked for the chance to talk to whoever was having problems, and was told that would be awkward.  A few days after that it was confirmed that they had already made the decision to have me step down, but that the team leadership was there to help me succeed however they could.

So after a little primal scream therapy, I sat down and really thought about what went right, what went wrong, and what really was important to me.

One of the factors that I think contributed was that we hired really experienced people in the industry, and I hadn't had experience building a small game from the ground up.  Instead of mandating how things should be done, or bringing a set of best practices, I tried to work collaboratively to help us develop the best tools and approaches for our unique needs.  I think this was a creative and good way for us to go given our talents, but I think it increased stress and was possibly interpreted as weakness on a team already feeling time pressure.

So, assuming that was a factor, and since I honestly did not know how other game companies work, I set out on a quest to tour the industry and learn as much as I could about how companies other than Blizzard operate.  I was interested in everything including technology, development processes, team composition, project planning, and business relationships.

I want to stop here and thank everyone who helped me in this quest.  Lots of companies opened their doors and under NDA showed me everything that I wanted to learn about.  Blizzard management too, went way above and beyond the call of duty and introduced me to people who could help me learn about the industry.  I learned that there are a ton of great companies out there, and lots of good ways to approach things.  Thank you all! <3 :)

Armed with a good perspective on the industry and how games are developed, I looked back over how we set up our project and came to the conclusion that it was entirely reasonable, from a tech perspective, and felt comforted.

With that awesome tour completed, I turned my attention back to my future.  My options were to stay on the team as an engineer, transition to another team at Blizzard, leave and start my own venture, or join another company in the industry.

I realized I didn't really want to stay on my team, because even though I love the team and I still think it's an awesome project, I didn't feel like I got any support in that role from management, and I never was told why I was asked to step down.  I don't feel like it was personal, that the management really was trying to do the right thing for the team, but I still didn't want to stay in that environment.

I thought a lot about joining another team at Blizzard, or joining another company, but I realized that after launching Wrath of the Lich King, booting engineering on a new team, and shipping new on StarCraft II, all in less than two years, I was starting to feel burned out.  I also looked at myself and realized that sometime in the last ten years I had become a family man, and what I wanted to do more than anything else in the world was spend time with my family.

I took a deep breath and relaxed, feeling my shoulders unclench and my face break out in a smile, and knew I had made the right decision.

Friday, March 11, 2011

Life and Love

Every once in a while you see something that takes you completely out of your day to day routine and reminds you that this world is a wide and wonderful place.

Spider Robinson is one of my all time favorite authors, and he has been entertaining me and teaching me about love and laughter most of my life.  I finished the Callahan series (again) tonight, and was touched by the plight of Doc Webster, who passed on with a brain tumor in the final story.  I finally noticed in the back of the book contact information for the American Brain Tumor Association:

I got on the 'Net and found his website and found that his wife had passed on, in peace, almost a year ago and was able to piece together a little of what his world has been like since then.  He is a brave man with a wonderful family. I'm lucky that I haven't had to deal with the death of someone close to me yet, but this world is richly textured, and people deal with pain and loss and anger and joy and triumph every single day around the world.

If you haven't seen his work, you can check it out in traditional and audio form here:

To all of you who are dealing with and have dealt with loss, I wish you the best, and while I know nothing can ever replace those you've loved, there are still more out there who have loved you, love you now, and will bring you joy in the future.