The 6Ps


The 6Ps is a useful framework for tackling problems and crafting durable solutions. The 6Ps are Premise, Purpose, Principles, Priorities, People and Plan.

J Allard introduced this approach at Microsoft in the 90s. His framework didn’t include Premise, so it was called the 5Ps. J’s conjecture was that any successful project eventually lands on the 5Ps—and the best projects do it up front.

I use this framework all the time, and not just at work. Whenever I’m crafting a plan I think about the 6Ps. The 6Ps work for big things like shipping game consoles to little things like organizing your garage. They work for organizations, for projects and for life goals. The most important Ps are also the hardest to get right: purpose and principles.

Here’s a quick example to make it real.

The Garage

Premise: The garage is a disaster. There’s no room for the car, and it’s hard to find the things we use.

Purpose: Permanently park the car in the garage while making it easy to find the garden tools.

Principles

  • Minimalist: Embrace living with less. Be willing to throw stuff out that we never use.
  • Organize for Efficiency: Put the things we use regularly where they’re easiest to find.
  • Low-cost: Do the work ourselves—or get the kids to do it!
  • Enduring: Assume we’ll acquire new stuff that needs to be integrated into the solution, but without losing space to park the car. 

Priorities:

  1. Parking the car in the garage
  2. Access to garden supplies
  3. Getting rid of stuff we don’t use
  4. Cost efficiency

People: Me (Directly Responsible Individual), spouse (Involved), kids (Assigned)

Plan: If we haven’t used garage item in 5 years, donate or toss. Spend May weekends cleaning. Spend June organizing. Park the car in the garage permanently no later than July 4.

For a small project, you can see that the 6Ps fit on a page. For a large project, the first four Ps should still fit within a page or two, but People and Plan can vary depending on the scope of the project.

Premise

The Premise defines the Why. The Premise describes the current state, the lay of the land and why the issue is worth solving. The Premise includes the problem statement, but also provides background and context. Defining the Premise first is important because there may be multiple solutions, so it can be worth reminding yourself of the context periodically. Ensure your team and/or customers agree the Premise is well defined before moving on to the Purpose.

Purpose

The Purpose defines the Goal. The Purpose is short and simple, no more than a few sentences. It’s the north star, the deliverable and the mantra for the team. It can be the mission statement, and like a good mission statement, it should be easy to remember—although it’s often challenging to formulate. The Purpose informs your team, your stakeholders, and your customers about the reason you have a Plan. Examples:

Project/Product/CompanyPurpose
Google ChromeProvide users with a fast, simple, and secure web browsing experience
NASA’s Gemini programDemonstrate the ability to orbit and maneuver manned spacecraft, rendezvous and dock with other spacecraft, and practice reentry and landing methods
Original Star Wars filmCreate a new kind of science fiction movie that is broadly appealing, pushes the boundaries of filmmaking technology, and promotes values such as courage and the struggle with tyranny
Sonos companyFill every home with music

Principles

The Principles define what’s truly important for success—and what’s not. Principles create guardrails for the project. Principles are sacrosanct. Principles help the team make quick and durable decisions. Principles define how the team will act. Principles create a shared consciousness within the team that allows the project to continue even if leadership changes. When the Plan needs to change, Principles help ensure that the Plan still addresses the Premise. When new people join the project, Principles are the most important thing for them to internalize. Principles are sometimes called tenets or core values.

Principles often involve deep discussion with stakeholders and customers. For example, should the product be high price and high quality but lower market share (principle: high quality), or should the product be lower price/quality but larger market share (principle: broad reach).

Principles are often adjectives or descriptive nouns. They can be single words or short phrases. Sometimes principles may conflict (see Priorities). A list of some potential principles appears below. All of these are valid, but they’re presented as inverse pairs to highlight the tradeoffs that come with good principles.

  • High Quantity/High Quality
  • Unfiltered/Curated
  • Inclusive/Exclusive
  • Experimental/Long Lasting
  • Open/Closed
  • Safe/Exciting
  • Transparent/Protected
  • Innovative/Comfortable
  • Simple/Powerful
  • Inexpensive/Boutique
  • Minimalist/Full featured

When you think of popular products, projects and even successful people, you can often derive the guiding principles that led to their success:

Product/Project/PersonPrinciples
iPodMinimalist, portable, integrated
Apollo space missionsInnovation, risk-taking, collaboration
DisneylandStorytelling, immersion, family-friendly
Twitter/XConciseness, openness, real-time communication
New York Times gamesApproachable, rewarding, curated, short time well spent
Dalai LamaCompassion, non-violence, universal responsibility, inner peace

Principles are important to get right because everything else is informed by the principles. Projects can struggle when principles are unwritten and develop organically over time. Ideal projects discuss and define principles early, then communicate them broadly.

Priorities

The Priorities define the important deliverables and the important customers. Priorities define how to make tradeoffs between conflicting principles. Priorities help control scope. Priorities can define the order in which things are done. Priorities can also indicate what’s not important (non-goals). Having clear Principles and Priorities enables teams to rapidly make decisions themselves. Prioritization is temporal and may change over the course of the project, but when priorities change they should be re-communicated to the team.

People

People includes everyone involved in the project. People includes ownership and accountability. The RACI model (responsible/approver/consulted/informed) is one useful framework for considering people for larger projects. The People component might also include roles that still need to be hired. People and Plan are usually closely related and often considered in tandem.

Plan

The Plan is the What. In the real world, people gravitate toward creating the plan first, but in the 6Ps framework the Plan is last. The Plan is informed by everything above. The Plan can change as long as it fulfills the Purpose, Principles and Priorities. The Plan includes milestones, dates, deliverables, and forcing functions such as conferences/events. The Plan can be complex and involve multiple approaches, but many other frameworks are available to help with this aspect.

More Examples

Original Xbox Console

Premise: Microsoft owns the office (“A computer on every desk”), but hasn’t made headway in the living room.

Purpose: Gain a competitive market share in the living room with an entertainment hub for gaming

Principles

  • Bet on broadband: embrace multiplayer gaming and social connection
  • Power and performance: smooth and immersive gameplay
  • Developer friendly: easy to port from PC, great tools, APIs, docs
  • Safe and secure: level the playing field; limit hacking, limit cheating
  • Leverage existing investments: NT kernel, Win32, DirectX, Visual Studio

Priorities

  • First party developers
  • Third party developers, ranked by potential game sales
  • Ship the console first, Xbox Live second

People: Build a team of experts from the gaming industry and from within Microsoft. Establish relationships with chip and hardware manufacturers to create a strong supply chain.

Plan: Ship a gaming console to compete with Sony and Nintendo. Ship a broadband multiplayer gaming service with secure networking.

C++ Language

Premise: There’s no programming language (circa 1980) that provides the performance of C combined with the abstractions of object oriented programming and generics.

Purpose: Deliver a general-purpose programming language with a bias towards systems development that supports data abstraction, OOP, and generic programming.

Principles

  • Practical: useful, mainstream, easy to adopt
  • Efficient: low-level control of system resources
  • Zero overhead: you don’t pay for what you don’t use
  • Compatible: a superset of C
  • Portable: works across a wide range of platforms
  • Object oriented: support reusable modular code with encapsulation, polymorphism and inheritance
  • Flexible: support multiple programming paradigms (functional, OOP, generics)

Priorities

Features

  1. Encapsulation and inheritance (C with Classes)
  2. Polymorphism (virtual functions, function overloading)
  3. Generics: templates and exception handling

Customers

  1. Bell Labs
  2. AT&T
  3. Universities

People: Bjarne Stroustrup and other Bell Labs contributors

Plan: Build C with Classes and go from there

References

J Allard’s original model: The 5Ps: Achieving Focus in Any Endeavor

Robbie Bach’s simplified model: Learning the 3P Strategy Framework

Posted in Uncategorized | Leave a comment

Tell Me About Yourself


I’ve been interviewing a lot of software engineers lately. I typically start my screen with a question like “Give me a two minute overview of your career.” The goal is to give the candidate some time to settle their nerves and learn something interesting about them at the same time.

I’m surprised how rare it is for candidates to stick to that “two minute overview” instruction. Five minutes isn’t uncommon, at which point I cut them off and move on to the next question, while writing down something in my notes about “doesn’t follow directions.”

I’m also surprised the way candidates interpret the question. Maybe it’s just nerves, but even experienced candidates don’t take advantage of the open-ended nature of the question. The typical answer is a historical recap. “I graduated with a degree in CS. My first job was at Amazing, Inc, where I worked on algorithms for efficient cargo ship loading. Then I moved on to ….”

I saw all of this on your resume already. This is your opportunity to go deeper. Here’s what I really want to know:

  • What 3-4 skills do you have that I most care about?
  • What are you really, really good at?
  • What gets you up in the morning excited to come to work?
  • What makes you special or even unique among candidates?

The best candidates will talk about their gaps, too. Examples:

  • “I’ve always wanted to work on machine learning, but I haven’t had the chance yet. I’ve been reading all the research.”
  • “I led a team, but what I learned is that I’m better at writing code.”
  • “I listed C# on my resume, but I haven’t used it three years. My best programming language is C.”

The next time you’re interviewing and you hear anything like “tell me about yourself” or “what’s your background” or “give me an overview of your career,” the interviewer is giving you chance to go much deeper than surface level. Take advantage.

Posted in C++, Computers and Internet, Management | Leave a comment

Quantum Management


At the recommendation of my current manager, Laura, I read this article from James Everingham, the head of engineering at Instagram. It’s an accurate description of my management philosophy. The gist is that managing by interjecting your views into the conversation typically results in non-optimal solutions. In the framework of quantum physics, you kill the cat. A better way to manage is to frame problems and define ambitious endpoints. Then let the team figure out the best way to get to the endpoint. Assuming you’ve followed another one of my principles — hire people smarter than you — you end up with an optimal solution.

 

Posted in Computers and Internet, Management | Leave a comment

References For The Win


I love single character code changes that generate massive performance improvements.

Unreal has had static code analysis passes not too long ago. It’s surprising that SA didn’t identify this issue. It’s also unlikely that programmer code reviews would have detected something as tiny as a missing ampersand.

Just another reminder of how difficult it is to identify performance problems.

Posted in C++, Computers and Internet, Game Programming, Unreal | 2 Comments

July 4, 1977


It was the summer before 4th grade. Jimmy Carter was president. Apple Computer had just incorporated, and the Atari 2600 video game console had been released, but personal computers were still a novelty. I wouldn’t use a PC for many more years.

I was a voracious reader who loved anything involving knights or pirates. I spent hours building castles and sailing ships with Legos. I watched Star Trek episodes, of course, but science fiction wasn’t really that compelling. The world of Star Trek was too obviously fake.

We were visiting my aunt and uncle in Eugene, Oregon. On the 4th of July, we all went to the movie theater for a matinee. It was a new movie everybody was talking about. The theater was packed. We split up to find seats. I sat next to my mom on the left hand side near the front.

87d7bda81df5d4d3a97312c601dbccec

Ten words in blue appeared: “A long time ago in a galaxy far, far away….”

Then that opening brass fanfare and the now famous title crawl began. I was transfixed. Mesmerized. Spellbound.

Fantastic images filled my impressionable brain. The unbelievably massive Imperial Star Destroyer chasing Princess Leia’s tiny Rebel Cruiser. Then robots. Now rebel troops. Stormtroopers. Blasters. Incredible sound effects and soundtrack.

Then the gong and ominous musical pause. Darth Vader. Hell, yeah!

In those five minutes, my life changed. I had never seen anything like this. Even watching that opening sequence 38 years later, I’m still hypnotized. I challenge you to find a better movie opening.

To a nine year old in 1977, the world of Star Wars was absolutely real. Watch the droids in that opening scene on the Rebel Cruiser. C3PO’s shiny skin is scuffed and marred. R2D2 is downright dirty. All those little details convinced me — and millions of others — that Luke and Leia and Ben and Han actually existed in some galaxy far away.

That summer, I started reading science fiction. I started building X-Wings and Star Destroyers and TIE Fighters. My bedroom was plastered with photos and drawings and newspaper clippings of anything and everything Star Wars. And when we finally got our first PC, I started programming (text-based) Star Wars simulations.

Which ultimately led me to programming jobs, then video game jobs, then Xbox, and now virtual reality.

George Lucas gets a bad rap these days, but I for one thank him and the entire original Star Wars production team. You transported this fan into a life and career he never would have foreseen on July 3, 1977.

 

Posted in Computers and Internet, Game Programming, VR | Tagged | Leave a comment

Baking with Swarm


One of the challenges in game development is pre-computing static lighting. Emulating photons bouncing around in a virtual environment can require legions of compute cycles. This process is called baking. The end result is a lightmap, a compact data structure that represents the way static objects are lit.

The Unreal Engine suite provides software that makes baking lightmaps easy and fast. There are three components involved:

  • Unreal Lightmass: the app that does the actual lighting computations. It’s designed to run on many different PCs simultaneously.
  • Swarm Agent: a background app that runs on any PC that has been designated to bake lighting. The Swarm Agent invokes Lightmass to do bakes.
  • Swarm Coordinator: an app that runs on a single always-on PC that receives requests for lighting builds and determines which Swarm Agents are available to participate in the bake.

To build lighting, anybody running the Unreal Editor can kick off a bake by selecting Build Lighting in the editor. This results in a number of Swarm Agents across the local network participating in the build. The number is a function of the size and quality of the bake, and also depends on which Agents have free CPU cycles. Each selected Swarm Agents receives Lightmass along with enough data to run its portion of the bake. The combined results are ultimately assembled back into a complete lightmap.

By running Swarm Agents on many computers on your local network, lighting builds can be performed very quickly. What might take hours on a single machine takes only a few minutes with Swarm.

The documentation on Swarm and Lightmass is sparse. The best reference is a single Unreal AnswerHub post.

Where I work at HBO, only a subset of employees use Unreal. However, we want every machine on our network to participate in lighting builds — without the hassle of installing Unreal. That goal turned into an interesting adventure. Although Epic recommends copying the Engine\Binaries\DotNET folder to install Swarm, that alone is insufficient.

Without rehashing what the existing documentation already covers, I’m including everything I’ve learned about Swarm as of Unreal version 4.9. This information is useful to anybody setting up Swarm on your network or wanting to understand how to best configure Swarm.

Swarm Coordinator

The Coordinator keeps track of all of the Swarm Agents on the network and the status of each. You can also use the Coordinator to restart Swarm Agent instances.

  • SwarmCoordinator.exe is designed to run on a single machine on your network. We run it on a low-end always-on Windows box. We also run Swarm Agent on the same machine.
  • Swarm Coordinator does not require that Unreal Engine be installed. I didn’t do the original setup of the Coordinator on our network, but based on empirical evidence, it requires the same files as the Swarm Agent (see below), plus the SwarmCoordinator.exe.config file.
  • We place a shortcut to SwarmCoordinator.exe in the Windows Startup folder so that the Coordinator always begins at system start.
  • In theory you could have multiple Coordinators on the network, each managing an independent set of machines/Agents. We haven’t tested this theory.

Unreal Lightmass

Lightmass is the workhorse application that bakes lighting. It is automatically distributed to Swarm Agents on a per bake basis.

  • Do not install UnrealLightmass.exe directly; it’s automatically copied and run whenever a bake is kicked off.
  • Because Lightmass is completely independent of Swarm, Swarm Agents can run different versions of Lightmass. That means you can have teams running different versions of Unreal and using different versions of Swarm Agents to bake lighting. Nice.
  • Lightmass will utilize multiple CPU cores (you can configure how many) and peg the CPU on most machines. However, because (by default) it is run at below normal priority, it doesn’t normally impact the productivity of anybody actually using the machine.
  • Lightmass is RAM intensive. It is not uncommon for it to use 1GB or more RAM depending on the bake size/quality.
  • Lightmass does not utilize the GPU (as of 4.9).
  • Depending on the size/quality of your bakes, Lightmass can require significant hard drive space. For instance, for our current small project, Lightmass uses many hundreds of MB per lighting build. See notes on configuring the Swarm cache for more details.
  • Lightmass source code is located at Engine\Source\Editor\UnrealEd\Private\Lightmass and Engine\Source\Programs\UnrealLightmass
  • The version of UnrealLightmass.exe that an Agent receives is dependent on processor architecture. There are 32 and 64-bit versions. Epic recommends only using Swarm/Lightmass on 64-bit systems for best performance, and we follow this advice.
  • Lightmass requires the following be installed in order to run. If you’re installing Swarm Agents on machines, you’ll need to be sure these components are also installed.
    • DirectX v9 extensions, specifically D3DX9_43.dll, DComp.dll and XInput1_3.dll
    • Microsoft C runtime for Visual Studio 2013, specifically MSVCP120.dll and MSVCR120.dll

Swarm Agent

Swarm Agent is designed to run on multiple machines across your network, building lighting on demand.

  • SwarmAgent.exe runs on Windows XP and up. It does not run on MacOS or Linux (as of 4.9).
  • Swarm is built with C# using the .NET 4.0 framework
  • SwarmAgent.exe and its associated DLLs are 32-bit images
  • We place a shortcut to SwarmAgent.exe in the Windows Startup folder so that the Agent always begins at system start.
  • Swarm can technically run a variety of distributed tasks, but as of 4.9 it only runs lighting calculations by distributing and spawning Lightmass
  • There are two types of logical Swarm Agents. The same SwarmAgent.exe client supports both modes.
    • The Local agent is the agent that kicked off the lighting bake
    • Remote agents running on other network PCs are assigned to participate in a given lighting bake
  • After initial communication with the Coordinator, Swarm Agents appear to perform all Agent-to-Agent communication in a peer-to-peer fashion.
  • The Local agent distributes UnrealLightmass.exe and lighting information to each assigned Remote agent. The Local agent also participates in the bake (unless otherwise configured), collects the work from remote agents and merges the results.
  • Swarm will not execute bakes on machines where it detects that the CPU usage has been, on average, above 20% for the past 10 seconds (see CPUBusyThreshold in Agent.cs)
  • Swarm spawns the Lightmass process at BELOW_NORMAL_PRIORITY_CLASS on the Local agent and IDLE_PRIORITY_CLASS on Remote agents to reduce the CPU impact on busy machines. These values are configurable.
  • Swarm will not necessarily use all available non-busy agents. The number of agents appears to depend on the size and complexity of the lighting bake.
  • Swarm does not require that Unreal Engine be installed.
  • Swarm requires the following files be present in the same folder as SwarmAgent.exe. These files are found in the Engine/Binaries/DotNET folder.
    • SwarmAgent.exe
    • AgentInterface.dll
    • SwarmCommonUtils.dll
    • SwarmCoordinatorInterface.dll
    • UnrealControls.dll
    • SwarmAgent.Options.xml
  • Swarm requires that the .NET 4.0 framework (or greater) be installed. If you’re running on Windows 8 or better, it’s already present. On Windows 7, it’s not preloaded, but comes automatically with Windows Update.
  • Although Swarm itself doesn’t require DirectX v9 extensions or the C runtime, Lightmass does need these, so if you’re installing Swarm, be sure to install those components, too.
  • Swarm Agent runs as a Notification (or System Tray) app. It is designed to run continuously in the background.
  • To view the Swarm Agent interface, locate the app in the system tray (yellow and black S icon) and double click the icon.
  • SwarmAgent.exe does not respond to WM_CLOSE. To close it manually, you must choose File->Exit from the main menu. To close it programatically, you can run taskkill.exe SwarmAgent.exe.
  • Empirically (using Microsoft Message Analyzer) SwarmAgent and SwarmCoordinator use the following protocols and ports:
    • ICMP protocol for ping requests between agents
    • TCP protocol for all other communication, using ports: 8008, 8009, 54430, 56574, 56587 and 56589
  • Some ports are configurable. For instance, SwarmCoordinator.exe.config allow you to remap port 8009.
  • Swarm will not activate on sleeping PCs. I recommend that if you always want your Swarm Agents to be available, those PCs should not enter power-saving sleep states. Alternatively, you can use Windows Task Scheduler to wake machines at the appropriate times to run your bakes. See powercfg command line tool for details. We’ve found that not all machines have BIOS configurations that support powercfg, so your mileage may vary.
  • Until we explored the issue in detail, the most common Swarm problem was blocked communication between different Swarm agents. Firewalls are the culprits. Windows Firewall will block Swarm by default, even if the user chooses to allow Swarm to do outbound communications. Because Swarm communicates in a peer-to-peer fashion, it must also be configured to allow inbound communication from other agents.
  • Programmatically, the Windows Firewall can be configured to permit Swarm usage with the following commands. We also do IP filtering to make sure these firewall exceptions only happen on our local network — replace x.y with your actual leading IP octets.
    • Allow Swarm ICMP pings: netsh advfirewall firewall add rule name=”Unreal Swarm” dir=in action=allow enable=yes remoteip=x.y.0.0-x.y.255.255 protocol=icmpv4 interfacetype=lan
    • Allow Swarm TCP commands: netsh advfirewall firewall add rule name=”Unreal Swarm” dir=in action=allow enable=yes remoteip=x.y.0.0-x.y.255.255 protocol=tcp interfacetype=lan localport=8008,8009,54430,56574,56587 remoteport=8008,8009,54430,56574,56587
    • Uninstall firewall exceptions: netsh advfirewall firewall delete rule name=”Unreal Swarm”

Swarm DiagramThis diagram shows a scenario where the PC in the bottom center has kicked off a Unreal lighting bake. Three PCs have Unreal installed, but all the PCs are running Swarm Agents. The top center PC is running the Swarm Coordinator. Five PCs were chosen to participate in the bake. The other PC was busy (compiling code, etc.). Communication between the local agent and the remote agents, represented by the arrows, happens in a peer-to-peer fashion.

Swarm Parameters

Many elements of how Swarm operates can be modified. The Settings tab in Swarm Agent allows you to change settings. Configuration information is stored in SwarmAgent.Options.xml and SwarmCoordinator.exe.config. Here are some of the most interesting elements, in order of importance.

  • CoordinatorRemotingHost: must match the Swarm Coordinator Computer name from Control Panel->System
  • CacheFolder: where Swarm caches job information. At install time, we set this to the equivalent of %temp%/SwarmCache
  • MaximumCacheSize: the maximum amount of HDD space that Swarm will consume in GB. We use 10 (the default).
  • MaximumJobsToKeep: the maximum number of cached jobs (and logs) to keep around. We use 5 (the default), but it’s extremely rare that we need to look at logs of old jobs, so I think 1 or 2 would be fine.
  • AllowedRemoteAgentNames: allows you to select a subset of Remote agents that you want to use. We always want to use all available, so we set this to *
  • AllowedRemoteAgentGroup: allows different PCs to participate in different Swarm groups. We currently have a single group with our team name.
  • AgentGroupName: this is the name of the group that the local agent belongs to. We use our team name.
  • LocalJobsDefaultProcessorCount and RemoteJobsDefaultProcessorCount (developer settings): the number of CPU cores used by Lightmass for local and remote bakes. These are set automatically to reasonable values, but can be reduced to lessen CPU impact at the cost of bake speed.

Swarm Installer

To make life easier for our extended team, I wrote a Windows installer that handles the complexity described above. The installer uses the WiX Toolset to generate a standard Windows MSI file. Notes about the installer:

  • The installer is a 64-bit package. This addresses: 1) Unreal’s recommendation that Lightmass run on 64-bit systems only, 2) a limitation of WiX that doesn’t allow 32-bit packages to include/install 64-bit binaries, plus 3) the fact that everybody in our office is running 64-bit Windows anyway 🙂
  • The following DirectX v9 files must be installed: dxsetup.exe, dxupdate.cab, dsetup.dll, dsetup32.dll, Jun2010_d3dx9_43_x64.cab and Apr2007_XInput_x64.cab. You can grab these files from the DirectX SDK. The WiX toolset page has good instructions for installing DX extensions.
  • In addition to the normal installer-y stuff (copying files and updating the registry), the installer:
  • For uninstalls, in addition to nuking the right files, the installer:

The Swarm requirements we publish to our team:

  • 64-bit Windows 7 or above
  • 4+ GB RAM
  • 10+ GB free hard drive space

We recommend that everybody on the team with machines that meet the spec — including people that already have Unreal installed — install Swarm Agent.

 

Posted in Game Programming, Unreal | Tagged , , | 5 Comments

Windows Installer: Removing Folders


Recently I put together a Windows Installer package (MSI file) for our team to use internally.

Without you having to do any extra work, the Windows Uninstaller automatically takes care of deleting any folders where it installed content. However, if an app writes to other folders that the installer doesn’t know about, it’s your responsibility to handle the removal of those folders.

In my case, the app that I was installing writes temporary files to a cache folder. The app could cache gigabytes of data, so it was key that the uninstall remove all those temporary files. Fortunately, WiX makes this easy. Here are the steps. First, we search the registry for the location of the cache folder and store it in a Property. We’ll use this if we’re uninstalling. Replace CompanyName and AppName with your company and app names.

<?define AppRegKey = "Software\CompanyName\AppName" ?>
<Property Id="CACHEFOLDER">
 <RegistrySearch Key="$(var.AppRegKey)" Root="HKCU" Type="raw"
 Id="CacheFolderRegSearch" Name="CachePath" />
</Property>

Second, in the case we’re installing, we need to save the location of the cache folder in the registry, so that we can invoke the “RemoveFolderEx” command on that folder during a future uninstall.

<?define AppCacheFolder = "AppCache" ?>
<Component Id="CacheCleanup" Guid="*">
 <RegistryValue Root="HKCU" Key="$(var.AppRegKey)"
 Name="CachePath" Type="string" Value="[TempFolder]$(var.AppCacheFolder)" KeyPath="yes" />
 <util:RemoveFolderEx On="uninstall" Property="CACHEFOLDER"/>
</Component>

We need to reference the component we’ve registered in a Feature block, along with other relevant components.

<Feature Id="MainApplication" Title="App Name" Level="1">
 <ComponentRef Id="OtherComponents" />
 <ComponentRef Id="CacheCleanup" />
</Feature>

To make this all work correctly, there are two more steps. First, ensure your WSX source references the utility extension where util:RemoveFolderEx is located.

<?xml version="1.0" encoding="UTF-8"?>
<Wix xmlns = "http://schemas.microsoft.com/wix/2006/wi"
     xmlns:util = "http://schemas.microsoft.com/wix/UtilExtension">

And second, reference the extension when you compile and link:

candle source.wxs -ext WixUtilExtension.dll
light source.wixobj -ext WixUtilExtension.dll

That’s it. Now the installation will correctly store a registry reference to the cache folder and clean up the cache folder on uninstall.

Posted in Computers and Internet, Installers, Microsoft, WiX | Leave a comment

Windows Installer: Firewall Settings


Recently I put together a Windows Installer package (MSI file) for our team to use internally.

The app I was installing communicates with other apps on our internal network, and it requires certain network ports be open for ICMP and TCP communication.

My first thought on solving this issue was to figure out where Windows Firewall Manager stores its firewall settings and then tell the firewall to read the new settings. Indeed, I found the location in the registry and I was able to successfully create the new settings. I was also able to kick the firewall using the commands “net stop “windows firewall”” followed immediately by “net start “windows firewall””. Everything seemed great.

But there were issues. For instance, during install, a little message showed up in the taskbar that the firewall was stopping. That’s confusing to the end user, especially because there’s no message that shows up saying the firewall is starting back up. Modifying the registry directly felt dirty. This was not a maintainable solution. The whole approach was a hack.

So I found a better way: the “netsh advfirewall” command. It’s available on Windows 7 and up.

Say you wanted to open ports 8008 and 8009 for inbound TCP traffic on the local network. Here’s the full command to add that rule:

netsh advfirewall firewall add rule name="My Firewall Rule" dir=in action=allow enable=yes protocol=tcp interfacetype=lan localport=8008,8009 remoteport=8008,8009

Even better, the firewall automatically starts applying this rule without a restart. There are many other advfirewall parameters, but you can explore those on your own.

Running these commands in the context of the installer was a little more challenging. I needed to add multiple rules, and I needed to handle removing the rules on uninstall. Let’s start with the variables I used to simplify things:

<?define Quote = """ ?>
<?define IpRange = "w.x.y.z-w.x.y.z" ?>
<?define OpenPorts = "8008,8009" ?>
<?define FirewallRuleName = "$(var.Quote)Open 8008 and 8009 for MyProject$(var.Quote)"?>
<?define FirewallAddRule = "advfirewall firewall add rule name=$(var.FirewallRuleName) dir=in action=allow enable=yes" ?>
<?define FirewallRuleICMP = "remoteip=$(var.IpRange) protocol=icmpv4 interfacetype=lan" ?>
<?define FirewallRuleTCP = "remoteip=$(var.IpRange) protocol=tcp interfacetype=lan localport=$(var.OpenPorts) remoteport=$(var.OpenPorts)" ?>
<?define FirewallRemoveRule = "advfirewall firewall delete rule name=$(var.FirewallRuleName)" ?>

The Quote variable handles the cases where we need to embed quotes in the command string. The IpRange variable locks this rule down to a specific set of IPs. Fill in your own values for w.x.y.z. Like many other advfirewall commands, it’s optional. FirewallAddRule has the meat of the command string, and then there are individual suffixes for the ICMP and TCP rules. The ICMP rule doesn’t accept ports. The FirewallRemoveRule ensures that all rules matching the given name will be removed on uninstall.

Here’s the code for executing these rules. First, for install:

<CustomAction Id="SetUpdateFirewallICMPCmd" Property="UpdateFirewallICMP" Execute="immediate"
 Value="&quot;netsh&quot; $(var.FirewallAddRule) $(var.FirewallRuleICMP)" />
<CustomAction Id="UpdateFirewallICMP" BinaryKey="WixCA" DllEntry="CAQuietExec" Execute="deferred"
 Return="ignore" Impersonate="no" />
<CustomAction Id="SetUpdateFirewallTCPCmd" Property="UpdateFirewallTCP" Execute="immediate"
 Value="&quot;netsh&quot; $(var.FirewallAddRule) $(var.FirewallRuleTCP)" />
<CustomAction Id="UpdateFirewallTCP" BinaryKey="WixCA" DllEntry="CAQuietExec" Execute="deferred"
 Return="ignore" Impersonate="no" />
<InstallExecuteSequence>
<Custom Action="SetUpdateFirewallICMPCmd" Before="InstallFinalize"><![CDATA[NOT REMOVE]]></Custom>
<Custom Action="UpdateFirewallICMP" After="SetUpdateFirewallICMPCmd"><![CDATA[NOT REMOVE]]></Custom>
<Custom Action="SetUpdateFirewallTCPCmd" After="UpdateFirewallICMP"><![CDATA[NOT REMOVE]]></Custom>
<Custom Action="UpdateFirewallTCP" After="SetUpdateFirewallTCPCmd"><![CDATA[NOT REMOVE]]><Custom>
</InstallExecuteSequence>

There are four custom actions that execute in sequence after the InstallFinalize stage. The first action establishes the command line for adding the ICMP rule using the variables defined above. CAQuietExec requires that the actual app command (netsh in this case) be quoted. The second action executes the command line established in the first command. Using multiple CAQuietExecs requires configuring the command line as a property with a matching ID in the following action. Changing firewall settings requires Windows elevated privilege, so we use deferred execution with no impersonation. The final two commands set the TCP rule in the same manner.

The funky notation ![CDATA[NOT REMOVE]] tells the installer that these commands should run whenever we’re not actually uninstalling the product (e.g. installs, reinstalls, upgrades, repairs).

Here’s uninstall:

<CustomAction Id="SetUpdateFirewallRemoveCmd" Property="UpdateFirewallRemove" Execute="immediate"
 Value="&quot;netsh&quot; $(var.FirewallRemoveRule)" />
<CustomAction Id="UpdateFirewallRemove" BinaryKey="WixCA" DllEntry="CAQuietExec" Execute="deferred"
 Return="ignore" Impersonate="no" />
<InstallExecuteSequence>
<Custom Action="SetUpdateFirewallRemoveCmd" Before="InstallFinalize">
<![CDATA[(NOT UPGRADINGPRODUCTCODE) AND (REMOVE="ALL")]]>
</Custom>
<Custom Action="UpdateFirewallRemove" After="SetUpdateFirewallRemoveCmd">
<![CDATA[(NOT UPGRADINGPRODUCTCODE) AND (REMOVE="ALL")]]>
</Custom>
</InstallExecuteSequence>

We set up the command line to remove the firewall settings, then invoke that command line with elevated privileges. We only remove the firewall settings when the product is being uninstalled (not installed, reinstalled, or upgraded)

One additional note. I really wanted to lock down the firewall rule to a specific app, so I was excited that advfirewall has just the option I was looking for:

allowedprogram program=C:\MyApp\MyApp.exe name="My Application"

But unfortunately in my particular case the app I was installing is not actually the app that does all the LAN communication. My app spawns other apps that do the communication, and the locations and names of those apps is not necessarily known at install time. Hence we chose to lock down the firewall rules to specific (small) range of IPs, so that if this install package gets out in the wild it’s not unnecessarily introducing security holes.

 

Posted in Computers and Internet, Installers, Microsoft, WiX | Leave a comment

Windows Installer: Modify Config Files


Recently I put together a Windows Installer package (MSI file) for our team to use internally.

The app I was installing writes to temporary storage, and the app stores a reference to the temporary storage folder in AppConfig.xml, which is included in the install manifest. AppConfig.xml contains a folder specifier like this:

<Options>
  <CacheFolder>c:\cache</CacheFolder>
</Options>

The problem I faced was that I wanted to avoid using a hardcoded “c:\cache.” Not every user has a C drive or wants a cache folder at the root. And I didn’t want the user to have to configure this manually. The installer should default to a reasonable location.

Fortunately, WiX handles this sort of thing with aplomb. You can modify the contents of XML files at install time. Perfect!

Here’s how. In the WXS manifest, add an XML setValue action to modify XML contents.

<?define AppConfigFile = "AppConfig.xml" ?>
<?define CacheFolder = "cache" ?>

<DirectoryRef Id="InstallDir">
  <Component Id="Config" Guid="<your-GUID-here>">
    <File Id="Config" Source="$(var.AppConfigFile)" KeyPath="yes"/>
    <util:XmlFile Id="Cache" File=[#MyAppConfig]"
      Action="setValue" ElementPath="//Options/CacheFolder"
      Value="[TempFolder]$(var.CacheFolder)" />
  </Component>
</DirectoryRef>

Here’s what’s happening. You define your file component as normal. After you’ve specified the file information, use the util:XmlFile specifier. XmlFile File points at the actual file by using the # specifier, which resolves to the full install path of AppConfig.xml. Change the XML value of to “%temp%cache”, which resolves to the user’s temporary folder plus “cache,” e.g. “c:\users\pete\AppData\Local\Temp\cache” — just what we want.

To make this all work correctly, there are two more steps. First, ensure your WSX source references the utility extension where util:XmlFile is located.

<?xml version="1.0" encoding="UTF-8"?>
  <Wix xmlns = "http://schemas.microsoft.com/wix/2006/wi"
       xmlns:util = "http://schemas.microsoft.com/wix/UtilExtension">

And second, reference the extension when you compile and link:

candle source.wxs -ext WixUtilExtension.dll
light source.wixobj -ext WixUtilExtension.dll

That’s it. Now the installation modifies the config file so your app automatically caches data to the user’s personalized temporary folder.

Posted in Computers and Internet, Installers, Microsoft, WiX | Tagged , , | Leave a comment

Windows Installer: Shut down System Tray Apps


Recently I put together a Windows Installer package (MSI file) for our team to use internally. The package installs a system tray app (sometimes called a notification app or notification area app). One common property of system tray apps is that they don’t actually close on exit — they simply return to the system tray and continue running in the background.

In order to properly uninstall this component, the app needs to be shut down by the installer/uninstaller. Typically, in WiX, you’d do this with the CloseApplication element, which sends a WM_CLOSE message to the system tray app, which it ignores completely. What to do?

Here was my solution, embedded in the WiX WXS source file (toolset v3.9):

<?define SysTrayAppExe = "mySysTrayApp.exe" ?>

<CustomAction Id="KillApp" Directory="InstallDir" Return="ignore" ExeCommand="&quot;[SystemFolder]taskkill.exe&quot; /F /IM &quot;$(var.SysTrayAppExe)&quot;" />

First, we define a custom action that will execute the taskkill Windows system command. We ignore the return value because we don’t care if the app was actually running. Just in case the Windows system folder has embedded spaces, we wrap the command in quotes using the XML quote specifier. The /F flag requests a forceful termination, and the /IM flag requests that we terminate the specifically named process, which is again wrapped in quotes in case it has embedded spaces.

Next, we set up the custom action to execute in the first phase of the install process. This ensures that the system tray app is always shut down when the install/uninstall begins.

Ideally, we’d use the CAQuietExec special custom action to avoid a command screen from briefly flashing on the screen, but CAQuietExec must be run after the InstallInitialize phase, by which time the Windows Installer has already detected that the system tray app is running, and it’s too late for us to do anything about it.

Posted in Computers and Internet, Installers, Microsoft, WiX | Tagged , , | 2 Comments