Lean Methodology: Before and After

Written by Guillermo Ortiz de Zarate on November 23, 2017

I first read The Lean Startup by Eric Ries in 2013. It was a lightbulb moment to see thoughts that had been percolating in my head over the years be articulated in such a way. Since then, I’ve been implementing the methodology with my team at the National Council of Architectural Registration Boards (NCARB) where I work as Chief Information and Innovation Officer, leading the modernization of the organization through the implementation of technology.

I believe the lean framework is the natural evolution of Agile, since it helps to identify what you should be working on. There is no bigger waste than spending time and resources on the wrong project. To convey the impact the principles had on my workplace, I have set out examples of two projects and how we approached them before and after adopting a lean mindset.


We were creating a set of tools for architectural licensure candidates to report their experience, so that a supervisor could approve their hours and they could fulfil the requirement necessary to gain a license. We launched a simple version of an app designed to let people record their hours in blocks, specifying a date range and the total hours by category during that time period. We received feedback that most people working in an architecture firm already report their hours once a week, to track their billable time. Early users suggested we transform the app to allow them to report in timesheet mode, mimicking their current practice.

It seemed like a good idea, and we spent six months developing this update to the tool, not a negligible timespan. Other projects were put on the backburner. In sympathetic hindsight, I can see why we jumped into the plan without considering the risks. Many members of NCARB staff work on outreach to understand what architects need from us. In these conversations, some people indicated they would prefer to use a timesheet, and this anecdotal evidence fell into the hands of leadership. Once the leadership wants to pursue an idea it can be hard to push them into experimental mode instead of execution. It made some logical sense, so we jumped right in.

The results didn’t vindicate the project. A few months after launching, the adoption of the time sheet mode was at 2-3%, and three years later it still sits at under 15%. The vast majority of people still use the original tool. What else could the team have achieved in the time they spent working on this feature?


Fast forward a few years, and I found an opportunity to try using the core principles from The Lean Startup. NCARB develops the registration exam for architects, required by all states to get their license. However, we do not train people to pass the exam, so third parties put out test preparation materials to help candidates study and pass all divisions of the exam. The Examination department asked if we could develop a tool that would allow candidates to research test preparation materials based on comments and reviews from peers, in the style of Yelp.

Typically, the conversation would either be dismissed due to lack of time, or jumped into feet first. But with lean principles fresh in my mind, I suggested we experiment with the idea and test assumptions about its potential success. They were willing to give it a shot, so we laid out the assumptions: candidates don’t know what test preparation materials to choose, and the price of materials and the urgency of exams add to the problem. We know that peer reviews are valuable in decision making, but a review engine only works if people are willing to help other and leave detailed comments. This was another assumption that needed validation.

Our goal was to help people make decisions about exam preparation, and we understood what success would look like. Early on we established metrics for measuring success, including how many people would use it and how many people would recommend it to a friend, as well as how many people were willing to leave reviews. We conducted a survey among recent test candidates to find out the materials they used and their opinion on them. As the exam body, we knew who had passed and who had failed, which proved valuable. We created a simple report based on this information and circulated it in an email marketing campaign to people who were set to take the test in the following months. None of these things took much time.

To measure success, we recorded who opened the email, clicked on the link and downloaded the packet, as well as who took part in the survey in the document designed to find out if it was valuable. The results were not promising. Of 99% who opened the email only half clicked on the link, only a fifth downloaded the document, and only a few told us it was valuable. At this point the question was: should we persevere, pivot to do something else, or abandon the project? We decided to abandon the project to focus on more pressing initiatives. Lean thinking helped us to dodge the bullet of wasting further time on a doomed project.

I used the principles from the book without any formal training and followed my instincts. I asked: what is within my circle of control? Even someone in middle management can enact lean principles in their own domain, by identifying assumptions and finding ways of validating them. As Eric Ries says in his book, don’t sell what you can make, but make what you can sell. Always ask, is this a real problem worth solving for our membership? The rest will follow.

If you want to find out more, the white paper I co-wrote with Elizabeth Engel, Innovate the Lean Way: Applying Lean Startup Methodology in the Association Environment can be downloaded for free here.

Does this topic intrigue you? Guillermo spoke on this subject at SURGE 2017, a free virtual summit we hosted over November 7-9th. Click here to access the replay of the session.