Successfully completed the SAFe SPC exam this morning. On the first attempt, thankfully. I am now a “Certified SAFe ® 5 Program Consultant.”
In a recent New York Times column, Adam Grant wrote:
Once upon a time – last century, actually – employers could rely on the conferring of a college degree as evidence of a certain level of competence in the degree subject. In some areas, this is probably still true. Generally speaking, this would apply to the scientific areas of study: chemistry, physics, mathematics, etc. Unfortunately, even these area are becoming suspect as academic rigor is eroded in the interests of removing perceived barriers to this or that special interest group. To be very clear, I’m referring to the importance of thorough and complete understanding of the subject. The mine field that academia has become is indeed rife with self-inflicted and often insurmountable barriers to learning. The egregious rise in the cost of tuition, grade inflation, and credential dilution are but a few examples.
There are other factors in play. The speed at which society moves in the 21st Century is simply too fast for the four-year degree to to have any hope of staying relevant, let alone keeping up. Almost every major university offers free courses in a wide variety of subjects so it is possible for a high school graduate to craft the equivalent of a Bachelors or Masters and complete it for a fraction of the cost and in half the time. Ah, but without having completed the paper chase, how can such an industrious individual establish for a potential employer that they have the requisite competence?
Adam Grant has it right. Credentials are overrated. So how can we assess the quality and potential of team candidates? Grant identifies three key mistakes interviewers make in the interview process.
- They ask they wrong kinds of questions.
- They focus on the wrong criteria.
- They’re overly influenced by the best talkers.
If, as Grant suggests, job interviews are broken than conducting remote job interviews in the midst of a pandemic are significantly more challenging. In this post, I wish to speak to the second mistake identified by Grant and write about what we can do to identify our criteria, what we can do during an interview to elicit information about the candidate’s qualifications, and a strategy for improving the efficacy of remote job interviews.
Identify Important Criteria
For the sake of example, we’ll engage in a little time travel into the future and imagine having hired the perfect product owner candidate. What tasks encountered in your work day are no longer an issue with the new candidate on board? Is the product backlog now well-maintained and in a healthy state? Does the sprint runway extend out 4-5 (or more) sprints? Has a stable sprint velocity emerged (suggesting that the user stories are of higher quality and understood better by the team)? Do conflicts between areas of the business occur less frequently than in the past? Are stakeholders pleased with the results they see at sprint and increment reviews?
If our example were for a scrum master candidate, we would ask ourselves different questions for eliciting important criteria for the position. Is there less conflict among team members? Does the team understand the purpose and value for determining the effort involved to complete a user story? And again, has a stable sprint velocity emerged?
In addition to considering what hasn’t been working well (and therefore illuminating what skills you want a candidate to bring to the table) it is also important to include what has been working. It will not serve the organization if one set of problems are swapped for another. Perhaps, for example, the previous product owner was well liked by the team and helped the team maintain a positive morale, but had a poorly maintained product backlog that prevented a good approximation for a release date. It wouldn’t be much of an improvement if the new product owner kept a healthy product backlog but did so by driving the team as a tyrant might.
Test for Matching Skills
With a good feel for the criteria needed to hire the best candidate you can then craft a strategy for determining how well the candidate’s abilities satisfy your criteria. Prepare tasks for the candidate that will verify congruity between what a candidate says they can do and what they can actually do. One approach, which I use frequently, is to present the candidate with a series of scenarios, each designed to build on how the candidate responded to the previous scenario. While I may only present a candidate 3-4 scenarios, I have several dozen in the queue and present the sequence based on how well the candidates responses to the challenge.
For example, for a scrum master role – a high-touch role that requires consummate communication skills, flexibility, and the ability to solve people problems – I may present an initial scenario as follows:
“I’m going to give you several scenarios. You are free to ask any questions you wish about the scenario and state any assumptions you are making in your responses.
You are being considered for a position as scrum master for a team that is developing a healthcare related web application for use in hospitals. This team is responsible for developing the UI/UX components and works closely with another team responsible for much of the database and middle tier components. As a new scrum master, what questions would you ask of anyone in the organization to help you quickly understand what you need to do to become effective as a scrum master for your team?”
There are many things I would hope to hear in the candidate’s answer. To mention a few, I’d like to hear that they want to speak to the product owner, the stakeholders, and, of course, each of the team members. I’d like to hear that they plan to spend time in information gathering mode rather than work immediately to shape the team into some version of teams they’ve worked with at other jobs. I’d like to hear questions from them about what kinds of metrics does the team use and what have they shown.
There are no right and wrong answers to a scenario like this. Just answers that are better than others. And I don’t expect the candidate to deliver an exhaustively thorough response.
From their responses, I might learn that they are a recipe follower or that they are flexible in adapting to the needs of the business while working to establish good scrum practices. I might learn that they really don’t know scrum at all and are only good at parroting text book examples and jargon. I might hear how they would attempt to leverage several things from previous experience while acknowledging those attempts would be experiments and subject to adaptation based on feedback.
Assuming the candidate responded to the first scenario in a way that scores high marks for satisfying my criteria, I might offer the next scenario as follows:
“Assume you have been serving successfully as scrum master for this team for six months now. The product owner calls the team together and says ‘I need to swap out some of the stories in the sprint for work that marketing wants done before the end of the week.’ As scrum master, how would you respond to this development?”
As with the previous scenario, the candidate’s response would be measured against the criteria I have established for the position. Depending on what I’ve heard, I may continue to offer additional scenarios that build on the candidates developing experience with the scenario scrum team.
This strategy is pursued until I’m satisfied the candidate knows what they claim to know or not. A short interview does not bode well for the candidate. A long interview does.
While the concept and practice is straightforward, shifting a team from intuitive guesses about story points to a more deliberate approach for determining effort value (a.k.a. story points) can be a challenge at first. The following approach may help start the process.
- Begin by focusing on product backlog items (PBIs) that the team has estimated using their previous approach at a 5 or greater. There isn’t much to be gained by applying this approach to PBIs estimated at 1 or 2. PBIs that the team knows are a bigger effort but may not be able to articulate why that is the case are good candidates for learning how to apply this technique.
- Ask the team how much time it may take to complete a PBI. While I have written before about the importance of excluding time criteria when determining effort values, this can be a good place to start. It is what teams are most familiar with – for better or worse. Teams usually have not problem throwing out a time: 8 hours, 16 hours, etc.
- With the time estimate in hand ask the team:
“If you sit in front of your computer and start the clock, will the PBI be done if you do nothing and the estimated time elapses?”
I would hope the team would answer “No.”
- With the answer to the first question in hand, ask the following question:
If the passage of time alone won’t get the PBI work completed, what will you be doing (actions and behaviors) to complete the work?
The conversation that follows from this questions is the basis for determining the effort criteria the team needs to better describe what they will be doing on their way to completing the PBI. The techniques around establishing effort criteria are described in an earlier post.
Build a Torii Gate, of course.
A vacation originally planed for this week in Utah was scrubbed. So a significant pivot was in order. Priorities, plans, and schedules shifted and forward motion was begun once again.
I wanted to build a Torii Gate on the East side of my property for several years. The gate and fence that was there worked well enough so it never made it very far up on the backlog. That changed last fall when the Chinook winds – which are frequent, sudden, and fierce in this part of the country – snapped the two supporting gate posts. (The same storm also blew off the gate on the North side of the property, but that’s another project.) The gate and fence have been braced up by 2×4’s all winter. Not a good look.
Worked on the hashira (posts) over the winter. They needed to withstand the Chinooks. So, 6′ steel post – 3′ bolted within 3 2x6x10s and 3′ sunk into a concrete base – ought to hold for a while.
Time to begin the outside work.
First post had to be set perfectly. This is after it had set for a few days and most of the supports had been pulled away.
Next, the nuki (lower beam) and the shimagi and kasagi (two upper beams.)
Add a little extra flair trim to the kasagi, stain, and seal.
All that was need to complete the Torii gate part of the gate was the gakuzuka – a small brace in the center between the shimagi and kasagi – with an inscription. The weather intervened and brought us about 9″ of fresh snow.
Weather cleared, snow melted, still self-isolating – back to work to build and install the new swinging gate.
Next, dress up the top of the swinging gate with a pattern to match the fence on the north side of the property.
Finally, add the gakuzuka. The Japanese kanji on the way into the gate is “Love.” Find love here, all ye who enter.
The kanji on the way out through the gate is “Peace.” Take peace with you into the world.
Add an exterior handle crafted from ceder and the gate is done. The street view is quite nice, even before the summer vines and surrounding flowers wake up.
Time now to clean up the work site and do a little path repair.
I’ve been reading “Hello World: Being Human in the Age of Algorithms” by Hannah Fry. She relates this story:
Unable to understand why their benefits had been reduced, or to effectively challenge the reduction, the residents turned to the American Civil Liberties Union (ACLU) for help.
…[The ACLU] began by asking for details on how the algorithm worked, but the Medicaid team refused to explain their calculations. They argued that the software that assessed the cases was a ‘trade secret’ and couldn’t be shared. Fortunately, the judge presiding over the case disagreed. The budget tool that wielded so much power over the residents was then handed over, and revealed to be – not some sophisticated AI, not some beautifully crafted mathematical model, but an Excel spreadsheet.
Within the spreadsheet, the calculations were supposedly based on historical cases, but the data was so badly riddled with bugs and errors that it was, for the most part, entirely useless. Worse, once the ACLU team managed to unpick the equations, they discovered ‘fundamental statistical flaws in the way that the formula itself was structured’. The budget tool had effectively been producing random results for a huge number of people. The algorithm – if you can call it that – was of such poor quality that the court would eventually rule it unconstitutional.
My first thoughts were, “How bad a spreadsheet hack do you gotta be to have your work be declared unconstitutional? And just how many hacks does it take to build an unconstitutional spreadsheet?”
To be fair, math is hard. Government is complex. And I’m comfortable with the assumption that everyone who had a hand in building this spreadsheet had good intentions. Venturing a guess, the breakdown happened at the manager/politician/lawyer level.
It is probable that the complexity of the task quickly overtook the abilities of the spreadsheet author(s) and the capabilities of the tool. Eventually, no single person understood how the whole thing worked. Consequently, making a change in one place affected how the spreadsheet worked in n other places and no one was capable of regression testing the beast. But the manager/politician/lawyer types knew what to do: Hide behind the “trade secret” smoke.
There are many lessons from this story. Plenty of points of failure. What I’m interested in writing about is the importance of transparency and how a good set of performance metrics can help in maintaining transparency.
The externally facing opacity in this story is readily apparent. What we don’t (and probably never will) see is the lack of transparency prevalent internally to the Idaho Department of Health and Welfare and whomever designed and built the spreadsheet tool. I’d bet a round of drinks that neither has heard of Agile much less employed its principles and practices. These by themselves – when actually practiced long term – go a long way toward establishing a culture of transparency. This is the key. Long term practice. A period of time is needed to change behaviors, mindsets, attitudes, beliefs, and when necessary, personnel. Even over the long term, implementing an Agile methodology isn’t improvisational theater. A strategy and a way to measure progress is needed.
Which gets me to metrics.
Selecting metrics and tuning them over time is critical to measuring team performance and developing improvement plans. Metrics that inform meaningful actions are the goal. Leave the vanity metrics that verify what managers want to hear or already “know” to the competition.
I’ve encountered my share of overly complex ways to measure the performance of individuals and teams. Often the metrics taken from machine-like task work (for example, assembly line work) are applied to creative or intellectual/knowledge tasks. This type of re-purposing results in, for example, counting lines of code or the number of source code check-ins as an indicator of software developer productivity. It never ends well.
When working to define a set of metrics to track an individual or team’s performance it is more effective to begin by asking several questions.
- What problems are you trying to solve?
- What questions will your chosen metrics answer?
- What questions will your chosen metrics not answer?
- How, specifically, will you know you can trust you metrics? How will you know when they are right and how will you know when they are wrong?
- How well do your metrics compliment each other? That is, by combining them do you end up with a much better picture of individual or team performance the you do by considering individual metrics?
- Do your metrics support any planned actions for improvement? Are you collecting actionable metrics or vanity metrics?
Finally, it is important to understand the limits of performance metrics. Displaying velocity charts that have fractions of story points implies an accuracy that simply isn’t there. Significantly adjusting project timelines based on the first three sprints worth of velocity data can have adverse secondary effects on the project.
There is no perfect set of metrics, no divine set of measures that match an impossible standard of perfect objectivity and fairness. The best possible set of metrics is one that supports useful decisions rather than simply instructs managers where to apply the stick. They should help show the way to performance improvement rather than simply report results.
I work to have 3-5 metrics, depending on the individual, the team, and the project. Less than 3 and the picture starts to look rather flat. More then 5 and the task of performance monitoring can become overly complicated and cumbersome. Keep it lean and manageable. That way, it’s easier to tell when things aren’t working and your metrics are much less likely to violate your team’s constitutional rights.
Are you here or there?
Working, in isolation.
The sprints continue.
If you’ve ever been fit for prescription glasses, you’ve no doubt had the experience of the eye exam where the doctor flips between different lens strengths and asks “Is this better or worse than before?” It’s basically A/B testing.
This came to mind after reading a research paper authored by Dan Gilbert and Jane Ebert  and listening to Gilbert’s TED Talk, “The surprising science of happiness.” The key bit, as described by Gilbert:
Sometime later — it could be 15 minutes, it could be 15 days — the same stimuli are put before the subject, and the subject is asked to re-rank the stimuli. “Tell us how much you like them now.”
The result was that their previous #3 was ranked as #2 and their previous #4 was ranked as #5. This reflects what Gilbert calls “synthetic happiness.” Having been denied their #1 and #2 choices, experiment participants was forced to “settle” for a lesser choice. However, having made the choice they increased they preference for the lesser choice and thereby synthesized happiness with that choice. Just as interesting, the previous #4 choice was pushed further down the scale as if to put some distance between the previous #3 choice. In effect, distinguishing the decision to take home #3 as clearly the better choice.
All this gave me an idea for something to try with a team I’ve working with that needed to rehabilitate their team identity into something healthier. Typically, teams sour on the idea of going through an exercise like this. The team I was working with was no exception. They likened it to defining team goals – a largely tedious and uninspiring chore.
I wanted to know if I could present two possible team identity statements – A/B style – of which one would be clearly undesirable and another more in line with what I suspect the team may be comfortable. The A/B presentation would keep this simple (presenting a selection of six team identity statements as in the experiment with pictures described by Gilbert would be a non-starter.)
Offering a choice should compel them to chose one over the other. I’m counting on their brains to do what brains do. When faced with a choice, they make one. If I were to present them with a single identity statement and ask “How would you like to change this to be more in line with the identity you want?”, I’ve every confidence the room would be filled with silence.
The very first presentation had a blank page on the left and my intentionally lame and inaccurate team goal on the right.
The team was well aware “no goal” wasn’t an option and wouldn’t reflect well on their performance review with HR and management. My theory was that when faced with an empty goal and one that was inaccurate, they’d suggest something, however minimal, that was an improvement on the initial goal. This is what happened and the team then spent a few minutes tuning the goal into something a little less cringe-worthy. This began the process of converting the goal from the scrum master’s goal to the team’s goal.
Then I deliberately let a week or more pass.
On next presentation, the goal on the left was the goal they chose and tuned previously. The second choice was similar but contained one or two slight modifications intended to move the team’s identity in a more positive and healthy direction. Over the course of several months I tested – A/B/Eye Exam style – numerous team goals. “Which goal do you prefer, the one on the right or the one on the left?”
So we had a start. From here on out it was just a matter of improvement. Keying off of things the team said or did, I’d modify the “accepted” goal and present it as an option at the next opportunity.
The key or driver in this approach, the hypothesis goes, is to set it up so that the team makes the decisions rather than having something foist upon them. The are virtually guaranteed to reject or strongly resist the latter. With the former, they have ownership in the decision. To reject their decision is to say, in essence, that they made a bad or wrong choice, a bad or wrong decision. In general, people don’t like to admit such a thing so they stick with a decision – for better or worse – if it’s a decision they made and are responsible for.
Another important element in play with this approach is the anchoring cognitive bias, particularly early on. People are much more comfortable making comparisons between things than they are with coming up with something original. By presenting a blank goal and one that reflects a direction in which I want the team to move – from nothing to something positive – the hypothesis is that the team will assimilate toward more positive goals and that this assimilation will become self-reinforcing over time.
References D. T. Gilbert, J. E. J. Ebert (2002) Decisions and Revisions: The Affective Forecasting of Changeable Outcomes, Journal of Personality and Social Psychology, Vol. 82, No. 4, 503–514
Photo credit: Max Pixel
I’ve coached and served as scrum master for dozens of remote teams over the years. In light of current events, I’ve posted a collation of the notes I have for helping distributed teams working effectively.
The rule is, if one person on a team is participating from a remote location, the team is a distributed team. Making distinctions such as “distributed team” vs. “fully distributed team” when some or all members of the team are working from remote locations risks marginalizing team members who are working at locations away from the home or central office.
To help ensure that the concept of “team” remains intact during distributed team conditions, this post is intended to serve as a guide for scrum masters for how best to facilitate and monitor team interactions and performance with distributed teams.
Scrum masters have the added burden of developing the skills within their teams for effective distributed team collaboration and communication. Distributed teams can function better than collocated teams when the team’s skills for organization and communication have been improved by necessity such that they are better positioned to function effectively as a team.
Even with ideal conditions (i.e. plenty of bandwidth for audio and video, intra-team familiarity, clear agenda, etc.) working with distributed teams will always have a degree of asynchronous communication. Fields of vision are limited to what’s in frame for the video camera, network latency issues, and a general lack of access to in-person non-verbal cues will cause people to speak over and interrupt each other and miss-read intent more than they would during in-person meetings. To mitigate these constraints, it is important to have a well understood and practiced set of etiquette rules for distributed team meetings.
- The principle challenge is to keep everyone on the team focused. Shorter, more frequent meetings are better than long meetings that invite participants to “multi-task” out of boredom. Invite essential people to the meeting. Use of a responsibility assignment matrix is a very useful tool for determining who to invite. People associated with the “consulted” or “informed” roles generally don’t need to be invited to formal Scrum meetings.
- Communicate an agenda – however sparse – prior to the meeting. Let the team know what the purpose and objectives of the meeting are so everyone knows when they are done. Agendas are scope management tools and help keep the meeting focused. Set the expectation the everyone on the team will be aware of the agenda prior to the start of the meeting.
- Ensure that all the necessary equipment is in working order and that all team members have verified connectivity. Test screen sharing capabilities and application accessibility prior to the meeting.
- It is preferred that the scrum master host the on-line meeting so that on the rare occasion they may need to take control of the meeting by muting or bumping someone from the meeting they may do so.
- Set the expectation that team members will participate from a location that is free from background noise and other distractions. Joining from coffee shops or while driving is to be discouraged as they end up being distractions for the entire team.
- Coach the team for how to take turns in a conversation.
- Coach the team on how to use any mute features.
- The rules of common courtesy apply to on-line meetings. If we wouldn’t accept a behavior in a face-to-face meeting, it shouldn’t be accepted in an on-line meeting either.
- Turn off notifications from applications such as email and instant messaging.
- Meeting participants need to be aware of what’s in the visual and auditory background and work to anticipate and mitigate potential interruptions.
In-Progress Meeting Activities
- Continue to stress that team members be on time for the meeting. Set the expectation that they will join on-line several minutes early to accommodate any last-minute technical issues.
- Meetings serve both a practical and social purpose for distributed teams. Allow time for casual conversation (and still expect everyone to be one time) as it is critical to building and maintaining rapport among distributed team members. Small talk, catching up, sharing interesting news bits, interesting (short) stories – any of these helps make the on-line meeting a more enjoyable experience and help set a positive tone for the duration.
- If a team member joins a meeting late, they should not announce to the team they have arrived. Doing will potentially disrupt a conversation already in progress.
- When a team member needs to contribute to a conversation while others are talking, the best approach is to state, “I have a comment.” or “I have a question.” and then pause. This serves as a signal to the team that you wish to speak. Either the conversation will naturally break, or the scrum master can listen for a break and offer the team member the opportunity to speak. The natural in-person behavior is to simply start talking. With distributed teams this frequently leads to several people talking at once and a degradation in communication.
- Ask for the opinion of someone on the team who has not yet participated in the meeting or has been silent for a period of time. This will help keep them focused on the meeting if they know they may be asked a question.
- Team members should leave a comment in the meeting chat window if they need to step away from the meeting for a short time or leave the meeting. Announcing such departures tend to be disruptive, but the team will likely need to know if any team members are not absent from the meeting.
- Remain attentive to anyone who is dominating the conversation or otherwise preventing others from contributing.
- For stand-ups, call out the person who is going to start the conversation. Assign that person the task of choosing the next person to speak and so on until everyone has had their turn at contributing. This will help keep each team member’s attention as they will not sure when they will be called on. Similarly (but less desirable) randomly call on team members and vary the pattern day-to-day.
- When conducting a meeting where only one or two people are attending remotely, ensure that both the conference room and the remote individuals are using video. This will allow for additional non-verbal cues to be included in the communication and help keep the conference room aware that one or more of their team members is attending remotely.
- Leverage chat for one-off conversations or bringing non-priority items to the host’s attention
- If video is in use, be aware of the need for patience when waiting for network latency to catch up with any shared screens.
- Start the meeting with everyone on video. After a few minutes for casual conversation and catching up, allow team members to switch off video to reduce bandwidth issues.
- Provide a recap of what was accomplished and identify any of the goal or agenda items were not addressed.
- State any next steps.
- State any action items and the names of people responsible for completing them. Action items without assigned names are in-action items.
- If a member of the team has a pattern of responding to questions with some version of “I didn’t understand the question. Can you restate it?” or they have to be prompted with “Are you there?” due to a non-response, there is a likelihood that they are not focused on the meeting and busy with something else. If the pattern is persistent, the scrum master will need to discuss this with the team member off-line.
- Be attentive to the introverts on the team and work to have them contribute to each meeting. Ask them for their opinion or any open-ended question that has more than a yes-no answer.
- The metrics a scrum master may use to assess team participation and performance will need to be more robust than the usual data-based and subjective measures. Assessing morale or the presence of intra-team conflict may not be as apparent with distributed teams. Scrum masters may need to meet virtually one-to-one with each of the team members more frequently and asses any time management or morale issues.
With apologies to Winston Churchill,
Agile in general, and scrum in particular, has suffered their share of hard yet deserved knocks. But many of these complaints come from people who are expecting perfection, some panacea or magic remedy to what ails their project management world. Often they want this perfection out of the box and miss the hard work needed to implement a relatively simple set of rules and guidelines while shifting from the “old ways” of getting work done.
Over the course of hundreds of thousands of years they have worked out an efficient way to migrate. Not perfect, but well adapted to the world in which they live. At the heart of this behavior are several important principles: Shared responsibility, clear communication, and coordinated effort.
Consider Agile similarly. It is a perfect system for an imperfect world. The principles found in the formation of a flock of geese can be found within the Agile Manifesto. Its foundation of assuming the need for experimentation, learning, and adaptation is central to it’s enduring success. If these values are absent from or poorly represented in an organization’s culture, the chances for sustainable success using any methodology are diminished.
Floating with the stream.
I’m coding my own design.
Who are you? Logjam!