White belt. All is new.
Brown belt. Ha! I know it all!
Black belt, open eyes.
White belt. All is new.
White belt. All is new.
Brown belt. Ha! I know it all!
Black belt, open eyes.
A presentation I gave last week sparked the need to reach back into personal history and ask when I first programed a computer. That would be high school. On an HP 9320 using HP Educational Basic and an optical card reader. The cards looked like this:
What occurred to me was that in the early days – before persistent storage like cassette tapes, floppy disks, and hard drives – a software developer could actually hold their program in their hands. Much like a woodworker or a glass blower or a baker or a candlestick maker, we could actually show something to friends and family! Woe to the student who literally dropped their program in the hallway.
Then that went away. Keyboards soaked up our coding thoughts and stored them in places impossible to see. We could only tell people about what we had created, often using lots of hand waving and so much jargon that it undoubtedly must have seemed as if we were speaking a foreign language. In fact, the effort pretty much resembled the same fish-that-got-away story told by Uncle Bert every Thanksgiving. “I had to parse a data file THIIIIIIIIIS BIG using nothing but Python as an ETL tool!”
Way bigger yawn.
I can compare this to woodworking. (Something I very much enjoy and from which I derive a great deal of satisfaction.) If I’m making something for someone else, I put in extra effort to make it beautiful and functional. To do that, I may need to make a number of tools to support the effort – saw fences, jigs, and clamps. These hand-made tools certainly don’t look very pretty. They may not even be distinguishable from scrap wood to anybody but myself. But they do a great job of helping me achieve greater things. Things I can actually show and handle. And if the power goes down in the neighborhood, they’ll still be there when the lights come back on.
In a Scientific American column delightfully named “The Artful Amoeba” there is an article on a little critter called the “fire chaser” beetle: How a Half-Inch Beetle Finds Fires 80 Miles Away – Fire chaser beetles’ ability to sense heat borders on the spooky
Fire chaser beetles are thus so hell bent on that objective that they have been known to bite firefighters, mistaking them, perhaps, for unusually squishy and unpleasant-smelling trees.
This part is interesting:
But the beetle has a good reason. It’s getting its legs out of the way of its heat eyes, pits filled with infrared sensors tucked just behind its legs.
A strategy suggested by the fire beetle life cycle is if you want to maximize a signal to noise ratio, iterate through three simple things:
Also, the “Way of the Amoeba” is now the “Way of the Artful Amoeba.” Update your phrase books accordingly.
Additional thoughts about signals and noise that have been rattling around in my brain since first posting on this topic.
At the risk of becoming too ethereal about all this, before there is signal and before there is noise, there is data. Cold, harsh, cruelly indifferent data. It is after raw data encounters some sort of filter or boundary, something that triggers a calculation to evaluate what that data means or whether it is relevant to whomever is on the other side of the filter, that it begins to be characterized as “signal” or “noise.”
Since we’re talking about humans in this series of posts, that filter is an amazingly complex system built from both physiological and psychological elements. The small amount of physical data that hits our senses and actually makes it to our brains is then filtered by beliefs, values, biases, attitudes, emotions, and those pesky unicorns that can’t seem to stop talking while I’m trying to think! It’s after all this processing that data has now been sorted according to “signal” (what’s relevant) and “noise” (what’s irrelevant) for any particular individual. Our individual systems of filters impart value judgments on the data such that each of us, essentially, creates “signal” and “noise” from the raw data.
That’s a long winded way to say:
Now apply this to everyone on the planet.
As an example, Google, itself a filter, is a useful one. Let’s assume for a moment that Google is some naturally occurring phenomenon and not a filter created by humans with their own set of filters driving what it means to create a
let’s be evil good search engine. To retrieve 1,000,000 pieces of information, my friend, Bob, entered search criteria of interest to him, i.e. “filter 1.” Maybe he searched for “healthy keto diet recipes”. Scanning those search results, I determine (using my “filter 2”) 100% of the search results are useless because my filter is “how do i force the noisy unicorns in my head to shut the hell up”. The Venn diagram of those two search results is likely to show a vanishingly small set of relationships between the two. (Disclaimer: I have no knowledge of the carbohydrate content of unicorns nor how tasty they may be when served with capers and a lemon dill sauce.)
Google may return 1,000,000 search results. But only a small subset is viewable at a time. What of the rest of the result set that I know nothing about? Is it signal? Is it noise? Is it just data that has yet to be subjected to anyone’s system of filters? Because Google found stuff, does that make it signal? Accepting all 1,000,000 search results as signal seems to require a willingness to believe that Google knows best when it comes to determining what’s important to me. This would apply to any filter not our own.
All systems for distinguishing signal from noise are imperfect and some of us on the Intertubes are seeking ways to better tune our particular systems. The system I use lets non-relevant data fall through the sieve so that the gold nuggets are easier to find. Perhaps at some future date I’ll unwittingly re-pan the same chunk of data through an experienced-refined sieve and a newly relevant gem will emerge from the dirt. But until that time, I’ll trust my filters, let the dirt go as noise, and lurch forward.
All signal all the time may not be a good thing. So I’d like to offer a defense for noise: It’s needed.
Signal is signal because there is noise. Without the presence of noise we risk living in the proverbial echo chamber. When we know what’s bad, we are better equipped to recognize what’s good. I deliberately tune into the noise on occasion for no other reason than to subject my ideas to a bit of rough and tumble. Its why I blog. Its why I participate in several select forums. “Here’s what I think, world. What say you?”
Of course, noise is noise because there is signal. Once we’ve had an experience of “better” we are then more skilled at recognizing what’s bad. I remember the food I grew up on as being good, but today I view some of it as poison (Wonder Bread anyone?) And there are subjects for which I no longer check out the noise. The exposure is too harmful.
There are subjects for which I seem to be swimming in noise and casting around for any sort of signal that suggests “better.” I’m recalling a joke about the two young fish who swim past an older fish. The older fish says to the younger fish, “The water sure is nice today.” A little further on, one of the young fish asks the other, “What’s water?” I’m hoping to catch that older fish in my net. He knows something I don’t.
To understand what I mean by noise being necessary it is important to understand the metaphor I’m using, where it applies and where it doesn’t.
Taking the metaphor literally, in the domain of electrical engineering, for example, the signal to noise ratio is indeed an established measure with clear unit definitions as to what is reflected by the ratio – decibels, for example. In this domain the goal is to push always for maximum signal and minimum noise.
In the world of biological systems, however, noise is most definitely needed. One of many examples I can think of is related to an underlying driver to evolution: mutations. In an evolving organism, anything that would potentially upset the genetic status quo is a threat to survival. Indeed, most mutations are at best benign or at worst lethal such that the organism or it’s progeny never survive and the mutation is selected against as evolutionary “noise.”
However, some mutations are a net benefit to survival and add to the evolutionary “signal.” We, as 21st Century homo sapiens, are who we are because of an uncountable number of noisy mutations that we’ll never know about because they didn’t survive. Even so, surviving mutations are not automatically “pure” signal. There are “noisy” mutations, such as that related to sickle cell anemia. Biological systems can’t recognize a mutation as “noise” or “signal” before the mutation occurs, only after, when they’ve been tested by the rough and tumble of life. This is why I speak in terms of “net benefit.”
For humans trying to find our way in the messy, sloppy world of human interactions and thought, pure signal can be just as undesirable as pure noise. I’ll defer to John Cook, who I think expresses more succinctly the idea I was clumsily trying to convey:
This is a metaphor for life. If you only value your own opinion, you’re an idiot in the oldest sense of the word, someone in his or her own world. Your work may have a strong signal, but it also has a lot of noise. Getting even one outside opinion greatly cuts down on the noise. But it also cuts down on the signal to some extent. If you get too many opinions, the noise may be gone and the signal with it. Trying to please too many people leads to work that is offensively bland.
The goal in human systems is NOT to push always for maximum signal and minimum noise. For example, this is reflected in Justice Brandeis’s comment: “If there be time to expose through discussion the falsehood and fallacies, to avert the evil by the process of education, the remedy to be applied is more speech, not enforced silence.” So my amended thesis is: In the domain of human interactions and thought, noise is needed by anyone seeking to both evaluate and improve the quality of the signal they are following.
A final thought…
If we were to press for eliminating as much “noise” as possible from human systems much like the goal for electrical noise, I’m left with the question “Who decides what qualifies as noise?”
A question was posed, “Why not be an information sponge?”
I’d have to characterize myself as more of an information amoeba – (IIRC, the amoeba is, by weight, the most vicious life form on earth) – on the hunt for information and after internalizing it, going into rest mode while I decompose and reassemble it into something of use to my understanding of the world. Yum.
More generally, to be an effective and successful consumer of information these days, the Way of the Sponge (WotS, passive, information washes through them and they absorb everything) is no longer tenable and the Way of the Amoeba (WotA, active, information washes over them and they hunt down what they need) is likely to be the more successful strategy. The WotA requires considerable energy but the rewards are commensurate with the effort. WotS…well, there’s your obsessive processed food eating TV binge-watcher right there. Mr. Square Bob Sponge Pants.
What’s implied by the WotA vs the WotS is that the former has a more active role in optimizing the informational signal to noise ratio than the latter. So a few thoughts to begin with on signals and noise.
Depending on the moment and the context, one person’s signal is another person’s noise. Across the moments that make up a lifetime, one person’s noise may become the same person’s signal and vice versa. When I was in high school, I found Frank Sinatra’s voice annoying and not something to be mingled with my collection of Mozart, Bach, and Vivaldi. Today…well, to disparage the Chairman of the Board is fightin’ words in my house. Over time, at least, noise can become signal and signal become noise.
But I’m speaking here of the signal quality and not it’s quantity (i.e. volume)
Some years ago I came across Stuart Kauffman’s idea of the adjacent possible:
This has been interpreted in a variety of ways. I carry this around in my head as a distillation from several of the more faithful versions: Expand the edge of what I know by studying the things that are close by. Over time, there is an accumulation of loosely coupled ideas and facts that begin to coalesce into a deeper meaning, a signal, if you will, relevant to my life.
With this insight, I’ve been able to be more deliberate and directed about what I want or need to know. I’ve learned to be a good custodian of the edge and what I allow to occupy space on that edge. These are my “internal gating mechanisms.” It isn’t an easy task, but there are some easy wins. For starters, learning to unplug completely. Especially from social media and what tragically passes for “news reporting” or “journalism”these days.
The task is largely one of filtering. I very rarely directly visit information sources. Rather, I leverage RSS feeds and employ filtering rules. I pull information of interest rather than have it pushed at me by “news” web sites, cable or TV channels, or newspapers. While this means I will occasionally miss some cool stuff, it’s more than compensated by the boost in signal quality achieved by excluding all the sludge from the edge. I suspect I still get the cool stuff, just in a slightly different form or revealed by a different source that makes it through the filter. In this way, it’s a matter of modulating the quantity such that the signal is easier to find.
There is a caution to consider while optimizing a signal-to-noise ratio, something reflected in Kauffman’s comments around the rate of exploration for new ideas: “If they did it too fast, they would destroy their own internal organization…”
Before the Internet, before PCs were a commodity, before television was popular it was much, much easier to find time to think. In fact, it was never something that had to be looked for or sought out. I think that’s what is different today. It takes WORK to find a quiet space and time to think. While my humble little RSS filters do a great job of keeping a high signal-to-noise ratio with all things Internet, accomplishing the same thing in the physical world is becoming more and more difficult.
The “attention economy,” or whatever it’s being called today, is reaching a truly disturbing level of invasion. It seems I’ve used more electrician’s tape to cover up camera lenses and microphones in the past year than I’ve used on actual electrical wires. The number of appliances and gadgets in the home with glowing screens crying out for bluetooth or wifi access like leaches seeking blood are their own source of noise. This is my current battleground for finding the signal within the noise.
Enough about filtering. What about boundaries. Fences make for good neighbors, said someone wise and experienced. And there’s a good chance that applies to information organization, too. Keeping the spiritual information in my head separate from my shopping list probably helps me stop short of forming some sort of cult around Costco. ( “All praise ‘Bulk,’ the God of Stuff!)
An amoeba has a much more develop boundary between self and other than a sponge and that’s probably a net gain even with the drawback of extra energy required to fuel that arrangement. Intellectually, we have our beliefs and values that mark where those edges between self and other are defined.
So I’ll stop for now with the question, “What are the strategies and mental models that promote permeability for desired or needed information while keeping, as much as possible, the garbage ‘out there?’”
I hear this (or some version) more frequently in recent years than in past:
I attribute the increased frequency to the increased popularity of Agile methods and practices.
That the “Responding to change over following a plan” Agile Manifesto value is cherry picked so frequently is probably due to a couple of factors:
The intent behind the “Responding to change” value, and the way successful Agile is practiced, does not allow for constant and unending change. Taken to it’s logical conclusion, nothing would ever be completed and certainly nothing would ever be released to the market.
I’m not going to rehash the importance of the preposition in the value statement. Any need to explain the relativity implied by it’s use has become a useful signal for me to spend my energies elsewhere. But for those who are not challenged by the grammar, I’d like to say a few thing about how to know when change is appropriate and when it’s important to follow a plan.
The key is recognizing and tracking decision points. With traditional project management, decisions are built-in to the project plan. Every possible bit of work is defined and laid out on a Gantt chart, like the steel rails of a train track. Deviation from this path would be actively discouraged, if it were considered at all.
Using an Agile process, decision points that consider possible changes in direction are built into the process – daily scrums, sprint planning, backlog refinement, reviews and demonstrations at the end of sprints and releases, retrospectives, acceptance criteria, definitions of done, continuous integration – these all reflect deliberate opportunities in the process to evaluate progress and determine whether any changes need to be made. These are all activities that represent decisions or agreements to lock in work definitions for short periods of time.
For example, at sprint planning, a decision is made to complete a block of work in a specified period of time – often two weeks. After that, the work is reviewed and decisions are made as to whether or not that work satisfies the sprint goal and, by extension, the product vision. At this point, the product definition is specifically opened up for feedback from the stakeholders and any proposed changes are discussed. Except under unique circumstances, changes are not introduced mid-sprint and the teams stick to the plan.
Undoing decisions or agreements only happens if there is supporting information, such as technical infeasibility or a significant market shift. Undoing decisions and agreements doesn’t happen just because “Agile is all about changing requirements.” Agile supports changing requirements when there is good reason to do so, irrespective of the original plan. With traditional project management, it’s all about following the plan and change at any point is resisted.
This is the difference. With traditional project management, decisions are built-in to the project plan. With Agile they are adapted in.
I’m often involved in discussions about Agile that question its efficacy in one way or another. This is, in my view, a very good thing and I highly encourage this line of inquiry. It’s important to challenge the assumptions behind Agile so as to counteract any complacency or expectation that it is a panacea to project management ills. Even so, with apologies to Winston Churchill, Agile is the worst form of project management…except for all the others.
Challenges like this also serve to instill a strong understanding of what an Agile mindset is, how it’s distinct from Agile frameworks, tools, and practices, and where it can best be applied. I would be the first to admin that there are projects for which a traditional waterfall approach is best. (For example, maintenance projects for nuclear power reactors. From experience, I can say traditional waterfall project management is clearly the superior approach in this context.)
A frequent challenge the idea that with Agile it is difficult to do any long-term planning.
Consider the notion of vanity vs actionable metrics. In many respects, large or long-term plans represent a vanity leading metric. The more detail added to a plan, the more people tend to believe and behave as if such plans are an accurate reflection of what will actually happen. “Surprised” doesn’t adequately describe the reaction when reality informs managers and leaders of the hard truth. I worked a multi-million dollar project many years ago for a Fortune 500 company that ended up being canceled. Years of very hard work by hundreds of people down the drain because projected revenues based on a software product design over seven years old were never going to materialize. Customers no longer wanted or needed what the product was offering. Our “solution” no longer had a problem to solve.
Agile – particularly more recent thinking around the values and principles in the Manifesto – acknowledges the cognitive biases in play with long-term plans and attempts to put practices in place that compensate for the risks they introduce into project management. One such bias is reflected in the planning fallacy – the further out the planning window extends into the future, the less accurate the plan. An iterative approach to solving problems (some of which just happen to use software) challenges development teams on up through managers and company leaders to reassess their direction and make much smaller course corrections to accommodate what’s being learned. As you can well imagine, we may have worked out how to do this in the highly controlled and somewhat predictable domain of software development, however, the critical areas for growth and Agile applicability are at the management and leadership levels of the business.
Another important aspect the Agile mindset is reflected in the Cone of Uncertainty. It is a deliberate, intentional recognition of the role of uncertainty in project management. Yes, the goal is to squeeze out as much uncertainty (and therefore risk) as possible, but there are limits. With a traditional project management plan, it may look like everything has been accounted for, but the rest of the world isn’t obligated to follow the plan laid out by a team or a company. In essence, an Agile mindset says, “Lift your gaze up off of the plan (the map) and look around for better, newer, more accurate information (the territory.) Then, update the plan and adjust course accordingly.” In Agile-speak, this is what is behind phrases like “delivery dates emerge.”
Final thought: You’ll probably hear me say many times that nothing in the Agile Manifesto can be taken in isolation. It’s a working system and some parts if it are more relevant than others depending on the project and the timing. So consider what I’ve presented here in concert with the Agile practices of developing good product visions and sprint goals. Product vision and sprint goals keep the project moving in the desired direction without holding it on an iron-rails-track that cannot be changed without a great deal of effort, if at all.
So, to answer the question in the post title, Agile helps with long term planning by first recognizing the the risks inherent in such plans and implementing process changes that mitigate or eliminate those risks. Unpacking that sentences would consist of listing all the risks inherent with long-term planning and the mechanics behind and reasons why scrum, XP, SAFe, LeSS, etc., etc., etc. have been developed.
Four little values.
Twelve principles to follow.
Why is this so hard?
Remember the phrases from earlier in the article series? Here they are again.
How much more meaningful these are to you now that you understand a little more about the system dynamics that drive projects. Choose just one of these and find where it’s reflected in the model. (Figure 1)1.
Now follow the impact and consequences around the various feedback loops. Reflect for a moment an ask yourself, “What can I do to help keep the system healthy and productive in light of what I now know may be happening?” There’s a lot to consider. We’ll cover several options in this article.
Moving from the outside in, the most visible nodes in the system are also influenced the least by direct intervention. These are Morale, Fatigue, and Experience. “The beatings will continue until morale improves” is, I hope, recognized as a cynical joke. While offering free coffee, Red Bull, and unlimited M&Ms may perk up employees in the short term, the long term health consequences are grim indeed. As for Experience, well, that just takes time and a great deal of effort to fully shape and mature.
Attempting to alter these nodes directly is likely to be wasted effort at best and more probably harmful. Even if some cursory improvement can be made, the underlying systemic influences – the true drivers – will still be present and will exert a far more powerful influence. It’s Conway’s Law, pure and simple. It’s better to thinking of Morale, Fatigue, and Experience as symptoms or indicators to be recognized and tracked rather than root causes to be treated. As indicators, they are incredibility powerful sources of information on whether or not changes made to other parts of the system are being successful. They are to be used, not abused.
We’ll begin by working backward from the disaster that was built up over the last several articles in the series. Let’s imagine we have a demoralized team (or teams) that are exhausted and burdened with an impossible delivery schedule. As it stands, it’s unfixable. A sprinter has a better chance of breaking the three minute mile than this team has in delivering their project by the stated delivery date.
Let’s also assume the choice is to continue the project. The two major actions for management at the is point are to move the Deadline and reduce the amount of Work to Do in the system. These aren’t choices, they’re actions that need to be engaged thoughtfully.
Simply moving the date to some point in the future that seems “doable” is yet another gamble. Neither will moving the date instantly resolve the other systemic issues. There is a considerable amount of recovery and rebuilding to be completed. It takes time to hire the people needed to rebuild the workforce. It takes time to rebuild trust and morale among the employees that remain. Moving the deadline out will begin to relieve pressure, but it will take time for the inflamed system to cool down and find an optimal working temperature.
The challenge for this first step is: How can you go about finding what is a reasonable date for the deadline? Answering this question is dependent on what is learned by looking to other parts of the system model for data.
Probably, the best thing to do is to declare that for some time boxed period, there will be no deadline date while these and many other questions are explored. This will have a side benefit of signaling to the development teams that management is serious about finding a realistic date. This will help to start rebuilding trust between management and the development teams.
One of the factors to consider in determining whether a new deadline can reliably be set is the Total Known Remaining Work in the system. As has been discussed previously, increasing the Total Known Remaining Work puts pressure on the completion date. Similarly, decreasing the
Total Known Remaining Work by some means will increase the likelihood that the completion date can be met. Actions to take that will allow management to regain control of the work flow include:
It’s been shown that changes in one part of the system will affect other parts of the system, whether by design or not. In this article we’ve discussed how adjusting the Deadline and Total Known Remaining Work can affect each other and the entire system. When adjusted in a way that considers system-wide effects, they can help restore balance and predictability to the overall system.
Previous article in the series: Assessing and Tracking Team Performance – Part 7: “Abandon All Hope,…”
1The core of the model I use to assess team and organization health is based on the work of James Lyneis and David Ford: System Dynamics Applied to Project Management, System Dynamics Review Volume 23 Number 2/3 Summer/Fall 2007