Wait for it ... process flow charts are really data flows?
"Everything Starts Out Looking Like a Toy" #111
Hi, I’m Greg 👋! I write essays on product development. Some key topics for me are system “handshakes”, the expectations for workflow, and the jobs we expect data to do. This all started when I tried to define What is Data Operations?
This week’s toy: creating vintage tintype style photography of She-Ra, He-Man, and Skeletor with Stable Diffusion. These generative AI models that you can run on your laptop are really going to change our perception of how to create art (and other things soon as well). Creating prompts for an AI is going to be a new skill for this decade. Edition 111 of this newsletter is here - it’s September 19, 2022.
The Big Idea
A short long-form essay about data things
⚙️ Wait for it ... process flow charts are really data flows?
Flow charts are straightforward. To define a business process, draw a box for the next thing that needs to happen, create a specialized shape when there is a decision, and annotate the lines to let the reader know what’s happening. What could be easier? I suppose you could invent a workflow that consisted of sentences or just watched you work and suggested an optimal flow.
But flow charts hide the most important piece of a process. They don’t explain in detail what’s happening to the underlying data and define how to resolve conflicts.
The main reason for this is visual complexity. Think about the start point of a process: you need the reader to understand when it starts and to identify that they are in the process. By definition a process needs one entry point. The best way tot do this in a visual display is to pick an entry point that is undeniable. For example, the figure above shows “New Lead”.
In a typical sales process, New Lead means “the first time we’ve encountered a person showing intent to purchase or convert.” The lead could be created from a form fill, attendance at an event, or a person starting a conversation. The starting point for handling that lead is pretty straightforward: they’ve raised their hand. After this happens, the go to market motion springs into action, considering what should be done next, how long it should take, and who needs to take the action.
When creating a process diagram, most people stop here. They use the process abstraction for a flow chart to generally say what needs to happen and when, and the overall disposition of the result. For example, the chart above details the deduping of a lead (perhaps on email address and name), the matching of that lead to an existing contact and appending data, and perhaps the creation of a new account and contact if qualification criteria is met.
But how does all of that actually happen? Flow creators stop with a pretty picture because they expect other owners (with more knowledge of the data) to fill in the details of how to connect systems. They use the picture of the flow to help stakeholders in other departments understand what’s going on with their intent. And they want the system to reinforce their intent so that metrics, playbooks, and outcomes will be congruent with the flow chart.
Flows hide information by design
Flows don’t tell the whole story. They are a simplification of the data flows that are going on to achieve the results of a flow chart. If you take the lead-to-account matching flow described above and show the the multiple data flows that result, you have a different picture. When a process flow executes as changes in data among objects, there are many other things going on.
We want to describe the process as a series of simultaneous data flows because data flows…
bring us closer to the actual data changes that happen across systems to enforce process
happen as a result of changes, including the changes invoked by this process
enable us to see when process flows overlap and potentially conflict with each other
Data flows let you drill down through the abstraction of the flow chart and use the actual data relationships and to demonstrate the changes that cause data to update across related objects to the current data flow. Because we model the simultaneous changes going on, it’s possible to understand the impact of the process flow and when and how it will be applied to the end state of the data.
The same process described as a data flow
How do we move from talking about process flows to talking about data flows?
For our example of lead to account matching, it’s not that much different than the original flow. The main difference is the chain of events that results from the original decision.
When we create the lead, the following happens:
We look up the lead, see if we have a duplicate based on criteria, and merge if the criteria is met based on the rule for all leads
If there is no duplicate contact, we look for an existing contact, and if we find one append data based on the rules for all contacts
If there is no duplicate contact, we start a lookup for an existing company and attach the new contact if found
If no company found, we check the criteria for all companies and check whether we need to create a new company
If that criteria is met, we create a new company, create a new contact, and associate that contact with the company
If not, we put that lead in a longer-term nurture flow
What’s happening? Each decision point in the flow chart corresponds to action in a continuous data flow like this one:
The swim lanes in the diagram above represent three simultaneous processes (and there are more executing at the field level for each of these objects, but you get the idea). Every cycle, each of the object data flows detect changes. When they see a change they execute actions in their pipeline by comparing the values to previously known values.
The basic flow we started with doesn’t show this complexity and tell you what’s really going on. To create the instructions for the input and output of the workflow, you need to define:
the actual data changes - what data moves from one object to another
resulting creates or deletes - how to create fields, combine information, and resolve conflicts
entry and exit points - defining the triggers to start and end data changes
Defining the data flows gives you the true picture of the movement for data inside your system. When is this really valuable? When you add the second flow that touches the same object. Flow charts fail to note any potential conflicts, while data flows are expressly designed to define and resolve these issues.
What’s the takeaway? When you’re building a process, go beyond the basic boxes and arrows and create data flows to demonstrate desired data changes, the way you will resolve those changes, and note the data field triggers that start and end a process. Data flows future-proof your organization for integration and help you diagnose issues faster.
Links for Reading and Sharing
These are links that caught my 👀
1/ AI will change the world… - in different ways than we think today. Ben Thompson writes in a fascinating essay on AI Unbundling that idea creation and manipulation is going to create some weird outcomes. The straightforward ones are that we will be able to auto-generate things that look a lot like things we rely on creators to visualize today. The not quite easy to see things look like entirely new forms of composing, authoring, and validating information.
2/ Open source design solutions - If you saw the news about Adobe buying Figma this week and want an open source alternative, check out Penpot. It seems to have many of the browser-based quick creation benefits of Figma. Without paying customers, it depends upon the community for support. On the other hand, the community can take this wherever it wants to go.
3/ Why are car wheels their current shape? - Yep, this one’s random. It turns out the design of car wheels has changed to match the forces present in the suspension design. TL;dr: making car wheels flatter instead of concave produces better suspension dynamics.
What to do next
Hit reply if you’ve got links to share, data stories, or want to say hello.
Want more essays? Read on Data Operations or other writings at gregmeyer.com.
The next big thing always starts out being dismissed as a “toy.” - Chris Dixon