Artificial Intelligence (AI) is the latest ‘gadget’ that companies want to add to every project. However, it’s not the silver bullet that many folks hope it to be. It has its strengths and limitations. To overcome AI’s limitations, Augmented Intelligence optimizes the strengths of Artificial Intelligence, Machine Learning (ML), and human capabilities.

The fundamental limitation of AI is not the programming, but the design of the AI algorithms. Artificial Intelligence is a rules-based solution, and a traditional rules-based AI requires complete and accurate rules. It’s up to the designers to accurately define the right rules. Historically, humans are not very good at predicting every eventuality or possibility, and thus do a poor job of specifying the rules.

And then there’s the issue of the data used to train and operate an AI system.

Humans routinely arrive at successful solutions based on ambiguous, inaccurate, and incomplete data. Computer and AI systems require accurate data to derive an accurate solution. Unfortunately, inaccurate data is a common occurrence, and it’s not always possible to identify the inaccuracies. For instance, there may be inherent biases in the data that will alter the results. These biases may be created by virtue of how the data was collected or due to an unrepresentative data set.

Accurate data is critical, especially if the tool includes a Machine Learning component. A learning engine is only as good as the data it learns from. A good AI solution should include a learning engine to constantly evolve its rules based on actual usage.

Creating a training data set requires a lot of human processing, which may account for much of the unintended biases in the data set. A human decides what data to use to train a learning engine. Without knowledge of how the training algorithm works, they may exclude data that would otherwise be useful and include data that confuses the learning engine.

Besides data accuracy issues, AI systems are not designed to understand inferences as well as humans. An AI engine might be good at taking ice cream orders, such as a chocolate ice cream cone with candy chunks on it, but the same engine would not know how to handle a request for a cone, “Just like that one only with sprinkles.”

AI systems are designed and trained to perform specific functions with specific data sets. They are not yet sophisticated enough to generalize across all inputs to learn everything. Humans are uniquely capable of transferring knowledge across domains to operate effectively in a novel environment based on what we have learned in other, non-related environments.

Humans have limitations, as well. Humans just don’t have memory processing capabilities as good as computers. We are not able to process large amounts of data, we cannot keep track of many things at one time, and we cannot recall things with perfect accuracy. These human limitations are AI strengths.

So, one of the big questions is how to leverage the strengths of humans to overcome the weaknesses of AI, and vice-versa. Augmented Intelligence is a model that emphasizes the assistive role AI can have in enhancing human cognition rather than trying to mimic or replace it. A good example of the use of Augmented Intelligence is in the Automated Readiness Forecasting (ARF) tool designed by the AF CyberWorx team.

ARF accepts mission parameters, such as an exercise several months away, and schedules qualification events to ensure that appropriate personnel and resources are available for the exercise. It also keeps track of all necessary equipment maintenance as well as personnel training and medical activities needed prior to the exercise. It suggests schedule changes which will ensure the required resources are ready in time. Affected personnel only have to approve or acknowledge any adjustments. The tool then tracks progress, highlighting any deviations to the plan and suggesting corrections along the way to stay on track. Machine Learning uses the repeated iterations and changes to evolve the scheduling algorithm, fine-tuning its capabilities and reducing reliance on human intervention.

The readiness tool assumes the mundane tasks humans would otherwise have to perform to keep track of readiness data and immediately adjusts the plan to accommodate any changes to resource availability. Normally, such adjustments would require dozens of people working hours to accomplish. Humans are still required to approve changes, but are relieved of the time-consuming and error-prone tasks associated with adjusting schedules manually. This tracking and automation saves hundreds of resource hours, eliminates errors, and ensures a higher level of readiness.

This example illustrates an Augmented Intelligence and Machine Learning paradigm that could be applied to many Air Force projects rather than a typical Artificial Intelligence approach. The point is, AI is not a panacea for all problems. AI is successful when the problem domain is well understood, the data set is accurate, and the AI is focused on a specific task. Knowing when to design for an Augmented Intelligence verses an Artificial Intelligence – and the difference between the two – is crucial to success. Not every problem needs an AI solution.

*The postings on this blog reflect individual team member opinions and do not necessarily reflect official Air Force positions, strategies, or opinions.

Dissecting the Design Sprint Event

PART FIVE: AFTER THE SPRINT: NEXT STEPS

A design sprint’s ultimate goal is to improve a situation, whether that means improving an existing process or developing a new product. Not acting on the solutions a team suggests means a design event has not fulfilled its purpose. AF CyberWorx continues working beyond the sprint itself to help the results of a design sprint reach implementation.

The last step of the event is the outbrief where participants present their findings to the stakeholders. With team suggestions in mind, AF CyberWorx uses their own crack team to work with stakeholders to determine the next steps.

Greg Bennett, AF CyberWorx Relationship Manager, explains, “Every single sprint is going to be different. The outcomes are going to be different. The resources required on the back end are going to be different.” As such, AF CyberWorx customizes a roadmap to the end result according to the needs, limitations, and available resources of the problem owner. “We will continue to stay engaged as they want us to stay engaged,” Greg reassures. However, “We will not fight the fight for them.” Even with CyberWorx assistance, “Transition will not happen without [the champion’s] direct involvement and advocacy.”

Each champion and problem owner has different levels of capability and needs. As such, AF CyberWorx has different tools at their disposal to help. While some projects proceed best after transitioning to a different Air Force agency (which is an option), others are best served by going through a research and contracting process to get industry assistance. AF CyberWorx acts as a bridge between the problem owner and the contracting office to assist with that.

Once the problem owner and AF CyberWorx decide to pursue contracting options, Erica Wilson, Contracting Officer, and Casey Pehrson, Contracting Specialist, go to work. Erica explains that with AF CyberWorx acting as their technical point of contact, “we connect market research to figure out if anybody’s doing what they’re wanting done, who’s doing it, and what types of businesses are doing it. From there, we decide on a contracting vehicle.” The contracting office looks for the most streamlined avenue based on available information, resources, and the problem owner’s timeline. To best do that, as Casey says, “We just need to come together as a government team and find the best way forward.”

The problem owner can help that process most by clearly defining their wants and needs: what they need, what they’re trying to buy or do, and what their limitations are. Information from the design sprint really helps with this, but the more specific the project parameters, the better the contracting office can find the best paths forward.

“Transition [to the end result] requires continued involvement with the customer,” Greg stresses, “to formulate strategies…and advocate with leadership, program offices, and sustainment functions.” It takes a dedicated team to champion a solution and transition it from the idea stage to implementation. From the discovery call laying the groundwork through the problem solving process to drawing out a roadmap to implementation, AF CyberWorx helps guide the process by connecting the right people at the right time. While we don’t do the fighting, we coach problem owners as needed to refine and mitigate Air Force challenges.

*The postings on this blog reflect individual team member opinions and do not necessarily reflect official Air Force positions, strategies, or opinions.

Dissecting the Design Sprint Event

STEP THREE: SOLUTION DESIGN

The solution design portion of a design sprint is often considered the most fun and free-flowing part of an event. This is what people come to the event for: to come up with solutions to their problem. What most newcomers to the design sprint process don’t realize is that the solution design portion is still part of a much larger process. Knowledge and artifacts from the previous steps in the event feed into ideating solutions and then paring them down for realistic solutions that will have a strong impact on the organization.

All the work the stakeholders and participants have done up to brainstorming solutions lead to creating a wide pool of potential solutions. Cole Stamm is an AF CyberWorx UX Research Fellow through the ORISE program. As he explains, “The personas, journey maps, task flows, task optimization – all that is the foundation for creating ideas off of.” From the initial discovery call to writing out personas and scenarios, the artifacts that map out the problem-solving journey the team has taken leads up to ideation. Cole went on, “You can’t create solutions without doing those first steps or they’re just arbitrary solutions.” Arbitrary solutions don’t provide the impact to the organization a team hopes to create.

The best solution design phases pull in focused information and give back actionable, well-developed solutions. Cole relates a specific event that had excellent results. Part of the key, he says, was because “it had a very specific focus…with very clear constraints and personas.” Another successful event “had a number of interesting solutions that benefited from the task analysis approach.” For each, the solutions were based on a strong foundation built throughout the event. The solutions focused on the end user and solving their pain points.

Ultimately, solution design is really just another step in the entire problem-solving process. Cole stresses, “Problem solving goes on throughout the process. It doesn’t really stop with the solution process.” AF CyberWorx facilitators are there every step of the way to assist the team with staying focused on the problem and reminding them of the gains they’ve made during their problem-solving journey to finish strong with impactful solutions.

The Solution Design Process

Once the team has a cohesive, shared vision of the problem, end goal, and requirements for success, they enter the ideation phase. This is the fun part. Cole outlines some key things participants need to know about ideating solutions: “Stay open to making connections between different peoples’ ideas…put off judgement until a later time…stay focused on the user at the center of the problem.” Keeping these ideas in mind, let the ideas flow freely, have fun, and trust that the process will weed out the ideas that are too complex, expensive, or off topic later.

The next step puts the products of previous steps to active use. Often, a team will find a process or design has specific requirements, objectives, or key questions that need to be addressed. After creating a large group of great ideas, the team needs to whittle down the number of solutions to narrow their focus. As Cole states, the group is looking for “the idea that’s really going to change the game.” The team votes, considering which solutions would be “easy wins” which would have the greatest impact on the problem, and which solutions generate the most passion among the participants. After that, asking which requirements each solution meets helps prioritize them and sometimes further weeds out unlikely solutions.

Now the team has a set of solutions that the majority agree meet the requirements, will be impactful to the problem and users, and are feasible. The team next needs to prepare those ideas for action. To do this, each solution gets fleshed out with a process Cole calls “make it or break it.” Here, the critical cap needs to go back on to determine the pros and cons of a solution and what acts as assists and barriers to the solution. Some examples of both include expenses, manpower, time to implement, or utilizing existing resources. Ultimately, it’s the decision-makers that decide if a solution is a go. It’s the participants’ job to give them the information they need to make the right decision.

AF CyberWorx guides participants through a tried-and-true process from discovery to solution design so a problem-solving team’s efforts result in finding a direction to improvement. Whether the solutions call for a change in process, prototypes for a new product, or a proof of concept, we help the team discover areas for improvement and develop paths towards success.

*The postings on this blog reflect individual team member opinions and do not necessarily reflect official Air Force positions, strategies, or opinions.

Dissecting the Design Sprint Event

PART TWO: BREAKOUT GROUPS

AF CyberWorx design sprints are tailored to the needs of the stakeholders. One crucial aspect is the number and diversity of participants in the group. Events at AF CyberWorx can have as few as five people and as many as forty or fifty. In each group, dynamics are determined by individual personalities, experience with the problem area, and strength of opinion as well as rank and positions (and perceived deferment to such). Each element may affect how people interact with one another.

However, each event is very short. The typical forming, storming, and norming growth of a group needs to happen as quickly and painlessly as possible while still gathering as much information and as many ideas as possible. The answer to this is to break a larger group into smaller diverse teams of four or five people.

Ideas are the meat of the problem-solving process. Ed Mikos, UX Design Analyst at AF CyberWorx, explains that “the information that’s created and captured [during an event] is going to be built on in a series of expansion and contraction steps leading to one final outcome.” In each step, ideas are generated from the group to expand potential. Those ideas are then vetted to contract focus down to the best, most impactful ideas that will push problem-solving efforts further. A team needs that pool of ideas to find which are the best.

Breaking the group into smaller focus teams gives everyone a voice. Ed states, “Everyone in the group brings something to the table.” Each person in the event is there for a reason and has good ideas from a different perspective. Smaller groups composed of diverse people takes out the possibility of a single, strong voice speaking for their peers in a larger crowd. Position becomes less of an issue when the higher ranked person is in a different group. The senior airman who actually uses a program has equal voice to the colonel who oversees the management of a similar office.

Another benefit of breakout groups comes when the groups return from a session and share their findings. Sometimes more than one group will have the same or similar ideas. That’s O.K.! Ed says when the same idea comes from multiple groups, “there’s probably something there [that needs to be explored].” Instead of being repetitive, the multiple groups reinforce each other, giving consensus and higher strength to that idea.

Groups increase idea generation, break down group dynamics to give everyone a voice, and often generate consensus faster than a large group as a whole. With those three benefits of breakout groups, Ed gives some key points for participants to remember when they move into the smaller focus groups during an event: “Everyone in the group brings something to the table. There are no bad ideas. Trust the process.”

He goes into more detail on some of these by explaining that “When someone is locked on an idea or someone is being too negative, like saying ‘Why are we doing this?’…that negativity is infectious.” Being a champion for your own idea is fine; that idea may be validated. However, keep an open mind. Each member of a group has a unique perspective and adds to the quality of the end product. “Allow for diversity of ideas…people come up with different ideas and know what the different feasibility and technical and organizational shortcomings will be.” When a diverse group works together with a positive attitude towards the process, the entire problem-solving team improves.

For those who are still hesitant to speak up or grab a marker and add their ideas, Ed reassures that “no one cares how good an artist you are or how eloquent you are. We need your ideas. You’re in the room for a reason.” Each idea adds to the whole. Each group’s findings add to the quality of the event. Every participant who speaks up adds valuable feedback that ensures a quality end product.

To assist the problem-solving team, AF CyberWorx provides experienced facilitators to encourage breakout teams to participate and stay on track. We know time is precious during a design sprint and how valuable each participant’s voice is in the group. Speak up and allow us to help you find the best possible solutions to your unique problem.

*The postings on this blog reflect individual team member opinions and do not necessarily reflect official Air Force positions, strategies, or opinions.

Dissecting the Design Sprint Event

PRE-GAME PART ONE: DISCOVERY CALL

When AF CyberWorx gets a request to run a design sprint, it’s like kicking an anthill…without all the biting that follows. There’s a series of meetings scheduled, logistics starts plans to make everything run smoothly, and team members are selected. One of the most important pieces to this initial organized scramble is the discovery call.

Dr. Dan Padgett, one of AF CyberWorx’s excellent user experience designers, describes the role of the discovery call as letting the facilitating team get a glimpse of the situation. This glimpse allows them to get into the stakeholders’ minds a little. As he explains, “At the end of the day, as UX designers, we’re not the subject matter experts in the field, so we need to try and understand as much as we can so that we can effectively plan for the problem solving session, whatever form that might take.”

The forms an AF CyberWorx event takes can be from a full five-day problem solving and design sprint to a simple site visit. The discovery call narrows down how long the process will take and what needs to be done.

One of the big pieces the team tries to discern is how solid a problem statement the stakeholders have. As Dr. Padgett explains, “If through a discovery session, we feel like we’ve got a really good handle on the scope of the problem and what they’re actually trying to solve, then we might do a lot less on the discovery portion of the design sprint.” The first day of an event is taken to refine the problem. How long the team spends on this exercise depends largely on how thoroughly the stakeholders have explained their situation and what they want to improve.

Paired with that is the need for clear goals. Everyone has problems, but fewer know what they want the end state to look like. “We want the problem to go away,” is not a goal. “We want the system to improve,” is not a goal, either. A goal needs to be actionable and measureable. The clearer the desired end state is, the more focused a problem solving team will be.

A piece of information that most people don’t think of is the understanding that not all AF CyberWorx team members are well steeped in every cyber career field. We don’t always understand the jargon. As Dr. Padgett describes it, some calls are extremely technical. The experts are on the line and use “acronyms without ceasing.” In the case that our team does not understand,the biggest help is to know what the acronyms mean and pass on that information early in the process. The discovery call is what shapes the overall event. It gives the team at AF CyberWorx the information it needs to build an event that will have the greatest impact. The best way to make the event as successful as possible as early as possible is for the discovery call to go well. Dr. Padgett explains some items to get the most out of discovery calls:

• Provide enough time before the event that an AF CyberWorx team can make a site visit. Seeing the work environment and the job actually being performed by the users themselves always gives information that someone used to the environment won’t recognize as significant.
• Ensure the right people are on the call. Stakeholders are important, yes; but having some of the users that experience the pain points normally adds more practical information.
• Know who the stakeholders are and who the decision makers are. They aren’t always the same. Knowing who will receive the information to make the final decision helps ensure that the final products are designed for maximum impact.
• Provide concrete information. Who are the users/customers of the system in question? What process/product/challenge is needing improvement or a solution? What are your goals?
• Exploring the situation and asking questions are part of every discovery call. Come with information and have people in the know on the call so the facilitating team can get a clear picture of what’s going on.
• A high-level comprehensive picture is more important than a detailed breakdown of the process. The detailed work will go into the event.

Ultimately, the discovery call is an introduction to the problem solving event. It shapes it, defines it, and prepares the AF CyberWorx team to guide a team of experts through the problem solving process. Information goes both ways. We also explain expectations while we’re gaining information. We aren’t the experts on the problem, but we are experts at guiding others through solving their problem. Help us help you.

*The postings on this blog reflect individual team member opinions and do not necessarily reflect official Air Force positions, strategies, or opinions.

Dissecting the Design Sprint Event

PRE-GAME PART TWO: EXPECTATIONS

“Our thoughts about an event can have a dramatic effect on how we go through the event itself.” –Martha Beck, life coach

Any time a group of people are about to embark on a new goal, there are expectations. As much as motivational quotes talk about having no expectations, they still exist in one form or another. A major part of the initial event planning is expectation management. Stakeholders have in mind what they think AF CyberWorx can do and will do. At the same time, the facilitation team has specific things they hope the stakeholders and participants will do. Sometimes the mystery continues on until the event itself. We hope this blog will work towards solving that mystery by setting realistic expectations.

When stakeholders request an event, they know the status quo is not working. They and the users they represent all have ideas of what might “fix” everything. Dr. Dan Padgett, a user experience designer on staff with AF CyberWorx, explains that the facilitation team does not “validate solutions they’ve already thought of.” Instead, they help with a process of thinking differently that gets the team to an answer. The key word being process. A problem solving event does not start with the solution. That being said, some participants and stakeholders come with the expectation that the AF CyberWorx team is going to either provide the answer to their problem or enable the idea they already have.

Vel Preston, Head of Innovation Design, explains it this way: “You scope a problem one way, and a lot of people come to the sprint who haven’t had the same background with new ideas. The group decides, ‘Well, we hadn’t considered that. Maybe we should scope this differently than we thought.’” The process AF CyberWorx guides event participants through leads experts and industry partners to refine the problem and find an impactful solution.

What the team thinks they’re going to fix at first may not be what needs to be focused on. Larry Marine, Lead UX Designer, explains why it’s necessary to lead participants away from their initial expectations as soon as possible: “Folks come in with a strong tendency towards the symptoms without fully understanding the problem.”

That being said, the facilitating team members are also not subject matter experts themselves. As Dr. Padgett says, “sometimes it takes participants a bit to realize that we’re there to facilitate the sprint, and that we’re not SMEs ourselves.” As designers, their specialty is facilitation and guiding teams to consider the user experience in their own designs. They are not experts in every field, nor are they experts on exactly what the stakeholders need to fix their problem. That’s why a group of experts from the field are brought together to solve the problem.

What AF CyberWorx cannot do for stakeholders is solve their problem. They will not immediately verify initial solutions before the process is followed, because they don’t know the current system well enough to say “yea” or “nay.” What we can do is assist a special focus team towards finding a desirable and feasible solution. We can provide facilities with a workspace, tools, and facilitators to make problem solving easier. We also provide networking with industry partners to broaden the knowledge and capabilities of the DoD talent pool. We facilitate improvement and growth, but the subject matter experts in the field are the real heroes that do the work.

Of course, expectations go both ways. Jayleen Guttromson-Johnson, Program Manager, lays out the AF CyberWorx expectations succinctly: “We expect full-time commitment while [participants are] in the design sprint. No email, phone calls, or disappearances. Also, be open to living with the uncomfortable. Our process isn’t like what most people are used to, so just trust the process.”

Stakeholders request AF CyberWorx facilitation and capabilities. We’re here for the team’s success. Let us help turn problems into solutions that go beyond simple expectations.

*The postings on this blog reflect individual team member opinions and do not necessarily reflect official Air Force positions, strategies, or opinions.