1.1 Project Planning

Before any software development (especially code writing) can begin, it is necessary to get a software project plan in place. This typically involves a project manager, who will scope the project correctly before getting started. This person can either be the software manager or a person who works closely with the software manager, thereby understanding task loads and capabilities.

Software projects are often labeled as “going to be delivered late” or “going to run over budget” before they have even started. Software projects are not the same as typical engineering projects because they are so diverse and have many acceptable approaches. Software is more of a hidden type of deliverable—it is not obvious to anyone outside software coding what is being done or to what level of quality the work has been completed. Sometimes software projects are unique, where the development team or even company hasn’t ever implemented anything similar before. They may represent big projects for a customer where there is no obvious possibility of using the software code elsewhere. And as mentioned, the software process itself may vary from project to project, based on software requirements. All these factors must be understood up front before the creation of a project schedule.

At the outset, when the project schedule is developed, there are a variety of inputs that need to be understood. First is understanding and translating the customer’s expectations. Is the completion date a date for a preliminary field integration test or do they expect a fully tested and validated product on that date? Understanding the software resources available and their capabilities is also important. Having software engineers with experience in similar projects or even projects for the same customer is better than having a staff that is entirely inexperienced. Having inexperienced engineers working on a project is not a bad thing as long as time is allocated for experienced staff to assist and bring them up to speed in readiness for the next project. Finally, the external factors that may impact a project need to be understood. Such factors may take the form of support from other functions within the organization, such as quality or risk management, or from outside regulatory or independent assessments that are required before the software can be delivered.

After these inputs are understood, the initial project plan can be put together. It is considered “initial” because a good software project plan will undergo numerous negotiated changes throughout its development life cycle. Too often arguments can occur because the project manager holds on to the original plan instead of updating it based on the realities of the project. Various factors that can affect an initial project plan include not initially evaluating the risks correctly, having resource issues occur that are not immediately controllable, or taking on an advanced feature that has not been done before, involving a great deal of “learning.”

Project plans should always have milestones in the schedule. These are points where a specific activity can be marked as complete. Avoid having milestones that are not specific to a deliverable, like a milestone stating that the design portion of the project is half done. There should be a measurable shift in activity or delivery when each milestone listed is completed. Tracking progress against the project plan and schedule can be done using a variety of tools, but typical granularity for project updates should be no less than 1 week. More often than not, the project manager will have a schedule that is constantly fluid or requiring a lot of time to status each of the pieces. Smaller software-specific plans can be done in conjunction with the larger project plan, as is done, for instance, in sprint planning for Agile.

Having a project plan that is understood and agreed upon up front, is regularly updated in terms of its status and progress, and has an easy way to assess its progress throughout the project is critical.

1.2 Risk Management

Risks will be present in any software project. Understanding those risks up front and determining mitigation plans helps in the development and then execution of the project plan. Not only are there project risks, like availability of resources, but there are business risks as well, such as having software that hits the market window. Other risks include changing the software requirements later in the software development process, hardware platform delays if developing an embedded product that the software needs to run on, or even outside influences, such as regulatory changes software needs to comply with. Risk classification, risk analysis, and risk mitigation plans need to be in place at the start of project planning and should be updated throughout development cycles.

Risk classification includes capturing all the possible risks that could influence the correct and complete delivery of the software project. They are typically categorized in groups that make sense to the project, such as resources, external groups, and even estimation risks. Each risk is preferably identified up front but risks can be added during the process of the project as well. Once they are documented and classified, a risk analysis can be performed.

A risk analysis involves taking a deeper look at each risk. This involves understanding how likely a risk is to occur, and then how much of a negative impact that risk will have if it does occur. The probability that the risk will occur is typically grouped as a percentage, with ratings for very high (maybe > 90%) all the way to very low (maybe < 10%). Sometimes factors are applied to each of these, with higher numbers associated with higher probabilities. Once this is complete, then the effect of that risk needs to be evaluated. This could be something severe in terms of impacting the project, to negligible. Again, a numerically weighted value can be associated with each of these assessment values. A simple way to order the risks is to multiply the two numbers for that risk together to get a score. However, a risk that has severe consequences may need special attention even if it not in the top 20% of risks—thus the suggestion of the “weighted” impact value.

Risk mitigation then involves developing a plan outlining what actions need to be taken if the risk identified occurs. This could involve early development of that feature so there is more time to address it as opposed to waiting until later in the schedule, or even developing a parallel path so that if one path for development fails the other may work. In any case, once the mitigation strategies for each of the risks is understood, then those strategies need to be executed and monitored throughout project development.

1.3 Kicking Off the Project

Ideally when the project plan including resource identification and schedule are complete, and a solid risk assessment is complete, then the project is ready to get started.

A project kickoff meeting is an effective way to get the project going in the right direction. Management can be brought in so that they understand the risks and resourcing. Developers should also participate to understand the project schedule and review the risks that were identified using the mitigation plans. The milestones (and how to evaluate whether they are complete) can also be reviewed.

It is important to get the stakeholders and the resources on the project to agree where they are, how the project will be measured, and the path that the project will take. If that is not done up front, there is little chance it will get any better after the project starts. A regular reporting rhythm for development can be agreed and getting people to buy into this idea is crucial. Once this kickoff occurs, the software development process can get started—with the software requirements.

2.1 User Requirements

User requirements are typically written when discussing the use cases for a project. The requirements definition is done with the customer or product managers that know how the embedded system will be used by the user.

Many user requirements deal with how a user will interact with a system and what that user expects. If there is a screen or human machine interface aspect to the system, a user requirement may be based on what happens when the user selects an action on the screen. Maybe with a button press not only does a process start, but it also switches to another screen and provides an audible notification. When user requirements such as these are written down, they can often break into multiple system requirements later due to switching of screens, the maximum delays in starting the process, and finally what the next screen should look like. One pitfall is starting to try to write the system requirements during a user requirement meeting. This often detracts from gaining insight into the requirements of the user, and key functionality pieces could be missed.

In fact, as alluded to earlier, it is often better to keep user requirements and system requirements separate in their tracking and reporting. The user requirements are often more readable, understandable, and provide a better sense of how the system will operate. Even though user requirements may lack specifics on what really needs to occur in the system, they are still valuable in that they can provide the overarching system functionality expectations.

Finally, when writing user requirements, it is a good idea to have traceability in terms of where the user requirement originated. Whether it is from a single customer or product manager, understanding where it came from is important. When capturing user requirements, there are times when separate user sessions may develop conflicting requirements. Being able to go back to the originator and understand the use case better can help facilitate the deciphering of conflicting user requirements. It could come from differing standpoints, like an operator vs. a maintenance person, so being able to go back and resolve those differences becomes important.

2.2 System Requirements

System requirements are requirements that may not necessarily be visible from a user standpoint. Many embedded systems have a much higher ratio of system requirements to user requirements, simply because embedded systems are often not seen. A braking system in an automobile, for instance, has few user inputs (the brake pedal) but many system requirements on how the brake system should function.

System requirements are often much more detailed than user requirements as well. Whether these originate from system or software engineers, they are often derived from understanding the interfaces that the embedded system must work with. So, there may be requirements to specific standards or timing requirements because of a deeper understanding of how the system must react.

One caution that should be exercised when writing system requirements is that these requirements should focus on how the embedded system works with and reacts to the external connections it has. If the embedded system must initiate or react to external signals, then this should be a primary focus for system requirement development. Too often, system requirements are written that specify the internals of the system. In some cases, this should be done, but in others it often restricts the software design unnecessarily. If it is important to include something from a product or system sense, then include it.

Once requirements are gathered up front, there are a variety of ways to manage those requirements. They should be revisited, will often change through a development process, and their management then validation is important. Because there are multiple ways to do this, one such method is described in a later section in this chapter covering Agile development.

3.1 Safety-Critical Elements

Safety-critical elements are requirements that the system must meet either from regulatory, customer, or internal safety design practices. Safety-critical software, because of the amount of testing and rigor of the software development process itself, should try to stay within the fewest number of subsystems as possible. When this is done, that subsystem can follow more rigor and be designed in such a way that the other subsystems cannot impact its execution.

Often, this type of safety-critical architecture drives the embedded software to live on its own hardware, a safety coprocessor of sorts that further helps guarantee its isolation. Its interface to the other subsystems is then managed through serial or discrete I/O, making it much easier to apply process rigor to one part of the embedded system and have isolation to the rest of the system.

3.2 Operating Performance

When certain performance (i.e., response time) targets for an embedded system need to be met, it is good to contain these requirements in a single subsystem if possible. If simple logic exists between getting an input signal and then driving an output signal in response, this could be architected in a single subsystem.

If the architecture of the embedded system, specifically because of its complexity, doesn’t allow for performance-oriented architecture items to be in an individual subsystem, then a different approach is required. There are two general types of messaging between subsystem interfaces, dynamic and periodic messaging. The most common approach is to minimize the number of interactions between subsystems when passing performance data between them, a.k.a. dynamic messaging. With dynamic messaging the response signal begins to be triggered as soon as the input stimuli is received and there is minimal delay between subsystem message transfer. This could be set up as a separate interface between subsystems.

With periodic messaging, a delay exits between subsystems as messages are transferred through. Periodic messaging could be a set of messages that is sent between subsystems every 100 ms therefore increasing the total amount of delay to receive a response output. Now imagine there are several subsystems where periodic messaging is used. Every time a message is transferred to a different subsystem, delay propagation continues to build. It is clear how this could affect performance-related requirements.

3.3 Security

Like safety-oriented information, it is important to try and isolate the security elements of the architecture. The goal would be to have multiple layers in the system before the secure data elements can be reached. In the security environment, embedded system layering could act as a “defense in depth” type of strategy.

In a simple example, an embedded system has layer A that interfaces externally, a layer B subsystem that it talks to, and then a layer C subsystem that contains the secure data elements. In the design, layer A receives a message, then creates a new message to layer B to request the required information. Layer B would process the request, but also check various performance and security requirements around the data being requested. This check system could also include network or formatting rules to make sure that numerous incorrect requests have been received recently. If this all checks out, then a separate message could be sent to layer C to get the data requested. Layer C could even include a check of some type of signature or encrypted part of the message that layers A and B do not know how to interpret, but the source of the request and layer C do.

Further checks, like creating an inability for layer A to talk to layer C, can provide security to the system. This could be done by using entirely different messaging checks for message formats so there exists no ability for the software to even interpret that type of message. Only layer B can interpret, check, and then reformat the message for the appropriate subsystem.

3.4 Reliability and Availability

The last of the large aspects that should be considered as part of the architectural design are the requirements for availability and reliability. These two terms are often mistaken as the same thing, but it is important to consider both for the architecture of the system.

Availability is the measurement of the uptime of the device performing a task. If a machine must spend 1 h for maintenance after every 9 h of operation, it would have an availability of 90%. Downtime of any kind, whether planned or unplanned, negatively impacts the availability of the system.

Reliability is the measurement that a system will perform its intended task when it is supposed to. Unexpected failures negatively impact a system’s reliability. Measurements, like mean time between failures, are often used for a reliability assessment.

Designing an architecture to meet availability requirements often includes elements of redundancy or efficiency in the way data is handled. If there is a strict availability requirement, duplication of either hardware or software in the design may be desired. If one data path does not work, the redundant path takes over and completes the particular “high-availability” aspect of the embedded system. This could be done at a process level within a hardware processing unit or separating hardware with checks between them to make sure that one has performed the task required.

The other aspects that come into subsystem and architecture design include updating the software and configurations. Does the system need to be taken all the way down to update a single subsystem or configuration in a subsystem? This impacts the availability of the system. Instead, can the architecture be designed in such a way that subsystems or hardware elements are hot-swappable, meaning a new item can be put in place dynamically with the system continuing to operate.

After considering safety-critical, operating performance, security, and reliability/availability aspects of the architecture, then the design will start falling into place. Using interface dataflow or specifications to understand the interactions between subsystems is important. It is possible that once the list of interactions is understood, then functions or data elements may move from one subsystem to another. All of this should be done before the embedded system design is started.

4.1 Object-Oriented Programming

Object-oriented programming starts with an object. An object maintains its local state and includes its own set of attributes that are specific to that object. Access to making changes in that object is handled by a set of the objects access functions, so its data can be tightly controlled. Object-oriented programming also deals with the term “class” that creates a new type, and objects are instances of a class. This would be similar in C programming, where the class could be unsigned long, and an object is each variable that is declared as an unsigned long.

Objects can store data in fields if they belong to that object. Objects can also have functionality that is specific to that object, if they belong to a class, called methods. All the fields and methods are referred to as attributes of that class.

There are two types of fields in objects. The first is a class variable which is shared data among all of the objects of that class. A change in a class variable in one object will also change in other objects of that same type. An object variable is specific to that object, so changes in an object variable do not change values in other instances of objects of that type. Consider a type called “student,” where we have an embedded system for a local school. A class variable could be the address of the school. Each student would have the same address, and in the event the address changes, it is desired that a change in address for one student would alter all the others. An object variable for the type “student” could be the student’s name. This would be unique to each object of the student type.

One of the major benefits to using object-oriented design is being able to reuse code in the system. This is typically gained through a term called inheritance. Consider inheritance as being able to add a subtype to any type that is specified in the system. As objects contain more and more data, they can become more difficult to manage. If object types have things in common, it is better to split this common data into a new class, and then have the appropriate classes inherit this new class as one of their attributes.

Consider a student and a teacher. There are attributes that are common to both, such as school address and possibly some type of ID number. Students have grades and teachers do not, and teachers have salaries where students do not. A “person” class could be created, where everything common to both students and teachers can be placed. Then, when the teacher class is created, it calls out to inherit all the attributes of the person class as well. The same occurs when the student class is created. The “student” class is known as the base class, and the student/teacher classes are known as subclasses.

With this basic understanding of fields, methods, classes, and objects, the object-oriented design process can proceed. Three of the key steps in object-oriented design include system context, class identification, and design modeling.

4.1.1 System Context

This first step in object-oriented design is helpful to understand the context and interactions of the system. Use case models are used to understand the interactions with the system and help identify objects and operations. The following is an example diagram to understand which functions an operator would perform with a machine (Fig. 1).

After putting together a diagram, specific use cases descriptions can then be written. The use case descriptions describe the data, the stimulus that enables the use case, and the response of the action. These help before taking on the next step in object-oriented design.

4.1.3 Design Modeling

During this step the relationships of the object classes are documented most typically using a documentation method called Unified Modeling Language™, or UML. This step bridges the requirements to the implementation of the system. UML diagrams need to show the relationships between the classes, and at the same time provide enough information for software programmers to implement. Consider the UML diagram in Fig. 2.

The object classes are listed along the top with a vertical line under each. For the sequence, time starts at the top of the diagram and moves down vertically. Arrows then show the interactions between the classes, with rectangles showing when an object of that class is in control of the system. If the object must wait for a response from another before continuing, then that rectangle is stretched down the line until the response is received.

Each major interaction of the system, especially those that were derived from the original use cases, should be documented. Other more detailed interactions that are not necessarily part of the use cases can be done later by the programmers once they understand more of the design. Once this last step is performed then object-oriented implementation can begin.

4.2 Real-Time Design

Real-time design is when there are specific system constraints that must be met or if a response must be generated within a very tight window from when stimuli are received. Object-oriented programming could certainly be used for some “soft” real-time systems, where a response has a window of time as opposed to being very tight or “hard.” A user touch screen is an example of a soft real- time system. A response is required because the screen cannot be perceived as being too slow, but it falls in the many tens of milliseconds from stimuli to screen change. An automobile motor with electronic injector control, however, must actuate for a specific period to inject fuel based on engine speed, load, and other dynamic values. The window for injector actuation is very tight, measured in microseconds, so it is considered a hard real-time system. Failure to hit the precise injection on/off points can result in mission failure, such as the automobile stalling or not meeting emission requirements. In these cases, using a real-time design approach is applicable.

Real-time design involves modeling system stimuli, the system itself, as well as the corresponding expected result for the system. By doing this, each data element, and where it is sampled or provided from in the system, is understood. A typical design phase for a real-time system involves a traditional dataflow diagram, where a process starts at the top and operations are performed based on conditions in the system. For example, if a value is greater than X, go down this path in the system. Otherwise, if the value is less than or equal to X, go down this other path. These dataflow diagrams can also be state machine diagrams, where the state of the system is documented, with the various elements that can change it to a different state. An example of a state machine diagram is given in Fig. 3.

Once dataflow and state machine diagrams are understood, then the design process shifts to looking specifically at the resources of the system. At this point, a real-time operating system can be selected, as this will have a bearing on the overall design. Real-time operating systems contain elements called signals, which can be sent from one process to another to initiate an action. That allows control from one process to another higher priority process to accomplish a task. For a smaller resourced processing system, a “roll your own” operating system can be created where an RTOS is not used. Instead, a rather simple scheduler for tasks can be used, with interrupts that wait for external stimuli before kicking off a specific sequence of tasks and then quickly returning to normal operation. Microcontrollers have built-in mechanisms that assist in both commercially available or open source RTOS products, and in systems where a simple execution method is created and used.

Real-time design focuses on speed of execution and minimizing delays. C ++ can be used for real-time programming, but certain aspects of C ++ should not be used, such as dynamic garbage collection, operator overloading, and dynamic creation/deletion of objects during operation. Aspects like these in the C ++ programming language can make execution timing inconsistent, which could lead to not meeting the real-time requirements. Typical programming languages for hard real-time systems include utilization of FPGAs for fast handling of data inputs, assembly language for a very specific sequence of timed events, and the C programming language which doesn’t have much of an overhead and has easily interpreted output when looking at its generated assembly language output when compiled.

Once either object-oriented, real-time, or possibly some other design method is complete, then the team can move on to implementation.

6.1 Validation and Verification

During and after the implementation phase of the project, verification and validation are performed on the embedded system. Verification and validation include testing the system behavior as specified by the original requirements of the system using elements of the use cases and any performance aspects that were added. Validation answers the question “are we building the correct system?” Verification answers the question “are we building the system correctly?”

Verification is more straightforward to perform. The embedded system requirements can be translated into test cases, and then a verification plan and procedure can be put in place. For each requirement, the system is tested to make sure it is met. Pass/fail or similar criteria can be used to assess the system.

Validation is a little more complicated. It attempts to truly answer whether the embedded system can be sold into the marketplace and might be required by customers. It takes use cases and customer feedback into consideration, as opposed to hard requirements, and includes user expectations, which may be a bit more subjective than hard requirements.

Software inspection and validation testing are performed to check each of these. Such processes may involve peer reviews of software in the deepest sense or higher level activities like performance testing and user reactions to using product prototypes. Verification and validation are the processes used to identify areas in the system that need improvement, with defect tracking and reporting used to understand stages in the process. Once a defect is identified, it can go to the appropriate stage in the process (even back to architecture or requirements) so that the change can be applied through the entire software development process. Once the issue is fixed, it can be revalidated or reverified before it is closed.

Automatic static analysis can also be a part of verification. Software code is fed into an automated tool that analyzes the software and points out potential defects or latent defects. This can include branches of code that cannot normally be executed, such as a branch of software that is only executed when X < 0, except that X is an unsigned integer. Static analysis can also point out if a software function is too complex, by applying a McCabe™ complexity factor which is dependent on the number of paths and exit points within a function. Finally, even if the software code compiles without error, the static analysis tool can find and flag pointers that are used before assignment or typecasting being performed that may not make sense for the situation.

6.2 Integration Testing

Integration testing involves testing the system after the software components are put together. It is the first phase of testing the complete system. It involves identifying which components are necessary for a given system function and putting those together to test them to make sure functional requirements are met. An issue that arises with integration testing is attempting to understand which software component is causing the error—often there may be complex interactions between components so that it is not clear which components are problematic.

Deciding the order of components to be integrated is an important part of this process. It is possible there are mature components in the system that were developed and used for a previous project. In this case, it may be a good idea to integrate these components first since there may be more confidence with this set of software. Another method, especially if the software components are all new, is to integrate the most commonly used components in the system first. As they are the most used, it will allow for more and more testing as other components are added.

When new components are added it is important to also retest use cases that affect the components already integrated. A simple addition may end up affecting the operation of a component underneath, due to interactions or timing execution issues. Once the components have been integrated and tested the next phase of testing can occur.

6.3 Release Testing

Release testing can begin once all the components have been integrated and it is a complete set of software. During release testing, the system requirements that have been translated into testing requirements are executed. Release testing is normally called “black box” testing, as the internals of the software components are not readily understood, rather the system is tested using varying inputs while looking at the outputs to understand how the system responds.

For a given set of inputs, a given set of outputs is expected. However, combinations of inputs that may not have been specifically stated should also be tested, to make sure the system does not respond in an adverse way. As a simple example, consider that a requirement specifies if A and B are true then output X should be on. Release testing should then create test cases where A and B are both off, in addition to only A on and only B on. During this phase different combinations should be performed to see if the system can be “broken.”

Testing with invalid inputs is just as important as testing with valid inputs. If a system login exists, then different combinations of usernames and passwords should be used. Even if the input is expecting alphanumeric values for a username, other types of characters should be used to make sure the system can still process those without malfunctioning.

Release testing is important to really understand how a system operates under proper, as well as adverse, inputs and conditions. This testing should be done before performance or stress testing.

6.4 Performance Testing

Performance testing involves putting the embedded system under stress to see if it can process input sets. A system may be designed to accept a serial message of 100 bytes, once per second. Performance testing takes this expectation but alters it to send 10,000 bytes per second, or maybe increases the frequency from once per second to once per every 10 ms. Even though the software design took the original typical state into consideration for the design, increasing the rate or the amount of data should not “crash” the system or cause undesirable behavior.

The first instance for performance testing is to evaluate the fault behavior of the system. Unexpected sequences of data, or sending corrupt data, should not cause additional failures to occur. Even though normal operation is a set of data that is properly formatted and sent at a precise periodic rate, the system must be able to handle nontypical conditions. This can include making sure the system throws away incorrect data, or is able to rate limit the data coming in so that it does not adversely affect the system.

The second case for doing performance testing is to make sure another defect is not caused somewhere else in the system. Even though the system may have a separate operation that is seemingly totally unrelated, understanding if there are defects because of timing becomes critical. Maybe time is being spent processing more data, so that a critical signal from one process to another cannot meet specific timing requirements and so goes to a fault state.

All these types of stress testing are particularly important for embedded systems where timing and resource constraints are more typical vs. an enterprise-oriented software architecture.

7.1 Scaling for Complexity/Organization

I work in a big organization therefore dataflows and infrastructure must be defined when thinking about “process.” Agile can still be applied when working in a company with 500 people … and no this doesn’t mean your Scrum Teams have hundreds of people on them with one Scrum Master. It means those 500 people organize into Scrum Teams with separate Scrum Masters (think multi-“core” processing here). Different Scrum Teams have different levels of expertise. Arbitrary to the principle of “all work can flow to any team,” the bottom line is: management is required to strategize about which work flows to which teams based on each team member’s strengths and weaknesses (as mentioned in the “Implementation” section earlier in the chapter). The Agile recommendation is team members should self-organize into teams. In larger organizations, silos of personnel with varied experience levels and various functional expertise may make self-organization an unrealistic goal. There must be a spread of expertise and skill sets across each team.

Another challenge that teams encounter when attempting to be purely Agile is that by becoming Agile, they assume they must become anti-Waterfall. Don’t be afraid to be the Agile team deemed as “too Waterfall-ish.” A team I once worked with was so adhered to the Agile principles that they refused to conceptualize and design an initial architecture because doing so would be “too Waterfall-ish.” Months and months passed, and the product was built vertical slice by vertical slice; minimal viable product (MVP) by MVP. It was not recognized until an undesired feedback cycle produced an enormous wave of defects (that were not initially caught because an integration test team would be “too Waterfall-ish”). This led me to the fact that it does not matter if you are Agile or not, complex systems must absolutely have forethought of architecture and conceptual design like the standard Waterfall practice of establishing requirements up front. We must not ignore all the system life cycle design concepts we have learned and benefited from simply to claim we operate in a particular way. We must use common sense (Fig. 4) !!!

That’s not to say some of the architecture and architectural documentation requirements may emerge as the team members design, develop, and test vertical slices of value in time-boxed increments. In Agile programs, parts of the architecture may absolutely be defined incrementally along the way, however, the team must carefully assess areas where an emerging architecture makes sense and other areas where it does not. As the picture shows below, prescriptive Waterfall design can be done in parallel with emergent Agile design and development. Teams working in each direction on each side of the boundary, must work together to ensure the emergent design jives with the prescriptive design. Agile teams are completing US abiding by constraints defined in the prescriptive design. Before beginning a program, ask yourself: What part of the product architecture and high-level system design should be defined up front?

Makes sense for architecture to emerge incrementally (emergent design):

• • Software application architecture
• • Internal interfaces and timing
• • Human machine interface screen design
• • Low-level configuration parameters

Does not make sense for architecture to emerge incrementally (prescriptive design defined up front):

• • Platform and framework architecture
• • External interfaces and timing (capture outer time boundaries)
• • Software application configuration schema
• • Key system dynamics
• • Software application system modes
• • Major software application state machines
• • Nonfunctional requirements:
  • • Performance characteristics
  • • Availability
  • • Maintainability

The meetings we have become accustomed to executing as part of the Waterfall process may not go away as we transition to a more Agile like execution. If the practices you have in place continue to serve you, do not drop them all to be Agile. Continue to hold a kickoff meeting to begin a new project. Continue to hold change control board meetings if you need them. Continue test readiness reviews. These meetings can be facilitated in addition to the Agile standups, sprint plannings, and retros (these are Agile meetings described later in this text). When applying Agile for complex integrated (and possibly multicore) embedded programs, you should merge Waterfall, Agile, or homegrown processes to come up with something that works for your team’s individual needs. Software release artifacts that capture configuration changes may now log user stories and defects instead of software change requests (SCRs). Requirements, test cases, code reviews, and test reviews can now all be linked to the user stories to provide traceability. Agile processes and practices are simply tools, the focus should always be on the product, manifesting the vision, and the individuals; not the process.

7.2 Roles

An organization’s roles will change when transitioning to the Agile development process. Teams adapt to Agile by using the prescribed roles, dataflows, and infrastructure prescribed by Scrum, Large Scale Scrum, eXtreme programming, and the Scaled Agile Framework model. Regardless of the framework your team chooses; it all comes down to the type of thinkers that exist in the organization. Considering which type of thinkers your team is made up of helps with mapping traditional roles to newer Agile framework roles. As you read this section of the chapter, proposal, conceptual, and detailed thinkers are flavors of these roles (not intended to replace traditional or Agile roles). This flexibility allows us to abstract the role from the individual to utilize individuals on our teams regardless of their traditional role or Agile role. For example, you could have a detailed thinker that is a Product Owner or a proposal phase thinker as a Scrum Master.

Proposal thinkers are often members of the commercial team, project or product managers, business analysts, architects, and/or principle engineers. These are often the people who know of the first news of a new program or system concept. Systems engineers, product owners, safety engineers, and quality assurance engineers are often well versed in this space, but it is not typically their primary focus. Proposal phase thinkers may be external facing or internal facing, meaning they collaborate with people either inside or outside of the organization. Externally facing product managers have the brunt of the work prior to program kickoff. Internally facing product managers have the brunt of the work after program kickoff. Proposal thinkers may support either side of a program kickoff. Proposal thinkers have a more intimate knowledge of a system’s use cases and user scenarios, as well as organizational details like team and supplier interfaces.

Regardless of the medium, proposal thinkers typically:

• • Communicate a transparent pipeline at the opportunity level
• • Participate in backlog grooming
• • Help to prioritize the backlog
• • Communicate a regularly cadenced vision
• • Negotiate what to work on and when to work on it
• • Decompose and refine at the portfolio level: During a weekly sync up with leadership higher level work items are groomed and sized. At this point modifications may be made to the overall strategy going forward.

Conceptual thinkers are the people who do not see value in memorizing detailed information that can be looked up online later. The thought of trudging through details without having ultimate clarity on the higher level conceptual structure drives them crazy. When solving a problem, conceptual thinkers are “systems” thinkers. They try to define and solve the web of complexity that connects the pieces of information; not the information itself. Conceptual thinkers are different from proposal thinkers in that they know the product, the system components, their interfaces, the functions, and the functional allocation to the components. The proposal thinkers may know some of this, but it is not their primary focus. Conceptual thinkers are involved in the headspace of the “what” (unlike detailed thinkers who only think about the how). The “what” may include understanding operational context, user activity flows, dependencies/dataflows with other systems, etc. Both proposal and conceptual thinkers actively drive the vision of the future product forward.

Conceptual thinkers may also be the team members who set objective learning goals for the group. Conceptual thinkers are typically focused on the high-level work items in the backlog (think capabilities or features). The conceptual thinker’s meetings are often focused around what work will go into the quarter (unlike the detailed thinkers who are digging in, focused on the user stories that will be assigned to sprints, which are 2-week cycles of work). Conceptual thinkers are your best bet to invite to feature writing and grooming sessions, as well as release planning sessions, described herein.

Detailed thinkers are focused on the details of implementation. Detailed thinkers crave order in a process. They crave determinism in operations. With Agile, some detailed thinkers may get turned off by all the flexibility and free-thinking that occurs. If you have a team of individuals that absolutely cannot stand flexibility, have them define how they want to do things and lock it in. For some teams just having this structure can make them operate better. Wrapping structure around Agile practices can help detailed thinkers feel more stability. Leaders of detailed thinkers should ensure the work to do is visible, the state of the work is visible, the metrics are made visible, and any impediments are cleared.

Traditional roles are mapped to Agile roles along with corresponding activities below.

• Traditional role: Architect
• Type of thinker: Proposal thinker
• Agile role: Product Owner or Business Analyst
• Typical duties (Agile + Waterfall):
• • Defines multigeneration product and technology plans (then the next three generations of the product look like this given the technology at hand)
• • Defines product vision and roadmap
• • Responsible for high-level system architecture
• • Designs validation architecture and strategy
• • Defines functional concept(s)
• • Performs risk management
• • Defines operational use cases
• • Defines use case models
• • Defines sprint entry/exit criteria
• • Prioritizes high-level work items in a backlog
• • Determines feature dependencies
• • Accepts user stories
• • Assists team by providing direction
• • Attends demos and provides feedback
• • Updates architectural documents
• Traditional role: Project leader or project manager
• Type of thinker: Proposal and conceptual thinker
• Agile role: RTE/Scrum Master
• Typical duties (Agile + Waterfall):
• • Facilitates creation of handoff package
• • Facilitates team coordination
• • Facilitates Agile meetings
• • Performs risk assessment/monitors risk log
• • Updates risk register, budget, plan, etc.
• • Attends demos
• • Attends technical readiness review
• • Ensures line of sight
• • Updates program management plan (iteratively)
• • May draft and update software quality assurance plan
• • May perform quality checks on user stories to ensure correct format with correct elements
• Traditional role: Systems engineer
• Type of thinker: Conceptual thinker
• Agile role: Product Owner or Business Analyst
• Typical duties (Agile + Waterfall):
• • Defines product level model: this model shows where the product fits in the product line it is part of
• • Drafts system engineering plan
• • Assists with definition of cyber security considerations
• • Assists with creation of architectural roadmap
• • Assists with development of the system architecture definition/high-level system architecture
• • Assists with development of the functional concept to enable generation of system requirements/acceptance criteria
• • Defines system functional specification
• • Defines system requirements/acceptance criteria for features and user stories with help from the team
• • Is responsible for ensuring the team understands the user’s intent
• • Performs system functional allocation (maps system functions to system components)
• • Assists with the identification of risks and assumptions
• • Assists with designing validation architecture and strategy with the system architect
• • Defines release and sprint DOR (definition of ready)
• • Drafts features (high-level requirements)
• • Determines feature dependencies with the system architect
• • Ensures backlog priority is correct
• • Assists with feature decomposition
• • Determines user story dependencies
• • Drafts a first release of user stories that demonstrate compliance to Safety objectives
• • Produces and gives demos with Scrum or Kanban Team.
• Traditional role: Software developer
• Type of thinker: Detailed thinker
• Agile role: Scrum Team member
• Typical duties (Agile + Waterfall):
• • Drafts software development plan
• • Develops software design
• • Drafts software design document
• • Assists with determining priorities by describing dependencies between user stories and level of effort
• • Sizes features/user stories with software tester(s)
• • Ensures the software build passes
• • Performs code reviews
• • Develops software
• • If tests fail, fixes software
• • Assists with writing acceptance criteria with the help of the team
• • Helps to prepare the sprint/release demo
• Traditional role: Software tester
• Type of thinker: Detailed thinker
• Agile role: Scrum Team member
• Typical duties (Agile + Waterfall):
• • Ensures software functionality is correct
• • Drafts test strategy and test plan
• • Establishes test equipment requirements
• • Identifies if requirement/acceptance criteria is testable
• • Sizes features/user stories with the software developer(s)
• • Assists with definition and decomposition of user stories
• • Commits to user stories in sprint with team
• • May assist with development of automated test scripts
• • May run automated test suite
• • Adds to regression test suite (manual or automated)
• • Drafts software test procedures
• • Writes software unit tests
• • Executes software tests
• • Reports software test results
• • Helps to prepare the sprint/release demo
• Traditional role: Safety engineer
• Type of thinker: Conceptual and detailed thinker
• Agile role: Supporting team member
• Typical duties (Agile + Waterfall):
• • Drafts system safety plan
• • Partakes in team planning (speaking on behalf of safety type work scope)
• • Partakes in sprint execution* (speaking on behalf of safety type work scope)
• • Identifies safety objectives
• • Drafts Preliminary Hazard Assessment (PHA), Functional Hazard Assessment (FHA), and System Hazard Analysis (SHA)
• • Drafts multiple Subsystem Hazard Assessments (SSHA)
• • Assesses if user stories have safety implications and if so tags them as such
• • Safety acceptance criteria
• • Attends demos

*Note: the safety team may work in a Kanban format a sprint behind the Scrum Team or may be embedded into the Scrum Team.

• Traditional role: Validation engineer
• Type of thinker: Conceptual and detailed thinker
• Agile role: Supporting team member
• Typical duties (Agile + Waterfall):
• • Validates software: ensures software meets user’s intent
• • Ensures traceability
• • Attends and participates in demos
• • May develop automated test scripts
• • May perform performance testing (verifies nonfunctional requirements/the “ilities”)
• • May perform sanity tests
• • Performs test reviews

Regardless of the Agile framework and practices implemented, knowing which type of thinkers exist helps in terms of efficient organization. When transitioning to Agile, do not blindly map these roles. Talk to team members. Some systems engineers may not want the duties of Product Owner. Also, the mapping of type of thinker to roles here is meant to cover cases 90% of the time. You may have some team members that fit into many or all of these categories. Each type of thinker may work in a team with members like them, sprinting through 2-week cycles making deliveries to downstream consumers. For example, think of the delivery boundaries as a “stage gate” [3] or serial structure. Proposal cycle sprinters are preparing a backlog of high-level work items to the conceptual sprinters. They do not know much about the work yet so maybe the backlog just consists of Market Functional Requirements (MFRs). The conceptual sprinters are working to flush out the operational context, that is, system level requirements. Conceptual sprinters preload the backlog for the detailed sprinters team. The work item type that conceptual thinkers typically work on are typically features or epics; detailed thinkers work on user stories (as referred to in most Agile toolsets). The developers work in a team concentrating on vertically slicing code within the user scenarios, system architectural constraints, and system interfaces which are predefined by the conceptual sprinters. The proposal thinkers are sprinting and delivering to the conceptual thinkers who are delivering to the detailed thinkers (Fig. 5).

However, this does not always have to be the case. Each type of thinker may work on a team that is “nested” (read: non-stage-gate) and has a mix of the various thinker types, sprinkling in their insight to various levels and phases of work. Each team has proposal, conceptual, and detailed thinkers either working in Scrum or Kanban configuration. For cases where the product is super mature or where bandwidth is slim the proposal, conceptual, and detailed thinkers may be the same people. Information exchange among the groups is highly interconnected and must be iterative. Quality is a by-product of how well the groups exchange information as they burn down their backlogs. The number of people on the team may influence whether your Agile teams will work in a stage gate or nested structure.

This “thinker” paradigm may also occur with development and/or testing. The test team can also work within a nested or stage gate structure. The detailed sprinters have detailed “testers” which may or may not be the developers (core Agile concepts recommend blurring developers and testers into one). Each level can have their set of tests to execute, with the proposal cycle testers focusing on user acceptance, the conceptual testers focusing on completeness and correctness of user scenarios, and the detailed testers ensuring that functionality is verified, designed, and developed correctly. Since the three thinker levels are sprinting concurrently there is a lot of knowledge sharing going on. Proposal testers are recommending conceptual functionality or maybe conceptual testers are recommending detailed functionality.

How deep the interfaces between these teams are depends on how new the technology and system is as well. For the implementation of an existing system the handoff from the proposal sprinters to the conceptual may be slim to none, information may go straight from the proposal team to the detailed team. For a brand-new system, the proposal and conceptual sprint cycles may take longer to get something ready for the detailed sprinters. Internal-facing proposal level thinkers help plan, flow information external to the organization into execution teams inside the organization, assist with updating/reworking plans, and stay in constant alignment.

7.3 Keep Your Plans!

A project plan may still be utilized along with Agile methodologies applied to pockets of execution in between planning and delivery. The entire life span of the project can be mapped out in the plan. Microsoft Project® can be used to plan the project including efforts for software and hardware. Various phases of this plan require different treatment. This allows for things like lead time on material procurement and shipments to be included in the overall vision for the program as well as aid in the understanding of the propagation of change. A preliminary project plan could be populated by the program manager who works with other members of the leadership team to define a set of preliminary target dates for each line item in the plan.

Once a draft of the overall project plan is developed, the three levels of thinkers get to work. The program manager communicates the preliminary plan to the other Agile leaders who then flow the plan to the team. The team plans out the “pockets of execution” in finer granularity. A “pocket” would correlate to a set of work items in the backlog and be represented as a line item in the project plan. JIRA® or Rally® are tools that can be used to track pockets of execution. These pockets may manifest themselves in the form of features that begin to be flushed out for each piece of functionality promised to the customer.

Planning is iterative and should change frequently as priorities are nonstatic. Involved parties should always work at reviewing and refining the plans. Once the team has filled in their part, this is a good point for the program manager to review what the team has claimed as part of the milestone deliverable and identify gaps, eliminate work that is not part of MVP, perform some prioritization, validate any assumptions, understand dependencies, etc. Changes are always flowed up to the plan to ensure everyone is in alignment. During sprint planning and refinement meetings, modifications may be made to the plan for a more granular strategy going forward. The commitment for a specified date may change as the team evaluates their story load versus velocity. This is an integral part of Agile, the team communicates to leadership what can be done but also more importantly communicates what cannot be done. It is then up to management to make the priority calls about what will be worked and what will slip. These updates can be made in the overall project plan. Once this is complete, program mangers really start to get a realistic picture of the path going forward. These steps are iterative in nature and occur with frequent cadence. This helps to keep the plan as accurate as possible. The more transparency between management and the team equates to a smoother execution and aligned expectations between the two parties. Continuous communication about what we are planning to work on and what our capabilities are makes everyone successful.

Program management and the team continue to stay in constant alignment by ensuring everyone has visibility in terms of plan progression. By this point dashboards can be set up for each project, so you’ll be able to tell if the initiatives are on track. Key views the Rally® tool supports include: initiative burnup, team commitment of initiative burnup, cumulative flow diagrams of scheduled states, team story point throughput, and scope creep. These help to forecast dates of completion and run simulations (trade out feature X for feature Y to meet completion date Z). A major part of the Agile process is that we want to make sure everyone’s voice is heard. It’s hard for proposal thinkers to attend every Scrum of Scrums and backlog refinement meeting so the best thing for them to do in that case is to identify another proposal thinker that can act as a proxy. During the Agile ceremonies where work is being ranked and claimed this proxy can speak up and ensure the features and/or user stories for the customer deliverables are getting boiled up to the top (higher priority) of the backlog and therefore getting executed on.

7.4 Meetings for Planning

Agile principles state that the teams (a.k.a. the knowledge workers) are great contributors in coming up with the plan! Teams know exactly what they need to do and can give a reasonable estimate of how long it will take. The beginning of planning for an Agile project typically entails drafting and prioritizing a backlog of work items to be accomplished by the team in a designated time frame. The work items drafted and prioritized by the proposal thinkers generally specify higher level system functionality that adds value to a user scenario of the system. The work items drafted and prioritized by the conceptual thinkers may be decomposed from the work items that the proposal thinkers have drafted. They typically specify mid-level functionality that adds value to a user activity of the system. Then the user scenarios and activities are decomposed further into user stories by the detailed thinkers. If the team is working in more of a nested structure, all types of thinkers may contribute to the user story’s acceptance criteria and/or requirement collectively.

7.5 Plan for Your Unplanned

Some of the details of higher level work items will most likely be unknown at the time of planning. Teams may add placeholders in their backlogs to account for the unknown. Some teams plan for a lower “loaded” capacity instead, intending unallocated capacity can be used for the unplanned. Either way, having a way for planning the unplanned scope helps the teams to be more predictable in executing their pocket of execution in the overall program plan mentioned earlier. As engineering discovery unfolds the problem space, so backlogs are rewritten and reprioritized to account for each new turn of events.

Flavors of “unplanned” work include:

• • Systems engineering work
• • Release preparation work
• • Research spikes
• • Additional validation
• • Defects
• • Field issues

Even though the teams plan out what they will work and plan for the unplanned, it is important that they always embrace change!

7.6 Documentation

The Agile Manifesto [2] states:

• Working software over comprehensive documentation

In organizations producing large integrated solutions, documentation still exists and, in some cases, comprehensive documentation must still exist. Many industry’s certification per safety standards still require a large set of documented artifacts to be produced. The difference when working in an incremental methodology is the documentation can be built incrementally as the program persists instead of being provided up front. Whereas with the Waterfall process documentation and meetings are tightly coupled, with the Agile process, this is not the case. In cases where it makes sense (emerging requirements) documentation becomes a by-product of the team’s working cycles rather than a driver. New tooling allows for documentation to be autogenerated directly from the work items that the team creates. Many companies are coming up with solutions for integrating life cycle tooling across the board, just Google integration of life cycle tooling and review what comes up. Having integrated requirements, safety artifacts, design and test artifacts, code, test procedures, and automation scripts is a foundational concept in both modern DevOps as well as MBSE techniques. It is ideal to produce documentation in this way if you can.

To produce the documentation incrementally, the team should be aware of documents that they will be responsible for updating along the way as well as any material they will need for updating such documentation. Material to consider includes any information for team to learn and access document generation tooling, document templates, and configuration management program to baseline revisions. The documents that can be updated incrementally are listed below. This is not an all-inclusive list—the standards your team work under may drive modifications to this list.

• • System/software interface specifications. Update as interfaces are created/modified while working user stories and features
• • System/software requirements. Update as requirements are created/modified while working user stories and features
• • Software design. Update as software design artifacts are created/modified while working user stories and features
• • System/software validation/verification plans. Update with additional test cases spawning from user stories
• • System/software integration test specifications. Update as integration test procedures are created/modified while working user stories and features
• • System/software architecture validation/verification report. Update as architecture is created/modified while working user stories and features
• • Functional hazard analysis. Update as user stories and features tagged with safety implications are worked

Usually conceptual and detailed thinkers are involved in the care and maintenance of these documents but in the Agile methodology the team may own this task collectively. There is no reason a software tester cannot update a requirements document. If this concept makes your team queasy, have a rule that the system engineer must review modifications made by anyone to the document. Many teams have leveraged online Wiki-type tools to capture documentation as well. Tools like Confluence® have revision tracking, data accessibility per profile permissions, searchability, and graphics editors. For most younger generations, online generation of documentation is all they know.

7.7 Go With the Flow

The “flow” or consistent throughput of value delivery is one of the greatest benefits of using the Agile process. Control points, phase gates, formal role definition, and stringent documentation do not block the Agile team. Transparency, adaptation, and improvement through regular retrospectives and small focused teams speed up flow. When thinking about flow in this way, there are a few key points that should be considered:

1. 1. Agile ceremonies promote and increase flow
2. 2. In Agile, flow exists within and between sprints
3. 3. Product vs. project teams have different flow
4. 4. Flow of supporting teams
5. 5. In a scaled Agile framework, flow exists in an Agile Release Train (ART)
   The types of thinkers discussed previously contribute to the flow by participating in the Agile ceremonies. Proposal thinkers contribute on the front end of the value flow, conceptual thinkers in the middle, and detailed thinkers at the end. As mentioned before, different types of thinkers may contribute in two important ways: (1) thinkers work in their own concentric team delivering to downstream cycles (stage gate), and (2) thinkers may participate in all levels contributing their own type of knowledge to the final product in various ways (nested).

7.7.2 Agile Flow Exists Within and Between Sprints

7.7.2.1 Within Sprints

7.7.2.1.1 Work Item Fields and States

We will now consider the user story life cycle as it is processed through the flow. The granularity of a user story plays a factor in promoting or decreasing flow. No matter the tooling your Agile team chooses, the chances are good that the tool is going to have some representation of a work item whether it be a feature, user story, defect, etc. Chances are also good that these work items will have a ton of attribute fields.

One of the challenges a new team may face is deciding which fields are necessary and which fields to leave blank. Good practice says we give the work item a name, description, identify the state it is in, identify a size estimate, and so on. There are other fields like parent work item, release, and planned time frame that come into play when considering traceability and metrics. There should be no prescribed data field requirements for a work item unless it is decided that the information entered benefits the team in some way. For example, features should only be sized by the team if the managers assess estimated effort accordingly and assign the appropriate workload based on team estimates. If the teams continually size features and the managers continually overload … the sizing does not add value.

User stories have transition states, owners, assigned sprints, and so on. It is up to the team to decide what each user story state transition means as well as what the accepted criteria and definition of done should be. A definition of done is a standard checklist the team commits to complete for each story.

For larger integrated systems requiring certification artifacts the definition of done may look like this:

• • Documentation updates
• • Acceptance criteria is met (story is accepted)
• • Review(s) have been performed (and documented, and location of review artifacts are accessible from user story)
• • Any remaining action items have been closed out
• • Safety engineering aspects are covered (including documentation or analysis)*
• • Traceability has been achieved wherever necessary
• • Code has been checked in (build is passing on master branch, automated test scripts checked in)

*If your safety team is running a Kanban approach this line item can be as simple as making sure the correct story exists and is closed in the safety engineering team’s backlog.

7.7.2.1.1.1 User Story Cycle

User stories have somewhat a life of their own. The life cycle for a story can look very similar to how SCRs are processed, except a user story’s life cycle is short, owned collectively by the team, and may spawn more user stories. It involves discovery, design, development, and testing, without phase gated control point boundaries. On the first day of the sprint, testing may be done that drives the development, generates the requirements in real time, and enhances everyone’s understanding of the problem space. There is no “right way” to work a story. When we had a “right way” defined in Waterfall processes and practices, everyone was waiting to learn or begin work, bugs were found right before releasing the software (instead of upstream), and designs were often overwritten and/or abandoned by the time coding began. Human behavior, as well as the work humans do, is diverse, so when considering our processes, the higher the level of abstraction the better. Design, code, and test reviews can still take place as the user stories are worked on by the team but their sequence can be flipped around if necessary (Fig. 6).

One pitfall I have seen some team’s fall into is to try to execute on every story consistently across the board. For example, say a team insists that every story must have a code review. At the end of the sprint no stories can be accepted because even though the software has been developed and tested, the code reviews for each of the stories were not completed. In the spirit of Agile, be flexible! It’s okay to have a code review story in the next sprint that will cover this task! Heck while reviewing eight code modules at one time the task may be completed quicker and similar inferences may emerge across the code modules (hence enhancing the review!) What I am getting at is that the old process tells us to do everything consistently across the board. In the true spirit of the Agile the only thing to keep consistent is to use common sense in all scenarios!

7.7.2.1.1.2 How Do We Write User Stories and How Granular Should They Be?

While it is ideal that stories are written in a way that allows developer(s) to take them and immediately go off and begin work, that is not always the case. Most of the time the Product Owner and team are scrambling to wrap up stories, get the demo prepared, and meet their sprint goals. This means, depending on the team, sometimes stories are written on the fly. Stories are initially drafted as reminders or placeholders, to capture fleeting thoughts from escaping and locking them into the backlog. Then if anyone has the time these stories may be given slightly more definition, say while a PO is facing a list of 20 of them or while the group is meeting during backlog grooming. In the embedded space, I have yet to see the case where an Agile team works off a perfectly defined backlog, and that’s ok! Why? Because stories are meant to be a “placeholder for a conversation” as many Agile references say. Be careful when being too nit-picky of how to accept a user story into the sprint based on a mile-long DOR list. This can slow flow down and become a bottleneck. There is a lot of evidence supporting how having a DOR can also help flow by reducing rework.

The other tricky part about stories is that they must be completed in a 2-week time frame. Again, be careful as the definition of done described above is an Agile tool that if not used correctly can really slow down your flow. In a system-of-systems team dynamic, a 2-week delivery is a lot harder than it sounds. Teams automatically assume they can do more than they can actually do, and they do not account for all the Murphy’s that come up. So, as the Agile coaches tell us, when writing your stories start small, very small. As an understanding of the problem evolves, and the work increases, add more user stories to the backlog.

7.7.2.1.2 Vertical Slices

The important thing to realize when working with Agile for complex systems such as these is that we do not get too wrapped around the axle in terms of what a vertical slice means to us. Wikipedia describes a vertical slice, sometimes abbreviated to VS, as “a type of milestone, benchmark, or deadline, with emphasis on demonstrating progress across all components of a project.” [6] For an embedded team, the scope boundary is different than the folks working with a software application. You can’t respin a board every sprint or quarter. However, hardware Kanban Teams can work concurrently with Scrum Teams to complete user stories for BOM creation, schematic creation, or procurement of materials.

At the start of development and design, a web application developer may understand the user scenario. The code can then be developed, integrated, and tested all the way to the database layer for a slice of application functionality in a 2-week cycle. A win is achieved when the web application developer(s) complete many of these vertical slices and the system is built. For an embedded developer, maybe the vertical slice only goes as deep as generating a binary (Fig. 7).

7.7.2.2 Between Sprints

After your team begins to work a few sprints in a row, it will become apparent to you that it makes sense to have what is called a HIP sprint or IP sprint (SAFe). Having additional time to perform some hardening, innovation, and (additional) planning is refreshing and welcomed after the team has been sprinting for the past 6 weeks. SAFe prescribes having one HIP sprint a quarter (after five normal sprints) but other texts recommend having one whenever the team needs it. It is up to your team to decide what makes sense for them. Some teams I have seen have thresholds for the amount of technical debt they will allow themselves to incur before making it mandatory to perform some clean up in a HIP sprint (Fig. 8).

Goals Agile teams should have for their HIP sprints include:

• • Documenting updates
• • Reducing technical debt
• • Reducing number of defects
• • Developing regression test automation
• • Changing impact analysis (as needed)
• • The last HIP sprint in a release could also be used for release planning for the next release

7.7.4 Supporting the Team’s Flow

In the embedded world you have software teams, where the Scrum mentality fits ok. Then you have many other development life cycle teams that also need to jive in the Agile flow. These are supporting teams like safety engineering, hardware engineering, configuration management, test automation, system integration test, proposal management, and laboratory support. These types benefit from working in the Kanban format as it is often difficult for these teams to plan what they will do next because their work runs parallel or in support of the software Scrum Teams. Supporting teams should hold their own Kanban Team meetings. At points where exchange between teams occurs, the Kanban principle classes of service [9] must be considered. These help teams to identify prioritization of supporting work items to the Scrum Teams. Kanban work in progress (WIP) limits can help a team to drive these items to completion (Fig. 9).

Common toolsets that support the Agile process provide teams with the ability to collect many metrics as work items are defined and executed on by the teams. Common Agile metrics include things like feature completion, US point completion, sprint velocity, or length of time in each Kanban state. Metrics help us to identify bottlenecks, blockers, steer the team, and get status. Setting up a framework and data model to gather metrics for Agile projects should be done up front.

Metrics can also be helpful for determining which work should be steered to the team and when. If a project or milestone is showing a projected end date that is past its due date, we can easily increase the slope of our trendline by taking more work for this project in the next couple of sprints. Comparing accepted vs. committed lets a manager see how good the team is at estimating the work, hence giving insight into the accuracy of the estimates for all current projects.

Commonly untracked metrics which provide an enormous amount of insight to a team’s flow are: amount of spanning work items, amount of unplanned work added after planning, and amount of work that is blocked.

• Spanning work items: Sometimes we forget to consider items spanning our formally defined time frames. Instead, we focus on work items that can be wholly completed in a designated time frame. For example, items that started in this quarter and will finish in the next quarter. Spanning items can have a significant impact on forecast completion and should not be disregarded while coming up with a metrics schema. Ideally, work items will be split to fit into the corresponding time frames, but this cannot always be done. Think about this when setting up your tooling and designating work to time frames.
• Amount of unplanned work: The amount and type of scope creep a team has from quarter to quarter can give insight into how predictable the team is. Ideally, if the team plans for the unplanned (see “Plan the unplanned” section) they will have a higher predictability than a team that does not.
• Amount of work that is blocked: Whether due to an external dependency or a resource constraint all teams have things that are out of their control, that block their flow. Metrics that provide visibility into how much of their defined work is out of their control is leverage when attempting to replan or negotiate a new end date with the customer.

8.2 Tooling

Tooling in this context refers to the tool your Agile teams are using to track their work, you know the one that has the user stories in it, where you move the stories through a sequence of states, assign them to iterations and owners, and so on. The data model represented by the tool should be apparent such that no one needs an explanation to understand it. The data model of your work items is the process these days. We do not require training when we get a new cell phone. Based on our general understanding of how to click through menus of various contexts we figure it out. Tooling for work item tracking and how it is configured for each team should be as intuitive as that.

Product architecture can be a driver for how the tool to track the work is set up. Tool configuration created for one team may not be easy to implement for another team. For example, if a team is working on a backlog for customer X, and customer Y has a separate software baseline, a duplicate backlog of similar work items will need to be created if customer Y decides on the same new feature set as customer X. This may not need to be thought through in excruciating detail if you have a couple of teams working concurrently to develop a product. However, when you have close to 70 teams syncing to develop a product for a large integrated system, the tooling schema must absolutely be cleanly defined and have team boundaries that make sense.

8.3 Agile and Alignments

Different teams and team members with different levels and areas of expertise require different Agile configurations. Staunch traditionalists might say that the foundations of Agile are static and should be used consistently across the board. I disagree. Different levels of system complexity and definition of architecture, software, or hardware could require different energy flows and a different focus in different areas. Hence, causing the team to use Agile practices differently. Tuning the team to create a better balance by selecting which Agile practices should be used, and when, may be necessary. Applying the Agile methods is not a one-size-fits-all process, all factors must be considered and balanced. You want to align Agile practices to work for the types of thinkers in your team (the organization) as well as to consider the phase of development (the life cycle) and lastly align Agile to the system you are building (whether a product or a project instantiation of the product).

What if we had foresight before the program started and selectively thought out the areas where we were strong and weak and then applied the Agile methodology accordingly. In Feng Shui you strive for balance in how you focus energy to enhance, produce, reduce, and control certain objectives. You let your current state speak to you and apply remedies to achieve the desired result. Consider the table below. In each cell, the maturity/knowledge level of the system construct on the x-axis is assessed for the type of thinker in the y-axis. Strong means the concept is understood and the requirements are known whether documented formally or not. Stable means the team has an awareness and perhaps begins to generate ideas but nothing is formalized yet. Weak means the concept is not understood and the requirements are not known or documented formally.

This snapshot shows a typical starting pattern for a product. The architecture concept may be strong to start off. At first the proposal and conceptual thinkers may be the only ones who know about the architectural concept while having a somewhat stable understanding of the software and hardware design. The detailed thinkers are left in the dark (with the purpose of keeping them focused on something already underway).

Starting off …

As time progresses, the contract is officially won. Now the conceptual thinkers really dig in and seek to understand what the architectural, software, and hardware design concepts will be. The detailed thinkers have probably been given a heads-up but still have limited or no knowledge of the new opportunity.

By the time the program has flowed into sprints, all the various thinkers understand the concepts. Each cell in the matrix eventually goes from weak to strong as the product is built out and delivered.

Agile best-practice recommendations for this case are:

• - Use Agile ceremonies to consistently promote flow
• - Vertical slices of functionality are developed incrementally to verify the functionality operates correctly on the desired platform
• - The customer is involved throughout the development to ensure intent is met

There are many different scenarios for how programs may play out. For example, another case may be as the team innovates, refactors a legacy product, or completes each vertical slices of the MVP, the manifestation of the architecture emerges along the way. Let’s dig into how we can balance the team’s Agile execution forward in this case. They have a weak definition of what the architecture should be when the program is proposed. This happens and is surprisingly common as detailed thinkers may make a solid business case for a particular functionality to marketing and sales who sells the idea before architecture is formally defined.

Starting off …

Recommendations for this case:

• - Commissioning a detailed Scrum Team on the hardware side would balance this out. Identifying the hardware includes considering operational environment, supplier constraints, and cost. As the detailed thinkers enlighten the conceptual thinkers, everyone begins to get a stronger understanding of the architectural, hardware, and software concepts. Once nailed down we know what platform we want to use, we know what our interfaces are, we know our incoming data, we know what our expected output data is, and therefore high-level requirements can begin to be generated.
• - Agile practices that promote knowledge sharing are team swarming and pair programming, these may be helpful as everyone is learning together.
• - Marketing and sales representatives could regularly attend system demos to ensure what is being developed is what was sold.

Let’s look at the opposite case. What if the product has been in the field for 10 years, the same team has been developing it for 10 years, very little change happens—maybe a few defects here and there when it is rolled out to a new customer.

Starting off …

What Agile practices are relevant in this case? All teams can improve! This is the perfect setup for migrating legacy development and test practices to Agile’s best friend, DevOps. In today’s age continuous integration and deployment, automated testing, and remote updates are not just nice to have … these methods are necessary for survival in the market. There is a ton of wonderful knowledge on this topic. If you are in a team that has this type of setup, your primary focus should be figuring out how to automate your infrastructure.