Automation Strategy - The Operational Objectives
The extent of the test coverage you desire for your product is one of the major operational objectives determining how much automation it requires. The coverage is generally driven by the type of system you aim to produce. For example, a medical equipment necessitates a much higher test coverage than a low-end consumer electronics product. We can generally agree that the more a product needs a high coverage, the more energy should be expended to automate the test. However, this is not always right. However if we are talking strictly of the extent of automation as a percentage of tests automated vs manual, then it is the low coverage products who will have a higher percentage.
It is easy to accept that an FM radio can be fully tested by a computer and test equipment, without human interaction. However if I delivered to you a nuclear power plant and told you no humans participated in any quality or safety verifications on it, you would clearly doubt its test coverage and its quality. Simply put, very high test coverage products still need human involvement to reach their quality objectives, even if they require a lot of energy to automate.
Unit delivery throughput
A straightforward operational objective is the target unit delivery throughput of the production line. This objective is often stated in units per week and/or per month.
The most important factor affecting the throughput is the test time. A main objective of the test automation should therefore be to allow a production line to reach its throughput target by reducing the total test time to a minimum, by removing time intensive manual operations and efficiently chaining test activities and measurements.
Other factors increasing the throughput are:
A high yield
Yield is primarily affected by the quality of the systems as they are manufactured prior to the tests. But a shoddy automation can drastically lower the yield and create false fails and unnecessary retests, which will cause the total test time to increase and the throughput to decrease.
Automation can help debugging by adding (or tweaking) tests to target debugging steps when a failure is detected. For example, it is possible to add measures taken at intermediary points to automatically investigate a failure of the measure taken at the end point of a sub-system. This targeting, as well as complete and standardized test outputs can increase the efficiency of the debugging phase and allow the turn-around of more units per week.
Full test cost per unit
Depending on the margin targeted for the product, a varying portion of the cost is earmarked to testing. The last main variable in the decision you face is therefore not the maximum per unit test cost itself - as this is a fixed parameter - but the allocation of this cost in the different activities and parts necessary to create the testbench. Let’s first define the full test cost of a system as the sum of the following parts:
The cost of test labor
The cost of testing 1 unit (time * salary) * volume
- including retests in case of less than 100% yield
- including maintenance and support from engineers
- including debugging
The cost of the bill of material
For all test stations on the production line:
- Test PCs
- Test equipment (automated or not)
- Mechanical fixtures
- Golden units of other parts of the system or electronic cards emulating them
The cost of design, development and deployment
Non-Recurring Engineering (NRE) for all testbenches on the production line:
- Design of electronic cards and mechanical fixtures
- Design and development of software tools
- Design and development of the test sequences
- Procedures, training and deployment
Total volume of production
To obtain the cost per unit we must divide the total cost by the target volume, which is our last operational objective affecting our decision. As we will see, the volume is the main driver of the choice we face.
How Much Automation - An Example
The decision on whether to automate the production of a system can be defined by the following sentence
Can you, with your automation plan,
- Reduce the active test time, and thus labor cost,
- While raising the BOM and development cost less than the amount saved on labor costs,
- While allowing you to meet your coverage objective,
- And to reach your unit throughput objective
A good way to go about it is to elaborate different strategies with different levels of automation which allow you to reach your targeted coverage. Then evaluate the costs and target optimizations that save time and money.
Let’s consider the following example. For its purpose, the operational objectives of the transfer to production are set as:
- A target volume of 1000 units
- A target cost per unit of < 40 $
- A target delivery throughput of 100 units per week
- A high target test coverage for a consumer electronics product, allowing the test to be fully automated and not require manual tests.
3 strategies have been evaluated, one not at all automated, one half-automated and one fully automated. For the sake of the example, let’s assume all three strategies meet the coverage objective, as establishing a failing strategy is not practical.
Cost of labor
The following table shows the cost of labor calculation as determined for our example.
- The not-automated strategy uses a fully manual test. It’s tester cost per hour is higher because engineers are required to perform and supervise some of the tests. Note that it fails to meet the throughput target.
- The half-automated strategy is based on a manual first half and automated second half. Test time gains are made, especially the active test time, leading to important labor cost savings.
- The fully automated strategy removes all manual tests. Slight test time and cost gains are made.
|Average test time||18 minutes||20 minutes||25 minutes|
|Active test time||5 minutes||10 minutes||25 minutes|
|Average tester (cost per hour)||35$||35$||45$|
|Cost per unit||2.92$||5.83$||18.75$|
|Total labor cost||2 920 $||5 830 $||18 750 $|
|Weekly throughput / bench (40h week)||133||120||96|
Cost of the testbench's Bill of Materials
The next table lists the bill of material cost for all strategies as determined for our example. Notice that the half and not-automated strategies share the same BOM, the difference being the operator of the equipment for some of the tests (a tester for the not-automated strategy or a computer for the half-automated strategy).
The fully automated strategy involves different equipment and fixtures to allow the complete automation which explains the cost increase.
|Test PCs||1 000 $||1 000 $||1 000 $|
|Test equipment||18 000 $||10 000 $||10 000 $|
|Mechanical fixture||6 000 $||1 000 $||1 000$|
|Electronic/Golden units||0 $||0 $||0 $|
|Total||25 000 $||12 000 $||12 000 $|
Cost of testbench development
The next table lists the cost of development of each strategy as determined for our example. The more complete the automation, the higher the investment in the development of the software tools, the test sequences and the mechanical fixtures to support them. However, procedures and training require more time when manual testing is involved.
|Mechanical fixture||3 000 $||600$||600$|
|Software tools||12 000 $||8 000 $||4 000 $|
|Test sequences||15 000 $||9 000 $||0 $|
|Procedures, training and deployment||1 200 $||2 000 $||3 000 $|
|Total||31 200 $||19 600 $||7 600 $|
Summary of costs
This last table recaps the sub-costs and calculates the total test cost per unit.
|Cost of labor||2 920 $||5 830 $||25 000 $|
|Cost of BOM||25 000 $||12 000 $||12 000 $|
|Cost of development||31 200 $||19 600 $||7 600 $|
|Total cost||59 120 $||37 430 $||38 350 $|
|Total cost per unit||59.12 $||37.43 $||38.35 $|
|Savings||-20 770 $||920 $||(baseline) 0 $|
As we can see from the above table, the half-automated and not-automated both meet the test cost target, while the fully automated does not. However, as calculated above, since the not-automated strategy does not meet the target throughput objective, only the half-automated strategy is a valid one.
Even if the not-automated strategy had reached required throughput, the cost savings made by the half-automated strategy would have led us to choose to automate.
Although the end cost looks similar, it is important to note that the half-automated strategy also leads to higher quality standards due to the standardization of procedure and results brought by the test software.
|Target cost per unit of < 40$||59.12 $||37.43 $||38.35 $|
|Target delivery throughput of 100 units per week||133||120||96|
|High test coverage||Reached||Reached||Reached|
By iterating the target volume parameter, we can calculate that for the fully automated strategy to reach the target cost of 40 $, the volume would have to be > 1515 units. For it to become the cheaper strategy, and therefore the selected one, the volume would need to be > 8110 units.
|Per unit test cost||Fully automated||Half-automated||Not-automated|
|Volume = 1000||59,12 $||38,43 $||38,35 $|
|Volume = 1516||39,99 $||27,34 $||31,68 $|
|Volume = 8110||9,85 $||9,85 $||21,17 $|
In conclusion, the decision of whether to automate your production line should be made with all the operational objective variables involved known and accounted for. It is a decision that cannot simply be made tentatively with preconceived intentions.