Deciding if Automated Test is Right for Your Application

Take the Next Step in Test.

Before advancing with a test project, it is important to understand the return on the investment that will be made. By understanding the concept of automated test, why is it used, when it should be used, it is easy to make the financial justification for it.

Successful product testing is critical to quality, but it also has a cost. Deciding whether the testing should be manual, automatic or a combination of both is based on many factors. Decision makers need to understand the high-level cost drivers as well as modern best practices.

Defining Automated Test

Automated test is the process of enabling a computer to run a series of individual tests on a product in a logical sequence. These tests take measurements using instrumentation hardware, which transmits the measured values to a computer with software (test executive) that can compare the results to pre-defined limits and determine whether the product has passed or failed. Using automation, the test procedure can be run very quickly, reliably and repeatedly, and with limited risk for human error. Automated test is now used in every industry that uses electronics and the number of automated testers is growing significantly.

Everyday Examples of Automated Test

A common example of the move from manual test to automated test can be found at the local mechanic’s. When cars were simple to repair, consumers would take them to a mechanic who would manually check items such as the distributor, spark plugs, and whether or not fuel was being delivered to the carburetor. These checks would be done one at a time until the fault was diagnosed and the repair carried out. These days, the mechanic plugs the automobile into a computer, presses a button, and then analyzes the test results that are generated after the computer has done a more comprehensive set of measurements. For example, during the testing of an Engine Control Unit (ECU), many tests are performed without the mechanic even needing to know what exactly is happening. This process is defined as automated test, where a series of tests is performed on a product in a sequential fashion.

The History of Automated Test

Products vary in their complexity, as does the way in which they should be tested. Often, the more complex the product, the more expensive it will be to manufacture and test. Automated test equipment (ATE) is used throughout the product development lifecycle from initial prototype development, through first batch testing and finally into full production. The same test systems are then quite often used in repair stations or at the site of contract manufacturers (CMs). Regardless of the manufacturing location, test equipment is generally designed and developed near the companies’ R&D center. This is to enable engineers to share initial ideas about how best to test a product and come up with a cost-effective solution for testing it.

It is often the case that the test development engineers install the equipment and train the local operators on how to use the equipment to obtain maximum performance. The decision on what test equipment to use is generally determined by the type of product being manufactured. Historically, ATEs were expensive and difficult to program. It was primarily used on mission-critical products. The early test equipment was affectionately known as “Rack and Stack,” and, as the name suggests, it was a rack-based test system stacked full of individual instruments. These were often connected together with General Purpose Interface Bus (GPIB), invented in the 1960s by Hewlett-Packard. GPIB is an 8-bit parallel bus that had the advantage of enabling the software developer to address each individual piece of instrumentation through a unique and intuitive addressing system. As such, a software developer could decide which instruments to control, in what order, and when to return the test results. GPIB created a revolution in test system design and is still in widespread use today.

Test equipment has evolved in leaps and bounds over the past three decades with faster buses such as PCI, PCI Express, PXI, Ethernet, USB, and many more. These newer standards enable much faster transfer of data between the test equipment instrumentation and the computer. As a result, automated test equipment has reduced in cost and increased in performance significantly. Currently, fully functional ATE can cost as little as a few thousands of dollars compared to hundreds of thousands in the 1980s and earlier. With the falling prices and technological improvements, there has been a proliferation in ATE adoption by companies everywhere. Once the domain of testing only mission-critical products, ATE systems are now commonplace in most high-tech manufacturing companies, producing a range of products from low cost to high value.

Manual Testing vs Automated Testing

The manual test process is normally handled by one or more individuals who perform one measurement task followed by another until the testing is complete. Quite often, companies have employed operators who sit in front of test equipment all day watching the instrumentation and writing down the test results. This process has been used for many years and continues to be used when labor costs are low. Manual tests are also used when the process of testing a product is a fairly simple one where an operator can perform the tests in a short period of time. Automated test, on the other hand, is when a computer-based test system performs the process of testing a product. Historically, automated test equipment has been expensive and complicated, and most automated test systems were built for the defense, automotive, and telecommunications industries. From the 1970s until the early 1990s, the cost of computing was extremely high relative to labor, and during those years the cost of automated test equipment far exceeded the cost of labor. Therefore, manual test was still dominant.

Test Sequences and Test Executive Software

Individual tests (referred to as test modules) perform specific functions and verifications. When these individual tests are put together, they are called a test sequence. An example of a test sequence might be (1) a power-on test, followed by (2) a voltage check test, followed by (3) a current test. With automated test equipment, multiple test sequences can be created to facilitate testing of many products within the same system. Test modules can be created in many programming languages, whereas the test sequence is typically created by a test executive software tool. A test executive tool has many functions, including the creation of individual test sequences, incorporating measurement limits to determine pass/fail conditions, and creating conditions where new test sequences may be called based on the measurements. This type of software also provides a framework for outputting test data, such that results can be uploaded to databases for further analysis.

Semi-Automated Test

When thinking about manual vs. automated test, it is important to not think in black and white terms. Many companies use a third option for their test strategy, called semi-automated test. This option is usually deployed when the product requires some form of configuration during testing. Semi-automated test is where an operator is present for some or all of the test cycle. The product is typically placed in a jig that is connected to a computer. During the test procedure, a graphical user interface (GUI) displayed on the computer screen guides the operator through the parts of the test procedure that require some form of manual adjustment to the product. This may mean adjusting a potentiometer or aligning a certain mechanical part. The computer can be reading a parameter while the operator is making the adjustments. Once the value is within the range required, the computer may prompt the operator by displaying the fact that this has been set correctly. Once these tests have been performed the computer once again takes over and the remaining tests are fully automated in the same way as described earlier.

Types of Automated Test

Automated test systems come in all shapes and sizes, leveraging all areas of expertise. Optical, RF, mechanical, electrical, software and other experts are all required at different points of different test projects.

In-Circuit Test (ICT)

In-circuit test has been around for many years. Its initial use was to test individual circuit boards that went into electronic products. An electrical probe tests a populated printed circuit board (PCB) checking for shorts, open circuits, resistance, capacitance, and other basic elements that show whether the assembly was correctly manufactured. The testing may be performed with a “bed-of-nails” test fixture and specialized test equipment, or with a fixtureless in-circuit test setup. A bed-of-nails tester is a traditional electronic fixture containing numerous pins inserted into holes in a glass-cloth-laminated sheet. The pins are aligned using tooling pins to make contact with test points on a PCB, which in turn are connected to a measuring unit by wires and cables. Each of the small, spring-loaded pins makes contact with one node in the circuitry of the device under test (DUT). By pressing the DUT down against the bed-of-nails, reliable contact can be quickly and simultaneously made with hundreds or even thousands of individual test points within the circuitry of the DUT. The hold-down force can be provided manually or by means of a vacuum, thus pulling the DUT downwards onto the nails. ICT systems often include built-in test data logging, with proprietary output reporting based on the manufacturer’s design. Many ICT systems are still in use today, although they are gradually being superseded by the boundary scan systems described below. ICT has many advantages including speed of test and the ability to test many parts of the PCB simultaneously, as well as being found off-the-shelf. They are normally used for testing populated individual PCBs prior to being assembled into the sub-assembly.

Boundary Scan

Boundary scan is a form of automated testing that has become extremely popular in recent years. Often known as JTAG (Joint Test Action Group), or by its IEEE standard (IEEE 1149.1), boundary scan offers many significant advantages over more traditional forms of testing and as such has become one of the major tools used in automated testing these days. The main reason that boundary scan testing was originally developed was to overcome the problems of lack of access to boards and integrated circuits for testing. As products have become smaller, the real estate available has become harder to find, meaning traditional methods of testing were unsuitable or impossible. Boundary scan overcomes this by having specific boundary-scan registers in large integrated circuits mounted on the board. With the board set to a boundary scan mode, serial data registers in the integrated circuits have data passed to them. The response and therefore the data passing out of this serial data chain enable the tester to detect any failures at relatively high speed. As a result of its ability to test boards and even integrated circuits with very limited physical test access, boundary scan has become widely used and its use continues to grow.

Automated Optical Inspection (AOI)

AOI is common in many manufacturing environments. It is basically a form of visual inspection, but is achieved automatically with cameras, computers, and software. Due to the automated nature of the inspection, it provides a much greater degree of repeatability and speed when compared to traditional manual inspection. AOI is particularly useful when situated at the end of a production line creating soldered PCBs, where it can quickly locate any production problems, including soldering defects, and can determine if the correct components are fitted and their orientation is correct. This type of inspection was traditionally carried out by operators who would look at the same type of boards over a period of many hours. One of the biggest problems with manual visual inspection is operator fatigue. Towards the end of a shift, the accuracy of manual inspection often decreases, causing bad products to pass or good products to fail. AOI successfully addresses these issues. Due to the high cost of AOI systems, they are often used only on high-volume production lines.

Automated X-Ray Inspection (AXI)

AXI has many similarities to AOI. However, with the advent of Ball Grid Array (BGA) packages, it was necessary to create a form of inspection that would enable the viewing of items that were not optically visible. AXI systems can look through Integrated Circuit (IC) packages to accurately evaluate the solder joints in the same way a doctor looks through an X-ray of a bone to look for a possible fracture.

Functional Automated Test Equipment

Functional test is, as its name suggests, for testing the functioning of a device. This is much further along the manufacturing process and is often the final test prior to the product being packed and shipped to the customer. Once the product has reached this stage of testing it is usually the most expensive to repair if something is found to be defective. The sooner a problem or design issue is tested and found, the less expensive it is to resolve. Most functional testers have to be custom-built, due to the fact that most functions of a company’s products are unique to the company. When a new product is designed and manufactured, it often has features that never existed before, meaning that no one has ever built a tester for those features.

Combinational Test

As you can see by the types of test already discussed, no single method of testing is able to provide a complete solution. To help overcome this problem, many ATE systems incorporate a variety of test approaches within one tester. By combining test functions, the test coverage is much higher. A combinational tester is also able to undertake a variety of different types of test without needing to physically move the board from one tester to another. This single suite of tests may include ICT as well as functional and JTAG boundary scan tests. Each type of automated test described has its strengths; therefore, it is necessary to choose the correct type of test approach for the testing that is required. By using a combination of all the different test techniques available, it is possible that any ATE built will be used to its fullest. This will enable tests to be executed swiftly, while still providing a high level of coverage for the successful testing of all product types.

Choosing Between Manual and Automated Test

By analyzing the big picture of the use-case of the product, in addition to the volume of manufacturing, the right path is generally clear. Often, mission-critical products are tested multiple times to ensure correct operation. Examples of such product types are those used for protecting the safety of people or those where defective functioning could be catastrophic, such as in the transportation, aerospace, medical, and defense industries. For example, companies manufacturing aircraft components know that any failure in their products could lead to death and destruction on a large scale, which is why these companies invest heavily in automated test equipment. Another example is a manufacturer of implanted medical devices, whose product quality can mean the difference between life and death for its users. Companies like these were early adopters of automated test technology for both safety reasons and affordability. Automated test never gets tired or has a bad day. When an operator performs the necessary tasks, it has been shown that results can vary from before lunchtime to after. Additionally, subjectivity begins to play a role. What may be suitable to one person may not be to the next. Products that should not pass to market have made it to the public and items that were rejected have been proven to be perfectly fine. These mistakes cost a fair amount in recalls, in repairs and in brand reputation. When working with objective limits, results are always dependable and repeatable.

Is the Application Right for Automation?

Manual test is traditionally used when the testing is a simple process and where labor costs are low. Manual testing is also frequently used in research and development departments where flexibility is an important factor when testing a new product under development. The ability to change a test process and try several different options lends itself to manual testing. Companies sometimes find it hard to move from a manual test process to an automated test process, as the perceived costs of the software development can seem high.

Automated test is now widely used for testing all sorts of electronic products due to the cost of test equipment and software tools reducing in line with Moore’s Law. As test equipment costs come down in price, the deployment of automated test equipment becomes more widespread. Another contributing factor to the spread of automated test is that labor costs are rising worldwide, causing manual testing to be more expensive. In niche industries where specialized skills are required, finding and training the right personal can be equally expensive and challenging. Automated test is used across all product types, whether high-volume/low-cost products or low-volume/high-value products. The factors for deciding which test equipment should be used are dependent on the type of products being manufactured. With high-volume/low-cost products, test times are often critical so saving a few seconds testing a product is a key driving factor when choosing the type of test equipment to use. With more complex and high-value products, the ability to perform many tests in depth is more important; therefore, the test equipment tends to be more sophisticated and of potentially higher cost.

Is Automation Worth the Financial Investment?

One of the biggest decisions business leaders need to make when starting out on the introduction of a new product is whether to invest in automated test. The cost of setting up an automated test system, while having come down in recent years, is still a significant investment. That said, once the initial investment is made there are ways of transforming it into a cost-saving platform. By designing and implementing test stations with a common core, they can be easily reused and recycled throughout manufacturing. The key to a common core or generic test station is grouping all of the expensive instrumentation into a base unit. From there it is simple to design flexible fixtures around them to accommodate different products and stages of the product lifecycle by swapping them in and out as required.

Opposition to introducing automated test equipment may also come from within the organization as some existing personnel may feel that their jobs are threatened by bringing in new technology. This perceived threat is often turned around when they realize that the mundane parts of their jobs will be replaced by more exciting roles where they can add more value to the business. Handled correctly, this situation can be turned into a positive experience for all involved.

The main differences between manual, semi-automated, and automated tests have been addressed, but the question remains of how to decide.

To determine which strategy is most suitable for any given situation, a number of questions need to be answered:

Is the product low cost or high value?
Will it be produced in high volume?
Is it complex or relatively simple to test?
Is it mission-critical for the end customer?
Do 100% of the units need to be tested or will batch testing suffice?

The answers to these questions provide the ability to make better decisions about whether automated test is a strategy that should be introduced. For companies manufacturing low-cost and simple electronic products, automated test is essential to keep the costs of manufacture and test down. In these cases, having an operator test every product by hand is far too expensive. Where the product is complex or of high value, it often makes sense to automate testing due to the high cost of specialized technician labor needed to test the product complexity. High-value products often have customers who expect high levels of testing and documented processes to ensure consistent quality. For mission-critical products, high quality and repeat testing are crucial during all stages of production. Testing should be performed at every level of assembly, from the testing of individual PCBs, through sub-system test, and finally to functional acceptance test. Failure in the field for these types of products (i.e., the cost of bad quality) is far more expensive than comprehensive testing.

To better understand, and thus justify, the differences in the test processes already discussed, it is helpful to apply an example of a typical test scenario. In the example scenario described above, a product that needs to be tested is broken down by potential cost per test for each of the three test processes. The critical calculations related to deciding on the best strategy are test time, operator costs, equipment costs, and engineering development (i.e., software design) needed. The example assumes that a product line builds 10,000 units per year according to the estimated fixed costs, development costs, and test times. This information is used to calculate the test costs per unit. The critical expense in this example is the test time. As it decreases, the cost per test decreases dramatically.

With manual test, operator costs are higher as the level of competence needs to be higher. A trained test technician is required to perform the test. This is not the case with the semi-automated or fully automated test scenarios where a less-skilled operator can be used. (In these cases, the operator could potentially work with multiple testers at the same time, but this multiplexing assumption is not used in the calculations.)The example presented in the above table provides a guide for calculating different test-type scenarios. In this case, due to unit volume (10,000 per year), it justifies moving from manual test to semi-automated test but not moving to fully automated test. That’s because the volume is high enough to justify some form of automation in order to significantly reduce labor costs, but the volume isn’t high enough to justify the more expensive costs of full automation.

Successful product testing is critical to quality, but there is a cost. Deciding whether product testing should be manual or automated must be weighed out carefully, using all the above considerations. Good planning will lead to effective results and accelerate manufacturing while improving overall results.