Sharpen The Blade

Procurement of A New Pistol, Part Two

Introduction:

In the last month’s article on the Procurement of a New Pistol I discussed the Performance Description which forms the basis of the requirements. This month I will continue the following:

· Getting manufacturers’ feed back on draft PDs

· ISO 9000 and other quality requirements

· How to conduct a weapon Test and Evaluation

· The ins & outs of Reliability and Durability Testing (as hard on the testers as on the pistols)

· Fending off protests from ‘evil’ gun manufacturers

· Tips on scoring (racking and stacking)

1. Getting Feedback

Now that the Team has determined what it wants through lengthy discussions and the use of a formal/informal Market Surveys, it’s time to find out what the firearms manufacturers have to say about all your hard work.

This is done by placing a ‘Draft” Performance Description in the solicitation web site for all to see and comment. The Performance Requirements, Validation and a Scoring Summary (to be discussed below) are offered for comments, usually in 10 to 15 business days. The comments received from the manufacturers range from helpful to hostile and arrogant. The smart, customer-focused manufacturers will make either few comments or phase their objections to certain items with pleasant, professional words, such as:

“We are able to supply you with all the features you are requesting, but if you made the pistol XX inches longer (or XX ounces heavier) we would be able to provide a wider section which we believe would be of great interest”.

I believe the above statement is something we could all relate to and it would make us take a second look at the requirements and what the manufacturer was suggestion. Under no circumstances are we required to change the requirements. Unfortunately, the gun world is filled with those who believe (incorrectly most of the time) that their firearms are the best and only a fool deviate from their standard offerings. Their comment letters are usually written by attorneys and may threaten legal action. Feel free to ignore them.

Both types of responses need to be thoroughly reviewed to be sure the requirements are not out-of-line with current technology or manufacturing capability and changes made accordingly.

2. Quality Requirements

The need for very specific quality requirements cannot be overstated. I have worked in industry for close to 20 years and am well aware of both commercial and Department of Defense quality levels and requirements, such as the old MIL-Q-9858/MIL-I-45208 as well as the latest ISO 9001 and 9002. I was, however, surprised by the low level of appreciation and understanding the firearms community has for these important standards.

The firearms user-community has for too long given the major manufacturers a pass on quality and more importantly, constancy. How many times has each one of us purchased a firearm from a well-established manufacturer and only to be disappointed when it failed to perform as advertised? We get more frustrated when the same firearm purchased by a friend works fine. The industry, as a whole, fails to deliver good products, day in, day-out. Since the buying community is so fragmented, the manufacturers continue to deliver junk without penalty. Well, only a big buyer has the clout to put an end to the junk deliveries.

The purpose of all the fancy ISO 9001/9002 quality stuff is insure constancy and to document changes. The ISO stuff will not make a bad manufacturing process better, but it’s job is to raise the ‘red flag’ so as to say, “You’re making stuff that doesn’t meet the drawing”. These ‘red flags’ can then be audited by a customer who can say “You’re making junk, I don’t want it.” The second job of the quality requirements is very important - if deliveries are to be made over a long period of time, say two to five years – that is to document changes. During a lengthy delivery, many things can change, such as the steel source, the design of critical item s such as the barrel, slide or frame (changes could result from ‘producibility’ – meaning the manufacturer has modified the part to make it easier and cheaper to make or the manufacturing process itself). The ISO system requires all this to be documented and big important customers can review this documentation.

Documenting the changes is important to the buyer. During the course of the Test and Evaluation, all the pistols have been subjected to a wide variety of expensive and extensive tests in order to select the best one. Now that all that has been done, to allow the manufacturer to make undocumented changes to the design or manufacturing process will cast a doubt on those test results. If changes have been made, how do we know the last pistol received is the same as the first? In Part One, the use of the First Article Test (FAT) helped to answer that question. However, the FAT and production sampling are catching the problem after-the-fact. The demand for the constancy must start deep within the manufacturer.

As a minimum, I would recommend, the supplier of any firearms have and maintain an ISO 9002 certification for the facility involved in the manufacture the firearm.

3. How to conduct a weapons Test & Evaluation

Here are some pointers to help conduct the T & E:

A. Be prepared. The T & E must exactly reflect the requirements and must be conducted in accordance with the Validation Section in the Performance Description. I would recommend the Test Evaluators prepare the following:

· A detailed plan for each requirement based on the words in the Validation Section. The plan should be so detailed as to allow anyone who reads it to perform the test.

· Each detailed plan should include physical space on the paper to record the test results, date and signatures of the evaluators as well as the pistol serial model and number.

· Rely on physical paper copies, not on electronic versions.

· Make a master notebook of all the test sheets. Could easily run 50 to 100 papers.

B. Have all the test equipment calibrated by a certified calibration company. This would include all calipers, weight scales, tape measures, force gages, etc. Keep a log book listing all calibrated items and the calibration date.

C. Conduct ‘dry-runs’ of all tests using all test personnel. For many tests, there is only one chance to get it right. Dry runs will quickly pickup any errors, confusion or lack of equipment.

D. Have a witness for all tests. The job of the ‘witness’ is to simply watch the evaluator and make sure he follows the test plan. The ‘witness’ should also countersign the test sheet.

E. Be organized and focus on only one test at a time.

F. Write down any and all results. Document!

G. Take it slow. Rushing will cause confusion and errors.

H. Label all magazines. Magazines can be a major source of test incidents.

4. The Ins & Outs of Reliability and Durability Testing

Fun part of the T & E is the part with all the shooting, or at least that what everyone expect until several days into the shoot.

In this test, the manufacturers submitted nine (9) pistols for testing. All nine were required to fire 3,000 rounds each and three (3) of the nine (9) fired 7,500 more, for a total of 10,500 rounds. Past testing experience showed the most a pistol should be shot before cooling and cleaning is 250 rounds. Thus, all shooting was broken into cycles of 250 rounds which means each pistol would fire 12 cycles to reach 3,000 rounds and some another 30 cycles. A cycle included: obtaining the pistols, ammo and mags; loading the mags; shooting and scoring, and cleaning the pistols. All this could be accomplished in 70 minutes.

To perform a Reliability and Durability Test of this magnitude will require the following:

· Each cycle will require a shooter as well as a ‘scorer’ (record keeper)

· One Range Officer

· Two Armorers

· Several Cameras to record incidences

· Secure boxes to store any broken items

· Sufficient ammunition which has been thoroughly tested

· A indoor or outdoor range large enough to safety accommodate all personnel

· Good hearing protection. Recommend double hearing protection of ear plugs and muffs

· Good ventilation if using an indoor range and range cleaning equipment

· Brass picker-upper. South Brass makes a nice model which works well on smooth concrete, but would not work on grass or gravel.

· Brass scrap barrel

· First Aid kit or if the number of personnel warrant, a EMT on site

· Plenty of targets

· A health baseline on all shooter which should include: Blood lead level, hearing, and lung capacity. Lung capacity is need if the shooters will be wearing respirators while cleaning an indoor range. Respirators are highly recommended.

· Pistol Cleaning stations, preferably away from the firing line stocked with all the patches, brushes, cleaning solvent and lubricates.

· A method of documenting all cycle results. The best way is to have pre-printed score sheets issued to each shooter/scorer team each cycle

· Written description of all potential malfunctions

· A couple of ‘dry runs’ helps get everyone on the same page

5. Protests

Regardless of how well you think you’re doing, someone isn’t going to like it and will file a ‘protest’. Usually, the protests will come in two categories: an objection to an item performance requirements or an objection to the test method/test results. Both are easy to rebut if you have done your homework.

The protest is usually written by the company’s console and will whine on and on about how the requirements/test placed his company at a disadvantage and how the government agency has conspired against them. The best advice is never to ‘bad-mouth’ any manufacturer during any part of the specification development, solicitation or test. This can be used to ‘prove’ the company’s case. Always be professional.

Other Suggestions in dealing with Protests include:

· As discussed in Part One, always have a good reason for each requirement. Never go looking for a requirement just to eliminate a particular manufacturer as this will nearly always backfire.

· Do what you say you’re going to do. Follow the validation test plan as closely as humanly possible. The manufacturers have all been given a chance to comment on the test plan and make recommendations. If they fail to provide feedback at the appropriate time, shame on them; if you modify the test plan without proper notification to the manufacturers, shame on you and you will loose.

· Document that you did what you said you would do. Document the tests using video tape, computer output and/or the official witness.

· Be prepared to write a coherent, factual, emotionless defense of the requirements and test plan/results.

6. Scoring

Scoring the test results is a difficult task. The following thoughts need to be kept in mind:

· Scoring must be designed to separate the ‘men from the boys’.

· At the end of the day, a pistol needs to be selected. Can’t go though all this for nothing.

· Too many pass/fail items might disqualify the best pistol for something relatively simple

· Pass/fails invite protests since it eliminates a pistol from the solicitation

· Try to adjust the scoring so pistols which miss the ‘little things’ can make it up on the big things, like the durability test

· Scoring methods need to be set prior to the start of the test to prevent a manipulation of the methods based on the test results

· A good total score of 1,000 points or less works well. Too much above 1,000 gets confusing

· Only release a summary of the scoring in the solicitation, not the scoring range or amount for each item

7. Summary:

I trust this has been helpful. I appreciate that all this is a major undertaking and requires a lot of time and money lacking in most small government agencies. However, even a small agency can perform simple tests with a handful of people if everything is well planned.

	Procurement of A New Pistol, Part Two Introduction: In the last month’s article on the Procurement of a New Pistol I discussed the Performance Description which forms the basis of the requirements. This month I will continue the following: · Getting manufacturers’ feed back on draft PDs · ISO 9000 and other quality requirements · How to conduct a weapon Test and Evaluation · The ins & outs of Reliability and Durability Testing (as hard on the testers as on the pistols) · Fending off protests from ‘evil’ gun manufacturers · Tips on scoring (racking and stacking) 1. Getting Feedback Now that the Team has determined what it wants through lengthy discussions and the use of a formal/informal Market Surveys, it’s time to find out what the firearms manufacturers have to say about all your hard work. This is done by placing a ‘Draft” Performance Description in the solicitation web site for all to see and comment. The Performance Requirements, Validation and a Scoring Summary (to be discussed below) are offered for comments, usually in 10 to 15 business days. The comments received from the manufacturers range from helpful to hostile and arrogant. The smart, customer-focused manufacturers will make either few comments or phase their objections to certain items with pleasant, professional words, such as: “We are able to supply you with all the features you are requesting, but if you made the pistol XX inches longer (or XX ounces heavier) we would be able to provide a wider section which we believe would be of great interest”. I believe the above statement is something we could all relate to and it would make us take a second look at the requirements and what the manufacturer was suggestion. Under no circumstances are we required to change the requirements. Unfortunately, the gun world is filled with those who believe (incorrectly most of the time) that their firearms are the best and only a fool deviate from their standard offerings. Their comment letters are usually written by attorneys and may threaten legal action. Feel free to ignore them. Both types of responses need to be thoroughly reviewed to be sure the requirements are not out-of-line with current technology or manufacturing capability and changes made accordingly. 2. Quality Requirements The need for very specific quality requirements cannot be overstated. I have worked in industry for close to 20 years and am well aware of both commercial and Department of Defense quality levels and requirements, such as the old MIL-Q-9858/MIL-I-45208 as well as the latest ISO 9001 and 9002. I was, however, surprised by the low level of appreciation and understanding the firearms community has for these important standards. The firearms user-community has for too long given the major manufacturers a pass on quality and more importantly, constancy. How many times has each one of us purchased a firearm from a well-established manufacturer and only to be disappointed when it failed to perform as advertised? We get more frustrated when the same firearm purchased by a friend works fine. The industry, as a whole, fails to deliver good products, day in, day-out. Since the buying community is so fragmented, the manufacturers continue to deliver junk without penalty. Well, only a big buyer has the clout to put an end to the junk deliveries. The purpose of all the fancy ISO 9001/9002 quality stuff is insure constancy and to document changes. The ISO stuff will not make a bad manufacturing process better, but it’s job is to raise the ‘red flag’ so as to say, “You’re making stuff that doesn’t meet the drawing”. These ‘red flags’ can then be audited by a customer who can say “You’re making junk, I don’t want it.” The second job of the quality requirements is very important - if deliveries are to be made over a long period of time, say two to five years – that is to document changes. During a lengthy delivery, many things can change, such as the steel source, the design of critical item s such as the barrel, slide or frame (changes could result from ‘producibility’ – meaning the manufacturer has modified the part to make it easier and cheaper to make or the manufacturing process itself). The ISO system requires all this to be documented and big important customers can review this documentation. Documenting the changes is important to the buyer. During the course of the Test and Evaluation, all the pistols have been subjected to a wide variety of expensive and extensive tests in order to select the best one. Now that all that has been done, to allow the manufacturer to make undocumented changes to the design or manufacturing process will cast a doubt on those test results. If changes have been made, how do we know the last pistol received is the same as the first? In Part One, the use of the First Article Test (FAT) helped to answer that question. However, the FAT and production sampling are catching the problem after-the-fact. The demand for the constancy must start deep within the manufacturer. As a minimum, I would recommend, the supplier of any firearms have and maintain an ISO 9002 certification for the facility involved in the manufacture the firearm. 3. How to conduct a weapons Test & Evaluation Here are some pointers to help conduct the T & E: A. Be prepared. The T & E must exactly reflect the requirements and must be conducted in accordance with the Validation Section in the Performance Description. I would recommend the Test Evaluators prepare the following: · A detailed plan for each requirement based on the words in the Validation Section. The plan should be so detailed as to allow anyone who reads it to perform the test. · Each detailed plan should include physical space on the paper to record the test results, date and signatures of the evaluators as well as the pistol serial model and number. · Rely on physical paper copies, not on electronic versions. · Make a master notebook of all the test sheets. Could easily run 50 to 100 papers. B. Have all the test equipment calibrated by a certified calibration company. This would include all calipers, weight scales, tape measures, force gages, etc. Keep a log book listing all calibrated items and the calibration date. C. Conduct ‘dry-runs’ of all tests using all test personnel. For many tests, there is only one chance to get it right. Dry runs will quickly pickup any errors, confusion or lack of equipment. D. Have a witness for all tests. The job of the ‘witness’ is to simply watch the evaluator and make sure he follows the test plan. The ‘witness’ should also countersign the test sheet. E. Be organized and focus on only one test at a time. F. Write down any and all results. Document! G. Take it slow. Rushing will cause confusion and errors. H. Label all magazines. Magazines can be a major source of test incidents. 4. The Ins & Outs of Reliability and Durability Testing Fun part of the T & E is the part with all the shooting, or at least that what everyone expect until several days into the shoot. In this test, the manufacturers submitted nine (9) pistols for testing. All nine were required to fire 3,000 rounds each and three (3) of the nine (9) fired 7,500 more, for a total of 10,500 rounds. Past testing experience showed the most a pistol should be shot before cooling and cleaning is 250 rounds. Thus, all shooting was broken into cycles of 250 rounds which means each pistol would fire 12 cycles to reach 3,000 rounds and some another 30 cycles. A cycle included: obtaining the pistols, ammo and mags; loading the mags; shooting and scoring, and cleaning the pistols. All this could be accomplished in 70 minutes. To perform a Reliability and Durability Test of this magnitude will require the following: · Each cycle will require a shooter as well as a ‘scorer’ (record keeper) · One Range Officer · Two Armorers · Several Cameras to record incidences · Secure boxes to store any broken items · Sufficient ammunition which has been thoroughly tested · A indoor or outdoor range large enough to safety accommodate all personnel · Good hearing protection. Recommend double hearing protection of ear plugs and muffs · Good ventilation if using an indoor range and range cleaning equipment · Brass picker-upper. South Brass makes a nice model which works well on smooth concrete, but would not work on grass or gravel. · Brass scrap barrel · First Aid kit or if the number of personnel warrant, a EMT on site · Plenty of targets · A health baseline on all shooter which should include: Blood lead level, hearing, and lung capacity. Lung capacity is need if the shooters will be wearing respirators while cleaning an indoor range. Respirators are highly recommended. · Pistol Cleaning stations, preferably away from the firing line stocked with all the patches, brushes, cleaning solvent and lubricates. · A method of documenting all cycle results. The best way is to have pre-printed score sheets issued to each shooter/scorer team each cycle · Written description of all potential malfunctions · A couple of ‘dry runs’ helps get everyone on the same page 5. Protests Regardless of how well you think you’re doing, someone isn’t going to like it and will file a ‘protest’. Usually, the protests will come in two categories: an objection to an item performance requirements or an objection to the test method/test results. Both are easy to rebut if you have done your homework. The protest is usually written by the company’s console and will whine on and on about how the requirements/test placed his company at a disadvantage and how the government agency has conspired against them. The best advice is never to ‘bad-mouth’ any manufacturer during any part of the specification development, solicitation or test. This can be used to ‘prove’ the company’s case. Always be professional. Other Suggestions in dealing with Protests include: · As discussed in Part One, always have a good reason for each requirement. Never go looking for a requirement just to eliminate a particular manufacturer as this will nearly always backfire. · Do what you say you’re going to do. Follow the validation test plan as closely as humanly possible. The manufacturers have all been given a chance to comment on the test plan and make recommendations. If they fail to provide feedback at the appropriate time, shame on them; if you modify the test plan without proper notification to the manufacturers, shame on you and you will loose. · Document that you did what you said you would do. Document the tests using video tape, computer output and/or the official witness. · Be prepared to write a coherent, factual, emotionless defense of the requirements and test plan/results. 6. Scoring Scoring the test results is a difficult task. The following thoughts need to be kept in mind: · Scoring must be designed to separate the ‘men from the boys’. · At the end of the day, a pistol needs to be selected. Can’t go though all this for nothing. · Too many pass/fail items might disqualify the best pistol for something relatively simple · Pass/fails invite protests since it eliminates a pistol from the solicitation · Try to adjust the scoring so pistols which miss the ‘little things’ can make it up on the big things, like the durability test · Scoring methods need to be set prior to the start of the test to prevent a manipulation of the methods based on the test results · A good total score of 1,000 points or less works well. Too much above 1,000 gets confusing · Only release a summary of the scoring in the solicitation, not the scoring range or amount for each item 7. Summary: I trust this has been helpful. I appreciate that all this is a major undertaking and requires a lot of time and money lacking in most small government agencies. However, even a small agency can perform simple tests with a handful of people if everything is well planned.