|
|
|
|
Procurement of A New Pistol, Part One
Introduction:I had the good fortune of assisting a major government agency with the selection of a new service pistol for its uniformed and plain-clothes officers – over 20,000 of them. My assistance included: the developing the solicitation performance description – which included the validation process and scoring parameters -, the Test and Evaluation plan and schedule and the intermediate and final test reports. What follows is an overview of the pistol procurement process, especially the formation of the Performance Specification and the Test and Evaluation Plan. Unfortunately, due to the nature for the procurement, I cannot release any test results or even the makes/models of the pistols offered for test. However, what follows can be used by any organization as a ‘blue print’ for the procurement of a pistol, rifle, shotgun or other duty weapon. As for my qualifications, I am a Licensed Professional Engineer with over 15 years experience in the design, manufacture and program management of military equipment. I have written several major proposals for a major defense contractor in response to Department of Defense (DoD) solicitations and have designed and supported several major Test and Evaluation programs for ground military vehicles. I have been involved in the shooting sports since my early years, have been active in the NTI study group for three years and am a graduate of numerous Insights and DTI shooting courses. I have a B.S. in Agricultural Engineering, M.S. in Industrial Engineering and a Master’s Certificate in Project Management.
The Performance Description1. What is a ‘PD’The ‘heart’ of any solicitation is the ‘Performance Description’ (PD). The PD spells out exactly what the end product must “do” or “perform” and how this “do” or “performance” will be measured and evaluated. First, a little history: In times past, the DoD spent much time and money writing specifications on everything from nuts and bolts to large mobile vehicles, like tanks and cannons. These specifications spelled out in minute detail not only what the equipment had to do (go so fast, shoot so far), but how this performance would be achieved. For example: “the tank must be made from steel meeting MIL-A-46100” and “the cannon tube must be hardened to 50 to 55 RHC”. These specs really tied the DoD contractor’s hands as steel, for example, would become obsolete or perhaps better steel (or some other material) might be developed which is better, stronger, cheaper. DoD since the early 1990’s has learned the error of its ways and has dropped almost all its detailed specifications, know a ‘mil-specs’, in favor of commercial specifications (such as ASTM, ASME, or SAE) and ‘performance descriptions. PDs don’t ‘engineer’ the product; they just describe what the user needs it to do. For example: “The tank must go 70 mph, weight less than 30 tons and be protected against Soviet anti-tank weapons.” The military doesn’t care how this is accomplished (if you can make tissue paper work, all the better) thus leaving the door open for many possible solutions from innovative contractors. Care must be taken not to ‘close the door’ with requirements which can be seen to favor one contractor over another or cannot be justified. There must be real reasons behind every performance requirement, not just opinions and wants. The same process was used for this pistol procurement. The question: “What do we really need in this pistol and what do we want it to do” had to be answered. All the personal desires (opinions?) of “I want a polymer frame or an aluminum frame or steel frame” needed to be discarded. The drive was not to become pistol designers, but to be good pistols users – users who know what they really need and why they need it.
2. Developing a PDThe PD I developed had three sections: Ø The Performance Requirements: All the physical dimensions and performance criteria are outlined in the first section. Example:
3.26.1: Plugged Barrel: The pistol shall be able to withstand the test firing of one round with a bullet lodged in the barrel. The pistol shall remain intact after firing. Ø The Validation: This section spells out the test method for each of the physical and performance criteria. Continuing with the above example:
4.20.1: A .40 caliber 155-grain bullet will be pressed into barrel from the breach to a depth of 1.160 inches as measured from the breach. The barrel will be re-installed and the pistol fired using one round of service ammunition. Any barrel deformation or damage will be noted. 4.20.2: The test will be repeated, with a second pistol, with a .40 caliber 155-grain bullet lodged 0.5 inch from the end of the muzzle. Any barrel deformation or damage will be noted.
Ø The Evaluation/Scoring: This section details the value of each physical and performance criteria relative to one another. Items can be scored “Pass/Fail” or “Scored” with points. A ‘Fail’ is ‘the kiss of death’ as the pistol cannot be selected regardless of how well is may do in meeting other requirements. Again, continuing with the above example:
Pass: The pistol remains intact (no parts or components are loosened, drop, fly or fall off), no shooting hazards or potential hazards noted or experienced. Barrel bulges acceptable; pistol may be nonfunctional after the test; Fail: Pistol does not remain intact, shooting hazard or potential hazard noted or experienced or barrel ruptures
A. Performance Requirements: Developing the PD requirements is the fun part. In this case, and it is the best way, all the users were brought together for a week to work together on determining what was really required for this pistol. Some basic ground rules guided our efforts: Ø Make it a Team Effort – input solicited from all program areas Ø Make the specification reflect the true needs of the users – Performance Based Ø Keep the field of potential vendors as wide as possible Ø Minimize use of strict “Pass/Fail” requirements – only where needed Ø Maximum use of scoring, Ø Use of market surveys and vendor comments Ø Design for “Best Value” – performance, delivery, life-cycle, past performance and price Ø Utilize life-cycle costs – spare part costs and frequency of repair Ø Have consistent, repeatable tests and test procedures Ø Impose restrict, fair quality requirements on the vendors to be sure the government get what it pays for Ø Be fair & balanced (sounds like a commercial for a news channel)
The group made use of
two aids: The Old SpecThe ‘old’ spec was a classic ‘MIL-SPEC’ style document, but it was useful as it provided good food for thought. The old style requirements were ‘converted’ in to modern performance requirements. Some examples: Old Style Specification: Barrel shall be made from AISI 4140 steel hardened to RHC of 45 to 50 with six right-hand, butt-rifled grooves.
Performance
Specification: The pistols shall fire 3.0-inch groups using
Old Style Specification: Firing pin shall leave a 0.020-inch deep indentation in the primer.
Performance Specification: The pistol shall safely and reliably operate with all SAAMI specification .40 S & W ammunition.
The Market SurveyThe purpose of a Market Survey is to gather information as to what is currently available and to learn the latest features and improvements. This can be done formally, by announcing a Market Survey through the procurement web site (www.fedbiz.gov). Such announcement would be pretty basic: “Tells us what you’ve got in semi-auto pistols in 5 pages or less”. Or the Market Survey can be done relatively informally: Go to the nearest large gun store(s) and purchase one of each semi-auto. Since pistols are relatively low-priced (at least as far as the government is concerned), this was the chosen route. This turned out to be the best method as it gave the group an opportunity to touch and feel (fondle?) a wide variety of pistols. This also helped determine requirements like the size/weight, sight parameters, etc. and helped to prevent indiscriminately eliminating a manufacturer for something like 1/16th of an inch in overall length or for a ¼ ounce in weight. The group reviewed numerous pistols representing all major manufacturers during this exercise.
The PD Requirements,
as determined by the group, were broken into three categories: Physical, Environmental and Performance
Characteristics. Note: Other user
groups may not have arrived at the same characteristics due to their individual
circumstances and this is OK. Physical Characteristics:
Environmental Characteristics: · Salt water immersion · High Temperature · Low Temperature
Performance Characteristics: · Magazine Drop Test · Sight Zero Workmanship
Justification for Requirements:q The Caliber was dictated by the selection of the .40 S&W round in the early 1990s. Logistical and training demanded it remain the same. q The caliber, mode of operation, magazine capacity, lack of safeties (grip or manual) and decocking/cocking levels were all a function of training. The group required the new pistol be easily introduced without the need for additional training. This dictated the pistol be of similar operation to the Berretta 96D and the HK USP Compact. The pistols purchased for the market survey showed that many manufacturers made a pistol with or without these features, thus not limiting the competition. q The physical dimensions were driven by a need to arm both uniformed and plain-clothed officers. Ideally, the selection of two pistols, one large and one medium would have been a better approach, but the group was limited to ‘one model fits all’ approach. The group determined it was best to procure a ‘compact’ size rather than a full size pistol (like the Beretta 96D). The size, weight, grip size, sights, etc. were determined by a review of the market survey pistols. q The environmental needs were determined from typical conditions experienced by the officers. The high temperature (of 200F) was the temperature of a car trunk in summer; the low temperature was typical of conditions in the northern U.S. Salt water was typical brief submersion as experienced when falling off a boat or dock. q The reliability test of 3,000 rounds was an attempt to assess the manufacturing quality of submitted pistols. In the past, only three pistols were selected for any amount of firing. A much better understanding of the quality level would be achieved by having the manufacturer submit nine (9) pistols and have all nine shoot some reasonable amount. The group decided a reasonable expectation would be for each of the nine to fire 3,000 rounds right out the box, without any major failures. I believe most civilian shooters would be happy with this quality level. q The durability test of 7,500 rounds – in addition to the 3,000 in the reliability test – is based on a number of rounds fired in a seven (7) year lifetime. This test was not meant to determine mean-time-before-failure (MTBF), but as a way to separate the ‘men from the boys’. A MTBF test could be performed after award. q Accuracy duplicated the Beretta 96D requirements.
B. Validation: Each requirement has a corresponding ‘validation’ section that tells exactly how the requirement will be tested and evaluated. The validation section describes in very great detail exactly how – what equipment, what method, how long, etc., the requirement will be evaluated/tested by the government. This detailed description allows the manufacturer to perform exactly the same tests and measures to ensure (hopefully) that their pistol meets the requirements prior to submittal. It is important to remember that different validation methods & techniques will render different results. The validation method can be as simple as “visually inspect for a grip safety, manual safety, or cocker/decocker” to a very detailed description of how the durability test will be conducted and how malfunctions will be classified. Each validation method, however, must be tested with a ‘dry run’ prior to being written into the final PD. It was not unusual for me to completely re-write validation method after a dry run due to inaccuracies or incompleteness. A good example of a validation method was the trigger pull requirement. To the average shooter, this doesn’t sound like a big deal, but it was. There are at least three (3) different methods of measuring trigger pull: A ‘Lyman’ type fish scale gauge (digital or spring), NRA method & weights, or free weights. All have their place, but all will give slightly different results due to such factors as loading rate, accuracy/repeatability and operator skill. For example, the Lyman digital trigger pull gauge is advertised as being the best thing since sliced bread. However, when I attempted to calibrate our lab unit using a calibrated force gauge during a dry run, I found the Lyman to be off by as much as 8 ounces in 10 pounds or 5%– clearly not acceptable. To avoid the deficiencies with the above trigger pull devices, I devised a special, but simple mechanism (to be explained in Part 2 of this essay). This devise was clearly spelled out in the validation so all could duplicate the equipment and adjust their triggers accordingly. The validation section also outlines the number of repetitions and the method of data reduction. For the trigger pull, three pistols were selected and each was tested five times. The pull was the average for all 15 trials. It is very important to provide the clearest and most complete validation section as possible and follow it to the letter. Any variation from the validation test procedures could lead to a legitimate protest by a wronged supplier. First Article Test (FAT)Validation isn’t something that stops after the contract is signed. No, we must always be vigilant and double check the manufacturer. A FAT is a nice way of saying: “After we select your pistol, we’re going to re-check many of the critical PD requirements on samples from the first production batch. This is to make sure you are supplying us with what was submitted in response to the solicitation. The first production models must perform the same, or we don’t pay. Also, just to keep you (the manufacturer) on your toes, we’re going to check some critical requirements on three pistols out of every 1,000 you send us and they better be the same.” The PD contained a validation matrix which tells which tests will be done on the FAT pistols (just about everything) and which tests will be done on the production samples (not too many as it’s very expensive).
Summary for Part OneTo summarize: · Performance Descriptions are concerned with the end product, not the method of getting there. · Don’t be try to be design engineers · Make it a Team effort · State what you need and be able to justify it · Dry run your validation processes · Make your validation methods clear, accurate and complete · Don’t forget the FAT. Next Month’s Topic:· Getting manufacturers’ feed back on draft PDs · ISO 9000 and other quality requirements · How to conduct a weapon Test and Evaluation · The ins & outs of Reliability and Durability Testing (as hard on the testers as on the pistols) · Fending off protests from ‘evil’ gun manufacturers · Tips on scoring (racking and stacking)
|
|||||||||||||||||||||||||||||||||||||||