[Stoves] Laboratory Comparison - but industrial standards for testing first!

[Charlie Sellers]

What a fierce level of introspection of our own community in the last few days!

Frank's comments come the closest to my interests so I replied to his email.  In industry (I have made alloys, metal powders, magnets, and medical devices around the world) we first develop a "standard operating procedure" (SOP) like the WBT, but an even more essential part of the quality control (QC) process (what we are talking about here) is to test the test - so see if it is capable of satisfying our needs for QC.  

Right now no one can quite say if the WBT can be used reliably around the world, by any person or lab, to measure a stove - even just  to determine firepower or  time to boil!    The results may not even be "publishable" since there is not enough supporting data - this happens with new tests and is normal.  My years working as a scientist at a gov't lab never exposed me to the real world of QC, lean manufacturing, six sigma, and all of the other things that are used to try and make your consumer products pretty darned reliable, but let me say that nothing in manufacturing is more important than the GUAGE REPEATABILITY AND REPRODUCIBILITY test methodology.  Anyone's carefully developed "test" (really an SOP - a test is when you actually apply the SOP to a sample) is meaningless until it is clearly (to the satisfaction of all) shown to be repeatable when YOU do it time after time and reproducible when OTHER PEOPLE do it independently.

Obviously there is a problem when only a lab or two has the capabilities to do the measurements (e.g.  emissions- but this realization permits us to prioritize spending for new lab equipment), but the stoving community would benefit from exposure to QC concepts for any and all of their testing.  I saw this issue at Stove Camp in 2006 and realized that it was impacting us in the Darfur stove project - just a few measurements here and there, under poorly controlled conditions and too few people  testing - so I wrote up an article/post on it here:
http://ewbappropriatetechnology4.blogspot.com/ (6th post from the top, called "Measurements and Statistics for Stovers").  Within the stove community we are just at Step 1 on the 4 step process (which is never finished) I describe - now we need to see how we can test our procedures (the WBT, the WBT plus emissions, etc.) to see if we are really capable of trusting our measurements.

Certainly a huge part that I see is missing is the use of the concept of a "normal distribution" - people are citing the standard deviation of 3 points, throwing out points arbitrarily, and worse without ever taking all the accumulated data on a stove and checking to see whether the distribution is such that ANY of the data can be trusted (Step 3).  Try not to average data or throw any away - you are losing information when you do this.

In the year that has passed since then I have seen all kinds of examples where "industrial" tried and true methods are missing - most recently in examining the failure rate of solar systems installed in developing countries (20% in one program).  Who is going to fund new programs when it is not clear that the implementers understand what they are doing?  There is no difference between a "new product introduction" in the consumer electronics field and one (such as a stove model) in the technical  humanitarian field - understand what the customer wants, develop a product, insist on perfect quality (through the use of capable testing), plan your introduction phase carefully, solicit feedback  from customers, etc.

I won't go on longer - if you read that post and need more information (R&R is not the best documented thing on the web, but search on it anyway) then maybe I can make a short presentation on it at ETHOS.  It comes down to having several operators (the inventor  can be one too!) test the same several stoves several times apiece, then crunch all the results through a spreadsheet and it produces a number (and supporting information, for improving) that says whether your testing is trustworthy.  If you have several labs then you can do this at each, either using the exact same stoves or different ones.  And if anyone is mass manufacturing stoves, then you use this same approach to determine if your manufacturing process is "within control" and your QC is where you want it to be (six sigma).

Thanks for tolerating an outside view.  Finger pointing about favoritism and mis-testing goes away completely when the testing is accepted by all - and concepts like guage R&R, normal distribution, and design of experiments (to examine the importance of different variables, as well as their interaction) are what it takes to get there.

Charlie
Rogue Stover




frank <frank at compostlab.com> wrote: Stovers,

The testing done during  Stove Camp should be taken with a grain of 
salt. To test something this complex I suggest there first needs to be 
about five tests done under the hood considered 'practice' tests with 
the designer involved. This is the time the designer learns that when 
fuel is applied the particles go up and learns about adjustments. When 
designing a stove without the use of the test hood the designer is only 
guessing as to final adjustments. Once adjustments for the best 
conditions are determined about twelve tests need be run. The best and worst eliminated and the ten averaged. This should be run by someone 
that is not the inventor IMO but the independent lab has had 
instructions from the inventor as how to use the stove.
This is a lot of work (money) but if the results are that the 'best 
show' is the winner of great rewards for contracts ( if that is what the 
investors base their decisions on) it may be worth it. And we will have 
'real' data and the truly best stove will win.




http://improvedstoves.blogspot.com/ - just R&D related to fuel efficient biomass stove issues
  http://travelswithcharlie.blogspot.com/ - most recent travel posts
http://www.flickr.com/photos/12820147@N07/sets/ - best of my travel photos
http://huiplesofguatemala.blogspot.com/ - my textile project in Guatemala - what colors!
http://travelswithcharlie2.blogspot.com/ - older travel posts, including Nepal travelogue 
http://ewbappropriatetechnology4.blogspot.com/ - just posts for the ATDT of the EWB-SFP; AT for developing countries

       
---------------------------------
Boardwalk for $500? In 2007? Ha! 
Play Monopoly Here and Now (it's updated for today's economy) at Yahoo! Games.