Re: ATM mirror mirror on the wall part 2

["Albert Highe" <ahighe@ix.netcom.com>]

Unfortunately, if you want statistically significant results, you can't run
just a few tests. If you want to have some confidence in your results,
statistical variation demands larger sample sizes. To achieve 95%
confidence, which means that 19 out of 20 times the true value will be
within one standard deviation of what you measured, you have to run a
minimum of thirty mirrors twice. Yes, I said 30. There is no way to be
"smarter" and run fewer samples. Fewer samples only gets you poorer
confidence in your results. So its only an issue of making a decision of how
confident you want to be in your results. Statistics dictates how much you
have to pay for that confidence. 99% confidence costs way too much for most
people. On the other hand, most people are uncomfortable being wrong more
than 5% of the time.

For 95% confidence, the best way is to make two sets of independent
measurements on thirty mirrors. To prevent bias in the results, the mirrors
need to be ordered randomly, measured, reordered randomly again, and
retested where the tester can't know the earlier results for each mirror.
All sixty measurements can be done by one person on one tester. Then you
simply compare the first results vs. the second. There is some simple math
to compare the results, but the key is to run enough samples. In the end,
you will have determined the sensitivity, within 95% confidence, achievable
by that person on that equipment. If you change person or tester, the series
needs to be repeated.

If you use a smaller sample size and/or deviate from the method, you are
wasting your time. The results won't be valid.

One variation is to have one person measure all thirty mirrors in random
order and then have another person (may be on another tester) run the second
set of measurements on the mirrors in another random order.  In this case,
the sensitivity of the method will be the worst of either person or tester.
If one person's method or tester does not measure effectively, you will not
know which person's method or tester is defective. So, its better to have
one person determine his or her sensitivity with a given piece of equipment
first.

It may surprise most of you that just about every piece of test equipment
and method, no matter how simple, fancy, expensive, or allegedly accurate,
fails this test. Most of us and most of industry rely on tests that are not
statistically valid and cannot possibly discriminate between "good" and
"bad".  That statement will undoubtedly provoke some controversy and
defensiveness, but it is nevertheless, true most of the time.

Albert

----- Original Message -----
From: "Scott Rychnovsky" <srychnov@chem.ps.uci.edu>
To: "Mel Bartels" <mbartels@efn.org>
Cc: <atm@shore.net>
Sent: Wednesday, June 06, 2001 7:45 PM
Subject: Re: ATM mirror mirror on the wall part 2


>
> Hello ATMers,
>
> I have volunteered to run a mirror testing round robin.  There are a
> few issues with need to deal with:
>
> (1) How many mirrors should we use.  I think we need to keep the
> number relatively small to minimize shipping and fatigue on the part
> of the testers. If every mirror is tested by interferometry the cost
> will be significant.
>
> (2) What should we use as a bookmark test?  Interferometry is the
> obvious choice.  Does anyone have any suggestions for whom to contact
> to purchase a professional interferometry evaluation of a parabolic
> telescope mirror?
>
> (3) What range of defects are we looking for?  How good and how bad
> should the test mirrors be?  I think the testing round robin would be
> interesting when trying to decide when a mirror is "good enough", but
> i am happy to hear your opinion.
>
> (4) I will put the results on a web site.  If you have suggestions on
> how to report the results I will be happy to hear them.
>
> Please contact me if you want to participate.
>
> Please contact me if you want to volunteer a mirror. You might give
> me some details about the mirror such as focal ratio, diameter and
> weight.
>
> I will be out of town until Sunday, so don't expect an answer to your
> e-mail messages immediately.
>
> Thank you for your interest.
>
> Scott
>
>
>
> >Scott Rychnovsky has volunteered a mirror and volunteered to run the
> >project.  Frank Ward and Bill Marriott have volunteered mirrors.
> >
> >If you wish to sign up, you should email Scott at
> >
> >srychnov@chem.ps.uci DOT edu    <- DOT edu == .edu
> >
>
>
> --
> Scott Rychnovsky
> srychnov@chem.ps.uci.edu
> ATM Page: http://www.mindspring.com/~tlepage/
>