Here's the forerunner to my edge measurement method (which I've described many times).
1). Take all of the long trades and add up the total profits plus commissions and slippage for a year.
2). Pull out the same number of trades with the same holding period randomly from the year. Add the total profits made on the random trades.
3). Do the random test 1000 times.
4). Sort the random tests by total profits from lowest to highest.
5). Locate where your total profits were in relation to the 1000 tests. This is the ranking. The closer to the middle e.g. 500, the less the edge. Ideally the test will be above 700 or upper 30%.
6). Do the same test using short side trades.
7). Do the same test over multiple years. You'll get a statistic that shows how well the system is doing independent of whether the overall market was up or down.
Here's a example for the daytrading continuation system that I posted in another thread. I didn't finish the system, so it only has trades for the long side. First here's the summary report from 1997 - present:
A friend of mine (with a phd in stats), told me I should've done a chi square goodness of fit test with a significance level of .01. The only problem I had was that it only told me if I had a edge...not the degree. Given that I now use 5,000 samples in the test and combined it with some Monte Carlo testing, I find it to have real value.
This can be combined with other tests to measure the mean and std. dev. to provide a confidence level, but since the number of samples is very small...it will probably only go out to the 95% confidence level that the mean is positive.
Attachment: 
Attachment: 