Thursday, January 28, 2016

63. THE PARETO LAW OF PREDICTABILITTY AND TIME SCALES. The multi-time-frames rainbow-walks stochastic processes.



SEE ALSO POSTS 7 AND 12


Preliminary Remark about Pareto and Log-normal distributions.

It is custom in the economist to model the financial inequalities with the Pareto distribution (see e.g. https://en.wikipedia.org/wiki/Pareto_distribution  ) which is essentially a polynomial function. The exact model of the inequalities is even worse and is closer to the log-normal distribution (see e.g. https://en.wikipedia.org/wiki/Log-normal_distribution ) where the severity of the inequalities is modeled with exponential functions.But here in this book we may keep talking about the Pareto distribution which is celebrated term among the economists


There is a thin line that separates business and investments as gambling that destroys the human spirit from business and investments as applications of scientific statistical knowledge under general principles that protects and reinforces the human spirit. This book contributes to see the difference and put the investors from the side of protected human spirit.



VALID STATISTICS AND PREDICTABILITY
We must make here some remarks about the robust application of statistical predictions in the capital markets.

1) The theory that the efficient markets and in particular that they follow a pure random walk is easy to refute with better statistical experiments and hypotheses tests. The random walk would fit to a market where the sizes of the economic organizations are uniformly random. But the reality is that they follow a Pareto or power distribution, therefore this is inherited in the distribution of the volumes of transactions and also in the emerging trends or drifts. 



2) The statistical models of time series  are more robust , when they apply to the entity MARKET as a whole and are better as  non-parametric , and not when they apply to single stocks and are linear or parametric. The reasons is that  a time series as a stochastic process , requires data of a sample of paths, and for a single stock is available only a single path. While for all the market the path of each stock or security is considered one path from the sample of all paths of all the stocks. Linear time series models or derived like ARMA, ARIMA, SARIMA etc are destined to fail for particular patterns like those described in the post 32, because the true equations are non-linear and in addition with random, time varying coefficients that derive the random emergence of the 4 basic observable patterns (see post 32 ). In addition the standard application of the time series by the researchers,  focuses  on stationary time series after they extract  a stable exponential trend, while in the reality the main concern should be the random path of the average value of the prices that shapes the patterns and is neither constant exponential trend neither zero ! The "statistical momentum conservation" (see post 10) might then be nothing else than an hypothesis that the random and time varying 1st order in time steps , partial correlation of the prices , is always positive. This can be easily tested statistically. E.g. in the cross exchange rate EURUSD but also in the indexes, the partial correlation of the current to the previous time step bar is measured indeed positive, in almost all time frames, except at the daily time frame, where the cyclic behavior prevails. In the daily time frame the partial correlation is negative , which means if one day is up the next day it is more probable that it is down. In addition, the cyclic behavior is even stronger in pairs of two days with negative partial correlation (two days up two days down etc)

3) The less ambitious the statistical application the more valid the result. E.g. applying a statistical hypothesis test, or analysis of variance   to test if there is an up or a down trend (drift) or none, is a more valid statistical deduction , than applying a linear model of a time series and requiring prediction of the next step price. 

4) Multivariate statistics, like factor analysis, discriminant analysis , logistic regression,  cluster analysis , goal programming etc are possible to utilize for a more detailed theory of predictability and of portfolio analysis, and sector analysis of the market and not only H. Markowitz theory. 

5) In applying of the above applications of statistics, the researcher must have at first a very good "feeling" of the data, and should verify rather with statistics the result rather than discover it. 

6) The "Pareto rule of complexity-results" also holds here. In other words with less than 20% of the complexity of the calculations is derived more than 80% of the deduction. The rest of the 20% requires more than 80% more complexity in the calculations.


The less hypothesis we use in applying statistical hypotheses, the better. That is why non-parametric statistics is better. An exception is our knowledge of the application of the Pareto distribution in various aspects of the market which we is parametric. 

That is why we avoid applying very complicated with many hypotheses and time consuming to estimate models to forecast the markets, but we prefer to respond to the market, by measuring only in a valid statistical way, the average position of the price, and the channel around it, the velocity (trend, 1st derivative) and acceleration-deceleration (2nd derivative)  of the prices. 

The statistical quantities from the front-office in trading need to me measured are 
1) the price position in the channel around the average, 2) the velocity (1st derivative) and 
3) the acceleration-deceleration (2nd derivative), which is done as statistical quantities by a hypothesis test or confidence interval. 
4) The support-resistance levels can be measured also by action-volume histograms.  The measurements are done with convenient indicators, and can also define in a statistically valid way, not only , the channels , the trend, reversal, and
5)  (Eliot or zig-zag) waves but also 
6) the spikes
7) It is required also an in advance in the past measurement and discovery of the basic stable cycles in the markets (see post 5)
8) An in advanced in the past measurement and discovery that trends duration and length, and volumes follow the Pareto distribution (see post 11,25 etc). 
In addition for the back-office of trading we need to measure the 
9) probability of success of trade based on the past history of trades, to apply the Kelly criterion, and also 
10) the average rate of increase of the trading funds and
11)  its variance again from the past history of trading.


THE OVER ALL STATISTICAL BEHAVIOR OF PRICE PATTERNS
We notice that although the price patterns are essentially  4 categories in details there are 6 distinct statistical patterns of the random or statistical position, velocity and acceleration of the prices (see post 32).
A class of stochastic processes can be defined as the behavior of the markets based on these   6 basic patterns P1, P2, P3, P4, P5, P5, P7. The Pi i=1,2,3,4,5,6,7 are essentially random  statistical patterns with random and variable parameters of size duration, and relative analogies that define them. E.g. we may have a Pareto distribution of the duration and price height of the patterns because of the inequalities in the markets. The 7th pattern P7 of stationary behavior  we may call intermittency pattern.  We may then assume a class of Markov processes  with random and variable transition probabilities, where each random type of pattern occurs and then a next one occurs. Bu the transition probabilities are not arbitrary! E.g. spikes occur usually at the begging (initial spikes)  and the end (terminal spikes) of trends, up and down trends with stationary channels in between them, shape cycles, that in the average of stable period, and some times of fixed beginning and end. 
The probability that a type of pattern occurs changes also according to the time scale. In 2, 5 or more years annual bars time frames, the non-waving trend pattern is dominating, while say in 5-minutes bars time-frame flat patterns are dominating. 
We notice that, by utilizing only pattern P3, of non-waving trend, and intermittency P7 we may derive all, other patterns with appropriate patterns of transition probabilities of the P3! I have coded a simulator of such a class of stochastic processes, superimposed on many time frames called Multi-time scales Rainbow Walks stochastic processes. 

This is the overall behavior of patterns of the markets, and there are some invariant properties  like 
1) Cyclic behavior as alternation with up and down trend patterns with flat channels at the bottoms and tops  
2) Statistical momentum conservation (see post 10) where the 1st time-step partial correlation of a price is almost always positive, 


3) A Pareto distribution of the duration and height of the patterns, due to the inequalities of the enterprises in the economic system (see post 10, 25, 57,63 )

There are many who complain that the indicators have lag, and prefer not to use indicators at all, but only the prices. This is rather stupid! The indicators when are measuring a statistical quantity  MUST have lag, because we are not interested for the price at the now only, but in a short-term past horizon too, which defines the statistical momentum which is statically conserved. The science of statistics is the best fr the moment one can have , from the collective scientific thinking, and we must  trust , respect it and be confident and humble when applying it. When we are applying the statistical mode of thinking for the markets, we never run serious dangers of being "burned"and "busted" in our deductions, as statistics claims everything only up to some probability, or probability inequality and interval.  


Most of the traders think that intraday manual trading is radically more profitable than day-to-day trading where only once per day (e.g. for 15 minutes) a control and a decision is taken. They think so because they estimate that profits will increase in direct analogy to the  shorter time scale they will use. But it is not so at least for two reasons. a) The shorter the scale the more the “noise” (=non profitable and non-tradable fluctuations of the markets due to unpredictability because of c) below)  b) In intraday manual trading, one has to spend many hours in front of the screen as if he was working at office while in day-to-day trading only 15-20 minutes per day, while he may have a normal work and normal day with other non-trading activities. c) Intraday and short time scales waves and patterns depend on a small number of people (mainly some packets of transactions by employees of the  big banks) and therefore are subject to unpredictable changes, of the actions of these employees. But long term waves depend on a very larger volumes , and global populations involved in the economy, therefore are more stable. From the psychological point of view, if someone believes in the natural guessing abilities of every one  , not though the observable patterns of the charts, but through  consciousness telepathy, then it is obvious that this small "window" of guessing which is not based on the observable patterns of the charts, works better for the larger scales where large populations are involved rather than to small intra-day scales where only a few people and in particular staff of the largest banks are involved. This is one more argument in favor of the better predictability of the larger time scales. Assuming that the ability of the small "window" of guessing  through  consciousness telepathy is inversely proportional to the probability of the size of the population that determines the movement of the prices, we get the Pareto diagram of the unpredictability among the  scales 

It is also important to realize, that this small "window"of guessing, based also on the news, is easy on the index funds as they represent, the overall feeling of the real business, of the economic system, and by far more difficult for individual enterprises and business. The trend of an index fund is macroscopic and artificial, as whenever an enterprise ceases to be growing according to its balance sheets, then it is substituted in the index by a new growing enterprise. Strictly speaking the tend of the index fund is the surplus demand in the society to put money in the big and larger business that  are represented with companies in the stock exchanges compared investing to smaller business outside the stock exchanges. It is like a  law of universal attraction where big money attracts more money. Strictly speaking again such a an excess demand for big and larger business is not the best in society, as it increases the inequalities, while it would be more desirable investing in small and medium size business usually outside the stock markets. But from the point of view of psychology the duality of the psychology in investing in index funds  is simple and asymmetric. When betting for up it is betting on hopes, when betting for down it is betting on fears
Similarly, for the same reason, using this factor of predictability, is by far more difficult, for the commodities compared again with the index funds. It is rare that there is a clear situation of demand-supply interplay, of a particular commodity, that can be found quantitatively in the news, and be early enough to apply  in trading. Most often there are no news useful, and there is no clear expected trend for a commodity. Cross-rates of foreign  exchange currencies are a little bit less difficult compared to other commodities. Currencies f go up or down compared to the totality of the rest following the growth of the underlying national economy. But cross-rates simply record their relative differences. Of course larger economies have a lower growth rate compared to smaller economies (exactly as in biological organisms, see end of the current post ) and the distribution of rates relative to sizes is a power low.  But again these cross-rate currencies with no strong  expected trend, compared to index funds. 

In the same rule we may be led even without assuming "intuitive guessing" abilities. It is known that the statistical distribution of the sizes enterprises follows the Pareto distribution, and so does the distribution of the size of packets of orders inherited from the enterprises,  therefore of the volumes too.  And the same we may assume for the size of populations of demand-supply that create the price patterns in the markets. Therefore that "inertia" which is due to the volumes ,of the movements of price patterns follows also the Pareto distribution. But this "inertia" due to the volumes,  is the basis of the "statistical momentum conservation", and also the basis of predictability of the moves of the markets. So large scale and large volume patterns and small scale and small volumes patterns predictability is like the predictability of the moves of the planets compared to the predictability of the moves say of asteroids that also collide more often between them.  Obviously the latter is more difficult to predict, even though the same physical Newton's law of momentum conservation holds.   Now as the difficulty of predictability of asteroids is proportional to the size and mass of the asteroids, and the size and mass follows a Pareto distribution, then so does the difficulty of predictability too (or unpredictability here).  The situation is also similar to the difference of predictability of say motion of celestial bodies in classical physics and the difficulty of predictability of particles in quantum physics of micro-world of atoms. 

The truth is as it seems to me  that for manual trading the golden scale is the seasonal cycles (2-6 months) as sub-cycles of a 5.55 Kitchin cycle or 22.2 years global climate cycle (Kuznet cycle, see post 5).  Of course by programming automate trading it is possible to trade intra-day without spending human time. But the rate of return of such automated intraday trading is not higher than the seasonal manual trading if in the latter, the human pattern recognition is involved which is superior to automated trading pattern recognition. Some sellers of automated trading show the results of their programs for a  limited time intervals (some months only) which appear very high so as to sell or rent them. But sooner or latter such automated trading had also significant failures so that in the long run (5 years or more) have less rate of return that the manual seasonal trading.
The way that unpredictability increases as the time scale becomes shorter, seems to be a result of the law of inequality. The very shape of the Pareto distribution could be used to present how unpredictability increases when the time sale decreases. In the next diagram of the Pareto distribution, the x-axis is the time scale, and the y-axis is the unpredictability. A hint of why this is so is the next: A wave in shorter time scale takes less volumes of transactions to be shaped. And the volumes of transactions, follow the Pareto or power distribution. The less the volumes the smaller the population (of transactions but also of traders) the less the predictability and stability. 

We may have a different situation of the above arguments if  the short term prediction of a trend (e.g. trend within  hours or within a day in 4-hours bars) is an inherited prediction from the larger scale trend (e.g. 6 months or 2.75 years) in which case the stability of the larger scale prediction is inherited also in the shorter scale, but for  sequences of  limited only small time intervals. But in such a case we are supposed to trade only one side of the trend (always the one that is in accordance with the seasonal trend) and it is not therefore a  genuine independent shorter scale prediction. In such cases the inverse pyramiding or moyen, or adjustments technique is the appropriate (see about this method in post  3). Notice that the adjustments or moyen or inverse pyramiding technique is based on the assumption of a constant infinite time trend. While the direct pyramiding technique is based on the assumption of a constant finite time trend following a power or Pareto distribution of time duration. 

It is crucial to realize that  cycles may emerge in the price changes , in a random way with a hazard rate of appearance at each period , and furthermore that they may appear not directly on the prices changes but on the rate of growth of prices.We must make clear here that we are not talking of exact periodicity but rather for randomly emerging temporary periodicity. 

From the  cycles (see post 5)  (not including their harmonics, that is their sub-multiples of the periods) the order of intensity of effect on the price movements, and therefore the order of predictability also is approximately the next:

Daily (1 Day earth)>> 
Year (12 months, earth)>> 
11 years global climate (Sunspots) >> 
Month (4 weeks, sun+moon)>>
2 weeks solar magnetic cycle (Parker Spiral)>>
160 mins Helioseismologic cycle >>
55 mins Helioseismologic cycle>>
5 mins Helioseismologic cycle.


And the  most predictable effect modulated by such cycles (after the long term permanent trend) is the reaction to an super-exponential moves (a blow-up at the end of trend in the form of super-exponential move or terminal spike).  (See e.g. https://www.ted.com/talks/didier_sornette_how_we_can_predict_the_next_financial_crisis    and http://www.er.ethz.ch/  Such super-exponential terminal patterns of trend may occur usually as result of overgrowth  of the one of the two populations in a demand-supply coupling rather that of domination and not so much of competition or cooperation. See also post 22. The frequency of emergence and the size of such  super-exponential blow-ups follows the law of inequalities in other words the Pareto or Log-normal distribution and is thus by fat more often than pure randomness would predict! )

THUS THE ORDER OF BETTER PREDICTABILITY IS 
1) LONG TERM PERMANENT TREND

2) A CYCLE IN THE ORDER OF PREDICTABILITY DESCRIBED ABOVE AND REALIZED AS REACTION TO SUPER-EXPONENTIAL TERMINAL MOVE (OR SPIKE).

Here in the chart below which is with the prices after taking the logarithm, we may watch deviations from the linear moves (super-exponential moves) and their highly predictable reaction lasting in the average 3.75 years. during 30 years!



The idea that the appearance of price patterns is the same among different time scales is literally wrong. It is true of course that the basic price patterns like waving channels , non-waving trend, spikes, and flat channels appear in almost all time scales. (For the 4 basic patterns see post 22) but it is not true that their relative probability of appearance is the same in all time scales! THERE IS SYSTEMATIC DEVIATION AS WE MOVE ACROSS THE TIME SCALES. The shorter the time scale the more often the patterns of flat channels and waving channels. The spikes appear in time sales of hourly or 4-hours bars, and appear less often as we go to large and smaller time scales.  As we go to larger time scales the pattern of constant non-waving  trend appears more and more often, and at a scale of months and years it seems to be the only pattern at annual tome intervals. 

From this we deduce easily that prediction and trading is much easier at the larger scales, than the short term intraday scales. The only phenomenon which is universal e.g. for index funds indexes, is that they reflect the laws of growth of enterprises and organizations, which is according to many studies Logy=a*Log x (super-linear of super-exponential growth) where x is the size of the organization, y is it time rate of growth (non-percentage but absolute growth) , and a is a constant. For biological organizations the a is less than 1 it is  about 75%  but for economic organizations the a is greater than 1 , it is about 115%  and then there is collapse of the growth. Nevertheless the index funds contain companies before their collapse, and any time a company starts to collapse, they substitute with a younger in the index fund. 



Deviations of the above law, is called by some researchers  Dragon-crises and are used to predict the reactive collapse either down or up.

EXPOSURE OF POSITIONS FOR THE LONG TERM TREND AS IT CHANGES BY TIME SCALE. 
Finally, we we remark that as the sample variance changes by the sample size by the rule of n^(-1/2), so also changes the ratio R/a^2 , where R is the average trend or drift, and a^2 is the variance of this trend. And this ratio determines the exposure of the portfolio, in the portfolio adjustment (see post 3) . E.g. as we calculated in post 3, if the annual  R is 10% and the adjustment is annual and a is 34%, then the exposure R/a^2 for the adjustment is 66%. But if the same trend and variance is calculated not annually but daily and the adjustment is daily, then  this exposure becomes about 3,5% !! See also post 12. 
Of course we are calculating here how the stable long term trend appears in short time scales. As we have seen in other posts, in short time scales there appear temporarily micro-trend patterns with much larger R/a^2, even larger than 66%. 

THE TOP 6 FACTORS OF ATTENTION IN MANUAL TRADING ARE

1) NEVER USE ALL YOUR FUNDS FOR TRADING.  DIVIDE THEM TO TRADING AND NON-TRADING FUNDS BY THE RATIO f=R/a^2 RULE (see below for this ratio or in posts 3,13,33). THE DIVISION OF FUNDS AT EACH PERIOD IS ADJUSTED TO CONFORM WITH  THIS PERCENTAGE RATIO. NEVER WITHDRAW PER PERIOD FROM THE NON-TRADING FUNDS MORE THAN HALF OF THE AVERAGE PROFITS OF THE TRADING FUNDS PER PERIOD. This division and adjustment of the funds has been applied for many years in buy and hold investments by  professor Michael LeBoeuf. 

2)  THE ONLY CERTAINTY, WHILE TRADING IS ALSO OUR  FIRST PRIORITY: WE MAY DETERMINE THAT OUR LOSSES AT EACH POSITION WILL NOT BE LARGER THAN A SPECIFIED PERCENTAGE DEFINED BY THE KELLY CRITERION (see  posts 3, 13, 33)

3) FOCUS ON MACROSCOPIC INSTRUMENTS LIKE  STOCK INDEXES WITH PERMANENT STRONG LONG TERM TREND, even if you want to trade at short time scales. (e.g. of the American Economy which is young and strong and indexes like Dow Jones, SnP500, Nasdaq etc)

4) FOR VERY LOW RISK AT OPENING POSITIONS ON THE PREVIOUS INDEXES WITH PERMANENT STRONG TREND, OPEN BETTING UPWARDS AT TERMINAL SPIKES AGAINST THE TREND. This is the Bill Williams technique. 

5) READ THE NEWS AND FINANCIAL STATEMENTS BUT THE ASSESSMENT OF THE PATTERNS OF THE MARKET REQUIRES THAT IT IS DONE IN MANY SUCCESSIVE TIME FRAMES CHARTS. This is a basic recommendation by Alexander Elder, which, by now, it is a common knowledge to traders


6) BE FLEXIBLE IN FORECASTING THE MARKET AND DO NOT HESITATE TO FOLLOW PROMPTLY ANY UNEXPECTED CHANGES OF THE TREND OF THE MARKET. 

The most essential tool for successful and profitable above the average,  trading from the three that the title of the book suggests (Law of growth, law of cycles, law of inequalities) is the law of cycles and the awareness to discover cycles in the charts of prices, that are not directly apparent. Especially when the cycles are 1) daily cycles to be traded with hourly or 4-hours bars and 2) Weekly cycles to be traded with hourly or 4-hours or daily bars  3) Monthly cycles to be traded  with daily bars 4) Seasonal 3-months cycles to be traded  with daily bars. 

Wednesday, January 6, 2016

62. Why most manual traders fail in creating a long term income starting from small amounts of money. A most difficult path to financial freedom.

ANYONE WHO WILL TRY TO MAKE MONEY SOLELY BY TRADING AND SUCH SYSTEMS OF TRANSACTIONS SHOULD BE AWARE THAT THERE IS A VERY POWERFUL AND ALMOST UNBEATABLE COLLECTIVE WILL SO AS NOT TO SUCCEED!  NO-ONE WANTS  PEOPLE TO QUITE THEIR JOBS AND MAKE MONEY THIS WAY AS IT IS SOMEHOW PARASITIC. IT IS IN SOME SENSE  UNETHICAL AS A PRACTICE ENFORCEABLE  TO  THE MAJORITY. AND OF COURSE NEITHER THOSE WHO HAVE LARGE CAPITAL  WANT THAT A MAJORITY WILL MAKE MONEY THIS WAY, AS THEY WOULD PREFER THAT THEY WORK IN THEIR COMPANIES FOR THEM. ONLY IN SPECIAL CONTINGENCIES AND SITUATIONS SOMETHING LIKE THIS WOULD BE ETHICAL. AND IN PARTICULAR A HIGHER MORALITY THAT WOULD SUPPORT SUCH A PRACTICE, WOULD BE PROVABLE WITH COLLECTIVELY BENEVOLENT DEEDS FROM A POSSIBLE SURPLUS OF SUCH MONEY!


Why most manual traders fail in creating a long term monthly income starting  from small amounts of money. A most difficult path to financial freedom.
There are plenty many reasons, of course that would make  manual trader to fail in trying systematic profits in capital markets, stock exchanges, forex etc.  The fact is nevertheless that the majority of traders fail in the long run when trying systematic profits that from little only money would create fast capitalization and a reliable monthly income, and this is exactly what the social system wants! 


it is not probable that automated trading would do so (see also post 6) . There is no serious social will to develop software for automated trading for fast capitalization and income! On the contrary the existing trading platforms, the last two decades still have serious drawbacks.  The possible development of a sufficient reliable software that would create in a fast way capitalization starting from very few money, and thus creating a monthly income for people ,even if it is possible, is at a deeper consideration, undesired from the economic system, itself which wants people to work for the business of those who have the capital, and thus control them  based on their need to survive. If such a software was to be developed, then this would give  access  to financial freedom to the majority, which we know is not in the intentions of the capital oligarchy than controls the economic system. And therefore a trader is left alone to develop such a system, which sincerely speaking is not really feasible, by one person only. Neither a collective open source volunteered development of such a software is really supported by the financial system.

 Therefore we go back to simplicity and simple only mainly manual practices that a single trader can do, that although simple are not probable for the majority of the people!


0) The reasons of failure in trading according to their power of effect may be classified to 

0.1) SOCIAL REASONS The possible development of a sufficient reliable trading system, manual or automated  that would create in a fast way capitalization starting from very few money, and thus creating a monthly income for people ,even if it is possible, is at a deeper consideration, undesired from the economic system itself!  As the economic system wants people to work for the business of those who have the capital, and thus control them  based on their need to survive. If such a software or manual trading system was to be developed and be easy to apply by the majority of the people, then this would give  access  to financial freedom to the majority (something like a guaranteed minimum income or subsidy for survival)  , which we know is not in the intentions of the capital oligarchy than controls the economic system. The awareness that the economic system is not much favorable for such possibilities is apparent e.g. in the writings of Bill Williams who has succeed to do it as "The alligator" and some times as "The gremlin." It is the old idea that the treasure (here of financial freedom) is guarded my a fierce and almost unbeatable dragon. A friend of mine, who is a professor and at the same time a successful trader, used to tell me that wanting to trade at the beginning  is like wanting to cross a field full of mines without any tool to detect them. Some how one could say that this is the usual situation with all businessmen  that want to create a capital. Nevertheless , judging from the scarcity of individuals who have succeeded in creating a reliable long term income, starting with a small amount of money though capitalization in the capital markets, compared to those that did it with ordinary real business, we conclude this this here is by far more difficult. On the other hand the fact that such a trading is after all capitalization as in many other types of business, is also an encouraging thought , because although business with successful and fast capitalization are not easy, they are not also impossible. 

0.2) The intellectual reasons. Not statistical valid thinking about random quantities in the markets, and not aligning with the scientific collective consensus, that may allow them to utilize the collective work of many thinkers. Many traders do not know how to think about random or half random quantities, unless they are educated in statistics and furthermore are willing to apply the statistical scientific mode of thinking. At this point we may also remark on the other hand ,that the science of applied statistics has not created a collective will to create fast capitalization in the capital markets, that would create income starting with a small amount of funds. In fact most scientists might suggest to avoid it. Therefore at this any trader is in his own. The successful trading is based on appropriate successful beliefs, and new habits of beliefs, and such a system of beliefs is rarely found as rare are the ones that have succeeded. But the scientific consensus of valid statistics, may hep on that. 
0.3) Psychological reasons. This is a very strong factor of failing, as most people are emotionally oriented. The successful trading as we said, is based on appropriate successful beliefs and habits of new beliefs, and such a system of beliefs is rarely found as rare are the ones that have succeeded systematically in trading. As the beliefs of a trader are directly affected by the impersonal beliefs, or beliefs of people of their environment, traders face a hard time creating the right beliefs and therefore the right emotions about trading. Furthermore the false beliefs of the majority of traders that are losing, either in their environment or even in the internet, is highly probable that will affect them, and make them lose too. Such false collective beliefs of the majority of the losing traders are often impersonal and written in many books of technical analysis, and have become so strong that the trader that joints and learns from the collective of other traders cannot but become a victim of them. In addition to that the close human environment of the trader may be hostile and obstructing to the desire of the trader to succeed and so as to allow  the trader to have the successful beliefs, even if the trader never talks about them. It may take years for some people to change the emotional and belief habits as correlated  with their open positions and the market moves. E.g. too many traders refuse to control the loss of a trade, at the worst case scenario for each trade, to be under 6% or under 2% , even when the percentage of their winning trades is as low as 51%-56%. They copy buy-and-hold investor that when they lose, they do not sell but they wait for the market to change direction. They find it very restrictive to bound loses to only 2% or 3%, and are based too much on the belief of their success in forecasting. They claim that by putting a tight stop loss,  they  will be losing almost always. There are many other examples where their habits of feelings and beliefs lead them systematically to loses.  Everybody knows how difficult is to change deeply rooted habits and beliefs. Just ask systematic smokers to give up smoking, or fat people to lose weight and become slim and one will also understand why it is difficult to practice successful trading. The more personal the trader takes his trading at the beginning (and this is the case because they are dealing usually with their own money) the more difficult it is to make it successful!  Some times their emotional habits are so inverse, that when they discover the trading that is winning in the long run,they discover also that it is not fun at all for every single trade, and emotionally it  is not even pleasant to proceed (apart from the final long term profit) ! The awareness that our sense of psychology and the world is often against us,   is apparent e.g. in the writings of Bill Williams, who has succeed to do be a winning trader, under the terms e.g.  like "The gremlins, the little devils".

0.4) Physical activities reasons. Trading which is intraday, requires many hours during the day, and  this is on often feasible to working people. Even daily trading which requires not more than half an hour per day, is already a strict demand of consistency, which for many people is not feasible. Furthermore requirements of sufficient good computer and internet support and even  , some funds to start , while at the same time one has a sufficient permanent income from a daily job is not the privilege of everybody. 



 In my opinion it is not that to learn the basics of technical analysis or statistics is difficult. There are other unexpected reasons too. I enumerate in my opinion some basic 4 of the reasons in order of significance apart from those mentioned hitherto.

     1)  One fundamental reasons is the brokers and the industry of trading itself. The brokers are interested for the profit of the transactions costs, or Bid-Ask profit, at the trading of traders which is independent to losing or gaining of the trader. Thus they push them to trade often, more often that it is necessary, which leads the traders to failure. They also corrupt them with trading competitions, where the winner is whoever will gain a usually extravagant rate of return within a fixed amount of time. This usually is a random effect, and the rate of return is ot at all the one that one can have in a systematic way. Thus the traders consider themselves a failure if the succeed in general a lower rate of return, which might be the normal and the one that they should conduct. So they push themselves to gain that very high rate of return, which in practice  impossible to sustain in a systematic way, thus they crash their account sooner or later. The brokers and other interested parts of the trading industry advertise very high unsustainable rates of return to attract customers and so a vicious cycle of trading failure is created. There are of course brokers that realize, that this tactic us not for their long term benefit, and try to help their customers to have systematic profits, and keep them for a long time. But the first time the broker will have economic difficulties, it will resort to the above pushing of their customers for high volume of transactions which sooner or later will lead the customers to crash their account. 
What is the normal sustainable rate of return of succseful trading. According to B. Williams, who has applied seasonal trading, with pyramiding (see post 61) it can raech at the maximum a 15% annually (and after a leverage between  20-100 if derivatives or CFD's are traded, it will be  about 300% annually, as Bill Williams reports). And a good intraday automated system will have sustainable monthly rate of return say for 5 years, of about 0,02%-0.1% (and after a leverage   of 100 if derivatives or CFD's, or forex  are traded, it will be  about 10% monthly, at a maximum draw down of 20%-30%). Now these values should not be considered high, if e.g. we compare them with the rate of return that a bank is making for example from a credit card, which is with an annual rate 10%-19%, but after the capital structure leverage of the banks which is 10 will be 100%-190% annually (and similarly for other banking products).
We should understand also that there is an indirect but everywhere existing hidden will in the impersonal economic system which wants the financially small to lose, and the financially  big to gain. It might be asked why one should not trade manually for example in 4-hours candles, and have the remarkable results of doubling the funds each month, as it is proved in the published results by the trading of the author himself at the end of the post 44. The reason is that this remarkable doubling in one month in manual trading, performed on 4-hours charts, is not something that can be done continuously. In order to have all these intraday fluctuations more or less predictable, you must look in the monitor every 4-hours, and have in your mind the evolution of the prices during the checking and this pushes almost all other daily activities aside. In addition this focus and concentration in the markets almost all the day, fills you with a rather toxic and negative energy which is not good for the body and the health. The situation is radically different for manual trading on daily charts with 15-20 minutes per day looking in the charts and seasonal trends. On the other hand if it is to code  an automated trading program, this has by far worse pattern recognition than the manual pattern recognition and the current state of the art of technology does not allow but a small fraction only of the manual trading doubling (1MD , where MD=Monthly Doubling)



2)     A second very significant reason in the same direction,  and with the same way of failure, is that it is not only the brokers and the industry that push the traders to extravagant unsustainable rates of return, but also the need of survival of the traders does too. Many traders are in an condition of unemployment, or with very low income of partial employment, and having some deposits in the bank , force themselves to trading for a living which is not possible with the amounts of funds they have in the bank, The result is a very risky trading, so as to attain very high rates of return, and thus a survival income, which crushes their account. In some cases the traders borrow from banks, and the payments that the banks require are by far higher that the systematic income it is possible to succeed in the markets with such funds, which leads again to crashing the account.


3)     A third reason is that the traders thing that intraday manual trading is radically more profitable than day-to-day trading where only once per day (e.g. for 15 minutes) a control and a decision is taken. They think so because they estimate that profits will increase in direct analogy to the  shorter time scale they will use. But it is not so at least for two reasons. a) The shorter the scale the more the “noise” (=non profitable and non-tradable fluctuations of the markets due to unpredictability)  b) In intraday manual trading, one has to spend many hours in front of the screen as if he was working at office while in day-to-day trading only 15-20 minutes per day, while he may have a normal work and normal day with other non-trading activities. The truth is that for manual trading the golden scale is the seasonal cycles (2-6 months) as sub-cycles of a 5.55 Kitchin cycle or 22.2 years global climate cycle (Kuznet cycle).  Of course by programming automate trading it is possible to trade intra-day without spending human time. But the rate of return of such automated intraday trading is not higher than the seasonal manual trading if in the latter, the human pattern recognition is involved which is superior to automated trading pattern recognition. Some sellers of automated trading show the results of their programs for a  limited time intervals (some months only) which appear very high so as to sell or rent them. But sooner or latter such automated trading had also significant failures so that in the long run (5 years or more) have less rate of return that the manual seasonal trading.
The way that unpredictability increases as the time scale becomes shorter, seems to be a result of the law of inequality. The very shape of the Pareto distribution could be used to present how unpredictability increases when the time sale decreases. In the next diagram of the Pareto didstribution, the x-axis is the time scale, and the y-axis is the unpredictability. A hint of why this is so is the next: A wave in shorter time scale takes less volumes of transactions to be shaped. And the volumes of transactions, follow the pareto or power distribution. The less the volumes the smaler the population (of transactions but also of traders) the less the predictability and stability. 



4)     Many traders apply only technical analysis, while they avoid fundamental analysis of the companies or of  the markets in general, and avoid news, as they consider them shifting sand. They are right in many cases. But if one has to obtain the maximum of the best results then all available relevant information should be used. For example in choosing a season trend,  it is better to confirm it with fundamental analysis (of the enterprise, the sector or macro-economic fundamentals) and news, rather than mere technical analysis indications.




There are 3 contexts of laws required in trading . The appropriate LAWS OF THINKING for trading, the appropriate LAWS OF FEELINGS for trading , and the appropriate LAWS OF ACTIONS for trading. 

The Successful trading is based according to these three laws on
1) POWER OF COLLECTIVE  SCIENTIFIC THINKING: A GREAT AND SIMPLE SCIENTIFIC PERCEPTION OF THE FUNCTION OF THE ECONOMY THROUGH SOME GLOBAL STATISTICAL LAW. E.g. The law of Universal attraction in economy: that big money attracts more big money in the capital markets, and this by the balance of demand and supply makes securities indexes of the companies , that are indeed the big money, to have mainly stable ascending trend, whenever one can observe such one. Valid statistical deductions can be obtained with simple statistical hypotheses tests about the existence or not of a trend, with sample size half the period of a dominating cycle). (STABLE GREAT SCIENTIFIC THOUGHT-FORM  OR BELIEF FACTOR IN TRADING. )

2) POWER OF COLLECTIVE PSYCHOLOGY: A LINK WITH THE POSITIVE COLLECTIVE PSYCHOLOGY.(E.g. that the growth of security indexes also represent the optimism of the growth and success of real business of the involved companies. And we bet or trade only on the ascension of the index, whenever  an ascending trend is observable). (STABLE GREAT POSITIVE COLLECTIVE   EMOTIONAL OR PSYCHOLOGICAL FACTOR IN TRADING. )


3) POWER OF INDIVIDUALS SIMPLE , CONSISTENT AND EASY TO CONDUCT PRACTICE. (e.g. a trading system with about 80% success  rate that utilizes essentially only one indicator in 3 time frames, simple risk management rules of stop loss, take profit, trailing and escalation, and time spent not more than 20 minutes per day. In this way there are not many opportunities of human errors in the conduction of the trading practice. Failed trades are attributed to the randomness and are not to blame the trader). (STABLE SIMPLE AND EASY PRACTICAL  FACTOR IN TRADING)

We may make the metaphor that successful trading is the ability to have successful resonance with the  activities of top minority of those who determine the markets.

In trading there are 3 components in the feelings that must be dealt with. 1) The feeling of MONEY itself, 2) The feeling of the UTILITY of the money 3) The feeling of the RISK of the money each time. What is called usually money management in trading is essentially RISK MANAGEMENT. 


REMARKS ABOUT VALID STATISTICS AND PREDICTABILITY
We must make here some remarks about the robust application of statistical predictions in the capital markets.

1) The theory that the efficient markets and in particular that they follow a pure random walk is easy to refute with better statistical experiments and hypotheses tests. The random walk would fit to a market where the sizes of the economic organizations are uniformly random. But the reality is that they follow a Pareto or power distribution, therefore this is inherited in the distribution of the volumes of transactions and also in the emerging trends or drifts. 

2) The statistical models of time series  are more robust , when they apply to the entity MARKET as a whole and are better as  non-parametric , and not when they apply to single stocks and are linear or parametric. The reasons is that  a time series as a stochastic process , requires data of a sample of paths, and for a single stock is available only a single path. While for all the market the path of each stock or security is considered one path from the sample of all paths of all the stocks. Linear time series models or derived like ARMA, ARIMA, SARIMA etc are destined to fail for particular patterns like those described in the post 32, because the true equations are non-linear and in addition with random, time varying coefficients that derive the random emergence of the 4 basic observable patterns (see post 32 ). In addition the standard application of the time series by the researchers,  focuses  on stationary time series after they extract  a stable exponential trend, while in the reality the main concern should be the random path of the average value of the prices that shapes the patterns and is neither constant exponential trend neither zero ! The "statistical momentum conservation" might then be nothing else than an hypothesis that the random and time varying 1st order in time steps , partial correlation of the prices , is always positive. This can be easily tested statistically. E.g. in the cross exchange rate EURUSD but also in the indexes, the partial correlation of the current to the previous time step bar is measured indeed positive, in almost all time frames, except at the daily time frame, where the cyclic behavior prevails. In the daily time frame the partial correlation is negative , which means if one day is up the next day it is more probable that it is down. In addition, the cyclic behavior is even stronger in pairs of two days with negative partial correlation (two days up two days down etc)

3) The less ambitious the statistical application the more valid the result. E.g. applying a statistical hypothesis test, or analysis of variance   to test if there is an up or a down trend (drift) or none, is a more valid statistical deduction , than applying a linear model of a time series and requiring prediction of the next step price. 

4) Multivariate statistics, like factor analysis, discriminant analysis , logistic regression,  cluster analysis , goal programming etc are possible to utilize for a more detailed theory of predictability and of portfolio analysis, and sector analysis of the market and not only H. Markowitz theory. 

5) In applying of the above applications of statistics, the researcher must have at first a very good "feeling" of the data, and should verify rather with statistics the result rather than discover it. 


6) The "Pareto rule of complexity-results" also holds here. In other words with less than 20% of the complexity of the calculations is derived more than 80% of the deduction. The rest of the 20% requires more than 80% more complexity in the calculations.


The statistical quantities from the front-office in trading need to me measured are 
1) the price position in the channel around the average, 2) the velocity (1st derivative) and 
3) the acceleration-deceleration (2nd derivative), which is done as statistical quantities by a hypothesis test or confidence interval. 
4) The support-resistance levels can be measured also by action-volume histograms.  The measurements are done with convenient indicators, and can also define in a statistically valid way, not only , the channels , the trend, reversal, and
5)  (Eliot) waves but also 
6) the spikes
7) It is required also an in advance in the past measurement and discovery of the basic stable cycles in the markets (see post 5)
8) An in advanced in the past measurement and discovery that trends duration and length, and volumes follow the Pareto distribution (see post 11,25 etc). 
In addition for the back-office of trading we need to measure the 
9) probability of success of trade based on the past history of trades, to apply the Kelly criterion, and also 
10) the average rate of increase of the trading funds and
11)  its variance again from the past history of trading.

The back-office statistical quantities in 9), 10), 11) are related , by simulation as e.g. in the simulator in post 43.

I have also created a simulator to experiment with different rules of withdrawals. 

THE OVER ALL STATISTICAL BEHAVIOR OF PRICE PATTERNS
We notice that although the price patterns are essentially  4 categories in details there are 6 distinct statistical patterns of the random or statistical position, velocity and acceleration of the prices (see post 32).
A class of stochastic processes can be defined as the behavior of the markets based on these   6 basic patterns P1, P2, P3, P4, P5, P5, P7. The Pi i=1,2,3,4,5,6,7 are essentially random  statistical patterns with random and variable parameters of size duration, and relative analogies that define them. E.g. we may have a Pareto distribution of the duration and price height of the patterns because of the inequalities in the markets. The 7th pattern P7 of stationary behavior  we may call intermittency pattern.  We may then assume a class of Markov processes  with random and variable transition probabilities, where each random type of pattern occurs and then a next one occurs. Bu the transition probabilities are not arbitrary! E.g. spikes occur usually at the begging (initial spikes)  and the end (terminal spikes) of trends, up and down trends with stationary channels in between them, shape cycles, that in the average of stable period, and some times of fixed beginning and end. 
The probability that a type of pattern occurs changes also according to the time scale. In 2, 5 or more years annual bars time frames, the non-waving trend pattern is dominating, while say in 5-minutes bars time-frame flat patterns are dominating. 
We notice that, by utilizing only pattern P3, of non-waving trend, and intermittency P7 we may derive all, other patterns with appropriate patterns of transition probabilities of the P3! I have coded a simulator of such a class of stochastic processes, superimposed on many time frames called Multi-time scales Rainbow Walks stochastic processes. 

This is the overall behavior of patterns of the markets, and there are some invariant properties  like 
1) Cyclic behavior as alternation with up and down trend patterns with flat channels at the bottoms and tops  
2) Statistical momentum conservation (see post 10) where the 1st time-step partial correlation of a price is almost always positive, 
3) A Pareto distribution of the duration and height of the patterns, due to the inequalities of the enterprises in the economic system (see post 10, 25, 57,63 )

There are many who complain that the indicators have lag, and prefer not to use indicators at all, but only the prices. This is rather stupid! The indicators when are measuring a statistical quantity  MUST have lag, because we are not interested for the price at the now only, but in a short-term past horizon too, which defines the statistical momentum which is statically conserved. It is not a race of speed, it is  a challenge of successful perception. The science of statistics is the best for the moment one can have , from the collective scientific thinking, and collective consensus, and we must  be honest and humble to admit is restricted abilities, but also trust , respect it and be confident for the success   when applying it. When we are applying the statistical mode of thinking for the markets, we never run serious dangers of being "burned"and "busted" in our deductions, as statistics claims everything only up to some probability, or probability inequality and intervals.  

There are some also that claim to have "intuitive guessing" about how the markets will move beyond the observable state of the markets. This of course cannot be included easily in  the standard statistical inference. But it seems to me that sometimes, this is in certain human and social environmental conditions is too much to ask from yourself, and it may fire-back to systematic opposite to the actual markets moves guessing! I believe that fortunately pure statistical inference from the observable states of the market only,  may be adequate for very profitable trading.   



MORE REASONS OF FAILURE WILL BE FOUND IF THE REASONS OF SUCCESSFUL TRADING ARE UNDERSTOOD AS WRITTEN IN THE POST 44

THE TOP 6 FACTORS OF ATTENTION IN MANUAL TRADING

1) NEVER USE ALL YOUR FUNDS FOR TRADING.  DIVIDE THEM TO TRADING AND NON-TRADING FUNDS BY THE RATIO f=R/a^2 RULE (see below for this ratio or in posts 3,13,33). THE DIVISION OF FUNDS AT EACH PERIOD IS ADJUSTED TO CONFORM WITH  THIS PERCENTAGE RATIO. NEVER WITHDRAW PER PERIOD FROM THE NON-TRADING FUNDS MORE THAN HALF OF THE AVERAGE PROFITS OF THE TRADING FUNDS PER PERIOD. This division and adjustment of the funds has been applied for many years in buy and hold investments by  professor Michael LeBoeuf. 

2)  THE ONLY CERTAINTY, WHILE TRADING IS ALSO OUR  FIRST PRIORITY: WE MAY DETERMINE THAT OUR LOSSES AT EACH POSITION WILL NOT BE LARGER THAN A SPECIFIED PERCENTAGE DEFINED BY THE KELLY CRITERION (see  posts 3, 13, 33)

3) FOCUS ON MACROSCOPIC INSTRUMENTS LIKE  STOCK INDEXES WITH PERMANENT STRONG LONG TERM TREND OR AT LEAST STRONG AND CLEAR SEASONAL TREND, even if you want to trade at short time scales. (e.g. of the American Economy which is young and strong and indexes like Dow Jones, SnP500, Nasdaq etc)

4) FOR VERY LOW RISK AT OPENING POSITIONS ON THE PREVIOUS INDEXES WITH PERMANENT STRONG TREND, OPEN BETTING UPWARDS , AT TERMINAL SPIKES AGAINST THE TREND. This is the Bill Williams technique. 

5) READ THE NEWS AND FINACIAL STATEMENTS BUT THE ASSESSMENT OF THE PATTERNS OF THE MARKET REQUIRES THAT IT IS DONE IN MANY SUCCESSIVE TIME FRAMES CHARTS. This is a basic recommendation by Alexander Elder, which, by now, it is a common knowledge to traders


6) BE FLEXIBLE IN FORECASTING THE MARKET AND DO NOT HESITATE TO FOLLOW PROMPTLY ANY UNEXPECTED CHANGES OF THE TREND OF THE MARKET.