Democrats and republicans

I have not been following the US presidential campaigns closely but lately I have been listening to some speeches by Barack Obama (thanks to my brother, Ganesh's enthusiasm in his speeches).

Just yesterday I listened to his (only part1) and John McCain's interviews with Rick Warren on CNN videos. Both speeches were good and clearly represented their parties' ideals. McCain's answers were short and clear while Obama's were long and complex. This was apparent starting with response to simplest questions like "State three wise men you would seek advice from" to complex issues like "Abortion". McCain's answers reflected gradient descent type approaches while those of Obama did MCMC type. No doubt both parties are dedicated to USA's growth and its benefit to the world. Both greedy deterministic approaches and probabilistic approaches have advantages and disadvantages depending on problem space: distribution of the problem instances. The main challenge in deciding whom to vote is what is to clearly understand the current problem instance and intelligently choose the better strategy. As is evident from the history republican strategy works most of the times. This is true in many optimization problems: gradient descent, though a local optimizer works in many many practical cases. But in hard cases when we need real "landscape shifts" in searching for the solutions we stand a better chance with probabilistic and "holistic/global" approaches. Such approaches tend to be computationally hard and require cleverer design of algorithms for feasibility.

Citizens of USA have been smart enough to choose Democrats a few times but I hope they realize this is one of those times again.

Faith in machines

In many of my previous posts I stressed, efficiency and local perspectives. Such emphasis can be traced back to faith in machines which reflects the faith of theoretical computer science community that polynomial amount of resources is efficient while exponential is not.

I love studying machines as the results can so much help understand ourselves (modulo communicable understanding). Computer Science provides a nice unified way for studying machine characteristics. This reminds me I have to finish one of my posts on language of thoughts :) It has a rough precursor which I posted about 2 years ago.

Practical and efficient statistics relies on Bayesian and Markovian principles which allow principled way of working by understanding limits of machines. It's amazing to know that lot of lower bound results in complexity theory and almost entire applied statistics can be traced back to the contributions of Markov brothers! All the extensions are non-trivial but behind the non-trivial efforts of later generations was the motivation based on faith in practicality or in more crisp words, machines.

Importance of being normal

In my previous post I discussed how variance in abilities and needs in a local neighborhood is necessarily a result of efficient resource management by Nature for sustenance of life. Then how change is essential to keep the variance (multiple hypotheses) over time to avoid getting stuck in local optima. I also mentioned that being around normal value is important.

In statistics filtering is a problem of estimating posterior of a random variable given observations correlated with the variable over time. Life can almost be treated as a random variable with some moments. In a more global perspective it's hard to characterize these moments and hence the posterior is usually represented using just random samples. I had a related post about six months ago. But our lives are mostly dominated by local perspectives. Actually if we had true global perspectives all the time we would be super natural!

In statistics a very popular technique of estimating a complex non-linear probability distribution of a random variable is non-parametric kernel density estimation. Intuitively it says that any complex distribution can be approximated using sum of Gaussians (or normal distributions). Let's say if we can track these individual Gaussians then we automatically track the overlaying complex distribution. Kalman filters are useful when the Gaussians undergo linear changes that is the mean and variance of the Gaussian only undergo linear transformations. For reasonable non-linear changes there are linear approximations resulting in extended Kalman filters. But for highly non-linear transformations the approximations made in extended Kalman filters are not good enough. Hence people developed unscented Kalman filter which is a combination of sampling based and closed form trackers. The key elements in unscented Kalman filters are the a set of sample around the mean of the distribution. These "sigma points" are the ones that undergo non-linear transformations which can then lead to recovering the necessary moments! See it's quite important to be around the normal distribution especially in the era of highly non-linear changes to actually "participate/contribute" in successful propogation of moments.

Need for change

Most of us agree that change is important yet hard. All of us can understand the benefits of change but the reason it seems hard is partly not having an "causal understanding" of why it is important. In this post I try to analyze why change is necessary for our survival. For that first I generalize the perspective of change to be more than just variance in temporal dimension of life. This means change characterizes any variance in the needs in any of our effective neighborhood. For e.g. it could be variance in the tastes of your roommates or variance in the goals of your friends etc. Having this generalized perspective, as I will try to argue, helps to see that variance in temporal dimension is not much harder than issues like tolerances etc. and that variance is essential.

Let's see why we need variance in the first place. There is high correlation between the composition of various chemicals in the body to the personalities and behaviors we manifest. Nature which has limited resources. For life to be persistent it is important that life can sustain on variety of resources so that Nature can efficiently "refill the resources". Based on the refilling abilities of Nature our bodies evolved to incorporate variance in the needs. We are not that variant in terms of needs for oxygen for e.g. since Nature seems to be very efficient in that resource regeneration. Hence the variance in our needs can be roughly traced to the variance in the availability of resources in our Natural neighborhood. Since Nature cannot handle all (varied) needs of all the life in a global way efficiently it decentralizes need fulfilling activities into the life forms itself. In other words the body chemical compositions are evolved in such a way as to locally have a cycle of supply and demand: starting from the most obvious examples, some are male, some are female, some have strong feelings about environment, some have strong feelings about high energy colliders creating black holes (see here), some are interested in making money, some are interested in education, some are spiritual, some are materialistic, some are good in theory, some are good in practice etc. etc. So to summarize we can think of local variances in needs is an efficient design of Nature for long-term sustenance of life. Something like: For a rope to be strong the individual fibers and yards and then strands have to be intertwined with friction among them. One other important thing with variance is that variance has to be "normal" locally so as to have the benefit of decentralization otherwise it would demand redundant effort. For e.g. if the friction between the fibers or yarns or strands is too high the rope might self-destruct under it's own friction without additional effort.

Now since the cycles of supply and demand usually are formed locally we might get stuck at local optima (which would be evident by diminishing returns in the cluster etc.). Change in temporal dimension would shuffle around the neighborhoods and gives us a chance to get out those local optima and form new cycles. Eventually we hope to find global optimum configuration. But as it is well known such optimization problems though can be "solved" require exponential amount of time in principle. So enjoy the journey and don't be scared to enter new cycles. It also helps to keep in mind an important property of stochastic optimization methods that not every move is better than the previous move which precisely is its strength. Of course blind (ignorant) change is not great since there are lot of probabilities (based on evidences, priors and likelihoods) you could compute to figure out the types of changes (moves) to reduce the mixing time.

I would ask the readers to pay attention to the words "efficient" and "local" as they carry the central message. Of course what all I discussed above (or in general in this blog) is not always new but is based on original thought. Lot of economists study such behaviors professionally and it would greatly help design your lives better by reading papers in such fields (I don't though): that's why the mathematicians who study and contribute to understanding such patterns are eligible Nobel prizes.

Causes of importance

Couple of weeks ago as part of our regular discussions Suzan and I were discussing about ones importance in society and service to humanity. She was saying that all kinds of work are equally important and that it's unfair to down weigh a particular type of work as easy and boring. Just recently Scott posted trying to show how it's meaningless to argue about one field of science being more fundamental (and hence superior). Whether something is important is not, makes only sense in the context of time and majority as per our current understanding of Nature. For eg. we need large number of particles acting at a micro-scale for the second law of thermodynamics to hold on the macro-scale and usually a theorem has to be checked over and over again independently for its correctness.

Monte Carlo methods are usually used for simulations of complex probability distributions without closed forms. Using such methods random samples are drawn from probability distributions to represent the distributions. Drawing random samples implies that all samples are equally likely according to the distribution. But the main problem is that it is seldom possible to draw samples from the complex distributions (partly because they don't have closed forms). Hence the samples are drawn from a heuristic approximation and then given importance weights according to a likelihood function. Metropolis-Hastings is a very famous algorithm for such simulations. If the distribution is dynamic then the simulation is called filtering and particle filtering is a very common tool. Essentially all such techniques depend on the Bayes rule. But that's not my point. For successful simulation of a distribution the crucial design aspects are the proposal and the likelihood functions. These functions can be chosen arbitrarily and usually domain specific knowledge is heavily needed. If they are designed properly then over a period of time the samples start behaving random with equal importance weights, meaning representing the true distribution.

If humans are supposed to represent uniformly drawn particles of the distribution of life energy then all humans will have equal chance of survival. But because of complex correlations and interdependences such uniform sampling is not possible and hence we have to design proper proposal and likelihood functions to give appropriate importance weights so that over time the chances of survival become uniform. We can already start seeing some of such changes based on today's ages of expectancy through out the world. Hence it is important to have such importances to different types of work like giving more importance to let's say medical work compared to the work that can be automated effectively. Thus it is not only not unfair to undermine some kind of work but in fact recommended for greater good. Once the humans reach certain peaks in the distribution then they would have similar weights as is the case for physicists vs. mathematicians vs. computer scientists vs. biologists and so on.