A set of stochastic algorithms to optimize the independence criterion for implicit imposition of the orthonormality constraint among the estimated sources will be presented. The major advantage of the proposed algorithms is the increased accuracy with which the weight matrix in the independent component analysis (ICA) model is estimated, compared to conventional schemes. In a pursuit to relax the orthonormality constraint, a design of the steepest descent (SD), conjugate gradient (CG) and quasi-Newton (QN) methods, intended for oblique manifold, will also be discussed with a few experimental results.