Hedge fund returns worsen: Is enormous unraveling near?

As a result, hedge funds have returned just 3 percent year to date, though assets under management have swelled to USD 2.56 trillion, according to eVestment, a data firm that tracks the industry.

The change in fortune for hedge funds, which had trounced stock performance over the past two decades, has market insiders searching for answers.

"Could the underperformance be cyclical or is there a structural change that has changed the return structure of returns for the hedge fund industry?" Mary Ann Bartels, technical analyst at Bank of America Merrill Lynch, wondered in a report Monday. "We continue to conclude as in prior research, there (are) likely too many hedge funds chasing (too) few returns."

There are about 8,300 active hedge funds, and while the financial crisis wiped took a large toll new funds continue to crop up.

BofA research shows that convertible arbitrage (simultaneously buying convertible shares and shorting common stock) and event-driven (headline-following) strategies fared best in September, with respective returns of 5.6 and 5 percent. Market neutral lagged, losing 5.31 percent.

For the year, this is the third-worst performance to date since BofA began tracking hedges in 1994.

Nevertheless, fixed income-based hedge funds have seen inflows of USD 38.8 billion this year, while equity funds have watched USD 7.4 billion come out.

That risk aversion during an aggressive stock market run-up may be what is holding back fund performance more than anything else.

Hedge fund manager and author James Altucher predicts more trouble ahead, in which there will be "an enormous unraveling of hedge fund assets at end of year when hedge funds open their doors and this will lead to a bad Q1 in 2013."

From the end of 1994 until August, globally diversified hedge funds have outperformed the SandP 500 by 130 percent, though the returns have waned as the American economy has sought to shake off the financial crisis.

"Generally speaking, hedge funds have delivered better risk-adjusted returns during this period," Bartels said. "Although long/short strategies were unscathed by the 2000-2003 downturn, they did not fare as well during the financial crisis and have underperformed the market since the end of 2010."

Moreover, industry insiders say managers have struggled to meet benchmark returns as the market has become far less fundamentally driven and more influenced by headline events such as the European debt crisis and fiscal peril in the U.S.

Aggressive monetary easing from global central banks also has made it more difficult for fund managers to anticipate market movements, with active fund managers overall suffering what could turn into their worst year ever against benchmark returns.

"I worry that crowded trades will become more crowded. We may see hedge fund hotels become more overbooked and that will make them roach motels. You can check in but you can`t check out," said Michael Block at Phoenix Partners. "That will make markets more vulnerable to exogenous shocks."

-John Melloy, executive producer for Fast Money and Halftime, contributed to this report.