The International Institute for Strategic Studies (IISS) lamented growing signs that the insurgency was expanding from the south of Afghanistan into northern provinces, with rebels learning lessons from Iraq.
Elsewhere in its report "The Military Balance 2008", the London-based think-tank noted progress by the so-called surge in Iraq, but warned that US and other troops face being in the country for a generation.
In Afghanistan, the annual study said the NATO-led operation was most at risk where its technical advantage was reduced, particularly in eastern Afghanistan, the scene of intense fighting with militia.
"Failure in these actions would risk boosting Islamic extremism (not just in Afghanistan), would produce a failed state in an area of strategic importance, and would offer safe haven to terrorist organisations and the narcotics trade.
"It would also undermine the credibility of NATO in its first major out-of-area combat operation," the study said on Tuesday.
The IISS said that although NATO's 41,000-strong force was bolstering President Hamid Karzai's fledgling government, the administration "still lacks authority in much of the country."
The report echoed warnings last week from two US think-tanks which said troop levels had to be ramped up and major changes had to be implemented urgently.