However, the California-headquartered company will not abandon its four-inch screen models.
"iOS posted the lowest positive growth for both quarter (Oct-Dec) at 6.7 percent and for the year at 12.9 percent, underperforming the overall market in both instances," IDC said in a statement.
For 2013 calendar year, Apple sold 793.6 million units of its smartphone operating system (iOS) against 500.1 million in 2012.
Shipments of iOS enabled smartphones stood at 51 million units in October-December last year from 47.8 million in the year-ago period.
In 2012, Apple's smartphone operating system shipments rose by 46 percent to 135.9 million units against 93.1 million units in 2011. For Q4 2012, shipments rose by 29.2 percent to 47.8 million units against 37 million in the same period of 2011.
According to research firm Canalys, iOS shipments in 2011 rose by 96 percent to 93.1 million units compared to 2010.
Although it remains widely popular in the smartphone market, Apple has been criticised for not offering any new low-cost iPhone or a large-screen iPhone in 2013 to compete with other OEMs, it added.
"IDC believes the company will release a large-screen version in 2014, but will not altogether abandon the smaller 4-inch screen version of previous models," the firm said.
While smartphone market growth remained strong in 2013, it should be noted that the era of double-digit annual growth has only a few years remaining, IDC said.
"In 2013 we saw the sub-USD 200 smartphone market grow to 42.6 percent of global volume or 430 million units," IDC Worldwide Quarterly Mobile Phone Tracker Programme Director Ryan Reith said.
IDC said that the global smartphone market, which crossed the important milestone in 2013 of 1 billion units, was driven by continued momentum from Android and iOS.
Android and iOS accounted for 95.7 percent of all the smartphone shipments in the fourth quarter of 2013, and for 93.8 percent of all smartphone shipments for the year.
This marked a 4.5 percent increase from 91.2 percent share that the two platforms shared in Q4 2012 and a 6.1 percent rise from the 87.7 percent share they had in 2012.