Crafted by European Commission chief Jean-Claude Juncker, the €351-billion ($387 billion) plan is based on €21 billion in seed money from the EU budget and the bloc's European Investment Bank (EIB).
Juncker expects this will trigger massive private investment over three years to take the plan to the total figure.
But the ILO warned that the plan would have only a marginal effect on unemployment if it does not focus on making credit accessible to job-rich small enterprises and provide more investment to countries with higher jobless rates.
"If the Juncker investment plan is not well designed, the employment benefits will be very small," said Raymond Torres, director of research at the ILO.
"By 2018, the employment impact will be only 400,000 new jobs, and given the 23 million unemployed this is a very small percent," he told reporters.
"If the plan is well designed, by contrast the number of jobs created could reach 2.1 million by 2018 and lower the unemployment rate by almost one percentage point," Torres said.
He said the plan had to focus on projects involving large economies of scale such as energy networks and green projects.
"The plan has to tackle the sources of low employment in Europe and in particular in small and medium-sized enterprises," he said.
"In high unemployment countries like Italy, Spain and so on there are small enterprises that cannot invest because they do not have proper access to bank
The Juncker plan is built on a complex mechanism intended to attract private investors to risky and overlooked projects that meet the EU's longer term goals.
The ILO also said countries that were in more need of aid should be allocated proportionately more resources, pointing to a North-South divide.
It said France, Germany, Italy and Britain received more than 45 percent of all funding by the European Investment Bank.
"In recent years, the disproportionate rise in unemployment levels has not been followed by a parallel increase in financing from the EIB," it said, citing Greece, which receives only 2.3 percent of EIB funding going to EU member states, while it is home to 5.1 percent of the bloc's unemployed.
"Similarly Spain receives 16.6 percent of total EIB funding within the EU but hosts 23.1 percent of the EU's total unemployed," it said.
"These developments have imposed huge economic and social costs, with the worst impacts in southern Europe but with damage to households and working
people across the region," said ILO deputy director for policy Sandra Polaski.
"The urgency to address these losses increases with every passing day," she said.