The reason why they have not returned is because of new entry requirement which include paying for work visas, health insurance and having at least £1200 to support themselves initially. This has left the UK economy with severe labour shortages whilst the loss of tax revenues generated by those EU workers.
The government has no idea how to solve this problem except by talking about 'training up our own people and paying them a lot more money' Many of 'our own people do not want the jobs previously done by EU workers and paying them more will only result in higher prices without an increase in productivity.
With an ageing population, we will need immigrants to fill the vacancies caused by Brexit and this will inevitably mean that we will have to find workers from Third World countries.