Automation is set to put a lot of people out of work - but machine learning could help them find their next career.
The typical Australian will change careers during their professional lifetime, by some estimates. And this is likely to increase as new technologies automate labour, production is moved abroad, and economic crises unfold.
Jobs disappearing is not a new phenomenon – have you seen an elevator operator recently? – but the pace of change is picking up, threatening to leave large numbers of workers unemployed and unemployable.
New technologies also create new jobs, but the skills they require do not always match the old jobs. Successfully moving between jobs requires making the most of your current skills and acquiring new ones, but these transitions can falter if the gap between old and new skills is too large.
We have built a system to recommend career transitions, using machine learning to analyse more than 8 million online job ads to see what moves are likely to be successful. The details are in PLOS ONE.
Our system starts by measuring similarities between the skills required by each occupation. For example, an accountant could become a financial analyst because the required skills are similar, but a speech therapist might find it harder to become a financial analyst as the skill sets are quite different.
Next, we looked at a large set of real-world career transitions to see which way around these transitions usually go: accountants are more likely to become financial analysts than vice versa.
Finally, our system can recommend a career change that’s likely to succeed – and tell you what skills you may need to make it work.
Our system uses a measure economists call “revealed comparative advantage” (RCA) to identify how important an individual skill is to a job, using online job ads from 2018. The map below visualises the similarity of the top 500 skills. Each marker represents an individual skill, coloured according to one of 13 clusters of highly similar skills.
The similarity between the top 500 skills in Australian job ads in 2018. Highly similar skills cluster together.
Once we know how similar different skills are, we can estimate how similar different professions are based on the skills required. The figure below visualises the similarity between Australian occupations in 2018.
Each marker shows an individual occupation, and the colours depict the risk each occupation faces from automation over the next two decades (blue shows low risk and red shows high risk). Visibly similar occupations are grouped closely together, with medical and highly skilled occupations facing the lowest automation risk.
The similarity between occupations, coloured by technological automation risk.
We then took our measure of similarity between occupations and combined it with a range of other labour market variables, such as employment levels and education requirements, to build our job transition recommender system.
Our system uses machine learning techniques to “learn” from real job transitions in the past and predict job movements in the future. Not only does it achieve high levels of accuracy (76%), but it also accounts for asymmetries between job transitions. Performance is measured by how accurately the system predicts whether a transition occurred, when applied to historic job transitions.
The full transitions map is big and complicated, but you can see how it works below in a small version that only includes transitions between 20 occupations. In the map, the “source” occupation is shown on the horizontal axis and the “target” occupation on the vertical axis.
If you look at a given occupation at the bottom of the map, the column of squares shows the probability of moving from that occupation to the one listed at the right-hand side. The darker the square, the higher the probability of making the transition.
A small piece of the transitions map, with 20 occupations. Transitions occur from columns to rows, and darker blue shades depict high transition probabilities. Source, Author provided
Sometimes a new career requires developing new skills, but which skills? Our system can help identify those. Let’s take a look at how it works for “domestic cleaners”, an occupation where employment has shrunk severely during COVID-19 in Australia.
New occupations and skills recommendations made by the Job Transitions Recommender System for ‘Domestic Cleaners’ – a ‘non-essential’ occupation that has experienced significant declines during the COVID-19 outbreak in Australia.
First, we use the transitions map to see which occupations it is easiest for a domestic cleaner to transition to. The colours split occupations by their status during the COVID-19 crisis – blue occupations are “essential” jobs that can continue to operate during lockdown, and red are “non-essential”.
We identify top recommended occupations, as seen on the right side of the flow diagram (bottom half of the image), sorted in descending order by transition probability. The width of each band in the diagram shows the number of openings available for each occupation. The segment colours represent whether the demand has increased or decreased compared with the same period of 2019 (pre-COVID).
The first six transition recommendations for are all “non-essential” services, which have unsurprisingly experienced decreased demand. However, the seventh is “aged and disabled carers”, which is classified as “essential” and grew significantly in demand during the beginning of the COVID-19 period.
Since your prospects of finding work are better if you transition to an occupation in high demand, we select “aged and disabled carers” as the target occupation for this example.
Our system can also recommend skills that workers need to develop to increase their chances of a successful transition. We argue that a worker should invest in developing the skills most important to their new profession and which are most different from the skills they currently have.
For a “domestic cleaner”, the top-recommended skills needed to transition to “aged and disabled carer” are specialised patient care skills, such as “patient hygiene assistance”.
On the other hand, there’s less need to develop unimportant skills or ones that are highly similar to skills from your current occupation. Skills such as “business analysis” and “finance” are of low importance for an “aged and disabled carer”, so they should not be prioritised. Similarly, skills such as “ironing” and “laundry” are required for the new job but it is likely that a “domestic cleaner” already possesses these skills (or can easily acquire them).
While the future of work remains unclear, change is inevitable. New technologies, economic crises and other factors will continue to shift labour demands, causing workers to move between jobs.
If labour transitions occur efficiently, there are significant productivity and equity benefits for everyone. If transitions are slow, or fail, it will have significant costs for both individuals and the state and the individual. The methods and systems we put forward here could significantly improve the achievement of these goals.
We thank Bledi Taska and Davor Miskulin from Burning Glass Technologies for generously providing the job advertisements data for this research and for their valuable feedback. We also thank Stijn Broecke and other colleagues from the OECD for their ongoing input and guidance in the development of this work.
, Honorary Scholar, ; , Lecturer in Computer Science, , and , Michael J Crouch Chair in Innovation,
This article is republished from under a Creative Commons license. Read the .