A Joint Learning Approach to Intelligent Job Interview Assessment

1 Introduction

As one of the most important functions in human resource management, talent recruitment aims on acquiring the right talents for organizations and always has direct impact on business success. As indicated in an article from Forbes, US corporations spend nearly 72 billion dollars each year on a variety of recruiting services, and the worldwide amount is likely three times bigger [Bersin, 2013] In particular, job interview, which is considered as one of the most useful tools and the final testing ground for evaluating potential employees in the hiring process, has attracted more and more attentions in human resource management. While substantial efforts have been made on the improvement of job interview process, traditional interview process has a substantial risk of bias due to the subjective nature of the process. This situation could be even more severe, since different interviewers may have different technical backgrounds or different experience levels in personal qualities. This may lead to a biased or incomplete assessment of job candidate.

∗Corresponding Author.

Recently, the Artificial Intelligence (AI) trend has made its way to talent recruitment, such as job recommendation [Malinowski et al., 2006; Paparrizos et al., 2011; Zhang et al., 2014], talent mapping [Xu et al., 2016], and market trend analysis [Zhu et al., 2016]. However, fewer efforts have been made on enhancing the quality and experience of job interview. A critical challenge along this line is how to reveal the latent relationships between job position and candidate, and further form perspectives for effective interview assessment. Intuitively, experienced interviewers could discover the topic-level correlation between job description and resume, and then design the interview details to measure the suitability of applicants. For example, a candidate for "Software Engineer", who has strong academic background, might be interviewed with questions not only about "Algorithm", "Programming", but also "Research". Meanwhile, compared with the technical interview, the vocabulary of comprehensive interview could be largely different.

To this end, we propose a novel approach to intelligent job interview assessment by learning the large-scale real-world interview data. Specifically, we develop a latent variable model named Joint Learning Model on Interview Assessment (JLMIA) to jointly model job description, candidate resume and interview assessment. JLMIA can effectively learn the representative perspectives of different job interview processes from the historical successful job interview records. Also, two categories of interviews, technical and comprehensive interviews, which are hosted by technical and managerial interviewers respectively, could be well differentiated. Furthermore, based on JLMIA, we also provide solutions for two applications named person-job fit and interview question recommendation. Extensive experiments conducted on realworld data clearly validate the effectiveness of JLMIA, which can lead to substantially less bias in job interviews and provide a valuable understanding of job interview assessment.

2 Problem Statement

Formally, our data set contains the recruitment documents of |M| unique jobs, i.e., S = {Sm = (Jm,Am)}|M| m=1 , where Jm is the job description of the m-th job and Am is the interview records of this job. Specifically, Am = {(Rmd, Emd)}|Dm| d=1 contains |Dm| interviews, where Rmd is the resume of candidate in d-th interview, and Emd is the corresponding interview assessment. Since all of the job descriptions, resumes, and interview assessments are textual data, we use bag-of-words to represent them, e.g., Jm = {wJmj }N Jm j=1 , similar to Rmd and Emd.

A job description Jm contains detailed job requirements, and a resume Rmd mainly consists of the past experiences of this candidate that can reflect her abilities. Meanwhile, the evaluation about a candidate in interview assessments bridges the gap between job requirements and her ability. And according to the goal of interviews, interview assessments can be further divided into technical and comprehensive interview.

As we known, during the interview, interviewers tend to ask questions related to the work experiences of candidates. Thus there often exits strong correlation between interview assessments and resumes. However, job description is usually more abstract than resumes, and candidates with different backgrounds may be suitable for the same job. Thus we think although there exists correlation between job descriptions and resumes, the diversity of job descriptions is less than that of resumes. In addition, it is obvious that the focus of interviews is different according their goals. Thus it is better to model the differences between technical and comprehensive interview.

Generally, the main tasks in this paper can be summarized as: Task 1, how to discover the strong correlation between resumes and interview assessments? Task 2, how to model the latent relationships between job descriptions and resumes? Task 3, how to distinguish the differences between different interview categories?

3 Technical Details of JLMIA

To solve the above tasks, we propose a novel joint learning model, namely JLMIA. In this section, we will formally introduce its technical details.

3.1 Model Formulation

To model the latent semantics in job description, resume, and interview assessment, we assume there exist latent topics, represented by ϕJ , ϕR and ϕE , in all of them. And

Algorithm 1: The Generative Process of JLMIA for Resume and Interview Assessment

1. For each topic k of candidate interview record:
2. (a) Draw ϕRk from the Dirichlet prior Dir(βR).
3. (b) Draw ϕET k and ϕEC k from the Dirichlet prior Dir(βE ).
4. For each job description Jm:
5. (a) Sample topic distribution θJm ∼Dir(α).
6. For each candidate interview record pair (Rmd,Emd, Imd):
7. (a) Sample topic distribution θAmd ∼ N(h(θJm,C), δ2 I)
8. (b) For the r-th word wRmdr in resume Rmd:
9. Draw topic assignment zRmdr ∼Multi(π(θAmd)).
10. Drawword wRmdr ∼Multi(ϕRzRmdr ).
11. (c) For the e-th word wEmde in interview assessment Emd:
12. Draw topic assignment zEmde ∼Multi(π(θAmd)).
13. Drawword wEmde ∼Multi(ϕET zEmde ) (Imde==TI).
14. Drawword wEmde ∼Multi(ϕEC zEmde ) (Imde==CI).

our tasks are further transformed to model the relationships among these latent topics. First, to model the strong correlation between resume Rmd and interview assessment Emd, we directly assume they share the same tuple-specific distribution θAmd over topics. Second, for revealing the relationships between job descriptions and resumes along with the differences between their diversity, we generate θAmd from the logistic-normal distribution with mean parameter related to the topic distribution of job description θJm. And the topic numbers of ϕJ , ϕR and ϕE are set as |kE | = |kR| = C · |kJ | = CK. In other words, for each topic in ϕJ , there are C topics in ϕR (ϕE ) related to it. Third, we use a label I ∈ {TI,CI} (e.g., Technical Interview or Comprehensive Interview) to indicate the type of interview for each interview assessment, where different types of interview assessment are generated from different topics ϕE ∈ {ϕET , ϕEC }. To simplify our model, we follow the idea in [Wang and McCallum, 2006], and set the interview label for each word in interview assessment instead of the entire interview assessment.

The graphical model of JLMIA is shown in Figure 1. Since the generative process of job description is the same as Latent Dirichlet Allocation (LDA) [Blei et al., 2003], here we only list the generative process for resume and interview assessment A = {Am}|M| m=1 , showed in Algorithm 1, where h(θ,C), in line 3.(a), is a vector concatenating C log vectors of θ, i.e., h(θJm,C)k = logθJm,k′ , k′ = k mod K, 1 ≤ k′ ≤ K, and π(θ), in line 3.(b).i and 3.(c).i, is the logistic

Due to the non-conjugacy of the logistic normal and multinomial, the latent parameters posterior is intractable. Thus we propose a variational inference algorithm for JLMIA.

3.2 Variational Inference for JLMIA

Here, we develop a variational inference algorithm for JLMIA based on mean-field variational families. The basic idea behind variational inference is to optimize the free parameters of a distribution over the latent variables, so that the distribution is close in Kullback-Liebler (KL) divergence to true posterior, which can be substituted. In our model, let us denote all latent variable parameters by Φ and all hyperparameters by Ω. Following the generative process, the joint distribution can be factored as:

where each component can be calculated by:

Then, corresponding to this joint distribution, we posit the fully factorized variational families as following, where the detail description of each term can be found in Appendix:

According to [Blei et al., 2017], minimizing the KL divergence between variational distribution and true posterior, is equivalent to maximize the log likelihood bound of job interview records, which is the evidence lower bound (ELBO):

where the expectation Eq[·] is taken with respect to the variational distribution in Equation 2, and H(q) denotes the entropy of that distribution.

The largest challenge to maximize ELBO is the nonconjugacy of logistic normal and multinomial, which leads to the difficulty in computing the excepted log probability of topic assignments in documents of each candidate interview records. Similar to [Wang and Blei, 2011], we introduce a new variational parameter ζ = {ζm1:|Dm|}m=1:|M| to preserve the lower bound of ELBO. Here we take the Eq[logp(zRmdr |θA)] as an example to explain it (the Eq[logp(zEmde |θA)] can be computed in a similar way):

For maximizing the ELBO, we develop an EM-style algorithm with coordinate ascent approach to optimize parameters, the details of which can be found in Appendix.

4 Application

Here, we will introduce two applications enabled by JLMIA, i.e., Person-Job Fit and Interview Question Recommendation.

4.1 Person-Job Fit

Person-Job Fit is the process of matching the right talent for the right job. Formally, given a job description Jg and a resume Rg, the objective is to measure their matching degree. Specifically, we need to first leverage JLMIA to infer the latent topic distributions of Jg and Rg respectively. However, our model cannot infer the topic distribution for an individual resume or job description. Thus we construct a S for resumes (job descriptions), where all of other data are set as empty, and infer the corresponding topic distribution.

After the variational parameters of topic assignment of each word, φJg and φRg , are learned, we can compute the document-topic distribution by:

Then, by computing the similarity between θJ g and θR g , we can measure the suitability between job description and resume. Actually, any distance calculation formulation between two probability distributions can be used here for measuring the similarity, such as Cosine distance and Kullback-Leibler divergence. Note that, since the dimension of θJ g and θR g may be different, here we have:

where Ck is a set of mapping index that satisfies θR g,c (c ∈ Ck) is generated from θJ g,k . In particular, the vectors θJ g and θR g learned by JLMIA can be regarded as low-rank semantic representations of job description and resume. Thus, using these representations instead of original bag-of-words as features for training a classifier (e.g., Random Forest) is another solution for Person-Job Fit.

4.2 InterviewQuestion Recommendation

During the interview, interviewers need to ask some questions to evaluate candidates. However, due to the limited expert resources, sometimes the interviewers may not have enough domain knowledge to prepare discriminative questions for systematically judging the competencies of candidates, especially from the view of Person-Job Fit.Thus, in this paper, we propose an effective algorithm for recommending interview questions based on JLMIA and interview questions accumulated in historical assessments.

To be specific, given a question database Q = {qi}Ni=1 , the problem of interview question recommendation is defined as retrieving a set of questions X that are related to a given query Υ (i.e., job requirement item or experience item of candidate). Similar to the process of computing the topic distributions θR g ,

we can compute the topic distribution θQ i of each question qi ∈ Q, through regarding interview questions as a part of interview assessment. Let θQ i denote a latent representation of question qi and the latent representation of the given query Υ as θΥ g ∈ {θJ g , θR g }.

To recommend high quality questions to interviewers, on the one hand, the selected question set X ⊂ Q, |X| = L should be relevant to the query θΥ g , on the other hand, we hope to avoid making the questions in X too similar to each other. To balance relevance and diversity of selected question set, we select questions X by maximizing the following objective function:

where, Rel(Υ,X) and Div(X) measure the relevance and diversity above, Sim(∗, ∗) is chosen as Cosine(∗, ∗) while Dis(∗, ∗) is set as 1-Cosine(∗, ∗), and Rel and Div are normalization factors, commonly chosen as the maximum possible values of Rel(Υ, X) and Div(X) respectively.

In general, the calculation of addressing F(X) is computationally prohibitive, since we will suffer the assemble explosion problem if we calculate Equation 6 for all subsets X. Fortunately, since F(X) defined in this paper is submodule [Tang et al., 2014], the simple greedy algorithm could achieve a (1 - 1/e) approximation of the optimal solution,

5 Experimental Results

In this section, we will introduce the performance of JLMIA based on a real-world interview data set.

5.1 Experimental Setup

The data set used in the experiments is the historical recruitment data provided by a high-tech company in China, which contains total 14,702 candidate interview records. To be specific, with the help of several staffing experts, we manually screened records with high quality interview assessment written by senior interviewers, and removed the records which lack details in job description or resume. After that, the filtered data set contains 4,816 candidate interview records related to 409 job positions. In JLMIA, we empirically set fixed parameters {δ2 , βJ , βR.βE } = {0.01, 0.1, 0.1, 0.1}. Note that, our model is trained with original Chinese words. And for facilitating demonstration, all experimental results were translated into English.

5.2 Evaluation of Topic Joint Learning

To evaluate the effectiveness of joint learned topics by JLMIA, we first trained our model on all successful job interview records. In particular, we set the parameters K = 10 and C = 2. Table 1 shows one randomly selected latent topic of job description and corresponding topics of resume and two types of interview assessments. Each topic is represented by several words with the highest probability.

We can observe that the topic of job description, containing "Experience" and "Foundation" of "PHP" and "Web", should be related to web development. Similarly, the corresponding topics of resume and technical interview assessment also contain front-end-related keywords, "HTTP", "Web Site", "Element","JS" and "CSS", which indicate the professional skills of candidates. Thus we believe that our model can effectively reveal the latent relationship among job description, resume and interview assessment. More interestingly, we can find that topic #1 and topic #11 of resume, which are both generated from topic #1 of job description, contain different keywords, which validate the assumption that the diversity of job description is less than that of resume. Meanwhile, compared with technical interview assessment, there are more keywords like "Communication", "Pressure" or "Work Overtime" in comprehensive interview assessment, which are related to the evaluation of personal qualities.

5.3 Performance of Person-Job Fit

Here, we evaluate the performance of JLMIA in terms of Person-Job Fit. Specifically, given a job description and a resume, we treat their latent topic distributions learned by our model as their representation vectors. Then, we train classic classifiers to predict the matching degree between the job and the candidate. Besides, to further demonstrate the effectiveness of our model, we also use the similarities between their representation vectors for measuring Person-Job Fit.

Benchmark Methods

We selected Latent Dirichlet Allocation (LDA) and bag-ofwords (BOW) vector representation as baselines. For LDA, we merged the resume of a candidate and the job she applied for as a document for learning the latent topics. And for bagof-words, where the i-th dimension of each vector is the frequency of the i-th word of the vocabulary, it itself is a kind of representation. Due to the limited space and similar trends of results, here we only selected Cosine and Kullback-Leibler similarity based approaches, and selected Random Forests and GBDT as classifiers. Please note that because the similarity between two BOW vectors is meaningless, we did not treat it as a baseline here.

Data Preparation

Different from the similarity based approaches, only one type of samples, i.e., positive samples, is required, the classifier based approaches need to prepare unsuitable pairs of job description and resume as negative samples to train classifiers. Although we can intuitively regard the historical failed job applications as negative samples, we do not know the exact reasons behind these failures. For example, some failed applications are just due to the low pay benefits, or other similar reasons in offer negotiation. Therefore, we manually generated the same number of negative samples to train classifier by randomly selecting resumes and job descriptions from the successful job interview records. Along this line, the experiments will only focus on the representation of latent topics, while interference from other factors will be impaired. After that, we randomly selected 80% data for model training and the other 20% data for test.

Performance Analysis

To evaluate the parameter sensitivity, we trained JLMIA by varying the parameter K from 10 to 50, and the parameter C from 1 to 5. The person-job fit performance of JLMIA based on Cosine similarity and different parameters is shown in Figure 2(a) and 2(b). We can find that the Receiver Operating Characteristic (ROC) AUCs and Precision-Recall (PR) AUCs are both better with small K, and reach the highest with K = 10 and C = 2. Therefore, we chose the best parameters K and C for the following experiments. Similarly, we also evaluated LDA model with different topic number parameters K, and chose K = 30 for other experiments.

Table 2 shows the Person-Job Fit performances of JLMIA and baselines. From the results, we find that our model consistently outperforms other baselines in both similarity based approaches and classifier based approaches. It indicates that JLMIA can effectively capture the latent relationship between job description and resume. More interestingly, the performances of JLMIA in similarity based approach is also higher than most of baselines. It clearly demonstrates the effectiveness of the representation learned by JLMIA.

5.4 Performance of Question Recommendation

To evaluate the performance of interview question recommendation of JLMIA. we first collected 1,085 interview questions as the candidate set from historical interview assessments, and then, compared JLMIA (K = 10 and C = 2) with BM25, a classic information retrieval model based on keywords matching which ignores the latent relationship between queries and questions. In our algorithm, the parameters are empirically set as Rel = 5, Div = 20 and µ = 0.9.

We randomly selected 100 experience items as the queries. For each query, we recommend 10 questions by JLMIA and BM25. Then, we asked 3 senior interviewers to evaluate the performance of recommendation questions. They were first required to judge which questions are relevant to this query, where the number of relevant questions is the relevance measure. Then, they needed to judge how many different technical aspects mentioned in those relevant questions, which is diversity measure, and how many questions are about personal quality, which should be different between technical interview (TI) and comprehensive interview (CI). As the average results shown in Table 3, we can find that compared with traditional keywords matching based approach BM25, JLMIA can recommend questions with more relevance and diversity. Meanwhile, JLMIA also can recommend more questions related to personal qualities, especially, the number of personal quality questions for comprehensive interview is more than technical interview, which distinguishes the different focuses of them two.

Further more, to illustrate the effectiveness of our question recommendation approach, we also show an example of top 4 questions recommended by different approaches in Table 4. Obviously, the given experience item is about web development. We find questions recommended by JLMIA contain all technical aspects mentioned in experience item (e.g., T1, T2 and T3 is about "CSS and HTML programming", and C4 is about "web development"). Also, JLMIA recommends questions designed for "HTTP" (i.e., T4 and C1), which is useful knowledge for web developers. Second, for the comprehensive interview, JLMIA also recommended questions to evaluate the personal qualities of candidates, such as C2, which is related to the communication ability and problem analysis ability. Last, for the questions recommended by BM25, since they must have the same words in given requirement, the semantic relationship between keywords are neglected (e.g.,

"HTTP" and "web"). Thus, the recommended questions by BM25 do not contain more technical details.

6 RelatedWork

Recruitment Analysis. With the importance of talents at an all time high and the availability of recruitment big data, recruitment analysis has been attracting more and more attentions [Xu et al., 2016; Zhu et al., 2016]. As early as 2006, Malinowski et al. tried to find a good match between talents and jobs by two distinct recommendation systems [Malinowski et al., 2006]. In 2011, Paparrizos et al. exploited all historical job transitions as well as the data associated with employees and institutions to predict the next job transition of employees [Paparrizos et al., 2011]. Recently, besides the match of talents and jobs [R´acz et al., 2016], researchers are also devoted to analyze recruitment market from more novel perspective, such as market trend analysis [Zhu et al., 2016; Lin et al., 2017], career development analysis [Li et al., 2017], talent circles [Xu et al., 2016] and popularity measure of job skills [Xu et al., 2018]. Although the above studies have explored different research aspects of recruitment market, few of them are developed for enhancing the quality and experience of job interviews. To this end, in this paper, we proposed a novel approach for intelligent job interview assessment by joint learning of multiple perspectives from large-scale real-world interview data.

Text Mining with Topic Model. Probabilistic topic models are capable of grouping semantic coherent words into human interpretable topics. As an important member of archetypal topic models, Latent Dirichlet Allocation (LDA) [Blei et al., 2003] has a lot of extensions [Zhu et al., 2014; Mimno et al., 2009; Pyo et al., 2015], etc.. Among them, some works focus on modeling shared latent topic distribution among multiple categories of documents, and have a wide range of practical applications. For example, Mimno et al. [Mimno et al., 2009] designed a polylingual topic model that discovers topics aligned across multiple languages. Pyo et al. [Pyo et al., 2015] proposed a novel model to learn the shared topic distribution between users and TV programs for TV program recommendation. Different from existing research efforts, in this paper we developed a novel model JLMIA to jointly model job description, candidate resume and interview assessment.

7 Concluding Remarks

In this paper, we proposed a novel approach for intelligent job interview assessment by learning the large-scale real-world interview data. To be specific, we first developed a latent variable model JLMIA to jointly model job description, candidate resume and interview assessment. JLMIA can effectively learn the representative perspectives of different job interview processes from the successful job interview records in history. Furthermore, we exploited JLMIA for two real-world applications, namely person-job fit and interview question recommendation. Extensive experiments conducted on real-world data clearly validate the effectiveness of JLMIA, which can lead to substantially less bias in job interviews and provide a valuable understanding of job interview assessment.

Acknowledgements

This work was partially supported by grants from the National Natural Science Foundation of China (Grant No.91746301, 61727809, 61703386).

A EMAlgorithm of Variational Inference

In this appendix we give some details of the EM-style algorithm of variational inference outlined in section 3.2

First of all, we define each variational distribution term of the variational families in Equation 2. To be specific, the variational distribution of each topic proportion vector θJm is Dirichlet parameterized by vector γ Jm. The variational distribution of θAmd,k , the k-th dimension of topic proportion vector θAmd , is univariate Gaussians {γAmd,k , δ2 }. The variational distribution of zJmj , zRmdr and zEmde are specified by free Multinomial with parameters φJmj,1:K , φRmdr,1:CK and φEmde,1:CK respectively. The variational distribution of ϕJk , ϕRk , ϕET k and ϕEC k are Dirithlet parameterized by λJk,1:|V J | , λRk,1:|V R| , λET k,1:|V E | and λEC k,1:|V E | , where |V J |, |V R| and |V E | are the lengths of vocabularies of job description, resume and interview assessment, respectively.

Actually, we find each term of ELBO in JLMIA is similar to some parts of ELBO in LDA model [Blei et al., 2003] or CTM model [Wang and Blei, 2011], except Eq[logp(θAmd |θJ , δ2 )], which can be computed by:

k mod K. Similar symbols are not described later for simplicity. And the Ψ(·) is Digamma function with derivative Ψ′ (·).

Then, we describe our EM-style algorithm. In E-step, we employ coordinate ascent approach to optimize all variational parameters. First, we optimize the ζmd in Equation 4:

Second, we optimize φJmj,1:K , φRmdr,1:CK and φEmde,1:CK for each coordinate. Assume that wJmj = c, wRmdr = t and wEmde = i, Imde = TI:

Third, we optimize γ Jm. Due to no analytic solution, we use Newton's method for each coordinate:

where function δyx = 1, only if x = y, otherwise, δyx = 0. Fourth, we optimize γAmd,1:CK . Due to no analytic solution, again, we use conjugate gradient algorithm with derivative:

Last, we optimize λJ , λR, λET and λEC . Their calculation process are similar, token λJk,c and λET k,i as examples:

In the M-step, we maximize the ELBO with respect to parameter α, similar to LDA, and regard the other hyperparameters in Ω as fixed parameters.