--- title: "Pre-DyGAE: Pre-training Enhanced Dynamic Graph Autoencoder for Occupational Skill Demand Forecasting" authors: - "Xi Chen" - "Chuan Qin" - "Zhigaoyuan Wang" - "Yihang Cheng" - "Chao Wang" - "Hengshu Zhu" - "Hui Xiong" doi: "10.24963/ijcai.2024/222" url: "https://ijcai.org/proceedings/2024/222" pdf: "https://ijcai.org/proceedings/2024/0222.pdf" venue: "Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence" year: 2024 pages: "2009-2017" track: "Main Track" keywords: - "Data Mining: DM: Mining spatial and/or temporal data" - "Data Mining: DM: Applications" --- # Pre-DyGAE: Pre-training Enhanced Dynamic Graph Autoencoder for Occupational Skill Demand Forecasting Xi Chen, Chuan Qin, Zhigaoyuan Wang, Yihang Cheng, Chao Wang, Hengshu Zhu, Hui Xiong *Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence*, pages 2009-2017. DOI: [10.24963/ijcai.2024/222](https://doi.org/10.24963/ijcai.2024/222) ## Abstract Occupational skill demand (OSD) forecasting seeks to predict dynamic skill demand specific to occupations, beneficial for employees and employers to grasp occupational nature and maintain a competitive edge in the rapidly evolving labor market. Although recent research has proposed data-driven techniques for forecasting skill demand, the focus has remained predominantly on overall trends rather than occupational granularity. In this paper, we propose a novel Pre-training Enhanced Dynamic Graph Autoencoder (Pre-DyGAE), forecasting skill demand from an occupational perspective. Specifically, we aggregate job descriptions (JDs) by occupation and segment them into several timestamps. Subsequently, in the initial timestamps, we pre-train a graph autoencoder (GAE), consisting of a semantically-aware cross-attention enhanced uncertainty-aware encoder and decoders for link prediction and edge regression to achieve graph reconstruction. In particular, we utilize contrastive learning on skill cooccurrence clusters to solve the data sparsity and a unified Tweedie and ranking loss for predicting the imbalanced distribution. Afterward, we incorporate an adaptive temporal encoding unit and a temporal shift module into GAE to achieve a dynamic GAE (DyGAE). Furthermore, we fine-tune the DyGAE with a two-stage optimization strategy and infer future representations. Extensive experiments on four real-world datasets validate the effectiveness of Pre-DyGAE compared with state-of-the-art baselines. ## Full text ## 1 Introduction With the rapid development of technology, the nature of occupation is continually changing [Bisello et al., 2022], e.g., the skill demand of the occupation. OSD forecasting plays an important role in research fields related to the labor market and economics, such as labor market evolution [Sus and Sylwestrzak, 2021], occupational mobility and development [Wang et al., 2021b; Wang et al., 2021a; Qin et al., 2019], and enterprise recruitment [Bilan et al., 2020; Qin et al., 2022]. In particular, dynamic OSD forecasting is helpful for the development of individual career construction and career planning [Guo et al., 2022a]. As for enterprises, forward-looking OSD is helpful for recruitment. ∗This work was done when Xi Chen was an intern at Career Science Lab, BOSS Zhipin supervised by Chuan Qin. †Corresponding Authors Figure 1: An illustration depicts the dynamic evolution of skill demands for NLP Algorithm Researchers (NLPer) and Algorithm Engineers (AE) from 2020 to 2023. The proportions of different colors on each donut chart represent the proportion of demand for different skills in the corresponding occupation. ![Image](https://ijcai.org/proceedings/2024/fulltext/222/image_000000_39a20924857505cc9d961095f150964feb6c6378f41bbed0ce38b096f619ce39.png) Previous studies on the analysis between occupation and skill mainly leverage the survey or interview-based methods, e.g., Occupational Information Network (O*NET), which is very labor-intensive and involves human biases [Felten et al., 2018]. Fortunately, recent years have witnessed the rapid development of online recruitment platforms, accumulating massive job postings across different occupations. This provides an unparalleled chance for implementing large-scale data-driven OSD forecasting. Along this line, although some research has been introduced for skill demand prediction [Wu et al., 2019; Li et al., 2020; Chao et al., 2024] and skill trend analysis [Mahdavimoghaddam et al., 2022], the skill demand forecasting at the occupational level is still largely under-explored. Predicting skill demand at an occupational granularity is essential as it acknowledges the varying trends in skill development across different occupations, highlighting that a singular prediction of overall skill trends falls short of meeting diverse demands. Indeed, there are three major challenges for OSD forecasting. Firstly, the trends of identical skills across various occupations exhibit diversification, while the demands for distinct skills in occupations are intricately interconnected [Langley et al., 2019]. This intricate interplay complicates the modeling in skill demand forecasting at the occupational level. Secondly, some pioneering skills often go unnoticed in enterprise recruitment job postings. This oversight makes it difficult to attain forward-looking forecasting on the skills that have not yet surfaced in occupations. Lastly, as shown in Figure 1 many skills occur in the long-tailed part, making accurate demand forecasting more challenging. To address the above challenges, we introduce PreDyGAE, pre-training enhanced dynamic graph autoencoder for OSD forecasting. To be specific, we aggregate JDs by occupation and partition them into several timestamps. Firstly, leveraging JDs from the initial timestamps, we pre-train the graph autoencoder (GAE), consisting of an encoder, a link prediction decoder, and an edge regression decoder. The encoder is enhanced by semantically-aware cross-attention and Graph Convolution Neural Networks (GCNs) for robust representations of both occupations and skills. Besides, GAE is pre-trained with a link prediction task enhanced by contrastive learning on skill clusters and an edge regression task to predict the distribution of skill demand using a unified Tweedie and ranking loss. Subsequently, in the following timestamps, we add an adaptive temporal encoding unit and a temporal shift operator into the pre-trained GAE to achieve a DyGAE. Furthermore, we fix the GAE and fine-tune the DyGAE with a two-stage optimization strategy. Finally, we obtain the evolved representations and achieve the forecasting of future OSD. Extensive real-world dataset experiments confirm Pre-DyGAE significantly outperforms state-of-theart baselines. The code is available online 1 . ## 2 RelatedWork ## 2.1 Occupational Skill Analysis Research on occupation and skill plays a crucial role in labor market analysis [Felten et al., 2018; Sun et al., 2021; Fang et al., 2023; Qin et al., 2024; Qin et al., 2023; Chen et al., 2021]. Recently, the labor market has undergone a rapid transformation driven by technological advancements. Several studies have examined the substitution of skills [Felten et al., 2018] and the evolution of occupations, while others have explored occupational mobility and market changes post-economic crisis [Bisello et al., 2022]. However, these studies often focus on localized phenomena of occupations and skills. The emergence of online recruitment platforms has enabled detailed statistical analysis of job postings. This data, collected from enterprise recruitment efforts, sheds light on the demand for talents and skills in the labor market [Zhu et al., 2016; Zhang et al., 2021; Guo et al., 2022b; 1 https://github.com/cx9941/Pre-DyGAE Chao et al., 2024], helping to predict skill demand for specific jobs [Wu et al., 2019]. Nonetheless, existing research tends to concentrate on skills or occupations separately, lacking a holistic analysis of both. Diverging from prior work, our study adopts an innovative perspective to explore the skill demand at the occupational level over time. ## 2.2 Dynamic Graph Learning Dynamic graph learning has garnered significant attention in recent research, which can model many real-world scenarios such as user-item interaction systems [Yu et al., 2023] and traffic networks [Zhao et al., 2020]. Many studies segment nodes based on temporal slices, analyzing static graphs within each slice and synthesizing temporal representations through methods like RNN or self-attention mechanisms, such as JODIE [Kumar et al., 2019], Dyrep [Trivedi et al., 2019], and CAWN [Wang et al., 2021d]. Specially, focus on spatio-temporal sequences, GConv [Seo et al., 2018], Dygrencoder [Taheri et al., 2019] and TGCN [Zhao et al., 2020] leveraged a combination of GCN and RNN to exploit spatial and temporal regularities. Besides, DCRNN [Li et al., 2018] captures the spatial dependency using random walks on the graph and A3TGCN [Bai et al., 2021] additionally adds the attention mechanism to assemble global temporal information. To learn the dynamic topology, EvolveGCN [Pareja et al., 2019] uses a recurrent model to evolve the GCN parameters, and AGCRN [Bai et al., 2020] captures fine-grained graph correlations in time series automatically. Some transformer-based methods, TCL [Wang et al., 2021c] and DyGFormer [Yu et al., 2023], learning from node historical first-hop interactions, efficiently benefit from long histories. Additionally, several studies have introduced temporal encoding mechanisms to effectively capture timing patterns in dynamic graphs, such as TGN [Rossi et al., 2020], TCL [Wang et al., 2021c] and GraphMixer [Cong et al., 2023]. ## 3 Preliminaries In this section, we define the key concepts integral to our study, present the problem formulation of OSD forecasting, and give an overview of our proposed Pre-DyGAE. Definition 1 (Occupational Skill Demand). At timestamp t, each occupation o in the occupation set O contains collected JDs P to . Each JD p encompasses a set of skills from the skill set S. Along this line, we define the OSD at timestamp t as: Rto,s = ∑p∈P to 1(s ∈ p)/|P to |, where Rt ∈ R|O|×|S| and | · | denotes the size of a set. Definition 2 (Occupational Skill Demand Graph). At timestamp t, OSD graph can be represented as an undirected bipartite graph G t = (V, E t , At ), where V = O ∪ S, E t is a set of edges between occupations and skills, and At ∈ R|V|×|V| is the weighted adjacency matrix. The edge eto,s between o and s is defined as {eto,s ∈ E t |o ∈ O, s ∈ S, and Rto,s ̸= 0}. The weighted adjacency matrix is defined as Ato,s = Ats,o = Rto,s if eto,s ∈ E t , otherwise 0. Problem 1 (Occupational Skill Demand Forecasting). Given an OSD graph sequence (G 1 , G 2 , ..., G T ), our objective becomes predicting the OSD graph G T+1 for the upcoming frame T + 1. Specifically, we validate the effectiveness of dynamic graph forecasting via two common tasks in dynamic graph learning: (i) dynamic link prediction, which estimates the existence of eT+1 o,s between o and s; (ii) dynamic edge regression, which infers the weighted adjacency matrix {AT +1 o,s |eT+1 o,s ∈ E T+1 }. Figure 2: An illustration of our proposed Pre-DyGAE. ![Image](https://ijcai.org/proceedings/2024/fulltext/222/image_000001_0861c1bd9050533fbe6132f95560cfcd76b8ffb5ac3a2942627f09d06e335686.png) Solution overview In this paper, we introduce Pre-DyGAE, a two-stage framework comprising pre-training and finetuning phases. An overview of our Pre-DyGAE is graphically depicted in Fig. 2. Considering the tradeoff between utilizing data integrity knowledge and capturing evolutionary patterns, we aggregate JDs from initial timestamps (P 1 , ..., P ∆t ) into P 0 to construct the initial graph G 0 for pre-training and finetune on the following OSD graphs (G∆t+1 , ..., G T ) for dynamic representations. In the pre-training stage, we employ a GAE to pre-train on G 0 , aiming to obtain robust representations. In particular, our encoder incorporates a semanticallyaware cross-attention mechanism, utilizing textual information stored in JDs to capture intricate relationships between occupations and skills. Additionally, GCNs are applied to model uncertainty. To guide comprehensive graph reconstruction, we introduce a contrastive loss on skill cooccurrence clusters for sparse data and a unified Tweedie loss and ranking loss for imbalanced prediction. In the fine-tuning stage, we enhance GAE with an adaptive temporal encoding unit and a temporal shift module for DyGAE. This dynamic model is fine-tuned using a two-stage optimization strategy, allowing it to evolve representations from timestamp ∆t + 1 to T. Finally, our framework enables the inference of future representations and the prediction of future OSD. ## 4 Pre-training Enhanced Graph Autoencoder In this section, our objective is to devise a pre-training model adept at capturing intricate relationships between occupations and skills to yield robust representations. The architecture consists of an encoder and two decoders. In the multi-task training framework, we jointly pre-train the model in a multitask framework to guide comprehensive graph reconstruction. ## 4.1 Encoder To fully leverage the semantic content of the data, we initial the primary embeddings of occupation o and skill s as Eo and Es with their corresponding JDs: where P 0 s represents {p|s ∈ p, p ∈ P 0 }, Bert(p) signifies the embedding of the p derived from a pretrained BERT model. Semantically-Aware Cross-Attention Besides, a crossattention mechanism is contributed to further harness the intricacies tied to skills and their representations in occupations. Let's represent the matrices of occupations and skills embeddings as EO and ES . We obtain the semanticallyaware cross-attention enhanced occupation and skill embeddings Z O as: where W Q, W K , and W V are trainable parameters and dK is the dimension of EK . Furthermore, we introduce a semantically-aware crossattention mechanism to further harness the semantic intricacies tied to skills and their representations in occupations. The mechanism can be reformulated as: where the bias Bbias ∈ R|O|×|S| is learnable, pivotal for instilling deep semantic insights into the representations of occupations and skills. With a cosine similarity function Sim(x, x′ ) = x·x′ ∥x∥·∥x′ ∥ , the bias Bbias is defined as: where FCi(·) is a linear fully connected layer. Uncertainy-Aware Modeling Furthermore, we utilize structural data to steer the integration of occupation and skill representations by GCN. The aggregation process is: where Z is the concatenation of Z O and Z S , f is a activation function, I is the identity matrix, W 0 is the trainable node transformation matrix, and D is the degree matrix of A0 + I. Considering the inherent uncertainty arising from multiple job postings associated with individual occupations, a deterministic representation may risk overfitting. To address this, we model the latent variable H0 to adhere to a Gaussian distribution for deriving more robust representations: where N(µ, σ) is the probability density function of the Gaussian distribution with mean µ and variance σ2 , and W µ and W σ are two trainable node transformation matrices. ## 4.2 Decoders Upon obtaining the representations, the decoders aim to reconstruct the input graph G 0 in two aspects, link prediction to predict the relationships between occupations and skills and edge regression for predicting the specific values. Link Prediction Decoder We model the likelihood P E of associating skill s with occupation o as follows: where sigmoid(·) is an activation function for calculating the probability, ⊙ denotes the element-wise multiplication operation, and MeanPooling(·) calculates the average value along the last dimension in the matrix. Edge Regression Decoder Extending the capabilities of GAE, we utilize a linear layer with an activation function to estimate R0o,s :ˆR0o,s = softplus(FC3(H0 o ⊙H0 s )), (8) where softplus(·) is an activation function as a smooth approximation of the rectifier linear unit activation function. Through this approach, GAE decodes H0 into distinct spaces for predicting both relationships between occupations and skills and OSD, facilitating mutual learning that contributes to the accomplishment of diverse tasks. The prediction within a static time slice also contributes to subsequent dynamic forecasting. ## 4.3 Pre-training Objectives The pre-training objective for graph reconstruction involves training the primary tasks of link prediction and edge regression, incorporating two auxiliary tasks. The loss function is composed of a negative log-likelihood loss (L0nll ) and a collaborative-aware contrastive loss (L0con ) for link prediction with sparse data challenge, as well as a Tweedie loss (L0rg ) and a ranking loss (L0rank ) for imbalanced regression. Negative Log-Likelihood Loss To achieve the accurate reconstruction of the graph edges E 0 , the negative loglikelihood loss Lnll is minimized: where y0 o,s = 1 if e0o,s ∈ E and y0 o,s = 0 otherwise. Collaborative-Aware Constrative Loss To solve the challenges of predicting relationships between occupations and less frequent skills, we further introduce a collaborativeaware contrastive loss. Specifically, by evaluating skill cooccurrence in P 0 , we apply a spectral clustering algorithm to divide the skills into C clusters and Ck is the k-th cluster contains skills with the closest co-occurrence degree. This approach tightens the representation of skills within a single cluster and thus addresses data sparsity, which is defined as: where γ is the temperature coefficient. Tweedie Loss Considering the OSD follows long-tailed distribution, directly employing regression loss functions i.e., MAE, may hinder the model's ability to effectively capture the imbalances present in both the head and tail data in training. Here, we employ the Tweedie loss to address this issue. where η ∈ (1, 2) is a hyperparameter defining the power relation between Tweedie distribution mean and variance, and E neg is the set of randomly sampled negative edges. RankingLoss Furthermore, the diverse skill demands of an occupation underscore the unique roles each skill plays. For instance, some foundational skills might be prevalent across different occupations, whereas the low occurrences could indicate some emerging demand. Therefore, we devise a ranking loss, as a pairwise loss [Wang et al., 2023], to better capture the distinctions between skills. where ϵ is the margin value and ρ0o,s is the rank. Specifically, if R0o,s ≥ 0.1, ρ0o,s = 0; if 0.1 > R0o,s > 0.01, ρ0o,s = 1; otherwise, ρ0o,s = 2. Finally, the pre-training loss function L0GAE is defined: where L0kl = 12 ∑i,j (µ0ij 2 + (σ0 ij )2 - 2 log(σ0 ij )-1) is a KL loss to minimize the discrepancy between the posterior and prior distributions of the latent variables, and λnll, λcon, λrg and λrank are the hyperparameters to balance the effects of different modules. ## 5 Dynamic Graph Autoencoder In this section, we incorporate an adaptive temporal encoding unit and a temporal shift module into GAE . Besides, we fix GAE and apply parameter efficient fine-tuning on DyGAE with a two-step optimization strategy. Finally, we can evolve the dynamic representations underlying a dynamic graph sequence (G∆t+1 , ..., G T ) and achieve OSD forecasting. ## 5.1 Temporal Feature Learning Specifically, we design an adaptive temporal encoding unit to facilitate the learning of temporal representations. Additionally, we model the Gaussian distribution of temporal features, enhancing our ability to capture and represent dynamic uncertainties. Notably, we introduce a temporal shift module to effectively explore the temporal evolution of features, thereby achieving dynamic representations. Adaptive Temporal Encoding Unit At timestamp t, the relationship between occupations and skills hinges on the prevailing graph structure G t and the stable graph structure G 0 . To harness the wealth of structural information embedded within the relationship between occupations and skills, we employ the dual-view GCN propagation method, integrating it with the specified adjacency matrix At and a learnable graph Aa initialized with A0 . With an initial learnable embedding Lt ∈ R(|O|+|S|)×d where d is the feature dimension, the dual-view GCN propagation layer is devised as: where M t ∈ R(|O|+|S|)×d is a trainable parameter, F t is the structure enhanced temporal hidden states, and [·; ·] denotes a concanate operation. Furthermore, We encode the temporal feature as a Gaussian distribution: Dynamic Uncertainty-Aware Modeling While the stable representations capture the essential characteristics of occupations and skills, learned temporal characteristics enrich the representation with time-varying nuances. Leveraging the superposition property of normal distributions, we model the dynamic representations by incorporating the temporal features into the stable initial representation: This strategy allows us to capture intricate temporal patterns and facilitate robust representation learning. Temporal Shift Module Following the assumption that variance does not depend on time [Gourru et al., 2022], we infer ˆ δt+1 = ∑ δt /T. To explore the evolution of ϕt , we first decompose ϕt into a stable term αt = FC8(ϕt ) and a trend term βt = FC9(ϕt ). To predict ϕt+1 , we add an MLP to evolve the trend term as ωt = MLP(βt ). Then we forecast the ϕt+1 as αt + ωt . ## 5.2 Parameter Efficient Fine-Tuning Leveraging the pre-trained GAE, we initially focus on optimizing the temporal features in each timestamp. Following this, we delve into exploring the evolution pattern. This two-step optimization strategy allows for a comprehensive exploitation of the pre-trained model's advantages and facilitates the accurate capture of evolution characteristics. Adaptive Temporal Encoding Unit Optimization For dynamic OSD, trend changes usually include four situations: emergence, disappearance, growth, and decline. To explicitly allow dynamic representations to perceive changes in trends, for each o and s, we additionally add a linear layer with an activation function to map temporal features into four trend labels τo,s as above. We estimate the likelihood of the trend label between o and s at t as P τ (o, s, t) = sigmoid(FC7(Hto ⊙Ht s )). The trendaware loss is defined as: In each timestamp t ∈ [∆t + 1, T], we fix the GAE and fine-tune the adaptive temporal encoding unit via Lttem = λtrendLttrend + LtGAE for the temporal feature distribution (ϕt , δt ), where λtrend is the hyperparameter, LtGAE is the pre-training loss in Eq.13 with Ht as the input of decoders and G t as the target graph. Temporal Shift Module Optimization Based on the temporal feature sequence ((ϕ∆t+1 , δ∆t+1 ), ..., (ϕT , δT )), we intend to predict the temporal feature in the next timestamp (ϕT+1 , δT+1 ). Firstly we reconstruct the {ϕt |t ∈ [∆t+1, T ]} with a reconstruction loss Lre: where MSE(x, x′ ) is a function to compute the mean squared error between x and x′ . Besides, to achieve next-step prediction, we introduce Lne as: Additionally, we formulate the loss λcom to keep the stable terms similar and distinguish trend terms. where Cos(x, x′ ) is a function, defined as Cos(x, x′ ) = 1m ∑mi=1 xi ·x′i ∥xi∥·∥x′i ∥ , to compute the average cosine similarity between x and x′ and m is the number of row in x. During the training process, the three loss functions are combined as Lshift = λreLre+λneLne+λcomLcom where λre, λne, and λcom are the hyperparameters. ## 5.3 Inference Procedure After the fine-tuning, we can infer the next step temporal features as ˆ ϕT+1 = αT + ωT . Combining with the pretrained representation distributions, we forecast the representation distributions of occupations and skills ˆHT+1 ∼ N(µ0 + ˆ ϕT+1 , σ0 + ˆ δT+1 ). Furthermore, the OSD graph ˆ G T+1 can be inferred with the inferred likelihood ˆ P E (eT +1 o,s ∈ E T+1 ) = sigmoid(MeanPooling( ˆHT+1 o ⊙ ˆHT+1 s )) and the inferred edge weight ˆ AT +1 o,s = softplus(FC3( ˆHT+1 o ⊙ ˆHT+1 s )). Table 1: OSD forecasting performance comparisons across four datasets. Each result was derived from four repetitive experiments. The best results are highlighted in bold, while the second-best results are underscored. Asterisks indicate statistical significance: * denotes performance improvement at the 0.05 level, and ** denotes a significant improvement at the 0.01 level, as determined by the paired t-test. | Data | Metric | DCRNN | DyGrEncoder | AGCRN | A3TGCN | TGAT | CAWN | DyGFormer | CHGH | Pre-DyGAE | |--------|-----------------------------------------|----------------------------------------------------------------|----------------------------------------------------------------|----------------------------------------------------------------|---------------------------------------------------------------|----------------------------------------------------------------------|---------------------------------------------------------------------|-----------------------------------------------------------------|----------------------------------------------------------------|---------------------------------------------------------------------------| | Dai | AUC%↑ Hits@1%↑ MRR%↑ EGM%↓ MAE%↓ RMSE%↓ | 66.42±0.25 11.99±0.44 16.62±1.01 0.09±0.00 0.74±0.02 3.58±0.06 | 72.84±2.89 9.52±0.88 13.60±2.44 0.23±0.05 0.93±0.05 2.97±0.10 | 64.05±0.91 12.59±0.98 17.24±2.11 0.09±0.00 0.89±0.02 3.31±0.24 | 61.91±11.11 1.93±1.91 4.00±6.52 0.15±0.08 1.10±0.10 4.17±0.13 | 90.47±0.09 38.62±7.50 41.29±3.29 0.07±0.00 1.16±0.00 5.17±0.00 | 91.17±0.43 39.30±1.90 43.06±0.05 1.12±0.55 1.83±0.47 5.05±0.01 | 90.83±0.38 29.05±0.85 41.16±0.21 1.17±0.97 2.02±0.80 5.16±0.02 | 70.76±1.18 21.90±1.56 25.00±1.74 0.11±0.06 0.35±0.01 1.14±0.02 | 96.96±0.04** 42.97±0.33** 51.92±0.19** 0.06±0.00** 0.30±0.04* 0.66±0.27* | | Fin | AUC%↑ Hits@1%↑ MRR%↑ EGM%↓ MAE%↓ RMSE%↓ | 67.58±0.49 15.08±0.45 20.52±1.26 0.12±0.00 0.67±0.03 3.00±0.14 | 70.15±2.02 15.07±1.85 19.05±5.44 0.17±0.01 0.88±0.01 2.74±0.03 | 65.87±0.95 14.99±0.85 20.47±1.38 0.14±0.01 1.02±0.03 3.45±0.16 | 68.39±2.06 5.11±2.86 8.84±6.62 0.39±0.08 1.58±0.11 4.28±0.23 | 92.17±0.20 46.14±2.66 48.91±0.64 0.13±0.00 1.54±0.00 6.12±0.00 | 92.26±0.03 45.20±5.16 48.65±0.72 16.18±10.27 16.70±10.24 17.19±9.87 | 92.26±0.17 28.25±20.81 45.09±3.85 1.29±1.06 2.71±1.17 6.38±0.26 | 71.51±2.54 23.37±3.84 26.73±4.17 0.18±0.04 0.49±0.05 2.31±0.01 | 96.16±0.09** 48.59±0.34* 57.89±0.37** 0.09±0.02** 0.34±0.06** 0.47±0.24** | | IT | AUC%↑ Hits@1%↑ MRR%↑ EGM%↓ MAE%↓ RMSE%↓ | 68.31±0.64 14.03±0.77 19.63±1.53 0.17±0.01 0.93±0.04 3.36±0.14 | 70.32±1.77 14.34±0.83 20.48±2.21 0.27±0.02 1.20±0.05 3.92±0.64 | 63.86±0.79 12.45±0.44 17.51±1.11 0.21±0.01 1.34±0.03 4.11±0.10 | 64.57±1.25 2.88±1.27 7.52±3.23 0.46±0.10 1.80±0.06 4.87±0.17 | 89.13±0.25 40.78±2.46 46.99±1.05 33.42±45.23 34.24±44.96 36.33±42.38 | 89.79±0.36 36.50±2.04 46.18±1.44 7.14±0.93 7.79±0.91 8.95±0.71 | 89.19±0.14 38.24±9.30 45.58±3.02 0.53±0.08 1.95±0.05 6.53±0.01 | 72.49±2.55 26.27±4.34 29.86±4.58 0.21±0.04 0.52±0.04 1.39±0.02 | 96.04±0.06** 44.34±0.67** 52.59±0.45** 0.11±0.02** 0.35±0.07** 0.73±0.34* | | Man | AUC%↑ Hits@1%↑ MRR%↑ EGM%↓ MAE%↓ RMSE%↓ | 68.20±0.89 12.87±0.99 17.12±1.73 0.12±0.00 0.66±0.01 2.42±0.05 | 72.73±3.24 13.74±1.62 18.25±3.90 0.23±0.06 0.89±0.07 2.64±0.14 | 65.45±1.39 14.04±1.71 18.40±4.06 0.14±0.01 0.94±0.04 3.11±0.17 | 69.75±1.73 3.13±1.26 6.86±2.44 0.32±0.05 1.29±0.03 3.96±0.08 | 89.18±0.24 35.07±2.21 45.86±0.85 0.12±0.00 1.31±0.00 5.18±0.00 | 90.12±0.11 41.80±1.51 49.03±1.86 2.58±1.22 3.20±1.14 5.35±0.42 | 90.47±0.28 28.54±1.90 44.86±0.80 7.69±9.58 8.51±9.22 10.29±7.23 | 71.14±1.76 22.47±2.39 25.69±2.66 0.16±0.03 0.40±0.03 1.27±0.02 | 96.92±0.07** 45.62±0.63** 53.85±0.46* 0.07±0.02** 0.30±0.06** 0.68±0.18** | ## 6 Experiments ## 6.1 Experimental Settings Datasets The datasets used in our experiments were collected from the public information of one of the largest online recruitment platforms. We gathered JDs spanning a diverse range of occupations, covering the period from January 2020 to December 2023. Along this line, we constructed largescale datasets from four industries, i.e., the Daily dataset (Dai), Finance dataset (Fin), IT dataset (IT), and Manufacturing dataset (Man). As mentioned above, we used half-year intervals as the basic time step, dividing each dataset into 8 timestamps. In our experiment setup, we selected ∆t = 4, which entailed using JDs from the first 4 timestamps for pretraining and the subsequent 3 timestamps for dynamic temporal feature learning. Besides, more parameter experiments are in the Appendix, and we choose λnll, λcon, λrg, λrank, λtrend, λre, λne, and λcom as 1, 0.05, 100, 0.1, 1, 1, 1, and 1. Meanwhile, we partitioned the JDs from the first month of the final timestamp to create a validation set, while the remaining five months' JDs were used as a test set. Detailed statistics and divisions of our datasets are provided in the Appendix. Evaluation Metrics We evaluated the effectiveness of our approach via dynamic link prediction and edge regression tasks. Therefore, we employed several link prediction metrics, including AUC [Zha et al., 2023], MRR, and Hits@1 [Wang et al., 2022], and regression metrics, including EGM [Yang et al., 2021], MAE [Zhu et al., 2016], and RMSE [Chang et al., 2014]. In particular, EGM was used to measure the heavy-tailed nature of our data distributions. Baselines We compared Pre-DyGAEwith a state-of-the-art method, i.e., CHGH [Chao et al., 2024]. In addition, we included dynamic graph learning baselines, including DCRNN [Li et al., 2018], DyGrEncoder [Taheri et al., 2019], AGCRN [Bai et al., 2020], A3TGCN [Bai et al., 2021], TGAT [Xu et al., 2020], CAWN [Wang et al., 2021d], and DyGFormer [Yu et al., 2023]. To maintain fairness in comparison, each baseline was trained with data within the former 7 timestamps. Besides, we also compared some traditional methods, but due to not achieving competitive results and page limitations, we have put their experimental results in the Appendix. ## 6.2 Experimental Results Performance Comparison Table 1 illustrates the comprehensive performance evaluation of Pre-DyGAE and baselines across four datasets. According to the results, there are several observations. Firstly, our proposed model has significant improvements over all baselines in all datasets, thus demonstrating the effectiveness of our framework in OSD forecasting. Specifically, the AUC metric showcases a remarkable performance of 96%. This confirms its capability to precisely predict future skills across a variety of occupations, a critical aspect for subsequent edge regression. Additionally, our approach demonstrates proficiency by improving the average RMSE in four datasets from 1.527% to 0.635%, an increase of 58% compared with the best-performing baseline, providing precise insights into the distribution of demand, and highlighting its capability to forecast future skill demand accurately. Moreover, the improvement in the EGM directly signifies the competence of our model in predicting unbalanced distribution. Thirdly, baseline models often show imbalanced results in both link prediction and regression metrics, indicating the complexity of integrating these two tasks. In contrast, our model excels in all metrics, demonstrating Pre-DyGAE's ability to improve performance on both tasks. Finally, it is noteworthy that RNN-based methods such as DCRNN and CAWN outperform Transformer-based methods such as DyGFormer in these datasets. This is attributed to the Transformer's primary focus on addressing long-horizon autocorrelation while overlooking variable trends, aligning with the unique characteristics of our forecasting problem. Table 2: OSD graph completion performance on Fin dataset. We introduced four variants of our GAE: "w/o rank", "w/o con", "w/o bias", and "L1". The first three variants denote our model without the Lrank (Eq.12), Lcon (Eq.10), and semantically-aware crossattention, respectively. The last variant indicates the substitution of the Tweedie Loss with L1 loss in Eq.11. | Metrics | AUC%↑ | MRR%↑ | EGM%↓ | MAE%↓ | |-----------|------------|------------|-----------|-----------| | NGCF | 70.15±0.09 | 14.23±0.10 | 0.03±0.00 | 0.51±0.01 | | LightGCN | 76.45±1.10 | 15.74±1.00 | 0.34±0.05 | 0.86±0.08 | | MultVAE | 77.22±0.00 | 12.08±0.00 | 0.02±0.00 | 0.51±0.00 | | w/o con | 93.38±0.06 | 19.46±0.25 | 0.09±0.04 | 0.22±0.08 | | w/o rank | 94.52±0.03 | 21.19±0.33 | 0.09±0.07 | 0.27±0.09 | | w/o bias | 94.37±0.11 | 21.16±0.31 | 1.09±0.62 | 1.49±0.73 | | L1 | 94.44±0.15 | 21.54±0.21 | 0.06±0.03 | 0.12±0.02 | | GAE | 95.56±0.06 | 22.64±0.27 | 0.02±0.02 | 0.09±0.05 | GAE Performance on OSD Graph Completion We conducted a validation experiment to assess the robustness of GAE's learned representations by predicting missing skill demand within the same time interval. JDs from the initial 4 timestamps were randomly split into training and test sets at a ratio of 50%, with corresponding occupational skill demand in triplet form (o, s,R0o,s ). For comparison, we selected NGCF, LightGCN, and MultVAE as the baselines [He et al., 2020]. The results in Table 2 show that GAE improved AUC and MRR compared to all baselines, indicating effective representations of occupations and skills. Specifically, "w/o bias" significantly impacts EGM and RMSE, demonstrating the importance of the semantically-aware cross attention. The ranking loss distinguishes feature spaces, and contrastive learning on skill clusters enhances metrics, guiding recognition of sparse skills. Additionally, comparing L1 loss and the Tweedie loss suggests that the latter is more suitable for predicting imbalanced distributions. Comprehensive results across all datasets are detailed in the Appendix. Ablation Studies We performed ablation studies to assess different modules in Pre-DyGAE, as depicted in Fig. 3. Results show Pre-DyGAE's performance compared to various backbones, highlighting GAE's robust representation and its ability to enhance capturing temporal features. Ablation experiments on the temporal module affirm our model's capability to capture demand distribution shifts caused by subsequent temporal changes based on the backbone representation. Parameter Experiment We have carefully considered the balance between leveraging data integrity knowledge and capturing evolutionary patterns. As shown in Fig. 4, the trend of rising first and then falling suggests the advantages of stable representations for capturing subsequent changes and the necessity of preserving some future time slices for mining evolutionary patterns. The consistency in the number of time frames across different metrics in parameter experiments further supports the effectiveness of our pre-training and finetuning framework. More details can be seen in the Appendix. Figure 3: OSD Forecasting Ablation Study on Fin dataset. The red histogram examines the impact of backbone ablation on OSD forecasting, while the green histogram assesses the adaptive temporal encoding unit and temporal shift module's effectiveness. Besides, "w/o pre-train" denotes GAE without pretraining directly applied to learn the temporal features. And "w/o adaptive" and "w/o trendweight" denote the adaptive temporal encoding unit without the learnable graphWa (Eq.14) and trend loss Ltrend, respectively. ![Image](https://ijcai.org/proceedings/2024/fulltext/222/image_000002_c6df0566ed7da009397fb692113c57ae2de5dd30ccd44c39aa76020baef4cd4e.png) Figure 4: The influence of ∆t on Fin dataset. We varied timestamps for aggregating JDs to construct G0 and pre-trained GAE with ∆t ranging from 1 to 6. Pre-DyGAE performed best with ∆t = 4. ![Image](https://ijcai.org/proceedings/2024/fulltext/222/image_000003_f883aff5f12124de554ddd2631b45595debe447b5d2fa2b3f1ca52eff2d7346d.png) ## 7 Conclusion In this paper, we addressed the critical task of forecasting skill demand in specific occupations, a key aspect in the dynamics of labor markets and economics. Specifically, we introduced Pre-DyGAE, a novel approach involving pre-training on initial timestamps, followed by parameterefficient fine-tuning in subsequent timestamps. This approach leveraged the overall interaction between occupations and skills over an extended period, enabling the discernment of stable associations between them. The pre-training of GAE incorporated a semantically-aware cross-attention enhanced uncertainty-aware encoder and a multi-task pretraining framework. These components proved effective in establishing fundamental representations. Based on the pretrained GAE, DyGAE excelled in capturing dynamic temporal features through a two-stage optimization process involving adaptive temporal encoding units and a temporal shift module. Extensive experiments on four real-world datasets have demonstrated the effectiveness of our Pre-DyGAE. ## Acknowledgements This work was partially supported by National Natural Science Foundation of China (Grant No.92370204), Guangzhou-HKUST(GZ) Joint Funding Program(Grant No.2023A03J0008), the National Key Research and Development Program of China (No. 2023YFF0725001), Education Bureau of Guangzhou Municipality, Guangdong Science and Technology Department, and Project funded by China Postdoctoral Science Foundation (Grant No.2023M730785). ## References 1. [Bai et al., 2020] Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. Advances in neural infor- mation processing systems, 2020. 2. [Bai et al., 2021] Jiandong Bai, Jiawei Zhu, Yujiao Song, Ling Zhao, Zhixiang Hou, Ronghua Du, and Haifeng Li. A3t-gcn: Attention temporal graph convolutional network for traffic forecasting. ISPRS International Journal of Geo-Information, 2021. 3. [Bilan et al., 2020] Yuriy Bilan, Halyna Mishchuk, Iryna Roshchyk, and Olena Joshi. Hiring and retaining skilled employees in smes: problems in human resource practices and links with organizational success. Business: Theory and Practice, 2020. 4. [Bisello et al., 2022] Martina Bisello, Vincenzo Maccarrone, and Enrique Fernández-Macías. Occupational mobility, employment transitions and job quality in europe: The impact of the great recession. Economic and Industrial Democracy, 2022. 5. [Chang et al., 2014] Biao Chang, Hengshu Zhu, Yong Ge, Enhong Chen, Hui Xiong, and Chang Tan. Predicting the popularity of online serials with autoregressive models. In CIKM, 2014. 6. [Chao et al., 2024] Wenshuo Chao, Zhaopeng Qiu, Likang Wu, Zhuoning Guo, Zhi Zheng, Hengshu Zhu, and Hao Liu. A cross-view hierarchical graph learning hypernet- work for skill demand-supply joint prediction. In AAAI, 2024. 7. [Chen et al., 2021] Liyi Chen, Chuan Qin, Ying Sun, Xin Song, Tong Xu, Hengshu Zhu, and Hui Xiong. Collaboration-aware hybrid learning for knowledge devel- opment prediction. In WWW, 2021. 8. [Cong et al., 2023] Weilin Cong, Si Zhang, Jian Kang, Baichuan Yuan, Hao Wu, Xin Zhou, Hanghang Tong, and Mehrdad Mahdavi. Do we really need complicated model architectures for temporal networks? In ICLR, 2023. 9. [Fang et al., 2023] Chuyu Fang, Chuan Qin, Qi Zhang, Kaichun Yao, Jingshuai Zhang, Hengshu Zhu, Fuzhen Zhuang, and Hui Xiong. Recruitpro: A pretrained lan- guage model with skill-aware prompt learning for intelli- gent recruitment. In KDD, 2023. 10. [Felten et al., 2018] Edward W. Felten, Manav Raj, and Robert Seamans. A method to link advances in artificial intelligence to occupational abilities. AEA Papers and Pro- ceedings, 2018. 11. [Gourru et al., 2022] Antoine Gourru, Julien Velcin, Christophe Gravier, and Julien Jacques. Dynamic gaus- sian embedding of authors. In Proceedings of the ACM Web Conference 2022, New York, NY, USA, 2022. 12. [Guo et al., 2022a] Pengzhan Guo, Keli Xiao, Zeyang Ye, Hengshu Zhu, and Wei Zhu. Intelligent career planning via stochastic subsampling reinforcement learning. Scien- tific Reports, 2022. 13. [Guo et al., 2022b] Zhuoning Guo, Hao Liu, Le Zhang, Qi Zhang, Hengshu Zhu, and Hui Xiong. Talent demand- supply joint prediction with dynamic heterogeneous graph enhanced meta-learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022. 14. [He et al., 2020] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, YongDong Zhang, and Meng Wang. Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2020. 15. [Kumar et al., 2019] Srijan Kumar, Xikun Zhang, and Jure Leskovec. Predicting dynamic embedding trajectory in temporal interaction networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019. 16. [Langley et al., 2019] Ann Langley, Kajsa Lindberg, Bjørn Erik Mørk, Davide Nicolini, Elena Raviola, and Lars Walter. Boundary work among groups, occupations, and organizations: From cartography to process. Academy of Management Annals, 2019. 17. [Li et al., 2018] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting. In International Conference on Learning Representations (ICLR '18), 2018. 18. [Li et al., 2020] Shan Li, Baoxu Shi, Jaewon Yang, Ji Yan, Shuai Wang, Fei Chen, and Qi He. Deep job understand- ing at linkedin. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2020. 19. [Mahdavimoghaddam et al., 2022] Jalehsadat Mahdavi- moghaddam, Ayush Bahuguna, and Ebrahim Bagheri. Exploring the utility of social content for understanding future in-demand skills. Proc. ACM Hum.-Comput. Interact., 2022. 20. [Pareja et al., 2019] Aldo Pareja, Giacomo Domeniconi, Jie Chen, Tengfei Ma, Toyotaro Suzumura, Hiroki Kaneza- shi, Tim Kaler, Tao B. Schardl, and Charles E. Leiserson. EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs, 2019. 21. [Qin et al., 2019] Chuan Qin, Hengshu Zhu, Chen Zhu, Tong Xu, Fuzhen Zhuang, Chao Ma, Jingshuai Zhang, and Hui Xiong. Duerquiz: A personalized question recommender system for intelligent job interview. In Proceedings of the 25th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, pages 2165-2173, 2019. 22. [Qin et al., 2022] Chuan Qin, Kaichun Yao, Hengshu Zhu, Tong Xu, Dazhong Shen, Enhong Chen, and Hui Xiong. Towards automatic job description generation with capability-aware neural networks. IEEE Transactions on Knowledge and Data Engineering, 2022. 23. [Qin et al., 2023] Chuan Qin, Hengshu Zhu, Dazhong Shen, Ying Sun, Kaichun Yao, Peng Wang, and Hui Xiong. Au- tomatic skill-oriented question generation and recommen- dation for intelligent job interviews. ACM Transactions on Information Systems, 2023. 24. [Qin et al., 2024] Chuan Qin, Le Zhang, Yihang Cheng, Rui Zha, Dazhong Shen, Qi Zhang, Xi Chen, Ying Sun, Chen Zhu, Hengshu Zhu, and Hui Xiong. A comprehensive sur- vey of artificial intelligence techniques for talent analytics. arXiv preprint arXiv:2307.03195v2, 2024. 25. [Rossi et al., 2020] Emanuele Rossi, Ben Chamberlain, Fab- rizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. Temporal graph networks for deep learning on dynamic graphs. In ICML 2020 Workshop on Graph Rep- resentation Learning, 2020. 26. [Seo et al., 2018] Youngjoo Seo, Michaël Defferrard, Pierre Vandergheynst, and Xavier Bresson. Structured Sequence Modeling with Graph Convolutional Recurrent Networks. In Neural Information Processing, 2018. 27. [Sun et al., 2021] Ying Sun, Fuzhen Zhuang, Hengshu Zhu, Qi Zhang, Qing He, and Hui Xiong. Market-oriented job skill valuation with cooperative composition neural net- work. Nature communications, 2021. 28. [Sus and Sylwestrzak, 2021] Aleksandra Sus and Bartosz Sylwestrzak. Evolution of the labor market and com- petency requirements in industry 4.0 versus the covid-19 pandemic. EUROPEAN RESEARCH STUDIES JOUR- NAL, 02 2021. 29. [Taheri et al., 2019] Aynaz Taheri, Kevin Gimpel, and Tanya Berger-Wolf. Learning to represent the evolution of dy- namic graphs with recurrent models. In Companion Pro- ceedings of The 2019 World Wide Web Conference, 2019. 30. [Trivedi et al., 2019] Rakshit Trivedi, Mehrdad Farajtabar, Prasenjeet Biswal, and Hongyuan Zha. Dyrep: Learn- ing representations over dynamic graphs. In International Conference on Learning Representations, 2019. 31. [Wang et al., 2021a] Chao Wang, Hengshu Zhu, Qiming Hao, Keli Xiao, and Hui Xiong. Variable interval time sequence modeling for career trajectory prediction: Deep collaborative perspective. In Proceedings of the Web Con- ference 2021, 2021. 32. [Wang et al., 2021b] Chao Wang, Hengshu Zhu, Peng Wang, Chen Zhu, Xi Zhang, Enhong Chen, and Hui Xiong. Per- sonalized and explainable employee training course rec- ommendations: A bayesian variational approach. ACM Transactions on Information Systems (TOIS), 2021. 33. [Wang et al., 2021c] Lu Wang, Xiaofu Chang, Shuang Li, Yunfei Chu, Hui Li, Wei Zhang, Xiaofeng He, Le Song, Jingren Zhou, and Hongxia Yang. TCL: transformer-based dynamic graph modelling via contrastive learning. CoRR, 2021. 34. [Wang et al., 2021d] Yanbang Wang, Yen-Yu Chang, Yunyu Liu, Jure Leskovec, and Pan Li. Inductive representa- tion learning in temporal networks via causal anonymous walks. In International Conference on Learning Represen- tations, 2021. 35. [Wang et al., 2022] Xintao Wang, Qianyu He, Jiaqing Liang, and Yanghua Xiao. Language models as knowledge em- beddings. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, 2022. 36. [Wang et al., 2023] Chao Wang, Hengshu Zhu, Chen Zhu, Chuan Qin, Enhong Chen, and Hui Xiong. Setrank: A setwise bayesian approach for collaborative ranking in rec- ommender system. ACM Transactions on Information Sys- tems, 2023. 37. [Wu et al., 2019] Xunxian Wu, Tong Xu, Hengshu Zhu, Le Zhang, Enhong Chen, and Hui Xiong. Trend-aware tensor factorization for job skill demand analysis. In Pro- ceedings of the 28th International Joint Conference on Ar- tificial Intelligence, 2019. 38. [Xu et al., 2020] Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. Inductive representa- tion learning on temporal graphs. In International Confer- ence on Learning Representations (ICLR), 2020. 39. [Yang et al., 2021] Yuzhe Yang, Kaiwen Zha, Ying-Cong Chen, Hao Wang, and Dina Katabi. Delving into deep im- balanced regression. In International Conference on Ma- chine Learning, 2021. 40. [Yu et al., 2023] Le Yu, Leilei Sun, Bowen Du, and Weifeng Lv. Towards better dynamic graph learning: New architec- ture and unified library. Advances in Neural Information Processing Systems, 2023. 41. [Zha et al., 2023] Rui Zha, Chuan Qin, Le Zhang, Dazhong Shen, Tong Xu, Hengshu Zhu, and Enhong Chen. Ca- reer mobility analysis with uncertainty-aware graph au- toencoders: A job title transition perspective. IEEE Trans- actions on Computational Social Systems, 2023. 42. [Zhang et al., 2021] Qi Zhang, Hengshu Zhu, Ying Sun, Hao Liu, Fuzhen Zhuang, and Hui Xiong. Talent demand fore- casting with attentive neural sequential model. In Proceed- ings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021. 43. [Zhao et al., 2020] Ling Zhao, Yujiao Song, Chao Zhang, Yu Liu, Pu Wang, Tao Lin, Min Deng, and Haifeng Li. T-gcn: A temporal graph convolutional network for traffic prediction. IEEE Transactions on Intelligent Transporta- tion Systems, 2020. 44. [Zhu et al., 2016] Chen Zhu, Hengshu Zhu, Hui Xiong, Pengliang Ding, and Fang Xie. Recruitment market trend analysis with sequential latent variable models. In Pro- ceedings of the 22nd ACM SIGKDD International Confer- ence on Knowledge Discovery and Data Mining, 2016.