Pre-DyGAE: Pre-training Enhanced Dynamic Graph Autoencoder for Occupational Skill Demand Forecasting

1 Introduction

With the rapid development of technology, the nature of occupation is continually changing [Bisello et al., 2022], e.g., the skill demand of the occupation. OSD forecasting plays an important role in research fields related to the labor market and economics, such as labor market evolution [Sus and Sylwestrzak, 2021], occupational mobility and development [Wang et al., 2021b; Wang et al., 2021a; Qin et al., 2019], and enterprise recruitment [Bilan et al., 2020; Qin et al., 2022]. In particular, dynamic OSD forecasting is helpful for the development of individual career construction and career planning [Guo et al., 2022a]. As for enterprises, forward-looking OSD is helpful for recruitment.

∗This work was done when Xi Chen was an intern at Career Science Lab, BOSS Zhipin supervised by Chuan Qin.

†Corresponding Authors

Previous studies on the analysis between occupation and skill mainly leverage the survey or interview-based methods, e.g., Occupational Information Network (O*NET), which is very labor-intensive and involves human biases [Felten et al., 2018]. Fortunately, recent years have witnessed the rapid development of online recruitment platforms, accumulating massive job postings across different occupations. This provides an unparalleled chance for implementing large-scale data-driven OSD forecasting. Along this line, although some research has been introduced for skill demand prediction [Wu et al., 2019; Li et al., 2020; Chao et al., 2024] and skill trend analysis [Mahdavimoghaddam et al., 2022], the skill demand forecasting at the occupational level is still largely under-explored. Predicting skill demand at an occupational granularity is essential as it acknowledges the varying trends in skill development across different occupations, highlighting that a singular prediction of overall skill trends falls short of meeting diverse demands.

Indeed, there are three major challenges for OSD forecasting. Firstly, the trends of identical skills across various occupations exhibit diversification, while the demands for distinct skills in occupations are intricately interconnected [Langley et al., 2019]. This intricate interplay complicates the modeling in skill demand forecasting at the occupational level. Secondly, some pioneering skills often go unnoticed in enterprise recruitment job postings. This oversight makes it difficult to attain forward-looking forecasting on the skills that have not yet surfaced in occupations. Lastly, as shown in Figure 1 many skills occur in the long-tailed part, making accurate demand forecasting more challenging.

To address the above challenges, we introduce PreDyGAE, pre-training enhanced dynamic graph autoencoder for OSD forecasting. To be specific, we aggregate JDs by occupation and partition them into several timestamps. Firstly, leveraging JDs from the initial timestamps, we pre-train the graph autoencoder (GAE), consisting of an encoder, a link prediction decoder, and an edge regression decoder. The encoder is enhanced by semantically-aware cross-attention and Graph Convolution Neural Networks (GCNs) for robust representations of both occupations and skills. Besides, GAE is pre-trained with a link prediction task enhanced by contrastive learning on skill clusters and an edge regression task to predict the distribution of skill demand using a unified Tweedie and ranking loss. Subsequently, in the following timestamps, we add an adaptive temporal encoding unit and a temporal shift operator into the pre-trained GAE to achieve a DyGAE. Furthermore, we fix the GAE and fine-tune the DyGAE with a two-stage optimization strategy. Finally, we obtain the evolved representations and achieve the forecasting of future OSD. Extensive real-world dataset experiments confirm Pre-DyGAE significantly outperforms state-of-theart baselines. The code is available online 1 .

2 RelatedWork

2.1 Occupational Skill Analysis

Research on occupation and skill plays a crucial role in labor market analysis [Felten et al., 2018; Sun et al., 2021; Fang et al., 2023; Qin et al., 2024; Qin et al., 2023; Chen et al., 2021]. Recently, the labor market has undergone a rapid transformation driven by technological advancements. Several studies have examined the substitution of skills [Felten et al., 2018] and the evolution of occupations, while others have explored occupational mobility and market changes post-economic crisis [Bisello et al., 2022]. However, these studies often focus on localized phenomena of occupations and skills. The emergence of online recruitment platforms has enabled detailed statistical analysis of job postings. This data, collected from enterprise recruitment efforts, sheds light on the demand for talents and skills in the labor market [Zhu et al., 2016; Zhang et al., 2021; Guo et al., 2022b;

1 https://github.com/cx9941/Pre-DyGAE

Chao et al., 2024], helping to predict skill demand for specific jobs [Wu et al., 2019]. Nonetheless, existing research tends to concentrate on skills or occupations separately, lacking a holistic analysis of both. Diverging from prior work, our study adopts an innovative perspective to explore the skill demand at the occupational level over time.

2.2 Dynamic Graph Learning

Dynamic graph learning has garnered significant attention in recent research, which can model many real-world scenarios such as user-item interaction systems [Yu et al., 2023] and traffic networks [Zhao et al., 2020]. Many studies segment nodes based on temporal slices, analyzing static graphs within each slice and synthesizing temporal representations through methods like RNN or self-attention mechanisms, such as JODIE [Kumar et al., 2019], Dyrep [Trivedi et al., 2019], and CAWN [Wang et al., 2021d]. Specially, focus on spatio-temporal sequences, GConv [Seo et al., 2018], Dygrencoder [Taheri et al., 2019] and TGCN [Zhao et al., 2020] leveraged a combination of GCN and RNN to exploit spatial and temporal regularities. Besides, DCRNN [Li et al., 2018] captures the spatial dependency using random walks on the graph and A3TGCN [Bai et al., 2021] additionally adds the attention mechanism to assemble global temporal information. To learn the dynamic topology, EvolveGCN [Pareja et al., 2019] uses a recurrent model to evolve the GCN parameters, and AGCRN [Bai et al., 2020] captures fine-grained graph correlations in time series automatically. Some transformer-based methods, TCL [Wang et al., 2021c] and DyGFormer [Yu et al., 2023], learning from node historical first-hop interactions, efficiently benefit from long histories. Additionally, several studies have introduced temporal encoding mechanisms to effectively capture timing patterns in dynamic graphs, such as TGN [Rossi et al., 2020], TCL [Wang et al., 2021c] and GraphMixer [Cong et al., 2023].

3 Preliminaries

In this section, we define the key concepts integral to our study, present the problem formulation of OSD forecasting, and give an overview of our proposed Pre-DyGAE.

Definition 1 (Occupational Skill Demand). At timestamp t, each occupation o in the occupation set O contains collected JDs P to . Each JD p encompasses a set of skills from the skill set S. Along this line, we define the OSD at timestamp t as: Rto,s = ∑p∈P to 1(s ∈ p)/|P to |, where Rt ∈ R|O|×|S| and | · | denotes the size of a set.

Definition 2 (Occupational Skill Demand Graph). At timestamp t, OSD graph can be represented as an undirected bipartite graph G t = (V, E t , At ), where V = O ∪ S, E t is a set of edges between occupations and skills, and At ∈ R|V|×|V| is the weighted adjacency matrix. The edge eto,s between o and s is defined as {eto,s ∈ E t |o ∈ O, s ∈ S, and Rto,s ̸= 0}. The weighted adjacency matrix is defined as Ato,s = Ats,o = Rto,s if eto,s ∈ E t , otherwise 0.

Problem 1 (Occupational Skill Demand Forecasting). Given an OSD graph sequence (G 1 , G 2 , ..., G T ), our objective becomes predicting the OSD graph G T+1 for the upcoming frame T + 1. Specifically, we validate the effectiveness of dynamic graph forecasting via two common tasks in dynamic graph learning: (i) dynamic link prediction, which estimates the existence of eT+1 o,s between o and s; (ii) dynamic edge regression, which infers the weighted adjacency matrix {AT +1 o,s |eT+1 o,s ∈ E T+1 }.

Solution overview In this paper, we introduce Pre-DyGAE, a two-stage framework comprising pre-training and finetuning phases. An overview of our Pre-DyGAE is graphically depicted in Fig. 2. Considering the tradeoff between utilizing data integrity knowledge and capturing evolutionary patterns, we aggregate JDs from initial timestamps (P 1 , ..., P ∆t ) into P 0 to construct the initial graph G 0 for pre-training and finetune on the following OSD graphs (G∆t+1 , ..., G T ) for dynamic representations. In the pre-training stage, we employ a GAE to pre-train on G 0 , aiming to obtain robust representations. In particular, our encoder incorporates a semanticallyaware cross-attention mechanism, utilizing textual information stored in JDs to capture intricate relationships between occupations and skills. Additionally, GCNs are applied to model uncertainty. To guide comprehensive graph reconstruction, we introduce a contrastive loss on skill cooccurrence clusters for sparse data and a unified Tweedie loss and ranking loss for imbalanced prediction. In the fine-tuning stage, we enhance GAE with an adaptive temporal encoding unit and a temporal shift module for DyGAE. This dynamic model is fine-tuned using a two-stage optimization strategy, allowing it to evolve representations from timestamp ∆t + 1 to T. Finally, our framework enables the inference of future representations and the prediction of future OSD.

4 Pre-training Enhanced Graph Autoencoder

In this section, our objective is to devise a pre-training model adept at capturing intricate relationships between occupations and skills to yield robust representations. The architecture consists of an encoder and two decoders. In the multi-task training framework, we jointly pre-train the model in a multitask framework to guide comprehensive graph reconstruction.

4.1 Encoder

To fully leverage the semantic content of the data, we initial the primary embeddings of occupation o and skill s as Eo and Es with their corresponding JDs:

where P 0 s represents {p|s ∈ p, p ∈ P 0 }, Bert(p) signifies the embedding of the p derived from a pretrained BERT model.

Semantically-Aware Cross-Attention Besides, a crossattention mechanism is contributed to further harness the intricacies tied to skills and their representations in occupations. Let's represent the matrices of occupations and skills embeddings as EO and ES . We obtain the semanticallyaware cross-attention enhanced occupation and skill embeddings Z O as:

where W Q, W K , and W V are trainable parameters and dK is the dimension of EK .

Furthermore, we introduce a semantically-aware crossattention mechanism to further harness the semantic intricacies tied to skills and their representations in occupations. The mechanism can be reformulated as:

where the bias Bbias ∈ R|O|×|S| is learnable, pivotal for instilling deep semantic insights into the representations of occupations and skills. With a cosine similarity function Sim(x, x′ ) = x·x′ ∥x∥·∥x′ ∥ , the bias Bbias is defined as:

where FCi(·) is a linear fully connected layer.

Uncertainy-Aware Modeling Furthermore, we utilize structural data to steer the integration of occupation and skill representations by GCN. The aggregation process is:

where Z is the concatenation of Z O and Z S , f is a activation function, I is the identity matrix, W 0 is the trainable node transformation matrix, and D is the degree matrix of A0 + I.

Considering the inherent uncertainty arising from multiple job postings associated with individual occupations, a deterministic representation may risk overfitting. To address this, we model the latent variable H0 to adhere to a Gaussian distribution for deriving more robust representations:

where N(µ, σ) is the probability density function of the Gaussian distribution with mean µ and variance σ2 , and W µ and W σ are two trainable node transformation matrices.

4.2 Decoders

Upon obtaining the representations, the decoders aim to reconstruct the input graph G 0 in two aspects, link prediction to predict the relationships between occupations and skills and edge regression for predicting the specific values.

Link Prediction Decoder We model the likelihood P E of associating skill s with occupation o as follows:

where sigmoid(·) is an activation function for calculating the probability, ⊙ denotes the element-wise multiplication operation, and MeanPooling(·) calculates the average value along the last dimension in the matrix.

Edge Regression Decoder Extending the capabilities of GAE, we utilize a linear layer with an activation function to estimate R0o,s :ˆR0o,s = softplus(FC3(H0 o ⊙H0 s )), (8)

where softplus(·) is an activation function as a smooth approximation of the rectifier linear unit activation function.

Through this approach, GAE decodes H0 into distinct spaces for predicting both relationships between occupations and skills and OSD, facilitating mutual learning that contributes to the accomplishment of diverse tasks. The prediction within a static time slice also contributes to subsequent dynamic forecasting.

4.3 Pre-training Objectives

The pre-training objective for graph reconstruction involves training the primary tasks of link prediction and edge regression, incorporating two auxiliary tasks. The loss function is composed of a negative log-likelihood loss (L0nll ) and a collaborative-aware contrastive loss (L0con ) for link prediction with sparse data challenge, as well as a Tweedie loss (L0rg ) and a ranking loss (L0rank ) for imbalanced regression.

Negative Log-Likelihood Loss To achieve the accurate reconstruction of the graph edges E 0 , the negative loglikelihood loss Lnll is minimized:

where y0 o,s = 1 if e0o,s ∈ E and y0 o,s = 0 otherwise.

Collaborative-Aware Constrative Loss To solve the challenges of predicting relationships between occupations and less frequent skills, we further introduce a collaborativeaware contrastive loss. Specifically, by evaluating skill cooccurrence in P 0 , we apply a spectral clustering algorithm to divide the skills into C clusters and Ck is the k-th cluster contains skills with the closest co-occurrence degree. This approach tightens the representation of skills within a single cluster and thus addresses data sparsity, which is defined as:

where γ is the temperature coefficient.

Tweedie Loss Considering the OSD follows long-tailed distribution, directly employing regression loss functions i.e., MAE, may hinder the model's ability to effectively capture the imbalances present in both the head and tail data in training. Here, we employ the Tweedie loss to address this issue.

where η ∈ (1, 2) is a hyperparameter defining the power relation between Tweedie distribution mean and variance, and E neg is the set of randomly sampled negative edges.

RankingLoss Furthermore, the diverse skill demands of an occupation underscore the unique roles each skill plays. For instance, some foundational skills might be prevalent across different occupations, whereas the low occurrences could indicate some emerging demand. Therefore, we devise a ranking loss, as a pairwise loss [Wang et al., 2023], to better capture the distinctions between skills.

where ϵ is the margin value and ρ0o,s is the rank. Specifically, if R0o,s ≥ 0.1, ρ0o,s = 0; if 0.1 > R0o,s > 0.01, ρ0o,s = 1; otherwise, ρ0o,s = 2.

Finally, the pre-training loss function L0GAE is defined:

where L0kl = 12 ∑i,j (µ0ij 2 + (σ0 ij )2 - 2 log(σ0 ij )-1) is a KL loss to minimize the discrepancy between the posterior and prior distributions of the latent variables, and λnll, λcon, λrg and λrank are the hyperparameters to balance the effects of different modules.

5 Dynamic Graph Autoencoder

In this section, we incorporate an adaptive temporal encoding unit and a temporal shift module into GAE . Besides, we fix GAE and apply parameter efficient fine-tuning on DyGAE with a two-step optimization strategy. Finally, we can evolve the dynamic representations underlying a dynamic graph sequence (G∆t+1 , ..., G T ) and achieve OSD forecasting.

5.1 Temporal Feature Learning

Specifically, we design an adaptive temporal encoding unit to facilitate the learning of temporal representations. Additionally, we model the Gaussian distribution of temporal features, enhancing our ability to capture and represent dynamic uncertainties. Notably, we introduce a temporal shift module to effectively explore the temporal evolution of features, thereby achieving dynamic representations.

Adaptive Temporal Encoding Unit At timestamp t, the relationship between occupations and skills hinges on the prevailing graph structure G t and the stable graph structure G 0 . To harness the wealth of structural information embedded within the relationship between occupations and skills, we employ the dual-view GCN propagation method, integrating it with the specified adjacency matrix At and a learnable graph Aa initialized with A0 . With an initial learnable embedding Lt ∈ R(|O|+|S|)×d where d is the feature dimension, the dual-view GCN propagation layer is devised as:

where M t ∈ R(|O|+|S|)×d is a trainable parameter, F t is the structure enhanced temporal hidden states, and [·; ·] denotes a concanate operation. Furthermore, We encode the temporal feature as a Gaussian distribution:

Dynamic Uncertainty-Aware Modeling While the stable representations capture the essential characteristics of occupations and skills, learned temporal characteristics enrich the representation with time-varying nuances. Leveraging the superposition property of normal distributions, we model the dynamic representations by incorporating the temporal features into the stable initial representation:

This strategy allows us to capture intricate temporal patterns and facilitate robust representation learning.

Temporal Shift Module Following the assumption that variance does not depend on time [Gourru et al., 2022], we infer ˆ δt+1 = ∑ δt /T. To explore the evolution of ϕt , we first decompose ϕt into a stable term αt = FC8(ϕt ) and a trend term βt = FC9(ϕt ). To predict ϕt+1 , we add an MLP to evolve the trend term as ωt = MLP(βt ). Then we forecast the ϕt+1 as αt + ωt .

5.2 Parameter Efficient Fine-Tuning

Leveraging the pre-trained GAE, we initially focus on optimizing the temporal features in each timestamp. Following this, we delve into exploring the evolution pattern. This two-step optimization strategy allows for a comprehensive exploitation of the pre-trained model's advantages and facilitates the accurate capture of evolution characteristics.

Adaptive Temporal Encoding Unit Optimization For dynamic OSD, trend changes usually include four situations: emergence, disappearance, growth, and decline. To explicitly allow dynamic representations to perceive changes in trends, for each o and s, we additionally add a linear layer with an activation function to map temporal features into four trend labels τo,s as above.

We estimate the likelihood of the trend label between o and s at t as P τ (o, s, t) = sigmoid(FC7(Hto ⊙Ht s )). The trendaware loss is defined as:

In each timestamp t ∈ [∆t + 1, T], we fix the GAE and fine-tune the adaptive temporal encoding unit via Lttem = λtrendLttrend + LtGAE for the temporal feature distribution (ϕt , δt ), where λtrend is the hyperparameter, LtGAE is the pre-training loss in Eq.13 with Ht as the input of decoders and G t as the target graph.

Temporal Shift Module Optimization Based on the temporal feature sequence ((ϕ∆t+1 , δ∆t+1 ), ..., (ϕT , δT )), we intend to predict the temporal feature in the next timestamp (ϕT+1 , δT+1 ). Firstly we reconstruct the {ϕt |t ∈ [∆t+1, T ]} with a reconstruction loss Lre:

where MSE(x, x′ ) is a function to compute the mean squared error between x and x′ . Besides, to achieve next-step prediction, we introduce Lne as:

Additionally, we formulate the loss λcom to keep the stable terms similar and distinguish trend terms.

where Cos(x, x′ ) is a function, defined as Cos(x, x′ ) = 1m ∑mi=1 xi ·x′i ∥xi∥·∥x′i ∥ , to compute the average cosine similarity between x and x′ and m is the number of row in x. During the training process, the three loss functions are combined as Lshift = λreLre+λneLne+λcomLcom where λre, λne, and λcom are the hyperparameters.

5.3 Inference Procedure

After the fine-tuning, we can infer the next step temporal features as ˆ ϕT+1 = αT + ωT . Combining with the pretrained representation distributions, we forecast the representation distributions of occupations and skills ˆHT+1 ∼ N(µ0 + ˆ ϕT+1 , σ0 + ˆ δT+1 ). Furthermore, the OSD graph ˆ G T+1 can be inferred with the inferred likelihood ˆ P E (eT +1 o,s ∈ E T+1 ) = sigmoid(MeanPooling( ˆHT+1 o ⊙ ˆHT+1 s )) and the inferred edge weight ˆ AT +1 o,s = softplus(FC3( ˆHT+1 o ⊙ ˆHT+1 s )).

6 Experiments

6.1 Experimental Settings

Datasets The datasets used in our experiments were collected from the public information of one of the largest online recruitment platforms. We gathered JDs spanning a diverse range of occupations, covering the period from January 2020 to December 2023. Along this line, we constructed largescale datasets from four industries, i.e., the Daily dataset (Dai), Finance dataset (Fin), IT dataset (IT), and Manufacturing dataset (Man). As mentioned above, we used half-year intervals as the basic time step, dividing each dataset into 8 timestamps. In our experiment setup, we selected ∆t = 4, which entailed using JDs from the first 4 timestamps for pretraining and the subsequent 3 timestamps for dynamic temporal feature learning. Besides, more parameter experiments are in the Appendix, and we choose λnll, λcon, λrg, λrank, λtrend, λre, λne, and λcom as 1, 0.05, 100, 0.1, 1, 1, 1, and 1. Meanwhile, we partitioned the JDs from the first month of the final timestamp to create a validation set, while the remaining five months' JDs were used as a test set. Detailed statistics and divisions of our datasets are provided in the Appendix.

Evaluation Metrics We evaluated the effectiveness of our approach via dynamic link prediction and edge regression tasks. Therefore, we employed several link prediction metrics, including AUC [Zha et al., 2023], MRR, and Hits@1 [Wang et al., 2022], and regression metrics, including EGM [Yang et al., 2021], MAE [Zhu et al., 2016], and RMSE [Chang et al., 2014]. In particular, EGM was used to measure the heavy-tailed nature of our data distributions.

Baselines We compared Pre-DyGAEwith a state-of-the-art method, i.e., CHGH [Chao et al., 2024]. In addition, we included dynamic graph learning baselines, including DCRNN

[Li et al., 2018], DyGrEncoder [Taheri et al., 2019], AGCRN [Bai et al., 2020], A3TGCN [Bai et al., 2021], TGAT [Xu et al., 2020], CAWN [Wang et al., 2021d], and DyGFormer [Yu et al., 2023]. To maintain fairness in comparison, each baseline was trained with data within the former 7 timestamps. Besides, we also compared some traditional methods, but due to not achieving competitive results and page limitations, we have put their experimental results in the Appendix.

6.2 Experimental Results

Performance Comparison Table 1 illustrates the comprehensive performance evaluation of Pre-DyGAE and baselines across four datasets. According to the results, there are several observations. Firstly, our proposed model has significant improvements over all baselines in all datasets, thus demonstrating the effectiveness of our framework in OSD forecasting. Specifically, the AUC metric showcases a remarkable performance of 96%. This confirms its capability to precisely predict future skills across a variety of occupations, a critical aspect for subsequent edge regression. Additionally, our approach demonstrates proficiency by improving the average RMSE in four datasets from 1.527% to 0.635%, an increase of 58% compared with the best-performing baseline, providing precise insights into the distribution of demand, and highlighting its capability to forecast future skill demand accurately. Moreover, the improvement in the EGM directly signifies the competence of our model in predicting unbalanced distribution. Thirdly, baseline models often show imbalanced results in both link prediction and regression metrics, indicating the complexity of integrating these two tasks. In contrast, our model excels in all metrics, demonstrating Pre-DyGAE's ability to improve performance on both tasks. Finally, it is noteworthy that RNN-based methods such as DCRNN and CAWN outperform Transformer-based methods such as DyGFormer in these datasets. This is attributed to the Transformer's primary focus on addressing long-horizon autocorrelation while overlooking variable trends, aligning with the unique characteristics of our forecasting problem.

GAE Performance on OSD Graph Completion We conducted a validation experiment to assess the robustness of GAE's learned representations by predicting missing skill demand within the same time interval. JDs from the initial 4 timestamps were randomly split into training and test sets at a ratio of 50%, with corresponding occupational skill demand in triplet form (o, s,R0o,s ). For comparison, we selected NGCF, LightGCN, and MultVAE as the baselines [He et al., 2020]. The results in Table 2 show that GAE improved AUC and MRR compared to all baselines, indicating effective representations of occupations and skills. Specifically, "w/o bias" significantly impacts EGM and RMSE, demonstrating the importance of the semantically-aware cross attention. The ranking loss distinguishes feature spaces, and contrastive learning on skill clusters enhances metrics, guiding recognition of sparse skills. Additionally, comparing L1 loss and the Tweedie loss suggests that the latter is more suitable for predicting imbalanced distributions. Comprehensive results across all datasets are detailed in the Appendix.

Ablation Studies We performed ablation studies to assess different modules in Pre-DyGAE, as depicted in Fig. 3. Results show Pre-DyGAE's performance compared to various backbones, highlighting GAE's robust representation and its ability to enhance capturing temporal features. Ablation experiments on the temporal module affirm our model's capability to capture demand distribution shifts caused by subsequent temporal changes based on the backbone representation.

Parameter Experiment We have carefully considered the balance between leveraging data integrity knowledge and capturing evolutionary patterns. As shown in Fig. 4, the trend of rising first and then falling suggests the advantages of stable representations for capturing subsequent changes and the necessity of preserving some future time slices for mining evolutionary patterns. The consistency in the number of time frames across different metrics in parameter experiments further supports the effectiveness of our pre-training and finetuning framework. More details can be seen in the Appendix.

7 Conclusion

In this paper, we addressed the critical task of forecasting skill demand in specific occupations, a key aspect in the dynamics of labor markets and economics. Specifically, we introduced Pre-DyGAE, a novel approach involving pre-training on initial timestamps, followed by parameterefficient fine-tuning in subsequent timestamps. This approach leveraged the overall interaction between occupations and skills over an extended period, enabling the discernment of stable associations between them. The pre-training of GAE incorporated a semantically-aware cross-attention enhanced uncertainty-aware encoder and a multi-task pretraining framework. These components proved effective in establishing fundamental representations. Based on the pretrained GAE, DyGAE excelled in capturing dynamic temporal features through a two-stage optimization process involving adaptive temporal encoding units and a temporal shift module. Extensive experiments on four real-world datasets have demonstrated the effectiveness of our Pre-DyGAE.

Acknowledgements

This work was partially supported by National Natural Science Foundation of China (Grant No.92370204), Guangzhou-HKUST(GZ) Joint Funding Program(Grant No.2023A03J0008), the National Key Research and Development Program of China (No. 2023YFF0725001), Education Bureau of Guangzhou Municipality, Guangdong Science and Technology Department, and Project funded by China Postdoctoral Science Foundation (Grant No.2023M730785).