--- title: "Enhancing Urban Flow Maps via Neural ODEs" authors: - "Fan Zhou" - "Liang Li" - "Ting Zhong" - "Goce Trajcevski" - "Kunpeng Zhang" - "Jiahao Wang" doi: "10.24963/ijcai.2020/180" url: "https://ijcai.org/proceedings/2020/180" pdf: "https://ijcai.org/proceedings/2020/0180.pdf" venue: "Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence" year: 2020 pages: "1295-1302" track: "Main track" keywords: - "Data Mining: Mining Spatial, Temporal Data" - "Machine Learning Applications: Applications of Supervised Learning" - "Data Mining: Feature Extraction, Selection and Dimensionality Reduction" --- # Enhancing Urban Flow Maps via Neural ODEs Fan Zhou, Liang Li, Ting Zhong, Goce Trajcevski, Kunpeng Zhang, Jiahao Wang *Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence*, pages 1295-1302. DOI: [10.24963/ijcai.2020/180](https://doi.org/10.24963/ijcai.2020/180) ## Abstract Flow super-resolution (FSR) enables inferring fine-grained urban flows with coarse-grained observations and plays an important role in traffic monitoring and prediction. The existing FSR solutions rely on deep CNN models (e.g., ResNet) for learning spatial correlation, incurring excessive memory cost and numerous parameter updates. We propose to tackle the urban flows inference using dynamic systems paradigm and present a new method FODE -- FSR with Ordinary Differential Equations (ODEs). FODE extends neural ODEs by introducing an affine coupling layer to overcome the problem of numerically unstable gradient computation, which allows more accurate and efficient spatial correlation estimation, without extra memory cost. In addition, FODE provides a flexible balance between flow inference accuracy and computational efficiency. A FODE-based augmented normalization mechanism is further introduced to constrain the flow distribution with the influence of external factors. Experimental evaluations on two real-world datasets demonstrate that FODE significantly outperforms several baseline approaches. ## Full text ## 1 Introduction Urban flow super-resolution (FSR) aims at inferring finegrained crowd flows in a city based on coarse-grained observations. As a variant of image SR in the traffic domain [Cai et al., 2019; Wang et al., 2019] it has practical significance in urban planning and traffic monitoring. Despite its close relationship to image SR, FSR has certain constraints the most important one being the structural constraint - i.e., the sum of the flow volumes in surrounding regions in the inferred fine-grained map strictly equals that of their corresponding superregion in the original map. However, in practice, the distribution of the flows in a given region is affected by many external factors, e.g., weather, time-of-day, etc. A recent work [Liang et al., 2019] addresses FSR problem based on the residual networks [He et al., 2016] and uses a simple normalization scheme to constrain the structural distribution of flows. However, the proposed architecture heavily relies on empirically stacking deep neural networks and, consequently, lacks principles to guide the design of effective and interpretable FSR networks. Furthermore, it pays no attention to the computational overheads as the network goes deeper, which requires significantly more memory cost and network parameters - making it hard to be optimized and prone to be overfitting. Recently, there has been a growing interest in bridging the gap between neural networks and dynamic systems [E, 2017; Lu et al., 2018]. In particular, a recent study [Chen et al., 2018] has shown that ResNet [He et al., 2016] can be interpreted as discretized Neural Ordinary Differential Equations (NODE), which provides a new perspective of improving stability and trainability of neural networks. For example, an additional state (called adjoint) is used in [Chen et al., 2018] to solve the ODEs. In this way, it does not need to store the intermediate states of the forward pass, resulting in O(1) memory cost in each ODE block. A few studies have been proposed to improve the performance of NODE [Liu et al., 2019b] and/or to apply NODE in different domains such as graph neural networks [Poli et al., 2019], time series learning [Rubanova et al., 2019; De Brouwer et al., 2019] and generative models [Heinonen and L¨ahdesm¨aki, 2019; Grathwohl et al., 2019]. However, the dynamics of either the hidden state or the adjoint are usually unstable, due to the numerical instability of solving the backward ODEs. ANODE [Gholami et al., 2019] and its variant [Zhang et al., 2019] improve the robustness and generalization of the adjoint method by adding a few of the intermediate states from the forward pass however, these cannot fundamentally solve the incorrect gradient problem. In this work, we introduce a new SR method - FODE (FSR with ODEs) for fine-grained urban flow inference. FODE is a more general neural ODE architecture that addresses the numerical instability problem of previous methods, while not incurring extra memory cost. The main idea of FODE is to incorporate an affine coupling layer in each ODE block to avoid the inaccurate gradient issue. The input to a FODE block can then be accurately reconstructed from its outputs without the need of storing intermediate states. We show theoretically and empirically that the proposed FODE model can be used to learn spatial correlations among urban regions and the influence of external features. The main contributions of this work can be summarized as: - We present a new neural ODE framework that can accu- rately compute the gradients in each block. - Our novel FSR method requires significantly less memory for fine-grained flow inference compared with ResNet-based deep neural network. - The augmented normalization scheme disentangles the influence of external factors on different regions. - Experiments conducted on two real-world datasets demonstrate that our FODE not only outperforms strong baselines on FSR task, but also provides a flexible balance between computational cost and inference accuracy. ## 2 Architecture andMethodology We now present the problem settings and discuss in detail the main aspects of the proposed FODE approach. We assume a grid-based segmentation, similar to [Zhang et al., 2017], which divides a city map M into H×W spatial grid-cells based on their geographical locations, i.e., M = {rij}H×M. Each cell rij corresponds to a spatial region - the ith row and the j th column of M. Let X ∈ RH×W + be the flow map at a given time, where each entry xij ∈ R+ denotes the volume of the flow in the region rij . Definition 1 (Urban Flow Super-Resolution (FSR)). Given a coarse-grained flow map Xc , an upscaling factor N as well as the external factors E (e.g., weather, events, etc.), the FSR task is to learn a model F mapping Xc into a fine-grained flow map ̂Xf ∈ RNH×NW + : where θ represent all learnable parameters. Figure 1 illustrates an example of converting a coarsegrained flow map to a fine-grained flow map in the city of Beijing. The coarse-grained flow map (20km×20km) consists of a total of 32×32 cells, each of which denotes a superregion. In the fine-grained flow map, there are 128×128 subregions in total. At the top of Figure 1, a superregion is composed of N 2 subregions (N = 4 here). Note that the sum of the flow volume in the N 2 subregions (top right) is equal to the flow volume of the corresponding superregion (top left). Figure 1: Coarse-grained and fine-grained flow maps in Beijing. ![Image](https://ijcai.org/proceedings/2020/fulltext/180/image_000000_379c4f75398ac90e88c8202eac4a07e77e41958d117d8dd80ea1b5409ea4e2d7.png) < l a t e x i s h 1 \_ b 6 4 = " C W Q G p Y S R r U I + m g F M > A B H c V N 8 E J 3 q / 9 L 0 v u w K 2 7 T d j f X o 5 D z n P y Z k O < l a t e x i s h 1 \_ b 6 4 = " C W Q G p Y S R r U I + m g F M > A B H c V N 8 E J 3 q / 9 L 0 v u w K 2 7 T d j f X o 5 D z n P y Z k O < l a t e x i s h 1 \_ b 6 4 = " C W Q G p Y S R r U I + m g F M > A B H c V N 8 E J 3 q / 9 L 0 v u w K 2 7 T d j f X o 5 D z n P y Z k O < l a t e x i s h 1 \_ b 6 4 = " C W Q G p Y S R r U I + m g F M > A B H c V N 8 E J 3 q / 9 L 0 v u w K 2 7 T d j f X o 5 D z n P y Z k O ## 2.1 Architecture Figure 2 illustrates the overall architecture of FODE, which consists of three main components: - FODE block which extracts spatial correlations among flow regions with the proposed new ODEs. - Feature fusion network (FFN) which combines the FODE blocks and fully connected networks for fusing the influence of external factors. - Fine-grained flow inference (FFI) which leverages an augmented N 2 -normalization scheme (AN 2 ) to estimate the distribution of both flows and external factors for generating the fine-grained flow maps. ## 2.2 FODE Block ResNet [He et al., 2016] and its variants have been widely employed for image super-resolution [Ledig et al., 2017; Zhang et al., 2018]. Recently, [Liang et al., 2019] exploited residual block for regional correlation learning and fine-grained urban flow inference, which share the same idea of image SR. Suppose we obtain the high-dimensional hidden state z0 from the coarse-grained flow Xc by some convolutional layers. The residual block then transforms the hidden states zn according to: While achieving promising results on SR tasks, residual networks still confront the problem of intensive computation and require to tune a huge amount of parameters. There is a growing interest in bridging the gap between discrete neural networks and continuous dynamic systems. For example, the iterative updates in the above residual block can be viewed as a discretization of a continuous ODE operator [Lu et al., 2018; Chen et al., 2018], if we take time t ∈ T as a continuous variable: Solving the above ODEs requires computing the gradients through backpropagation, which would be memory prohibitive if the time is reduced into infinite steps. In principle, it requires O(N) cost to store the all N intermediate activations. NODE [Chen et al., 2018] proposes to address this problem with the adjoint method. Considering a loss function L, an additional state referred to as the adjoint a (t) = ∂L/∂z (t) can be used to compute the gradient w.r.t. parameters θ: where we only need to store the final state z(T ). Hence this strategy successfully reduces the memory cost. However, it also introduces other problems - the inaccurate gradient and < l a t e x i s h 1 \_ b 6 4 = " U p X O S H f y v j M V c P 8 > A B 9 D L N F 2 r Z d u g k w R I Q W + 0 T J / C n m 3 o 7 E q z G Y K 5 < l a t e x i s h 1 \_ b 6 4 = " U p X O S H f y v j M V c P 8 > A B 9 D L N F 2 r Z d u g k w R I Q W + 0 T J / C n m 3 o 7 E q z G Y K 5 < l a t e x i s h 1 \_ b 6 4 = " U p X O S H f y v j M V c P 8 > A B 9 D L N F 2 r Z d u g k w R I Q W + 0 T J / C n m 3 o 7 E q z G Y K 5 < l a t e x i s h 1 \_ b 6 4 = " U p X O S H f y v j M V c P 8 > A B 9 D L N F 2 r Z d u g k w R I Q W + 0 T J / C n m 3 o 7 E q z G Y K 5 < l a t e x i s h 1 \_ b 6 4 = " H m X 0 9 B d v P o y 3 D f W g > A 8 c V L S N F u r Y U Q E R + C T M j O / n K w z p G I k 7 2 q J Z 5 < l a t e x i s h 1 \_ b 6 4 = " H m X 0 9 B d v P o y 3 D f W g > A 8 c V L S N F u r Y U Q E R + C T M j O / n K w z p G I k 7 2 q J Z 5 < l a t e x i s h 1 \_ b 6 4 = " H m X 0 9 B d v P o y 3 D f W g > A 8 c V L S N F u r Y U Q E R + C T M j O / n K w z p G I k 7 2 q J Z 5 < l a t e x i s h 1 \_ b 6 4 = " H m X 0 9 B d v P o y 3 D f W g > A 8 c V L S N F u r Y U Q E R + C T M j O / n K w z p G I k 7 2 q J Z 5 < l a t e x i s h 1 \_ b 6 4 = " 0 N o O Z f E S L P U 8 / Q Y + n G I > A B 3 c V D F W u r X R C m g d T M j 9 K w z q 2 v 7 5 k J p y H < l a t e x i s h 1 \_ b 6 4 = " 0 N o O Z f E S L P U 8 / Q Y + n G I > A B 3 c V D F W u r X R C m g d T M j 9 K w z q 2 v 7 5 k J p y H < l a t e x i s h 1 \_ b 6 4 = " 0 N o O Z f E S L P U 8 / Q Y + n G I > A B 3 c V D F W u r X R C m g d T M j 9 K w z q 2 v 7 5 k J p y H < l a t e x i s h 1 \_ b 6 4 = " 0 N o O Z f E S L P U 8 / Q Y + n G I > A B 3 c V D F W u r X R C m g d T M j 9 K w z q 2 v 7 5 k J p y H < l a t e x i s h 1 \_ b 6 4 = " Q d m M y 7 8 X P q c G D C W H f K A I Y U > B / 3 V L S N F J 2 v O T r k o g n R w E 5 9 + j Z 0 u p z < l a t e x i s h 1 \_ b 6 4 = " Q d m M y 7 8 X P q c G D C W H f K A I Y U > B / 3 V L S N F J 2 v O T r k o g n R w E 5 9 + j Z 0 u p z < l a t e x i s h 1 \_ b 6 4 = " Q d m M y 7 8 X P q c G D C W H f K A I Y U > B / 3 V L S N F J 2 v O T r k o g n R w E 5 9 + j Z 0 u p z < l a t e x i s h 1 \_ b 6 4 = " Q d m M y 7 8 X P q c G D C W H f K A I Y U > B / 3 V L S N F J 2 v O T r k o g n R w E 5 9 + j Z 0 u p z < l a t e x i s h 1 \_ b 6 4 = " P J 3 I 8 Q u S z H 9 k L d > A B c V N E W X O C o 0 g U D 2 w m 7 G / v T y 5 r + M n j Z p q K F R f < l a t e x i s h 1 \_ b 6 4 = " P J 3 I 8 Q u S z H 9 k L d > A B c V N E W X O C o 0 g U D 2 w m 7 G / v T y 5 r + M n j Z p q K F R f < l a t e x i s h 1 \_ b 6 4 = " P J 3 I 8 Q u S z H 9 k L d > A B c V N E W X O C o 0 g U D 2 w m 7 G / v T y 5 r + M n j Z p q K F R f < l a t e x i s h 1 \_ b 6 4 = " P J 3 I 8 Q u S z H 9 k L d > A B c V N E W X O C o 0 g U D 2 w m 7 G / v T y 5 r + M n j Z p q K F R f < l a t e x i s h 1 \_ b 6 4 = " y m P u 5 r v C U J K 9 g W L R F c > A B + X V D S N 2 p f Z d o q Q Y M 3 T E / z 0 G w n 7 H I 8 j k O < l a t e x i s h 1 \_ b 6 4 = " y m P u 5 r v C U J K 9 g W L R F c > A B + X V D S N 2 p f Z d o q Q Y M 3 T E / z 0 G w n 7 H I 8 j k O < l a t e x i s h 1 \_ b 6 4 = " y m P u 5 r v C U J K 9 g W L R F c > A B + X V D S N 2 p f Z d o q Q Y M 3 T E / z 0 G w n 7 H I 8 j k O < l a t e x i s h 1 \_ b 6 4 = " y m P u 5 r v C U J K 9 g W L R F c > A B + X V D S N 2 p f Z d o q Q Y M 3 T E / z 0 G w n 7 H I 8 j k O Figure 2: Overview of the proposed FSR architecture. ![Image](https://ijcai.org/proceedings/2020/fulltext/180/image_000001_13c6d6391784b933c7f75e3126ff3c3e69fdc602fda342672ed3fc71b9e75a1b.png) numerical instability. Taking Eq. (2) for example, we compare the difference between the two calculated gradients: (1) Calculate the gradient w.r.t. zn by Eq. (6): (2) Calculate the exact gradient w.r.t. zn by chain rule: As can be seen, the second terms of Eq. (7) and Eq. (8) are different, which, therefore, makes the state a (tn) ̸= an. In other words, the adjoint method leads to inaccurate and unstable gradient ∂L/∂θ - compared to direct backpropagation that computes correct and stable gradient, but suffers a prohibitive memory cost. ANODE and its variant [Gholami et al., 2019; Zhang et al., 2019] mitigate this problem by splitting a block into time batches and stores in memory a few intermediate states from the forward pass, which, in a sense, is a compromise between time and storage. In this work, we bridge this gap by introducing an affine coupling layer in the computations of ODEs, to avoid the inaccurate gradient issue while only requiring O(1) memory in each block. Affine coupling layer referred to a family of neural network whose forward function is a bijective mapping and has been widely used in invertible generative models [Dinh et al., 2015; Dinh et al., 2017; Kingma and Dhariwal, 2018], where the input to a bijective block can be accurately reconstructed from its outputs. As illustrated in Figure 3, we divide the input zt (let zt = z (t)) into two parts of same size zat , zbt ∈ RC/2×H×W - where C is the number of channels. In the forward pass functions in each FODE block with time step size ∆t, we have: Figure 3: Computational graph for a FODE block. ![Image](https://ijcai.org/proceedings/2020/fulltext/180/image_000002_fcdb70d002fd73434edc6b63cd4c2121a536dc176bd97f8a8804e674098a9cf6.png) where hat and hbt are the intermediate states, zat+∆t and zbt+∆t denote the outputs, I, J , S and K are differentiable neural networks. The reverse computations are therefore: To ensure the reversibility of Eq. (10), all elements in S (hbt ) and I (zat ) need to be greater than zero, which can be met by making a simple transformation for each element e as exp(loge), e ∈ I or e ∈ S. Then we have following property which ensures that the affine transformations in the forward pass in Eq. (9) are reversible: Proposition 1. The forward pass functions Eq. (9) in the FODE block is reversible as long as each element of I(zat ) and S(hbt ) is non-zero. Proof. Let JF and JG be the Jacobians of the transformations in Eq. (9), which can be computed as: where I and O are identity and zero matrix, respectively, and diag(·) denotes the diagonal matrix whose diagonal elements correspond to the elements in I (zat ) or S(hbt ). Since JF and JG are lower triangular matrices, their determinants are < l a t e x i s h 1 \_ b 6 4 = " d V Z j r z k g D 0 X K 7 8 O f n S > A B 9 c J N E 2 W o M F T p R m Q 5 + q / 3 w U P H Y y v u G I L C < l a t e x i s h 1 \_ b 6 4 = " o r k q 9 f X z n + c B M D A K W d y 0 > V L S g N E J G Z Y 2 C 7 Q 3 T j U v w H R / O I p u 5 m P F 8 < l a t e x i s h 1 \_ b 6 4 = " o r k q 9 f X z n + c B M D A K W d y 0 > V L S g N E J G Z Y 2 C 7 Q 3 T j U v w H R / O I p u 5 m P F 8 < l a t e x i s h 1 \_ b 6 4 = " r T E 2 K H f q N Z / w J 5 c o g > A B 9 X V C 7 D z O Q S F R Y U 0 v + 8 m W P y n I L k j d 3 G u p M < l a t e x i s h 1 \_ b 6 4 = " T E K 8 j z 0 B d Y k 5 g 9 q v > A X c V C 7 S N F L 3 r M Z J m U w I H p P D / G Q f n O + W 2 u R y o < l a t e x i s h 1 \_ b 6 4 = " Q j D T P k M v o N X 7 2 S Y d g A > B 9 c V L F r q p 0 E y C w z u K m 5 8 + O G Z J H I R / 3 n W U f < l a t e x i s h 1 \_ b 6 4 = " Q j D T P k M v o N X 7 2 S Y d g A > B 9 c V L F r q p 0 E y C w z u K m 5 8 + O G Z J H I R / 3 n W U f < l a t e x i s h 1 \_ b 6 4 = " P I q G v E o p X F L 5 A y n M Q g > B 9 c V C 7 T w 0 r J U Y W K z u R S d f / 8 3 + D O j 2 m N k Z H < l a t e x i s h 1 \_ b 6 4 = " 0 W u Y M k V L O 5 n J + o G N z 7 D > A B 8 c S F 3 r y K 9 g P p m Z E f q X / j C w H I U 2 R v Q T d < l a t e x i s h 1 \_ b 6 4 = " 0 W u Y M k V L O 5 n J + o G N z 7 D > A B 8 c S F 3 r y K 9 g P p m Z E f q X / j C w H I U 2 R v Q T d < l a t e x i s h 1 \_ b 6 4 = " 0 W u Y M k V L O 5 n J + o G N z 7 D > A B 8 c S F 3 r y K 9 g P p m Z E f q X / j C w H I U 2 R v Q T d < l a t e x i s h 1 \_ b 6 4 = " c H r V S k d g D W v B M 2 p Y O U > A 8 n L N F 3 u 7 K C f y m Q J R + G / o T P w 5 I Z z X q j 9 E 0 < l a t e x i s h 1 \_ b 6 4 = " o z 7 r C E j g J G N 9 d D m 2 Q 3 0 > A B 8 n c V S F L X + q O y T Y R Z p M P k w f 5 U H v W K u I / < l a t e x i s h 1 \_ b 6 4 = " D A L I v k 0 f Q 7 w + G y p 3 E > B 8 n c V S g M F U 2 5 C R Z o Y K X 9 q N J u O r j z P m W T H / d < l a t e x i s h 1 \_ b 6 4 = " D A L I v k 0 f Q 7 w + G y p 3 E > B 8 n c V S g M F U 2 5 C R Z o Y K X 9 q N J u O r j z P m W T H / d < l a t e x i s h 1 \_ b 6 4 = " P O Y p M n A I z v 9 y U X 2 m B 8 > c V N S F H 7 + T C R 0 L g W Q r Z q w K J / 3 o 5 j d u f E k D G < l a t e x i s h 1 \_ b 6 4 = " o O R Q z B C S N 8 w P / k g F A L > 9 X c V 7 0 G + p Y 2 H J u m D E n d 5 q v y j U 3 r M I W K Z f T < l a t e x i s h 1 \_ b 6 4 = " T J 3 U F m D C n 5 I R O 8 / o G > A B 9 X c V L S N q 7 W + d u g 0 v k E Z w 2 r M p Q f H z K j P y < l a t e x i s h 1 \_ b 6 4 = " T J 3 U F m D C n 5 I R O 8 / o G > A B 9 X c V L S N q 7 W + d u g 0 v k E Z w 2 r M p Q f H z K j P y < l a t e x i s h 1 \_ b 6 4 = " N B j U I L D 2 Z n y O T P v g M > A 9 X c V S F 3 W u r Y d f q Q J m H R / + G o w 7 p k z K 8 C 0 E 5 < l a t e x i s h 1 \_ b 6 4 = " y 2 P F R K m 7 D 5 v I Z V L G k C n w > A B 9 X c S g N 0 z + o p Y H Q J u E 8 d q j U 3 O r M W f T / < l a t e x i s h 1 \_ b 6 4 = " w G Y A I + 0 H n W 2 p R u T k Z J 8 > B 9 X c V D L S g M F j m C 5 N 3 q f E K o v d U 7 O r y P z / Q < l a t e x i s h 1 \_ b 6 4 = " w G Y A I + 0 H n W 2 p R u T k Z J 8 > B 9 X c V D L S g M F j m C 5 N 3 q f E K o v d U 7 O r y P z / Q < l a t e x i s h 1 \_ b 6 4 = " I D u B 7 8 R G k z p Z 2 L g J K y c > A 9 X V S N F 3 W r Y d f P q Q O m H / + o T w v n j U M E 0 C 5 < l a t e x i s h 1 \_ b 6 4 = " F R C Z W f 0 w 3 L P O N k u S 5 g > A B 8 n c V D 2 p r d I o 7 K 9 J m Q z / + G M H Y j U E T X q v y < l a t e x i s h 1 \_ b 6 4 = " F R C Z W f 0 w 3 L P O N k u S 5 g > A B 8 n c V D 2 p r d I o 7 K 9 J m Q z / + G M H Y j U E T X q v y < l a t e x i s h 1 \_ b 6 4 = " F R C Z W f 0 w 3 L P O N k u S 5 g > A B 8 n c V D 2 p r d I o 7 K 9 J m Q z / + G M H Y j U E T X q v y < l a t e x i s h 1 \_ b 6 4 = " F R C Z W f 0 w 3 L P O N k u S 5 g > A B 8 n c V D 2 p r d I o 7 K 9 J m Q z / + G M H Y j U E T X q v y < l a t e x i s h 1 \_ b 6 4 = " G 5 Q D z E d o I y c V r M n 0 > A B 8 L S N F 2 p f Z u g k w R 7 Y / j K T q O U 9 3 J P C X W v H m + < l a t e x i s h 1 \_ b 6 4 = " G 5 Q D z E d o I y c V r M n 0 > A B 8 L S N F 2 p f Z u g k w R 7 Y / j K T q O U 9 3 J P C X W v H m + < l a t e x i s h 1 \_ b 6 4 = " G 5 Q D z E d o I y c V r M n 0 > A B 8 L S N F 2 p f Z u g k w R 7 Y / j K T q O U 9 3 J P C X W v H m + < l a t e x i s h 1 \_ b 6 4 = " G 5 Q D z E d o I y c V r M n 0 > A B 8 L S N F 2 p f Z u g k w R 7 Y / j K T q O U 9 3 J P C X W v H m + < l a t e x i s h 1 \_ b 6 4 = " j m p o 3 A K z 7 f T 0 M P y I 9 k > C n c V S w N B E J 2 L X F + R Z g D u Q d 5 r Y G / v W H q U O 8 < l a t e x i s h 1 \_ b 6 4 = " 2 u z + D q 0 Y N I f A 8 J / d 5 Q E c > C n V 9 S w B L X F P W R Z v o O 7 H j g p G M m K k U y r T 3 < l a t e x i s h 1 \_ b 6 4 = " 2 u z + D q 0 Y N I f A 8 J / d 5 Q E c > C n V 9 S w B L X F P W R Z v o O 7 H j g p G M m K k U y r T 3 < l a t e x i s h 1 \_ b 6 4 = " m L n 7 8 D U O Y 0 / g v T E d + M > A C c V J S N B 2 W Q y I Z 3 o F w j G q H X f u p K R r z P 9 k 5 < l a t e x i s h 1 \_ b 6 4 = " T G 3 n u O F w j N X Q r + M J A 5 p E > C c V 7 S B L f Z g o 9 z y / 2 D U W P m H I 8 d v Y q k K 0 R < l a t e x i s h 1 \_ b 6 4 = " R K P D f Z 0 N m H W T 7 j v y r M > A C n c V 9 S w B E 2 L X z F + p J U g o I 8 Q 5 u / k O 3 Y G d q < l a t e x i s h 1 \_ b 6 4 = " R K P D f Z 0 N m H W T 7 j v y r M > A C n c V 9 S w B E 2 L X z F + p J U g o I 8 Q 5 u / k O 3 Y G d q < l a t e x i s h 1 \_ b 6 4 = " L D O 2 Y Z H o v 3 P 5 y B N M K c > A C n V S F J r U d u g 8 R X w T W E 9 m Q / 0 z 7 G p q + j k I f < l a t e x i s h 1 \_ b 6 4 = " U z O G 9 m H 0 d n r W 2 P X y u j M 8 > A C c V D L S N F J 3 v q g I E B o Q Z p w T 7 Y f + k K R 5 / < l a t e x i s h 1 \_ b 6 4 = " U z O G 9 m H 0 d n r W 2 P X y u j M 8 > A C c V D L S N F J 3 v q g I E B o Q Z p w T 7 Y f + k K R 5 / < l a t e x i s h 1 \_ b 6 4 = " U z O G 9 m H 0 d n r W 2 P X y u j M 8 > A C c V D L S N F J 3 v q g I E B o Q Z p w T 7 Y f + k K R 5 / < l a t e x i s h 1 \_ b 6 4 = " U z O G 9 m H 0 d n r W 2 P X y u j M 8 > A C c V D L S N F J 3 v q g I E B o Q Z p w T 7 Y f + k K R 5 / ## Algorithm 1 Gradient calculation in FODE. Input: Initial value: za0 , zb0 ; parameters θ; integration time t ∈ T ; time step size: ∆t Output: Gradient dL dθ ; - 1: Forward Pass: - 2: for t := 0 to T do - 4: Delete zat , zbt and all intermediate activations; - 3: zat+∆t , zbt+∆t ←ODESolve (Eq.(9), [zat , zbt ], θ); - 5: t ← t + ∆t; - 6: end for - 7: return zaT and zbT . - 8: Backward: - 9: for t := T to 0 do - 10: Restore ˆzat-∆t and ˆzbt-∆t with zat and zbt by Eq.(10); - 11: Let zat-∆t = ˆzat-∆t , zbt-∆t = ˆzbt-∆t ; - 12: Compute ˆzat and ˆzbt by Eq.(9) ; - 13: Compute gradients ▽t by Eq. (13) and Eq. (14); - 15: Delete zat , zbt ; - 14: Update model gradients dL dθ ← dL dθ +▽t; - 16: t ← t - ∆t; - 17: end for - 18: return dL dθ . computed as |JF | = diag(I (zat )) and |JG| = diag(S(hbt )). While each element e in S(hbt ) and I (zat ) is greater than 0 after transformation exp(loge), Jacobians JF and JG are therefore invertible. Due to the reversibility of FODE, in the forward pass with ODE solver, we only save the output (zat+∆t ,zbt+∆t ) without the needs to store other variables and intermediate activations. In the backward stage, we first restore the input (ˆzat , ˆzbt ) from the output (zat+∆t , zbt+∆t ) by Eq. (10). Then, we perform one-time step forward pass to obtain the output (ˆzat+∆t ,ˆzbt+∆t ), and then calculate corresponding gradients ∂[ˆzat+∆t ,ˆzbt+∆t] ∂[ˆzan ,ˆzbn ] and ∂[ˆzat+∆t ,ˆzbt+∆t] ∂[θI,θJ ,θS,θK] . Subsequently, the gradients w.r.t. zt and θ of one-time step are computed as: The process of calculating accurate gradients in FODE is summarized in Algorithm 1, where ODESolve can be any numerical solutions, e.g., Euler, Runge-Kutta [Butcher and Wanner, 1996] and Dopris Solver [Ascher et al., 1997]. After FODE block, layer normalization [Ba et al., 2016] is employed to normalize the feature maps in the channel. Then we leverage SubPixel blocks [Liang et al., 2019] to upscale the hidden state from coarse-grained to fine-grained with upscaling factor N, which obtains the fine-grained hidden state output Hf after the convolutional layer. < l a t e x i s h 1 \_ b 6 4 = " d R 7 2 S w Z c Q Y V f L + M y 0 > A B n N 8 E J 3 U r q / 9 G P g o z T O F D I m u j X C W k p v K 5 H < l a t e x i s h 1 \_ b 6 4 = " d R 7 2 S w Z c Q Y V f L + M y 0 > A B n N 8 E J 3 U r q / 9 G P g o z T O F D I m u j X C W k p v K 5 H < l a t e x i s h 1 \_ b 6 4 = " d R 7 2 S w Z c Q Y V f L + M y 0 > A B n N 8 E J 3 U r q / 9 G P g o z T O F D I m u j X C W k p v K 5 H < l a t e x i s h 1 \_ b 6 4 = " d R 7 2 S w Z c Q Y V f L + M y 0 > A B n N 8 E J 3 U r q / 9 G P g o z T O F D I m u j X C W k p v K 5 H < l a t e x i s h 1 \_ b 6 4 = " d R 7 2 S w Z c Q Y V f L + M y 0 > A B n N 8 E J 3 U r q / 9 G P g o z T O F D I m u j X C W k p v K 5 H < l a t e x i s h 1 \_ b 6 4 = " d R 7 2 S w Z c Q Y V f L + M y 0 > A B n N 8 E J 3 U r q / 9 G P g o z T O F D I m u j X C W k p v K 5 H < l a t e x i s h 1 \_ b 6 4 = " d R 7 2 S w Z c Q Y V f L + M y 0 > A B n N 8 E J 3 U r q / 9 G P g o z T O F D I m u j X C W k p v K 5 H < l a t e x i s h 1 \_ b 6 4 = " d R 7 2 S w Z c Q Y V f L + M y 0 > A B n N 8 E J 3 U r q / 9 G P g o z T O F D I m u j X C W k p v K 5 H < l a t e x i s h 1 \_ b 6 4 = " d R 7 2 S w Z c Q Y V f L + M y 0 > A B n N 8 E J 3 U r q / 9 G P g o z T O F D I m u j X C W k p v K 5 H < l a t e x i s h 1 \_ b 6 4 = " d R 7 2 S w Z c Q Y V f L + M y 0 > A B n N 8 E J 3 U r q / 9 G P g o z T O F D I m u j X C W k p v K 5 H < l a t e x i s h 1 \_ b 6 4 = " d R 7 2 S w Z c Q Y V f L + M y 0 > A B n N 8 E J 3 U r q / 9 G P g o z T O F D I m u j X C W k p v K 5 H < l a t e x i s h 1 \_ b 6 4 = " d R 7 2 S w Z c Q Y V f L + M y 0 > A B n N 8 E J 3 U r q / 9 G P g o z T O F D I m u j X C W k p v K 5 H < l a t e x i s h 1 \_ b 6 4 = " H r 7 y n w g 8 F U S p K j q Y > A B X c V N + T W C 5 I o M 2 Q 9 m 0 3 / z R G d k u O f J Z v L D P E < l a t e x i s h 1 \_ b 6 4 = " H r 7 y n w g 8 F U S p K j q Y > A B X c V N + T W C 5 I o M 2 Q 9 m 0 3 / z R G d k u O f J Z v L D P E < l a t e x i s h 1 \_ b 6 4 = " H r 7 y n w g 8 F U S p K j q Y > A B X c V N + T W C 5 I o M 2 Q 9 m 0 3 / z R G d k u O f J Z v L D P E < l a t e x i s h 1 \_ b 6 4 = " H r 7 y n w g 8 F U S p K j q Y > A B X c V N + T W C 5 I o M 2 Q 9 m 0 3 / z R G d k u O f J Z v L D P E < l a t e x i s h 1 \_ b 6 4 = " c S w y O I o 9 + Q A R 5 g H M / u Y > B V D L F 3 j Z X W k d U 2 J 0 m N p C G f q P T v 7 r K n E z 8 < l a t e x i s h 1 \_ b 6 4 = " c S w y O I o 9 + Q A R 5 g H M / u Y > B V D L F 3 j Z X W k d U 2 J 0 m N p C G f q P T v 7 r K n E z 8 < l a t e x i s h 1 \_ b 6 4 = " c S w y O I o 9 + Q A R 5 g H M / u Y > B V D L F 3 j Z X W k d U 2 J 0 m N p C G f q P T v 7 r K n E z 8 < l a t e x i s h 1 \_ b 6 4 = " c S w y O I o 9 + Q A R 5 g H M / u Y > B V D L F 3 j Z X W k d U 2 J 0 m N p C G f q P T v 7 r K n E z 8 < l a t e x i s h 1 \_ b 6 4 = " X 2 g K r S R U W 7 Z N 8 + P I > A B 3 c V D L F u Y Q E n q C p T d M k z 5 j G / o w J m H v 9 0 y O f < l a t e x i s h 1 \_ b 6 4 = " X 2 g K r S R U W 7 Z N 8 + P I > A B 3 c V D L F u Y Q E n q C p T d M k z 5 j G / o w J m H v 9 0 y O f < l a t e x i s h 1 \_ b 6 4 = " X 2 g K r S R U W 7 Z N 8 + P I > A B 3 c V D L F u Y Q E n q C p T d M k z 5 j G / o w J m H v 9 0 y O f < l a t e x i s h 1 \_ b 6 4 = " X 2 g K r S R U W 7 Z N 8 + P I > A B 3 c V D L F u Y Q E n q C p T d M k z 5 j G / o w J m H v 9 0 y O f Figure 4: Feature fusion network. ![Image](https://ijcai.org/proceedings/2020/fulltext/180/image_000003_cc621056b021610f989283593d9649d8f0d7e66f0649baf743e3b35d9faf577b.png) ## 2.3 Feature Fusion Network It has been demonstrated that external features (e.g., wind speed, temperature, weather and holidays) affect the traffic distribution of the flows [Liang et al., 2019]. Here we also take these factors into consideration for improving performance. In addition to a simple fully connected network (FCN) for feature fusion, we use the proposed FODE blocks to improve the ability of estimating the feature influence. As shown in Figure 4, we obtain the coarse-grained feature map Ec ∈ RH×W + and fine-grained feature map Ef ∈ RNH×NW + by appending two FODE blocks: where SP indicates a SubPixel block used for upsampling. ## 2.4 Augmented FlowNormalization One of the main differences between FSR and image SR tasks is that there is a structural constraint in FSR, i.e., the amount of flows in the subregions should equal the flows in the respective superregion: where xcij (resp. xfi′ j ′ ) denotes the flow volume in a superregion (resp. subregion) of the coarse-grained (resp. finegrained) grid map. This is a straightforward method that normalizes the flow in the N 2 subregions to meet the constraint, a.k.a. N 2 -Normalization [Liang et al., 2019]. However, it ignores the influence of external factors on the subregions, and to address this problem, we propose an augmented N 2 normalization method which takes the distribution of external factors into account when constraining the flow, as illustrated in Figure 5. More specifically, we replace the xfi′ j ′ in a subregion with probability value αi′ ,j ′ , and we modify Eq. (17) as follows: where eci,j indicates the degree of influence of complex factors on the region ri,j. αi′ ,j ′ and βi′ ,j ′ denote the proportion of flow and factor influence assigned to the subregions from corresponding superregion, respectively. Now, we learn the joint distribution of factors' influence Dfe and flow Dfh to obtain the final flow distribution Dfπ . Here we present a Distribution Gating Mechanism (DGM) < l a t e x i s h 1 \_ b 6 4 = " J S 9 u r k d F 8 2 B c 0 n y q o > A 3 V N E U / L G P R 7 W m p Z I C j T z X M Y H + O Q D g 5 w f K v < l a t e x i s h 1 \_ b 6 4 = " J S 9 u r k d F 8 2 B c 0 n y q o > A 3 V N E U / L G P R 7 W m p Z I C j T z X M Y H + O Q D g 5 w f K v < l a t e x i s h 1 \_ b 6 4 = " J S 9 u r k d F 8 2 B c 0 n y q o > A 3 V N E U / L G P R 7 W m p Z I C j T z X M Y H + O Q D g 5 w f K v < l a t e x i s h 1 \_ b 6 4 = " J S 9 u r k d F 8 2 B c 0 n y q o > A 3 V N E U / L G P R 7 W m p Z I C j T z X M Y H + O Q D g 5 w f K v < l a t e x i s h 1 \_ b 6 4 = " S E 7 C D y / N I m T Z Y F 2 g r > A B 8 3 c V U q 9 L 0 G R W k p J P j z X M H n K o u O d w + v 5 f Q < l a t e x i s h 1 \_ b 6 4 = " S E 7 C D y / N I m T Z Y F 2 g r > A B 8 3 c V U q 9 L 0 G R W k p J P j z X M H n K o u O d w + v 5 f Q < l a t e x i s h 1 \_ b 6 4 = " S E 7 C D y / N I m T Z Y F 2 g r > A B 8 3 c V U q 9 L 0 G R W k p J P j z X M H n K o u O d w + v 5 f Q < l a t e x i s h 1 \_ b 6 4 = " S E 7 C D y / N I m T Z Y F 2 g r > A B 8 3 c V U q 9 L 0 G R W k p J P j z X M H n K o u O d w + v 5 f Q < l a t e x i s h 1 \_ b 6 4 = " C I r E n q j g c k 8 v N O K F Y > A B Z 7 S P o 3 W 0 G Q y V d 5 w H 2 m T J L / R f z U X p 9 D + u M < l a t e x i s h 1 \_ b 6 4 = " v C T L H W z R o 8 c k n P 7 + K d Q U > A B Z S g N F I u M / X 2 w G r p O E D J 9 Y m 3 j 5 q f y V 0 < l a t e x i s h 1 \_ b 6 4 = " v C T L H W z R o 8 c k n P 7 + K d Q U > A B Z S g N F I u M / X 2 w G r p O E D J 9 Y m 3 j 5 q f y V 0 < l a t e x i s h 1 \_ b 6 4 = " F g + j y D G V U T 8 d S 3 u Y z R B v > A n c N E J r q / 9 L 0 W X 5 m 2 Z K M w P O k 7 H o Q f I p C < l a t e x i s h 1 \_ b 6 4 = " C I r E n q j g c k 8 v N O K F Y > A B Z 7 S P o 3 W 0 G Q y V d 5 w H 2 m T J L / R f z U X p 9 D + u M < l a t e x i s h 1 \_ b 6 4 = " v C T L H W z R o 8 c k n P 7 + K d Q U > A B Z S g N F I u M / X 2 w G r p O E D J 9 Y m 3 j 5 q f y V 0 < l a t e x i s h 1 \_ b 6 4 = " v C T L H W z R o 8 c k n P 7 + K d Q U > A B Z S g N F I u M / X 2 w G r p O E D J 9 Y m 3 j 5 q f y V 0 < l a t e x i s h 1 \_ b 6 4 = " F g + j y D G V U T 8 d S 3 u Y z R B v > A n c N E J r q / 9 L 0 W X 5 m 2 Z K M w P O k 7 H o Q f I p C < l a t e x i s h 1 \_ b 6 4 = " L I f K U p S 0 X F Q M B R P c n D > A 9 V g N E O y r Y o H j G / J k d Z v w T q 7 u 3 8 W + C 5 2 z m < l a t e x i s h 1 \_ b 6 4 = " L I f K U p S 0 X F Q M B R P c n D > A 9 V g N E O y r Y o H j G / J k d Z v w T q 7 u 3 8 W + C 5 2 z m < l a t e x i s h 1 \_ b 6 4 = " L I f K U p S 0 X F Q M B R P c n D > A 9 V g N E O y r Y o H j G / J k d Z v w T q 7 u 3 8 W + C 5 2 z m < l a t e x i s h 1 \_ b 6 4 = " L I f K U p S 0 X F Q M B R P c n D > A 9 V g N E O y r Y o H j G / J k d Z v w T q 7 u 3 8 W + C 5 2 z m < l a t e x i s h 1 \_ b 6 4 = " X j u E p + S 2 v W K P 7 L O g > A B 9 c V D N J y r f U Y o M H G / I z k d n Z w R T q 3 8 Q m 5 C 0 F < l a t e x i s h 1 \_ b 6 4 = " X j u E p + S 2 v W K P 7 L O g > A B 9 c V D N J y r f U Y o M H G / I z k d n Z w R T q 3 8 Q m 5 C 0 F < l a t e x i s h 1 \_ b 6 4 = " X j u E p + S 2 v W K P 7 L O g > A B 9 c V D N J y r f U Y o M H G / I z k d n Z w R T q 3 8 Q m 5 C 0 F < l a t e x i s h 1 \_ b 6 4 = " X j u E p + S 2 v W K P 7 L O g > A B 9 c V D N J y r f U Y o M H G / I z k d n Z w R T q 3 8 Q m 5 C 0 F < l a t e x i s h 1 \_ b 6 4 = " f y j E p Z S G V U X D H 2 5 z B d > A 7 c C g N F I P o 3 r L W M q 0 Y Q u m J v k + / w 9 K R O n T 8 < l a t e x i s h 1 \_ b 6 4 = " T n X y o 5 J d p M 9 u I 2 V E > A B 7 c Z C S g N F P L m 8 W G 3 H O w z K Q r D j Y U + / f k q v R 0 < l a t e x i s h 1 \_ b 6 4 = " T n X y o 5 J d p M 9 u I 2 V E > A B 7 c Z C S g N F P L m 8 W G 3 H O w z K Q r D j Y U + / f k q v R 0 < l a t e x i s h 1 \_ b 6 4 = " j N X k p r 9 Y Q G 0 o W c f g d u 7 > A B V D L S E O z M H 8 v R P C Z w 2 m / + n y q F 3 K T 5 U I J < l a t e x i s h 1 \_ b 6 4 = " W J n 7 X K 9 o v 3 I D + u G A k d > B c V L S g N E O y r U j F 8 R z 5 m Y 2 Z p / P T H q M 0 f C Q w < l a t e x i s h 1 \_ b 6 4 = " r B p L P + q K V 2 9 O v D F H E > A 7 X c C T N z M I Q j S R Y 0 5 f y o 8 g G J / k U u d Z n 3 m w W < l a t e x i s h 1 \_ b 6 4 = " r B p L P + q K V 2 9 O v D F H E > A 7 X c C T N z M I Q j S R Y 0 5 f y o 8 g G J / k U u d Z n 3 m w W < l a t e x i s h 1 \_ b 6 4 = " m Q p g D v O + K T Y C W Z n u I d 8 E > A B 7 X c V 9 S w N J 3 z M q 2 k y j 5 G f / 0 R L r o U H F P < l a t e x i s h 1 \_ b 6 4 = " 5 D w Y n A N j T P X k g 7 H 3 + 2 C > B c V L S E O y r o 9 Z M K u F z 0 G v W p f / I U d Q q R J m 8 < l a t e x i s h 1 \_ b 6 4 = " A C X w 2 Q p z 7 Z r m q u 9 0 D k B g > H c V L S N E O y + o M K F G v W d J / 5 f U 3 Y P R I 8 j T n < l a t e x i s h 1 \_ b 6 4 = " A C X w 2 Q p z 7 Z r m q u 9 0 D k B g > H c V L S N E O y + o M K F G v W d J / 5 f U 3 Y P R I 8 j T n < l a t e x i s h 1 \_ b 6 4 = " X R C q I 5 r Q 0 z v A Y L H d T U > B c V N S 8 E J 3 / 9 Z p j y 2 D + g W n u w F K k O P G 7 o m f M < l a t e x i s h 1 \_ b 6 4 = " C I r E n q j g c k 8 v N O K F Y > A B Z 7 S P o 3 W 0 G Q y V d 5 w H 2 m T J L / R f z U X p 9 D + u M < l a t e x i s h 1 \_ b 6 4 = " v C T L H W z R o 8 c k n P 7 + K d Q U > A B Z S g N F I u M / X 2 w G r p O E D J 9 Y m 3 j 5 q f y V 0 < l a t e x i s h 1 \_ b 6 4 = " v C T L H W z R o 8 c k n P 7 + K d Q U > A B Z S g N F I u M / X 2 w G r p O E D J 9 Y m 3 j 5 q f y V 0 < l a t e x i s h 1 \_ b 6 4 = " F g + j y D G V U T 8 d S 3 u Y z R B v > A n c N E J r q / 9 L 0 W X 5 m 2 Z K M w P O k 7 H o Q f I p C < l a t e x i s h 1 \_ b 6 4 = " L K g N Y 2 G T J 3 / z n w y I U 7 m F > A B + X c V S 8 E W O Q P R C 0 d M u p k D 9 Z j o r v H 5 f q < l a t e x i s h 1 \_ b 6 4 = " L K g N Y 2 G T J 3 / z n w y I U 7 m F > A B + X c V S 8 E W O Q P R C 0 d M u p k D 9 Z j o r v H 5 f q < l a t e x i s h 1 \_ b 6 4 = " L K g N Y 2 G T J 3 / z n w y I U 7 m F > A B + X c V S 8 E W O Q P R C 0 d M u p k D 9 Z j o r v H 5 f q < l a t e x i s h 1 \_ b 6 4 = " L K g N Y 2 G T J 3 / z n w y I U 7 m F > A B + X c V S 8 E W O Q P R C 0 d M u p k D 9 Z j o r v H 5 f q Figure 5: Illustration of AN 2 -Normalization. ![Image](https://ijcai.org/proceedings/2020/fulltext/180/image_000004_dae067054b526819bf903f5b07f78699bedfaf962f7988debde9093a40fd7b5b.png) to explicitly capture the dynamic spatial dependence between the two distributions: where σ denotes distribution gate, Dfe and Dfh are generated by N 2 -Normalization with input Ef and Hf . AN 2 - Normalization not only inherits all the advantages of N 2 - Normalization (e.g., parallelizable computation and no extra parameters), but also bridges gaps between external factors and urban flows when normalizing the values of subregions. Fine-grained flow inference. At this point, we are ready to generate the fine-grained flow ̂Xf from the learned joint distribution as: where Xcup ∈ RNH×NW + is produced by nearest-neighbor upsampling with scaling factor N. Optimization. Finally, we minimize the mean squared error between the ground truth fine-grained flow map Xf and the model inferred output ̂Xf : where θ represents all learnable parameters in our model. ## 3 Experiments We now present the details of our experimental evaluations. ## 3.1 Experimental Settings Datasets. We evaluate all the methods using two real-world urban flow datasets: (1) TaxiBJ [Liang et al., 2019] - a taxi GPS data including taxi flows from July 1, 2014 to October 31, 2014; and (2) BikeNYC - collected from an open website 1 which contains data from January 1, 2019 to June 30, 2019. Each dataset contains two sub-datasets: coarse-grained and fine-grained flows. A detailed description of the datasets is shown in Table 1. Note that the scaling factors are different for the two datasets, i.e., N = 4 and N = 2 for TaxiBJ and BikeNYC, respectively. Baselines. We compare FODE with the following 10 baselines: - Mean Partition (Mean) evenly distributes the flow volume in each subregion. 1 https://www.citibikenyc.com/system-data Table 1: Statistics of datasets. | Dataset | TaxiBJ | BikeNYC | |-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------| | Time range | 7/1/2013-10/31/2013 | 1/1/2019-3/31/2019 | | Time interval Coarse-grained size Fine-grained size Upscaling factor (N) Latitude range Longitude range External Factors Temperature / ◦ C Wind speed / mph Weather conditions Holidays | 30 minutes 32×32 128×128 4 39.82◦N - 39.99◦N 116.26◦E-116.49◦E (meteorology, time (e.g., [-24.6,41.0] [0,48.6] types (e.g., Rainy,Sunny) 18 | 1 hour 40×16 80×32 2 40.65◦N-40.81◦N 74.00◦E-74.07◦E hourofday, dayofweek)) \ \ \ 10 | - Historical Average (HA) models historical average data to predict the flow in the subregion. - SRCNN [Dong et al., 2015] is a classic model for image SR based on CNNs. - ESPCN [Shi et al., 2016] introduces a sub-pixel convolutional layer for image SR. - VDSR [Kim et al., 2016] employs residual networks to solve the slow convergence and limited representation problems in SR. - SRResNet [Ledig et al., 2017] is a ResNet-based variant of VDSR model, which allows stacking more network layers. - OISR [He et al., 2019] is an ODE-inspired SR model using Runge-Kutta (RK3) method [Butcher and Wanner, 1996] as ODE solver. - NODE [Chen et al., 2018] is a neural ODE method that uses adjoint method for discretization and optimal control of ODEs. - ANODE [Gholami et al., 2019] is an improved version of NODE by introducing checking points for alleviating the incorrect gradient issue. - UrbanFM [Liang et al., 2019] infers fine-grained urban flow with external factors by stacking ResNet-based neural networks. Metrics. We evaluate different methods with three widely used metrics: Root Mean Squared Error (RMSE), Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE). Implementation details. Adam [Kingma and Ba, 2014] is adopted to train FODE with batch size 16 and learning rate e-4 . We leverage Dopri5 numerical method, which can adaptively choose the step size, as ODESovle in FODE. FODE consists of 128 channels and 1 ODE block. We also present a simplified version S-FODE which contains 64 channels while other components are the same as FODE. During training, we halve the learning rate and perform a test on the validation set every 20 epochs. For all image SR models numerical methods (NODE and ANODE), we use N 2 -normalization to constrain the inferred flow distribution. We note that the details of other network settings are described in the source-implementation 2 . 2 https://github.com/Anewnoob/FODE Table 2: Performance comparisons on TaxiBJ and BikeNYC. | Datasets - Method | TaxiBJ - RMSE | TaxiBJ - MAE | TaxiBJ - MAPE | BikeNYC - RMSE | BikeNYC - MAE | BikeNYC - MAPE | |---------------------|-----------------|----------------|-----------------|------------------|-----------------|------------------| | Mean | 20.918 | 12.019 | 4.469 | 4.554 | 1.379 | 0.678 | | HA | 4.741 | 2.214 | 0.332 | 2.414 | 0.676 | 0.216 | | SRCNN | 4.297 | 2.491 | 0.714 | 2.385 | 0.821 | 0.433 | | ESPCN | 4.206 | 2.497 | 0.732 | 2.356 | 0.825 | 0.441 | | VDSR | 4.159 | 2.213 | 0.467 | 2.344 | 0.734 | 0.285 | | SRResNet | 4.164 | 2.457 | 0.713 | 2.355 | 0.741 | 0.430 | | OISR | 4.126 | 2.134 | 0.421 | 2.318 | 0.683 | 0.246 | | NODE | 4.058 | 2.125 | 0.408 | 2.309 | 0.671 | 0.243 | | ANODE | 3.967 | 2.043 | 0.351 | 2.246 | 0.635 | 0.217 | | UrbanFM | 3.951 | 2.011 | 0.327 | 2.234 | 0.627 | 0.209 | | S-FODE | 3.941 | 2.001 | 0.322 | 1.951 | 0.517 | 0.129 | | FODE | 3.860 | 1.963 | 0.313 | 1.916 | 0.512 | 0.115 | ## 3.2 Results Overall performance. Table 2 reports the FSR results of all the methods, demonstrating that FODE and its variant outperform all baselines on all metrics across both datasets. Taking TaxiBJ for example, FODE yields 14.9%, 19.1%, and 41.6% improvement on average in terms of RMSE, MAE, and MAPE, respectively. In a relatively smaller area of NYC, FODE achieves larger improvement for inferring the bicycle flow. The performance gain of FODE over baselines demonstrates the effectiveness of continuous-time ODEs, which provides an alternative view of improving fine-grained flow inference. Comparison analysis. Image SR methods are usually not comparable even with N 2 normalization, due to the structural constraints and the influence of external factors in FSR application. This implies that FSR requires specific model design that seamlessly taking constraints and factors into account. As a specifically tailored FSR model, UrbanFM achieves best performance among baselines. However, as a ResNetbased model, it models the urban flow in a discrete manner by stacking deep neural networks, which could be problematic since urban flow inherently can be viewed as a continuous dynamic system. OISR, NODE and ANODE are numerical methods tailored for FSR. OISR uses RK-block as the network structure, which suffers from a huge number of parameters and numerical errors that significantly affect the performance. Additionally, it requires a significant amount of memory for storing intermediate quantities during backpropagation (cf. Table 3). NODE, in contrast, uses adjoint method for solving ODEs, which only needs O( ˜ L) memory as FODE (note that ˜ L = 1 in FODE), while ANODE requires O( ˜ L)+O(N) memory. Nevertheless, the dynamics of either the hidden state or the adjoint might be unstable, which incurs inaccurate gradient computation, as we analyzed in previous section. The performance gain of FODE over these numerical methods indicates that our method estimates the gradients more accurately, due to the introduced affine coupling layer while incurring no memory cost. Memory efficiency. In addition to FSR performance, FODE has a significant memory and parameter efficiency, compared to ResNet-based models. Table 3 outlines the memory cost and parameters required for different methods (we omit other image SR methods due to the similar architectures as SRResNet). In particular, FODE requires only 1/3× parameters of UrbanFM and reduces the memory cost to O(1). It is worthwhile to note that a simplified version S-FODE contains only 0.7M parameters while still outperforming all the baseline methods on FSR task. Table 3: Comparisons of parameters and memory cost. C: the number of channels; L( ˜ L): the number of ResNet (ODE blocks); H: the number of layers in each ResNet; N: the number of function evaluations. | Method | C | #Params (M) | memory | time | |----------|-----|---------------|--------------|--------| | SRResNet | 128 | 5.5 | O(LH) | O(LH) | | OISR | 128 | 15.7 | O(LH) | O(LH) | | NODE | 128 | 2.1 | O( ˜ L) | O(N) | | ANODE | 128 | 2.4 | O( ˜ L)+O(N) | O(N) | | UrbanFM | 128 | 6.2 | O(LH) | O(LH) | | S-FODE | 64 | 0.7 | O( ˜ L) | O(N) | | FODE | 128 | 2.1 | O( ˜ L) | O(N) | Ability of factor fusion. External factors play important roles and should be carefully considered in FSR models. Figure 6(a) and 6(b) compare the influence of factors learned by UrbanFM and FODE, respectively. UrbanFM uses a FCN to fuse external factors, which is too weak to correlate complex factors with flow distributions. For example, the impact of factors concentrates in a smaller area for UrbanFM - i.e., two main roads are more affected by external factors than other regions. In contrast, FODE estimates the influence of factors by evenly distributing the external influences and is therefore more robust. This can be further verified by the results shown in Figure 6(c) and Figure 6(d), where we observe that FODE consistently converges while UrbanFM, surprisingly, achieves best performance using temperature only, rather than all the factors. Error analysis. Figure 7(a) shows the inference errors of UrbanFM and FODE on a data sample, where a brighter pixel indicates a larger error. To better visualize the quality of inference, we select four subregions (A, B, C, and D), from which we clearly see that the flow inference by FODE performs better than UrbanFM in crowded areas. Similarly, Figure 7(b) depicts the overall inference error using two different normalization schemes. We observe that in most areas, especially in E, F, G subregions, the distribution generated by AN 2 -normalization is closer to the ground-truth, which demonstrates that the proposed AN 2 method is a more effective way of constraining the flow distributions. This improvement can be attributed to jointly modeling the distribution of external factors and urban flows in AN 2 , compared to simply constraining the flows in subregions in N 2 -normalization. Accuracy vs. efficiency. Another merit of FODE is that it allows to balance the trade off between the flow inference accuracy and the computational overhead, by varying the number of function evaluations N. As shown in Figure 8, the more function evaluated in the forward pass, the lower the MSE loss. Accordingly, the time required for training the model increases linearly with the number of function evaluations. This result is important since downstream applications UrbanFM Figure 6: Analysis of external factor fusion. ![Image](https://ijcai.org/proceedings/2020/fulltext/180/image_000005_d7c2218cf24ccbf46e7c7f9eaa8418efc5b48f2780e72eeafcf346c4319ddcd3.png) ![Image](https://ijcai.org/proceedings/2020/fulltext/180/image_000006_012af777e98dc56644669eb71130b071488f74153978a30261eb5f82350ee93f.png) ![Image](https://ijcai.org/proceedings/2020/fulltext/180/image_000007_3f5b00f65b3a75400b5bab864d22b943f9d36a13472d7f7f2043176f7d280d3f.png) (a) Comparison of inference errors. ![Image](https://ijcai.org/proceedings/2020/fulltext/180/image_000008_3ce5a96078f502ab191c947a9d4a84e44643176e72226afb096c172d612be6c6.png) -Normalization ![Image](https://ijcai.org/proceedings/2020/fulltext/180/image_000009_df64b14bbcec1e2c4dc5b953d12a2a9765497f4acecacc0486eb3d1640154e83.png) -Normalization AN 2 (b) N 2 vs. AN 2 . Figure 7: Inference error visualization. ![Image](https://ijcai.org/proceedings/2020/fulltext/180/image_000010_1b00e6c79b04edeacdb5856e504664dc1bbdd153305c24f581a3e934dcae575d.png) Figure 8: Tradeoff between training time and inference accuracy. may make flexible solutions by conciliating inference accuracy with computational cost. ## 4 Conclusion We proposed a novel method FODE for inferring fine-grained urban flow. FODE learns urban flow distribution through a new ODEs parameterized by affine coupling neural networks alleviating the numerical instability gradient computation issue, which allows for both memory and model parameter savings. In addition, it is capable of explicitly providing more flexible prediction performance by adaptively balancing prediction accuracy and computation overheads. Furthermore, we believe that FODE is a more general ODE-based architecture and can be better exploited for time series prediction or other fine-grained inference tasks such as single image SR [Cai et al., 2019; Wang et al., 2019] and air quality inference [Liu et al., 2019a]. ## Acknowledgments This work was supported by National Natural Science Foundation of China (Grant No.61602097) and NSF grant CNS 1646107. ## References 1. [Ascher et al., 1997] Uri M Ascher, Steven J Ruuth, and Raymond J Spiteri. Implicit-explicit runge-kutta methods for time-dependent partial differential equations. Applied Numerical Mathematics, 25(2-3):151-167, 1997. 2. [Ba et al., 2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Ge- offrey E Hinton. Layer normalization. arXiv: 1607.06450, 2016. 3. [Butcher and Wanner, 1996] John Charles Butcher and Ger- hard Wanner. Runge-kutta methods: some historical notes. Applied Numerical Mathematics, 22(1-3):113-151, 1996. 4. [Cai et al., 2019] Jianrui Cai, Shuhang Gu, Radu Timofte, Lei Zhang, and et al. Ntire 2019 challenge on real image super-resolution -methods and results. In CVPR Chal- lenge, pages 2211-2223, 2019. 5. [Chen et al., 2018] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. In NeuIPS, pages 6572-6583, 2018. 6. [De Brouwer et al., 2019] Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes -contin- uous modeling of sporadically-observed time series. In NeurIPS, pages 7377-7388, 2019. 7. [Dinh et al., 2015] Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non-linear independent compo- nents estimation. In ICLR, 2015. 8. [Dinh et al., 2017] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In ICLR, 2017. 9. [Dong et al., 2015] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. TPAMI, 38(2):295-307, 2015. 10. [E, 2017] Weinan E. A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1-11, 2017. 11. [Gholami et al., 2019] Amir Gholami, Kurt Keutzer, and George Biros. Anode: Unconditionally accurate memory- efficient gradients for neural odes. In IJCAI, pages 730- 736, 2019. 12. [Grathwohl et al., 2019] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. FFJORD: free-form continuous dynamics for scalable re- versible generative models. In ICLR, 2019. 13. [He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016. 14. [He et al., 2019] Xiangyu He, Zitao Mo, Peisong Wang, Yang Liu, Mingyuan Yang, and Jian Cheng. Ode-inspired network design for single image super-resolution. In CVPR, pages 1732-1741, 2019. 15. [Heinonen and Lähdesmäki, 2019] Markus Heinonen and Harri Lähdesmäki. ODE 2 VAE: Deep Generative Second Order ODEs with Bayesian Neural Networks. In NeurIPS, pages 13412-13421, 2019. 16. [Kim et al., 2016] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, pages 1646-1654, 2016. 17. [Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv: 1412.6980, 2014. 18. [Kingma and Dhariwal, 2018] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 con- volutions. In NIPS, pages 10236-10245, 2018. 19. [Ledig et al., 2017] Christian Ledig, Lucas Theis, Ferenc Huszár, Caballero, and et al. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR, pages 105-114, 2017. 20. [Liang et al., 2019] Yuxuan Liang, Kun Ouyang, Lin Jing, Sijie Ruan, Ye Liu, Junbo Zhang, David S Rosenblum, and Yu Zheng. Urbanfm: Inferring fine-grained urban flows. In KDD, pages 3132-3142, 2019. 21. [Liu et al., 2019a] Ning Liu, Rui Ma, Yue Wang, and Lin Zhang. Inferring fine-grained air pollution map via a spa- tiotemporal super-resolution scheme. In UbiqComp, pages 498-504, 2019. 22. [Liu et al., 2019b] Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsieh. Neural SDE: Stabi- lizing neural ode networks with stochastic noise. arXiv: 1906.02355v1, 2019. 23. [Lu et al., 2018] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridg- ing deep architectures and numerical differential equa- tions. In ICML, pages 3282-3291, 2018. 24. [Poli et al., 2019] Michael Poli, Stefano Massaroli, Jun- young Park, Atsushi Yamashita, Hajime Asama, and Jinkyoo Park. Graph neural ordinary differential equa- tions. arXiv: 1911.07532v1, 2019. 25. [Rubanova et al., 2019] Yulia Rubanova, Ricky T.Q. Chen, and David Duvenaud. Latent ordinary differential equa- tions for irregularly-sampled time series. In NeurIPS, pages 5321-5331, 2019. 26. [Shi et al., 2016] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single im- age and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, pages 1874- 1883, 2016. 27. [Wang et al., 2019] Zhihao Wang, Jian Chen, and Steven CH Hoi. Deep learning for image super-resolution: A survey. arXiv:1902.06068, 2019. 28. [Zhang et al., 2017] Junbo Zhang, Yu Zheng, and Dekang Qi. Deep spatio-temporal residual networks for citywide crowd flows prediction. In AAAI, pages 1655-1661, 2017. 29. [Zhang et al., 2018] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In CVPR, pages 2472-2481, 2018. 30. [Zhang et al., 2019] Tianjun Zhang, Zhewei Yao, Amir Gho- lami, Kurt Keutzer, Joseph Gonzalez, George Biros, and Michael Mahoney. Anodev2: A coupled neural ode evolu- tion framework. In NeurIPS, 2019.