Enhancing Urban Flow Maps via Neural ODEs

1 Introduction

Urban flow super-resolution (FSR) aims at inferring finegrained crowd flows in a city based on coarse-grained observations. As a variant of image SR in the traffic domain [Cai et al., 2019; Wang et al., 2019] it has practical significance in urban planning and traffic monitoring. Despite its close relationship to image SR, FSR has certain constraints the most important one being the structural constraint - i.e., the sum of the flow volumes in surrounding regions in the inferred fine-grained map strictly equals that of their corresponding superregion in the original map. However, in practice, the distribution of the flows in a given region is affected by many external factors, e.g., weather, time-of-day, etc.

A recent work [Liang et al., 2019] addresses FSR problem based on the residual networks [He et al., 2016] and uses a simple normalization scheme to constrain the structural distribution of flows. However, the proposed architecture heavily relies on empirically stacking deep neural networks and, consequently, lacks principles to guide the design of effective and interpretable FSR networks. Furthermore, it pays no attention to the computational overheads as the network goes deeper, which requires significantly more memory cost and network parameters - making it hard to be optimized and prone to be overfitting.

Recently, there has been a growing interest in bridging the gap between neural networks and dynamic systems [E, 2017; Lu et al., 2018]. In particular, a recent study [Chen et al., 2018] has shown that ResNet [He et al., 2016] can be interpreted as discretized Neural Ordinary Differential Equations (NODE), which provides a new perspective of improving stability and trainability of neural networks. For example, an additional state (called adjoint) is used in [Chen et al., 2018] to solve the ODEs. In this way, it does not need to store the intermediate states of the forward pass, resulting in O(1) memory cost in each ODE block. A few studies have been proposed to improve the performance of NODE [Liu et al., 2019b] and/or to apply NODE in different domains such as graph neural networks [Poli et al., 2019], time series learning [Rubanova et al., 2019; De Brouwer et al., 2019] and generative models [Heinonen and L¨ahdesm¨aki, 2019; Grathwohl et al., 2019]. However, the dynamics of either the hidden state or the adjoint are usually unstable, due to the numerical instability of solving the backward ODEs. ANODE [Gholami et al., 2019] and its variant [Zhang et al., 2019] improve the robustness and generalization of the adjoint method by adding a few of the intermediate states from the forward pass however, these cannot fundamentally solve the incorrect gradient problem.

In this work, we introduce a new SR method - FODE (FSR with ODEs) for fine-grained urban flow inference. FODE is a more general neural ODE architecture that addresses the numerical instability problem of previous methods, while not incurring extra memory cost. The main idea of FODE is to incorporate an affine coupling layer in each ODE block to avoid the inaccurate gradient issue. The input to a FODE block can then be accurately reconstructed from its outputs without the need of storing intermediate states. We show theoretically and empirically that the proposed FODE model can be used to learn spatial correlations among urban regions and the influence of external features. The main contributions of this work can be summarized as:

• We present a new neural ODE framework that can accu-

rately compute the gradients in each block.

• Our novel FSR method requires significantly less memory for fine-grained flow inference compared with ResNet-based deep neural network.
• The augmented normalization scheme disentangles the influence of external factors on different regions.
• Experiments conducted on two real-world datasets demonstrate that our FODE not only outperforms strong baselines on FSR task, but also provides a flexible balance between computational cost and inference accuracy.

2 Architecture andMethodology

We now present the problem settings and discuss in detail the main aspects of the proposed FODE approach.

We assume a grid-based segmentation, similar to [Zhang et al., 2017], which divides a city map M into H×W spatial grid-cells based on their geographical locations, i.e., M = {rij}H×M. Each cell rij corresponds to a spatial region - the ith row and the j th column of M. Let X ∈ RH×W + be the flow map at a given time, where each entry xij ∈ R+ denotes the volume of the flow in the region rij .

Definition 1 (Urban Flow Super-Resolution (FSR)). Given a coarse-grained flow map Xc , an upscaling factor N as well as the external factors E (e.g., weather, events, etc.), the FSR task is to learn a model F mapping Xc into a fine-grained flow map ̂Xf ∈ RNH×NW + :

where θ represent all learnable parameters.

Figure 1 illustrates an example of converting a coarsegrained flow map to a fine-grained flow map in the city of Beijing. The coarse-grained flow map (20km×20km) consists of a total of 32×32 cells, each of which denotes a superregion. In the fine-grained flow map, there are 128×128 subregions in total. At the top of Figure 1, a superregion is composed of N 2 subregions (N = 4 here). Note that the sum of the flow volume in the N 2 subregions (top right) is equal to the flow volume of the corresponding superregion (top left).

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

C

W

Q

G

p

Y

S

R

r

U

I

+

m

g

F

M

>

A

B

H

c

V

N

8

E

J

3

q

/

9

L

0

v

u

w

K

2

7

T

d

j

f

X

o

5

D

z

n

P

y

Z

k

O

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

C

W

Q

G

p

Y

S

R

r

U

I

+

m

g

F

M

>

A

B

H

c

V

N

8

E

J

3

q

/

9

L

0

v

u

w

K

2

7

T

d

j

f

X

o

5

D

z

n

P

y

Z

k

O

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

C

W

Q

G

p

Y

S

R

r

U

I

+

m

g

F

M

>

A

B

H

c

V

N

8

E

J

3

q

/

9

L

0

v

u

w

K

2

7

T

d

j

f

X

o

5

D

z

n

P

y

Z

k

O

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

C

W

Q

G

p

Y

S

R

r

U

I

+

m

g

F

M

>

A

B

H

c

V

N

8

E

J

3

q

/

9

L

0

v

u

w

K

2

7

T

d

j

f

X

o

5

D

z

n

P

y

Z

k

O

2.1 Architecture

Figure 2 illustrates the overall architecture of FODE, which consists of three main components:

• FODE block which extracts spatial correlations among flow regions with the proposed new ODEs.
• Feature fusion network (FFN) which combines the FODE blocks and fully connected networks for fusing the influence of external factors.
• Fine-grained flow inference (FFI) which leverages an augmented N 2 -normalization scheme (AN 2 ) to estimate the distribution of both flows and external factors for generating the fine-grained flow maps.

2.2 FODE Block

ResNet [He et al., 2016] and its variants have been widely employed for image super-resolution [Ledig et al., 2017; Zhang et al., 2018]. Recently, [Liang et al., 2019] exploited residual block for regional correlation learning and fine-grained urban flow inference, which share the same idea of image SR. Suppose we obtain the high-dimensional hidden state z0 from the coarse-grained flow Xc by some convolutional layers. The residual block then transforms the hidden states zn according to:

While achieving promising results on SR tasks, residual networks still confront the problem of intensive computation and require to tune a huge amount of parameters. There is a growing interest in bridging the gap between discrete neural networks and continuous dynamic systems. For example, the iterative updates in the above residual block can be viewed as a discretization of a continuous ODE operator [Lu et al., 2018; Chen et al., 2018], if we take time t ∈ T as a continuous variable:

Solving the above ODEs requires computing the gradients through backpropagation, which would be memory prohibitive if the time is reduced into infinite steps. In principle, it requires O(N) cost to store the all N intermediate activations. NODE [Chen et al., 2018] proposes to address this problem with the adjoint method. Considering a loss function L, an additional state referred to as the adjoint a (t) = ∂L/∂z (t) can be used to compute the gradient w.r.t. parameters θ:

where we only need to store the final state z(T ). Hence this strategy successfully reduces the memory cost. However, it also introduces other problems - the inaccurate gradient and

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

X

O

S

H

f

y

v

j

M

V

c

P

8

>

A

B

9

D

L

N

F

2

r

Z

d

u

g

k

w

R

I

Q

W

+

0

T

J

/

C

n

m

3

o

7

E

q

z

G

Y

K

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

X

O

S

H

f

y

v

j

M

V

c

P

8

>

A

B

9

D

L

N

F

2

r

Z

d

u

g

k

w

R

I

Q

W

+

0

T

J

/

C

n

m

3

o

7

E

q

z

G

Y

K

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

X

O

S

H

f

y

v

j

M

V

c

P

8

>

A

B

9

D

L

N

F

2

r

Z

d

u

g

k

w

R

I

Q

W

+

0

T

J

/

C

n

m

3

o

7

E

q

z

G

Y

K

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

p

X

O

S

H

f

y

v

j

M

V

c

P

8

>

A

B

9

D

L

N

F

2

r

Z

d

u

g

k

w

R

I

Q

W

+

0

T

J

/

C

n

m

3

o

7

E

q

z

G

Y

K

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

m

X

0

9

B

d

v

P

o

y

3

D

f

W

g

>

A

8

c

V

L

S

N

F

u

r

Y

U

Q

E

R

+

C

T

M

j

O

/

n

K

w

z

p

G

I

k

7

2

q

J

Z

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

m

X

0

9

B

d

v

P

o

y

3

D

f

W

g

>

A

8

c

V

L

S

N

F

u

r

Y

U

Q

E

R

+

C

T

M

j

O

/

n

K

w

z

p

G

I

k

7

2

q

J

Z

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

m

X

0

9

B

d

v

P

o

y

3

D

f

W

g

>

A

8

c

V

L

S

N

F

u

r

Y

U

Q

E

R

+

C

T

M

j

O

/

n

K

w

z

p

G

I

k

7

2

q

J

Z

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

m

X

0

9

B

d

v

P

o

y

3

D

f

W

g

>

A

8

c

V

L

S

N

F

u

r

Y

U

Q

E

R

+

C

T

M

j

O

/

n

K

w

z

p

G

I

k

7

2

q

J

Z

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

N

o

O

Z

f

E

S

L

P

U

8

/

Q

Y

+

n

G

I

>

A

B

3

c

V

D

F

W

u

r

X

R

C

m

g

d

T

M

j

9

K

w

z

q

2

v

7

5

k

J

p

y

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

N

o

O

Z

f

E

S

L

P

U

8

/

Q

Y

+

n

G

I

>

A

B

3

c

V

D

F

W

u

r

X

R

C

m

g

d

T

M

j

9

K

w

z

q

2

v

7

5

k

J

p

y

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

N

o

O

Z

f

E

S

L

P

U

8

/

Q

Y

+

n

G

I

>

A

B

3

c

V

D

F

W

u

r

X

R

C

m

g

d

T

M

j

9

K

w

z

q

2

v

7

5

k

J

p

y

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

N

o

O

Z

f

E

S

L

P

U

8

/

Q

Y

+

n

G

I

>

A

B

3

c

V

D

F

W

u

r

X

R

C

m

g

d

T

M

j

9

K

w

z

q

2

v

7

5

k

J

p

y

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Q

d

m

M

y

7

8

X

P

q

c

G

D

C

W

H

f

K

A

I

Y

U

>

B

/

3

V

L

S

N

F

J

2

v

O

T

r

k

o

g

n

R

w

E

5

9

+

j

Z

0

u

p

z

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Q

d

m

M

y

7

8

X

P

q

c

G

D

C

W

H

f

K

A

I

Y

U

>

B

/

3

V

L

S

N

F

J

2

v

O

T

r

k

o

g

n

R

w

E

5

9

+

j

Z

0

u

p

z

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Q

d

m

M

y

7

8

X

P

q

c

G

D

C

W

H

f

K

A

I

Y

U

>

B

/

3

V

L

S

N

F

J

2

v

O

T

r

k

o

g

n

R

w

E

5

9

+

j

Z

0

u

p

z

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Q

d

m

M

y

7

8

X

P

q

c

G

D

C

W

H

f

K

A

I

Y

U

>

B

/

3

V

L

S

N

F

J

2

v

O

T

r

k

o

g

n

R

w

E

5

9

+

j

Z

0

u

p

z

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

P

J

3

I

8

Q

u

S

z

H

9

k

L

d

>

A

B

c

V

N

E

W

X

O

C

o

0

g

U

D

2

w

m

7

G

/

v

T

y

5

r

+

M

n

j

Z

p

q

K

F

R

f

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

P

J

3

I

8

Q

u

S

z

H

9

k

L

d

>

A

B

c

V

N

E

W

X

O

C

o

0

g

U

D

2

w

m

7

G

/

v

T

y

5

r

+

M

n

j

Z

p

q

K

F

R

f

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

P

J

3

I

8

Q

u

S

z

H

9

k

L

d

>

A

B

c

V

N

E

W

X

O

C

o

0

g

U

D

2

w

m

7

G

/

v

T

y

5

r

+

M

n

j

Z

p

q

K

F

R

f

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

P

J

3

I

8

Q

u

S

z

H

9

k

L

d

>

A

B

c

V

N

E

W

X

O

C

o

0

g

U

D

2

w

m

7

G

/

v

T

y

5

r

+

M

n

j

Z

p

q

K

F

R

f

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

y

m

P

u

5

r

v

C

U

J

K

9

g

W

L

R

F

c

>

A

B

+

X

V

D

S

N

2

p

f

Z

d

o

q

Q

Y

M

3

T

E

/

z

0

G

w

n

7

H

I

8

j

k

O

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

y

m

P

u

5

r

v

C

U

J

K

9

g

W

L

R

F

c

>

A

B

+

X

V

D

S

N

2

p

f

Z

d

o

q

Q

Y

M

3

T

E

/

z

0

G

w

n

7

H

I

8

j

k

O

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

y

m

P

u

5

r

v

C

U

J

K

9

g

W

L

R

F

c

>

A

B

+

X

V

D

S

N

2

p

f

Z

d

o

q

Q

Y

M

3

T

E

/

z

0

G

w

n

7

H

I

8

j

k

O

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

y

m

P

u

5

r

v

C

U

J

K

9

g

W

L

R

F

c

>

A

B

+

X

V

D

S

N

2

p

f

Z

d

o

q

Q

Y

M

3

T

E

/

z

0

G

w

n

7

H

I

8

j

k

O

numerical instability. Taking Eq. (2) for example, we compare the difference between the two calculated gradients: (1) Calculate the gradient w.r.t. zn by Eq. (6):

(2) Calculate the exact gradient w.r.t. zn by chain rule:

As can be seen, the second terms of Eq. (7) and Eq. (8) are different, which, therefore, makes the state a (tn) ̸= an. In other words, the adjoint method leads to inaccurate and unstable gradient ∂L/∂θ - compared to direct backpropagation that computes correct and stable gradient, but suffers a prohibitive memory cost. ANODE and its variant [Gholami et al., 2019; Zhang et al., 2019] mitigate this problem by splitting a block into time batches and stores in memory a few intermediate states from the forward pass, which, in a sense, is a compromise between time and storage.

In this work, we bridge this gap by introducing an affine coupling layer in the computations of ODEs, to avoid the inaccurate gradient issue while only requiring O(1) memory in each block. Affine coupling layer referred to a family of neural network whose forward function is a bijective mapping and has been widely used in invertible generative models [Dinh et al., 2015; Dinh et al., 2017; Kingma and Dhariwal, 2018], where the input to a bijective block can be accurately reconstructed from its outputs.

As illustrated in Figure 3, we divide the input zt (let zt = z (t)) into two parts of same size zat , zbt ∈ RC/2×H×W - where C is the number of channels. In the forward pass functions in each FODE block with time step size ∆t, we have:

where hat and hbt are the intermediate states, zat+∆t and zbt+∆t denote the outputs, I, J , S and K are differentiable neural networks. The reverse computations are therefore:

To ensure the reversibility of Eq. (10), all elements in S (hbt ) and I (zat ) need to be greater than zero, which can be met by making a simple transformation for each element e as exp(loge), e ∈ I or e ∈ S. Then we have following property which ensures that the affine transformations in the forward pass in Eq. (9) are reversible:

Proposition 1. The forward pass functions Eq. (9) in the FODE block is reversible as long as each element of I(zat ) and S(hbt ) is non-zero.

Proof. Let JF and JG be the Jacobians of the transformations in Eq. (9), which can be computed as:

where I and O are identity and zero matrix, respectively, and diag(·) denotes the diagonal matrix whose diagonal elements correspond to the elements in I (zat ) or S(hbt ). Since JF and JG are lower triangular matrices, their determinants are

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

d

V

Z

j

r

z

k

g

D

0

X

K

7

8

O

f

n

S

>

A

B

9

c

J

N

E

2

W

o

M

F

T

p

R

m

Q

5

+

q

/

3

w

U

P

H

Y

y

v

u

G

I

L

C

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

o

r

k

q

9

f

X

z

n

+

c

B

M

D

A

K

W

d

y

0

>

V

L

S

g

N

E

J

G

Z

Y

2

C

7

Q

3

T

j

U

v

w

H

R

/

O

I

p

u

5

m

P

F

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

o

r

k

q

9

f

X

z

n

+

c

B

M

D

A

K

W

d

y

0

>

V

L

S

g

N

E

J

G

Z

Y

2

C

7

Q

3

T

j

U

v

w

H

R

/

O

I

p

u

5

m

P

F

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

r

T

E

2

K

H

f

q

N

Z

/

w

J

5

c

o

g

>

A

B

9

X

V

C

7

D

z

O

Q

S

F

R

Y

U

0

v

+

8

m

W

P

y

n

I

L

k

j

d

3

G

u

p

M

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

T

E

K

8

j

z

0

B

d

Y

k

5

g

9

q

v

>

A

X

c

V

C

7

S

N

F

L

3

r

M

Z

J

m

U

w

I

H

p

P

D

/

G

Q

f

n

O

+

W

2

u

R

y

o

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Q

j

D

T

P

k

M

v

o

N

X

7

2

S

Y

d

g

A

>

B

9

c

V

L

F

r

q

p

0

E

y

C

w

z

u

K

m

5

8

+

O

G

Z

J

H

I

R

/

3

n

W

U

f

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

Q

j

D

T

P

k

M

v

o

N

X

7

2

S

Y

d

g

A

>

B

9

c

V

L

F

r

q

p

0

E

y

C

w

z

u

K

m

5

8

+

O

G

Z

J

H

I

R

/

3

n

W

U

f

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

P

I

q

G

v

E

o

p

X

F

L

5

A

y

n

M

Q

g

>

B

9

c

V

C

7

T

w

0

r

J

U

Y

W

K

z

u

R

S

d

f

/

8

3

+

D

O

j

2

m

N

k

Z

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

W

u

Y

M

k

V

L

O

5

n

J

+

o

G

N

z

7

D

>

A

B

8

c

S

F

3

r

y

K

9

g

P

p

m

Z

E

f

q

X

/

j

C

w

H

I

U

2

R

v

Q

T

d

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

W

u

Y

M

k

V

L

O

5

n

J

+

o

G

N

z

7

D

>

A

B

8

c

S

F

3

r

y

K

9

g

P

p

m

Z

E

f

q

X

/

j

C

w

H

I

U

2

R

v

Q

T

d

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

0

W

u

Y

M

k

V

L

O

5

n

J

+

o

G

N

z

7

D

>

A

B

8

c

S

F

3

r

y

K

9

g

P

p

m

Z

E

f

q

X

/

j

C

w

H

I

U

2

R

v

Q

T

d

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

c

H

r

V

S

k

d

g

D

W

v

B

M

2

p

Y

O

U

>

A

8

n

L

N

F

3

u

7

K

C

f

y

m

Q

J

R

+

G

/

o

T

P

w

5

I

Z

z

X

q

j

9

E

0

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

o

z

7

r

C

E

j

g

J

G

N

9

d

D

m

2

Q

3

0

>

A

B

8

n

c

V

S

F

L

X

+

q

O

y

T

Y

R

Z

p

M

P

k

w

f

5

U

H

v

W

K

u

I

/

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

D

A

L

I

v

k

0

f

Q

7

w

+

G

y

p

3

E

>

B

8

n

c

V

S

g

M

F

U

2

5

C

R

Z

o

Y

K

X

9

q

N

J

u

O

r

j

z

P

m

W

T

H

/

d

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

D

A

L

I

v

k

0

f

Q

7

w

+

G

y

p

3

E

>

B

8

n

c

V

S

g

M

F

U

2

5

C

R

Z

o

Y

K

X

9

q

N

J

u

O

r

j

z

P

m

W

T

H

/

d

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

P

O

Y

p

M

n

A

I

z

v

9

y

U

X

2

m

B

8

>

c

V

N

S

F

H

7

+

T

C

R

0

L

g

W

Q

r

Z

q

w

K

J

/

3

o

5

j

d

u

f

E

k

D

G

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

o

O

R

Q

z

B

C

S

N

8

w

P

/

k

g

F

A

L

>

9

X

c

V

7

0

G

+

p

Y

2

H

J

u

m

D

E

n

d

5

q

v

y

j

U

3

r

M

I

W

K

Z

f

T

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

T

J

3

U

F

m

D

C

n

5

I

R

O

8

/

o

G

>

A

B

9

X

c

V

L

S

N

q

7

W

+

d

u

g

0

v

k

E

Z

w

2

r

M

p

Q

f

H

z

K

j

P

y

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

T

J

3

U

F

m

D

C

n

5

I

R

O

8

/

o

G

>

A

B

9

X

c

V

L

S

N

q

7

W

+

d

u

g

0

v

k

E

Z

w

2

r

M

p

Q

f

H

z

K

j

P

y

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

N

B

j

U

I

L

D

2

Z

n

y

O

T

P

v

g

M

>

A

9

X

c

V

S

F

3

W

u

r

Y

d

f

q

Q

J

m

H

R

/

+

G

o

w

7

p

k

z

K

8

C

0

E

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

y

2

P

F

R

K

m

7

D

5

v

I

Z

V

L

G

k

C

n

w

>

A

B

9

X

c

S

g

N

0

z

+

o

p

Y

H

Q

J

u

E

8

d

q

j

U

3

O

r

M

W

f

T

/

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

w

G

Y

A

I

+

0

H

n

W

2

p

R

u

T

k

Z

J

8

>

B

9

X

c

V

D

L

S

g

M

F

j

m

C

5

N

3

q

f

E

K

o

v

d

U

7

O

r

y

P

z

/

Q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

w

G

Y

A

I

+

0

H

n

W

2

p

R

u

T

k

Z

J

8

>

B

9

X

c

V

D

L

S

g

M

F

j

m

C

5

N

3

q

f

E

K

o

v

d

U

7

O

r

y

P

z

/

Q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

I

D

u

B

7

8

R

G

k

z

p

Z

2

L

g

J

K

y

c

>

A

9

X

V

S

N

F

3

W

r

Y

d

f

P

q

Q

O

m

H

/

+

o

T

w

v

n

j

U

M

E

0

C

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

F

R

C

Z

W

f

0

w

3

L

P

O

N

k

u

S

5

g

>

A

B

8

n

c

V

D

2

p

r

d

I

o

7

K

9

J

m

Q

z

/

+

G

M

H

Y

j

U

E

T

X

q

v

y

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

F

R

C

Z

W

f

0

w

3

L

P

O

N

k

u

S

5

g

>

A

B

8

n

c

V

D

2

p

r

d

I

o

7

K

9

J

m

Q

z

/

+

G

M

H

Y

j

U

E

T

X

q

v

y

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

F

R

C

Z

W

f

0

w

3

L

P

O

N

k

u

S

5

g

>

A

B

8

n

c

V

D

2

p

r

d

I

o

7

K

9

J

m

Q

z

/

+

G

M

H

Y

j

U

E

T

X

q

v

y

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

F

R

C

Z

W

f

0

w

3

L

P

O

N

k

u

S

5

g

>

A

B

8

n

c

V

D

2

p

r

d

I

o

7

K

9

J

m

Q

z

/

+

G

M

H

Y

j

U

E

T

X

q

v

y

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

G

5

Q

D

z

E

d

o

I

y

c

V

r

M

n

0

>

A

B

8

L

S

N

F

2

p

f

Z

u

g

k

w

R

7

Y

/

j

K

T

q

O

U

9

3

J

P

C

X

W

v

H

m

+

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

G

5

Q

D

z

E

d

o

I

y

c

V

r

M

n

0

>

A

B

8

L

S

N

F

2

p

f

Z

u

g

k

w

R

7

Y

/

j

K

T

q

O

U

9

3

J

P

C

X

W

v

H

m

+

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

G

5

Q

D

z

E

d

o

I

y

c

V

r

M

n

0

>

A

B

8

L

S

N

F

2

p

f

Z

u

g

k

w

R

7

Y

/

j

K

T

q

O

U

9

3

J

P

C

X

W

v

H

m

+

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

G

5

Q

D

z

E

d

o

I

y

c

V

r

M

n

0

>

A

B

8

L

S

N

F

2

p

f

Z

u

g

k

w

R

7

Y

/

j

K

T

q

O

U

9

3

J

P

C

X

W

v

H

m

+

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

j

m

p

o

3

A

K

z

7

f

T

0

M

P

y

I

9

k

>

C

n

c

V

S

w

N

B

E

J

2

L

X

F

+

R

Z

g

D

u

Q

d

5

r

Y

G

/

v

W

H

q

U

O

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

2

u

z

+

D

q

0

Y

N

I

f

A

8

J

/

d

5

Q

E

c

>

C

n

V

9

S

w

B

L

X

F

P

W

R

Z

v

o

O

7

H

j

g

p

G

M

m

K

k

U

y

r

T

3

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

2

u

z

+

D

q

0

Y

N

I

f

A

8

J

/

d

5

Q

E

c

>

C

n

V

9

S

w

B

L

X

F

P

W

R

Z

v

o

O

7

H

j

g

p

G

M

m

K

k

U

y

r

T

3

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

m

L

n

7

8

D

U

O

Y

0

/

g

v

T

E

d

+

M

>

A

C

c

V

J

S

N

B

2

W

Q

y

I

Z

3

o

F

w

j

G

q

H

X

f

u

p

K

R

r

z

P

9

k

5

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

T

G

3

n

u

O

F

w

j

N

X

Q

r

+

M

J

A

5

p

E

>

C

c

V

7

S

B

L

f

Z

g

o

9

z

y

/

2

D

U

W

P

m

H

I

8

d

v

Y

q

k

K

0

R

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

R

K

P

D

f

Z

0

N

m

H

W

T

7

j

v

y

r

M

>

A

C

n

c

V

9

S

w

B

E

2

L

X

z

F

+

p

J

U

g

o

I

8

Q

5

u

/

k

O

3

Y

G

d

q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

R

K

P

D

f

Z

0

N

m

H

W

T

7

j

v

y

r

M

>

A

C

n

c

V

9

S

w

B

E

2

L

X

z

F

+

p

J

U

g

o

I

8

Q

5

u

/

k

O

3

Y

G

d

q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

D

O

2

Y

Z

H

o

v

3

P

5

y

B

N

M

K

c

>

A

C

n

V

S

F

J

r

U

d

u

g

8

R

X

w

T

W

E

9

m

Q

/

0

z

7

G

p

q

+

j

k

I

f

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

z

O

G

9

m

H

0

d

n

r

W

2

P

X

y

u

j

M

8

>

A

C

c

V

D

L

S

N

F

J

3

v

q

g

I

E

B

o

Q

Z

p

w

T

7

Y

f

+

k

K

R

5

/

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

z

O

G

9

m

H

0

d

n

r

W

2

P

X

y

u

j

M

8

>

A

C

c

V

D

L

S

N

F

J

3

v

q

g

I

E

B

o

Q

Z

p

w

T

7

Y

f

+

k

K

R

5

/

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

z

O

G

9

m

H

0

d

n

r

W

2

P

X

y

u

j

M

8

>

A

C

c

V

D

L

S

N

F

J

3

v

q

g

I

E

B

o

Q

Z

p

w

T

7

Y

f

+

k

K

R

5

/

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

U

z

O

G

9

m

H

0

d

n

r

W

2

P

X

y

u

j

M

8

>

A

C

c

V

D

L

S

N

F

J

3

v

q

g

I

E

B

o

Q

Z

p

w

T

7

Y

f

+

k

K

R

5

/

Algorithm 1 Gradient calculation in FODE.

Input: Initial value: za0 , zb0 ; parameters θ; integration time t ∈ T ; time step size: ∆t Output: Gradient dL

dθ ;

• 1: Forward Pass:
• 2: for t := 0 to T do
• 4: Delete zat , zbt and all intermediate activations;
• 3: zat+∆t , zbt+∆t ←ODESolve (Eq.(9), [zat , zbt ], θ);
• 5: t ← t + ∆t;
• 6: end for
• 7: return zaT and zbT .
• 8: Backward:
• 9: for t := T to 0 do
• 10: Restore ˆzat-∆t and ˆzbt-∆t with zat and zbt by Eq.(10);
• 11: Let zat-∆t = ˆzat-∆t , zbt-∆t = ˆzbt-∆t ;
• 12: Compute ˆzat and ˆzbt by Eq.(9) ;
• 13: Compute gradients ▽t by Eq. (13) and Eq. (14);
• 15: Delete zat , zbt ;
• 14: Update model gradients dL dθ ← dL dθ +▽t;
• 16: t ← t - ∆t;
• 17: end for
• 18: return dL dθ .

computed as |JF | = diag(I (zat )) and |JG| = diag(S(hbt )). While each element e in S(hbt ) and I (zat ) is greater than 0 after transformation exp(loge), Jacobians JF and JG are therefore invertible.

Due to the reversibility of FODE, in the forward pass with ODE solver, we only save the output (zat+∆t ,zbt+∆t ) without the needs to store other variables and intermediate activations. In the backward stage, we first restore the input (ˆzat , ˆzbt ) from the output (zat+∆t , zbt+∆t ) by Eq. (10). Then, we perform one-time step forward pass to obtain the output (ˆzat+∆t ,ˆzbt+∆t ), and then calculate corresponding gradients ∂[ˆzat+∆t ,ˆzbt+∆t] ∂[ˆzan ,ˆzbn ] and ∂[ˆzat+∆t ,ˆzbt+∆t] ∂[θI,θJ ,θS,θK] . Subsequently, the gradients w.r.t. zt and θ of one-time step are computed as:

The process of calculating accurate gradients in FODE is summarized in Algorithm 1, where ODESolve can be any numerical solutions, e.g., Euler, Runge-Kutta [Butcher and Wanner, 1996] and Dopris Solver [Ascher et al., 1997]. After FODE block, layer normalization [Ba et al., 2016] is employed to normalize the feature maps in the channel. Then we leverage SubPixel blocks [Liang et al., 2019] to upscale the hidden state from coarse-grained to fine-grained with upscaling factor N, which obtains the fine-grained hidden state output Hf after the convolutional layer.

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

d

R

7

2

S

w

Z

c

Q

Y

V

f

L

+

M

y

0

>

A

B

n

N

8

E

J

3

U

r

q

/

9

G

P

g

o

z

T

O

F

D

I

m

u

j

X

C

W

k

p

v

K

5

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

d

R

7

2

S

w

Z

c

Q

Y

V

f

L

+

M

y

0

>

A

B

n

N

8

E

J

3

U

r

q

/

9

G

P

g

o

z

T

O

F

D

I

m

u

j

X

C

W

k

p

v

K

5

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

d

R

7

2

S

w

Z

c

Q

Y

V

f

L

+

M

y

0

>

A

B

n

N

8

E

J

3

U

r

q

/

9

G

P

g

o

z

T

O

F

D

I

m

u

j

X

C

W

k

p

v

K

5

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

d

R

7

2

S

w

Z

c

Q

Y

V

f

L

+

M

y

0

>

A

B

n

N

8

E

J

3

U

r

q

/

9

G

P

g

o

z

T

O

F

D

I

m

u

j

X

C

W

k

p

v

K

5

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

d

R

7

2

S

w

Z

c

Q

Y

V

f

L

+

M

y

0

>

A

B

n

N

8

E

J

3

U

r

q

/

9

G

P

g

o

z

T

O

F

D

I

m

u

j

X

C

W

k

p

v

K

5

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

d

R

7

2

S

w

Z

c

Q

Y

V

f

L

+

M

y

0

>

A

B

n

N

8

E

J

3

U

r

q

/

9

G

P

g

o

z

T

O

F

D

I

m

u

j

X

C

W

k

p

v

K

5

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

d

R

7

2

S

w

Z

c

Q

Y

V

f

L

+

M

y

0

>

A

B

n

N

8

E

J

3

U

r

q

/

9

G

P

g

o

z

T

O

F

D

I

m

u

j

X

C

W

k

p

v

K

5

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

d

R

7

2

S

w

Z

c

Q

Y

V

f

L

+

M

y

0

>

A

B

n

N

8

E

J

3

U

r

q

/

9

G

P

g

o

z

T

O

F

D

I

m

u

j

X

C

W

k

p

v

K

5

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

d

R

7

2

S

w

Z

c

Q

Y

V

f

L

+

M

y

0

>

A

B

n

N

8

E

J

3

U

r

q

/

9

G

P

g

o

z

T

O

F

D

I

m

u

j

X

C

W

k

p

v

K

5

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

d

R

7

2

S

w

Z

c

Q

Y

V

f

L

+

M

y

0

>

A

B

n

N

8

E

J

3

U

r

q

/

9

G

P

g

o

z

T

O

F

D

I

m

u

j

X

C

W

k

p

v

K

5

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

d

R

7

2

S

w

Z

c

Q

Y

V

f

L

+

M

y

0

>

A

B

n

N

8

E

J

3

U

r

q

/

9

G

P

g

o

z

T

O

F

D

I

m

u

j

X

C

W

k

p

v

K

5

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

d

R

7

2

S

w

Z

c

Q

Y

V

f

L

+

M

y

0

>

A

B

n

N

8

E

J

3

U

r

q

/

9

G

P

g

o

z

T

O

F

D

I

m

u

j

X

C

W

k

p

v

K

5

H

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

r

7

y

n

w

g

8

F

U

S

p

K

j

q

Y

>

A

B

X

c

V

N

+

T

W

C

5

I

o

M

2

Q

9

m

0

3

/

z

R

G

d

k

u

O

f

J

Z

v

L

D

P

E

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

r

7

y

n

w

g

8

F

U

S

p

K

j

q

Y

>

A

B

X

c

V

N

+

T

W

C

5

I

o

M

2

Q

9

m

0

3

/

z

R

G

d

k

u

O

f

J

Z

v

L

D

P

E

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

r

7

y

n

w

g

8

F

U

S

p

K

j

q

Y

>

A

B

X

c

V

N

+

T

W

C

5

I

o

M

2

Q

9

m

0

3

/

z

R

G

d

k

u

O

f

J

Z

v

L

D

P

E

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

H

r

7

y

n

w

g

8

F

U

S

p

K

j

q

Y

>

A

B

X

c

V

N

+

T

W

C

5

I

o

M

2

Q

9

m

0

3

/

z

R

G

d

k

u

O

f

J

Z

v

L

D

P

E

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

c

S

w

y

O

I

o

9

+

Q

A

R

5

g

H

M

/

u

Y

>

B

V

D

L

F

3

j

Z

X

W

k

d

U

2

J

0

m

N

p

C

G

f

q

P

T

v

7

r

K

n

E

z

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

c

S

w

y

O

I

o

9

+

Q

A

R

5

g

H

M

/

u

Y

>

B

V

D

L

F

3

j

Z

X

W

k

d

U

2

J

0

m

N

p

C

G

f

q

P

T

v

7

r

K

n

E

z

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

c

S

w

y

O

I

o

9

+

Q

A

R

5

g

H

M

/

u

Y

>

B

V

D

L

F

3

j

Z

X

W

k

d

U

2

J

0

m

N

p

C

G

f

q

P

T

v

7

r

K

n

E

z

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

c

S

w

y

O

I

o

9

+

Q

A

R

5

g

H

M

/

u

Y

>

B

V

D

L

F

3

j

Z

X

W

k

d

U

2

J

0

m

N

p

C

G

f

q

P

T

v

7

r

K

n

E

z

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

X

2

g

K

r

S

R

U

W

7

Z

N

8

+

P

I

>

A

B

3

c

V

D

L

F

u

Y

Q

E

n

q

C

p

T

d

M

k

z

5

j

G

/

o

w

J

m

H

v

9

0

y

O

f

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

X

2

g

K

r

S

R

U

W

7

Z

N

8

+

P

I

>

A

B

3

c

V

D

L

F

u

Y

Q

E

n

q

C

p

T

d

M

k

z

5

j

G

/

o

w

J

m

H

v

9

0

y

O

f

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

X

2

g

K

r

S

R

U

W

7

Z

N

8

+

P

I

>

A

B

3

c

V

D

L

F

u

Y

Q

E

n

q

C

p

T

d

M

k

z

5

j

G

/

o

w

J

m

H

v

9

0

y

O

f

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

X

2

g

K

r

S

R

U

W

7

Z

N

8

+

P

I

>

A

B

3

c

V

D

L

F

u

Y

Q

E

n

q

C

p

T

d

M

k

z

5

j

G

/

o

w

J

m

H

v

9

0

y

O

f

2.3 Feature Fusion Network

It has been demonstrated that external features (e.g., wind speed, temperature, weather and holidays) affect the traffic distribution of the flows [Liang et al., 2019]. Here we also take these factors into consideration for improving performance. In addition to a simple fully connected network (FCN) for feature fusion, we use the proposed FODE blocks to improve the ability of estimating the feature influence. As shown in Figure 4, we obtain the coarse-grained feature map Ec ∈ RH×W + and fine-grained feature map Ef ∈ RNH×NW + by appending two FODE blocks:

where SP indicates a SubPixel block used for upsampling.

2.4 Augmented FlowNormalization

One of the main differences between FSR and image SR tasks is that there is a structural constraint in FSR, i.e., the amount of flows in the subregions should equal the flows in the respective superregion:

where xcij (resp. xfi′ j ′ ) denotes the flow volume in a superregion (resp. subregion) of the coarse-grained (resp. finegrained) grid map. This is a straightforward method that normalizes the flow in the N 2 subregions to meet the constraint, a.k.a. N 2 -Normalization [Liang et al., 2019]. However, it ignores the influence of external factors on the subregions, and to address this problem, we propose an augmented N 2 normalization method which takes the distribution of external factors into account when constraining the flow, as illustrated in Figure 5.

More specifically, we replace the xfi′ j ′ in a subregion with probability value αi′ ,j ′ , and we modify Eq. (17) as follows:

where eci,j indicates the degree of influence of complex factors on the region ri,j. αi′ ,j ′ and βi′ ,j ′ denote the proportion of flow and factor influence assigned to the subregions from corresponding superregion, respectively.

Now, we learn the joint distribution of factors' influence Dfe and flow Dfh to obtain the final flow distribution Dfπ . Here we present a Distribution Gating Mechanism (DGM)

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

S

9

u

r

k

d

F

8

2

B

c

0

n

y

q

o

>

A

3

V

N

E

U

/

L

G

P

R

7

W

m

p

Z

I

C

j

T

z

X

M

Y

H

+

O

Q

D

g

5

w

f

K

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

S

9

u

r

k

d

F

8

2

B

c

0

n

y

q

o

>

A

3

V

N

E

U

/

L

G

P

R

7

W

m

p

Z

I

C

j

T

z

X

M

Y

H

+

O

Q

D

g

5

w

f

K

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

S

9

u

r

k

d

F

8

2

B

c

0

n

y

q

o

>

A

3

V

N

E

U

/

L

G

P

R

7

W

m

p

Z

I

C

j

T

z

X

M

Y

H

+

O

Q

D

g

5

w

f

K

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

J

S

9

u

r

k

d

F

8

2

B

c

0

n

y

q

o

>

A

3

V

N

E

U

/

L

G

P

R

7

W

m

p

Z

I

C

j

T

z

X

M

Y

H

+

O

Q

D

g

5

w

f

K

v

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

S

E

7

C

D

y

/

N

I

m

T

Z

Y

F

2

g

r

>

A

B

8

3

c

V

U

q

9

L

0

G

R

W

k

p

J

P

j

z

X

M

H

n

K

o

u

O

d

w

+

v

5

f

Q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

S

E

7

C

D

y

/

N

I

m

T

Z

Y

F

2

g

r

>

A

B

8

3

c

V

U

q

9

L

0

G

R

W

k

p

J

P

j

z

X

M

H

n

K

o

u

O

d

w

+

v

5

f

Q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

S

E

7

C

D

y

/

N

I

m

T

Z

Y

F

2

g

r

>

A

B

8

3

c

V

U

q

9

L

0

G

R

W

k

p

J

P

j

z

X

M

H

n

K

o

u

O

d

w

+

v

5

f

Q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

S

E

7

C

D

y

/

N

I

m

T

Z

Y

F

2

g

r

>

A

B

8

3

c

V

U

q

9

L

0

G

R

W

k

p

J

P

j

z

X

M

H

n

K

o

u

O

d

w

+

v

5

f

Q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

C

I

r

E

n

q

j

g

c

k

8

v

N

O

K

F

Y

>

A

B

Z

7

S

P

o

3

W

0

G

Q

y

V

d

5

w

H

2

m

T

J

L

/

R

f

z

U

X

p

9

D

+

u

M

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

v

C

T

L

H

W

z

R

o

8

c

k

n

P

7

+

K

d

Q

U

>

A

B

Z

S

g

N

F

I

u

M

/

X

2

w

G

r

p

O

E

D

J

9

Y

m

3

j

5

q

f

y

V

0

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

v

C

T

L

H

W

z

R

o

8

c

k

n

P

7

+

K

d

Q

U

>

A

B

Z

S

g

N

F

I

u

M

/

X

2

w

G

r

p

O

E

D

J

9

Y

m

3

j

5

q

f

y

V

0

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

F

g

+

j

y

D

G

V

U

T

8

d

S

3

u

Y

z

R

B

v

>

A

n

c

N

E

J

r

q

/

9

L

0

W

X

5

m

2

Z

K

M

w

P

O

k

7

H

o

Q

f

I

p

C

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

C

I

r

E

n

q

j

g

c

k

8

v

N

O

K

F

Y

>

A

B

Z

7

S

P

o

3

W

0

G

Q

y

V

d

5

w

H

2

m

T

J

L

/

R

f

z

U

X

p

9

D

+

u

M

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

v

C

T

L

H

W

z

R

o

8

c

k

n

P

7

+

K

d

Q

U

>

A

B

Z

S

g

N

F

I

u

M

/

X

2

w

G

r

p

O

E

D

J

9

Y

m

3

j

5

q

f

y

V

0

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

v

C

T

L

H

W

z

R

o

8

c

k

n

P

7

+

K

d

Q

U

>

A

B

Z

S

g

N

F

I

u

M

/

X

2

w

G

r

p

O

E

D

J

9

Y

m

3

j

5

q

f

y

V

0

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

F

g

+

j

y

D

G

V

U

T

8

d

S

3

u

Y

z

R

B

v

>

A

n

c

N

E

J

r

q

/

9

L

0

W

X

5

m

2

Z

K

M

w

P

O

k

7

H

o

Q

f

I

p

C

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

I

f

K

U

p

S

0

X

F

Q

M

B

R

P

c

n

D

>

A

9

V

g

N

E

O

y

r

Y

o

H

j

G

/

J

k

d

Z

v

w

T

q

7

u

3

8

W

+

C

5

2

z

m

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

I

f

K

U

p

S

0

X

F

Q

M

B

R

P

c

n

D

>

A

9

V

g

N

E

O

y

r

Y

o

H

j

G

/

J

k

d

Z

v

w

T

q

7

u

3

8

W

+

C

5

2

z

m

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

I

f

K

U

p

S

0

X

F

Q

M

B

R

P

c

n

D

>

A

9

V

g

N

E

O

y

r

Y

o

H

j

G

/

J

k

d

Z

v

w

T

q

7

u

3

8

W

+

C

5

2

z

m

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

I

f

K

U

p

S

0

X

F

Q

M

B

R

P

c

n

D

>

A

9

V

g

N

E

O

y

r

Y

o

H

j

G

/

J

k

d

Z

v

w

T

q

7

u

3

8

W

+

C

5

2

z

m

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

X

j

u

E

p

+

S

2

v

W

K

P

7

L

O

g

>

A

B

9

c

V

D

N

J

y

r

f

U

Y

o

M

H

G

/

I

z

k

d

n

Z

w

R

T

q

3

8

Q

m

5

C

0

F

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

X

j

u

E

p

+

S

2

v

W

K

P

7

L

O

g

>

A

B

9

c

V

D

N

J

y

r

f

U

Y

o

M

H

G

/

I

z

k

d

n

Z

w

R

T

q

3

8

Q

m

5

C

0

F

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

X

j

u

E

p

+

S

2

v

W

K

P

7

L

O

g

>

A

B

9

c

V

D

N

J

y

r

f

U

Y

o

M

H

G

/

I

z

k

d

n

Z

w

R

T

q

3

8

Q

m

5

C

0

F

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

X

j

u

E

p

+

S

2

v

W

K

P

7

L

O

g

>

A

B

9

c

V

D

N

J

y

r

f

U

Y

o

M

H

G

/

I

z

k

d

n

Z

w

R

T

q

3

8

Q

m

5

C

0

F

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

f

y

j

E

p

Z

S

G

V

U

X

D

H

2

5

z

B

d

>

A

7

c

C

g

N

F

I

P

o

3

r

L

W

M

q

0

Y

Q

u

m

J

v

k

+

/

w

9

K

R

O

n

T

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

T

n

X

y

o

5

J

d

p

M

9

u

I

2

V

E

>

A

B

7

c

Z

C

S

g

N

F

P

L

m

8

W

G

3

H

O

w

z

K

Q

r

D

j

Y

U

+

/

f

k

q

v

R

0

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

T

n

X

y

o

5

J

d

p

M

9

u

I

2

V

E

>

A

B

7

c

Z

C

S

g

N

F

P

L

m

8

W

G

3

H

O

w

z

K

Q

r

D

j

Y

U

+

/

f

k

q

v

R

0

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

j

N

X

k

p

r

9

Y

Q

G

0

o

W

c

f

g

d

u

7

>

A

B

V

D

L

S

E

O

z

M

H

8

v

R

P

C

Z

w

2

m

/

+

n

y

q

F

3

K

T

5

U

I

J

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

W

J

n

7

X

K

9

o

v

3

I

D

+

u

G

A

k

d

>

B

c

V

L

S

g

N

E

O

y

r

U

j

F

8

R

z

5

m

Y

2

Z

p

/

P

T

H

q

M

0

f

C

Q

w

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

r

B

p

L

P

+

q

K

V

2

9

O

v

D

F

H

E

>

A

7

X

c

C

T

N

z

M

I

Q

j

S

R

Y

0

5

f

y

o

8

g

G

J

/

k

U

u

d

Z

n

3

m

w

W

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

r

B

p

L

P

+

q

K

V

2

9

O

v

D

F

H

E

>

A

7

X

c

C

T

N

z

M

I

Q

j

S

R

Y

0

5

f

y

o

8

g

G

J

/

k

U

u

d

Z

n

3

m

w

W

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

m

Q

p

g

D

v

O

+

K

T

Y

C

W

Z

n

u

I

d

8

E

>

A

B

7

X

c

V

9

S

w

N

J

3

z

M

q

2

k

y

j

5

G

f

/

0

R

L

r

o

U

H

F

P

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

5

D

w

Y

n

A

N

j

T

P

X

k

g

7

H

3

+

2

C

>

B

c

V

L

S

E

O

y

r

o

9

Z

M

K

u

F

z

0

G

v

W

p

f

/

I

U

d

Q

q

R

J

m

8

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

A

C

X

w

2

Q

p

z

7

Z

r

m

q

u

9

0

D

k

B

g

>

H

c

V

L

S

N

E

O

y

+

o

M

K

F

G

v

W

d

J

/

5

f

U

3

Y

P

R

I

8

j

T

n

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

A

C

X

w

2

Q

p

z

7

Z

r

m

q

u

9

0

D

k

B

g

>

H

c

V

L

S

N

E

O

y

+

o

M

K

F

G

v

W

d

J

/

5

f

U

3

Y

P

R

I

8

j

T

n

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

X

R

C

q

I

5

r

Q

0

z

v

A

Y

L

H

d

T

U

>

B

c

V

N

S

8

E

J

3

/

9

Z

p

j

y

2

D

+

g

W

n

u

w

F

K

k

O

P

G

7

o

m

f

M

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

C

I

r

E

n

q

j

g

c

k

8

v

N

O

K

F

Y

>

A

B

Z

7

S

P

o

3

W

0

G

Q

y

V

d

5

w

H

2

m

T

J

L

/

R

f

z

U

X

p

9

D

+

u

M

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

v

C

T

L

H

W

z

R

o

8

c

k

n

P

7

+

K

d

Q

U

>

A

B

Z

S

g

N

F

I

u

M

/

X

2

w

G

r

p

O

E

D

J

9

Y

m

3

j

5

q

f

y

V

0

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

v

C

T

L

H

W

z

R

o

8

c

k

n

P

7

+

K

d

Q

U

>

A

B

Z

S

g

N

F

I

u

M

/

X

2

w

G

r

p

O

E

D

J

9

Y

m

3

j

5

q

f

y

V

0

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

F

g

+

j

y

D

G

V

U

T

8

d

S

3

u

Y

z

R

B

v

>

A

n

c

N

E

J

r

q

/

9

L

0

W

X

5

m

2

Z

K

M

w

P

O

k

7

H

o

Q

f

I

p

C

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

K

g

N

Y

2

G

T

J

3

/

z

n

w

y

I

U

7

m

F

>

A

B

+

X

c

V

S

8

E

W

O

Q

P

R

C

0

d

M

u

p

k

D

9

Z

j

o

r

v

H

5

f

q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

K

g

N

Y

2

G

T

J

3

/

z

n

w

y

I

U

7

m

F

>

A

B

+

X

c

V

S

8

E

W

O

Q

P

R

C

0

d

M

u

p

k

D

9

Z

j

o

r

v

H

5

f

q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

K

g

N

Y

2

G

T

J

3

/

z

n

w

y

I

U

7

m

F

>

A

B

+

X

c

V

S

8

E

W

O

Q

P

R

C

0

d

M

u

p

k

D

9

Z

j

o

r

v

H

5

f

q

<

l

a

t

e

x

i

s

h

1

_

b

6

4

=

"

L

K

g

N

Y

2

G

T

J

3

/

z

n

w

y

I

U

7

m

F

>

A

B

+

X

c

V

S

8

E

W

O

Q

P

R

C

0

d

M

u

p

k

D

9

Z

j

o

r

v

H

5

f

q

to explicitly capture the dynamic spatial dependence between the two distributions:

where σ denotes distribution gate, Dfe and Dfh are generated by N 2 -Normalization with input Ef and Hf . AN 2 - Normalization not only inherits all the advantages of N 2 - Normalization (e.g., parallelizable computation and no extra parameters), but also bridges gaps between external factors and urban flows when normalizing the values of subregions.

Fine-grained flow inference. At this point, we are ready to generate the fine-grained flow ̂Xf from the learned joint distribution as:

where Xcup ∈ RNH×NW + is produced by nearest-neighbor upsampling with scaling factor N.

Optimization. Finally, we minimize the mean squared error between the ground truth fine-grained flow map Xf and the model inferred output ̂Xf :

where θ represents all learnable parameters in our model.

3 Experiments

We now present the details of our experimental evaluations.

3.1 Experimental Settings

Datasets. We evaluate all the methods using two real-world urban flow datasets: (1) TaxiBJ [Liang et al., 2019] - a taxi GPS data including taxi flows from July 1, 2014 to October 31, 2014; and (2) BikeNYC - collected from an open website 1 which contains data from January 1, 2019 to June 30, 2019. Each dataset contains two sub-datasets: coarse-grained and fine-grained flows. A detailed description of the datasets is shown in Table 1. Note that the scaling factors are different for the two datasets, i.e., N = 4 and N = 2 for TaxiBJ and BikeNYC, respectively.

Baselines. We compare FODE with the following 10 baselines:

• Mean Partition (Mean) evenly distributes the flow volume in each subregion.

1 https://www.citibikenyc.com/system-data

Table 1: Statistics of datasets.

Dataset	TaxiBJ	BikeNYC
Time range	7/1/2013-10/31/2013	1/1/2019-3/31/2019
Time interval Coarse-grained size Fine-grained size Upscaling factor (N) Latitude range Longitude range External Factors Temperature / ◦ C Wind speed / mph Weather conditions Holidays	30 minutes 32×32 128×128 4 39.82◦N - 39.99◦N 116.26◦E-116.49◦E (meteorology, time (e.g., [-24.6,41.0] [0,48.6] types (e.g., Rainy,Sunny) 18	1 hour 40×16 80×32 2 40.65◦N-40.81◦N 74.00◦E-74.07◦E hourofday, dayofweek)) \ \ \ 10

• Historical Average (HA) models historical average data to predict the flow in the subregion.
• SRCNN [Dong et al., 2015] is a classic model for image SR based on CNNs.
• ESPCN [Shi et al., 2016] introduces a sub-pixel convolutional layer for image SR.
• VDSR [Kim et al., 2016] employs residual networks to solve the slow convergence and limited representation problems in SR.
• SRResNet [Ledig et al., 2017] is a ResNet-based variant of VDSR model, which allows stacking more network layers.
• OISR [He et al., 2019] is an ODE-inspired SR model using Runge-Kutta (RK3) method [Butcher and Wanner, 1996] as ODE solver.
• NODE [Chen et al., 2018] is a neural ODE method that uses adjoint method for discretization and optimal control of ODEs.
• ANODE [Gholami et al., 2019] is an improved version of NODE by introducing checking points for alleviating the incorrect gradient issue.
• UrbanFM [Liang et al., 2019] infers fine-grained urban flow with external factors by stacking ResNet-based neural networks.

Metrics. We evaluate different methods with three widely used metrics: Root Mean Squared Error (RMSE), Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE).

Implementation details. Adam [Kingma and Ba, 2014] is adopted to train FODE with batch size 16 and learning rate e-4 . We leverage Dopri5 numerical method, which can adaptively choose the step size, as ODESovle in FODE. FODE consists of 128 channels and 1 ODE block. We also present a simplified version S-FODE which contains 64 channels while other components are the same as FODE. During training, we halve the learning rate and perform a test on the validation set every 20 epochs. For all image SR models numerical methods (NODE and ANODE), we use N 2 -normalization to constrain the inferred flow distribution. We note that the details of other network settings are described in the source-implementation 2 .

2 https://github.com/Anewnoob/FODE

Table 2: Performance comparisons on TaxiBJ and BikeNYC.

Datasets	TaxiBJ			BikeNYC
Method	RMSE	MAE	MAPE	RMSE	MAE	MAPE
Mean	20.918	12.019	4.469	4.554	1.379	0.678
HA	4.741	2.214	0.332	2.414	0.676	0.216
SRCNN	4.297	2.491	0.714	2.385	0.821	0.433
ESPCN	4.206	2.497	0.732	2.356	0.825	0.441
VDSR	4.159	2.213	0.467	2.344	0.734	0.285
SRResNet	4.164	2.457	0.713	2.355	0.741	0.430
OISR	4.126	2.134	0.421	2.318	0.683	0.246
NODE	4.058	2.125	0.408	2.309	0.671	0.243
ANODE	3.967	2.043	0.351	2.246	0.635	0.217
UrbanFM	3.951	2.011	0.327	2.234	0.627	0.209
S-FODE	3.941	2.001	0.322	1.951	0.517	0.129
FODE	3.860	1.963	0.313	1.916	0.512	0.115

3.2 Results

Overall performance. Table 2 reports the FSR results of all the methods, demonstrating that FODE and its variant outperform all baselines on all metrics across both datasets. Taking TaxiBJ for example, FODE yields 14.9%, 19.1%, and 41.6% improvement on average in terms of RMSE, MAE, and MAPE, respectively. In a relatively smaller area of NYC, FODE achieves larger improvement for inferring the bicycle flow. The performance gain of FODE over baselines demonstrates the effectiveness of continuous-time ODEs, which provides an alternative view of improving fine-grained flow inference.

Comparison analysis. Image SR methods are usually not comparable even with N 2 normalization, due to the structural constraints and the influence of external factors in FSR application. This implies that FSR requires specific model design that seamlessly taking constraints and factors into account. As a specifically tailored FSR model, UrbanFM achieves best performance among baselines. However, as a ResNetbased model, it models the urban flow in a discrete manner by stacking deep neural networks, which could be problematic since urban flow inherently can be viewed as a continuous dynamic system. OISR, NODE and ANODE are numerical methods tailored for FSR. OISR uses RK-block as the network structure, which suffers from a huge number of parameters and numerical errors that significantly affect the performance. Additionally, it requires a significant amount of memory for storing intermediate quantities during backpropagation (cf. Table 3). NODE, in contrast, uses adjoint method for solving ODEs, which only needs O( ˜ L) memory as FODE (note that ˜ L = 1 in FODE), while ANODE requires O( ˜ L)+O(N) memory. Nevertheless, the dynamics of either the hidden state or the adjoint might be unstable, which incurs inaccurate gradient computation, as we analyzed in previous section. The performance gain of FODE over these numerical methods indicates that our method estimates the gradients more accurately, due to the introduced affine coupling layer while incurring no memory cost.

Memory efficiency. In addition to FSR performance, FODE has a significant memory and parameter efficiency, compared to ResNet-based models. Table 3 outlines the memory cost and parameters required for different methods (we omit other image SR methods due to the similar architectures as SRResNet). In particular, FODE requires only 1/3× parameters of UrbanFM and reduces the memory cost to O(1). It is worthwhile to note that a simplified version S-FODE contains only 0.7M parameters while still outperforming all the baseline methods on FSR task.

Table 3: Comparisons of parameters and memory cost. C: the number of channels; L( ˜ L): the number of ResNet (ODE blocks); H: the number of layers in each ResNet; N: the number of function evaluations.

Method	C	#Params (M)	memory	time
SRResNet	128	5.5	O(LH)	O(LH)
OISR	128	15.7	O(LH)	O(LH)
NODE	128	2.1	O( ˜ L)	O(N)
ANODE	128	2.4	O( ˜ L)+O(N)	O(N)
UrbanFM	128	6.2	O(LH)	O(LH)
S-FODE	64	0.7	O( ˜ L)	O(N)
FODE	128	2.1	O( ˜ L)	O(N)

Ability of factor fusion. External factors play important roles and should be carefully considered in FSR models. Figure 6(a) and 6(b) compare the influence of factors learned by UrbanFM and FODE, respectively. UrbanFM uses a FCN to fuse external factors, which is too weak to correlate complex factors with flow distributions. For example, the impact of factors concentrates in a smaller area for UrbanFM - i.e., two main roads are more affected by external factors than other regions. In contrast, FODE estimates the influence of factors by evenly distributing the external influences and is therefore more robust. This can be further verified by the results shown in Figure 6(c) and Figure 6(d), where we observe that FODE consistently converges while UrbanFM, surprisingly, achieves best performance using temperature only, rather than all the factors.

Error analysis. Figure 7(a) shows the inference errors of UrbanFM and FODE on a data sample, where a brighter pixel indicates a larger error. To better visualize the quality of inference, we select four subregions (A, B, C, and D), from which we clearly see that the flow inference by FODE performs better than UrbanFM in crowded areas. Similarly, Figure 7(b) depicts the overall inference error using two different normalization schemes. We observe that in most areas, especially in E, F, G subregions, the distribution generated by AN 2 -normalization is closer to the ground-truth, which demonstrates that the proposed AN 2 method is a more effective way of constraining the flow distributions. This improvement can be attributed to jointly modeling the distribution of external factors and urban flows in AN 2 , compared to simply constraining the flows in subregions in N 2 -normalization.

Accuracy vs. efficiency. Another merit of FODE is that it allows to balance the trade off between the flow inference accuracy and the computational overhead, by varying the number of function evaluations N. As shown in Figure 8, the more function evaluated in the forward pass, the lower the MSE loss. Accordingly, the time required for training the model increases linearly with the number of function evaluations. This result is important since downstream applications UrbanFM

(a) Comparison of inference errors.

-Normalization

-Normalization

AN 2

(b) N 2 vs. AN 2 .

Figure 8: Tradeoff between training time and inference accuracy.

may make flexible solutions by conciliating inference accuracy with computational cost.

4 Conclusion

We proposed a novel method FODE for inferring fine-grained urban flow. FODE learns urban flow distribution through a new ODEs parameterized by affine coupling neural networks alleviating the numerical instability gradient computation issue, which allows for both memory and model parameter savings. In addition, it is capable of explicitly providing more flexible prediction performance by adaptively balancing prediction accuracy and computation overheads. Furthermore, we believe that FODE is a more general ODE-based architecture and can be better exploited for time series prediction or other fine-grained inference tasks such as single image SR [Cai et al., 2019; Wang et al., 2019] and air quality inference [Liu et al., 2019a].

Acknowledgments

This work was supported by National Natural Science Foundation of China (Grant No.61602097) and NSF grant CNS 1646107.