Stroke-Based Stylization Learning and Rendering with Inverse Reinforcement Learning

1 Introduction

Artistic stylization in non-photorealistic rendering enables users to stylize pictures with the appearance of traditional art forms, such as pointillism painting, line sketching, or brush stroke drawing. Among them, brush stroke drawing is one of the widely used art styles across different cultures in history. In computer-generated painterly rendering, stroke placement is a big challenge and significant efforts have been made to investigate how to draw a stroke with realistic brush texture in a desired shape and how to organize multiple strokes [Fu et al., 2011].

The goal of this paper is to develop an AI-aided art authoring system for artistic brush stroke generation. In this section, we first review backgrounds in computer graphics and artificial intelligence, and then give an overview of our proposed system.

1.1 Background in Computer Graphics

The most straightforward approach for painterly rendering would be physics-based painting, i.e., giving users an intuitive feeling just like drawing with a real brush. Some works modeled physical virtual brushes including its 3D structure, dynamics, interaction with paper surface [Chu and Tai, 2004] and simulating the physical effect of the ink dispersion [Chu and Tai, 2005]. These virtual brushes can be used to draw various styles of strokes with a digital pen or mouse. However, it is very complex to control a virtual brush. Furthermore, since the computational cost is often very high to achieve satisfactory visual effects to human eyes, some physics-based painting approaches rely on graphics processing units (GPUs) for obtaining reasonable performance [Chu et al., 2010].

∗Email: ningxie@tongji.edu.cn

†Email: tingting@tust.edu.cn

‡Email: ftian@bournemouth.ac.uk

§Email: zhxh@cc.it-hiroshima.ac.jp

¶Email: sugi@k.u-tokyo.ac.jp

To address these issues associated with physics-based painting, the stroke-based rendering approach was proposed to directly simulate rendering marks (such as lines, brush strokes, or even larger primitives such as tiles) on a 2D canvas. This stroke-based rendering [Hertzmann, 2003] underpins many artistic rendering algorithms, especially on those emulating traditional brush-based artistic styles such as oil painting and watercolor.

Although physics-based painting and stroke-based rendering are useful for (semi-)professional usage, often users who have no painting expertise are only interested in final results rather than the painting process itself [Kalogerakis, 2012]. To make the painterly rendering system more accessible to novice users, several researchers investigated beautification. The early work [Theodosios and Van, 1985] explored automatic techniques to beautifying geometric drawings by enforcing various relations such as the collinearity of lines and the similarity of their lengths. The approach by Igarashi et al. [T.Igarashi et al., 1997] offered users several choices in the beautifying process.

Filter-based methods are also widely used for building artistic rendering algorithms applied in the image manipulation software such as Photoshop and Gimp. The main task is to find out the nice or beautiful outlines/edges based on novel filters, such as bilateral filter [Pham and Vliet, 2005], DoG filter [S´ykora et al., 2005], morphological filter [Bousseau et al., 2007], shock filter [Kang and Lee, 2008] and Kuwahara filters [Kyprianidis et al., 2009]. These techniques are usually based on heuristics developed through hands-on experience, showing that certain combinations of filters produce an artistic look, more precisely, called stylized cartoon rendering, pen-and-ink illustration, or watercolor painting. However, the connection between the edge-preserving image simplification and the artistic rendering is less obvious, because the significant artistic look is often achieved or further reinforced by taking the local image structure and brush stroke details into account, rather than the global image abstraction. In practice, the designers usually firstly apply the painting style filter on the real photo in order to imagine the whole art authoring in terms of the entire layout. Then, another layer is created on the top to emphasize the local parts of the image that are important to the users by hand or stroke-based methods.

More recently, methods that attempt not only to beautify generated artistic images, but also to maintain users' personal styles are pursued. Studies of style imitation in artistic rendering focused on ink sketching. Baran et al. [Baran et al., 2010] proposed a method to draw smooth curves while maintaining the details. The sketch beautification approach by Orbay et al. [Orbay and Kara, 2011] used the model that automatically learns how to parse a drawing. Zitnick [Zitnick, 2013] proposed a general purpose approach to handwriting beautification using online input from a stylus. Since techniques of line sketching style imitation are not suitable to synthesizing brush strokes, quite a few previous works [Xu et al., 2006; Zeng et al., 2010; Lu et al., 2012] tried to reproduce brush stroke texture as reference samples.

1.2 Background in Artificial Intelligence

Differently from the above approaches developed in computer graphics, the system we propose in this paper trains a virtual brush agent to learn the stroke drawing model according to particular artists' styles using their stroke drawing videos. The problem on truncated fault texture can be solved by using our learned stroke drawing behavior model. Since the brush agent is trained locally with the data set of basic stroke shapes, we can create strokes in new shapes even when they are quite different from an artist's examples. This is eminently suitable for the artistic stylization of images when nonexpert users try to render their favorite photos into a particular artist's style with just a few button clicks.

Our proposed system is based on the reinforcement learning (RL) method to artistic brush stroke generation [Xie et al., 2012], which allows users to automatically produce consistent and smooth brush strokes. In this RL approach, the task of synthesizing the texture of each individual stroke is formulated as a sequential decision making problem based on the Markov decision process, where a soft-tuft brush is regarded as an RL agent. Then, to sweep over the shape closed by the contour, the agent is trained by a policy gradient method [Williams, 1992] to learn which direction to move and how to keep the stable posture while sweeping over various stroke shapes provided as training data. Finally, in the test phase, the agent chooses actions to draw strokes by moving a virtual inked brush within a newly given shape represented by a closed contour.

In this paper, we extend this RL-based approach to be able to incorporate personal artistic stylization. More specifically, we propose to use a method of inverse RL [Abbeel and Ng, 2004] to learn the reward function from stroke data videocaptured from artists: we first invite artists to draw strokes using our handcrafted device for recording the brush movement. The brush footprint in each key frame of the captured strokedrawing video is then extracted, and time series data are obtained by assembling the extracted posture configuration of each footprint including the motion attitude, pose, and locomotion of the brush. The data are used to mimic the artist's stroke drawing style through the reward function learned by inverse RL (IRL).

1.3 Overview of Our Proposed System

An overview of our system, called AI-aided art authoring (A4) system for artistic brush stroke generation, is illustrated in Figure 1. Our system consists of two phases: an online synthesis phase and an offline training phase.

In the online synthesis phase, A4 provides a fast and easyto-use graphical-user interface so that users can focus on developing art work concepts just by sketching the position and attitude of desired strokes. Given an input picture or photo, even non-expert users can sketch the shapes of desired strokes using either closed contours or simple curves.

In the offline training phase, the main task is to train the virtual agent so as to synthesize strokes in an artist's drawing style. Instead of the classical policy gradient method [Williams, 1992], we use the state-of-the-art policy gradient algorithm called importance-weighted policy gradients with parameter-based exploration [Zhao et al., 2013], which allows stable policy update and efficiently reuse of previously collected data.

Through experiments, we demonstrate that the proposed system is promising in producing stroke placement with a personalized style.

2 Reinforcement Learning Formulation of Brush Agent

In order to synthesize the painting imagery of an artist's personal style, we construct our brush agent equipped with the style learning ability by extending the existing RL-based approach [Xie et al., 2012] as illustrated in Figure 2.

We assume that our stroke drawing problem is a discretetime Markov decision process. At each time step t, the agent observes a state st ∈ S, selects an action at ∈ A, and then receives an immediate reward rt resulting from a state transition. The state space S and action space A are both defined as continuous spaces in this paper. The dynamics of the environment is characterized by unknown conditional density p(st+1|st, at), which represents the transition probability density from current state st to next state st+1 when action at is taken. The initial state of the agent is determined following unknown probability density p(s1). The immediate reward rt is given according to the reward function R(st, at, st+1). The agent's decision making procedure at each time step t is characterized by a parameterized policy p(at|st, θ) with parameter θ, which represents the conditional probability density of taking action at in state st.

A sequence of states and actions forms a trajectory denoted by

where T denotes the number of steps called the horizon length. Given policy parameter θ, trajectory h follows

The discounted cumulative reward along h, called the return, is given by

where γ ∈ [0, 1) is the discount factor for future rewards.

The goal is to optimize the policy parameter θ so that the expected return is maximized:

where J(θ) is the expected return for policy parameter θ:

In the previous work [Xie et al., 2012], the reward function was manually designed to produce "nice" drawings. On the other hand, we aim to learn the reward function from an artist's drawing data D in this paper. We assume that data D = {τ1, . . . , τN} is generated following optimal policy π∗ , where the n-th trajectory τn is a T-step sequence of stateaction pairs τi = {(si,1, ai,1), . . . , (si,T , ai,T )}. In Section 3, we describe our device to capture an artist's drawing to obtain D and explain an inverse RL method [Abbeel and Ng, 2004]. Then in Section 4, a policy learning method [Zhao et al., 2013] is introduced.

3 Reward Function Learning fromArtist

To learn a particular artist's stroke drawing style, we collect stroke data from brush motion and drawings on the canvas and then learn the reward function from the collected data. In this section, we first describe the details of the data collection procedure and then introduce our reward function.

3.1 Data Collection

We designed a device shown in Figure 3 to video-record brush motion. A digital single-lens reflex camera is mounted at the bottom of the frame of the device. Data collection is carried out under normal in-door lighting and thus there is no need for automatic camera calibration in real-time. A traditional Asian calligraphy paper is placed on the transparent glass panel on the top of the device. In each data-collection session, an artist is asked to draw a panda with various strokes on the glass panel. The brush motion of an artist is captured when they dip the brush into the traditional calligraphy ink and start drawing strokes.

We split the recorded video of the stroke drawing into frames to analyze brush movement (Figure 4 (c)). To each frame, we apply the model-based tracking technique [Davies, 2005] and detect the posture configuration of brush footprints such as the brush movement information (the velocity, heading direction and pose) and the relative location information to the target desired shape over time. We then apply principal component analysis [Jolliffe, 2002] to compute the principal axis of the footprint which defines the direction of the footprint. Finally, the configuration of the footprint is determined by matching the template of the footprint which consists of a tip Q and a circle with center C and radius r.

3.2 Reward Function Design

We design the reward function to measure the quality of the brush agent's stroke drawing movement. First of all, a smoother movement should yield a higher immediate reward value. We calculate the immediate reward value by considering (i) the distance between the center of the brush agent and the nearest point on the medial axis of the shape at the current time step, and (ii) the change of the local configuration of the brush agent after an action:

where ft and ft+1 are footprints at time steps t and t + 1, respectively. This reward design means that the immediate reward is zero when the brush is blocked by a boundary as ft = ft+1 or the brush is going backward to a region that has already been covered by previous footprints fi for i < t + 1. C(st, at, st+1) calculates the cost of the transition of footprints from time t to t + 1 as

where the first two terms measure the cost regarding the location of the agent, while the last three terms measure the cost regarding the posture when the agent moves from time t to t + 1. More specifically, ∆ωt,t+1,∆φt,t+1, and ∆dt,t+1 are normalized changes in angle ω of the velocity vector, heading directions φ, and ratios d of the offset distance between time t and time t + 1:

∆φt,t+1 and ∆dt,t+1 are defined in the same way. To set the values of five parameters α1, α2, . . . , α5, we use the maximum-margin inverse reinforcement learning method [Abbeel and Ng, 2004]. This allows us to learn the artist's personal style based on an his/her drawing data through inferring appropriate values for α1, α2, . . . , α5.

4 Policy Learning

The previous work [Xie et al., 2012] learned policies by the classical policy gradient method [Williams, 1992]. However, this algorithm is often unreliable due to the large variance of the policy gradient estimator [Zhao et al., 2011].

To mitigate the large variance problem, an alternative method called policy gradients with parameter based exploration (PGPE) was proposed [Sehnke et al., 2010]. The basic idea of PGPE is to use a deterministic policy and introduce stochasticity by drawing parameters from a prior distribution. More specifically, parameters are sampled from the prior distribution at the start of each trajectory, and thereafter the controller is deterministic. Thanks to this per-trajectory formulation, the variance of gradient estimates in PGPE does not increase with respect to trajectory length [Zhao et al., 2011]. The gradient estimation of PGPE can be further stabilized by subtracting a baseline [Zhao et al., 2011].

However, (baseline-subtracted) PGPE still requires a relatively large number of samples to obtain accurate gradient estimates, which can be a critical bottleneck for our application due to the large costs and time in data collection. To cope with this problem, we use a variant called importanceweighted policy gradients with parameter-based exploration (IW-PGPE) [Zhao et al., 2013], which allows efficient reuse of previously collected data. In the online synthesis phase illustrated in Figure 1, the user is allowed to choose one or several learned policies to control the drawing behavior for each input shape.

5 Stroke Texture Rendering

We use both the raster brush texture mapping and the physical pigment dispersion simulation to generate both dry and wet textures. Rendering is carried out by capturing single footprints of a brush and then stamping them along the trajectory obtained by the brush agent's learned policy. The scanned footprint images are used as the reference texture of brush footprints and sampled with different contents of ink of hollow strokes for rendering the change of the stroke texture. Then, we save them into raster textures to create our brush footprint texture libraries as shown in Figure 5(a). For the drying stroke rendering, given the parameters of the brush ink style, footprint texture images with different ink contents are affinely transformed and then mapped onto the optimal sequential collection of footprints according to the shapes and orientations of the footprints.

Discrete series of footprint images need to be interpolated to render strokes with smooth textures. To do so, each intermediate pixel on the resulting stroke texture is linked by a pair of points on the two nearest footprints using the interval piecewise B´ezier splines. Figure 5 (b) illustrates a dry stroke. Wet stroke rendering is carried out by adding ink and pigment dispersion into the brush texture mapping. We adopt the water-based paint flow physical simulation [Chu and Tai, 2005]. The quantity of ink and pigment on the paper canvas is initialized according to the current sampled brush texture images. Figure 5 (c) illustrates a wet stroke where its shape and trajectory is the same as those in Figure 5 (b).

6 Experiments and Results

Figure 6 plots the average return over 16 trials as the function of policy update iterations, obtained by the policies learned by our approach. Returns at each trial are computed over 300 training episode samples. This graph shows that the average return sharply increases in an early stage and then converges at about the 20th iteration.

Stroke drawing results by an artist, the agent trained with the learned reward function, and the agent trained with the manually designed reward function [Xie et al., 2012] are compared in Figure 7. The results show that the proposed method imitates the real artist's stroke drawing better than the previous method. More specifically, the two results marked with red in the right-most column show that our rendered stroke texture is much smoother than the one obtained with the manually designed reward function.

Finally, we applied the policy obtained by our method to photo artistic conversion system [Xie et al., 2011] (Figure 8), where we manually sketched contours from the original pictures that represent the boundaries of desired strokes. The results in Figure 8 (c) show that shapes are filled with smooth strokes by our IRL method and visually reasonable drawings are obtained.

To further investigate our IRL-based method, we performed the user study on the aesthetic assessment of the traditional oriental ink painting simulation between the proposed A4 system and the brush stroke (Sumie) filter of the stateof-the-art commercial software (Adobe Photoshop CC 2014). We invited 318 individuals to take the online questionnaire survey. We conducted a quantitative user study following the same approach as in [Xu et al., 2008]. We asked the participants to tell which one is more like the oriental ink painting style for each pairs (shown as (b) and (c) in Figure 8) among four pairs of paintings. We include this question in the user study to directly compare subjective aesthetic assessment of the viewer by selecting which images they like. The aesthetic scores are given by participants shown in Figure 9. Obviously, our results obtained higher aesthetic scores than Photoshop.

7 Conclusion and FutureWork

We have proposed an AI-aided art authoring system (A4) so as to fast and easy creation of stylized stroke-based paintings. Our main contributions in this papers are (i) we developed a device to capture artists' brush strokes, (ii) we collected training data in various styles, (iii) we applied inverse reinforcement learning to learn the reward function from the data provided by artists, (iv) we applied the state-of-the-art reinforcement learning method, IW-PGPE (importance-weighted policy gradients with parameter-based exploration), to accurately learning the policy function by efficiently reusing previously collected data, and (v) we demonstrated through experiments the effectiveness of our proposed approach in converting photographs into stroke drawings.

In the future, an automatic contour extraction from pictures may be explored to simplify the process of photo stylization for non-expert users to both learn and detect local contourbased representations for mid-level feature information in the form of hand drawn contours in images.

Acknowledgments

We would like to thank Laga Hamid for his suggestions and advices at different stages of this research. We also thanks the reviewers for their helpful comments. Ning Xie was supported by National Science Foundation of China (No.61272276) and Tongji University Young Scholar Plan (No.2014KJ074), Tingting Zhao was supported by SRF for ROCS,SEM., and Masashi Sugiyama was supported by KAKENHI 23120004.