---
title: "Unsupervised Domain Adaptation with Dual-Scheme Fusion Network for Medical Image Segmentation"
authors:
  - "Danbing Zou"
  - "Qikui Zhu"
  - "Pingkun Yan"
doi: "10.24963/ijcai.2020/455"
url: "https://ijcai.org/proceedings/2020/455"
pdf: "https://ijcai.org/proceedings/2020/0455.pdf"
venue: "Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence"
year: 2020
pages: "3291-3298"
track: "Main track"
keywords:
  - "Machine Learning: Deep-learning Theory"
  - "Machine Learning: Deep Learning"
  - "Machine Learning: Deep Learning: Convolutional networks"
---

# Unsupervised Domain Adaptation with Dual-Scheme Fusion Network for Medical Image Segmentation

Danbing Zou, Qikui Zhu, Pingkun Yan

*Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence*, pages 3291-3298. DOI: [10.24963/ijcai.2020/455](https://doi.org/10.24963/ijcai.2020/455)

## Abstract

Domain adaptation aims to alleviate the problem of retraining a pre-trained model when applying it to a different domain, which requires large amount of additional training data of the target domain. Such an objective is usually achieved by establishing connections between the source domain labels and target domain data. However, this imbalanced source-to-target one way pass may not eliminate the domain gap, which limits the performance of the pre-trained model. In this paper, we propose an innovative Dual-Scheme Fusion Network (DSFN) for unsupervised domain adaptation. By building both source-to-target and target-to-source connections, this balanced joint information flow helps reduce the domain gap to further improve the network performance. The mechanism is further applied to the inference stage, where both the original input target image and the generated source images are segmented with the proposed joint network. The results are fused to obtain more robust segmentation. Extensive experiments of unsupervised cross-modality medical image segmentation are conducted on two tasks -- brain tumor segmentation and cardiac structures segmentation. The experimental results show that our method achieved significant performance improvement over other state-of-the-art domain adaptation methods.

## Full text

## 1 Introduction

Deep learning has achieved state-of-the-art results in many fields. However, the model trained on one dataset cannot be directly applied to other datasets with similar contents and various distribution. Since the different distribution brings a domain shift problem and further leads to identification failure. Solving the identification failure caused by domain shift is an important issue, which can improve the generalization as well as portability and further solve the problem of lacking of labeled data and expensive cost of manual annotation. Unsupervised domain adaptation methods are an effective method to solve those problems, which enables one labeled dataset to serve multiple other unlabeled ones and greatly improve the practical value of deep learning network. The unsupervised domain adaptation methods can be categorized into two types: image adaptation and feature adaptation. [Chen et al., 2018; Bousmalis et al., 2017; Hoffman et al., 2017; Chen et al., 2019] are the representative articles of image adaptation. In [Chen et al., 2018; Bousmalis et al., 2017], segmentation network is first trained using source domain dataset, and then target data are translated into source domain for segmentation using the trained segmentation network. In [Hoffman et al., 2017; Chen et al., 2019], the source domain data are first translated into target domain, and then the segmentation network are trained by the generated target-like data with corresponding source domain annotation. The representative articles of feature adaptation are [Tsai et al., 2018; Dou et al., 2018]. Those methods directly perform feature adaptation without cross-domain translation of the image.

∗Corresponding Authors.

However, current works based on image adaptation only conduct the two-direction translation separately. This imbalanced source-to-target or target-to-source one way connection may not eliminate the domain gap, which limits the performance of the pre-trained model. To overcome this problem, we propose to combine these two complementary directions into one unified framework for acquiring a better result. Our proposed model, named Dual-Scheme Fusion Network (DSFN), adopts CycleGAN[Zhu et al., 2017]'s architecture as base structure. To make the network can simultaneously conduct the tasks of image translation and segmentation, we modified the generator to a joint translation-segmentation network. The translation and segmentation parts share a same encoder. This setting can not only reduce network's parameters, but also enable the two tasks to enhance each other. Moreover, the segmentation network also can help reserve content of the generated image. During training, the first joint cross-domain translation and segmentation network is trained by source domain images. The task of translation branch is generating target-like images and the segmentation branch aims to segment the source domain image. The first trained segmentation network can be used to segment the source-like image generated by the target domain image. Then, the generated target-like image is used to train the joint network for converting target-like image to source domain and segmen-

s

Figure 1: Overall structure of our proposed network. Es + Ts→t is the source-to-target image translation branch while Et + Tt→s conduct target-to-source translation. Es + Ss and Et + St are the two segmentation network that we aim to get. We add two other discriminators Ds f , Dtf and auxiliary tasks for Dt, Ds to perform feature adaptation. The red arrows represent different losses, the yellow arrows show the data flows from source or source-like images, the green arrows show the data flows from target or target-like images, and the black arrows mean the data flows for feature adaptation.

![Image](https://ijcai.org/proceedings/2020/fulltext/455/image_000000_211c26b2a58ea22aa009820b3e811184e2d842e831c11b92d770287d4d80f76e.png)

tation. The second obtained segmentation branch can be directly used to segment the target domain data. Finally, we merge the two segmentation outputs of target image to obtain a more accurate result. Moreover, we also introduce feature adaptation loss referring[Chen et al., 2019] to further narrow the feature-level gap between target and target-like data, as well as source and source-like data. The whole network is designed subtly, the backward process of our network is also the test procedure. The main contributions of our paper are as follows:

- 1) We propose an end-to-end unsupervised parallel dualscheme fusion network for solving the problem of unsupervised domain adaptation from two perspectives by building both source-to-target and target-to-source connections, this balanced joint information flow helps bridge the domain gap to further improve the network performance.
- 2) We train our network in a cycle manner and the backward process is also the testing process. The structure of our network is effective and elegant.
- 3) We validate our method on the BRATS[Menze et al., 2014] and MM-WHS[Zhuang and Shen, 2016] dataset. Experimental results show that our proposed approach exceed current state-of-the-art domain adaptation methods, effectively alleviating the expensive cost of manual label-

ing cross-domain medical data, which has great clinical practical value.

## 2 RelatedWork

Typically, domain adaptation methods generalize to other tasks by minimizing the distance of feature space between the source and the target domain. For example, Tzeng et al. [Tzeng et al., 2014] tries to learn a domain-invariant feature representation between source and target by proposing an adaptation layer and a domain confusing loss. Long et al. [Long et al., 2015] matches source and target distribution by embedding their hidden features in a reproducing kernel Hilbert space.

Inspired by the advanced of generative adversarial network [Goodfellow et al., 2014], adversarial learning becomes the main training strategy for domain adaptation methods. Some of them directly map feature representations between source and target domain, for example, Tsai et al .[Tsai et al., 2018] conducted different feature levels' domain adaptation by introducing multi-level discriminators to the segmentation network and Dou et al. [Dou et al., 2018] shared a similar idea with an innovative network structure. Some conduct pixellevel domain transfer, which has two directions as mentioned above. One is to perform target-to-sourced translation, such as [Chen et al., 2018], and the other is to perform source-totarget translation, such as [Hoffman et al., 2017] and [Zhu et al., 2020].

Table 1: Comparison of Dice score and Hausdorff distance between our proposed method and other state-of-the-art methods on whole tumor segmentation in the BRATS dataset.

| Method                         |   Dice Score [%] - T1 |   Dice Score [%] - FLAIR |   Dice Score [%] - T1CE |   Dice Score [%] - Average |   Hausdorff Distance [mm] - T1 |   Hausdorff Distance [mm] - FLAIR |   Hausdorff Distance [mm] - T1CE |   Hausdorff Distance [mm] - Average |
|--------------------------------|-----------------------|--------------------------|-------------------------|----------------------------|--------------------------------|-----------------------------------|----------------------------------|-------------------------------------|
| Without domain adaptaion       |                   6.8 |                     54.4 |                     6.7 |                       22.6 |                           58.7 |                              21.5 |                             60.2 |                                46.8 |
| CycleGAN[Zhu et al., 2017]     |                  38.1 |                     63.3 |                    42.1 |                       47.8 |                           25.4 |                              17.2 |                             23.2 |                                21.9 |
| CyCADA [Hoffman et al., 2017]  |                  49.6 |                       72 |                    51.7 |                       57.8 |                           20.2 |                              14.9 |                             18.4 |                                17.8 |
| AdaptSegNet[Tsai et al., 2018] |                  42.6 |                     67.8 |                    33.1 |                       47.8 |                           23.4 |                              16.6 |                             28.8 |                                22.9 |
| SIFA[Chen et al., 2019]        |                  51.7 |                       68 |                    58.2 |                       59.3 |                           19.6 |                              16.9 |                            15.01 |                                17.1 |
| DSFN(our proposed)             |                  57.3 |                     78.9 |                    62.2 |                       66.1 |                           17.5 |                              13.8 |                             15.5 |                                15.6 |

Due to the expensive cost of manual annotation, there is a lot of research on domain adaptation in the medical field. For example, Nie et al. [Nie et al., 2017] proposed a context-aware GAN to generate PET images from MRI images. Wolterink et al. [Wolterink et al., 2017] used CycleGAN to convert between unpaired CT and MRI images. Similarly, Chen et al. [Chen et al., 2019] conducted the MRI to CT translation and CT splenomegalia segmentation at the same time. Zhao et al. [Zhao et al., 2018] proposed a RsGAN technique to synthesize fundus image data set and generalize its segmentation network to other data sets. Chen et al. [Chen et al., 2018] adopted target-to-source translation to segment left/right lung using chest X-ray datasets. Kamnitsas et al. [Kamnitsas et al., 2017] proposed a multi-connected domain discriminator to segment unannotated MR images with traumatic brain injuries. [Dou et al., 2018] transfers a cardiac segmentation network trained with MRI to unannotated CT images. Fang and Yan [Fang and Yan, 2020] propose a new multi-scale paramid network and loss function to segment multiple partially labeled datasets.

## 3 Methods

The overall framework of our proposed Dual-Scheme Fusion Network for unsupervised domain adaptation is shown in Fig.1, which contains two joint translation-segmentation modules. Each joint network has an encoder-decoder structure. The decoder is divided into two branches-translation branch and segmentation branch, which share one same encoder. Each translation branch corresponds to a discriminator to distinguish the generated image from the real image. The design of our network structure is a transformation based on CycleGAN[Zhu et al., 2017]. The difference is that we replace the generator with a joint translation-segmentation module.

The background setting is that we have a source domain dataset {xs} with corresponding annotation {ys} and want to annotate the unlabeled target domain dataset {xt}. We solve this problem from two perspectives: (a) Target-to-Source: We train a segmentation network Es + Ss using source domain dataset {xs, ys}, then translate target image xt into the source domain through Et + Tt→s and segment it through the segmentation network Es + Ss; (b) Source-to-Target: we first translate the source domain image xs into the target domain through Es + Ts→t, then a segmentation network Et + St is trained using the generated xs→t with corresponding annotation ys, so the real xt can be directly segmented by Et + St. We integrate these two streams into a unified network and fuse their results to form a new prediction as final output.

The flow of data through our network is described as follows. In the training process of forward, xs is inputed into the joint network G1 to obtain the cross-domain generated image xs→t and the segmentation prediction ˆ ys. Discriminator Dt will distinguish between xt and xs→t. The generated image xs→t is then putted into the joint network G2 to obtain reconstructed image xs→t→s and the segmentation prediction ˆ ys→t .

And in the training process of backward (test process follows the same data flow of backward process), the target domain image xt is inputted into joint network G2, and we can get the xs-like cross-modality image xt→s and segmentation prediction ˆ yt. Discriminator Ds distinguish between xs and xt→s. Then, the generated xt→s is putted into the joint network G1 which outputs the reconstructed image xt→s→t and the segmentation prediction ˆ yt→s. As for feature adaptation, Discriminator Ds f distinguishes between ˆ ys and ˆ yt→s while Dtf will distinguish between ˆ ys→t and ˆ yt. Dt and Ds also have an auxiliary task to respectively distinguish between xs→t and xt→s→t, xs→t→s and xt→s. In the test process, ˆ yt and ˆ yt→s are two segmentation predictions for the target domain image xt from different perspective. We fuse the two predictions to get a higher-accuracy result fusion(ˆ yt, ˆ yt→s) as our final result for target image xt. For the selection of the fusion strategy, we find that averaging the prediction probabilities of these two results can obtain better performance.

## 3.1 Adversarial Loss

We train our unified network in a cycle manner, which means that we have a source to target translation and a target to source translation as well. To generate more realistic images, we adopt the adversarial loss as loss function. The adversarial learning consists of two modules: a generator and a discriminator. Our problem is an image-to-image domain translation problem. So in source to target translation, we forward a source image xs to generator Es+Ts→t and it's responsible for generating an image xs→t that looks like target image and can fool the discriminator Dt, while the discriminator Dt aims to distinguish whether the image is actually derived from target domain or synthesized by the generator. This forms a process of adversarial learning, which can be described as minEs+Ts→t maxDt Ladv(Es, Ts→t, Dt) and:

Figure 2: Visual display of the results for comparing different methods on BRATS dataset. From left to right are the target domain input images, results without domain adaptation (as the baseline), CycleGAN, CYCADA, Adapt Segnet, SIFA and our proposed method. The last column is ground truth. The first row is T1 images with segmentation in green, the second row is FLAIR images with segmentation in blue, and the last row is T1CE images with segmentation in red.

![Image](https://ijcai.org/proceedings/2020/fulltext/455/image_000001_5f5fb78d984fc78a03d9fdc7a2d10c7ea9005a280318c6823e7ab0de82218008.png)

<!-- formula-not-decoded -->

The target to source translation is likewise. The objective function is minEt+Tt→s maxDs Ladv(Et, Tt→s, Ds) and:

<!-- formula-not-decoded -->

## 3.2 Reconstruction Loss

In order to make the generated image preserve structure, shape and other content information of the original image, we introduce reconstruction loss in our model to constraint the training process.

Taking the forward process as example, reconstruction means that xs→t translated by xs can be reconstructed back to source domain by generator Et+Tt→s and will be consistent with the original xs, ie Tt→s (Et (xs→t)) ∼ xs. Its objective function is:

<!-- formula-not-decoded -->

Similarly, in the backward process, it is Ts→t (Es (xt→s)) ∼ xt:

<!-- formula-not-decoded -->

## 3.3 Segmentation Loss

As mentioned above, our network contains two segmentation networks to solve the unsupervised domain adaptation problem. As shown in Fig. 1, in the forward process, Es + Ss is our first segmentation network by training the source domain image xs, Et + St is the second segmentation network obtained by training the generated image xs→t. The loss function of two segmentation network is:

<!-- formula-not-decoded -->

As for backward process, segmentation network is employed in the test procedure to get the final segmentation result of the target domain image xt. First, feeding target domain data xt into the joint network G2 through the segmentation branch Et + St in the joint network for getting the first segmentation prediction ˆ yt. At the same time, xt→s is generated by the translation branch Et+Tt→s. Then xt→s will be fed into the joint network G1, so the segmentation branch Es+Ss of G1 can output the second segmentation prediction for xt, denoted as ˆ yt→s. Finally, we fuse ˆ yt and ˆ yt→s to get the final segmentation result fusion{ˆ yt, ˆ yt→s}.

## 3.4 Feature-adaptation Loss

In order to further improve the segmentation results, we introduce feature adaptation loss referring to [Chen et al., 2019]. As mentioned above, the segmentation network Es + Ss is trained on xs and used to induce xt→s. But there is still domain gap between them that makes the segmentation result of xt→s inaccurate. To reduce their differences in feature level, we introduce a discriminator Ds f and an auxiliary task for Dt, when Ds f cannot distinguish between segmentation predictions from xs or xt→s, and Dt can't differentiate between xs→t and xt→s→t, the features of xs and xt→s are consistent. The corresponding loss function is:

Table 2: Comparison of Dice score and Hausdorff distance between our proposed method and other state-of-the-art methods for cardiac structures segmentation on MM-WHS dataset.

| Method                         |   Dice Score [%] - AA |   Dice Score [%] - LAC |   Dice Score [%] - LVC |   Dice Score [%] - MYO |   Dice Score [%] - Average |   Hausdorff Distance [mm] - AA |   Hausdorff Distance [mm] - LAC |   Hausdorff Distance [mm] - LVC |   Hausdorff Distance [mm] - MYO |   Hausdorff Distance [mm] - Average |
|--------------------------------|-----------------------|------------------------|------------------------|------------------------|----------------------------|--------------------------------|---------------------------------|---------------------------------|---------------------------------|-------------------------------------|
| without domain adaptation      |                  28.4 |                   27.7 |                    4.0 |                    8.7 |                       17.2 |                           32.6 |                            35.1 |                            54.2 |                            57.4 |                                44.8 |
| CycleGAN[Zhu et al., 2017]     |                  73.8 |                   75.7 |                   52.3 |                   28.7 |                       57.6 |                           16.2 |                            15.4 |                            20.8 |                            27.7 |                                20.0 |
| CyCADA [Hoffman et al., 2017]  |                  72.9 |                   77.0 |                   62.4 |                   45.3 |                       64.4 |                           14.9 |                            12.8 |                            18.3 |                            22.9 |                                17.2 |
| AdaptSegNet[Tsai et al., 2018] |                  69.4 |                   73.3 |                   54.9 |                   36.7 |                       58.6 |                           17.1 |                            17.6 |                            22.8 |                            25.4 |                                20.7 |
| SIFA[Chen et al., 2019]        |                  81.1 |                   76.4 |                   75.7 |                   58.7 |                       73.0 |                            8.2 |                            10.5 |                            12.2 |                            17.9 |                                12.2 |
| DSFN(our proposed)             |                  84.7 |                   76.9 |                   79.1 |                   62.4 |                       75.8 |                            7.4 |                            11.9 |                            10.6 |                            15.7 |                                11.4 |

<!-- formula-not-decoded -->

and

<!-- formula-not-decoded -->

Similarly, to reduce the domain gap between xt and xs→t, we also introduce a discriminator Dtf and an auxiliary task for Ds. And the loss function is:

<!-- formula-not-decoded -->

and

<!-- formula-not-decoded -->

In summary, the overall objective function is:

<!-- formula-not-decoded -->

where we set λrec = 10, λf = 0.1, λaux = 0.1 and λseg = 1 to balance different losses.

## 3.5 Network Structure and Training Details

Our translation-segmentation joint network follows the structure settings from [Chen et al., 2019]. Overall, our joint network contains an encoder connecting to two decoder branches. The encoder contains three residual blocks[He et al., 2016] and down-sampling layers in sequence, then 8 residual blocks are followed and two dilated residual blocks continue to enlarge the receptive field, and two convolutional layers are connected at the end. For the translation branch, decoder contains a convolutional layer, 4 residual blocks and 3 deconvolutional layers to restore the resolution, then a convolutional layer and tanh activation function is followed. As for the segmentation branch, it contains a convolutional layer and an upsampling layer.

For the discriminator, we follow the setting of patchGAN[Isola et al., 2017] and use five convolutional layers to convolve in turn, with the first three stride being 2 and the last two being 1. Their kernel size is 4×4 and the channel number is 64, 128, 256, 512 and 1, respectively.

During the training process, we substitute the least-squares loss for the log likelihood objective in the adversarial loss referring to CycleGAN[Zhu et al., 2017], which can make the training process more stable[Mao et al., 2017]. We use the Adam solver[Kingma and Ba, 2014] with a batch size of 8 and initialize the learning rate to 0.0002 for translation task and 0.001 for segmentation task. The various parts of the network are trained in the following order: Es+Ss → Ts→t → Dt → Ds f →Et+St →Tt→s →Ds →Dtf .

## 4 Experiments

We validate our proposed method on two datasets - MultiModality Brain Tumor Segmentation Challenge 2018[Menze et al., 2014] and the Multi-Modality Whole Heart Segmentation Challenge 2017[Zhuang and Shen, 2016]. The first dataset contains four modalities of MRI imaging: T1, T1CE, T2, and FLAIR. There are a total of 75 patient data. Because experts always annotate whole tumor on T2 modality, we take T2 as the source domain with whole tumor annotation and aims to segment three target domain - T1, T1CE, FLAIR. Our experiments are conducted in an unpaired manner.

The second dataset is composed of 20 unpaired MR and CT imaging data, and their ground truth contains the annotation of four heart structures - the ascending aorta (AA), the left atrium blood cavity (LAC), the left ventricle blood cavity (LVC), and the myocardium of the left ventricle (MYO). We use MR images and their corresponding labels as the source domain dataset to segment the target domain CT images.

For both datasets, we randomly select 80% patient data for each modality as the training set and 20% as the testing set. All data subtract their mean and are divided by their standard deviation for normalization, then we switch them to [-1,1] before feeding to the network. We augment the data by rotation, cropping, etc., and each slice is resized to 256×256. The target labels of these two datasets are only used for evaluation.

Figure 3: Visual display of the cross-domain cardiac structures segmentation results for comparing different methods on MM-WHS dataset.

![Image](https://ijcai.org/proceedings/2020/fulltext/455/image_000002_46b66e78413fff93e95cc0c8bd0c612b510257ce79caca37f0ef1b4d2056abd1.png)

Table 3: Ablation study of fusion strategy and feature adaptation on BRATS dataset. Result1 and Result2 represent the results of these two schemes, respectively.

| Method                     |   Dice Score [%] - Result1 |   Dice Score [%] - Result2 |   Dice Score [%] - Fusion |
|----------------------------|----------------------------|----------------------------|---------------------------|
| Without feature adaptation |                       60.6 |                       61.8 |                      64.2 |
| With feature adaptation    |                       62.5 |                       63.4 |                      66.1 |

Table 4: Ablation study of fusion strategy and feature adaptation on MM-WHS dataset. Result1 and Result2 represent the results of these two schemes, respectively.

| Method                     |   Dice Score [%] - Result1 |   Dice Score [%] - Result2 |   Dice Score [%] - Fusion |
|----------------------------|----------------------------|----------------------------|---------------------------|
| Without feature adaptation |                       58.3 |                       60.1 |                      64.7 |
| With feature adaptation    |                       70.9 |                       73.7 |                      75.8 |

Our experiments are conducted from two aspects: comparison with other state-of-the-art unsupervised domain adaptation methods and ablation experiments. For evaluation, we select two metrics commonly used in medical image segmentation tasks: Dice coefficient[%] and Hausdorff Distance. Dice coefficient measures the overlapping part of our prediction results and the groundtruth. Hausdorff distance is a distance defined between two sets in the metric space.

## 4.1 Comparison Experiments

The methods we compare with are: CycleGAN[Zhu et al., 2017], CyCADA [Hoffman et al., 2017], AdaptSegNet[Tsai et al., 2018], SIFA[Chen et al., 2019]. In addition, we also compare the results without domain adaptation (directly input the target image into the trained source domain segmentation network).

Table 1 and Table 2 respectively show the comparison result between our method and others on the brain tumor segmentation dataset and cardiac segmentation dataset. It can be seen that our method has a significant improvement compared to no domain adaptation result by improving Dice Score from 22.6% to 66.1% on tumor segmentation task and from 17.2% to 75.8% on cardiac structures segmentation task, and reducing the Hausdorff Distance from 46.8mm to 15.6mm and from 44.8mm to 11.4mm respectively, which shows that our method is very effective in solving domain degradation problem. More importantly, our proposed method achieves better results than the current state-of-the-art methods on these two datasets. Especially when compared with SIFA[Chen et al., 2019], which only consider the source-to-target direction, we can increase the Dice by 6.8% on the BRATS dataset and by 2.8% on the MM-WHS datasetet after adopting both source-to-target and target-to-source direction and fusing their results, demonstrating the effectiveness of our proposed method.

Fig.2 and Fig.3 present the visual results compared with other methods. We can note that our outputs are not only closer to the ground truth, but also the wrong semantic prediction results are far less than others, indicating that our fusion strategy can indeed make the results more robust.

## 4.2 Ablation Experiments

We perform ablation experiments on feature adaptation and fusion to verify the influence of each component for our proposed Dual-Scheme Fusion Network.

The ablation experiments are set as follows, we conduct experiments respectively with or without feature adaptation, in each case, we compare the results of three schemes: sourceto-target scheme, target-to-source scheme, and their fusion solution. Table 3 and Table 4 present the ablation study results, Result1 represents the target-to-source method, Result2 represents the source-to-target method and Fusion represents our Dual-Scheme Fusion method.

It can be seen that, in both datasets, the results of adding feature adaptation are improved in all three cases compared with no feature adaptation. Among them, the improvement on the MM-WHS dataset is particularly obvious, and the Dice score can increase over 10% compared to no feature adaptation, demonstrating the effectiveness of feature adaptation.

Figure 4: Ablation studies on (a) BRATS dataset and (b) MM-WHS dataset. The first and second rows present the results without and with feature adaptation, respectively. The second and third columns are the segmentation prediction results of two schemes, and the fourth column is their fusion result, respectively.

![Image](https://ijcai.org/proceedings/2020/fulltext/455/image_000003_d62b7489027eaf6e6af788c483efb63e33c5faa94bcadce299f1ef4d264ef452.png)

And regardless of whether feature adaptation is added or not, compared with experiments conducted from source-totarget or target-to-source perspective alone, the experimental results are more accurate after using our dual-scheme fusion strategy on both datasets. And Dice score can improve about 3% on BRATS dataset and 4% on MM-WHS dataset without feature adaptation. If feature adaptation is added, our method can further improve the Dice score by about 3% on BRATS and 2% on MM-WHS, which reflects the effectiveness of fusion strategy.

Fig.4(a) and Fig.4(b) present the visual results on BRATS dataset and MM-WHS dataset respectively. The first row shows the results without feature adaptation while the second row shows them with feature adaptation. It can be seen that overall the results of adding feature adaptation are more accurate than not adding. The first column is the input images, prediction1 is the results from target-to-source scheme, prediction2 is from source-to-target scheme and fusion result means adopting our fusion strategy. We can see that the edges of fusion images are smoother than the two separate prediction results and are closer to the ground truth. Moreover, from Fig.4(b) we can see that the misprediction outside the cardiac structures from the two prediction results also disappear after fusion strategy, which means through our fusion strategy, the results from target-to-source and source-to-target perspectives can complement each other and a smoother and more robust result can be obtained.

## 5 Conclusion

In this paper, we address the problem of domain adaptation from two complementary perspectives and unify these two approaches into an end-to-end network in an elegant and efficient way. Through the cycle structure, not only can the unpaird data training problem be solved, but also a clever combination of these two perspectives is obtained. Finally, we choose an effective fusion method to acquire the final result. For further improvement, we also add feature adaptation loss. The experimental results show that our proposed method greatly surpasses the baseline and is superior to other state-of-the-art methods.

## Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grants 62041105, 6182211, the Science and Technology Major Project of Hubei Province (Next-Generation AI Technologies) under Grant 2019AEA170, Natural Science Foundation of Hubei Province under Grants 2018CFA050. The numerical calculations in this paper have been done on the supercomputing system in the Supercomputing Center of Wuhan University.

## References

1. [Bousmalis et al., 2017] Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Kr- ishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 3722-3731, 2017.
2. [Chen et al., 2018] Cheng Chen, Qi Dou, Hao Chen, and Pheng-Ann Heng. Semantic-aware generative adversar- ial nets for unsupervised domain adaptation in chest x- ray segmentation. In International Workshop on Machine Learning in Medical Imaging, pages 143-151. Springer, 2018.
3. [Chen et al., 2019] Cheng Chen, Qi Dou, Hao Chen, Jing Qin, and Pheng-Ann Heng. Synergistic image and fea- ture adaptation: Towards cross-modality domain adap- tation for medical image segmentation. arXiv preprint arXiv:1901.08211, 2019.
4. [Dou et al., 2018] Qi Dou, Cheng Ouyang, Cheng Chen, Hao Chen, and Pheng-Ann Heng. Unsupervised cross- modality domain adaptation of convnets for biomedical image segmentations with adversarial loss. arXiv preprint arXiv:1804.10916, 2018.
5. [Fang and Yan, 2020] Xi Fang and Pingkun Yan. Multi- organ segmentation over partially labeled datasets with multi-scale feature abstraction. arXiv preprint arXiv:2001.00208, 2020.
6. [Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
7. [He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
8. [Hoffman et al., 2017] Judy Hoffman, Eric Tzeng, Tae- sung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell. Cycada: Cycle- consistent adversarial domain adaptation. arXiv preprint arXiv:1711.03213, 2017.
9. [Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with con- ditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-1134, 2017.
10. [Kamnitsas et al., 2017] Konstantinos Kamnitsas, Christian Baumgartner, Christian Ledig, Virginia Newcombe, Joanna Simpson, Andrew Kane, David Menon, Aditya Nori, Antonio Criminisi, Daniel Rueckert, et al. Unsuper- vised domain adaptation in brain lesion segmentation with adversarial networks. In International conference on in- formation processing in medical imaging, pages 597-609. Springer, 2017.
11. [Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
12. [Long et al., 2015] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I Jordan. Learning transferable fea- tures with deep adaptation networks. arXiv preprint arXiv:1502.02791, 2015.
13. [Mao et al., 2017] Xudong Mao, Qing Li, Haoran Xie, Ray- mond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In ICCV, pages 2794-2802, 2017.
14. [Menze et al., 2014] Bjoern H Menze, Andras Jakab, Ste- fan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slot- boom, Roland Wiest, et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE transactions on medical imaging, 34(10):1993-2024, 2014.
15. [Nie et al., 2017] Dong Nie, Roger Trullo, Jun Lian, Caro- line Petitjean, Su Ruan, Qian Wang, and Dinggang Shen. Medical image synthesis with context-aware generative adversarial networks. In International Conference on Medical Image Computing and Computer-Assisted Inter- vention, pages 417-425. Springer, 2017.
16. [Tsai et al., 2018] Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmo- han Chandraker. Learning to adapt structured output space for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7472-7481, 2018.
17. [Tzeng et al., 2014] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confu- sion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.
18. [Wolterink et al., 2017] Jelmer M Wolterink, Anna M Din- kla, Mark HF Savenije, Peter R Seevinck, Cornelis AT van den Berg, and Ivana Išgum. Deep mr to ct synthesis using unpaired data. In International Workshop on Sim- ulation and Synthesis in Medical Imaging, pages 14-23. Springer, 2017.
19. [Zhao et al., 2018] He Zhao, Huiqi Li, Sebastian Maurer- Stroh, Yuhong Guo, Qiuju Deng, and Li Cheng. Su- pervised segmentation of un-annotated retinal fundus im- ages by synthesis. IEEE transactions on medical imaging, 38(1):46-56, 2018.
20. [Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image transla- tion using cycle-consistent adversarial networks. In ICCV, pages 2223-2232, 2017.
21. [Zhu et al., 2020] Qikui Zhu, Bo Du, and Pingkun Yan. Boundary-Weighted Domain Adaptive Neural Network for Prostate MR Image Segmentation. IEEE Transactions on Medical Imaging, 39(3):753-763, March 2020.
22. [Zhuang and Shen, 2016] Xiahai Zhuang and Juan Shen. Multi-scale patch and multi-modality atlases for whole heart segmentation of mri. Medical image analysis, 31:77- 87, 2016.