TRANSFER LEARNING WITH PRE-TRAINED CONDITIONAL GENERATIVE MODELS

Abstract

Transfer learning is crucial in training deep neural networks on new target tasks. Current transfer learning methods always assume at least one of (i) source and target task label spaces overlap, (ii) source datasets are available, and (iii) target network architectures are consistent with source ones. However, holding these assumptions is difficult in practical settings because the target task rarely has the same labels as the source task, the source dataset access is restricted due to storage costs and privacy, and the target architecture is often specialized to each task. To transfer source knowledge without these assumptions, we propose a transfer learning method that uses deep generative models and is composed of the following two stages: pseudo pre-training (PP) and pseudo semi-supervised learning (P-SSL). PP trains a target architecture with an artificial dataset synthesized by using conditional source generative models. P-SSL applies SSL algorithms to labeled target data and unlabeled pseudo samples, which are generated by cascading the source classifier and generative models to condition them with target samples. Our experimental results indicate that our method can outperform the baselines of scratch training and knowledge distillation.

1. INTRODUCTION

For training deep neural networks on new tasks, transfer learning is essential, which leverages the knowledge of related (source) tasks to the new (target) tasks via the joint-or pre-training of source models. There are many transfer learning methods for deep models under various conditions (Pan & Yang, 2010; Wang & Deng, 2018) . For instance, domain adaptation leverages source knowledge to the target task by minimizing the domain gaps (Ganin et al., 2016) , and fine-tuning uses the pre-trained weights on source tasks as the initial weights of the target models (Yosinski et al., 2014) . These existing powerful transfer learning methods always assume at least one of (i) source and target label spaces have overlaps, e.g., a target task composed of the same class categories as a source task, (ii) source datasets are available, and (iii) consistency of neural network architectures i.e., the architectures in the target task must be the same as that in the source task. However, these assumptions are seldom satisfied in real-world settings (Chang et al., 2019; Kenthapadi et al., 2019; Tan et al., 2019) . For instance, suppose a case of developing an image classifier on a totally new task for an embedded device in an automobile company. The developers found an optimal neural architecture for the target dataset and the device by neural architecture search, but they cannot directly access the source dataset for the reason of protecting customer information. In such a situation, the existing transfer learning methods requiring the above assumptions are unavailable, and the developers cannot obtain the best model. To promote the practical application of deep models, we argue that we should reconsider the three assumptions on which the existing transfer learning methods depend. For assumption (i), new target tasks do not necessarily have the label spaces overlapping with source ones because target labels are often designed on the basis of their requisites. In the above example, if we train models on StanfordCars (Krause et al., 2013) , which is a fine-grained car dataset, there is no overlap with ImageNet (Russakovsky et al., 2015) even though ImageNet has 1000 classes. For (ii), the accessibility of source datasets is often limited due to storage costs and privacy (Liang et al., 2020; Kundu et al., 2020; Wang et al., 2021a) , e.g., ImageNet consumes over 100GB and contains person faces co-occurring with objects that potentially raise privacy concerns (Yang et al., 2022) . For (iii), the consistency of the source and target architectures is broken if the new architecture is specialized for < l a t e x i t s h a 1 _ b a s e 6 4 = " K L T 7 c K n U F y k d 5 Q T Q k / M B x T v G / Q o = " > A A A C Z n i c h V G 7 S g N B F D 1 Z X z G + o i I K N s E Q s Q o 3 I i p W g T S W i T E q x B B 2 1 z E u 2 e w u u 5 O A B n 9 A s D W F l Y K I + B k 2 / o C F f 6 B Y K t h Y e L N Z E A 3 q H W b m z J l 7 7 p y Z 0 R z T 8 C T R Y 0 j p 6 e 3 r H w g P R o a G R 0 b H o u M T W 5 5 d d 3 V R 0 G 3 T d n c 0 1 R O m Y Y m C N K Q p d h x X q D X N F N t a N d P e 3 2 4 I 1 z N s a 1 M e O q J U U y u W s W / o q m Q q n y n L c j R O S f I j 1 g 1 S A Y g j i K w d v c Y u 9 m B D R x 0 1 C F i Q j E 2 o 8 L g V k Q L B Y a 6 E J n M u I 8 P f F z h G h L V 1 z h K c o T J b 5 b H C q 2 L A W r x u 1 / R 8 t c 6 n m N x d V s a Q o A e 6 o V e 6 p 1 t 6 p o 9 f a z X 9 G m 0 v h z x r H a 1 w y m M n M / n 3 f 1 U 1 n i U O v l R / e p b Y x 6 r v 1 W D v j s + 0 b 6 F 3 9 I 2 j 1 m t + b S P R n K d L e m H / F / R I d 3 w D q / G m X + X E x j k i / A G p n 8 / d D b Y W k 6 n l 5 F J u K Z 5 e D b 4 i j F n M Y Y H f e w V p r C O L A p 9 b w S n O 0 A o 9 K a P K l D L d S V V C g W Y S 3 0 K J f Q J 9 / Y q p < / l a t e x i t > C t

Supervised Loss

Unsupervised Loss < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 W z e B u T S n r i n Y W w d l e d q C N 0  1 Z 4 M = " > A A A C g H i c h V H L L g R B F D 3 a a 4 z X Y C O x m Z g Q q 1 E t g l i I h I W l 1 y A x M q l u N a a j + p H u m p H R m Y W t H 7 C w I h E R G 7 7 B x g 9 Y + A S x J L G x c K e n E 0 F w K 1 V 1 6 t Q 9 t 0 5 V G Z 6 0 A s X Y Y 5 P W 3 N L a 1 p 7 o S H Z 2 d f f 0 p v r 6 N w K 3 7 J s i Z 7 r S 9 b c M H g h p O S K n L C X F l u c L b h t S b B r 7 C / X 9 z Y r w A 8 t 1 1 l X V E z s 2 3 3 O s o m V y R V Q h N Z S 3 u S q Z X I a L t U I Y 5 K U o K u 7 7 7 k F a 1 Q q p D M u y K N I / g R 6 D D O J Y d l O X y G M X L k y U Y U P A g S I s w R F Q 2 4 Y O B o + 4 H Y T E + Y S s a F + g h i R p y 5 Q l K I M T u 0 / j H q 2 2 Y 9 a h d b 1 m E K l N O k V S 9 0 m Z x g h 7 Y F f s h d 2 z a / b E 3 n + t F U Y 1 6 l 6 q N B s N r f A K v c e D a 2 / / q m y a F U q f q j 8 9 K x Q x E 3 m 1 y L s X M f V b m A 1 9 5 f D k Z W 1 2 d S Q c Z e f s m f y f s U d 2 R z d w K q / m x Y p Y P U W S P k D / / t w / w c Z E V p / K T q 5 M Z u b n 4 q 9 I Y A j D G K P 3 n s Y 8 l r C M H J 1 7 h E v c 4 F b T t D F t X N M b q V p T I m q Y Y / t p R W Z R T a Z Z h Y Y z 9 B H I E 0 o h i w 0 5 d Y R d 7 s K G h D h M c F g R h A w o 8 a k X I Y H C I K 8 E n z i W k h / s c R 0 i Q t k 5 Z n D I U Y v d p r N K q G L E W r Y O a X q j W 6 B S D u k v K M U O o = " > A A A C Z n i c h V F N S w J B G H 7 c v s x K r Y i C L p I Y n W Q M q e g Q Q o c 6 + p E f Y C K 7 2 2 i L 6 + 6 y u w o m / Y G g a x 4 6 F U R E P 6 N L f 6 C D / 6 D o a N C l Q 6 / r Q p R U 7 z A z z z z z P u 8 8 M y M Z q m L Z j H U 9 w s j o 2 P i E d 9 I 3 N T 3 j D w R n 5 3 K W 3 j B l n p V 1 V T c L k m h x V d F 4 1 l Z s l R c M k 4 t 1 S e V 5 q b b b 3 8 8 3 u W k p u n Z g t w x e q o t V T a k o s m g T l d k r W + V g m E W Z E 6 F h E H N B G G 4 k 9 e A t D n E E H T I a q I N D g 0 1 Y h Q i L W h E x M B j E l d A m z i S k O P s c p / C R t k F Z n D J E Y m s 0 V m l V d F m N 1 v 2 a l q O W 6 R S V u k n K E C L s i d 2 x H n t k 9 + y F f f x a q + 3 U 6 H t p 0 S w N t N w o B 8 6 W M u / / q u o 0 2 z j + U v 3 p 2 U Y F W 4 5 X h b w b D t O / h T z Q N 0 8 6 v c x 2 O t J e Z d f s l f x f s S 5 7 o B t o z T f 5 J s X T l / D R B 8 R + P v c w y K 1 H Y x v R e C o e T u y 4 X + H F M l a w R u + 9 i Q T 2 k U S W z q 3 i H B f o e J 4 F v 7 A g L A 5 S B Y + r m c e 3 E E K f h d O K s g = = < / l a t e x i t > G s < l a t e x i t s h a 1 _ b a s e 6 4 = " x I e W D X R V n y 7 X Z R B g z 5 X W 3 p z i B o M = " > A A A C d n i c h V H L S s N A F D 2 N r 1 p f V T e C I M V S d V V u R V R c S M G N y / p o K 6 i U J E 5 r M E 1 C M m 3 V 4 g / 4 A y 7 E h Y K K + B l u / A E X f o K 4 V N C F C 2 / T g G h R 7 z A z Z 8 7 c c + f M j O a Y h i e J H k N K W 3 t H Z 1 e 4 O 9 L T 2 9 c / E B 0 c y n l 2 x d V F V r d N 2 9 3 Q V E + Y h i W y 0 p C m 2 H B c o Z Y 1 U + S 1 v a X G f r 4 q X M + w r X V 5 4 I j t s l q y j K K h q 5 K p Q n R w v 1 D 3 t k x R l K r r 2 r W Y P C p E 4 5 Q k P 2 K t I B W A O I L I 2 N F r b G E H N n R U U I a A B c n Y h A q P 2 y Z S I D j M b a P O n M v I 8 P c F j h B h b Y W z B G e o z O 7 x W O L V Z s B a v G 7 U 9 H y 1 z q e Y 3 F 1 W x p C g B 7 q h F 7 q n W 3 q i j 1 9 r 1 f 0 a D S 8 H P G t N r X A K A 8 c j a 2 / / q s o 8 S + x + q f 7 0 L F H E v O / V Y O + O z z R u o T f 1 1 c O T l 7 W F 1 U R 9 g i 7 o m f 2 f 0 y P d 8 Q 2 s 6 q t + u S J W T x H h D 0 j 9 f O 5 W k J t O p m a T M y s z 8 f R i 8 B V h j G I c U / z e c 0 h j G R l k + d w a z n C F 6 9 C 7 M q Y k l M l m q h I K N M P 4 F g p 9 A j A M k V 4 = < / l a t e x i t > x s t < l a t e x i t s h a 1 _ b a s e 6 4 = " m z W W i s g k H v H j I E p 4 x t q T a s 3 0 5 O w = " > A A A C Z n i c h V G 7 S g N B F D 1 Z 3 / G R q I i C T T A o V u F G g o q F C D a W e R g T U A m 7 6 x i X b H a X 3 U l Q g z 8 g 2 G p h p S A i f o a N P 2 C R P 1 A s F W w s v N k s i I p 6 h 5 k 5 c + a e O 2 d m N M c 0 P E n U D C k d n V 3 d P b 1 9 4 f 6 B w a F I d H h k w 7 N r r i 7 y u m 3 a b l F T P W E a l s h L Q 5 q i 6 L h C r W q m K G i V 1 d Z + o S 5 c z 7 C t d X n g i O 2 q W r a M X U N X J V O 5 / Z I s R e O U I D 9 i P 0 E y A H E E k b a j 1 9 j C D m z o q K E K A Q u S s Q k V H r d N J E F w m N t G g z m X k e H v C x w h z N o a Z w n O U J m t 8 F j m 1 W b A W r x u 1 f R 8 t c 6 n m N x d V s Y w T Q 9 0 Q y 9 0 T 7 f 0 R O + / 1 m r 4 N V p e D n j W 2 l r h l C L H E 7 m 3 f 1 V V n i X 2 P l V / e p b Y x a L v 1 W D v j s + 0 b q G 3 9 f X D s 5 f c U n a 6 M U O X 9 M z + L 6 h J d 3 w D q / 6 q X 2 V E 9 h x h / o D k 9 + f + C T b m E s n 5 R C q T i q 8 s B 1 / R i 0 l M Y Z b f e w E r W E M a e T 6 3 j B O c 4 i z 0 q A w p Y 8 p 4 O 1 U J B Z p R f A k l 9 g H q N Y r k < / l a t e x i t > x t < l a t e x i t s h a 1 _ b a s e 6 4 = " 9 H V W x C 0 Z d f G e q g 9 2 a C i 7 7 x Q 3 H 0 k = " > A A A C d n i c h V H L S s N A F D 2 N 7 / q q u h E E K Z a q q 3 I j o u J C B D c u q 7 V V U C l J n G v X f G G Z t s h K U 1 p i y / W E V t Q t s a k f r j T 3 N 8 v C 8 0 3 H 3 p A V V + w W t X 3 b L J i G J p n K x 4 Y q + a q / Y 4 m C 1 D z P O Y r L k 3 w s Q S k K I v 4 T q C F I I I y 0 E 6 t h B 3 t w Y K C E I g R s S M Y W N P j c t q G C 4 D K 3 i y p z H i M z 2 B c 4 Q Z S 1 J c 4 S n K E x e 8 j j P q + 2 Q 9 b m d b O m H 6 g N P s X i 7 r E y j i Q 9 0 i 0 1 6 I H u 6 I n e f 6 1 V D W o 0 v V R 4 1 l t a 4 e Y H T 0 c z r / + q i j x L H H y q / v Q s U c B C 4 N V k 7 2 7 A N G 9 h t P T l 4 7 N G Z n E 9 W Z 2 k K 3 p m / 5 d U p 3 u + g V 1 + M a 7 X x P o 5 o v w B 6 v f n / g l y M y l 1 L j W 7 N p t Y X g q / o h t j m M A 0 v / c 8 l r G K N L J 8 7 h E u c I N a 5 E 0 Z V 5 L K V C t V i Y S a E X w J h T 4 A M h y R X w = = < / l a t e x i t > y s t < l a t e x i t s h a 1 _ b a s e 6 4 = " x I e W D X R V n y 7 X Z R B g z 5 X W 3 p z i B o M = " > A A A C d n i c h V H L S s N A F D 2 N r 1 p f V T e C I M V S d V V u R V R c S M G N y / p o K 6 i U J E 5 r M E 1 C M m 3 V 4 g / 4 A y 7 E h Y K K + B l u / A E X f o K 4 V N C F C 2 / T g G h R 7 z A z Z 8 7 c c + f M j O a Y h i e J H k N K W 3 t H Z 1 e 4 O 9 L T 2 9 c / E B 0 c y n l 2 x d V F V r d N 2 9 3 Q V E + Y h i W y 0 p C m 2 H B c o Z Y 1 U + S 1 v a X G f r 4 q X M + w r X V 5 4 I j t s l q y j K K h q 5 K p Q n R w v 1 D 3 t k x R l K r r 2 r W Y P C p E 4 5 Q k P 2 K t I B W A O I L I 2 N F r b G E H N n R U U I a A B c n Y h A q P 2 y Z S I D j M b a P O n M v I 8 P c F j h B h b Y W z B G e o z O 7 x W O L V Z s B a v G 7 U 9 H y 1 z q e Y 3 F 1 W x p C g B 7 q h F 7 q n W 3 q i j 1 9 r 1 f 0 a D S 8 H P G t N r X A K A 8 c j a 2 / / q s o 8 S + x + q f 7 0 L F H E v O / V Y O + O z z R u o T f 1 1 c O T l 7 W F 1 U R 9 g i 7 o m f 2 f 0 y P d 8 Q 2 s 6 q t + u S J W T x H h D 0 j 9 f O 5 W k J t O p m a T M y s z 8 f R i 8 B V h j G I c U / z e c 0 h j G R l k + d w a z n C F 6 9 C 7 M q Y k l M l m q h I K N M P 4 F g p 9 A j A M k V 4 = < / l a t e x i t > x s t < l a t e x i t s h a 1 _ b a s e 6 4 = " W y u e G G E x p J B F M R p h e z j n M f s 8 the new tasks like the above example. Deep models are often specialized for tasks or computational resources by neural architecture search (Zoph & Le, 2017; Lee et al., 2021) in particular when deploying on edge devices; thus, their architectures can differ for each task and runtime environment. Since existing transfer learning methods require one of the three assumptions, practitioners must design target tasks and architectures to fit those assumptions by sacrificing performance. To maximize the potential performance of deep models, a new transfer learning paradigm is required. C C A = " > A A A C b X i c h V H L S g M x F D 0 d X 7 U + W h V B U K R Y f I G U V I q K C y m 4 c d l W q 8 U q Z W a M d e h 0 Z p h J i 7 X 4 A 6 4 F F 6 K g I C J + h h t / w E U / Q V y 4 U H D j w t v p g K i o N y Q 5 O b n n 5 i R R L F 1 z B G N 1 n 9 T S 2 t b e 4 e 8 M d H X 3 9 A Z D f f 3 r j l m 2 V Z 5 R T d 2 0 s 4 r s c F 0 z e E Z o Q u d Z y + Z y S d H 5 h l J c b u x v V L j t a K a x J q o W 3 y 7 J B U P b 1 V R Z E L U 5 t Z 8 X M + F q X k z n Q x E W Z W 6 E f 4 K Y B y L w I m m G r r G F H Z h Q U U Y J H A Y E Y R 0 y H G o 5 x M B g E b e N G n E 2 I c 3 d 5 z h E g L R l y u K U I R N b p L F A q 5 z H G r R u 1 H R c t U q n 6 N R t U o Y x z h 7 Y D X t h 9 + y W P b L 3 X 2 v V 3 B o N L 1 W v G r C q K m 4 F Z I 0 / O q T i k q C 0 V N c 2 3 Z E W + 9 F v d R q 4 g i i I J 8 y 4 o p a C A N K p B 7 g t 2 0 E Q A B z E 8 E H x o x i 4 s K G 7 b K E E g Z G 4 X H e Y i R j L Z J 3 S R Z W 3 M W c Q Z F r M H P L Z 4 t Z 2 y P q 9 7 N V W i d v g U l 3 v E S h O T 4 k x 8 F V f i V H w T F + L v f q i s + 3 H W P X F U V h G Y o o P d l X 0 W W q I I W T K j i G 2 r M M u 7 p 5 p E s 9 m W Y Z F k b q J 8 h G I I 0 o F r 3 k J Q r Y g Q c L J T j g c K E I 2 z A g q W 0 h C w a f u G 1 U i Q s I i X C f 4 w g J 0 p Y o i 1 O G Q e w + O o = " > A A A C Z n i c h V F N S w J B G H 7 c v s x K r Y i C L p I Y n W Q M q e g Q Q o c 6 + p E f Y C K 7 2 2 i L 6 + 6 y u w o m / Y G g a x 4 6 F U R E P 6 N L f 6 C D / 6 D o a N C l Q 6 / r Q p R U 7 z A z z z z z P u 8 8 M y M Z q m L Z j H U 9 w s j o 2 P i E d 9 I 3 N T 3 j D w R n 5 3 K W 3 j B l n p V 1 V T c L k m h x V d F 4 1 l Z s l R c M k 4 t 1 S e V 5 q b b b 3 8 8 3 u W k p u n Z g t w x e q o t V T a k o s m g T l d k r W + V g m E W Z E 6 F h E H N B G G 4 k 9 e A t D n E E H T I a q I N D g 0 1 Y h Q i L W h E x M B j E l d A m z i S k O P s c p / C R t k F Z n D J E Y m s 0 V m l V d F m N 1 v 2 a l q O W 6 R S V u k n K E C L s i d 2 x H n t k 9 + y F f f x a q + 3 U 6 H t p 0 S w N t N w o B 8 6 W M u / / q u o 0 2 z j + U v 3 p 2 U Y F W 4 5 X h b w b D t O / h T z Q N 0 8 6 v c x 2 O t J e Z d f s l f x f s S 5 7 o B t o z T f 5 J s X T l / D R B 8 R + P v c w y K 1 H Y x v R e C o e T u y 4 X + H F M l a w R u + 9 i Q T 2 k U S W z q 3 i H B f o e J 4 F v 7 A g L A 5 S B Y + r m c e 3 E E K f h d O K s g = = < / l a t e x i t > G s < l a t e x i t s h a 1 _ b a s e 6 4 = " 8 H p Y e G 6 R s P B C n r m y A B A 9 4 L z c H 6 8 = " > A A A C e H i c h V H L S g M x F D 0 d X 7 W + q t 0 I b o r F 1 6 a k p a i I i 0 o 3 L m t r V f A x z I x R h 8 6 L m b R Q h / 6 A P + D C h S i I i p / h x h 9 w 4 S e I S w V B X H g 7 H R A V 9 Y Y k J y f 3 3 J w k q m P o n m D s I S J 1 d H Z 1 9 0 R 7 Y 3 3 9 A 4 N D 8 e G R N c + u u R q v a L Z h u x u q 4 n F D t 3 h F 6 M L g G 4 7 L F V M 1 + L p a L b T 2 1 + v c 9 X T b W h U N h 2 + b y r 6 l 7 + m a I o i S 4 4 m C 7 O 3 4 W 6 Y i D j T F 8 J e a s m j K 8 R R L s y C S P 0 E m B C m E U b T j l 9 j C L m x o q M E E h w V B 2 I A C j 9 o m M m B w i N u G T 5 x L S A / 2 O Z q I k b Z G W Z w y F G K r N O 7 T a j N k L V q 3 a n q B W q N T D O o u K Z O Y Y P f s m j 2 z O 3 b D H t n 7 r 7 X 8 o E b L S 4 N m t a 3 l j j x 0 N F p + / V d l 0 i x w 8 K n 6 0 7 P A H u Y D r z p 5 d w K m d Q u t r a 8 f H j + X F 0 o T / i Q 7 Z 0 / k / 4 w 9 s F u 6 g V V / 0 S 5 W e O k E M f q A z P f n / g n W s u n M b D q 3 k k v l F 8 O v i G I M 4 5 i m 9 5 5 D H s s o o k L n N n C K K 1 x H 3 q S k N C X N t F O l S K h J 4 E t I 2 Q / O k J I V < / l a t e x i t > C At s < l a t e x i t s h a 1 _ b a s e 6 4 = " b c z 4 3 l F q d S b + h Z G A c H Q b b l b J m F c = " > A A A C e H i c h V H L S g M x F D 0 d X 7 W + q m 4 E N 8 V S H 5 u S S l E R F 4 o b l 7 6 q g q 3 D z J j W w X k x k x b q 0 B / w B 1 y 4 E A V R 8 T P c + A M u + g n i s o I g L r y d D o g W 9 Y Y k J y f 3 3 J w k q m P o n m C s H p E 6 O r u 6 e 6 K 9 s b 7 + g c G h + P D I j m e X X Y 3 n N N u w 3 T 1 V 8 b i h W z w n d G H w P c f l i q k a f F c 9 X m 3 u 7 1 a 4 6 + m 2 t S 2 q D i + Y S s n S i 7 q m C K L k + O i q L A 7 8 v K m I I 0 0 x / J W a L G p y P M n S L I h E O 8 i E I I k w 1 u 3 4 D f I 4 h A 0 N Z Z j g s C A I G 1 D g U d t H B g w O c Q X 4 x L m E 9 G C f o 4 Y Y a c u U x S l D I f a Y x h K t 9 k P W o n W z p h e o N T r F o O 6 S M o E U e 2 J 3 r M E e 2 T 1 7 Z h + / 1 v K D G k 0 v V Z r V l p Y 7 8 t D p 2 N b b v y q T Z o G j L 9 W f n g W K W A i 8 6 u T d C Z j m L b S W v n J y 1 t h a 3 E z 5 k + y K v Z D / S 1 Z n D 3 Q D q / K q X W / c = " > A A A C d H i c h V H L S s N A F D 2 N r 1 o f r b o R d F E s F Q U p U y k q L q T g x m U f V g W V k M S x D U 2 T k E y L t f Q H / A E X X S In this paper, we shed light on an important but less studied problem setting of transfer learning, where (i) source and target task label spaces do not have overlaps, (ii) source datasets are not available, and (iii) target network architectures are not consistent with source ones (Tab. 1). To transfer source knowledge while satisfying the above three conditions, our main idea is to leverage source pre-trained discriminative and generative models; their architectures differ from that of target tasks. We focus on applying the generated samples from source class-conditional generative models for target training. Deep conditional generative models precisely replicate complex data distributions such as ImageNet (Brock et al., 2018; Karras et al., 2020; Dhariwal & Nichol, 2021) , and the pre-trained models are widely used for downstream tasks (Wang et al., 2018; Zhao et al., 2020a; Patashnik et al., 2021; Ramesh et al., 2022) . Furthermore, deep generative models have the potential to resolve the problem of source dataset access because they can compress information of large datasets into much smaller pre-trained weights (e.g., about 100MB in the case of a BigGAN generator), and safely generate informative samples without re-generating training samples by differential privacy training techniques (Torkzadehmahani et al., 2019; Augenstein et al., 2020; Liew et al., 2022) . By using conditional generative models, we propose a two-stage transfer learning method composed of pseudo pre-training (PP) and pseudo semi-supervised learning (P-SSL). Figure 1 illustrates an overview of our method. PP pre-trains the target architectures by using the artificial dataset generated from the source conditional generated samples and given labels. This simple pre-process provides effective initial weights without accessing source datasets and architecture consistency. To address the non-overlap of the label spaces without accessing source datasets, P-SSL trains a target model with SSL (Chapelle et al., 2006; Van Engelen & Hoos, 2020) by treating pseudo samples drawn from the conditional generative models as the unlabeled dataset. Since SSL assumes the labeled and unlabeled datasets are drawn from the same distribution, the pseudo samples should be target-related samples, whose distribution is similar enough to the target distribution. To generate target-related samples, we cascade a classifier and conditional generative model of the source domain. Specifically, we (a) obtain pseudo source soft labels from the source classifier by applying



r B n A l 9 B m P w D L 7 Z S 9 < / l a t e x i t > D s t < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 Z Z I z I Y v G q i p y g L a u G s S t b 3 e 8 h M = "> A A A C c H i c h V G 7 S g N B F D 1 Z X z E + E r U R L H w E R S z C r I i K h Q h a W P r K A 2 I I u + s k W d y X u 5 N A X P w B f 8 D C R g U R 8 T N s / A E L P 0 H s V L C x 8 O 5 m Q V T U O 8 z M m T P 3 3 D k z o z q G 7g n G H m J S W 3 t H Z 1 e 8 O 9 H T 2 9 e f T A 0 M 5 j y 7 7 m o 8 q 9 m G 7 R Z U x e O G b v G s 0 I X B C 4 7 L F V M 1 e F 7 d X w 3 2 8 w 3 u e r p t 7 Y i m w 0 u m U r X 0 i q 4 p g q j S r q m

y y e 3 b N X t g d u 2 G P 7 P 3 X W n 5 Y I / D S p F l t a b l T T h 4 P b 7 / 9 q z J p F q h 9 q v 7 0L F D B Y u h V J + 9 O y A S 3 0 F r 6 x u H J y / b S 1 q Q / x S 7 Y E / k / Z w / s l m 5 g N V 6 1 y 0 2 + d Y o E f Y D 8 / b l / g t x s R p 7 P z G 3 O p V e W o 6 + I Y w Q T m K b 3 X s A K 1 r G B L J 1 7 g B O c 4 T z 2 L A 1 L o 9 J 4 K 1 W K R Z o h f A lp 5 g N z A 4 9 C < / l a t e x i t > D t < l a t e x i t s h a 1 _ b a s e 6 4 = " H 9 o 4 G f Z U X Z + O W 9 m h l c y c 0 l w 3 d

o w T U I y r d T i D / g D L s S F g h b x M 9 z 4 A y 7 6 C e K y g i 5 c e J s G R E W 9 w 8 y c O X P P n T M z u m u Z v i S q R 5 S 2 9 o 7 O r u 6 e a G 9 f / 8 B g b G g 4 5 z s l z x B Z w 7 E c b 0

a l a a W W / n g 0 d D q 2 7 + q E s 0 C e 5 + q P z 0 L 7 G L B 9 a q R d 8 t l G r d Q m / r K w c n L 6 m J 6 v D b B L t k T + b 9 g d X Z H N z A q r + p V i q d P E a A P i H 1 / 7 p 9 g f T Y a m 4 v G U / F I Y s n 7 C j + G M Y Y p e u 9 5 J L C C J D J 0 r o F j n O H c 9 y w N S i P S a D N V 8 n m a A X w J a f I D z 4 m N E w = = < / l a t e x i t > (x t , y t ) < l a t e x i t s h a 1 _ b a s e 6 4 = " m D Z c x 7 i 3 0 8D t A D S L v w O 8 A 1 Y z N s s = " > A A A C g 3 i c h V H L S g M x F D 2 O r 1 p f V T e C I M W i V M S S S t E i C A U 3 L l t r V V A p M 9 N Y g 9 O Z Y S Y t 1 t K d K 3 / A h S s F E X W p f + D G H 3 D R T x C X C m 5 c e D s d E B X 1 h i Q n J / f c n C S a b Q h X M t Z o U 9 o 7 O r u 6 A z 3 B 3 r 7 + g c H Q 0 P C 6 a 5 U d n e d 0 y 7 C c T U 1 1 u S F M n p N C G n z T d r h a 0 g y + o e 0 v N / c 3 K t x x h W W u y a r N d 0 p q 0 R S 7 Q l c l U f n Q + L b k B 7 I W V a f D a Z e X C 1 Y 4 7 f B Z 6 a j C F G a x n g 9 F W I x 5 E f 4 J 4 j 6 I w I + 0 F b r E N g q w o K O M E j h M S M I G V L j U t h A H g 0 3 c D m r E O Y S E t 8 9 R R 5 C 0 Z c r i l K E S u 0 9 j k V Z b P m v S u l n T 9 d Q 6 n W J Q d 0 g Z x i R 7 Z F f s h T 2 w G / b E 3 n + t V f N q N L 1 U a d Z a W m 7 n B 4 9 H s 2 / / q k o 0 S + x 9 q v 7 0 L L G L p O d V k H f b Y 5 q 3 0 F v 6 y u H J S 3 Z x d b I 2 x c 7 Z M / k / Y w 1 2 T z c w K 6 / 6 R Y a v n i J I H x D / / t w / w f p c L D 4 f S 2 Q S k V T S / 4 o A x j C B K L 3 3 A l J Y Q R o 5 O v c I 1 7 j F n d K p z C h z S q K V q r T 5 m h F 8 C W X p A5 w y l P 4 = < / l a t e x i t > (a) Pseudo Pre-training < l a t e x i t s h a 1 _ b a s e 6 4 = " g 4 m l I r B G G M d z A j 2 1 5 T s R v c T X z s g = " > A A A C j 3 i c h V H P S x t B G H 1 Z 2 6 p p r Y m 9 C L 0 s B k U F w 0 S C i o c S 6 M V C D 9 E Y F V T C 7 u Y z D u 4 v d m a D G v I P 9 N R b D z 1 V K K X 0 3 m s L v f g P e P B P E I 8 W e u n B L 5 s F s V L 9 h p l 5 8 + Z 7 3 7 y Z s U N X K i 3 E e c Y Y e P T 4 y e D Q c P b p s 5 H n o 7 n 8 2 I Y K 4 s i h u h O 4 Q b R l W 4 p c 6 V N d S + 3 S V h i R 5 d k u b d o H r 3 v 7 m 2 2 K l A z 8 d X 0 U 0 q 5 n t X y 5 J x 1 L M 9 X I z e 5 o O t S d a X

2 t 1 k h o 9 L 0 c 8 2 3 0 t h Y 3 R d + O 1 P w + q P J 4 1 9 m 9 U 9 3 r W 2 M N S 4 l W y 9 z B h e r d w + v r 2 8 Y e r 2 v L a Z G d K n I h L 9 v 9 J n I t f f A O / / d v 5 v E p r H 5 H l D y j 9 + 9 x 3 w c Z 8 s b R Q L K + W C 5 W l 9 C u G 8 B I T m O b 3 X k Q F K 6 i i z u e + x 3 f 8 w E 8 j b y w a r 4 x K P 9 X I p J o X u B X G m 2 t M n J o c < / l a t e x i t > (b) Pseudo Semi-supervised Learning Supervised Loss < l a t e x i t s h a 1 _ b a s e 6 4 = " b K D h q y t f M L u 6 5 9 N m 7 v k M I B u a R e Q = " > A A A C e 3 i c h V H L S g M x F D 0 d X 7 W + q o I I b o p F E Z G S S n 3 g Q g Q 3 L n 3 V B 1 a G m T H V 4 L y Y p I V a / Q F / w I U r B R H R v 3 D j D 7 j w E 8 S l g h s F b 6 c D o q L e k O T k 5 J 6 b k 8 T 0 b S E V Y w 8 x r a G x q b k l 3 p p o a + / o 7 E p 2 9 6 x J r x R Y P G 9 5 t h d s m I b k t n B 5 X g l l 8 w 0 / 4 I Z j 2 n z d 3 J + v 7 a + X e S C F 5 6 6

j b u 0 2 o p Y l 9 a 1 m j J U W 3 S K T T 0 g Z Q p D 7 J 5 d s W d 2 x 6 7 Z I 3 v 7 t V Y 1 r F H z U q H Z r G u 5 r 3 c d 9 6 + 8 / q t y a F b Y + 1 T 9 6 V m h i O n Q q y D v f s j U b m H V 9 e W D k + e V m e W h 6 j A 7 Z 0 / k / 4 w 9 s F u 6 g V t + s S 6 W + P I p E v Q B 2 e / P / R O s j W e y k 5 n c U i 4 9 Nx t 9 R R w D G M Q I v f c U 5 r C A R e T p 3 E O c 4 x o 3 s X c t r Y 1 q Y / V U L R Z p e v El t I k P D A a T E Q = = < / l a t e x i t > y s ⇠ Y s < l a t e x i t s h a 1 _ b a s e 6 4 = " H 9 o 4 G f Z U X Z + O W 9 m h l c y c 0 l w 3 d

w z X P E 6 A M y P 5 + 7 H e z M p j N z 6 e x G N r m 8 F H 5 F F O O Y w D S 9 9 z y W s Y Z 1 5 O j c K i 5 w i 7 v I u 5 S Q pq S Z V q o U C T W j + B b S 7 C f Q o J I W < / l a t e x i t > C At t < l a t e x i t s h a 1 _ b a s e 6 4 = " m z V O V 2 V A L 7 d m c Z 0 + g m W 8 Z n b d 6 B U = " > A A A C e H i c h V H L S g M x F D 0 d X 7 W + q m 4 E N 8 V S H 5 u S S l E R F 4 o b l 7 6 q g q 3 D z J j W w X k x k x b q 0 B / w B 1 y 4 E A V R 8 T P c + A M u + g n i s o I g L r y d D o g W 9 Y Y k J y f 3 3 J w k q m P o n m C s H p E 6 Or u 6 e 6 K 9 s b 7 + g c G h + P D I j m e X X Y 3 n N N u w 3 T 1 V 8 b i h W z w n d G H w P c f l i q k a f F c 9 X m 3 u 7 1 a 4 6 + m 2 t S 2 q D i + Y S s n S i 7 q m C K L k + O i q 7 B 3 4 e V M R R 5 p i + C s 1 2 a v J 8 S R L s y A S 7 S A T g i T C W L f j N 8 j j E D Y 0 l G G C w 4 I g b E C B R 2 0 f G T A 4 x B X g E + c S 0 o N 9 j h p i p C 1 T F q c M h d h j G k u 0 2 g 9 Z i 9 b N m l 6 g 1 u g U g 7 p L y g R S 7 I n d s Q Z 7 Z P f s m X 3 8 W s s P a j S 9 V G l W W 1 r u y E O n Y 1 t v / 6 p M m g W O v l R / e h Y o Y i H w q p N 3 J 2 C a t 9 B a + s r J W W N r c T P l T 7 I r 9 k L + L 1 m d P d A N r M q r d r 3 B N 8 8 R o w / I / H z u d r A z m 8 7 M p b M b 2 e T y U v g V U Y x j A t P 0 3 v N Y x h r W k a N z q 7 j A L e 4 i 7 1 J C m p J m W q l S J N S M 4 l t I s 5 / M j 5 I U < / l a t e x i t > C As s < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 T S M o / 3 n f u t J 4 t m H / w / P 8 d 7 b o X

Figure 1: Proposed transfer learning methods leveraging conditional source generative model Gs. Red color represents given source models, light blue represents target models and datasets, and dark blue represents the output of the proposed methods. (a) We produce initial weights of a target architecture At by training a source classifier C A t s with pairs of conditional sample xs ∼ Gs(ys) and uniformly sampled source label ys. (b) We penalize a target classifier C A t t with unsupervised loss derived from SSL method by applying a pseudo sample xs←t while supervised training on target dataset Dt. xs←t is sampled from Gs conditioned by pseudo source label ys←t = C As s (xt).

Comparison of transfer learning settings

