Mode Collapse (Lack of Diversity) This is a common problem with GAN training. Mode collapse occurs when the model does not care about the input random noise, and it keeps generating the same sentence regardless of the input. In this sense, the model is now trying to fool the discriminator, and finding a single point is sufficient enough to do so.

模式崩潰(缺乏多樣性)這是GAN訓(xùn)練中的常見(jiàn)問(wèn)題。 當(dāng)模型不關(guān)心輸入的隨機(jī)噪聲時(shí)，就會(huì)發(fā)生模式崩潰，并且不管輸入如何，它都會(huì)不斷生成相同的句子。 從這個(gè)意義上講，模型現(xiàn)在正試圖欺騙鑒別器，找到一個(gè)單點(diǎn)足以做到這一點(diǎn)。

Unstable Training. The most important problem is to ensure that the generator and the discriminator are on par to each other. If either one outperforms each other, the whole training will become unstable, and no useful information will be learned. For example, when the generator’s loss is slowly reducing, that means the generator starts to find a way to fool the discriminator even though the generation is still immature. On the other hand, when discriminator is OP, there is no new information for the generator to learn. Every generation will be evaluated as fake; therefore, the generator will have to rely on changing the word randomly in searching for a sentence that may fool the D.

培訓(xùn)不穩(wěn)定。 最重要的問(wèn)題是確保生成器和鑒別器彼此相同。 如果任何一個(gè)人的表現(xiàn)都超過(guò)對(duì)方，則整個(gè)培訓(xùn)將變得不穩(wěn)定，并且將不會(huì)學(xué)習(xí)有用的信息。 例如，當(dāng)發(fā)電機(jī)的損耗逐漸降低時(shí)，這意味著即使發(fā)電機(jī)還不成熟，發(fā)電機(jī)也開始尋找一種欺騙鑒別器的方法。 另一方面，當(dāng)判別器為OP時(shí)，沒(méi)有新的信息可供生成器學(xué)習(xí)。 每一代都將被視為假貨； 因此，生成器將不得不依靠隨機(jī)更改單詞來(lái)搜索可能使D蒙蔽的句子。

Intuition is NOT Enough. Sometimes, your intended modeling is correct, but it may not work as you want it to be. It may require more than that to work. Frequently, you need to do hyperparameters tuning by tweaking learning rate, trying different loss functions, using batch norm, or trying different activation functions.

直覺(jué)還不夠 。 有時(shí)，您預(yù)期的建模是正確的，但可能無(wú)法按您希望的那樣工作。 可能需要更多的工作。 通常，您需要通過(guò)調(diào)整學(xué)習(xí)率，嘗試使用不同的損失函數(shù)，使用批處理規(guī)范或嘗試使用不同的激活函數(shù)來(lái)進(jìn)行超參數(shù)調(diào)整。

Lots of Training Time. Some work reported training up to 400 epochs. That is tremendous if we compare with Seq2Seq that might take only 50 epochs or so to get a well-structured generation. The reason that causes it to be slow is the exploration of the generation. G does not receive any explicit signal of which token is bad. Rather it receives for the whole generation. To able to produce a natural sentence, G needs to explore various combinations of words to reach there. How often do you think G can accidentally produce <eos> out of nowhere? If we use MLE, the signal is pretty clear that there should be <eos> and there are <pad> right after it.

大量的培訓(xùn)時(shí)間。 一些工作報(bào)告說(shuō)培訓(xùn)了多達(dá)400個(gè)紀(jì)元。 如果我們與Seq2Seq進(jìn)行比較，那可能只花費(fèi)50個(gè)紀(jì)元左右即可得到結(jié)構(gòu)良好的世代，這是巨大的。 導(dǎo)致它變慢的原因是一代人的探索。 G沒(méi)有收到任何明顯的信號(hào)，指出哪個(gè)令牌不好。 相反，它為整個(gè)世代所接受。 為了產(chǎn)生自然的句子，G需要探索各種單詞組合以到達(dá)那里。 您認(rèn)為G多久會(huì)偶然地偶然產(chǎn)生<eos>？ 如果我們使用MLE，則信號(hào)很清楚，應(yīng)該有<eos>，緊隨其后的是<pad>。

Use ADAM Optimizer. Some suggest using the ADAM for the generator and SGD for the discriminator. But most importantly, some paper starts to tweak the beta for the ADAM. betas=(0.5, 0.999)

使用ADAM優(yōu)化器 。 有些人建議使用ADAM作為生成器，使用SGD作為鑒別器。 但最重要的是，一些論文開始調(diào)整ADAM的beta版本。 beta =(0.5，0.999)

Wasserstein GAN. Some work reports using WGAN can stabilize the training greatly. From our experiments, however, WGAN can not even reach the quality of regular GAN. Perhaps we are missing something. (See? It’s quite difficult)

瓦瑟斯坦甘 。 使用WGAN的一些工作報(bào)告可以大大穩(wěn)定培訓(xùn)。 但是，根據(jù)我們的實(shí)驗(yàn)，WGAN甚至無(wú)法達(dá)到常規(guī)GAN的質(zhì)量。 也許我們?nèi)鄙倭艘恍〇|西。 (看？這很困難)

GAN Variation. Some suggest trying KL-GAN, or VAE-GAN. These can make the models easier to train.

GAN變化 。 有些人建議嘗試KL-GAN或VAE-GAN。 這些可以使模型更容易訓(xùn)練。

Input Noise to the Discriminator. To make the discriminator’s learning on par with the generator, which in general have a harder time than the D, we input some noise along with the input as well as using dropout to make things easier.

鑒別器的輸入噪聲 。 為了使鑒別器的學(xué)習(xí)與生成器(通常比生成器困難)相提并論，我們?cè)谳斎氲耐瑫r(shí)輸入一些噪聲，并使用壓差使事情變得更容易。

DCGAN (Deep Convolutional GAN). This is only for computer vision tasks. However, this model is known to avoid unstable training. The key in this model is to not use ReLU, use BatchNorm, and use Strided Convolution.

DCGAN(深度卷積GAN) 。 這僅用于計(jì)算機(jī)視覺(jué)任務(wù)。 但是，已知該模型可以避免不穩(wěn)定的訓(xùn)練。 該模型的關(guān)鍵是不使用ReLU，使用BatchNorm和使用Strided Convolution。

Ensemble of Discriminator. Instead of having a single discriminator, multiple discriminators are trained with different batch, to capture different aspects of respect. Thus, the generator can not just fool a single D, but to be more generalized so that it can fool all of them. This is also related to Dropout GAN (many D and dropout some during training).

鑒別器合奏 。 代替單個(gè)鑒別器，而是用不同的批次訓(xùn)練多個(gè)鑒別器，以捕獲尊重的不同方面。 因此，生成器不僅可以欺騙單個(gè)D，還可以對(duì)其進(jìn)行更廣泛的概括以使其欺騙所有D。 這也與輟學(xué)GAN(許多D，并且在培訓(xùn)期間輟學(xué))有關(guān)。

Parameter Tuning. With the learning rate, dropout ratio, batch size, and so on. It is difficult to determine how much a model is better than another. Therefore, some would test on multiple parameters and see whichever works best. One bottleneck is there is no evaluation metric for GAN, which results in a lot of manual checks to determine the quality.

參數(shù)調(diào)整 。 具有學(xué)習(xí)率，輟學(xué)率，批量大小等。 很難確定一個(gè)模型比另一個(gè)模型好多少。 因此，有些人會(huì)在多個(gè)參數(shù)上進(jìn)行測(cè)試，然后看哪個(gè)效果最好。 一個(gè)瓶頸是沒(méi)有針對(duì)GAN的評(píng)估指標(biāo)，這導(dǎo)致需要大量手動(dòng)檢查來(lái)確定質(zhì)量。

Scheduling G and D. Trying to learn G 5 times followed by D 1 times are reported to be useless in many work. If you want to try scheduling, do something more meaningful.

安排G和D。 據(jù)報(bào)告，嘗試學(xué)習(xí)G 5次然后學(xué)習(xí)D 1次在許多工作中是沒(méi)有用的。 如果您想嘗試安排時(shí)間，請(qǐng)做一些更有意義的事情。