In a context where products need to be continuously fed with effective marketing descriptions to fit into a good SEO strategy, Artificial Intelligence with Natural Language Generation (NLG) technology appears to be an ideal solution to automatically generate quality and quantity content. This rapidly growing technology has become common in many industries and can become a real ally for your data-driven content marketing strategy.

The code used in this article can be found here — How to fine-tune GPT-2. Also For testing GPT2-Home on a live Demo, you can visit the model page on Huggingface. Any question? Post a Github issue.

Why is product content important?

Product descriptions are essential for a successful online business because they help you become more visible online, share the information needed by consumers to make a buying decision, and therefore, have more sales.

Transformers

The paper Attention Is All You Need introduces a novel architecture called Transformer. As the title indicates, it uses the attention mechanism. The focus of the original research was on translation tasks. This was followed by the introduction of several influential models, including GPT, BERT, GPT-2, DistilBERT, BART, T5, and GPT-3. This list is far from comprehensive, and is just meant to highlight a few of the different kinds of Transformer models.

Like LSTM, Transformer is an architecture for transforming one sequence into another one with the help of two parts (Encoder and Decoder), but it differs from the previously existing sequence-to-sequence models because it does not imply any Recurrent Networks (GRU, LSTM, etc.). The Encoder takes the input sequence and maps it into a higher dimensional space (n-dimensional vector). That abstract vector is fed into the Decoder which turns it into an output sequence. The team presenting the paper proved that architecture with only attention mechanisms without any RNN (Recurrent Neural Networks) can improve the results in translation tasks and other tasks.

What is GPT-2

GPT-2 is a large transformer-based language model with 1.5 billion parameters, trained on a dataset of 8 million web pages. GPT-2 is trained with a simple objective: predict the next word, given all of the previous words within some text. The diversity of the dataset causes this simple goal to contain naturally occurring demonstrations of many tasks across diverse domains. GPT-2 is a direct scale-up of GPT, with more than 10X the parameters and trained on more than 10X the amount of data.

GPT-2 displays a broad set of capabilities, including the ability to generate conditional synthetic text samples of unprecedented quality, where we prime the model with input and have it generate a lengthy continuation. In addition, GPT-2 outperforms other language models trained on specific domains (like Wikipedia, news, or books) without needing to use these domain-specific training datasets. On language tasks like question answering, reading comprehension, summarization, and translation, GPT-2 begins to learn these tasks from the raw text, using no task-specific training data. While scores on these downstream tasks are far from state-of-the-art, they suggest that the tasks can benefit from unsupervised techniques, given sufficient (unlabeled) data and computing.

What is GPT2-Home?

GPT2-Home is a fine-tuned version of GPT-2 on Amazon home products metadata. It can generate descriptions for your home products by simply getting a text prompt. It has been fine-tuned on various types of home products to cover all types of product descriptions you need. It can generate descriptions from main home products like the furniture to home decorative products like Wall Decor, Candle, Posters and etc.

Fastai with 🤗Transformers

GPT2-Home is finetuned using fastai and transformers. fastai is a deep learning library that provides practitioners with high-level components that can quickly and easily provide state-of-the-art results in standard deep learning domains, and provides researchers with low-level components that can be mixed and matched to build new approaches.

Hugging Face, a company with open-source NLP technologies, provides thousands of pre-trained models to perform tasks on different modalities such as text, vision, and audio.

📑 Future Improvements

Adding 14 new categories of products besides the only home products.