Dialogue Summarization

Goal: Fine-tune summarization model for Dialogue dataset.

Model (base) : PEGASUS 

Dataset: SAMSum people dialogues with summaries

Steps:

Joe : I baked cookies. Do you want some?
Bob: Sure! Let's have a breakfast together
Joe: OK, I'll bring you cookies tomorrow :)
Bob: Perfect! I'll prepare the rest
Joe: Sounds good! See you then!

Summary: Joe will bring baked cookies for breakfast with Bob

DATASET 1 : CNN / DailyMail dataset

Train examples: 287,133

Validation examples: 13,368

Test examples: 11,490

Example: 

Harry Potter star Daniel Radcliffe gains access to a reported £20 million ($41.1 million) fortune as he turns 18 on Monday, but he insists the money won't cast a spell on him. Daniel Radcliffe as Harry Potter in "Harry Potter and the Order of the Phoenix" To the disappointment of gossip columnists around the world, the young actor says he has no plans to fritter his cash away on fast cars, drink and celebrity parties. "I don\'t plan to be one of those people who, as soon as they turn 18, suddenly buy themselves a massive sports car collection or something similar. He told an Australian interviewer earlier this month. "I don\'t think I\'ll be particularly extravagant. "The things I like buying are things that cost about 10 pounds -- books and CDs and DVDs." At 18, Radcliffe will be able to gamble in a casino, buy a drink in a pub or see the horror film "Hostel: Part II," currently six places below his number one movie on the UK box office chart. Details of how he'll mark his landmark birthday are under wraps. His agent and publicist had no comment on his plans. "I\'ll definitely have some sort of party," he said in an interview. "Hopefully none of you will be reading about it." Radcliffe\'s earnings from the first five Potter films have been held in a trust fund which he has not been able to touch. Despite his growing fame and riches, the actor says he is keeping his feet firmly on the ground. "People are always looking to say \'kid star goes off the rails,\'" he told reporters last month. "But I try very hard not to go that way because it would be too easy for them." His latest outing as the boy wizard in "Harry Potter and the Order of the Phoenix" is breaking records on both sides of the Atlantic and he will reprise the role in the last two films.  Watch I-Reporter give her review of Potter\'s latest » . There is life beyond Potter, however. The Londoner has filmed a TV movie called "My Boy Jack," about author Rudyard Kipling and his son, due for release later this year. He will also appear in "December Boys," an Australian film about four boys who escape an orphanage. Earlier this year, he made his stage debut playing a tortured teenager in Peter Shaffer\'s "Equus." Meanwhile, he is braced for even closer media scrutiny now that he\'s legally an adult: "I just think I\'m going to be more sort of fair game," he told Reuters. E-mail to a friend . Copyright 2007 Reuters. All rights reserved.This material may not be published, broadcast, rewritten, or redistributed.

Summary:


Rouge scores for summarization

Ground Truth:

Baseline (first 3 sentences)

Harry Potter star Daniel Radcliffe gains access to a reported £20 million ($41.1 million) fortune as he turns 18 on Monday, but he insists the money won't cast a spell on him. Daniel Radcliffe as Harry Potter in "Harry Potter and the Order of the Phoenix" To the disappointment of gossip columnists around the world, the young actor says he has no plans to fritter his cash away on fast cars, drink and celebrity parties

GPT-2 : <Article> + "TL;DR"

Rudyard Kipling's youngest son is coming of age in \"Harry Potter and the Order of the Phoenix.\"Expect Radcliffe's first Potter film to break box office records in the UK, in spite of poor box office and ratings.If Radcliffe can't hit box office records, can the franchise live on in a different form?Thanks

T5 (CNN included)

the young actor says he has no plans to fritter his cash away on fast cars, drink and celebrity parties .he will be able to gamble in a casino, buy a drink or see the horror film \"Hostel: Part II\" his agent and publicist had no comment on his plans .his latest outing as the boy wizard is breaking records on both sides of the Atlantic

BART (CNN only)

Harry Potter star Daniel Radcliffe turns 18 on Monday.He gains access to a reported £20 million ($41.1 million) fortune.Radcliffe's earnings from the first five Potter films have been held in a trust fund.Details of how he'll mark his landmark birthday are under wraps

Pegasus (CNN only)

Rouge for Single "Harry Potter" example

Rouge on entire CNN dataset

DATASET 2: SAMSum

Example: 

Hannah: Hey, do you have Betty's number?
Amanda: Lemme check
Hannah: <file_gif>
Amanda: Sorry, can't find it.
Amanda: Ask Larry
Amanda: He called her last time we were at the park together
Hannah: I don't know him well
Hannah: <file_gif>
Amanda: Don't be shy, he's very nice
Hannah: If you say so..
Hannah: I'd rather you texted him
Amanda: Just text him 🙂
Hannah: Urgh.. Alright
Hannah: Bye
Amanda: Bye bye

Rouge Pegasus(FT:CNN) on SAMSum['test']

Ground Truth summary:

Hannah needs Betty's number but Amanda doesn't have it. She needs to contact Larry.

Pegasus (FT:CNN) zero-shot summary:

Amanda: Ask Larry
Amanda: He called her last time we were at the park together.
Hannah: I'd rather you texted him.
Amanda: Just text him .

Remarks:

Pegasus (FT:SAMSum) zero-shot summary:

Amanda can't find Betty's number. Larry called Betty last time they were at the park together. Hannah wants Amanda to text Larry. Amanda will text Larry.

Fine-Tune Pegasus on SAMSum

Rouge Pegasus(FT:SAMSum) on SAMSum['test']

Conclusion