SHONGLAP — A Large Bengali Open-Domain Dialogue Corpus

Abstract

We introduce SHONGLAP, a large annotated open-domain dialogue corpus in Bengali language. Due to unavailability of high-quality dialogue datasets for low-resource languages like Bengali, existing neural open-domain dialogue systems suffer from data scarcity. We propose a framework to prepare large-scale open-domain dialogue datasets from publicly available multi-party discussion podcasts, talk-shows and label them based on weak-supervision techniques which is particularly suitable for low-resource settings. Using this framework, we prepared our corpus, the first reported Bengali open-domain dialogue corpus (7.7k+ fully annotated dialogues in total) which can serve as a strong baseline for future works. Experimental results show that our corpus improves performance of large language models (BanglaBERT) in case of downstream classification tasks during fine-tuning.

Highlights

A pipeline that turns raw multi-party discussion podcasts and TV debates into fully annotated dialogue data — denoising, speaker diarization, speech-to-text, punctuation restoration, then weak-supervision (via Snorkel) to label speaker roles without manual annotation.
The first reported large-scale, fully-labeled open-domain Bengali dialogue corpus: 7,703 dialogues, 66,413 sentences, 138,445 unique tokens, averaging 2.68 speakers and 7.6 turns per dialogue.
Fine-tuning BanglaBERT on a downstream speaker-bias classification task using this corpus reaches 73.5% accuracy, and performance improves consistently as more of the corpus is used for training — direct evidence the corpus is useful for real downstream tasks, not just descriptive statistics.

Method

High-quality transcribed, annotated dialogue data is scarce for Bengali, while manual transcription and labeling at scale is expensive. SHONGLAP’s pipeline instead builds the corpus from Bengali political talk-show and debate audio that is already public, using weak supervision in place of manual labeling.

Pipeline: raw dialogue audio is cleaned, diarized, and transcribed via Speech2Text, then multiple label functions vote on speaker roles, a generative model learns each label function's accuracy, and a noise-aware discriminative model produces the final annotated dialogues

Fig. 1 — Corpus preparation workflow. Raw audio is cleaned, diarized, and transcribed; several weak label functions then vote on speaker roles, a generative model learns each function’s reliability on the fly, and a noise-aware discriminative model produces the final probabilistically-labeled dialogues.

From Raw Audio to Transcribed Dialogue

Podcast and TV debate audio is converted to 16kHz mono and denoised with a 20-layer Deep Complex U-Net. Speaker diarization uses ECAPA-TDNN (with spectral clustering) to determine who spoke when, and audio is segmented by speaker turn and further split on silence. Transcription uses wav2vec 2.0 (XLSR-53, pretrained across 53 languages and fine-tuned on 400 hours of Bengali audio) with a KenLM language model, followed by an XLM-RoBERTa-based punctuation restoration model fine-tuned to 97.2% accuracy on Bengali.

Weak Supervision for Speaker-Role Labeling

Rather than manually label which speaker is the “host” versus a guest in each dialogue, several heuristic label functions vote based on signals like number of questions asked, utterance sentiment, and reply descriptiveness (hosts tend to ask more questions and stay more neutral; guests give longer, descriptive answers). A generative model (via Snorkel) learns each label function’s accuracy from their agreement patterns and combines them into probabilistic labels, which train a final noise-aware discriminative model — avoiding the cost of manually labeling speaker roles across thousands of dialogues.

A Sample Annotated Dialogue

Each dialogue is stored as parallel lists of utterances and speaker labels:

{
  "utts": [
    "নির্বাস্তু হচ্ছে",
    "সেটা আমি ... হচ্ছে ভোটের পাশে",
    "ভোটের পাশে যে মানুষের নির্বাচন ... করে না",
    "২০ তারিখের নিউইয়র্কের উপনির্বাচনের ... পেছনে কত টাকা",
    "এখনো ... বড় টাকা",
    "এখন এখন এই যে ... এটা কথা আমি প্রশ্ন করি",
    "এই জায়গাটা",
    "এই দেশের ট্যাক্স ... আমার স্ত্রী বা আমার",
    "ইনভেস্ট করেন আবার ইনভেস্ট ... ইনভেস্টমেন্ট হবে"
  ],
  "speakers": [
    "HOST", "SPEAKER_1", "HOST", "SPEAKER_1", "SPEAKER_0",
    "SPEAKER_1", "SPEAKER_0", "SPEAKER_1", "SPEAKER_0"
  ]
}

Fig. 3 — A sample annotated dialogue. The utts and speakers lists have equal length, pairing each utterance with its speaker role.

Corpus Description

Statistic	Value
Total dialogues	7,703
Total sentences	66,413
Total unique tokens	138,445
Avg. duration (minutes)	5.62
Avg. speakers per dialogue	2.68
Avg. turns per dialogue	7.6
Avg. questions per dialogue	2.14

Sentiment analysis (via a pretrained Bengali BERT sentiment model, 93.2% accuracy) on the 66,413 sentences finds a fairly balanced emotional tone: 39.44% neutral, 36.20% positive, 24.36% negative — consistent with debate-style political discussion rather than uniformly contentious argument.

Topic modeling with Latent Dirichlet Allocation over the corpus surfaces five dominant themes — Election, International Affairs, Bangladesh, Economics, and a miscellaneous/functional-word cluster — matching the corpus’s political-discussion origins. A GloVe-embedding analysis of the most frequent political terms shows a long-tailed frequency distribution dominated by words directly related to “politics” itself:

Bar chart of top political keywords in Bengali script and their frequencies, dominated by variations of the word for politics, with a long tail of related terms

Fig. 4 — Top political keywords by frequency (English, left to right): politics, in politics, for liberation, for politics, social, economics, society, culture, practice, rules, leftist, then, haven’t (done) yet.

Evaluation: Speaker-Bias Classification

To test whether the corpus is actually useful downstream — not just descriptively rich — the authors fine-tune BanglaBERT (an ELECTRA-based Bengali language model) on a biased-speaker classification task built from the corpus: hosts are assumed to stay neutral (negative/unbiased label) while guest speakers present a particular view (positive/biased label), using 10,000 training, 500 validation, and 500 test utterances.

Metric	Score
Accuracy	0.735
Precision	0.7328
Recall	0.735
F1 Score	0.733

More tellingly, F1 score on this task rises steadily as more of the corpus is used for fine-tuning — from 0.45 at 1,000 training samples to over 0.75 at 6,000 — direct evidence that the corpus itself, not just the base language model, is driving the improvement.

Line chart showing F1 score rising steadily from about 0.45 at 1000 training samples to over 0.75 at 6000 samples

Fig. 5 — Test F1 score on the speaker-bias task rises steadily as more SHONGLAP data is used for fine-tuning BanglaBERT.

SHONGLAP: A Large Bengali Open-Domain Dialogue Corpus