0:42 · 1.3 MB · MP4 Download the video
Walkthrough 1
Segment a hashtag
Type a hashtag, step through forward and reverse MaxMatch one match at a time, and see which split the Brown unigram language model prefers.
- 1Open the Hashtag Segmenter. Both MaxMatch directions run in the browser.
- 2Type a hashtag: #1stsundayofoctober
- 3Step through forward and reverse MaxMatch, one dictionary match at a time
- 4Play to the end. Forward MaxMatch ends with ofo, c, tobe and r
- 5The unigram LM scores both splits. The higher log-probability wins
- 6Per-token Brown counts and log-probabilities explain the verdict
- 7Across the notebook's hashtags, win counts come with Wilson 95% intervals
Transcript
Silent screen recording of segment a hashtag on the NLP Playground, captioned step by step. A highlighted circle shows the pointer.
- 0:00Open the Hashtag Segmenter. Both MaxMatch directions run in the browser.
- 0:02Type a hashtag: #1stsundayofoctober
- 0:07Step through forward and reverse MaxMatch, one dictionary match at a time
- 0:17Play to the end. Forward MaxMatch ends with ofo, c, tobe and r
- 0:21The unigram LM scores both splits. The higher log-probability wins
- 0:27Per-token Brown counts and log-probabilities explain the verdict
- 0:34Across the notebook's hashtags, win counts come with Wilson 95% intervals