ciseau:在Python中标记和清理字符串 源码

塞绍 Python中的单词和句子标记化。 用法 使用此程序包可根据句子和单词的边界拆分字符串。 例如,简单地将字符串分解为令牌: tokenize("Joey was a great sailor.") #=> ["Joey ", "was ", "a ", "great ", "sailor ", "."] 还可以检测句子边界: sent_tokenize("Cat sat mat. Cat's named Cool.", keep_whitespace=True) #=> [["Cat ", "sat ", "mat", ". "], ["Cat ", "'s ", "named ",