스페인어 단어 토큰 화자

Question

스페인어 문장을 단어로 토큰 화하고 싶습니다. 다음과 같은 올바른 접근 방식인가요? 아니면이를 수행하는 더 좋은 방법이 있습니까?

import nltk 
from nltk.tokenize import word_tokenize 

def spanish_word_tokenize(s): 
    for w in word_tokenize(s):  
     if w[0] in ("¿","¡"): 
      yield w[0] 
      yield w[1:] 
     else: 
      yield w   

sentences = "¿Quién eres tú? ¡Hola! ¿Dónde estoy?" 

spanish_sentence_tokenizer = nltk.data.load('tokenizers/punkt/spanish.pickle') 

sentences = spanish_sentence_tokenizer.tokenize(sentences) 
for s in sentences: 
    print([s for s in spanish_word_tokenize(s)]) 

Answer 1

f. NLTK github 발행 #1214, NLTK에는 꽤 많은 대체 토큰 화가가 있습니다.)

예. @jonsafari toktok tokenizer의 사용 NLTK 포트 : 당신이 코드를 조금 해킹 및 https://github.com/nltk/nltk/blob/develop/nltk/tokenize/toktok.py#L51에 u'\xa1'를 추가하는 경우

>>> import nltk 
>>> nltk.download('perluniprops') 
[nltk_data] Downloading package perluniprops to 
[nltk_data]  /Users/liling.tan/nltk_data... 
[nltk_data] Package perluniprops is already up-to-date! 
True 
>>> nltk.download('nonbreaking_prefixes') 
[nltk_data] Downloading package nonbreaking_prefixes to 
[nltk_data]  /Users/liling.tan/nltk_data... 
[nltk_data] Package nonbreaking_prefixes is already up-to-date! 
True 
>>> from nltk.tokenize.toktok import ToktokTokenizer 
>>> toktok = ToktokTokenizer() 
>>> sent = u"¿Quién eres tú? ¡Hola! ¿Dónde estoy?" 
>>> toktok.tokenize(sent) 
[u'\xbf', u'Qui\xe9n', u'eres', u't\xfa', u'?', u'\xa1Hola', u'!', u'\xbf', u'D\xf3nde', u'estoy', u'?'] 
>>> print " ".join(toktok.tokenize(sent)) 
¿ Quién eres tú ? ¡Hola ! ¿ Dónde estoy ? 

>>> from nltk import sent_tokenize 
>>> sentences = u"¿Quién eres tú? ¡Hola! ¿Dónde estoy?" 
>>> [toktok.tokenize(sent) for sent in sent_tokenize(sentences, language='spanish')] 
[[u'\xbf', u'Qui\xe9n', u'eres', u't\xfa', u'?'], [u'\xa1Hola', u'!'], [u'\xbf', u'D\xf3nde', u'estoy', u'?']] 

>>> print '\n'.join([' '.join(toktok.tokenize(sent)) for sent in sent_tokenize(sentences, language='spanish')]) 
¿ Quién eres tú ? 
¡Hola ! 
¿ Dónde estoy ? 

, 당신은 얻을 수 있어야합니다 :

[[u'\xbf', u'Qui\xe9n', u'eres', u't\xfa', u'?'], [u'\xa1', u'Hola', u'!'], [u'\xbf', u'D\xf3nde', u'estoy', u'?']]