Sklearn CountVectorizer를 'word'및 'char'분석기와 함께 어떻게 사용합니까? - python

Question

Sklearn CountVectorizer를 'word'와 'char'분석기와 함께 어떻게 사용합니까? http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html

단어 나 문자로 텍스트 기능을 추출 할 수 있지만 어떻게 charword_vectorizer을 만들 수 있습니까? 벡터 라이 제이션을 결합하는 방법이 있습니까? 또는 하나 이상의 분석기를 사용합니까?

>>> from sklearn.feature_extraction.text import CountVectorizer 
>>> word_vectorizer = CountVectorizer(analyzer='word', ngram_range=(1, 2), min_df=1) 
>>> char_vectorizer = CountVectorizer(analyzer='char', ngram_range=(1, 2), min_df=1) 
>>> x = ['this is a foo bar', 'you are a foo bar black sheep'] 
>>> word_vectorizer.fit_transform(x) 
<2x15 sparse matrix of type '<type 'numpy.int64'>' 
    with 18 stored elements in Compressed Sparse Column format> 
>>> char_vectorizer.fit_transform(x) 
<2x47 sparse matrix of type '<type 'numpy.int64'>' 
    with 64 stored elements in Compressed Sparse Column format> 
>>> char_vectorizer.get_feature_names() 
[u' ', u' a', u' b', u' f', u' i', u' s', u'a', u'a ', u'ac', u'ar', u'b', u'ba', u'bl', u'c', u'ck', u'e', u'e ', u'ee', u'ep', u'f', u'fo', u'h', u'he', u'hi', u'i', u'is', u'k', u'k ', u'l', u'la', u'o', u'o ', u'oo', u'ou', u'p', u'r', u'r ', u're', u's', u's ', u'sh', u't', u'th', u'u', u'u ', u'y', u'yo'] 
>>> word_vectorizer.get_feature_names() 
[u'are', u'are foo', u'bar', u'bar black', u'black', u'black sheep', u'foo', u'foo bar', u'is', u'is foo', u'sheep', u'this', u'this is', u'you', u'you are'] 

Answer 1

analyzer 인수가 모든 권한을 얻을로 당신은 호출을 전달할 수 토큰 화를 통해, 예를 들어,

>>> from pprint import pprint 
>>> import re 
>>> x = ['this is a foo bar', 'you are a foo bar black sheep'] 
>>> def words_and_char_bigrams(text): 
...  words = re.findall(r'\w{3,}', text) 
...  for w in words: 
...   yield w 
...   for i in range(len(w) - 2): 
...    yield w[i:i+2] 
...    
>>> v = CountVectorizer(analyzer=words_and_char_bigrams) 
>>> pprint(v.fit(x).vocabulary_) 
{'ac': 0, 
'ar': 1, 
'are': 2, 
'ba': 3, 
'bar': 4, 
'bl': 5, 
'black': 6, 
'ee': 7, 
'fo': 8, 
'foo': 9, 
'he': 10, 
'hi': 11, 
'la': 12, 
'sh': 13, 
'sheep': 14, 
'th': 15, 
'this': 16, 
'yo': 17, 
'you': 18} 

Answer 2

당신은 FeatureUnion 추정으로 임의의 특징 추출 단계를 결합 할 수 있습니다 :이 경우 http://scikit-learn.org/dev/modules/pipeline.html#featureunion-combining-feature-extractors

을이 아마 larsmans 솔루션에 비해 덜 효율적이지만, 쉽게 사용할 수 있습니다.