0
파이썬에서 큰 XML 파일을 구문 분석하고 있습니다. 나는이 XML을 구문 분석 iterparse 모듈을 사용하고iterparse를 사용하는 중에 XMLSyntax 오류가 발생했습니다.
<?xml version="1.0" encoding="utf-8"?>
<posthistory>
<row Id="1332647" PostHistoryTypeId="5" PostId="723397" RevisionGUID="cd3aafe8-47ee-497d-a4e8-948c2a769d7e" CreationDate="2009-04-06T22:27:07.567" UserId="40414" Comment="Added examples of articles and filenames" Text="I have a large number of text files (1000+) containing articles from academic journals. Each article's file contains a "stub" from the end of the previous article (at the beginning) and from the beginning of the next article (at the end). I need to remove these stubs in preparation for running a frequency analysis on the articles because the stubs are duplicate data. 

I know I can use diff to output the differences between the files, ignore whitespace and treat the files as text, and then compare them manually, but this doesn't male much sense when dealing with this much data. Accuracy does not have to be 100%, so a script that compared each file to the next file and then removed 1 copy of the duplicate text would be perfect. This seems like it would be a pretty common issue when programming so I am surprised that I haven't been able to find anything that does this.

The file names sort in order, so a script that compares each file to the next sequentially should work. E.G.

<pre>bul_9_5_181.txt
bul_9_5_186.txt
</pre>

are two articles, one starting on page 181 and the other on page 186. 

Note: I am an academic doing content analysis of old journal articles for a project in the history of psychology. I am no programmer, but I do have 10+ years experience with linux and can usually figure things out as I go. 

Thanks for your help

<b>---example stub at beginning of file: everything before "AFFECTIVE PHENOMENA — EXPERIMENTAL" is duplicate from previous file----</b>

SYN&STHESIA

ISI

the majority of Portugese words signifying black objects or ideas relating to black. This association is, admittedly, no true synsesthesia, but the author believes that it is only a matter of degree between these logical and spontaneous associations and genuine cases of colored audition.
REFERENCES

DOWNEY, JUNE E. A Case of Colored Gustation. Amer. J. of Psycho!., 1911, 22, S28-539MEDEIROS-E-ALBUQUERQUE. Sur un phenomene de synopsie presente par des millions de sujets./. de psychol. norm, et path., 1911, 8, 147-151. MYERS, C. S. A Case of Synassthesia. Brit. J. of Psychol., 1911, 4, 228-238.

AFFECTIVE PHENOMENA — EXPERIMENTAL
BY PROFESSOR JOHN F. .SHEPARD
University of Michigan

Three articles have appeared from the Leipzig laboratory during the year. Drozynski (2) objects to the use of gustatory and olfactory stimuli in the study of organic reactions with feelings, because of the disturbance of breathing that may be involved. He uses rhythmical auditory stimuli, and finds that when given at different rates and in various groupings,

<b>---this is from the end of the same file, everything AFTER "1911. Pp.39" is duplicate from the next article---</b>

Pleasantness of Colors. Arner. J. of Psychol., 1911, 22, 578-579. 8. WASHBURN, M. F . and CRAWFORD, D . Fluctuations in the Affective Value of Colors During Fixation for One Minute. Amer. J. of Psychol., 1911, 22, 579-J82. 9. WELLS, F . L. and FORBES, A. On Certain Electrical Processes in the Human Body and their Relation to Emotional Reactions. (No. 16 of Archives of Psychology). New York: The Science Press, 1911. Pp. 39.

AFFECTIVE PHENOMENA — DESCRIPTIVE AND THEORETICAL
BY PROFESSOR H. N. GARDINER Smith College

Fundamental questions are discussed systematically by Rehmke (18) in a second edition of a well-digested treatise, a characteristic feature of which is its attempt to relate feeling, emotion and mood.
Feeling (Gefuhl) is defined as a Bestimmtheitsbesonderheit des zustdnd-

lichen Bewusstseins. Consciousness being conceived as the individual soul, its state is assumed to be at any given moment simple and unique; hence the momentary feeling is always one of pleasure or displeasure, never "mixed." It is determined, not by any one, but by the totality of the objective factors, those being massgebend which are in the focus of attention. A " feeling," in the ordinary sense, is a complex of the affective state and the "determining" and "accompanying" objective components, the "determining" objects of attention giving the kind of feeling, the "accompanying" organic sensations being mainly responsible for its obscure "coloring" and its degree. Mood (Stimmung) appears in a certain contrast to "feeling" in that in it organic sensation is the "determining" factor and no particular object occupies the focus of attention. Emotion {Affeki) is not contrasted with "feeling," but is "feeling" characterized by the intensity of the "accompanying" organic sensations, which are rightly included in the emotion; we must not, however, confuse, with James and Lange, the bodily changes which give rise

<b>---end example---</b>" />
</posthistory>
다음과 같이 큰 XML 파일의 관련 부분이다.
'''
Function provides fast iteration of XML files via iterparse
Source - Listing 5 at Source - http://www.ibm.com/developerworks/xml/library/x-hiperfparse/
'''
def fast_iter(context, func):
try:
for event, elem in context:
func(elem)
elem.clear()
while elem.getprevious() is not None:
del elem.getparent()[0]
del context
except etree.XMLSyntaxError, e:
print e
나는 다음과 같은 추적과
Input is not proper UTF-8, indicate encoding !
Bytes: 0x97 0x20 0x45 0x58, line 3, column 1694
몇 가지 중요한 포인트를 다음 XMLSyntaxError을받을 -
[1]은 XML이 큰 3기가바이트 (때문에 iterparse 모듈을 사용하는 것이 필수입니다 또는 그 이상).
[2] 구문 오류를 발생시키는 XML 파일의 일부만 제공했습니다.
[1] 내가이 문제를 자동으로 해결할 수
다음과 같이내 질문
은? 그렇다면 어떻게?[2]이 문제를 수동으로 해결할 수 있습니까? 그렇다면 어떻게?
[3]이 문제를 무시하고 대용량 XML 파일을 구문 분석 할 수 있습니까? 그렇다면 어떻게? 나는 다음과 같은 자료를 본
:
How should I deal with an XMLSyntaxError in Python's lxml while parsing a large XML file?
Ignore encoding errors in Python (iterparse)?
Is there a way to recover iterparse on invalid Char values?
그러나 아무도 내 질문에 대답 수 없었다.
ISO-8859-1에서 '0x97' 바이트는 "EM DASH"가 아닌 "END OF GUARDED AREA"제어 문자를 나타냅니다. 그러나 CP1252에서는 '0x97'이 실제로 "EM DASH"입니다. – mzjn
당신 말이 맞아요. 사실 [이 페이지] (http://en.wikipedia.org/wiki/ISO/IEC_8859-1)에 따르면 ISO-8859-1은'0x7F-0x9F' 범위의 문자를 정의하지 않습니다. –