Python subprocess.popen 및 rdiff-backup

Question

Windows 컴퓨터를 Linux 서버에 백업하기 위해 python으로 rdiff-backup 래퍼 프로그램을 만들고 싶습니다.

래퍼 프로그램에서 rdiff-backup의 출력을 처리하려고합니다. 그러나 서브 프로세스 모듈로 rdiff-backup을 실행하고 stdout과 stderr을 래퍼로 파이핑하면 stdout은 항상 파이프의 끝에서 끝납니다.

파이프없이 서브 프로세스를 사용할 때 stdout과 stderr는 콘솔에서 올바른 순서로 인쇄됩니다.

로컬에서 rdiff-backup을 사용하고 ssh 파이프를 사용하지 않을 때 stdout 및 stderr이 올바른 순서로 나타났습니다. Rdiff-backup은 또한 subprocess.popen을 사용하여 ssh 세션을 열고 데이터를 서버로 파이핑합니다. 나는 어떤 이유로 stdout이 ssh 세션이 닫힐 때까지 차단 될 것이라고 생각한다.

import sys 
import subprocess 

class Rdiffbackup(object): 
    def __init__(self): 
     #self.io_q = Queue() 
     self.exe = 'F:\\workspace\\pysubprocess\\bin\\rdiff-backup\\rdiff-backup.exe' 

     self.verbosity = '-v5' 

     self.ssh_exe = './bin/openssh/bin/ssh' 
     self.quiet = '-q' 
     self.compression = '-C' 
     self.port = '-p 5555' 
     self.key = '-i ./keys/rdiffbackup' 
     self.options = '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' 
     self.remote_schema = self.ssh_exe + ' ' + self.quiet + ' ' + self.compression + ' ' + self.port + ' ' + self.key + ' ' + self.options + ' %s rdiff-backup --server' 

    def start(self,source,dest): 
     with subprocess.Popen([self.exe,self.verbosity,'--remote-schema',self.remote_schema,source,dest],stdout=subprocess.PIPE,stderr=subprocess.STDOUT) as self.proc:  
      for line in self.proc.stdout: 
       sys.stdout.write(line.decode("utf-8"))  

if __name__ == '__main__': 
    rdb = Rdiffbackup() 
    source = "C:/Users/vdrmrt/Desktop/data"  
    dest = "vdrmrt@hostname::backup" 
    rdb.start(source,dest) 

출력 :

----------------------------------------------------------------- 
Detected abilities for source (read only) file system: 
    Access control lists       Off 
    Extended attributes       Off 
    Windows access control lists     On 
    Case sensitivity        Off 
    Escape DOS devices       Off 
    Escape trailing spaces      Off 
    Mac OS X style resource forks    Off 
    Mac OS X Finder information     Off 
----------------------------------------------------------------- 
Unable to import win32security module. Windows ACLs 
not supported by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0 
escape_dos_devices not required by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0 
----------------------------------------------------------------- 
Detected abilities for destination (read/write) file system: 
    Ownership changing       Off 
    Hard linking         On 
    fsync() directories       On 
    Directory inc permissions     On 
    High-bit permissions       On 
    Symlink permissions       Off 
    Extended filenames       On 
    Windows reserved filenames     Off 
    Access control lists       On 
    Extended attributes       On 
    Windows access control lists     Off 
    Case sensitivity        On 
    Escape DOS devices       Off 
    Escape trailing spaces      Off 
    Mac OS X style resource forks    Off 
    Mac OS X Finder information     Off 
----------------------------------------------------------------- 
Backup: must_escape_dos_devices = 0 
Using rdiff-backup version 1.2.8 
Executing ./bin/openssh/bin/ssh -q -C -p 5555 -i ./keys/rdiffbackup -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null vdrmrt@hostname rdiff-backup --server 
Hardlinks disabled by default on Windows 
Unable to import module xattr. 
Extended attributes not supported on filesystem at C:/Users/vdrmrt/Desktop/data 
Unable to import module posix1e from pylibacl package. 
POSIX ACLs not supported on filesystem at C:/Users/vdrmrt/Desktop/data 
escape_dos_devices not required by filesystem at C:/Users/vdrmrt/Desktop/data 
Symbolic links excluded by default on Windows 
Starting increment operation C:/Users/vdrmrt/Desktop/data to backup 

올바른 출력 :

Using rdiff-backup version 1.2.8 
Executing ./bin/openssh/bin/ssh -q -C -p 5555 -i ./keys/rdiffbackup -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null vdrmrt@hostname rdiff-backup --server 
Hardlinks disabled by default on Windows 
Unable to import module xattr. 
Extended attributes not supported on filesystem at C:/Users/vdrmrt/Desktop/data 
Unable to import module posix1e from pylibacl package. 
POSIX ACLs not supported on filesystem at C:/Users/vdrmrt/Desktop/data 
escape_dos_devices not required by filesystem at C:/Users/vdrmrt/Desktop/data 
----------------------------------------------------------------- 
Detected abilities for source (read only) file system: 
    Access control lists       Off 
    Extended attributes       Off 
    Windows access control lists     On 
    Case sensitivity        Off 
    Escape DOS devices       Off 
    Escape trailing spaces      Off 
    Mac OS X style resource forks    Off 
    Mac OS X Finder information     Off 
----------------------------------------------------------------- 
Unable to import win32security module. Windows ACLs 
not supported by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0 
escape_dos_devices not required by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0 
----------------------------------------------------------------- 
Detected abilities for destination (read/write) file system: 
    Ownership changing       Off 
    Hard linking         On 
    fsync() directories       On 
    Directory inc permissions     On 
    High-bit permissions       On 
    Symlink permissions       Off 
    Extended filenames       On 
    Windows reserved filenames     Off 
    Access control lists       On 
    Extended attributes       On 
    Windows access control lists     Off 
    Case sensitivity        On 
    Escape DOS devices       Off 
    Escape trailing spaces      Off 
    Mac OS X style resource forks    Off 
    Mac OS X Finder information     Off 
----------------------------------------------------------------- 
Backup: must_escape_dos_devices = 0 
Symbolic links excluded by default on Windows 
Starting increment operation C:/Users/vdrmrt/Desktop/data to backup 

Answer 1

마침내 문제를 해결할 수있었습니다.
Rdiff 백업은 stdout 및 stderr이 다른 프로그램으로 파이프되었을 때 버퍼링되었습니다. 나는 또한 7
Py2exe - win32api.pyc ImportError DLL load failed

가 여기 창문을 구축 할 때 작동하도록 추가 옵션을 추가했다
진정한 : '버퍼링' 이 솔루션은 별도의 py2exe에 옵션으로 rdiff는 백업을 다시했다 rdiff-backup 빌드의 마지막 setup.py에서 가져온 스 니펫.

if '--single-file' in sys.argv[1:]: 
      sys.argv.remove('--single-file') 
      extra_options.update({ 
       'options': {'py2exe': {'bundle_files': 1, 
             'unbuffered': True, 
             'dll_excludes': [ "mswsock.dll", "powrprof.dll" ]}}, 
       'zipfile': None 
      }) 

Answer 2

여기

이 실제 프로그램은 표준 출력을 듣고 스레드를 사용하는 단순화 된 버전을 내 코드입니다

stdout 및 stderr에 대해 별도의 파이프를 사용해보십시오. 먼저 stdout 줄을 읽은 다음 stderr 줄을 읽습니다.