0
업데이트 됨. 나는 Ubuntu 14.04를 실행 중이었고 다음 스크립트를 문제없이 실행했습니다. 여기서 EC2 인스턴스에있는 파일은 put입니다. 참고 : 나는 동일한 IP를 사용하여 성공과 실패를 모두 보여 주었지만 처음부터이 스크립트를 두 번 실행하여 항상 새 IP를 생성 한 것처럼 가장했습니다. 이제EC2에서 Python 패브릭 병렬 실행 실패 :
In [25]: execute(upload_scripts_data, rps_file) # send review_page_scraper helpers
[[email protected]] Executing task 'upload_scripts_data'
[[email protected]] Executing task 'upload_scripts_data'
[[email protected]] Executing task 'upload_scripts_data'
[[email protected]] put: /home/rerwin21/amazon_proj/amazon/review_page_scraper.py -> /home/ubuntu/review_page_scraper.py
[[email protected]] put: /home/rerwin21/amazon_proj/amazon/review_page_scraper.py -> /home/ubuntu/review_page_scraper.py
[[email protected]] put: /home/rerwin21/amazon_proj/amazon/review_page_scraper.py -> /home/ubuntu/review_page_scraper.py
Out[25]:
{u'[email protected]': None,
u'[email protected]': None,
u'[email protected]': None}
, 문제 : 다음
import boto.ec2 import os from fabric.api import run, parallel, env, sudo from fabric.tasks import execute from fabric.operations import put # file path python scripts and data rps_file = "review_page_scraper.py" # make sure hosts are clear before we add to them env.hosts = [] # how many instances to start and how to split up the data frame num_instances = 3 # EC2 access keys access_key = 'my_access_key' secret_key = 'my_secret_key' # get a connection to the east region conn = boto.ec2.connect_to_region("us-east-1", aws_access_key_id=access_key, aws_secret_access_key=secret_key) # create the reservation of instances reservation = conn.run_instances('my_ami_id', key_name='original_key', # my original key security_groups=['my_sec_grp'], instance_type='t2.micro', min_count=num_instances, max_count=num_instances) # get list of instances instance_lst = reservation.instances # get a status update and wait if the instance isn't up and running yet for instance in instance_lst: while instance.state != "running": sleep(5) instance.update() print "%s is running" % instance.ip_address # get username and host, add 'ubuntu' as username hosts = ["[email protected]" + ip.ip_address for ip in instance_lst] env.hosts = hosts # set environment variable @parallel def upload_scripts_data(file_name): path = "~/amazon_proj/amazon/" put(path + file_name, "~") # put it in the home dir of EC2 instance # execute functions w/ rps_file execute(upload_scripts_data, rps_file) # send review_page_scraper helpers ssh-keygen -t rsa을 사용하여 생성 된 내 키 쌍을 분실했을 때 공개 키를 AWS로 가져 왔습니다. 그래서 우분투를 다시 설치해야하고 16.04를 선택했습니다.
ssh-keygen -t rsa을 사용하여 새 키를 생성하고 개인 키와 공개 키 각각에 대해
~/.ssh/id_rsa 및 ~/.ssh/id_rsa.pub에 저장했습니다.
그런 다음 공용 키를 가져 와서 "id_rsa_pub"이름으로 저장했습니다. 이제 위의 스크립트를 실행하여 key_name 인수를 "id_rsa_pub"로 변경합니다. 또한 AWS의 지침에 따라 chmod 0400 id_rsa을 실행했습니다. 출력은 다음과 같습니다.
In [22]: execute(upload_scripts_data, rps_file)
[[email protected]] Executing task 'upload_scripts_data'
[[email protected]] Executing task 'upload_scripts_data'
[[email protected]] Executing task 'upload_scripts_data'
!!! Parallel execution exception under host u'[email protected]':
Process [email protected]:
Traceback (most recent call last):
File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/tasks.py", line 242, in inner
submit(task.run(*args, **kwargs))
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/tasks.py", line 174, in run
return self.wrapped(*args, **kwargs)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/decorators.py", line 181, in inner
Traceback (most recent call last):
File "<ipython-input-22-ed18eb00cc62>", line 1, in <module>
execute(upload_scripts_data, rps_file)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/tasks.py", line 412, in execute
ran_jobs = jobs.run()
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/job_queue.py", line 168, in run
self._fill_results(results)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/job_queue.py", line 191, in _fill_results
datum = self._comms_queue.get_nowait()
File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/queues.py", line 152, in get_nowait
return self.get(False)
File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/queues.py", line 135, in get
res = self._recv()
TypeError: ('__init__() takes exactly 2 arguments (3 given)', <class 'paramiko.ssh_exception.NoValidConnectionsError'>, (None, 'Unable to connect to port 22 on or 54.165.186.168'))
return func(*args, **kwargs)
File "<ipython-input-19-ed4344124d24>", line 4, in upload_scripts_data
put(path + file_name, "~")
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 677, in host_prompting_wrapper
return func(*args, **kwargs)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/operations.py", line 345, in put
ftp = SFTP(env.host_string)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/sftp.py", line 33, in __init__
self.ftp = connections[host_string].open_sftp()
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 159, in __getitem__
self.connect(key)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 151, in connect
user, host, port, cache=self, seek_gateway=seek_gateway)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 603, in connect
raise NetworkError(msg, e)
NetworkError: Low level socket error connecting to host 54.165.186.168 on port 22: Unable to connect to port 22 on or 54.165.186.168 (tried 1 time)
[[email protected]] put: /home/rerwin21/amazon_proj/amazon/review_page_scraper.py -> /home/ubuntu/review_page_scraper.py
!!! Parallel execution exception under host u'[email protected]':
Process [email protected]:
Traceback (most recent call last):
File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/rerwin21/anaconda2/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/tasks.py", line 242, in inner
submit(task.run(*args, **kwargs))
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/tasks.py", line 174, in run
return self.wrapped(*args, **kwargs)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/decorators.py", line 181, in inner
return func(*args, **kwargs)
File "<ipython-input-19-ed4344124d24>", line 4, in upload_scripts_data
put(path + file_name, "~")
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 677, in host_prompting_wrapper
return func(*args, **kwargs)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/operations.py", line 345, in put
ftp = SFTP(env.host_string)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/sftp.py", line 33, in __init__
self.ftp = connections[host_string].open_sftp()
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 159, in __getitem__
self.connect(key)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 151, in connect
user, host, port, cache=self, seek_gateway=seek_gateway)
File "/home/rerwin21/anaconda2/lib/python2.7/site-packages/fabric/network.py", line 603, in connect
raise NetworkError(msg, e)
NetworkError: Timed out trying to connect to 54.173.57.59 (tried 1 time)
그런 자세한 질문과 결과는 사과드립니다. 나는 내 지식을 다 써 버렸고 온라인에서 내 문제와 완전히 똑같은 것을 보지 못했습니다.
업데이트 :
몇 가지 유의 사항. 이전 보안 키 그룹과 함께 사용한 보안 그룹과 AMI를 사용하고 있습니다. 다음으로, 훨씬 더 혼란 스럽습니다. execute(upload_scripts_data, rps_file) 명령을 다시 실행하면 오류없이 실행됩니다.
예를 들어 IP 주소가 '54.173.57.59' 인 보안 그룹을 확인하십시오. 포트 22에서 들어오는 tcp 연결을 허용합니까? 그것은 ssh 데몬을 실행 중입니까? –
Thanks @NehalJWani, 위의 업데이트로 응답 해 드리겠습니다.보세요. –