I have +20 scrapy crawlers that I want to deploy manually from a browser webpage.
In order to achieve this, I have created a simple twisted server that executes in a shell process the following commands:
scrapyd-deploy default -p $project
curl http://127.0.0.1:6800/schedule.json -d project=$project -d spider=$spider
These commands are executed in twisted using utils.getProcessOutput(scriptname). The two previous commands are inside the script given as a parameter.
When trying to execute the twisted server with twistd -y <server.py>, it gives the following error: [Failure instance: Traceback (failure with no frames): : got stderr: 'Packing version 1399464111\n'
].
Here is the twisted server's code:
#/usr/bin/python
from twisted.internet import utils, reactor
from twisted.web import server, resource
from twisted.application import internet, service
class CrawlerResource(resource.Resource):
isLeaf = True
script = "./script2.sh"
def render_GET(self, request):
request.write("<pre>\n")
deferred = utils.getProcessOutput(self.script)
deferred.addCallback(lambda s: (request.write(s+"success!\n"), request.finish()))
deferred.addErrback(lambda s: (request.write(str(s)), request.finish()))
return server.NOT_DONE_YET
# factory: create a protocol for each connection
resource = CrawlerResource()
factory = server.Site(resource)
# application & service: run server in the background as a service
application = service.Application("Crawlers deployer")
service = internet.TCPServer(8000, factory)
service.setServiceParent(application)
What's causing this error (it isnt very verbose)?