Home:ALL Converter>Writing a web crawler using python twisted

Writing a web crawler using python twisted

Ask Time:2015-04-14T03:36:30         Author:puruki123

Json Formatter

I'm using Twisted to write a web crawler driven with Selenium. The idea is that I spawn twisted threads for a twisted client and a twisted server that will proxy HTTP requests to the server. Something that looks like this:

    +--------+       +--------+
    |        +------>+        |
 -->| Client |       | Server |---> WWW
    |        +<------+        |
    +--------+       +--------+

All this is running in the same process, though. The question is whether twisted allows this kind of applications or is only thought to run Client and Server as different processes (as this is the typical case I've seen everywhere).

Author:puruki123,eproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article:https://stackoverflow.com/questions/29613616/writing-a-web-crawler-using-python-twisted
yy