python multiprocessing scheduling task -


i have 8 cpu core , 200 tasks done. each tasks isolate. there no need wait or share result. i'm looking way run 8 tasks/processes @ time (maximum) , when 1 of them finished. remaining task automatic start process.

how know when child process done , start new child process. first i'm trying use process(multiprocessing) , it's hard figure out. try use pool , face pickle problem cause need use dynamic instantiate.

edited : adding code of pool

class collectorparallel():  def fire(self,obj):     collectorcontroller = collectorcontroller()     collectorcontroller.crawltask(obj)  def start(self):     log_to_stderr(logging.debug)     pluginobjectlist = []     pluginname in self.settingmodel.getallcollectorname():         name = pluginname.capitalize()         #get plugin class , instanitiate object         module = __import__('plugins.'+pluginname,fromlist=[name])         pluginclass = getattr(module,name)         pluginobject = pluginclass()         pluginobjectlist.append(pluginobject)        pool = pool(8)     jobs = pool.map(self.fire,pluginobjectlist)     pool.close()      print pluginobjectlist

pluginobjectlist got 

[<plugins.name1.name1 instance @ 0x1f54290>, <plugins.name2.name2 instance @ 0x1f54f38>]

picklingerror: can't pickle : attribute lookup builtin.instancemethod failed

but process version work fine

warning kinda subjective deployment , situation current setup follows

i have worker program, fire 6 copies (i have 6 cores). each worker following;

  1. connect redis instance
  2. try , pop work of specific list
  3. pushes logging information
  4. either idles or terminates on lack of work in 'queue'

then each program standalone while still doing work require separate queuing system. have no go-between on processes, might solution problem.


Comments

Post a Comment

Popular posts from this blog

curl - PHP fsockopen help required -

i trying work on cloudflare api , have found in documentation. unfortunately have no idea how can use in fsockopen or through curl. here example of post request. need know how can make request below data post using curl or fsockopen curl https://www.cloudflare.com/api_json.html -d 'a=cache_lvl' \ -d 'tkn=8afbe6dea02407989af4dd4c97bb6e25' \ -d 'email=sample@example.com' \ -d 'z=example.com' \ -d 'v=agg' here link cloudflare https://www.cloudflare.com/docs/client-api.html#s4.2 try one: $ch = curl_init("https://www.cloudflare.com/api_json.html"); curl_setopt($ch, curlopt_returntransfer,1); curl_setopt($ch, curlopt_followlocation, 1); curl_setopt($ch, curlopt_ssl_verifypeer,false); // handling of -d curl parameter here. $param = array( 'a' => 'cache_lvl', 'tkn' => '8afbe6dea02407989af4dd4c97bb6e25', 'email' => 'sample@example.com', 'z' =...
Read more

HTTP/1.0 407 Proxy Authentication Required PHP -

i working in restricted network/limited access internet. since have use proxy make web service call. here php code calling web service , response . $client = new soapclient("http: //www.xyz.com/abc/wsdls/login.wsdl", array("trace" => 1, "exceptions" => 1, 'proxy_host' => $proxy_host, 'proxy_port' => $proxy_port, 'proxy_login' => $proxy_login, 'proxy_password' => $proxy_password)); response is fatal error: uncaught soapfault exception: [wsdl] soap-error: parsing wsdl: couldn't load ' http://www.xyz.com/abc/wsdls/login.wsdl ' : failed load external entity if copy url error message , paste in browser(with same proxy settings) works. 2nd scenario is, if pass values instead of variables giving me diffarant response $client = new soapclient("http: //www.xyz.com/abc/wsdls/login.wsdl", array("trace" => 1, "exceptions" => 1, 'proxy_host' ...
Read more

java - More than one row with the given identifier was found: 1, for class: com.model.Diagnosis -

below model, service implementation , error trace of code. model class: @entity @table(name = "ph_diagnosis_history") @jsonidentityinfo(generator = objectidgenerators.propertygenerator.class, property = "id",scope=diagnosis.class) public class diagnosis extends abstractentity { @column(name = "diagnosis_notes", length=255) @length(max = 255, message = iprprop.max_len_255_msg) private string diagnosisnote; @enumerated(enumtype.string) private diagnosistypeenum type; @temporal(temporaltype.date) @column(name = "diagnosis_date") @jsonserialize(using = jsondateserializer.class) private date diagnosisdate; @manytoone @joincolumn(name = "patient_chart_id") private patientchart patientchart; public string getdiagnosisnote() { return diagnosisnote; } public void setdiagnosisnote(string diagnosisnote) { this.diagnosisnote = diagnosisnote; } public d...
Read more