Scrapy python : unicode links error -


link encoding

when scraping site scrapy extracts links containing &amd , throws excption: not instantiate link objects unicode urls. assuming utf-8 encoding (which wrong) how can fix error?

i had same problem character → inserted on links. found this related commit on github , used this advice write file link_extractors.py with:

from scrapy.selector import htmlxpathselector scrapy.contrib.linkextractors.sgml import sgmllinkextractor scrapy.utils.response import get_base_url   class customlinkextractor(sgmllinkextractor): """need fix encoding error."""      def extract_links(self, response):         base_url = none         if self.restrict_xpaths:             hxs = htmlxpathselector(response)             base_url = get_base_url(response)             body = u''.join(f x in self.restrict_xpaths                            f in hxs.select(x).extract())             try:                 body = body.encode(response.encoding)             except unicodeencodeerror:                 body = body.encode('utf-8')         else:             body = response.body          links = self._extract_links(body, response.url, response.encoding, base_url)         links = self._process_links(links)         return links

afterwards used in spiders.py:

rules = (     rule(customlinkextractor(allow=('/gp/offer-listing*', ),                            restrict_xpaths=("//li[contains(@class,'a-last')]/a", )),          callback='parse_start_url', follow=true,           ), )

Comments

Post a Comment

Popular posts from this blog

curl - PHP fsockopen help required -

i trying work on cloudflare api , have found in documentation. unfortunately have no idea how can use in fsockopen or through curl. here example of post request. need know how can make request below data post using curl or fsockopen curl https://www.cloudflare.com/api_json.html -d 'a=cache_lvl' \ -d 'tkn=8afbe6dea02407989af4dd4c97bb6e25' \ -d 'email=sample@example.com' \ -d 'z=example.com' \ -d 'v=agg' here link cloudflare https://www.cloudflare.com/docs/client-api.html#s4.2 try one: $ch = curl_init("https://www.cloudflare.com/api_json.html"); curl_setopt($ch, curlopt_returntransfer,1); curl_setopt($ch, curlopt_followlocation, 1); curl_setopt($ch, curlopt_ssl_verifypeer,false); // handling of -d curl parameter here. $param = array( 'a' => 'cache_lvl', 'tkn' => '8afbe6dea02407989af4dd4c97bb6e25', 'email' => 'sample@example.com', 'z' =...
Read more

HTTP/1.0 407 Proxy Authentication Required PHP -

i working in restricted network/limited access internet. since have use proxy make web service call. here php code calling web service , response . $client = new soapclient("http: //www.xyz.com/abc/wsdls/login.wsdl", array("trace" => 1, "exceptions" => 1, 'proxy_host' => $proxy_host, 'proxy_port' => $proxy_port, 'proxy_login' => $proxy_login, 'proxy_password' => $proxy_password)); response is fatal error: uncaught soapfault exception: [wsdl] soap-error: parsing wsdl: couldn't load ' http://www.xyz.com/abc/wsdls/login.wsdl ' : failed load external entity if copy url error message , paste in browser(with same proxy settings) works. 2nd scenario is, if pass values instead of variables giving me diffarant response $client = new soapclient("http: //www.xyz.com/abc/wsdls/login.wsdl", array("trace" => 1, "exceptions" => 1, 'proxy_host' ...
Read more

c# - Resource not found error -

i have problem accessing method on homecontroller. show code of method : [httpget] public actionresult decriptidentifiantetredirige(string login_crypter, string mdp_crypter) { string loginacrypter = _globalmanager.protegemotdepasse(login_crypter); string mdpacrypter = _globalmanager.protegemotdepasse(mdp_crypter); user userapp = new models.user(login_crypter, mdp_crypter); if (userapp.authentificationvalidee(userapp.userlogin, userapp.password)) { session["name"] = userapp.userlogin; return redirecttoaction("accueil", "home"); } else { return redirecttoaction("validerauthentification", "home"); } } then in routeconfig.cs wrote route : routes.maproute( name: "authentificationapresdecryptage", url: "{controller}/{action}/{login_crypter}/{mdp_crypter}", defaults: new...
Read more