PHP Classes
elePHPant
Icontem

That loop is not right

Recommend this page to a friend!

      Fast PHP CURL Multiple Requests  >  All threads  >  That loop is not right  >  (Un) Subscribe thread alerts  
Subject:That loop is not right
Summary:Will not scale / v. inefficient
Messages:7
Author:Colin McKinnon
Date:2018-09-12 21:45:14
 

  1. That loop is not right   Reply   Report abuse  
Colin McKinnon - 2018-09-12 21:45:14
The curl_multi_exec() function executes the requests asynchronously. The first call launches the requests and subsequent calls poll the status of the requests. But the subsequent requests do not block - i.e. the thread of execution will go around that loop as fast as it can, stealing CPU cycles which could be used more productively elsewhere.

A better solution is to inject a [u]sleep() in the loop:

do {
curl_multi_exec($mh,$running);
$running && usleep(50000);
} while($running);

Further, in a general purpose library, it would be helpful to limit the amount of time the script will wait for responses. Currently, if a response is delayed, the script will bomb out when it reaches the time limit defined for PHP execution - not exit cleanly.

Since integer arithmetic is faster then dealing with times and clocks, the duration doesn't have to be exact, and curl_multi_exec() will return relatively quickly...

$sleep_microseconds=50000;
$max_duration_seconds=10;

$max_iteractions=(int)($max_duration_seconds * 1000000 / $sleep_microseconds);

$iterations=0;

do {
curl_multi_exec($mh,$running);
$running && usleep(50000);
} while($running && ++$iterations<$max_iterations);

(code not tested)

  2. Re: That loop is not right   Reply   Report abuse  
Colin McKinnon - 2018-09-12 21:46:49 - In reply to message 1 from Colin McKinnon
See also https://lampe2e.blogspot.com/2015/03/making-stuff-faster-with-curlmultiexec.html

  3. Re: That loop is not right   Reply   Report abuse  
riccardo castagna - 2018-09-12 22:18:07 - In reply to message 2 from Colin McKinnon
The loop is right, make a test and than we speak

  4. Re: That loop is not right   Reply   Report abuse  
riccardo castagna - 2018-09-12 23:02:43 - In reply to message 2 from Colin McKinnon
And yes, I already knew the point of view of this solution (https://lampe2e.blogspot.com/2015/03/making-stuff-faster-with-curlmultiexec.html), which I had tested before and I had tested many others but they were all solutions for unsatisfactory performance standards for me.
Personally, the use I make of this class, and I use it a lot, is to receive several individual files from different servers and at the moment has always done its job very well.

A different thing is if the class is used to receive from different servers whole web pages that have other resources to load inside them (files css, files js ...). In this case, since the requests are asynchronous and simultaneous codes knead between them.

Is my opinion that the curl multy should not be used for resource requests that have other resources to load within them, I think it was not born for this but for other uses.

But this is only my opinion.


  5. Re: That loop is not right   Reply   Report abuse  
riccardo castagna - 2018-09-13 09:33:05 - In reply to message 4 from riccardo castagna
I can add that the Curl_Multi can also be used for requests for entire web contents that have other contents to load inside them (for example whole web pages or APIs) but in this case we must take precautions.

It is necessary to make an inventory of all the internal resources of each web content that you want to receive through the multi-curl, make the request through the cURL_Multi of the main resource and all internal resources that are within them, then from the main resource via ( for example) str_replace remove the internal resources that would have to be loaded and reinstate them from those that we have inventoried and inserted in the requests of the multi curl. This way you have complete control of all resources.

Fortunately, programming in php is not a philosophy that is the art of contradicting, there is the possibility of doing tests and tests prove the validity of something and tell us if something works or does not work.

  6. Re: That loop is not right   Reply   Report abuse  
riccardo castagna - 2018-09-14 13:21:23 - In reply to message 2 from Colin McKinnon

  7. Re: That loop is not right   Reply   Report abuse  
riccardo castagna - 2018-09-17 04:59:05 - In reply to message 1 from Colin McKinnon
In order to remove any doubt, in the link below, I posted the video of the CPU usage test of the cUrl_Multi class published by me.

In order to avoid that the result of the test could be distorted by the use of other software I avoided using the PC software to capture the screen and I used the smartphone to make the video.

Buena Vista !!!

link to the cpu test: https://www.youtube.com/watch?v=e1kGLYjgn_g