php-development

How to crawl to 3rd party sites [PHP code]

So you want to crawl to others sites? PHP has a built it function – CURL – that lets the code “see URLs”. The following generic code will let you crawl into other URLs with two methods – GET or POST:

function curl_to_api($url,$method="GET",$args=array())
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //Set curl to return the data instead of            printing it to the browser.
if (count($args) > '0')
{
$argsChain="";
$the_keys=array_keys($args);
$num=count($the_keys);
for ($i=0; $i<$num; $i++)
{
$argsChain = $argsChain.$the_keys[$i]."=".$args[$the_keys[$i]]."&";
}
rtrim($argsChain, '&');
if ($method == "POST")
{
curl_setopt($ch, CURLOPT_POST, count($argsChain));
curl_setopt($ch, CURLOPT_POSTFIELDS, $argsChain);
}
else
{
$url = $url . "?".$argsChain;
}
}
curl_setopt($ch, CURLOPT_URL, $url);
$response = curl_exec($ch);
curl_close($ch);
return $response;
}

Enjoy it 🙂

Comments

comments

3 comments

Leave a Reply

Your email address will not be published. Required fields are marked *