PHP Classes

Content downloader: Download content from many sites at the same time

Recommend this page to a friend!
  Info   View files Example   View files View files (2)   DownloadInstall with Composer Download .zip   Reputation   Support forum (1)   Blog    
Ratings Unique User Downloads Download Rankings
StarStar 26%Total: 539 This week: 1All time: 5,586 This week: 560Up
Version License PHP version Categories
content-downloader 0.1GNU General Publi...5.0PHP 5, Web services
Description 

Author

This class can download content from multiple sites at the same time.

It can add to a queue one or more URLs of remote sites from which it is necessary to download content.

The class uses the Curl multiple functions to send the requests to all remote sites at once and returns the downloaded content or the error messages in case it was not possible to download the content from any of the sites.

Picture of magog
  Performance   Level  
Name: magog <contact>
Classes: 3 packages by
Country: Russian Federation Russian Federation
Age: 47
All time rank: 285978 in Russian Federation Russian Federation
Week rank: 411 Up22 in Russian Federation Russian Federation Up
Innovation award
Innovation award
Nominee: 2x

Example

<?php

/*... some code ...*/

include_once 'CContentDownloader.php';

// Create CContentDownloader instance
$contRecivier = new CContentDownloader();

// Get contents of all pages in some array
foreach($pages as $k => $page){
   
$uri = $page['host'].$page['uri'];
   
// Add source & remember index in stack
   
$pages[$k]['uri_id'] = $contRecivier->addSource($uri);
}

// Get contents of requested url`s
$contents = $contRecivier->getContents();

// Proccess contents
if($contents)
    foreach (
$pages as $k => $page){
        if(
            isset(
$contents[$page['uri_id']]) &&
           
is_array($contents[$page['uri_id']]) &&
           
$contents[$page['uri_id']]['error'] === false
       
){

           
$content = $contents[$page['uri_id']];
           
           
// Process curl info array (headers)
           
if($content['headers']['http_code'] != 200){
                throw new
Exception("Incorrect HTTP code ({$content['headers']['http_code']}).");
            }
           
            if(
strpos( $content['headers']['content_type'], 'text/html' ) === FALSE){
                throw new
Exception("Unsupported format ({$content['headers']['content_type']}).");
            }
           
           
/* ... do something with content */
           
echo $content['content'];
           
        } else {
           
/* proccess curl error */
           
throw new Exception("Curl error #({$contents[$page['uri_id']]['error']}).");
        }
    }

/*... some code ...*/

?>


  Files folder image Files  
File Role Description
Plain text file CContentDownloader.php Class Content Downloader class
Accessible without login Plain text file example.php Example How to use

 Version Control Unique User Downloads Download Rankings  
 0%
Total:539
This week:1
All time:5,586
This week:560Up
 User Ratings  
 
 All time
Utility:41%StarStarStar
Consistency:33%StarStar
Documentation:-
Examples:25%StarStar
Tests:-
Videos:-
Overall:26%StarStar
Rank:4335