PHP Classes

Content downloader: Download content from many sites at the same time

Recommend this page to a friend!
     
  Info   Example   View files Files   Install with Composer Install with Composer   Download Download   Reputation   Support forum   Blog    
Ratings Unique User Downloads Download Rankings
StarStar 26%Total: 539 All time: 5,587 This week: 455Up
Version License PHP version Categories
content-downloader 0.1GNU General Publi...5.0PHP 5, Web services
Description 

Author

magog

This class can download content from multiple sites at the same time.

It can add to a queue one or more URLs of remote sites from which it is necessary to download content.

The class uses the Curl multiple functions to send the requests to all remote sites at once and returns the downloaded content or the error messages in case it was not possible to download the content from any of the sites.

Picture of magog
  Performance   Level  
Name: magog <contact>
Classes: 3 packages by
Country: Russian Federation Russian Federation

Level 3 StarStarStar

 Innovation award
Innovation award
Nominee: 2x

Example

<?php

/*... some code ...*/

include_once 'CContentDownloader.php';

// Create CContentDownloader instance
$contRecivier = new CContentDownloader();

// Get contents of all pages in some array
foreach($pages as $k => $page){
    $uri = $page['host'].$page['uri'];
    // Add source & remember index in stack
    $pages[$k]['uri_id'] = $contRecivier->addSource($uri);
}

// Get contents of requested url`s
$contents = $contRecivier->getContents();

// Proccess contents
if($contents)
    foreach ($pages as $k => $page){
        if(
            isset($contents[$page['uri_id']]) &&
            is_array($contents[$page['uri_id']]) &&
            $contents[$page['uri_id']]['error'] === false
        ){

            $content = $contents[$page['uri_id']];
           
            // Process curl info array (headers)
            if($content['headers']['http_code'] != 200){
                throw new Exception("Incorrect HTTP code ({$content['headers']['http_code']}).");
            }
           
            if(strpos( $content['headers']['content_type'], 'text/html' ) === FALSE){
                throw new Exception("Unsupported format ({$content['headers']['content_type']}).");
            }
           
            /* ... do something with content */
            echo $content['content'];
           
        } else {
            /* proccess curl error */
            throw new Exception("Curl error #({$contents[$page['uri_id']]['error']}).");
        }
    }

/*... some code ...*/

?>

  Files folder image Files (2)  
File Role Description
Plain text file CContentDownloader.php Class Content Downloader class
Accessible without login Plain text file example.php Example How to use

The PHP Classes site has supported package installation using the Composer tool since 2013, as you may verify by reading this instructions page.
Install with Composer Install with Composer
Downloadcontent-downloader-2012-04-06.zip 1KB
Downloadcontent-downloader-2012-04-06.tar.gz 1KB
Install with ComposerInstall with Composer
 Version Control Unique User Downloads Download Rankings  
 0%
Total:539
This week:0
All time:5,587
This week:455Up
 User Ratings  
 
 All time
Utility:41%StarStarStar
Consistency:33%StarStar
Documentation:-
Examples:25%StarStar
Tests:-
Videos:-
Overall:26%StarStar
Rank:4422
  

  Applications that use this package  
No pages of applications that use this class were specified.

Add link image If you know an application of this package, send a message to the author to add a link here.