Saturday, June 30, 2012

Detecting Incomplete File Uploads in PHP


Detecting Incomplete File Uploads in PHP

PHP has no good mechanism for doing this.  The best solution I have been able to come up with is to check the file size, wait some amount of time, and check it again.  If I were processing an upload directory for instance, I would sample the file size as I was building the file list, save the size, and then check it again when I go to actually process the files.  But even so, you likely need to add a sleep command in there somewhere.  The resulting code looks something like this:

   const FILE_DETECTION_DELAY = 5;
  
   /**
    * Gets a list of files in a given directory.  The directory
    * must exist or this function will throw an exception
    *
    * @param string $path
    *
    * @return array
    */
   private function _getFilesInDirectory($path) {
  
      $fileList = array();
      $fileSize = array();
      $handle = opendir($path);
      if($handle) {
         while(false !== ($file = readdir($handle))) {
            if($file != "." && $file != "..") {
               $fileName[] = $path . $file;
               $fileSize[] = filesize($path . $file);
            }
         }
         closedir($handle);
      } else {
         $this->logger->log('[' . __METHOD__ . '] ' .
                      'Error: Failed to find directory at: ' . $path);
         throw new Exception("Failed to find directory: " . $path);
      }
      sleep (self::FILE_DETECTION_DELAY);
      $fileList = array();
      for ($i=0;$i<count($fileName);$i++) {
         if (filesize($fileName[$i]) == $fileSize[$i]) {
            $fileList[] = $fileName[$i];
         }
      }
      return $fileList;
   }


This is just a quick attempt at creating this code.  I’ve never actually run this function.  So it’s possible there may be in an error in it. 

The reason I don’t use this function is there’s still a possibility that the file may be under upload, but the upload has temporarily stalled.  Longer delays reduce the risk of this, but they slow down code execution.   You can trade off portability for a more definitive solution to this problem (providing you are running under Unix or Linux) by using the “lsof” command.  “lsof” stands for “list open files”.  Run with no arguments, it will do exactly that.  But, if you run it with a specific file as an argument, it will return a list of processes using that file (like an FTP upload for instance).  If no process is using the file, it returns nothing.  Therefore, the following simple function can definitively detect an open file on Unix flavored systems.

   /**
    * REQUIRES: Unix OS
    * Checks for an open file with the UNIX "lsof" command
    * If the file is open, this command will return process
    * information for the process using it.
    * If the file is not open, this command returns nothing.
    * This makes the code dependent on a Unix system supporting the "lsof" command
    *
    * @param string $file
    * @return bool
    */
   private function _fileIsOpen($file) {
      $status = system('lsof ' . $file);
      if($status) {
         return true;
      } else {
         return false;
      }
   }

So if we wanted to produce something similar to what we have in the first example, we’d simply add a function to find all the files:

   /**
    * Gets a list of files in a given directory.  The directory
    * must exist or this function will throw an exception
    *
    * @param string $path
    *
    * @return bool
    */
   private function _getFilesInDirectory($path) {
  
      $fileList = array();
      $handle = opendir($path);
      if($handle) {
         while(false !== ($file = readdir($handle))) {
            if($file != "." && $file != "..") {
               $fileList[] = $path . $file;
            }
         }
         closedir($handle);
         return $fileList;
      } else {
         $this->logger->log('[' . __METHOD__ . '] ' .
               'Error: Failed to find directory at: ' . $path);
         throw new Exception("Failed to find directory: " . $path);
      }
   }

And finally, some kind of overall controller to call these:

   /**
    * A function to retrieve all uploaded files not still
    * in the process of being uploaded.
    *
    * @param string $path
    *
    * @return array
    */
   private function _getCompletedUploadList($path) {
      $fileList = $this->_getFilesInDirectory($path);
      $validFiles = array();
      foreach ($fileList as $file) {
         if (!$this->_fileIsOpen($file)) {
            $validFiles[] = $file;
         }
      }
      return $validFiles();
   }

Sunday, June 17, 2012

Create a Stand-Alone Magento Web Page


Sometimes, it’s nice to just quickly whip out a single page Magento script to test something such as new API you are trying to talk to.  But, if that page is going to use Magento’s functionality, it needs to load up all the XML control files first so Magento knows where to find its classes. 

The following script does just that:

<?php

/*
 * place this in a subdirectory under docroot
 * i.e. docroot/script/testClass.php
 * http://your-test-server/script/testClass.php
 *
 */

$baseDir = dirname(dirname(__FILE__));

include_once $baseDir . "/app/Mage.php";

class testClass {

   public function run() {
      Mage::App();
      $config = Mage::GetConfig();
      $config->init();

      // your test code here 
     
   }
}

$testClass = new testClass;
$testClass->run();

?>

Saturday, June 2, 2012

Flush Magento Cache from Command Line or Standalone Script

There are several scripts floating around for this.  But unfortunately, if you have any kind of sophisticated cache (i.e. not using the Magento defaults) none of the scripts I could find will work.

The problem with them is they get a handle to the default Magento cache and attempt to flush it.  But if you are using Memcache or Redis, you’ve altered the “local.xml” and the default cache flush will not hit the proper cache.  When you do a manual cache flush via the control console, the “index.php” file calls Mage::run() which conditions the cache variables for you.  But since Mage::run() contains a dispatch() call, you cannot make a standalone page or a standalone script using this technique.

If you have reconfigured your cache, your local.xml has a section that looks something like this:


<config>
    <global>
        <cache>
            <backend>Zend_Cache_Backend_Redis</backend>
            <backend_options>
                <server>some_server_name.com</server>
                <port>some_port</port>
                <database>2</database>
                <force_standalone>0</force_standalone> 
                <automatic_cleaning_factor>20000</automatic_cleaning_factor>
            </backend_options>
        </cache>
        <full_page_cache>
            <backend>Zend_Cache_Backend_Redis</backend>
            <backend_options>
                <server>some_server_name.com</server>
                <port>some_port</port>
                <database>3</database>
                <force_standalone>0</force_standalone> 
                <automatic_cleaning_factor>20000</automatic_cleaning_factor>
            </backend_options>
        </full_page_cache>

In order to clean out that full page cache, you need  to read in this XML and use it as a parameter when you instantiate the cache model prior to attempting to a cache flush. 

The following script does this.  This script is meant to be called from “docroot/some_dir” via a standard “http” access as a standalone web page.  You’d probably want to set “.htaccess” to limit access to some kind of automated agent because if somebody hits this page manually, it’s going to dump your cache on the spot.

<?php

$baseDir = dirname(dirname(__FILE__));

include_once $baseDir . "/app/Mage.php";

$cacheType = 'full_page';

// We only really need the Mage::setRoot() function here, but the problem is
// that the underlying $cache=Mage_Core_Model_Cache function calls
// Mage::getConfig()->getOptions()in its constructor to get the cache parameters
// So we have to first use the Mage::App() function to create Mage::$_config
// We also get the needed Mage::setRoot() as a byproduct of Mage::App()  

Mage::App();

// grab a handle to the Mage::$_config variable with Mage::GetConfig(). 
$config = Mage::GetConfig();

// load the Base directories so config knows where to find the configuration files
// loads "docroot/app/etc/*" for later use with $config->getNode
$config->loadBase();

// getNode pulls the $options from the "local_xml" file 
// Change 'full_page_cache' here to 'cache' to get first level cache.
$options = $config->getNode('global/full_page_cache');
if ($options) {
  $options = $options->asArray();
} else {
  $options = array();
}

$cache = $config->getModelInstance('core/cache', $options);

$tags = $cache->cleanType($cacheType);

Mage::log('Full Page Cache Cleared via CL | Type: full_page | Tags: FPC');
echo "Cleared Full Page Cache";

?>

Saturday, May 12, 2012

MySQL select “MAX” row from “GROUP BY” result set


Let’s say you have a database containing orders, which map to shipments via a shipment map.  You want to find the most recently created shipment for every product in the database and determine if it was for a quantity > 1. 

So you do a “GROUP BY product_id”, and you then do a “MAX(created_date)” to find the newest entry.  So far: so good.  But now what?  If you simply try and grab the “quantity” field along with the “MAX(created_date)”, those two values will not come from the same row of the table, and you will not get what you are looking for.

The trick is to use the results of this query to create a new table containing just the “product_id” and “created_date” and then match that table against the original table to find the row with the same “product_id” and “created_date” from which we can take “quantity” and anything else we want.  In the example, I just wanted the product_id’s for which the last shipment had a quantity greater than one.  So that’s the only value I picked off.

SELECT ol.product_id FROM order_line ol
INNER JOIN order_line_shipment_map olsm using (order_line_id)
INNER JOIN
    (
        SELECT ol.product_id, max(olsm.created_date) AS max_created_date
         FROM order_line_shipment_map olsm
        INNER JOIN order_line ol USING (order_line_id)
         GROUP BY ol.product_id
    )    AS rtab
ON rtab.vendor_product_id = ol.vendor_product_id
AND olsm.created_date = rtab.max_created_date
WHERE ol.quantity > 1;