Flush Magento Cache from Command Line or Standalone Script

There are several scripts floating around for this.  But unfortunately, if you have any kind of sophisticated cache (i.e. not using the Magento defaults) none of the scripts I could find will work.

The problem with them is they get a handle to the default Magento cache and attempt to flush it.  But if you are using Memcache or Redis, you’ve altered the “local.xml” and the default cache flush will not hit the proper cache.  When you do a manual cache flush via the control console, the “index.php” file calls Mage::run() which conditions the cache variables for you.  But since Mage::run() contains a dispatch() call, you cannot make a standalone page or a standalone script using this technique.

If you have reconfigured your cache, your local.xml has a section that looks something like this:

<config>

    <global>

        <cache>

            <backend>Zend_Cache_Backend_Redis</backend>

            <backend_options>

                <server>some_server_name.com</server>

                <port>some_port</port>

                <database>2</database>

                <force_standalone>0</force_standalone> 

                <automatic_cleaning_factor>20000</automatic_cleaning_factor>

            </backend_options>

        </cache>

        <full_page_cache>

            <backend>Zend_Cache_Backend_Redis</backend>

            <backend_options>

                <server>some_server_name.com</server>

                <port>some_port</port>

                <database>3</database>

                <force_standalone>0</force_standalone> 

                <automatic_cleaning_factor>20000</automatic_cleaning_factor>

            </backend_options>

        </full_page_cache>

In order to clean out that full page cache, you need  to read in this XML and use it as a parameter when you instantiate the cache model prior to attempting to a cache flush. 

The following script does this.  This script is meant to be called from “docroot/some_dir” via a standard “http” access as a standalone web page.  You’d probably want to set “.htaccess” to limit access to some kind of automated agent because if somebody hits this page manually, it’s going to dump your cache on the spot.

<?php

$baseDir = dirname(dirname(__FILE__));

include_once $baseDir . "/app/Mage.php";

$cacheType = 'full_page';

// We only really need the Mage::setRoot() function here, but the problem is

// that the underlying $cache=Mage_Core_Model_Cache function calls

// Mage::getConfig()->getOptions()in its constructor to get the cache parameters

// So we have to first use the Mage::App() function to create Mage::$_config

// We also get the needed Mage::setRoot() as a byproduct of Mage::App()  

Mage::App();

// grab a handle to the Mage::$_config variable with Mage::GetConfig(). 

$config = Mage::GetConfig();

// load the Base directories so config knows where to find the configuration files

// loads "docroot/app/etc/*" for later use with $config->getNode

$config->loadBase();

// getNode pulls the $options from the "local_xml" file 

// Change 'full_page_cache' here to 'cache' to get first level cache.

$options = $config->getNode('global/full_page_cache');

if ($options) {

  $options = $options->asArray();

} else {

  $options = array();

}

$cache = $config->getModelInstance('core/cache', $options);

$tags = $cache->cleanType($cacheType);

Mage::log('Full Page Cache Cleared via CL | Type: full_page | Tags: FPC');

echo "Cleared Full Page Cache";

?>

MySQL select “MAX” row from “GROUP BY” result set

Let’s say you have a database containing orders, which map to shipments via a shipment map.  You want to find the most recently created shipment for every product in the database and determine if it was for a quantity > 1. 

So you do a “GROUP BY product_id”, and you then do a “MAX(created_date)” to find the newest entry.  So far: so good.  But now what?  If you simply try and grab the “quantity” field along with the “MAX(created_date)”, those two values will not come from the same row of the table, and you will not get what you are looking for.

The trick is to use the results of this query to create a new table containing just the “product_id” and “created_date” and then match that table against the original table to find the row with the same “product_id” and “created_date” from which we can take “quantity” and anything else we want.  In the example, I just wanted the product_id’s for which the last shipment had a quantity greater than one.  So that’s the only value I picked off.

SELECT ol.product_id FROM order_line ol

INNER JOIN order_line_shipment_map olsm using (order_line_id)

INNER JOIN

    (

        SELECT ol.product_id, max(olsm.created_date) AS max_created_date

         FROM order_line_shipment_map olsm

        INNER JOIN order_line ol USING (order_line_id)

         GROUP BY ol.product_id

    )    AS rtab

ON rtab.vendor_product_id = ol.vendor_product_id

AND olsm.created_date = rtab.max_created_date

WHERE ol.quantity > 1;

Converting MAC “CR” Line Endings to UNIX “LF”

If your file loads in looking something like this:

it is because you’ve saved it using a MAC formatted file structure, probably using MAC Excel, as MACOS 10.x uses UNIX style line endings.  If you then open this file under VIM running in Linux and VIM is not set to automatically detect it, you get this mess.   This happens because old-school MACs used CR line endings, and Unix expects LF type line endings.

Most UNIX based utilities and scripting languages do not know how to interpret a CR type line ending.  If you try to parse the CSV file with any UNIX based utility or scripting language, you are going to have trouble because the function is going to treat the entire file as a single line.  Most PHP file handling functions also have trouble with it.

So you need to convert the line endings.  There are numerous ways to do this such as using TR or Perl:

tr '\n' '\r' < mac_formatted _file.csv

perl -ne 's/([^\r])\r/$1\n/g; s/\r//g; print;'  mac_format.csv

But the easiest way I have found is to just use the VIM editor. 

First, check your current file format settings with a:   

:set ffs? 

If you are having trouble reading MAC formatted files, you will likely find this produces a:

Fileformats=unix,dos

A simple execution of:

:set ffs=unix,dos,mac

will generally easily fix this.   Exit the file and open it again, and you will now be able to see your text correctly.  However, you haven’t actually converted it.  You’ve just taught VIM how to automatically understand the CR type line endings.

VIM can be in any one of three file editing modes: dos, unix, or mac.  VIM attempts to figure out the correct mode when it loads the file.  But if the file format settings (ffs) do not contain the mac entry, VIM does not know how to read MAC files.

You can convert the line endings by first switching the editor into MAC mode so it can understand the file, switching the file format over to UNIX, and then writing out the converted file:

1. :e ++ff=MAC
2. :setlocal ff=unix
3. :w 

The VIM “:e ++ff=mac” command switches VIM over to look for ‘\r’ (Carriage Return or CR) characters as line endings.  This will show up as “^M” in the file if the file is being interpreted as a UNIX file.  Unix uses a pure ‘\n’ (Line Feed or LR) as the only line-ending indicator.  The UNIX LF character shows up as a “^J” if you switch the editor over to UNIX mode.  If you switch a UNIX file to DOS mode, it will display correctly, but it will show a display at the bottom saying “[CR missing][dos]” right after you execute the “:e ++ff=dos” command.

You can switch back and forth into any mode you want by executing:

1. :e ++ff=current_format
2. :setlocal ff=target_format
3. :w

where “current_format” and “target_format” are:  dos, unix, or mac

RROD XBOX-360 Fix Using 5.25 Fan.doc

I added a 5 ¼” fan to the top of my XBOX 360 because it kept getting the RROD.  And I couldn’t find any other way to seemingly permanently fix it.  At the time I did this, I was working under contract for a company in the Philippines, and I couldn’t get a replacement US console to play my US region coded games that I had brought with me.  It’s a lot of work just to fix a console that can be replaced for $150.  But it did produce what appeared to be either a long-term or perhaps permanent fix.  The fix goes beyond just adding a fan.  I shimmed the GPU with a 0.5mm aluminum shim, X-Clamp fixed both the GPU and CPU, reflowed the solder by CAREFULLY overheating the box as detailed below, put the DVD drive outside the case by extending its cables, and then finally, I cut in a 5 ¼” fan over the top of the GPU.

I am on my seventh and eighth X-BOX 360 (I have two working consoles that have never failed, but they are both relatively new models). 

In order to fix this, you have to stop the PCB from flexing, and you need to dissipate the heat somehow.  The GPU and CPU both run very hot—too hot for the thermal characteristics of the case.  The problem is exacerbated by the RHOS compliant non-leaded solder used on the package.  The primary reason that lead is used in solder is to make it more flexible and resistant to cracking. 

The “X-Clamp” fix basically pins the motherboard to the case.  The metal on the bottom of the case is very thick.  This by itself goes a long way towards stopping the constant PCB flexing due to thermal expansion that is largely responsible for eventually breaking the solder connections on the GPU and to a lesser extent CPU packaging.   There are very good articles on how to do the X-Clamp fix here: http://xbox-experts.com/tutorial/team-hybrids-ultimate-xclamp-fix-released/   WARNING::Before you just launch in and repair it this way, you should know I followed this to a T, and it didn’t last.  It’s not sufficient in and of itself to permanently fix an XBOX.  And further, there’s a problem because the GPU and the CPU packages are not the same height off the motherboard.  This article discusses using a shim on the GPU which I strongly recommend:

http://blog.reloadsystems.net/2007/08/14/x-clamp-replacement-and-shroud-mod-360-rol-fix/

The X-Clamp fix calls for putting a 2mm stack of washers on top of the motherboard.  The heat sinks sit on the washers.  However, the GPU is only about 1.5mm high.  So this leaves a 0.5mm space between the top of the GPU and the heat sink which relies on the solder paste as a filler.  This is very bad.  And we want to get mechanical pressure on the GPU to keep the solder balls from separating in the event they crack.  So the shim is essential to the fix.

I couldn’t find the aluminum shim anywhere in the Philippines.  I ended up sanding the paint off a soda can and cutting a couple pieces to get the requisite shim height.  I then put down a thin layer of solder paste between the aluminum sheets.  Obviously, I don’t recommend this, but it did seem to work.

Lastly, once you finish the fix, you should follow the directions at Llamma for safely overheating the GPU by pulling the fan shroud and putting the fan directly on the CPU as detailed in these step-by-step instructions:

http://www.llamma.com/xbox360/repair/ring_of_light_x-clamp_fix.htm

They make a good argument for using this method, and I agree with it.  And I can attest it does work.  I did it three times before I finished fully modding this box as it kept breaking again every few days until I finally went all out with the shim and the new fan.

In order to add a 12v fan, you need to find a 12v source on the motherboard.  I used the source from here: http://www.llamma.com/xbox360/mods/images/Fan_Mod/360-124.jpg.  That solder joint in the photo looks terrible.  You only need a small amount of solder.  And flux is your friend.

A huge set of Kudos are due to all of these sites for their help fixing this problem.  There are a ton of bogus X-Box repair links out there.  But these are are the gold amongst the slag, and they helped me hugely.

However, even after I finished all these fixes, I still didn’t feel very good about my 360.  I figured that the board is bound to flex due to thermal expansion, and obviously, the GPU balls were already cracked.  I had already RROD’ed three times and supposedly “fixed it” by this time.  So I decided I needed a better way to keep the GPU cool.

After looking the box over for a while, I came to the conclusion that the only way to cool the GPU is to remove the DVD drive.  The DVD drive is sitting right on top of the GPU and almost completely blocking the airflow to it.  Now mind you, this isn’t going to be the world’s most beautiful 360 once it’s finished.  The DVD sitting off to the side isn’t very nice to look at.  But in a pinch, you have to do what you have to do.

So I salvaged a 5 ¼” fan from a dead PC power supply.  I also cut off the wire harness to give me wire to work with in order to extend the DVD power cable which I had zero chance of finding another of in the Philippines.  The power cable does not carry any frequency dependent information, so it’s perfectly acceptable to lengthen it.  The SATA cable on the other hand is quite short and likely won’t respond well to being messed with.  Luckily, it’s a standard SATA cable.  So you can just replace it with a SATA cable made for a PC which is plenty long.

You also need to drill a hole in the back of the XBOX (in addition to the giant hole you need to make for the fan).  I strongly recommend wrapping the power cable and SATA cable with electrical tape where they pass through the 360 case to prevent abrasion of the wires over time or you can put a grommet in the hole.  I went the tape route.

There is a natural step in the heat shield for the top of the case.  I set my 5 ¼” fan right in this natural flat spot.  But when I put it back into the case, it hit on the heatsink shroud, and I had to cut the shroud and then repair it as best I could with index cards and tape.  So, if you are going to do this, make sure you position your fan far enough forward in the case to miss the X-box fan shroud or realize you are going to have to modify the fan shroud.

You need 12 volts for the 51/4” fan.  I you don’t like the connection point above, there are other places you can get it.  Use a voltmeter and fish around for it.  Or find some other sites with alternate connection points.  BE CAREFUL WHICH WAY YOU CONNECT THE LEADS.  You want the fan to blow down into the case.  If you reverse the leads, it’s going to blow outward.  You want the air to flow down from the top case, over the GPU, and out through the back via the two existing exhaust fans.

I set the DVD drive on the carrier.  Opening the DVD drive becomes a bit of a problem.  The button to open the DVD drive is actually on the main case.  And it relies on a plastic piece that is affixed to the DVD drive.  So you have to replace the button somehow if you still want it.  However, the software method of opening the drive works fine.

I used this 360 for about five months after I made this fix.  I played all the way through Mass Effect as well as a couple other games.  The 360 is still in the Philippines.  I haven’t been back there to test that it is still working, but I expect it is.  But being an engineer, I am not going to say it’s permanently fixed without a much larger sample size.  But it definitely created a much longer term fix than any other solution I tried.

When Microsoft brought out the 360 “slim”, they merged the GPU and CPU onto one chip.  The units made just prior to this had half the power consumption (and half the heat generation) of the original units.  I searched around and bought one of these last two chip units when I realized what MS had done.  Now that they have merged the GPU and CPU, the concentrated heat signature around that single package is the same as the original GPU heat signature.  So it remains to be seen whether the slim units are reliable or not.  Also, they scream like a banshee when the DVD drive is on high speed.  You are almost forced to put the games on the hard drive.

Let me also say that putting your games on your hard drive is a very good idea anyway.  My console number five bit the dust because the DVD stopped working.  I tried to change the resistor setting to increase the laser strength, but it didn’t help in the slightest.  That console still “works”, but it can’t read the DVD’s.  Even before the DVD died, the output was getting flaky. I think the graphics scalar chip is going out.  It only worked reliably via HDMI even before the DVD drive died.

When that unit died, I was forced to buy a MS slim (I have two consoles in the US and one in the Philippines) to replace it.  The first “slim” unit I bought died after about three weeks.  It started freezing up and resetting.  I took it back where I bought it and exchanged it.  The replacement unit is fine…so far. 

At the end of the day, it’s hard to argue that MS knows anything about hardware design.   It’s a software company.  I’ll probably go with Sony here on out.  The only reason I haven’t is I have an immense library of games with double copies of a lot of the network games.