Converting MAC “CR” Line Endings to UNIX “LF”

If your file loads in looking something like this:

it is because you’ve saved it using a MAC formatted file structure, probably using MAC Excel, as MACOS 10.x uses UNIX style line endings.  If you then open this file under VIM running in Linux and VIM is not set to automatically detect it, you get this mess.   This happens because old-school MACs used CR line endings, and Unix expects LF type line endings.

Most UNIX based utilities and scripting languages do not know how to interpret a CR type line ending.  If you try to parse the CSV file with any UNIX based utility or scripting language, you are going to have trouble because the function is going to treat the entire file as a single line.  Most PHP file handling functions also have trouble with it.

So you need to convert the line endings.  There are numerous ways to do this such as using TR or Perl:

tr '\n' '\r' < mac_formatted _file.csv

perl -ne 's/([^\r])\r/$1\n/g; s/\r//g; print;'  mac_format.csv

But the easiest way I have found is to just use the VIM editor. 

First, check your current file format settings with a:   

:set ffs? 

If you are having trouble reading MAC formatted files, you will likely find this produces a:

Fileformats=unix,dos

A simple execution of:

:set ffs=unix,dos,mac

will generally easily fix this.   Exit the file and open it again, and you will now be able to see your text correctly.  However, you haven’t actually converted it.  You’ve just taught VIM how to automatically understand the CR type line endings.

VIM can be in any one of three file editing modes: dos, unix, or mac.  VIM attempts to figure out the correct mode when it loads the file.  But if the file format settings (ffs) do not contain the mac entry, VIM does not know how to read MAC files.

You can convert the line endings by first switching the editor into MAC mode so it can understand the file, switching the file format over to UNIX, and then writing out the converted file:

1. :e ++ff=MAC
2. :setlocal ff=unix
3. :w 

The VIM “:e ++ff=mac” command switches VIM over to look for ‘\r’ (Carriage Return or CR) characters as line endings.  This will show up as “^M” in the file if the file is being interpreted as a UNIX file.  Unix uses a pure ‘\n’ (Line Feed or LR) as the only line-ending indicator.  The UNIX LF character shows up as a “^J” if you switch the editor over to UNIX mode.  If you switch a UNIX file to DOS mode, it will display correctly, but it will show a display at the bottom saying “[CR missing][dos]” right after you execute the “:e ++ff=dos” command.

You can switch back and forth into any mode you want by executing:

1. :e ++ff=current_format
2. :setlocal ff=target_format
3. :w

where “current_format” and “target_format” are:  dos, unix, or mac

RROD XBOX-360 Fix Using 5.25 Fan.doc

I added a 5 ¼” fan to the top of my XBOX 360 because it kept getting the RROD.  And I couldn’t find any other way to seemingly permanently fix it.  At the time I did this, I was working under contract for a company in the Philippines, and I couldn’t get a replacement US console to play my US region coded games that I had brought with me.  It’s a lot of work just to fix a console that can be replaced for $150.  But it did produce what appeared to be either a long-term or perhaps permanent fix.  The fix goes beyond just adding a fan.  I shimmed the GPU with a 0.5mm aluminum shim, X-Clamp fixed both the GPU and CPU, reflowed the solder by CAREFULLY overheating the box as detailed below, put the DVD drive outside the case by extending its cables, and then finally, I cut in a 5 ¼” fan over the top of the GPU.

I am on my seventh and eighth X-BOX 360 (I have two working consoles that have never failed, but they are both relatively new models). 

In order to fix this, you have to stop the PCB from flexing, and you need to dissipate the heat somehow.  The GPU and CPU both run very hot—too hot for the thermal characteristics of the case.  The problem is exacerbated by the RHOS compliant non-leaded solder used on the package.  The primary reason that lead is used in solder is to make it more flexible and resistant to cracking. 

The “X-Clamp” fix basically pins the motherboard to the case.  The metal on the bottom of the case is very thick.  This by itself goes a long way towards stopping the constant PCB flexing due to thermal expansion that is largely responsible for eventually breaking the solder connections on the GPU and to a lesser extent CPU packaging.   There are very good articles on how to do the X-Clamp fix here: http://xbox-experts.com/tutorial/team-hybrids-ultimate-xclamp-fix-released/   WARNING::Before you just launch in and repair it this way, you should know I followed this to a T, and it didn’t last.  It’s not sufficient in and of itself to permanently fix an XBOX.  And further, there’s a problem because the GPU and the CPU packages are not the same height off the motherboard.  This article discusses using a shim on the GPU which I strongly recommend:

http://blog.reloadsystems.net/2007/08/14/x-clamp-replacement-and-shroud-mod-360-rol-fix/

The X-Clamp fix calls for putting a 2mm stack of washers on top of the motherboard.  The heat sinks sit on the washers.  However, the GPU is only about 1.5mm high.  So this leaves a 0.5mm space between the top of the GPU and the heat sink which relies on the solder paste as a filler.  This is very bad.  And we want to get mechanical pressure on the GPU to keep the solder balls from separating in the event they crack.  So the shim is essential to the fix.

I couldn’t find the aluminum shim anywhere in the Philippines.  I ended up sanding the paint off a soda can and cutting a couple pieces to get the requisite shim height.  I then put down a thin layer of solder paste between the aluminum sheets.  Obviously, I don’t recommend this, but it did seem to work.

Lastly, once you finish the fix, you should follow the directions at Llamma for safely overheating the GPU by pulling the fan shroud and putting the fan directly on the CPU as detailed in these step-by-step instructions:

http://www.llamma.com/xbox360/repair/ring_of_light_x-clamp_fix.htm

They make a good argument for using this method, and I agree with it.  And I can attest it does work.  I did it three times before I finished fully modding this box as it kept breaking again every few days until I finally went all out with the shim and the new fan.

In order to add a 12v fan, you need to find a 12v source on the motherboard.  I used the source from here: http://www.llamma.com/xbox360/mods/images/Fan_Mod/360-124.jpg.  That solder joint in the photo looks terrible.  You only need a small amount of solder.  And flux is your friend.

A huge set of Kudos are due to all of these sites for their help fixing this problem.  There are a ton of bogus X-Box repair links out there.  But these are are the gold amongst the slag, and they helped me hugely.

However, even after I finished all these fixes, I still didn’t feel very good about my 360.  I figured that the board is bound to flex due to thermal expansion, and obviously, the GPU balls were already cracked.  I had already RROD’ed three times and supposedly “fixed it” by this time.  So I decided I needed a better way to keep the GPU cool.

After looking the box over for a while, I came to the conclusion that the only way to cool the GPU is to remove the DVD drive.  The DVD drive is sitting right on top of the GPU and almost completely blocking the airflow to it.  Now mind you, this isn’t going to be the world’s most beautiful 360 once it’s finished.  The DVD sitting off to the side isn’t very nice to look at.  But in a pinch, you have to do what you have to do.

So I salvaged a 5 ¼” fan from a dead PC power supply.  I also cut off the wire harness to give me wire to work with in order to extend the DVD power cable which I had zero chance of finding another of in the Philippines.  The power cable does not carry any frequency dependent information, so it’s perfectly acceptable to lengthen it.  The SATA cable on the other hand is quite short and likely won’t respond well to being messed with.  Luckily, it’s a standard SATA cable.  So you can just replace it with a SATA cable made for a PC which is plenty long.

You also need to drill a hole in the back of the XBOX (in addition to the giant hole you need to make for the fan).  I strongly recommend wrapping the power cable and SATA cable with electrical tape where they pass through the 360 case to prevent abrasion of the wires over time or you can put a grommet in the hole.  I went the tape route.

There is a natural step in the heat shield for the top of the case.  I set my 5 ¼” fan right in this natural flat spot.  But when I put it back into the case, it hit on the heatsink shroud, and I had to cut the shroud and then repair it as best I could with index cards and tape.  So, if you are going to do this, make sure you position your fan far enough forward in the case to miss the X-box fan shroud or realize you are going to have to modify the fan shroud.

You need 12 volts for the 51/4” fan.  I you don’t like the connection point above, there are other places you can get it.  Use a voltmeter and fish around for it.  Or find some other sites with alternate connection points.  BE CAREFUL WHICH WAY YOU CONNECT THE LEADS.  You want the fan to blow down into the case.  If you reverse the leads, it’s going to blow outward.  You want the air to flow down from the top case, over the GPU, and out through the back via the two existing exhaust fans.

I set the DVD drive on the carrier.  Opening the DVD drive becomes a bit of a problem.  The button to open the DVD drive is actually on the main case.  And it relies on a plastic piece that is affixed to the DVD drive.  So you have to replace the button somehow if you still want it.  However, the software method of opening the drive works fine.

I used this 360 for about five months after I made this fix.  I played all the way through Mass Effect as well as a couple other games.  The 360 is still in the Philippines.  I haven’t been back there to test that it is still working, but I expect it is.  But being an engineer, I am not going to say it’s permanently fixed without a much larger sample size.  But it definitely created a much longer term fix than any other solution I tried.

When Microsoft brought out the 360 “slim”, they merged the GPU and CPU onto one chip.  The units made just prior to this had half the power consumption (and half the heat generation) of the original units.  I searched around and bought one of these last two chip units when I realized what MS had done.  Now that they have merged the GPU and CPU, the concentrated heat signature around that single package is the same as the original GPU heat signature.  So it remains to be seen whether the slim units are reliable or not.  Also, they scream like a banshee when the DVD drive is on high speed.  You are almost forced to put the games on the hard drive.

Let me also say that putting your games on your hard drive is a very good idea anyway.  My console number five bit the dust because the DVD stopped working.  I tried to change the resistor setting to increase the laser strength, but it didn’t help in the slightest.  That console still “works”, but it can’t read the DVD’s.  Even before the DVD died, the output was getting flaky. I think the graphics scalar chip is going out.  It only worked reliably via HDMI even before the DVD drive died.

When that unit died, I was forced to buy a MS slim (I have two consoles in the US and one in the Philippines) to replace it.  The first “slim” unit I bought died after about three weeks.  It started freezing up and resetting.  I took it back where I bought it and exchanged it.  The replacement unit is fine…so far. 

At the end of the day, it’s hard to argue that MS knows anything about hardware design.   It’s a software company.  I’ll probably go with Sony here on out.  The only reason I haven’t is I have an immense library of games with double copies of a lot of the network games. 

Accessing MySQL Using Bash Scripting

Accessing MySQL Using Bash Scripting

We have a spreadsheet in which the first column contains a customer’s email addres.  We want to perform some processing on each customer, but the script that does the processing only recognizes the customerId.  What’s the quickest way to get the information we need?  You are probably thinking we run a query in MySQL and dump the data.  And that would also work.  But we need some kind of bash shell to run the processing against customer’s ID anyway.  So why not just extract the MySQL information we need at the same time.  The following script does just that.

It assumes a five column table, but can be easily modified for more or less:

CSV Data format:

------------------------------------------------------------------------------

Email, First Name, Last Name, Address, Birthdate

------------------------------------------------------------------------------

whoever@wherever.com, Ming, Nguyen, wherever, 4/6/1958

someoneelse@someother.com, John, Doe, somewhere, 4/6/1981

------------------------------------------------------------------------------

#!/bin/bash

# Processes customer Id’s via a CLI PHP Script

# 120406

PHP_DIR=/usr/bin

CLI_DIR=/home/user/site/public

AQCOMMAND=cli.queue-add.message

TQCOMMAND=cli.queue-run.listener

QUEUE_NAME=blog-test-queue

MTYPE=sendCustomerEmail

MESSAGE=customerId
USER=Your_MySQL_User
PASSWORD=Your_MySQL_Password

# Store file name

file="$1"

# make sure file exist and readable

if [ ! -f $file ]; then

   echo "$file : does not exist"

   exit 1

elif [ ! -r $file ]; then

   echo "$file: not readable"

   exit 2

fi

# read one line of file into $line variable

while read -r line

do

   # set the field separator to a “,” character

   # and read in five fields assigning each field

   # to $f1, $f2, $f3, etc for five fields  

   IFS=, read -r f1 f2 f3 f4 f5 <<<"$line"

   

     # execute a mysql select using $f1 (customer email)

     # to get the customer ID and assign it to $result

     # The “-N –r –B” prevents MYSQL from outputting header

     # information that we would have to strip out anyway

    result=$(

        mysql -N -r -B -u $USER –p$PASSWORD db_name << eof

           select customer_id from customer WHERE 

           email = '$f1'

        eof

    )

    # let’s echo it just so we can see our progress

    echo $result

       someCommand argument=$result;

    sleep 3

done <"$file"

exit 0

This method can be used to perform any kind of query against the database.  I have some examples of doing something similar from within a DOS batch file here.

Activating Zend Debugger via Mule

Mule can provide translation services between applications while acting as a common Enterprise Service Bus.   It’s very useful, but it’s also aptly named.  It’s stubborn, and it has the worst documentation, bar none, of any tool I have ever used.  

But all griping aside, the fact is, it’s sitting between two applications I need to communicate with each other.  And I want to debug the receiving application.  So how can I cause the receiving application to trigger Zend Debugger when it receives the data from Mule?  

The original mule configuration looks like this:

<http:outbound-endpoint

   host="receiving-application"

   port="80"

   path="index.php/catalog/updatePrice"

   contentType="text/xml"

   mimeType="text/plain"

   exchange-pattern="one-way"

/> 

We simply add the cookies into the header using a <properties> statement.  Note that a lot of the time, you will see endpoints defined as <http:outbound-endpoint host port etc/> with no closing tag because the "/" at the end of the tag closed it already.  In that case, just remove the trailing "/" and put a </http:outbound-endpoint> tag at the end as I have done here.

<http:outbound-endpoint

   host="receiving-application"

   port="80"

   path="index.php/catalog/updatePrice"

   contentType="text/xml"

   mimeType="text/plain"

   exchange-pattern="one-way"

> 

   <properties>

      <spring:entry key="cookies">

         <spring:map>

            <spring:entry key="start_debug" value="1" />

            <spring:entry key="debug_fastfile" value="1" />

            <spring:entry key="debug_host" value="127.0.0.1" />

            <spring:entry key="use_remote" value="1" />

            <spring:entry key="debug_port" value="10137" />

            <spring:entry key="original_url" value="http://receiving_application" />

            <spring:entry key="debug_stop" value="1" />

            <spring:entry key="debug_session_id" value="1005" />             

         </spring:map>

      </spring:entry>

   </properties

</http:outbound-endpoint>

Don’t forget to include the spring definitions at the top of the file:

xmlns:spring="http://www.springframework.org/schema/beans"

and:

http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd

  

If you need help setting up remote debugging in general, see my blog here: http://www.klugedeforce.com/2012/02/remote-debugging-cli-over-ssh-tunnel-in.html

I particularly like this method because it doesn’t require modification of the outgoing data in the sending application.  I have had problems with Mule stripping out the cookie information.  I am sure there is a way to convince Mule not to strip the cookies, but I haven't invested any serious effort in figuring it out because this method is generally easier.  And even if you were say polling an FTP server for a file and then posting the results as PHP, you can still intercept the incoming transaction this way.  So it's more universally applicable as well.

Codeigniter load->model and load->library ideosyncracies

Codeigniter's load model and load library commands both create Singletons, which are global in scope.  This can produce difficult to diagnose bugs if you don't realize it.

When you execute a $this->load->model('model_name', 'instance_name') , it invokes the "model" method in the CI_Loader class.  This method first checks that no other instance of the model exists with the same name.  If you do not give a name, then it uses "model_name" as the "instance_name".  This method stores the name of every model it has created in an array named "_ci_models".  If it does not find the name there, it then looks for it in $CI which is the super object you access via "$CI=& get_instance();". 

Because the model method executes an "if (isset($CI->name))" prior to instantiating the class, there cannot be any any object of any type anywhere within codeigniter that has the same name.  Hence it is global in scope.

And you cannot overwrite the assignment directly.  So let's say for example we are trying to make a "$Vendor_details" object.  And we want to assign one of two different address storage models to it based on the "$vendor_type".  So we create code something like the following.

switch ($vendor_type) {

case "domestic" :
$this->load->model('models/vendor_domestic', 'vendorModel');
break;
case "international" :
$this->load->model('models/vendor_international', 'vendorModel');
break;
...

And we try to access this model via "$this->vendorModel->someFunction".  Let's say the first time through, "vendorModel" is set to reference an instance of "Vendor_domestic", but the next time through, $vendor_type = "international".  We try to create an instance of  the "Vendor_international" model and assign it to $this->vendorModel.  We will get no warning, but we will in fact end up accessing the "Vendor_domestic" instance we created in the first pass, because "$this->vendorModel" cannot be reassigned, and it is pointing to the instance we first created. 

Once the name checks pass, the model method executes a "require_once" on the model name and then creates an instance of it references to either "instance_name" or the model name if no "instance_name" argument was passed in.   It also opens the database connection. 

The library load operates in a very similar way, but of course it does not open the database (which is probably already open anyway in most cases). 

If you need multiple copies of a model or library, you will need to instantiate them by including the referenced class in the top of the file with a:
require_once('path/modelName.php') in the top of the file

and instantiate the class with the standard:
$instance_name = new modelClassName();

Repairing Model 77 Skill Saw

Recently, my Mag 77 Skill saw suddenly stopped working.  I made a cut.  I put the saw down (gently), and when I picked it up, it sputtered for a second and stopped.  I hit the switch again.  It sputtered slightly again, and that was the last sign of life from it.  Luckily, I have another older saw, so I finished the project with that.

I was just going to buy another saw, but when I saw that the price tag on this saw model, which Skill still sells was $189, I figured it was worth the trouble of trying to fix it.

I framed houses for about three years when I was (much) younger.  We used these worm drive Skill saws exclusively.  They are beasts.  They can take ridiculous amounts of punishment.  I have seen them fail, but it usually comes on gradually and involves the triggers or brushes.  However, I use this saw so rarely, it seemed highly improbable that it would be either the trigger or the brushes.  Nevertheless, I inspected the brushes and tested the trigger, and you should too.  

Neither of these was the culprit here.  So I had to dig in and crack the case.  It’s actually quite easy to get into the motor on this saw.  The only caveat is it uses T20 torx screws.  I had a driver laying around, but if not, you are going to have to buy it.  There are some larger Torx screws on the case, but you don’t actually need to pull them in order to get to the motor.

Before you begin pulling the screws, you should remove the brushes if you have not already done so.  They are removed by unscrewing two black plastic plugs from each side of the case.  You can see a picture here of one of the brushes that has been removed.  Notice that I damaged the cap slightly in the process.  Use a screwdriver that fits well.  They are a bit delicate.

Notice that the brush is quite long.  This brush is barely worn.  If you use a saw heavily, you can wear out your brushes.  So if need be, you might have to replace yours.  You risk damaging the brushes if you pull the saw apart before you remove them, and you absolutely cannot put it back together without taking them out anyway as they are spring loaded and  will pop in and block the motor armature from going back in.

You need to pull these five screws (green arrows) to get the saw apart. 

You can see three of them here, and the other two are on the back side.  The four that go into the back of the case are very long.   Notice that small part marked with the blue arrow.  That fell out of somewhere at some point.  After studying the saw a bit, I decided it must be a tensioner for the motor armature.  I’ll show you where I put it when I was reassembling here in a second.

Once I got the saw apart, I saw the problem immediately.  There were two wires not connected to anything.  It was obvious they had been connected together, but had broken.  Unfortunately, it occurred to me to write this up after the fact, and I didn't take a picture of it.  They had been arcing to the case for some time before the saw failed.

If you look at the picture of the cooling fins, you can see the electrical arc damage on them.  

 This explains why this saw had been popping the GFI for about a year before it failed.  I had gotten in the habit of just plugging it inside because the outdoor GFI plug would just trip every time I tried to use it.

What happened was the manufacturer ran one of the supply wires to the forward end of the motor and connected it to the enamel coated wire from the windings.  They taped the two wires down with strapping tape.  The strapping tape failed after a few years, and the wires were long enough that they eventually pulled away from the windings far enough (or maybe got sucked into) the cooling fins.  When they started shorting to the fins, it caused the GFI to pop. And eventually the mechanical strain of hitting the cooling fins eventually broke the connection causing the saw to die.

The fix was very easy.  I cleaned up the enamel coated wire with some sandpaper, and I soldered it back onto the supply wire.  I then taped the connection with electrical tape, and then I taped that to the windings by threading through the electrical tape through the loops using a paperclip.  I put the saw back together, but I still wasn’t feeling too good about it.  Then it occurred to me that what it really needed was a more positive non-conductive fastener like a zip-tie (Panduit).  So I tore the saw back apart, this time not removing any unnecessary screws, and that’s when I also decided to photograph it.

The final work looks like this.  Notice that I didn't clamp the zip-ties directly on the wiring.  They are sitting on electrical tape.  I was a bit concerned about vibration causing them to damage the enamel on the windings.  I think it's probably better to not to tighten them down directly on the windings.  And I snugged them, but I didn't really zip them down as tight as I possibly could.  Maybe it would be better to just tape it.  I really wrestled with that decision before deciding to use the ties.  But I don't trust that electrical tape not to fail after a couple years.  And it's very hard to thread it through the windings.  So two turns was about all I could get per piece of tape.

I think that tensioner marked with the blue arrow above goes where the blue arrow is on this photo.  It's the only place that it fits.  Pay close attention when you pull your saw apart, and you might not have to guess at it.  Mine just fell out in the process.

The saw now works perfectly, and it no longer pops the GFI.

SVN: Merging Trunk into Branch

In my case, the repository in question is on Jira.  So instead of using svn+ssh://, I use https:// to access the repository.  I am assuming you already have a branch in your local directory, and you want to merge trunk into it.  This is a good practice.  Otherwise, if you have CSS files with revisions for instance, it's quite possible that both you and another developer might mod those files and advance the revision.  This creates a "local add, incoming add upon merge" tree conflict that subversion doesn't know how to resolve forcing you to resolve it manually.  This sort of problem cannot be completely avoided, but at least you can pick up any changes that have already been put back before you alter the files.  Over the course of a long project, you can end up very far behind trunk with a big job to merge if you don't do this occasionally.

Before you start this process, remember to do a commit of your branch picking up any changes that have made.  This keeps the log files clean when you commit the trunk merge, and it allows you to completely roll back the changes in case something goes horribly wrong.  Assuming you have already committed your branch, proceed as follows:

Check out a copy of trunk if you don't already have one.  Otherwise, just cd to the root of it and proceed to step 2.

          1.  svn co https://server/path/to/project/trunk

cd to the root directory of trunk

          2.  svn up

this will already be up to date if you just checked it out in step one, but note down the revision number (YYYY) it spits out.

cd to your branch root directory

          3.  svn log --stop-on-copy

This will display all the revision that have occurred since this branch was cut from the main one.  Find the lowest numbered revision (the oldest one), and make a note of it (XXXX).

Now execute the merge with the following:

           4.  svn merge --dry-run -r XXXX:YYYY https://server/path/to/project/trunk

Note the "--dry-run".  That's completely optional of course, but if you think there may be conflicts, it's not a bad idea to get an idea of what they may be before you start.  

once you are happy with that, run it for real:

           4.  svn merge -r XXXX:YYYY https://server/path/to/project/trunk

resolve all the conflicts.

           5. svn commit -m "Merged trunk into branch"