Sunday, November 24, 2013

#define logging class for C++

The intent of this class is to lookup the identifier (the GET_USER_DATA part) by the token string (the value the text part gets converted to).

So if you have:

#define                        GET_USER_DATA      0x1FD02000

Given, “0x1FD02000”, I need to find “GET_USER_DATA”.  I tried to find some elegant want to do this, but there simply is not any possible way to do this.  The compiler replaces those identifiers at compile time.  And the information is simply not in the finished program. 

I wanted to do this to assist in debugging a multi-part networked application, for which I was not the original author.  The various application servers send data packets back and forth with the message type as the first DWORD in the packet.  I printed out the “dwMsgId” hex values from the packets, each of which corresponded to some identifier in the messages.h file, but there were so many message types that I couldn’t remember them all.  So I had to constantly keep doing searches on the Hex values to look them up and see what I was looking at.  So I tried to find some more readable way of logging these values.

One easiest solution is something like this:

switch (dwMsgId) {
   case GET_USER_DATA:
      strcpy (cText, "GET_USER_DATA");
      ...
      break;
   case NOTIFY_USER_LOG:
      strcpy (cText, "NOTIFY_USER_LOG");
      ...
      break;

But you can't swap that in and out easily between the debug and release versions, and it's just generally messy when you start talking about 80+ message types.  So instead, I decided to make a map based logging class.  And I figured I'd write a perl script to parse the message.h file to get the #defines and format them to fit the map class.  With that method, we get a call in the target application that looks something like:

#ifdef _DEBUG_XSOCKET
            wsprintf (cHex, "0X%.8X", * dwMsgId);
            cText = m_pLogXSock->getText(cHex);
            wsprintf(cLogMsg, "Message Received (0x%.8X) : %s", dwMsgId, cText );
#endif

Using this, we can set the value "_DEBUG_XSOCKET" in the preprocessor directives for the debug version of the project so the block gets automatically swapped in and out depending on the build type.

The actual class to support this follows.  Because the program I was trying to debub is an old-school C program, it uses character pointers for everything rather than std::string.  The std::map class does not support arguments of type *char by default.  So I defined a custom comparison function.  One thing to note here is that I tried to use the "unordered_map" library first.  I didn't really check it that closely, and I started getting a weird linker error about something to do with a single argument function. I then recreated this class using std::string.  But that didn't fix the problem.  So finally I broke down and read through the MSDN docs on the function.  The unordered_map library does not support customer comparison functions (i.e. <map> and <unordered_map> have different signatures).  So in the end, I went with the <map> version and swapped this back to the *char version.

The map class "find()" function either returns an iterator to the target entry, or it sends back something that is effectively an invalid pointer.  If you try and access a nonexistent entry with "whatever = it->second", it will compile fine, but it will throw an exception and crash your program when you try and use it.  So you have to compare the returned value to "tMap.end()" before you try and use it to access anything.

LogXSock.h
================
#if !defined(AFX_LOGXSOCK_H)
#define AFX_LOGXSOCK_H

#include <windows.h>
#include <stdio.h>
#include <stdlib.h>
#include "Messages.h"
#include <map>

class CLogXSock 
{
public:
       char * cGetValue(char * cXkey);    
       CLogXSock();
       virtual ~CLogXSock();

       char * cEmpty;

private:
       struct cmp_str : std::binary_function<char *, char *, bool> {
        bool operator() (char *a, char *b)  {
           return std::strcmp(a, b) < 0;
        }
    };

    std::map<char *, char *, cmp_str> tMap;
    std::map<char *, char *, cmp_str>::iterator it;
      
};

#endif // !defined(AFX_LOGXSOCK_H)

LogXSock.cpp
================
#include "LogXSock.h"

CLogXSock::CLogXSock()
{
       tMap["0X1DF00013"] = "MSGID_SERVERSTOCKMSG";
       tMap["0X1DF4D009"] = "MSGID_ITEMCREATE";
       tMap["0X1DF23500"] = "MSGID_REQUEST_REGISTERSERVER";
       tMap["0X1DF00000"] = "MSGID_SENDSERVERSHUTDOWNMSG";
       tMap["0X1DF10002"] = "MSGID_SERVERALIVE";

       cEmpty = new char[2];
       strcpy(cEmpty, "");
}

char * CLogXSock::cGetValue (char * cXkey) {
       it = tMap.find(cXkey);
       if (it != tMap.end()) {
              return(it->second);
       } else {
              return (cEmpty);
       }
}

CLogXSock::~CLogXSock()
{

}


And lastly, here is the perl script I used to find the defines and pull out the values:

extract_define_from_header.pl
========================
#!usr/bin/perl

use File::Basename;
#use Switch;
$outputFile = "map_defines.txt";
$open = "UPDATE country SET country_code = '";
$middle = "' WHERE  country_name = '";
$end = "';";
$num_args = $#ARGV + 1;

$scriptName = fileparse($0, ".pl"); #get the base name of the calling script

if ($ARGV[0] eq "-h" or $ARGV[0] eq "--h") {

  print"This script ingests a country_code file and writes it out'\n";
  print"\n";
  print"usage: $scriptName.pl input.txt\n";
  print"usage: $scriptName.pl input.txt output.txt\n";

  print"\n";
  exit;
}
if ($num_args < 1) {
   die ("Must pass the input file as the first argument.  You passed nothing.\nTry 'perl $scriptName.pl -h' for help.\nExiting...\n");
}
if ($num_args > 0) {
  $inputFile = $ARGV[0];
}
if ($num_args > 1) {
  $outputFile = $ARGV[1];
}
 
open INFILE, "$inputFile" or die "Failed to open inputFile $inputFile; Exiting...\n";
open OUTFILE, ">$outputFile" or die "Failed to open outputFile $outputFile for writing; Exiting...\n";

while ($line = <INFILE>) {
   chomp ($line);
   @split = split(/\s+/,$line);
   $define = $split[0];
   if ($define eq "#define") {
       $name = $split[1];
       $value = $split[2];
       $length = @split;
        print (OUTFILE "tMap[\"".uc($value)."\"] = \"" . uc($name) . "\";\n");
   }
}
close (INFILE);
close (OUTFILE);



Run the perl script as:

perl extract_define_from_header.pl   targetDefineFile.h outputFile.txt

Then copy the results from "outputFile.txt" and paste them into the constructor for LogXSock.cpp.

In practical application, I moved the logging into the class as well so that it reduces the footprint in the target application.  But I've found that tends not to be very portable.  So this remains the base class for this type of logging functionality.  If you use strings instead of chars, then you can just use the built in comparison functionality rather than a custom comparison function, and you can use <unordered_map> which is faster.  We don't need any type of sorting in this application, but I'm guessing, and it is a guess, that the std::map using the target application’s native *char arguments is going to be faster than a std::string based function where we have to translate back and forth, but where we can use <unordered_map> rather than <map>.  But regardless, I also have a string based version of this in my toolbox as well.




Thursday, July 18, 2013

Fix Magento catalog_product_flat Indexer Stuck on Processing

Occasionally, the magento “catalog_product_flat” indexer will hang.  Recently, this happened to a store I was working on that had several hundred thousand products with 225 catalog_product_flat_x entries. 

The fix for this problem is normally to BACKUP THE DATABASE, truncate all the “catalog_product_flat_x” tables, and then reindex the catalog product entries.

However, I am not so hot to manually truncate over 200 tables by hand.  So I used a bash script along with a configuration file.

One thing interesting to note here, I flat out could not get “mysql” on my Mac (installed with Zend server) to recognize the “mysql –defaults-extra-file=whatever” command.  It works fine on the AWS boxes, but on my Mac, it just keeps giving the error:

/usr/local/zend/mysql/bin/mysql.client: unknown variable 'defaults-extra-file=magento.cnf'

which is pretty annoying.  And yes, it was the first argument after the “mysql” command.  I created a SQL script to create a dummy local magento database with a couple “catalog_product_flat_x” tables as well as some other differently named tables to test it on.  But I ended up having to test it on an AWS dev server because of this.

Giving credit where it is due, I modified a bash script from here: https://gist.github.com/marcanuy/5977648

The modified script is:

#!/bin/bash
# Truncate database tables from console
DATABASE=magento
DEFAULTS_FILE=$DATABASE.cnf

mysql --defaults-extra-file="$DEFAULTS_FILE" -Nse 'SELECT table_name FROM information_schema.tables WHERE table_schema = "magento" AND table_name like ("catalog_product_flat_%")' $DATABASE | while read table; do mysql --defaults-extra-file="$DEFAULTS_FILE" -e "truncate table $table" $DATABASE; done

And the config file which needs to be named “magento.cnf”, looks like:
[client]
host=localhost
user=magento
password=magento

where of course you have to substitute the credentials for your mysql server in place of the test credentials above.

Then finally, go to the Magento docroot directory, and re-run the indexer:


php shell/indexer.php --reindex catalog_product_flat

Sunday, June 9, 2013

Bash Download of M3U MP3 Playlist


I have a list of MP3’s I regularly listen to which are all pulled from a website via a “*.m3u” list.  My wife wanted the songs, but she wants to listen to them on her IPhone.  So I whipped up this Bash script to take a “*.m3u” file as an argument, and pull all the tracks in the playlist down to the current directory.

If you look at the m3u playlist in Itunes, you may see some streaming files that are of duration “continuous”.  Those will not download correctly using this method.

#!/bin/bash

# Takes a list of MP3's (generally an m3u file) and downloads all to current directory
# www.klugedeforce.com
#
if [ $# != 1 ] ; then
  echo "usage $0 mp3_list_file.txt"
  echo "file list can have any extension, but should contain list of files one per line:"
  echo ""
  echo "http://downloads.hpmgl.net/mp3song_file1.mp3"
  echo "http://downloads.hpmgl.net/song_file2.mp3"
  echo "http://downloads.hpmgl.net/song_file3.mp3"

elif [ -e $1 ] ; then
   IFS=$'\r\n'
   for line in $(cat $1); do
       trimmed_line="$(echo ${line} | sed 's/^[ \t\r\n]*//g' | sed 's/[ \t\r\n]*$//g')"
       # trimmed_line=$line | sed 's/^[ \t\r\n]*//g' | sed 's/[ \t\r\n]*$/b/g'
       outfile=${trimmed_line##http://*/}
       if [ -n "$outfile" ] ; then
       `curl "${trimmed_line}" > "${outfile}"`;
          echo $outfile;
       fi
   done
else
   echo "Filename $1 not found"
fi          


Wednesday, June 5, 2013

Magento XMLRPC Credit Memo List with Filter


Magento XMLRPC Credit Memo List with Filter

There are some examples of using credit memos on the Magento API documentation web page:


But the only examples there are for the SOAP implementation.  However, the Magento API also supports the XMLRPC protocol. 

The following code works as is for the Zend_XmlRpc_Client, but the key thing is the parameter formatting, which is not platform specific.

$params = array( array(‘order_id’ => ‘12345678’),
                  array(‘whatever_field’ => $target_val),
                  array(‘another_field’ => $another_val),
          );

$this->client = new Zend_XmlRpc_Client($apiUrl);

$session = $this->client->call('login', array($this->apiUser, $this->apiKey));

$response = $this->client->call('call',
                 array($session, 'sales_order_creditmemo.list', $params));

Saturday, April 20, 2013

Bypass Kinetico Plus Vx Gx MACguard PureMometer





Luckily, it has a rubber o-ring in it, so it’s very easy to put back provided you don’t do something foolish and damage it.  After studying it a bit, I came to the conclusion that there must be something in the center hole of the final filter cartridge which pushes the plastic pin from the PureMometer up.  I tried putting a wooden skewer down the center hole of the cartridge.  I hit some slight resistance, and then I punched through.  When I put the filter back on, instead of a trickle, we had virtually nothing.  So it seems I made it worse.

The mechanism looks like this internally:


I think this thing works as follows.  As the water comes in from the tank, it passes over the water wheel.  The water wheel has a gear on it which hits the large gear on the top of the nearest gear stack.  That in turn has a smaller gear on it that hits the top large gear on the opposite gear stack.  And they ping pong back and forth downwards over three sets of gears.  Finally the bottom gear on the opposite stack drives the white gear in the foreground of the picture and spins the blue PureMometer valve around…very, very slowly.  The blue PureMometer valve is spring loaded.  When nothing is pressing up on it, it comes down and cuts the water flow off nearly completely. 

So the blue PureMometer pin is spring loaded and it turns.  The bottom of it is also fluted and shaped a bit like a drill bit.  I think it is pressing down on something or some substance in the center hole of the filter, and it slowly drills through it over time.  Once it gets through, the spring pushes the blue pin all the way down and no more water.

So what we need is something to press that blue pin back up into the housing and allow the water to flow.  After looking around for a bit, I came up with a simple paperclip bent as follows:



I then pressed it down into the top of the filter as shown here:





I was concerned that drill end of the spinning PureMometer pin might catch on the paperclip, and with that massive gear reduction, it could easily snap off that rather spindly pin.  So I also pulled the center gear out so the water wheel motion could not transfer to the blue pin. 



Now the blue pin is pushed up by the paperclip, and it is no longer able to spin.  I think that with a new cartridge, even in the absence of the paperclip, removing the gear should keep the drill bit from spinning, and the cartridge wouldn’t wear out, but because I don’t have the new cartridge, I can’t inspect it closely enough to be sure.  As you can see, the paperclip fully resets the pin.



Here is a shot of the faucet.  It’s running like a champ.



So I successfully bypassed it.  Now what about the water quality?  It stands to reason that the filter is designed to clean 500 gallons of pretty much any household water.  We already have a salt-based whole house Kinetico softener/filter system ahead of this filter anyway.  And we’ve got fairly decent city water.  Without the whole house filter, the water smells of chlorine, but otherwise, it’s fine.  And it’s just statistically improbable we’d have the worst case water in the nation.  We likely have fairly average water.  If you figure this filter can handle 500 gallons of worst case water, it should not have a problem with 1000 gallons of ours.  So I figure to extend this from three months to six months.  But now you do need to keep an eye on it since it will never shut off on its own. 

I bypassed this because I suspect this rather clever mechanism is almost surely for the benefit of Kinetico, not really for me, the consumer.  It probably easily doubles the filter revenue they take in.  Once the quality of the water noticeably deteriorates, then I will drop the $90 on a new filter at my discretion.  I don’t feel inclined to let Kinetico force me to do it before I think it’s necessary.

Your mileage may vary, and it’s up to you to make sure your water is still clean should you choose to bypass this shutoff mechanism.  I take no responsibility for you, your actions, Kinetico’s revenue, etc.  So proceed with care and good luck.