Sunday, November 24, 2013

#define logging class for C++

The intent of this class is to lookup the identifier (the GET_USER_DATA part) by the token string (the value the text part gets converted to).

So if you have:

#define                        GET_USER_DATA      0x1FD02000

Given, “0x1FD02000”, I need to find “GET_USER_DATA”.  I tried to find some elegant want to do this, but there simply is not any possible way to do this.  The compiler replaces those identifiers at compile time.  And the information is simply not in the finished program. 

I wanted to do this to assist in debugging a multi-part networked application, for which I was not the original author.  The various application servers send data packets back and forth with the message type as the first DWORD in the packet.  I printed out the “dwMsgId” hex values from the packets, each of which corresponded to some identifier in the messages.h file, but there were so many message types that I couldn’t remember them all.  So I had to constantly keep doing searches on the Hex values to look them up and see what I was looking at.  So I tried to find some more readable way of logging these values.

One easiest solution is something like this:

switch (dwMsgId) {
   case GET_USER_DATA:
      strcpy (cText, "GET_USER_DATA");
      ...
      break;
   case NOTIFY_USER_LOG:
      strcpy (cText, "NOTIFY_USER_LOG");
      ...
      break;

But you can't swap that in and out easily between the debug and release versions, and it's just generally messy when you start talking about 80+ message types.  So instead, I decided to make a map based logging class.  And I figured I'd write a perl script to parse the message.h file to get the #defines and format them to fit the map class.  With that method, we get a call in the target application that looks something like:

#ifdef _DEBUG_XSOCKET
            wsprintf (cHex, "0X%.8X", * dwMsgId);
            cText = m_pLogXSock->getText(cHex);
            wsprintf(cLogMsg, "Message Received (0x%.8X) : %s", dwMsgId, cText );
#endif

Using this, we can set the value "_DEBUG_XSOCKET" in the preprocessor directives for the debug version of the project so the block gets automatically swapped in and out depending on the build type.

The actual class to support this follows.  Because the program I was trying to debub is an old-school C program, it uses character pointers for everything rather than std::string.  The std::map class does not support arguments of type *char by default.  So I defined a custom comparison function.  One thing to note here is that I tried to use the "unordered_map" library first.  I didn't really check it that closely, and I started getting a weird linker error about something to do with a single argument function. I then recreated this class using std::string.  But that didn't fix the problem.  So finally I broke down and read through the MSDN docs on the function.  The unordered_map library does not support customer comparison functions (i.e. <map> and <unordered_map> have different signatures).  So in the end, I went with the <map> version and swapped this back to the *char version.

The map class "find()" function either returns an iterator to the target entry, or it sends back something that is effectively an invalid pointer.  If you try and access a nonexistent entry with "whatever = it->second", it will compile fine, but it will throw an exception and crash your program when you try and use it.  So you have to compare the returned value to "tMap.end()" before you try and use it to access anything.

LogXSock.h
================
#if !defined(AFX_LOGXSOCK_H)
#define AFX_LOGXSOCK_H

#include <windows.h>
#include <stdio.h>
#include <stdlib.h>
#include "Messages.h"
#include <map>

class CLogXSock 
{
public:
       char * cGetValue(char * cXkey);    
       CLogXSock();
       virtual ~CLogXSock();

       char * cEmpty;

private:
       struct cmp_str : std::binary_function<char *, char *, bool> {
        bool operator() (char *a, char *b)  {
           return std::strcmp(a, b) < 0;
        }
    };

    std::map<char *, char *, cmp_str> tMap;
    std::map<char *, char *, cmp_str>::iterator it;
      
};

#endif // !defined(AFX_LOGXSOCK_H)

LogXSock.cpp
================
#include "LogXSock.h"

CLogXSock::CLogXSock()
{
       tMap["0X1DF00013"] = "MSGID_SERVERSTOCKMSG";
       tMap["0X1DF4D009"] = "MSGID_ITEMCREATE";
       tMap["0X1DF23500"] = "MSGID_REQUEST_REGISTERSERVER";
       tMap["0X1DF00000"] = "MSGID_SENDSERVERSHUTDOWNMSG";
       tMap["0X1DF10002"] = "MSGID_SERVERALIVE";

       cEmpty = new char[2];
       strcpy(cEmpty, "");
}

char * CLogXSock::cGetValue (char * cXkey) {
       it = tMap.find(cXkey);
       if (it != tMap.end()) {
              return(it->second);
       } else {
              return (cEmpty);
       }
}

CLogXSock::~CLogXSock()
{

}


And lastly, here is the perl script I used to find the defines and pull out the values:

extract_define_from_header.pl
========================
#!usr/bin/perl

use File::Basename;
#use Switch;
$outputFile = "map_defines.txt";
$open = "UPDATE country SET country_code = '";
$middle = "' WHERE  country_name = '";
$end = "';";
$num_args = $#ARGV + 1;

$scriptName = fileparse($0, ".pl"); #get the base name of the calling script

if ($ARGV[0] eq "-h" or $ARGV[0] eq "--h") {

  print"This script ingests a country_code file and writes it out'\n";
  print"\n";
  print"usage: $scriptName.pl input.txt\n";
  print"usage: $scriptName.pl input.txt output.txt\n";

  print"\n";
  exit;
}
if ($num_args < 1) {
   die ("Must pass the input file as the first argument.  You passed nothing.\nTry 'perl $scriptName.pl -h' for help.\nExiting...\n");
}
if ($num_args > 0) {
  $inputFile = $ARGV[0];
}
if ($num_args > 1) {
  $outputFile = $ARGV[1];
}
 
open INFILE, "$inputFile" or die "Failed to open inputFile $inputFile; Exiting...\n";
open OUTFILE, ">$outputFile" or die "Failed to open outputFile $outputFile for writing; Exiting...\n";

while ($line = <INFILE>) {
   chomp ($line);
   @split = split(/\s+/,$line);
   $define = $split[0];
   if ($define eq "#define") {
       $name = $split[1];
       $value = $split[2];
       $length = @split;
        print (OUTFILE "tMap[\"".uc($value)."\"] = \"" . uc($name) . "\";\n");
   }
}
close (INFILE);
close (OUTFILE);



Run the perl script as:

perl extract_define_from_header.pl   targetDefineFile.h outputFile.txt

Then copy the results from "outputFile.txt" and paste them into the constructor for LogXSock.cpp.

In practical application, I moved the logging into the class as well so that it reduces the footprint in the target application.  But I've found that tends not to be very portable.  So this remains the base class for this type of logging functionality.  If you use strings instead of chars, then you can just use the built in comparison functionality rather than a custom comparison function, and you can use <unordered_map> which is faster.  We don't need any type of sorting in this application, but I'm guessing, and it is a guess, that the std::map using the target application’s native *char arguments is going to be faster than a std::string based function where we have to translate back and forth, but where we can use <unordered_map> rather than <map>.  But regardless, I also have a string based version of this in my toolbox as well.




Thursday, July 18, 2013

Fix Magento catalog_product_flat Indexer Stuck on Processing

Occasionally, the magento “catalog_product_flat” indexer will hang.  Recently, this happened to a store I was working on that had several hundred thousand products with 225 catalog_product_flat_x entries. 

The fix for this problem is normally to BACKUP THE DATABASE, truncate all the “catalog_product_flat_x” tables, and then reindex the catalog product entries.

However, I am not so hot to manually truncate over 200 tables by hand.  So I used a bash script along with a configuration file.

One thing interesting to note here, I flat out could not get “mysql” on my Mac (installed with Zend server) to recognize the “mysql –defaults-extra-file=whatever” command.  It works fine on the AWS boxes, but on my Mac, it just keeps giving the error:

/usr/local/zend/mysql/bin/mysql.client: unknown variable 'defaults-extra-file=magento.cnf'

which is pretty annoying.  And yes, it was the first argument after the “mysql” command.  I created a SQL script to create a dummy local magento database with a couple “catalog_product_flat_x” tables as well as some other differently named tables to test it on.  But I ended up having to test it on an AWS dev server because of this.

Giving credit where it is due, I modified a bash script from here: https://gist.github.com/marcanuy/5977648

The modified script is:

#!/bin/bash
# Truncate database tables from console
DATABASE=magento
DEFAULTS_FILE=$DATABASE.cnf

mysql --defaults-extra-file="$DEFAULTS_FILE" -Nse 'SELECT table_name FROM information_schema.tables WHERE table_schema = "magento" AND table_name like ("catalog_product_flat_%")' $DATABASE | while read table; do mysql --defaults-extra-file="$DEFAULTS_FILE" -e "truncate table $table" $DATABASE; done

And the config file which needs to be named “magento.cnf”, looks like:
[client]
host=localhost
user=magento
password=magento

where of course you have to substitute the credentials for your mysql server in place of the test credentials above.

Then finally, go to the Magento docroot directory, and re-run the indexer:


php shell/indexer.php --reindex catalog_product_flat

Sunday, June 9, 2013

Bash Download of M3U MP3 Playlist


I have a list of MP3’s I regularly listen to which are all pulled from a website via a “*.m3u” list.  My wife wanted the songs, but she wants to listen to them on her IPhone.  So I whipped up this Bash script to take a “*.m3u” file as an argument, and pull all the tracks in the playlist down to the current directory.

If you look at the m3u playlist in Itunes, you may see some streaming files that are of duration “continuous”.  Those will not download correctly using this method.

#!/bin/bash

# Takes a list of MP3's (generally an m3u file) and downloads all to current directory
# www.klugedeforce.com
#
if [ $# != 1 ] ; then
  echo "usage $0 mp3_list_file.txt"
  echo "file list can have any extension, but should contain list of files one per line:"
  echo ""
  echo "http://downloads.hpmgl.net/mp3song_file1.mp3"
  echo "http://downloads.hpmgl.net/song_file2.mp3"
  echo "http://downloads.hpmgl.net/song_file3.mp3"

elif [ -e $1 ] ; then
   IFS=$'\r\n'
   for line in $(cat $1); do
       trimmed_line="$(echo ${line} | sed 's/^[ \t\r\n]*//g' | sed 's/[ \t\r\n]*$//g')"
       # trimmed_line=$line | sed 's/^[ \t\r\n]*//g' | sed 's/[ \t\r\n]*$/b/g'
       outfile=${trimmed_line##http://*/}
       if [ -n "$outfile" ] ; then
       `curl "${trimmed_line}" > "${outfile}"`;
          echo $outfile;
       fi
   done
else
   echo "Filename $1 not found"
fi