Zend Debugger with HTTP Post and SOAP

In previous blogs, I have discussed methods for doing remote debugging vi SSH and Zend built-in tunneling as well as for debugging CLI driven functions with Zend Debugger by setting environment variables.  But there sometimes comes the case that you need to debug a SOAP or HTTP post initiated transaction.

The way to do this is by putting cookies in the header of the incoming http transaction that tell Zend to trigger the debug session.  I generally do most of my system development on an AWS instance.  So I also use remote debugging in conjunction with this.   The other day, I did a writeup on how to get SSH tunneling setup.  This is the same method you would use to remotely debug here.  So if you need to get an SSH tunnel setup for remote debugging, please refer to my other blog post here and get your tunnel working before proceeding.

Assuming you have your remote debug environment setup or are working locally, we now want to proceed to debug an HTTP post or SOAP initiated transaction.  Generally, I use CURL to setup my transactions.   So I would produce code something like the following:

$debug_cookie = "start_debug=1; debug_fastfile=1; debug_host=127.0.0.1; use_remote=1; debug_port=10137; original_url=http://test-server; debug_stop=1; debug_session_id=1005;";

curl_setopt($ch, CURLOPT_COOKIE, $debug_cookie);

and inject it into my curl creation flow.  if you use some other method of setting up your transactions, that's fine.  At the end of the day, you need to produce a line in your headers that looks something like:

Cookie: start_debug=1; debug_fastfile=1; debug_host=127.0.0.1; use_remote=1; debug_port=10137; original_url=http://test-server; debug_stop=1; debug_session_id=1005;

And however you go about doing it is fine as long as the header actually gets passed in with the cookie in it.  

I defined a PHP server for "http://test-server", setup appropriate path mapping, and set it as the default server.  I am not 100% sure how Zend goes about resolving path mapping as to whether this is necessary or not, but it worked correctly.  So if you are having path mapping problems, it's worth a shot to try that and see if it helps.  

Remote Debugging CLI over SSH Tunnel in Zend Studio

I had trouble getting the path mapping to work on this.  I originally posted the blog entry last night stating I could not get it working.  This morning, I got it working, but the problem is I can't get it to stop working now, so I don't know exactly what fixed it.

The changes I made were: 1) I added an "original_url=http//test-server" to the QUERY_STRING environment variable.  I then created a new server under Studio Admin | Preferences | PHP DEBUG | PHP Servers | New.  I put "http://test-server" as the URL of that points to the document root of this server (which it actually is).  I then created a path mapping all the way down to the public directory.  Previously, my path mapping looked like this:

/home/server_name/site/current/releases/bootstrap => ProjectRoot

I left that in place, but I added another one with:
/home/server_name/site/current/releases/bootstrap/public => ProjectRoot/public

Once I did this, my path mapping suddenly started working.  However, I also closed out Zend Studio last night and reopened it this morning, and I didn't try to run this.  So maybe resetting studio captured some setting that wasn't sticking.  I just don't know.  And since I can't break it now, I can't figure out what the critical parameter is.  So best bet is to just follow all of the above.

ORIGINAL BLOG CONTENT FOLLOWS:

For remote debugging in general, the SSH tunnel works very well.  I prefer it to the so-called "tunneling" option native to Zend Studio, which I discussed in a another blog recently here.

This discussion assumes that you already have a working SSH connection to the remote server you want to debug on.

ESTABLISH A TUNNEL
Execute: SSH -R 10137:127.0.0.1:10137 user@remote-server

This will log you into the remote server.  Verify that this is working by running:
netstat -tln | grep 10137

on the remote server.  It should come back with something very similar to the following:


tcp        0      0 127.0.0.1:10137         0.0.0.0:*               LISTEN    
tcp6       0      0 ::1:10137               :::*                    LISTEN


If you don't get this, then maybe the user account you are using doesn't have adequate access to set this up.  It doesn't need all root privileges, but it does need some.  It's very possible that the account on which your application is setup does not have adequate access, and you need to tunnel in as "ubuntu" or the equivalent for your OS.

You need to keep this terminal session open as long as you are communicating with the remote machine.  I created a very simple bash file named "tunnel" with one argument "user@remote-server" to do this so I can easily shell into various machines.


#!/bin/bash
# Opens tunnel for use by Zend Studio

SSHCOMMAND="SSH -R 10137:127.0.0.1:10137"

echo "$SSHCOMMAND $1"
$SSHCOMMAND $1

I am also assuming you already have Zend Debugger setup on the remote machine.  I have some decent instructions for doing this in this blog: http://www.klugedeforce.com/2012/02/remote-debugging-over-tunnel-in-zend.html.  So I won't repeat them in detail here.  You need to put the "ZendDebugger.so" on the server, put "dummy.php" in the base directory, and configure the "php.ini" file or a "conf.d/*.ini" with the needed settings.  Make sure 127.0.0.1 is in the allowed hosts list.  You don't need the tunnel entry in the .ini at all, but having it won't hurt.


Go to Zend Studio | Preferences | PHP | Debug | Debug Servers.  Select the Zend Debugger entry and click "Configure".  Set "Debug Port: 10137" and set "Client Host/IP: 127.0.0.1"

Create a remote server with the address of the remote server as "http://test-server" or something along those lines.  If the remote server has a real address, use it.   Just be sure you set the identical address in the "original_url=http://test-server" in the QUERY_STRING environment variable (discussed later).  Set this new server to default.

To test you settings, you can go to Run | Debug Configurations and setup a debug session using the server you configured above (or just use any one you already have setup).  Click "test debugger".  It should give you a success message.  This uses dummy.php so it must be in the root html directory (probably */public) for this to work.  At this point, you can debug the remote server if you choose in a normal way.  As long as that terminal with the SSH session is open, all your Zend communication will route through it, and you can remote debug with no issues.  Disable the tunnel (i.e. exit from it), and change the server to a local one, and you can debug the local copy as well with the same settings.

I tried the following CLI debugging with the built in "tunnel" option, but it didn't even think about working, which is why I decided to try the SSH tunnel option.

DEBUGGING REMOTE CLI CALLS
You need to configure an Environment Variable named "QUERY_STRING" in order to debug command line PHP calls.  Zend debugger will pick up that QUERY_STRING and stop the session accordingly.  You can either set it when you call the script or export it from a BASH file (my preferred method).  I generally use a bash file to kick off the CLI processes and events and pass in any variables from the command line.

So I might end up calling something like:
X-Devent newPurchaseOrder 3048272 DEBUG, which will cause a PHP call to a cli.php file.  And before I do that, I can easily have:

export QUERY_STRING="start_debug=1&debug_fastfile=1&debug_host=127.0.0.1&use_remote=1&debug_port=10137&debug_stop=1&original_url=http://test-server&debug_session_id=1002"

at the top of the bash file that sets the QUERY_STRING environment variable a-priori.

Thanks to the "debug_stop=1", the code execution will stop in the debugger at the first line in the first PHP files it hits.  "use_remote=1" is taken from the server's perspective.  It actually means use the local copies of the files from Zend Studio (so you can set breakpoints).

I struggled to find the right permutations to get this to work, but now that it does, it's a great capability to have.  I can now debug queue events by intercepting the handler calls.

Remote Debugging Over Built-in "Tunnel" in Zend Studio v7.2

The goal here is to setup remote debugging on an AWS EC2 instance or any other machine which restricts outgoing ports using Zend Studio's tunneling option.  I set this up on v7.2  It's likely similar on the newer versions as well but could vary because Zend loves to move configuration settings around between versions.

This does not use SSH tunneling.  It tunnels the data back out over port 80.  So it does not require much of any sort of access to the web server once you get the web server configured (or if you build the configuration into your deployment scripts).  I had a hard time getting this working.  This is because of the allow_hosts and allow_tunneling configuration.  So pay special attention to getting that configured right.

Also, the tunnel connection seems to time out occasionally even though it shows it as still connected.  So if your debug session suddenly won't work, try resetting it. I also have to say this runs painfully slow, but it's better than nothing.  I need to open up the remote ports and get an apples to apples speed comparison.  Maybe remote debugging is just inherently slow.  Local debug definitely runs substantially, maybe 4x or 5x faster than this.

CONFIGURE THE TUNNEL SERVER IN ZEND STUDIO

Click "Zend Studio" in the menu bar and select "Preferences"

Go to the PHP section and expand it.  Look for PHP Servers and select it.

This brings up a dialog box with a couple menu buttons on the right side.  Select "New".

Give the server a name "NewServerName" and a URL (http://test-server)

Click "Next"

Setup the server path mapping:

Path on Server:

/site/current

Path in Workspace:

/ExchangeISIS

Click Next

I'm not running Zend Server so I did not select Zend Server Integration.  

So just click Next

Now we finally come to the Tunneling Settings screen

Tick the check box for "Enable Tunneling"

Click "Finish" to complete setup

Hit OK to get out of the preferences Dialog box

CONFIGURE THE SERVER

Now you have to setup the server to actually support debugging assuming you don't already have it in the image which you probably don't or you wouldn't be reading this right now.

SO: You need to download the debug DLL extensions from Zend Studio at:

http://www.zend.com/en/products/studio/downloads

If you are not sure what version of Linux you are running on the server do a:

file /sbin/init

This will produce something like:

/sbin/init: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), ...

or

/sbin/init: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), ...

 and obviously, if it says 32, then you are running a 32 bit OS so get the 32 bit version of the 

At this point, you have to register with Zend to complete the download (annoying, yes I know):

Note that the download page has tabs on it, and it will default to the OS you are running.  But that's probably not the OS you want to run it on.  So make sure you get the correct file.  I got an ELF error the first time I tried to run this (visible in the /var/logs/apache2/error.log".  I did a "file ZendDebugger.so", and it said "Mach-O" something or other.  That's a MAC format...because my work development machine is a MAC.  Oops.

Upload the file to the server.  Gunzip it.  tar -xvf the tar file.  This should produce a directory similar to:

ZendDebugger-20110410-linux-glibc23-i386

Navigate into this directory.  You will see multiple directories named for various PHP versions, a README.txt and a dummy.php file in this directory.

Select the proper PHP file  

Do a "php -v" from the command line on the remote server.

This will tell you what version of PHP you are running.  Select the appropriate Zend Debugger version.

In my case, I got:

PHP 5.3.2

So I took the ZendDebugger.so from the "5_3_x_comp" directory.

Move the file to a suitable location

I usually use: /usr/local/lib/zend (I have to create the zend directory)

If you don't know where the php.ini file is on the system in question, you can use phpinfo (detailed below) to find it.  I got smart here and initially did a: php -i | grep php.ini and got an /etc/php5/cli/php.ini  (uh...NO!  Note the CLI?)  This is the php.ini for php invoked from the command line interface.  It's not the same as when invoked from the debugger.  My actual one is in /etc/php5/apache2/php.ini.  If you can't find it, look at the top of the phpinfo output and note the location for "Loaded Configuration File".  Take note of the extensions directory which is, in this case, /etc/php5/apache2/conf.d.  

Go to the extension directory and create a new file named zend.ini (should be a couple other *.ini files in this directory already).  Whenever php starts up, it searches this directory and pulls in all the .ini files.  Now alter the contents of zend.ini to point to the "ZendDebugger.so" file with the following:

zend_extension=/usr/local/lib/zend/ZendDebugger.so

zend_debugger.allow_hosts=67.164.28.0/24, 127.0.0.1

zend_debugger.allow_tunnel=67.164.28.0/24, 127.0.0.1

zend_debugger.expose_remotely=always

Of course your IP addresses will be different, but you need both your ISP address as well as the 127.0.0.1 address.  This gave me a headache trying to figure out how to get it to work.

Note that "67.164.28.0/24" should be replaced with your external IP as the ISP has assigned it to you and Zend expects it in CDIR format.  Google "what is my ip address" to find out what it is.  Let's say it's 69.127.36.13.  You could write this as:

zend_debugger.allow_hosts=69.127.36.0/24, and any IP address with 67.127.36.0 to 69.127.36.255 will match it.

Copy the dummy.php file which is in the root directory where you untarred the ZendDebugger file to the document root of your server (probably something/something/public)

Save the settings and restart your web server:

"apache2ctl restart" or whatever that command happens to be on your server

Go to the document root of the web server (where you index.php file reside).

Create a new file named "phpinfo.php"

Edit it and add this single line at the top:

<?php phpinfo(); ?>

Go to  a web browser and access this file.  If you can't access, and you don't have it intentionally blocked via .htaccess, that's a separate problem.  Get your web server working to the point it can at least serve a page.

Assuming you can access it, you will (should) get a page full of information about your PHP installation with each function in a separate section.  Scroll to the bottom and find the "Zend Debugger" section.  If it's not in there, then you don't have ZendDebugger setup properly in the "php.ini" file, so at least get this working before trying to proceed.

You can also check the apache2 error log (or whatever server log you are using).  On ubuntu, the apache log file is located in /var/log/apache2/error.log.  

There is a good discussion of how to get this working at: http://forums.zend.com/viewtopic.php?f=59&t=962#p4190 which I got benefit from and I hope you do as well should you need it.

CONFIGURE ZEND STUDIO

Go to Zend Studio | Preferences | PHP | Debug | Installed Debuggers and select "Zend Debugger" and press the "Configure" button.  The default port for this is 10137, which is fine.  Set the Client Host IP to 127.0.0.1.  Make sure the Dummy File Name is still "dummy.php"

Now, enable the tunneling by activating the tunnel from the menu bar.  Here I had to attach a screenshot as there's no good way to describe this. 

Now go to Run | Debug Configurations.  Select PHP Web page, right click and create a "new" entry.  Set the PHP server to your newly created server: "NewServerName" i.e. whatever you named it when you created it above.  Pick the starting file from your project, and enter the matching URL (assuming auto generate didn't work correctly which it never does for me).

Now hit the "Test Debugger" button.  You should get success.  If you didn't, then check your "allowed hosts" again and go back over the steps.  

I verified that this setup is definitely working on my remote AWS EC2 development server.  When I get time, I am going to get NetOps to open the required ports or cede their tightfisted control so I can test non-tunneled remote debugging on the same machine.  But that's a task for another day.

Remote File Synch in Zend Studio

This remote file synch method works for any version of Zend Studio.  I originally started doing it in v7.2, and I just carried it forward.  It relies on rsync which is part of the standard Unix distribution.  If you are running Linux or MACOSX, you most likely already have it as /usr/bin/rsync (do a "which rsync" to verify it's installed and note down the location).  If you are running windows, you'd have to install it.  There is a version for windows, but I have not tested it.

I first tried using the built in remote server functionality in Zend Studio 7.2.  You can access this by going to the "Servers" tab which is in the lower right corner of the default PHP perspective.  If you select servers and then connection, you can configure a remote server.  You can then highlight the project, right click it to get the right click menu, and select Team|Share Project, and away you go.  The problem with this is it's not very configurable, and it insists on instantiating the project directory "SuperCoolTool" (or whatever) on the remote server, which is probably not what you want.  Also, if you select any files to publish initially, they ignore the local file path and end up going into the server's base directory.  While this would have been nice to have, it clearly wasn't ready for prime time in v7.2.

You'll generally manage an AWS EC2 box through SSH with RSA keys.  If you haven't used an RSA key before on your current machine, then you need to set it up.  There's a ton of info on that out there, so I am not going to go into that here.  For this sync method to work, you need to be able to SSH into the target server using a public/private key pair, and I assume you already have this setup for the remainder of this tutorial.

Go to a command prompt, and execute:

ssh remote_username@remote_server

You should be sitting in the base user directory for "remote_username" on the remote machine (remote server=either the IP address or domain if you have it aliased in /etc/hosts).  If you get access denied, you don't have your public key setup properly. So you need to fix it before proceeding.

While you are on the remote server, navigate to the base directory when you want your synched code to end up (maybe "site/root" or whatever) and make a note of the path.

The final server name you will use for rsync is:

remote_username@remote_server:site/root

so make a note of it.

You need to advise Zend Studio where those SSH keys are to use them.  Go to the "Zend Studio" on the menu bar and open the "Preferences" menu.  Select SSH2, and then (under the General Tab), select the SSH2 home directory which should already exist if you have working SSH and is located in your /Users/username/.ssh directory.

Select "Add Private Key" and select your private key which is probably named "id_rsa" (I imagine you can use id_dsa as well.  I have both on my machine).

Hit "Apply" and then "OK" and quit out of the menu.  

Now we are going to configure rsync so it will run automatically in the background any time we make any change to a file in our project.

Since we likely don't want every single file change to be mirrored, we need an exclusion list of files we don't want Zend Studio to mess with.

So, create an exclusion file in the base of your Zend Studio project.

Name the file: rsyncExclusion.txt.

It should contain the following:

.htaccess

.settings

.project

.buildpath

.settings

.cache

.svn

Rakefile.rb

Gemfile

Gemfile.lock

.externalToolBuilders

*.gif

/test/

/tools/

docroot

app.ini

Save it.  You will probably need to adjust this later, but it's a decent start.

Highlight the project.  Right click it.  Select properties from the right click menu.  Navigate to Builders.  Select the "New..." button on the right side of the dialog box.  Select "Program" as the option.  This produces another dialogue box.

In the "Main" tab (default starting point)

Name: New Builder of some kind (whatever you want)

Location: user/bin/rsync

Working Directory: (select the "Variables... button and choose "build_project"

Arguments: (going to run through these in a second)

-vcrz 

--copy-links

--delete-after

--exclude-from=rsyncExclude.txt

.

remote_username@remote_server:site/root

(Before you proceed further, you may want to change "-vcrz" to "-nvcrz" for testing.

-n = dry run (won't actually alter any files)

-v = verbose error messages

-c = use a checksum to identify files (vs name and date)

-r = recurse subdirectories and synch everything

-z = compress the file stream for better performance

--copy-links copy any symbolic links on the local copy as real directories on the remote server

--delete-after delete any files on remote server that are not on the local one

--exclude-from=rsyncExclude.txt

In the "Build Options" tab (tick the following):

Allocate Console

Launch in background

After a "Clean"

During manual builds

During auto builds

During a "Clean"

Uncheck anything else and hit OK

This should start synching immediately.

As noted, rsyncExclude.txt  is a file in the base of your project.  It has all file exclusions you want.  Anything in the list will be excluded from synch.  So if .htaccess is on the remote machine, but not on the local one, it will not be deleted IF IT IS ON THE LIST.  Otherwise, kiss it goodbye.  If you don't put the "--delete-after" option in there, it will not delete files remotely when you delete them locally.

Let's go over the exclude format a bit:

/tools/

docroot

app.ini

The option I have used here will only exclude tools/ if it is in the base directory

If you want any directory named tools excluded even if it's not at the base, it would be:

tools/

"docroot" in the list above is meant to protect a link such as:

ln -s public docroot

that is only on the remote server and not on the local one.

As can be seen by this example, you can protect files from deletion from the server as well as prevent inclusion from the local machine with the same list.

Once you hit okay, Zend Studio will now manage the file process in the background.  Any trouble you run into can likely be solved by adjusting the file exclusion list (assuming you actually had SSH working before you started attempting rsync).  I have found this to be a very reliable method of keeping a remote server in sync, and it doesn't require the remote server to be setup beforehand.  You can bring it online later in the project without any problems, and it doesn't clutter up your Zend interface with file status icons.