Don't Believe Me? Check Your Browser.
1 comment
Tuesday, November 11, 2008
Once upon a time, I owned a digital race photography company. We employed a national network of professional freelance photographers who took digital photos of participants at marathons, triathlons, bike races, snow shoe races, and more. They delivered those photos to our servers for sale. I was responsible for all of the custom software that kept the company and website humming.
We had an office in Seattle, so we shot lots of races in that area. There's a large software company headquartered not far from there that I'll call Tinysoft. Tinysoft is huge, and to their credit, many of their employees participated in races we photographed.
Shortly after one of the races, we received a large number of customer service complaints from Tinysoft employees, claiming that they were unable to purchase photos from us. It was easy to detect a pattern because they were sending their complaints from their tinysoft.com email addresses. But it was more difficult to track down the root of the problem. Since we weren't receiving similar complaints from other customers, I recommended that the Tinysoft employees attempt to purchase their photos from home, rather than work. That worked.
My focus shifted to find the cause of the problems on the Tinysoft campus. First, I sent email replies to the Tinysoft customers suggesting that they contact their internal tech support group to investigate the issue. I was convinced that the issue was not on our server side. I added much more verbose logging to the server and discovered that all of the failed page requests had RED-PRXY in a "via" field in the headers.
Todd
I have been to your site 5 different times in the last two weeks to order pictures. Unfortunately, not one time during my last 5 visits has your site worked properly. Everytime I try to order pictures, I get an error msg, "An error occurred processing your request." I mentioned having problems to several of my friends that have also used your site and they concurred that your site is always failing.
I'm not writing this msg to have you help me resolve the issue… I'm past that. You have great pictures and provide a wonderful service, but your web site makes it impossible to do business with you. If you outsource your site, I highly recommend looking at other options. You're loosing a lot more business than you think.
Me
We have been working to resolve the specific problem you are reporting. However, we have been unable to make much headway because we can't reproduce the errors. The problems are not universal. In fact, in nearly every case (at least 5 today), the feedback we've received has come from Tinysoft employees. As far as I can tell, the trouble seems to be related to the Tinysoft proxy servers. Although this will probably not make you feel better, we have been consistently receiving orders from other customers who have no trouble with the site.
Despite sending specific troubleshooting information to Tinysoft employees about the issue, it was not resolved. Their campus proxy server was truncating requests to our server, but I was unable to convince anyone of that. Since it was easily reproducible, I added a special error message to the site whenever the server detected the RED-PRXY value:
Special Note
You are accessing this site through a proxy server (RED-PRXY) that has been known to cause problems for our customers. As far as we can tell, this issue only affects Tinysoft employees accessing this site from the company network. Your proxy server sits on your network and acts like a middleman between your computer and our server. Proxy servers should pass data from to our server without modification. However, your proxy server has been known to truncate data. Our server is unable to process the truncated requests. You will most likely have trouble using this site, including purchasing your photos, as long as you access the site from your current network. Please contact your network administrator or our customer service department if you need additional information. You will have a better experience if you visit this site from outside your current network (e.g. from a library or home). If you need additional assistance, we are available to process your order over the phone if necessary. You may reach us at 206.555.1212 Monday - Friday 8AM to 5PM (Pacific).
Within 24 hours of adding the special note to the site, the issue was resolved by someone on the Tinysoft staff and Tinysoft employees were successfully purchasing photos again.
I hope you take a few lessons from this tale:
1. In Yuma, use verbose logging to review values in request headers that may expose trends while you are trouble-shooting issues for a small group of visitors to your site.
2. Trust what the data tells you. Even if the customer is Goliath to your David, the data doesn't lie.
3. Sadly, your customers may trust messages they see in their browsers more than information they receive via email or on the phone from a human.
Web development is fun! I mean that. I've been doing it for over 10 years and continue to find new and interesting challenges. I hope you are enjoying your development experience with Yuma.
We had an office in Seattle, so we shot lots of races in that area. There's a large software company headquartered not far from there that I'll call Tinysoft. Tinysoft is huge, and to their credit, many of their employees participated in races we photographed.
Shortly after one of the races, we received a large number of customer service complaints from Tinysoft employees, claiming that they were unable to purchase photos from us. It was easy to detect a pattern because they were sending their complaints from their tinysoft.com email addresses. But it was more difficult to track down the root of the problem. Since we weren't receiving similar complaints from other customers, I recommended that the Tinysoft employees attempt to purchase their photos from home, rather than work. That worked.
My focus shifted to find the cause of the problems on the Tinysoft campus. First, I sent email replies to the Tinysoft customers suggesting that they contact their internal tech support group to investigate the issue. I was convinced that the issue was not on our server side. I added much more verbose logging to the server and discovered that all of the failed page requests had RED-PRXY in a "via" field in the headers.
Todd
I have been to your site 5 different times in the last two weeks to order pictures. Unfortunately, not one time during my last 5 visits has your site worked properly. Everytime I try to order pictures, I get an error msg, "An error occurred processing your request." I mentioned having problems to several of my friends that have also used your site and they concurred that your site is always failing.
I'm not writing this msg to have you help me resolve the issue… I'm past that. You have great pictures and provide a wonderful service, but your web site makes it impossible to do business with you. If you outsource your site, I highly recommend looking at other options. You're loosing a lot more business than you think.
Me
We have been working to resolve the specific problem you are reporting. However, we have been unable to make much headway because we can't reproduce the errors. The problems are not universal. In fact, in nearly every case (at least 5 today), the feedback we've received has come from Tinysoft employees. As far as I can tell, the trouble seems to be related to the Tinysoft proxy servers. Although this will probably not make you feel better, we have been consistently receiving orders from other customers who have no trouble with the site.
Despite sending specific troubleshooting information to Tinysoft employees about the issue, it was not resolved. Their campus proxy server was truncating requests to our server, but I was unable to convince anyone of that. Since it was easily reproducible, I added a special error message to the site whenever the server detected the RED-PRXY value:
Special Note
You are accessing this site through a proxy server (RED-PRXY) that has been known to cause problems for our customers. As far as we can tell, this issue only affects Tinysoft employees accessing this site from the company network. Your proxy server sits on your network and acts like a middleman between your computer and our server. Proxy servers should pass data from to our server without modification. However, your proxy server has been known to truncate data. Our server is unable to process the truncated requests. You will most likely have trouble using this site, including purchasing your photos, as long as you access the site from your current network. Please contact your network administrator or our customer service department if you need additional information. You will have a better experience if you visit this site from outside your current network (e.g. from a library or home). If you need additional assistance, we are available to process your order over the phone if necessary. You may reach us at 206.555.1212 Monday - Friday 8AM to 5PM (Pacific).
Within 24 hours of adding the special note to the site, the issue was resolved by someone on the Tinysoft staff and Tinysoft employees were successfully purchasing photos again.
I hope you take a few lessons from this tale:
1. In Yuma, use verbose logging to review values in request headers that may expose trends while you are trouble-shooting issues for a small group of visitors to your site.
2. Trust what the data tells you. Even if the customer is Goliath to your David, the data doesn't lie.
3. Sadly, your customers may trust messages they see in their browsers more than information they receive via email or on the phone from a human.
Web development is fun! I mean that. I've been doing it for over 10 years and continue to find new and interesting challenges. I hope you are enjoying your development experience with Yuma.
Kim
Tuesday, July 21, 2009
Yuma dead??