In May last year I updated our CPAN installation as I normally do every 3 months or so. The routine will seem familiar to all Perl developers out there. On the CPAN shell just run install Bundle::* for most core modules, test, be happy. This one particular time it got a little more complicated than that. It was my own Perl Chernobyl – a simple procedure that should have taken 30 minutes became a system meltdown.

Flashback
Since 1999 my trust in CPAN has been unshakable, the Perl community itself has been one of the biggest quality assurance teams ever grouped together(perhaps rivaled only by Linux, but that was a few years after Perl was out there). What’s most impressive is a complete absence of micro-management by Larry Wall. Perl is an entirely autonomous slightly organized anarchic community of hackers that do what they do because they love Perl, not because a czarish group leader is spiking them with a fork.

Larry Wall’s object encapsulation principle, that you shouldn’t go into other people’s houses because you weren’t invited, not because the door is locked and the owner has a gun, prevails in the Perl community as well.

I was involved with Hibernate from early 2006 up until late 2007 in many projects. When I finished my work with Java I went looking for similar ORM alternatives in Perl, to try out on my own projects. Heck, even Dr. Dobbs had me featured as a Hibernate developer in Brazil…so I desperately needed to get back to Perl.

I tested Rose::DB(fast!) and Class::DBI without much success, not that I have anything against these great modules, just that I was missing something…and even I didn’t know exactly what it was.

 My homegrown can of wORMs
I also forgot the “lazyness” virtue for a bit and tried my own solution, which unfortunately ended up on a production system. My custom ORM was becoming a monster, its shadow pending over me scared me to heck and just the thought of maintaining my own ORM code freaked me out. It was dirty, clever code, full of hacks, cryptic variables, and too many specifics for the system it was running under(a large tourism system that grouped years and years of heterogenous data), so I couldn’t release it as a CPAN module to share my development nor reuse it on my other projects. All in all, the lesson was taken: lazyness IS a programmer virtue. Cultivate it. Time to find a Perl ORM on CPAN.

So…to make a long story short, fast forward and for over a year now I’ve been in love with the DBIx::Class ORM framework.

It doesn’t feature “too much magic”, it lets you hack some SQL in there when you need it, it reuses other CPAN modules for the SQL generation and is overall a very granular, very well architected system. All this without losing the Perl attitude, it feels like Perl and it just works, period.

I went full throttle with DBIx::Class, or DBIC as they call it. Since I couldn’t find any books on the subject(are there any yet?) I read all I could read about it online. The DBIC documentation has matured with time, as did my own projects using the modules. Slowly I’ve been migrating thousands and thousands of lines of code from my older projects to DBIC code. Right now I’ve developed this natural feel for DBIC. It’s consistent, works a dandy. The DBIC community is alive and active, giving support and kindly answering all kinds of questions on the dbix-class@lists.scsys.co.uk mailing list.

Back to May 2008 and my CPAN upgrade
So, as I settled with DBIx::Class, I got used to upgrading the system every so often. DBIC is still under 1.0, so it’s changing constantly. I kept my routine upgrades blindly, religiously, happening until May 2008.

That’s when an otherwise tranquil CPAN upgrade went sour. It was the first time a CPAN module in release status broke my system.

After the CPAN ritual was over everything seemed fine until screens went blank on my browser. No data was returned from my HTTP requests. I went over to check the Apache server logs and this is the war zone I found:
[Tue May 27 06:01:16 2008] [notice] child pid 14013 exit signal Segmentation fault (11)
[Tue May 27 06:01:47 2008] [notice] child pid 13698 exit signal Segmentation fault (11)
[Tue May 27 06:02:04 2008] [notice] child pid 13678 exit signal Segmentation fault (11)
[Tue May 27 06:56:07 2008] [notice] child pid 13999 exit signal Segmentation fault (11)
[Tue May 27 06:57:28 2008] [notice] child pid 14000 exit signal Segmentation fault (11)
[Tue May 27 06:58:10 2008] [notice] child pid 14004 exit signal Segmentation fault (11)
[Tue May 27 06:59:48 2008] [notice] child pid 14008 exit signal Segmentation fault (11)
[Tue May 27 07:00:17 2008] [notice] child pid 14002 exit signal Segmentation fault (11)
(Actually, this sample of log entries was taken at the NEXT DAY from the upgrade…. I had been getting my butt kicked by this problem since the day before…if you got this far, then read on.)

I suspected my application first, went over several pieces of it and couldn’t replicate the problem. Sometimes it worked, sometimes it didn’t. Worst kind of bug…these have a way to freak our mental logic unit(worse, it blows the otherwise slo-blo patience fuse).

I added a forced DBI reconnect to every page and the problem seemed solved, until it happened yet again. Blank page, segmentation fault. The absence of some proper error messages on the main httpd log meant it crashed before the logging phase of the Apache request loop.

This couldn’t be good, but I hadn’t realised that much at this point.

At first I was sure it was the Linux instance of DLL hell …I thought maybe my latest DBD::mysql linked to some conflicting libmysqlclient version. Maybe I forgot to run ldconfig… But then ldconfig didn’t work… Slowly I tried all the simpler tricks, none worked.

So I recompiled MySQL. It didn’t matter really, it wasn’t a critical MySQL host. It was, though, the main Apache server front, and it was down… Pressure everywhere, customers, everyone asking when the system would be back up…and there I was recompiling Apache, MySQL and other 20 or so libraries, hoping the impedance would match at some point and everything would just work again.

Each full system recompilation took over 20 minutes. It’s a custom Apache server wholly patched by yours truly to work as an optimized(hopefully, when it works) application server for Perl apps.

Was it PHP that was messing around with mod_perl? Tried it. Was it mod_ssl meddling with the Apache guts and knocking me down? Tried it. Nothing seemed to work.

The dreaded gdb session
I finally realized I’d have to feed a httpd core into gdb and try to find out what was happening from the recovered black box. Hopefully the httpd process core would reveal the culprit.

Recompile Apache one more time, this time editing line 282 of the configure script, adding CFLAGS=-g to add debug symbols and information to the binary. (In the meantime the server is down, and the cell phone is on silent, 27 calls unanswered.)

So, a few minutes later we have a brand new httpd binary, nice and stuffed with debug symbols.

Adding
CoreDumpDirectory /usr/local/apache/coredump/
to our httpd.conf would guarantee we get some juicy cores dumped for us at that location.

We tell the kernel we want core dumps by using ulimit -c (not activated on the Fedora family by default, but if you’re lucky to use Slackware you’ll find the cores are always there).

ulimit -c 10240 (10 MB limit for core files)

And Bob’s your uncle. A few minutes later and we have a TON of cores under /usr/local/apache/coredump/ to examine. I go with the lowest PID first to examine the earliest possible casualties, then I chose some random PID’s to examine. Here we go…

# gdb ../bin/httpd -c core.1035
GNU gdb 6.8
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show copying”
and “show warranty” for details.
This GDB was configured as “i686-pc-linux-gnu”…

[... snip ....]

Program terminated with signal 11, Segmentation fault.
[New process 1045]
#0 0×084b739c in Perl_av_undef ()

OK. What the heck is Perl_av_undef()?! I didn’t have much time to investigate. Phone calls were piling up as quickly as core files…and as curious as I was, I needed solutions, not overnight hacking sessions into the Perl source code…

I know DBIC uses some large data structures passed in and out of functions. Class instances can be HUGE data structures that reference a deeply nested spiderweb of related classes, on and on recursively.

Researching Perl_av_undef() on Google gives me the clue I need: it’s something related to deeply nested recursive data structures blowing up the Perl stack, AV probably means ArrayValue and so on.

So I restart the Apache server without a kernel stack limit:

# ulimit -s unlimited
(Actually, I chose 100MB for the original test. But I later found that when testing, you better exaggerate to really eliminate a possibility. This time we’re trying to rule out stack problems here.)

Restarted the httpd server, and for a few minutes I thought we’d solved the problem… Relief only lasted about 5 minutes…and the core files reappeared.

By now you could tell I was a pretty desperate man. The stack problem was only part of the overall grim picture.

I decided to check the core files and see what had changed.

# gdb /usr/local/apache/bin/httpd -c core.1047
[... snip ...]
Program terminated with signal 11, Segmentation fault.
[New process 1047]
#0 0×00b9e46b in mysql_ping () from /usr/local/mysql//lib/mysql/libmysqlclient.so.15

And that’s when things started to turn around. Mysql was my first suspect after all….

Back to gdb:
(gdb) where
#0 0×00b9e46b in mysql_ping () from /usr/local/mysql//lib/mysql/libmysqlclient.so.15
#1 0×004d99f2 in XS_DBD__mysql__db_ping (cv=0×94d2680) at mysql.xs:519
#2 0×001edbf4 in XS_DBI_dispatch () from /usr/lib/perl5/site_perl/5.10.0/i686-linux/auto/DBI/DBI.so
#3 0×084bfa43 in Perl_pp_entersub ()
#4 0×084be32e in Perl_runops_standard ()
#5 0×084ba15d in Perl_call_sv ()
#6 0×0043ae8f in EMBPERL2_CallStoredCV () from /usr/lib/perl5/site_perl/5.10.0/i686-linux/auto/Embperl/Embperl.so
#7 0×00453cf0 in embperl_Execute () from /usr/lib/perl5/site_perl/5.10.0/i686-linux/auto/Embperl/Embperl.so
#8 0×0045a008 in ProviderEpRun_GetContentIndex () from /usr/lib/perl5/site_perl/5.10.0/i686-linux/auto/Embperl/Embperl.so
#9 0×0045891a in Cache_GetContentIndex () from /usr/lib/perl5/site_perl/5.10.0/i686-linux/auto/Embperl/Embperl.so
#10 0×00434a63 in ProcessFile () from /usr/lib/perl5/site_perl/5.10.0/i686-linux/auto/Embperl/Embperl.so
#11 0×004355da in embperl_RunRequest () from /usr/lib/perl5/site_perl/5.10.0/i686-linux/auto/Embperl/Embperl.so
#12 0×00435b8c in embperl_ExecuteRequest () from /usr/lib/perl5/site_perl/5.10.0/i686-linux/auto/Embperl/Embperl.so
#13 0×0043233f in XS_Embperl__Req_ExecuteRequest () from /usr/lib/perl5/site_perl/5.10.0/i686-linux/auto/Embperl/Embperl.so
#14 0×084bfa43 in Perl_pp_entersub ()
#15 0×084be32e in Perl_runops_standard ()
#16 0×084ba15d in Perl_call_sv ()
#17 0×08371d3b in perl_call_handler (sv=0×96a8d20, r=0×9541c14, args=0×0) at mod_perl.c:1668
#18 0×083723f9 in perl_run_stacked_handlers (hook=0×85ef03b “PerlHandler”, r=0×9541c14, handlers=0×96a8cb0)
at mod_perl.c:1381
#19 0×08373861 in perl_handler (r=0×9541c14) at mod_perl.c:904
#20 0×08395497 in ap_invoke_handler (r=0×9541c14) at http_config.c:476
#21 0×083acd2f in process_request_internal (r=0×9541c14) at http_request.c:1299
#22 0×083ad1a0 in ap_internal_redirect (new_uri=0×951427c “/index.html”, r=0×9512ae4)
at http_request.c:1440
#23 0×00386fe4 in mod_gzip_redir1_handler () from /usr/local/apache/modules/mod_gzip.so
#24 0×00385219 in mod_gzip_handler () from /usr/local/apache/modules/mod_gzip.so
#25 0×08395497 in ap_invoke_handler (r=0×9512ae4) at http_config.c:476
#26 0×083acd2f in process_request_internal (r=0×9512ae4) at http_request.c:1299
#27 0×083acd8c in ap_process_request (r=0×9512ae4) at http_request.c:1315
#28 0×083a2ef6 in child_main (child_num_arg=12) at http_main.c:4971
#29 0×083a3227 in make_child (s=0×87be05c, slot=12, now=1211894763) at http_main.c:5150
#30 0×083a32c2 in startup_children (number_to_start=23) at http_main.c:5178
#31 0×083a3a66 in standalone_main (argc=3, argv=0xbfe95934) at http_main.c:5525
#32 0×083a434f in main (argc=3, argv=0xbfe95934) at http_main.c:5883

The call stack trace is very clear: we get that the problem happens with the native mysql_ping() C function called from an XS stub.

Now Google is our friend and we quickly arrive at this bug report from May 20 2008, precisely a week before my fantastic upgrade voyage.

Suggested solution is regression to DBD::mysql version 4.006… How frustrating. CPAN only installs the latest version, so we gotta do this one manually.

# tar xzvf DBD-mysql-4.006.tar.gz
# cd DBD-mysql-4.006
# perl Makefile.PL
# make
# make install
# /usr/local/apache/bin/apachectl restart

It’s been a year since….and the bug is still there. A poster on the original bug report discussion thread says 4.010 fixes the issue. I haven’t tested that one yet.

2009-05-18 – Version 0.10

This is release 0.10 of Zen Twitter Tools and the Zen::Twitter library, which currently covers the following:

• Search
• Timeline
• Status
• User
• Direct Message
• Friendship
• Social Graph
• Account
• Favorite
• Notification
• Block
• Help

You can read more about the above function groups at the Twitter API documentation site.

The following scripts are currently included on Zen Twitter Tools v 0.10:

Account Management
account_rate_limit_status.pl – Check your current API limits and how many credits you have left.
account_update_delivery_device.pl – Update delivery device for Twitter notifications.
account_update_profile_color.pl – Update your profile colors.
account_update_profile.pl – Update your profile data.
account_verify_credentials.pl – Test the data on credentials.xml by logging in from the command line.

Block Management
block_blocking.pl – List users you are currently blocking.
block_create.pl – Block a user
block_destroy.pl – Unblock a user
block_exists.pl – Check whether you are currently blocking a user.
block_ids.pl – List IDs of all users you are blocking.

Direct Messages
direct_message_destroy.pl – Delete a Direct Message
direct_message_send.pl – Send a Direct Message
direct_messages.pl – List Direct Messages inbox
direct_messages_sent.pl – List Sent Direct Messages

Favorites
favorite_create.pl – Add a status to your Favorites
favorite_destroy.pl – Remove a status from your Favorites
favorites.pl – List your Favorites

Followers and Friends
followers_not_friends.pl – Lists which followers you have not yet befriended.
friendship_create.pl – Follow a user
friendship_destroy.pl – Unfollow a user
friendship_exists.pl – Check whether user_a follows user_b
friends_not_followers.pl – Lists which friends do not yet follow you back.
friends_timeline.pl – Your friends’ 20 latest updates
mentions.pl – The 20 latest mentions of @you
public_timeline.pl – The Twitter-wide latest 20 updates, updated every 60 seconds.

Notification (Mobile)
notification_follow.pl – Receive mobile notifications for a user.
notification_leave.pl – Stop receiving mobile notifications for a user.

Search
search.pl – CLI interfact to search.twitter.com. Searches Tweets for keywords. Example: search.pl ‘twitter tools’

Social Graph
socialgraph.pl – Simple friend/follower count.

Status
status_destroy.pl – Delete one of your status messages.
status_show.pl – Show the status of a user.
status_update.pl – Update your status.

Help
test.pl – Tests a simple exchange with the Twitter server to check connectivity.

Users
user_timeline.pl – Retrieve a USER’s timeline(Same data you see on http://twitter.com/USER’s page)
user.pl – Reports the current status of the user identified by screen_name

Sample Scripts
Sample scripts are provided with each module so you can implement/customize your own tools. The scripts can also be used to directly manipulate your account. All functions requiring authentication need a properly configured credentials.xml file. Setup is very simple, you should not have any problems using the scripts and library.

Not Implemented on v 0.10
The following functions were not implemented on 0.10 :
- Search trends (Reason: did not implement the JSON-only requests yet.)
- Profile image uploads(2 functions: background image and user image).
- Account signoff function. (Not applicable on a stateless CLI environment.)

Download: zen-twitter-tools-0.10.tar.gz

Note: I’ll be working on improving the documentation for the library. For now I’ve quickly annotated each function but did not provide standard CPAN-quality POD’s for all modules.

2009-05-14

I found some Twitter modules on CPAN but the Twitter API seemed fun, it’s well documented and getting more popular by the day, so I felt like having a go at it in Perl.

Bug reports, suggestions and constructive criticism are always welcome.

So far we have some base classes, authentication, HTTP and XML modules in place. The only demos available at this time are the Friend/Follower monitoring scripts.

The Zen::Twitter::User Object
Using Class::Accessor we automate the creation of accessor methods for each field returned in the Twitter user data structure. See below for the vast ammount of user information you can get from a single Zen::Twitter::User object.

Accessor Method	Description
id	Numeric Twitter ID
name	Users’s real name
screen_name	Twitter nickname, what goes twitter.com/HERE
location	Geolocation of the user
description	Self-description
profile_image_url	User’s photo image url
url	The user’s home page
protected	String boolean literally ‘true’ or ‘false’, whether user’s updates are protected
followers_count	Number of people following this user
profile_background_color	6 character html hex code without the traditional hash# (E.g. ffcc00 for bright orange)
profile_text_color	6 char html hex code for his text color
profile_link_color	6 char html hex code for profile link
profile_sidebar_fill_color	6 char html hex code for sidebar fill
profile_sidebar_border_color	6 char html hex code for profile border color
friends_count	Number of friends (people this user is following)
created_at	Signup date in format DAYOFWEEK MON DAYOFMONTH HH:MM:SS +-OFFSET YEAR
favourites_count	Number of favourites
utc_offset	UTC offset
time_zone	Time zone
profile_background_image_url	Profile background image URL
profile_background_tile	String boolean ‘true’ or ‘false’ whether the bg image is tiled
statuses_count	Number of updates
notifications	String boolean ‘true’ or ‘false’ whether user receives notifications
following	Number of people following this user

Zen::Twitter
The Zen::Twitter module is the general front-end employing direct Twitter functions to create added functionality. For instance, friends_not_followers() and it’s counterpart followers_not_friends() are implemented in it, using 2 direct Twitter functions from Zen::Twitter::Friends to obtain the respective lists and then diff them from each other.

Zen::UserAgent
This is currently a single-function stub really, all it does is create a single point from where our UserAgents are created. We might need to add functionality, so right now it might seem like overkill but it just might come in handy to have all our HTTP User Agent stuff tucked in one namespace.

Zen::Twitter::Authentication
Hides the details of finding and parsing credentials.xml and gives us a Credentials object to pass around wherever authentication is required.

Zen::Twitter::Credentials
A simple Credentials interface. Currently offers username() and password() methods only. Created it so we wouldn’t be passing a hashref around accessing its internals without a proper interface.

Zen::Twitter::Friends
Maybe this one should have been called Zen::Twitter::Users, keeping the names sync’d with Twitter’s. It contains 2 direct Twitter querying functions: friends() and followers(), both return an arrayref of Zen::Twitter::User objects for the user identified by the Credentials passed in.

Zen::Twitter::HTTP
Queries the Twitter XML services and returns Perl data structures parsed by XML::Simple. A Zen::Twitter::User can be directly instantiated from each user tag on the XML returned by the functions described here.

Download
zen-twitter-tools-0.01.tar.gz

That’s all folks! I hope you can use some of this code to create your own Twitter apps(I placed it all under the BSD License, just in case).

2009-05-17 – Version 0.08

- Implemented Block methods, with sample scripts.
Twitter API Sections now Available in Zen::Twitter :

• Search(without ‘trends’)
• Timeline
• Status
• User
• Direct Message
• Friendship
• Social Graph
• Favorite
• Notification
• Block
• Help

Next up in TO DO list: Account Methods

Download: zen-twitter-tools-0.08.tar.gz

Version 0.07

- Implemented Friendship methods, with samples.
- Implemented Social Graph methods. Included sample scripts.
- Implemented Favorite methods, with sample scripts.
- Implemented Notification methods, with sample scripts.

Twitter API Sections now Available in Zen::Twitter :

• Search(without ‘trends’)
• Timeline
• Status
• User
• Direct Message
• Friendship
• Social Graph
• Favorite
• Notification
• Help

Next up in TO DO list: Block Methods

Download: zen-twitter-tools-0.07.tar.gz

Version 0.04

- Implements the 4 Direct Message methods: direct_messages(), sent(), new() and destroy() which allow you to create and send direct messages as well as view the lists of sent and received messages.

Twitter API Sections now Available in Zen::Twitter : Search(without ‘trends’), Timeline, Status, User and Direct Message Methods.
Next up in TO DO list: Friendship Methods

Download: zen-twitter-tools-0.04.tar.gz

Version 0.03

- Implements all 3 Status methods: show(), update() and destroy() which allow you to view any status messages(authentication needed for protected updates), update your own status and destroy your own unwanted updates. Includes 3 example scripts implementing each.

- Implements the single test() Help method. Sample script allows you to “ping” the Twitter servers and test connectivity.

Twitter API Sections now Available in Zen::Twitter : Search(without ‘trends’), Timeline, Status and User Methods.
Next up in TO DO list: Direct Messaging.

Download: zen-twitter-tools-0.03.tar.gz

Version 0.02

- Implements all 4 Timeline Methods : Display public timeline, friends’ timeline, a specific user’s timeline and mentions of you.
- Zen:Twitter::User object is employed on all Timeline results where a user is mentioned
- 4 scripts added to demonstrate the Timeline methods

- Implements Search (No ‘trends’ functionality yet.)
- search.pl script added to demonstrate search functionality

- Filtering/sorting HTTP parameters unimplemented by the library are passed on to Twitter.

Download: zen-twitter-tools-0.02.tar.gz

For quite a few days now I’d been getting 500 Server Errors on my main Apache server, which is an EC2 instance at Amazon Web Services.

The traffic on some of the sites is quite respectable, so i was definitely not a “morning bug” issue.

The messages I was receiving varied, and the error seemed random – it wasn’t the first connection, it wasn’t a last, not related to long queries, not related to a certain section of the web site and so on. Worst kind of problem to debug.

Some of the messages I received included:

DBI Connection failed
or
Can't connect to MySQL server on [...]
or
DBI Connection failed: Lost connection to MySQL server at 'reading authorization packet', system error: 0 at [...]


Further testing led me to several MySQL bug reports, some of which were legitimate. Others were just support requests disguised as bugs.

It turns out sometimes the network latency of EC2 instances can get quite high. I’d heard such comments before, but never had any issues.

Raising connect_timeout from the default of 5 seconds to 30 solved the ‘connection failed’ problem. Raising the net_read_timeout solved the ‘Lost connection’ issue. All three problems were related to network latency.

Added to my MySQL server’s /etc/my.cnf [mysqld] section:

connect_timeout=30
net_read_timeout=45