TravisSwicegood.com

It's 37º F (2.7º C) and dropping. It's going to hover near freezing tonight, and come midnight... flip a coin - head's it rains, tails it's dry. And he's out there.

I know he's not my responsibility. But isn't he?

There's a big guy that's homeless here in Lawrence. You know him if you've lived in or around downtown Lawrence. The guy's really big. He started hanging around South Park shortly after the Salvation Army closed their shelter. We noticed him hanging out early in the morning and in the evenings. It wasn't long before we put two and two together. He'd taken up residence. At least over night.

As the temperature started dropping, he and the other few souls that would spend the night in South Park on the benches, in the gazebo, or under the stairs at the gazebo started disappearing. I assumed they'd found shelter somewhere warm, or moved south following the warmer temperatures. Tonight, as it warms up to a balmy 37, he was back.

We noticed him earlier this evening as we walked the dogs. Meg pointed him out as we started in to South Park. He was headed down the red-bud path. As he got closer, we could see the two pieces of cardboard he was carrying. My heart went out to him as I realized what he had done. He'd scavenged up some insulation for the night to come.

Continuing on our walk, he milled around South Park some more. As we finished our loop through the park with our dogs, I saw him lumbering off back toward downtown. "Well, maybe he isn't spending the night in the park," I reassured myself and didn't give it another thought. Meg and carried on our evening, including some pizza and beer at the Oread and a Jayhawks game.

Walking back down the mountain tonight, I couldn't help but scan the park. The benches were all empty, as was the gazebo. Maybe he had found a spot in the shelter. Then I saw under the stairs. His unmistakable girth, under a pile of blankets, huddled up against the gazebo.

I quietly motioned to Meg. Our banter, lively all the way home, died.

After we had passed the gazebo I said, "I know he's not my responsibility. But if he isn't mine, who's is he?"

I don't know what I could have done. I don't know his story. Could I approach him, or is he unstable? Why's he homeless? Is it a "lifestyle choice", as certain organizations would have you believe of the majority of Lawrence's homeless, or is he one of those guys who just had one too many blows and hasn't been able to get back up?

I wonder though, what is my responsibility? What is our responsibility? If you believe in a higher power, when he comes asking "where's the big guy?", are you going to respond "am I my brother's keeper?", or will you response "I am my brother's keeper, he's over here"?

Want to help?

Head over to the Lawrence Community Shelter website. They can use cash or supplies.

As someone noted the other day on one of my Facebook posts, I've been doing a lot of Python development. I've moved almost entirely to Python for development, web and otherwise. Instead of PHP, I reach for Django when I need to prototype an application quickly.

One of the things I've been struggling with is how to build re-usable applications that are testable without having the entire Django stack running. Until recently, I've used buildout to handle this. There's a djangorecipe for creating a Django repository. I include that, a sample project, the necessary requirements in a buildout.cfg and away we go.

All was well, until I included two project that had a sample project/ directory. The base Django project couldn't figure out what was what, problems abound. There are other solutions. I could have set the project (doesn't this seem like the Misses Bennett in Pride and Prejudice?) variable to change it on a per-app basis, but that still left me with some problems.

Namely, I don't like the default layout of djangorecipe's Django project. I wanted to change it, but after some digging around in buildout's internals, I realized it wasn't going to be a solution I could live with long term. I'd heard a lot of people (by that, I mean James) state their preference for virtualenv and pip. The separation of concerns (one application for isolation, another for installation) instead of the all-in-one approach of buildout felt better to me, so I started exploring.

And I came up empty.

A lot of people talk about using virutalenv and pip together; pip documents how to install into a virtual environment; but no one talks about how to use everything together with Django. Specifically, no one mentions what to include in your repository. Until now. :-)

Setting up the repository

The most important part of this for me was what to store in the repository. It's simple enough, really. First, you need a requirements.txt file. For most simple Django apps, it contains one line. For example, this is what the requirements.txt file for d51.django.apps.tagging looks like:

Django

The next thing I need is a simple .gitignore file. My mantra is to not commit anything that I can generate. This means all of the files generated by virtualenv and pip need to be ignored. I also ignore the swap files created by Vim (hey, I'm not the only one at the company who uses it, so might as well ignore it) and I ignore all .pyc and .pyo files. The resulting .gitignore file looks like:

bin/*
include/*
lib/*
.*.swp
*.py[co]

Now we're ready. Of course, you need to have virtualenv and pip installed, but once you've done that, running tests are pretty simple. First, you have to initialize the environment:

prompt> git clone git://github.com/domain51/d51.django.apps.tagging.git
prompt> cd d51.django.apps.tagging
prompt> virtualenv .
prompt> pip install -E . -r requirements.txt

The observant might notice my call to virtualenv. I've left out the parameter --no-site-packages. Two reasons. First, I don't keep things like Django installed at the site-packages level. Second, the things I do install, tools like Fabric, I want access to them while in the virtual environment without having to re-install them.

Now that the virtualenv has been initialized, now you need to activate the virtual environment:

prompt> source ./bin/activate
(d51.django.apps.tagging)prompt>

Notice that the prompt changes. It's prefixed with the name of the directory you're in to signify that you're inside virtualenv. Now running the tests are dead simple:

(d51.django.apps.tagging)prompt> python ./run_tests.py

Testing Django apps inside virtualenv

I need audio that plays when you get to this line. That screeching record player coming to halt. The visual question mark. What's this ./run_tests.py file you ask? The secret sauce.

Django wants to be setup in order to run. Normally that's requires a project, settings.py, and a partridge in a pear tree. Unless you call settings.configure. You can use that to mimic the normal Django settings, tweaking the settings to match your needs for testing.

For d51.django.apps.tagging, the settings are pretty simple. I need to make sure that my app is available along with django.contrib.contenttypes since I make use of the generic relationship code. There's also some cargo culting required, as Django won't run without a DATABASE_ENGINE specified. The end result looks like this:

from django.conf import settings
from django.core.management import call_command

def main():
    # Dynamically configure the Django settings with the minimum necessary to
    # get Django running tests
    settings.configure(
        INSTALLED_APPS=(
            'django.contrib.contenttypes',
            'd51.django.apps.tagging',
        ),
        # Django replaces this, but it still wants it. *shrugs*
        DATABASE_ENGINE='sqlite3'
    )

    # Fire off the tests
    call_command('test', 'tagging')

if __name__ == '__main__':
    main()

Running without activating virtualenv

This works, but requires that you always have virtualenv activated. For example, if you deactivate virtualenv and try to run the test, you get an ImportError:

(d51.django.apps.tagging)prompt> deactivate
prompt> python run_tests.py 
Traceback (most recent call last):
  File "run_tests.py", line 1, in <module>
    from django.conf import settings
ImportError: No module named django.conf

You can programmatically activate virtualenv, however, by including this snippet of code in a .py file located in the root of your repository:

execfile('./bin/activate_this.py',
         dict(__file__='./bin/activate_this.py'))

You can add that line to the top of the file and execute run_tests.py without needing to activate the virtualenv before hand. The line needs to go before the from django.conf line to make sure that Python knows where to find Django and any other requirements of the test.

This requires that you activate the virtual environment prior to running the test, or have Django installed at the system level. This can be further simplified and remove the need to activate and deactivate the environment prior to test runs by executing the bin/activate_this.py file that virtualenv ships with.

Making this reusable

There's a lot of code in that run_tests.py file that is going to be duplicated for every project. Actually, there's only three lines, well, two lines and one variable in a line, in it that are unique: the two line of INSTALLED_APPS and the app name to test in the call_command line.

To keep from repeating that for every single project, I created a simple harness for initializing tests. Introducing d51.django.virtualenv.test_runner, a very small package for running Django tests inside virtualenv.

Using this, run_tests.py now looks like:

try:
    from d51.django.virtualenv.test_runner import run_tests
except ImportError:
    print "Please install d51.django.virtualenv.test_runner to run these tests"

def main():
    settings = {
        "INSTALLED_APPS": (
            "django.contrib.contenttypes",
            "d51.django.apps.tagging",
        ),
    }
    run_tests(settings, 'tagging')

if __name__ == '__main__':
    main()

The first four lines give the user some input when they run the tests without test harness. That's optional, depending on how user friendly you want to be. After that, all I do is call run_tests with a settings dictionary and the name of the app I want to test.

There is one downside, though. You have have it installed outside of your virtualenv in order to run your tests. Personally, I'm not to worried about it, as I have it installed, but if you're really paranoid, you could include it in your requirements.txt file, which would require that the user be inside the virtualenv to run your tests.

Wrapping it all up in a Fabric cloth

The final step is to make all of this bullet proof is creating a Fabric file that handles all of the initialization and running of the tests for me. For good measure, it should also be capable of cleaning up after itself. I don't need a bazillion copies of Django laying around, afterall.

The end result looks something like this (using Fabric 1.0a):

from fabric.api import local

def test():
    """
    Run tests for d51.django.apps.schedules
    """
    local("python ./run_tests.py")

def init():
    """
    Initialize a virtualenv in which to run tests against this
    """
    local("virtualenv .")
    local("pip install -E . -r requirements.txt")

def clean():
    """
    Remove the cruft created by virtualenv and pip
    """
    local("rm -rf bin/ include/ lib/")

Now you can run the initialize the environment, run the tests, and clean up after yourself with three commands:

prompt> fab init
prompt> fab test
prompt> fab clean

One drawback to this method, Fabric's local command swallows the output of the test. This isn't a problem until you have a failure. local does contain a capture parameter, but it doesn't display the output from an failed command. That's fixable, but for the time being, my recommendation is to use Fabric as your quick sanity check, but rely on straight python ./run_tests.py for your real testing.

Conclusion

That brings us to the end of our quick tour. Hopefully this provides you with the information you need to get started using virtualenv and pip with Django. It's not that complicated. Actually, once you have your bearings, it's downright easy. The problem is more that people who have trodden down this path haven't documented their way. Hopefully, this post helps serve as a rough map.

Thanks

I'd like to thanks James Bennett for illuminating a few pieces of this puzzle for me (in particular, pointing me toward settings.configure) and his preview a draft of the article before I posted. I'd also like to thank Jeff Triplett for his comments on a draft and pointing out an unexplained inconsistency with the rest of the world's examples of virtualenv. And, as always, my good buddy Roder for his constant encouragement.

Yesterday I lamented the issues with namespaces in Python. It's not really the namespaces, it's the marketing of namespaces. Newbies to the community (something I still consider myself for most purposes) are drawn to modules thinking that there's a one-to-one relationship between file hierarchy and namespaces. And there is. Well, sort of.

You have to read the entire manual or happen to have someone to point out the difference between namespaces and modules to even realize there is a difference. Under most circumstances, you won't even realize they are different until you start to do something slightly complex. Say, for example, building an application with multiple modules inside a similar namespace, each module in a separate repository and its own history. There needs to be a "there be dragons" warning to let people know.

Those dragons are such: you have two paths inside sys.path that contain similar code. Such as I do with all of the Domain51 code. Package one has foo as a module while package two contains bar. The assumption would be that Python acts like most other languages and would exhaust it's sys.path trying to find both packages, but that's not the case. It'll get to the first one, then pretend the second doesn't exist.

The fix for this is the explicitness, one of Python's cardinal virtues. You have to declare the namespace in order for it to work. As you don't see this hardly anywhere in Python because Pythonistas feel that namespaces are a bad idea, here's the code you need to include in your __init__.py file to make it declare itself as a namespace:

import pkg_resources
pkg_resources.declare_namespace(__name__)

That's it. Now Python becomes smart again, and you can have real namespaces with similar directory structures existing side-by-side. Python is perfectly capable of finding them - now. Which raises an interesting question: why does Python scan the entire sys.path looking for files, building up this list of what declares what namespaces and where only to ignore it later unless they're explicit about it? I haven't dove into the source to be sure, but it seems it would have to scan the __init__.py files in order to know whether there's something there.

But I digress. There's a bigger dragon that's not even hinted at. Python's inability to find modules.

Take, for example, my python-stupidity repository on GitHub. Run the test.py file and you can see the error for yourself. There are two barfoo modules within the path, but Python decides to act the villagedolt and stop as soon as it hits the first one that might match. This particular case is caused by foobar trying to import a method from barfoo that doesn't exist in foobar.barfoo

This is, in my opinion, a huge issue. Note that foobar.barfoo declared it's namespace. It said loudly, "I am me", and Python ignored that fact in favor of relative includes. Not only that, but it stopped and started pouting as soon as one module that said it was foobar.barfoo couldn't match.

Why not finishing looking through the rest of the sys.path? Why not pay attention to that precious declaration Python wants you to add to explicitly become a namespace?

Like almost all problems with programming languages, however, there is a fix. At first glance, I thought it might be the way PHP handled it - just include a separator at the beginning of the import. That didn't work, but in searching for the solution, I found out that Python supports relative imports through it's support of intra package references. The fix within the foobar module is to do from ..barfoo import base_barfoo, but this only covers you if you're in Python 2.5 or later.

According to my understanding of it, you use it to explicitly say I want a sibling module names X without having to declare the entire namespace or accidentally picking up a module from the global namespace. Fair enough, but my solution to the problem above is to use the relative import to trick Python into thinking it couldn't find a module named the same.

You can see the code in the d51.django.auth package. I have a d51.django.auth.facebook module which takes precedent over PyFacebook's facebook module, but only inside the d51.django.auth.

I'm not saying that namespaces are a bad idea in Python or any other language. I'll gladly take namespaces, in any form I can get them and use them. They provide a great way to segregate code into small, independent, re-usable packages while continuing to say "I'm from over here." They allow facebook to be used as a module in multiple places without causing an issue, other than the ones listed here.

No, my problem is not with namespaces. My problem is with Python's current method for searching for them; it's lack of exposing namespaces and modules and their differences up front, and it's brain dead way of halting on the first partial hit. I'm amazed that a language that prides itself on explicitness—on not doing anything that's not asked for—decides that it's ok to stop looking for matching code just because it found one thing that doesn't match. It smacks of premature optimization.

Keith Casey's recent post on a book recommendations got me thinking. I get asked what books I recommend. I whole-heartedly agree with his first recommendation. The Pragmatic Programmer sits on my desk and is often in my laptop bag. I reach for it if I have five minutes and want to flip through something technical without having to load up my Google Reader.

Not on his list is the Passionate Programmer. I highly recommend this book to anyone working in the industry. I've reviewed it on Amazon, feel free to check that out for more information.

But this post isn't about what I recommend, it's what I don't. There are two books that Keith recommends that I no longer recommend off the bat to programmers. PEAA and Refactoring. Both are excellent books, but neither should be read by people starting out.

Why? The books are both excellent. I've read both. I own both. I do recommend them, some of the time. Both are geared toward programmers that already know how to program. Both contain a wealth of information which, in the right hands, can take the development of the reader to the next level.

But they also contain information that can be abused tremendously. For example, Active Record in it's abused, single-world form of ActiveRecord has caused a tremendous about of scaling issues. All because it was a Fowler endorsed method for dealing with data in a database. Developers without an understanding of its limitations started copying what Rails did and in the process mixed business logic and database logic into the same object. What happens when your data needs to be stored in a flat file or a nosql style database? Your business logic is now all tied up in your database and its painful to extract that and retrofit your application to be scalable.

Of course, since you have Refactoring you can fix that. Right? Well, sort of. Patterns and refactoring go hand in hand. Ideally, refactoring code should lead to some more understandable, reusable code; something that's more easily explained to another developer. Patterns fill the bill. Nowhere in Refactoring or Joshua Kerievsky's Refactoring to Patterns will you find anything about pattern hoping. To fix an improper use of ActiveRecord by moving to another pattern, however, you end up doing exactly that.

Couple that with the sweater string issue. We've all had a loose string on a sweater, right? You give it a tug instinctively to get rid of it, and before you know it a hem in gone and the sweater is ruined. Refactoring, in the wrong hands, causes the same thing.

Well, I need to extract this logic out. Next this piece needs to be mutable. I should inject this object so I can replace it out later if I need to. And so on, and so on, ad infinitum.

Both patterns and refactoring are powerful tools, but not in the hands of a novice. Both books should be read, but not when you're starting out. Get your hands on Pragmatic and Passionate, spend some time looking for a mentor to help guide you down the right path. Once you get your feet under you as a programmer, then start learning about the mechanics of patterns and refactoring. Once you're to that point, PEAA and Refactoring are great material.

Wow! Has it already been nearly a week since ZendCon wrapped up?! Time is flying right now, and with IPC and the holidays right around the corner, I don't see it getting any less hectic.

ZendCon was fun. This was my first time attending/presenting this particular conference. It had a distinctly different feel too it than the conferences put on by Marco and company. ZC felt more corporate than tek does, but that's not necessarily a bad thing. We tend to forget that if it weren't for the business interests at play none of us would have jobs.

I made it to only a handful of talks, but everything I attended was top-notch. I'd highly recommend checking out Josh Holmes' The Lost Art of Simplicity talk. He presented it in the uncon, but accept it as a keynote if he submits it to your conference.

Speaking of Microsoft (Josh works for them) they stepped up in a huge way Wednesday night. The topic of what to do after the Ask Zend session that night and the idea of heading up to San Francisco came up. Josh jumped into gear and not only got us into the Bing party that was going on in conjunction with Web 2.0, but also scored us all copies of Windows 7 and a party bus to get us there and back!

Microsoft catches a lot of crap from the open-source community, some of it warranted, but they're heading in the right direction as long as they keep hiring guys like Josh and making them the face of Microsoft. The company will have no choice to change if it's made up guys who are really taking the blue monster to heart.

The cynic in me has to ask if this isn't just a long-view version of embrace, extend, and extinguish via slowly causing liver failure in the main contributors to open source! :-D

I also finally got a Fork You shirt from the guys over at GitHub. I'm thinking of forking the shirt design and updating it to say:

Fork Me

On the Git front, I might have made Elizabeth M. Smith mad enough at the state of the current Git Windows tools to actually do something about it. She started hacking a bit on it at the conference, and I'm sure we'll see something by the end of the year (right, Liz?).

Next up for me (and my final conference of 2009) is the International PHP Conference in Germany. I'll be giving a talk titled Building Real-Time Applications using XMPP. Meg is tagging along and we're going to see some of southern Germany after the conference. Definitely looking forward to the time off on another continent.

So I've been kicking around the idea of starting a hacking night/weekend type thing. I think it'd be a blast to take one evening or afternoon a month or maybe even ever week, commit to a particular project and get a bunch of people in the same room and hack on it. We do a lot of yacking at LPDN, this would be an opportunity to do a lot of coding. Ya know, less bitching, more fixing.

There's a couple of possible formats. I'm curious to get some feedback. What do you think would be more fun?

• First, we pitch around a few ideas over at Lawrence Programmers, come up with some new library or software, hack it together, release it, and all feel good about contributing to the world at large.
• Second, we could pick open source projects and contribute to them. PHP has had some big successes with their Test Fests. There's plenty of open source projects in every language that would love more unit tests or some bug squashing. We could trade languages, each month doing something different with people that know that language taking the lead. Get someone from each language to chaperon their own language.
• Third, we could do something GiveCamp style. Get people to donate a weekend and hack out as much as we can for non-profits. There's plenty of non-profits that could use a hand. I know of one (*wink* *wink*) that has a ton of really cool projects, including a few big ones, that they need done and hacking together something to get it kick started would be awesome.
• Or finally, we could create an evening every week where people bring their laptops, Bananajour prepared, and hack on whatever they're up to.

Having a hacking night sounds kind of fun to me. The big projects definitely have a lot of potential for great things coming out of them and I plan on getting a Kansas Give Camp organized sometime this winter, but based on the experience I had getting Lawrence Programmers off the ground, I think we're probably better off to start slow, maybe declaring one night a week hack night at a coffee shop or some such.

On Twitter the other day, I said:

I think I might be the first person in history to use John Deer and the iPhone together in the same sentence to illustrate a point.

Yeah, I know, there's a typo there; but I fixed it in the title, that counts for something, right? Anyhow, I thought it would do to explain what I meant.

I read How you know a phone is rubbish the other day talking about the difference between iPhone ads and everything else. The gist: iPhone ads show you an iPhone, Pre, Blackberry, etc., etc., etc., ad infinitum, show you what your life will be like if you purchase their phone.

My favorite has to be this Pre commercial (which for some reason doesn't want to embed correctly). It seems to suggest that all you need to do is purchase a Pre and the world starts bowing down before you in perfect harmony. If only.

John Deere commercials tend toward the same path as an iPhone. Granted, I seem to recall (can't verify as John Deere doesn't seem to post their commercials anywhere online) a few commercials that show husbands getting that little bit of extra adoration from their wives after mowing the lawn and such, but generally John Deere commercials focus on John Deere tractors and the satisfaction you get using them.

I think its fair to say that both John Deere (yes, yes, I know. I'm showing my midwest colors right about now) and the iPhone are iconic symbols. Both are instantly recognizable, and both have rabid followers. To top it off, both rely on images of using their product in their marketing, not trailers for faux-movies or promises of a more balanced life perched upon a rock, to convince you that you should be using their products.

That's going to be the goal for any software products that I create from here on out. No gimmicks to give you a subminial message that the package is superior. Instead, it's going to be all about the package. Use this software because it's good at what it does and you need what it does.

Productivity has been one of my biggest challenges. Well, let me rephrase. Consistently producing extremely productive days has been one of my biggest challenges. I'm an overachiever and since that came naturally, I've never had to be scientific about getting things done. Until lately, at least.

Photo by rintakumpu

The last few years, I've noticed its become harder and harder stay on track. Twitter, IRC, Digg, Email, IM, Facebook, etc., etc., ad infinitum, are always competing for my attention, and when they're all a keystroke away, it becomes too easy to check it "just this once"... again.

They say realization is the first step to fixing the problem. I've been working on that lately. Between launching multiple websites, speaking at multiple conferences this fall, and launching a new company... well, let's just say my time is at a premium. Figuring out how to use my time efficiently has been forced to the top of my priorities.

One tool I've started using is Cultured Code's Things. I switched from OmniFocus because Things focuses on being simple. Dead simple. Yes, I know you use the standard GTD workflow in OmniFocus, but I'm a categorizer. I have over 1,000 tags in my delicious. That's just bookmarks!

Another thing I'm getting ready to start this next week is something I pulled out of Scott H Young's post. He says to setup your tasks in weekly and daily goals. Each day, pick the things from your weekly list that have to get completed, then do them. You don't stop working until those are done, but when they are, you stop regardless of whether it is 10 AM, or 10 PM.

Me: "I see why he left. Go inside the next time you check in." So much for friendly skies. #americanairlines

Jeffery Carouth: What did you do?

Started to respond on Facebook, but ran into their limit, so I'm responding on my blog...

Went to take my boarding pass that he was handing to me, then snapped that back at me. Let me back up with the context.

I was standing in line at the curbside checkin with everyone else from CodeWorks '09. The guy who's line I'm standing in takes a bag back to the conveyor, then disappears through another door. I'm standing around looking dumbfounded while the other two guys are continuing to check people that came in after us.

Finally, I asked him if we lost the other guy or not? "I don't know" then after he finishes with another person who just walked up he proclaims "next". I walk over, start the process, everything's fine.

Then he calls me down to get my boarding pass and baggage check ticket. He starts handing it to me (I thought) and is pointing out what's what. I go to take it from him and he jerks it back and says "I see why the other guy left!" Then mumbles something about this being for express check-in and next time I shouldn't park illegally and tie it up for people who know what they're doing. He was old and not particularly articulate, so that combined with being floored by the lecture I didn't catch exactly what he said.

I rarely fly American anyhow, but you can bet I won't be flying on them again where I pay for the flight.

Twitter has been taken over this morning by #wtc, #sept11, and a host of other related hash tags. Everyone, understandably, wants to share where they were and what they were doing when America's outlook changed forever. Even "God Bless America" has made it onto the trending topics of the day.

We all know the numbers. 3,017 dead, over 6,000 physically injured and many millions more left with scares of a different type, but let's look at some other numbers.

• 1 injured
• 1 dead, 1 injured
• 3 dead, 3 injured
• 101 dead, 536 injured

That's the numbers from the terrorist attacks around the world in the last 3 weeks.

Terrorism isn't going away easily. We can't "God Bless" ourselves out of it. Instead of just remembering where you were that morning, think about what you've done these past 8 years to help create a world where terrorism isn't a normal part of life. Figure out what you can do to make tomorrow better than today, then do it.