Last week I did a post on the many reasons to use Postgres. My goal with the post was two fold:

• Call out some of the historical arguments against it that don’t hold any more
• Highlight some of the awesome but more unique features that are less commonly found in databases.

Once I published the post it was clear and was immediately pointed out in the comments and on hacker news that I missed quite a few features that I’d mostly come to take for granted. Perhaps this is due to so much awesomeness existing within Postgres. A large thanks to everyone for calling these out. To help consolidate many of these, here’s a second list of the many reasons to use Postgres:

Create Index Concurrently

On most traditional databases when you create an index it holds a lock on the table while it creates the index. This means that the table is more or less useless during that time. When you’re starting out this isn’t a problem, but as your data grows and you then add indexes later to improve performance it could mean downtime just to add an index. Not surprisingly Postgres has a great means of adding an index without holding that lock. Simply doing CREATE INDEX CONCURRENTLY instead of CREATE INDEX will create your index without holding the lock.

Of course with many features there are caveats, in the case of creating your index concurrently it does take somewhere on the order of 2-3 times longer, and cannot be done within a transaction

Transactional DDL

If you’ve ever run a migration had something break mid-way, either due to a constraint or some other means you understand what pain can come of quickly untangling such. Typically migrations on a schema are intended to be run holistically and if they fail you want to fully rollback. Some other databases such as Oracle in recent versions and SQL server do support, this. And of course Postgres supports wrapping your DDL inside a transaction. This means if an error does occur you can simply rollback and have the previous DDL statements rolled back with it, leaving your schema migrations as safe as your data, and your application in a consistent state.

Foreign Data Wrappers

I talked before about other languages within your database such as Ruby or Python, but what if you wanted to talk to other databases from your database. Postgres’s Foreign Data Wrapper allows you to fully wrap external data systems and join on them in a similar fashion to as if they existed locally within the database. Here’s a sampling of just a few of the foreign data wrappers that exist:

• Oracle
• MySQL
• Redis
• Twitter

In fact you can even use Multicorn to allow you to write other foreign data wrappers in Python. An example of how this can be done, in this case with Database.com/Force.com can be found here

Conditional Constraints and Partial Indexes

In a similar fashion to affecting only part of your data you may care about an index on only a portion of your data. Or you may care about placing a constraint only where a certain condition is true. Take an example case of the white pages. Within the white pages you only have one active address, but you’ve had multiple ones over recent years. You likely wouldn’t care about the past addresses being indexed, but would want everyones current address to be indexed. With Partial Indexes becomes simple and straight forward:

1

CREATE INDEX idx_address_current ON address (user_id) WHERE current IS True;

Postgres in the Cloud

Postgres has been chosen by individual shops and been proven to scale by places such as Instagram and Disqus. Perhaps even more importantly it’s becoming easy to use Postgres due to the many clouds that are running Postgres as a Service, such as:

• Heroku Postgres
• VMware vFabric
• Enterprise DB
• Cloud Postgres

Full disclosure, I work at Heroku, and am also a large fan of their database service

Listen/Notify

If you want to use your database as a queue there’s some cases where it just won’t work, as heavily discussed in a recent write-up. However, much of this could be discarded if you included Postgres in this discussion due to Listen/Notify. Postgres will allow you to LISTEN to an event and of course NOTIFY for when the event has occurred. A great example of this in action is Ryan Smith’s Queue Classic.

Fast column addition/removal

Want to add a column or remove one. With millions of records this modification in some databases could take seconds or even minutes, in cases I’ve even heard horror stories of adding a column taking hours. With Postgres this is a near immediate action. The only time you pay a higher price is when you choose to write default data to a new column.

Table Inheritance

Want inheritance in your database just like you have in with models inside your application code? Not a problem for Postgres. You can have one table easily inherit for another, leaving a cleaner data model within your database while still giving all the flexibility you’d like on your data model. The Postgres docs on DDL Inheritance do a great job of documenting how to use this and giving a very simple but clear use case.

Per transaction synchronous replication

The default mode for Postgres streaming replication is asynchronous. This works well when you want to maintain performance, but also care about your data. There are cases where you want your replication to be synchronous though. Furthermore, for some cases asynchronous may work well enough where as other data you may care more about the data and want synchronous replication, within the same database. For example, if you care about user sign-ups and purchases, but ratings of products and comments is less important Postgres provides the ability to treat them as such. With Postgres you can have per transaction synchronous replication, this means you could have strong data guarantee on your user and purchase transactions, and less guarantees on others. This means you only pay the extra performance cost where you really care about versus an all or nothing approach you have with other databases.

Conclusion

Hopefully you’re convinced on why Postgres is a great tool, if not take a look back at my previous post.

As for some guidance. I’ve been on nearly all sides of the conference with the exception of organizing, so again organizers please don’t take offense to the feedback.

First on timeline

Please publish this early on your site and on lanyrd

• Deadline for CFP
• Deadline for Call for Sponsorship
• Publish speaker list
• Early bird registration ends
• Regular registration ends

As a sponsor

Early dates and times

As a sponsor please give very early notice of your conference times. As an attendee I can determine within a couple months or even sometimes a few weeks if I’m able to make it to the conference. As a sponsor unfortunately I’m more restricted by budgets and timelines.

Give me options on sponsorship

A prospectus is great, and often times I’m completely happy with it. Other times there’s things I may want to sponsor that are not on the list. If videos aren’t already being recorded I’d love to see the content live on, and this is an immediate place that jumps out as valuable to sponsor. Often times flexibility doesn’t cause me to sponsor or not, but it does leave a clear reminder of my experience.

Location of exhibit hall

Sometimes the exhibit hall is hidden away and only visited by attendees that really seek it out. Obviously this is less than ideal for a sponsor with a booth. The obvious solution is a central expo hall, but many conferences can go one further and put lunch and the expo hall in the same area. Having foot traffic near and through the expo hall gives slightly more exposure, letting sponsors get a better value.

As a speaker

Have previous years talks available

If you are interested in attracting new speakers to your conference, please include last years talks. As a speaker there’s often a new conference I’d be interested in attending, if I’d not attended or spoken I may not know appropriate context. Keeping last years speaker list and talk topics helps me elect whether it might be a fit.

Publish the CFP

This is especially key to have on lanyrd, above all the other timelines. There’s currently no better place for me to look today to get an idea of when CFPs are coming up. As a tip for others looking to submit talks to conferences check out their list of upcoming calls.

Bonus if you give me a signup form to get notified via email when the CFP is open

Turnaround

I understand there’s a lot to do when organizing a conference. As much as possible keep the turnaround fast on these. There’s a lot of effort involved in reviewing talk submissions, so I understand that its not a 1 day activity. However, far too often I’ve reviewed talks for conferences and given feedback, and seen most of the activity occur in a mad sprint at the end of the planned time. If this is going to occur at least compress that time.

Speaker dinner

As a speaker, I’ll be missing a few talks as I prep for mine and of course give mine. Additionally there’s a lot in common with other speakers at the conference often. A good speakers dinner with an opportunity to connect with them can be what sets the conference apart for me. Of course good food and drinks always helps this, but most importantly it does a good job of bringing all the speakers together.

Notice

If your CFP is well in advance of the conference, advertise it early and often.

As an attendee

Power and Internet

Power any and everywhere. Internet that works. I understand its hard, but everyone remember when it works well, so its one straight forward way to make your conference stand out.

If you really want to deliver a great experience, have a charging valet, let attendees drop off a device and pick it up an hour later.

Hallway Tracks

Talks are awesome, but give opportunity to connect with everyone there. Many of the people at a conference I see once at year at that particular conference. If my choice is between a talk and catching up with an old friend, the old friend may win out. Give me the opportunity to do both.

Evening events

I don’t want to get into the debate about drinking (coffee or alcohol), but evening events encourage socialization. Adobe’s MAX conference was a great example of this, while there was beer, I never saw over the top drinking at the after conference event. To keep it fun there were Xboxes, PS3s, Segway obstacle courses and much more. If this isn’t something the conference wants to condone or organize itself, there’s likely an event happening somewhere, help with the publicity of those. With any luck the non-conference evening events are well done, and the conference spirit continues into the evening.

UPDATE: A part 2 has been posted on Why Use Postgres

Very often recently I find myself explaining why Postgres is so great. In an effort to save myself a bit of time in repeating this, I though it best to consolidate why Postgres is so great and dispel some of the historical arguments against it.

Replication

For some time the biggest argument for MySQL over Postgres was the lack of a good replication story for Postgres. With the release of 8.4 Postgres’s story around replication quickly became much better.

While replication is indeed very important, are users actually setting up replication each time with MySQL or is it to only have the option later?

Window functions

This is a feature those familiar with Oracle greatly missed in Postgres. In fact even SQL Server had some form of them, though it was with T-SQL, which adds a bit more complexity to the feature. This is a feature that once you have you can’t live without; the other options that existed before were slower and much more complicated. With the release of 8.4 window functions were added to bring Postgres on par with Oracle in this area. For more info on using them check the Postgres docs above or PostgresGuide.com.

Flexible Datatypes

Creating a custom column is simpler in Postgres than any other database I’ve used by far. Excluding custom datatypes, even Postgres’s out of the box datatypes make Postgres stand out far ahead of other databases. In particular the ability to create a column as an Array of some datatype. Want to store a game of tic-tac-toe in a database, or an array of 1 user’s phone numbers? It simply becomes a single column that can contain multiple phone numbers for a user.

Functions

Need to do some logic outside of standard SQL? Postgres likely has a function already available to do it for you. What about the time you wanted to take all rows returned by a query and combine them into a function? Give array_agg a look. Need to split a string and grab a part of it or some other string action, there’s a function for that.

Custom Languages

Want to use another language inside your database? Postgres probably supports it:

• Python in Postgres
• Ruby in Postgres
• R in Postgres
• V8 in Postgres

Extensions

Need to go beyond standard Postgres? There’s a good chance that someone else has, and that there’s already an extension for it. Extensions take Postgres further with things such as Geospatial support, JSON data types, Key Value Stores, and connecting to external data sources (Oracle, MySQL, Redis). I could easily have a full post on extensions available alone, fortunately someone else has already created an awesome one - PostgreSQL Most Useful Extensions.

NoSQL gives flexibility

I don’t want to get too NoSQL versus SQL debate…. no matter which side you fall on you can get both in Postgres. With hstore and PLV8 you’ll get the flexibility in your data that you would with Mongo along with all of the above features. Will Leinweber has a talk that he’s given at several conferences recently that highlights Schemaless SQL.

Custom Functions

Didn’t find the function you wanted in the above? Try creating it yourself:

1
2
3
4
5

CREATE FUNCTION awesomeness(varchar) RETURNS boolean
    AS 'CASE WHEN $1 == \'postgres\' THEN TRUE ELSE FALSE END;'
    LANGUAGE SQL
    IMMUTABLE
    RETURNS NULL ON NULL INPUT;

Common Table Expressions

Often times when exploring data or creating a new view you’ll want to load data into a temporary table. When exploring data you only need this for a temporary time. Why actually go through the effort of putting it into a temporary table, especially if you only need it for a single query. Common Table Expressions let you accomplish just that.

Development Pace

For some period of time MySQL and Postgres were both moving at fast paces. In recent years though Postgres has rapidly picked up its pace of how much gets packed into a single release. Just take a look at the Major Releases.

Conclusion

Hopefully you’re convinced on why Postgres is a great tool. Next take a visit to PostgresGuide if you need some direction on where to start or how to use many of the above features.

When I first came across Django I was an immediate fan. It featured:

• Good documentation
• Steady but stable progress
• Community around apps which encouraged DRY

I’ve been a user off and on depending on my needs for nearly four years since discovering it, and throughout that time all of the above have remained true. However, as I’ve worked on and encountered more complex applications there’s one thing that has time and again broke down for me, which is the Django apps model. It hasn’t broken down due to Django only though, I’ve seen it break down in Ruby (Rails), Java, .Net, take you’re pick of language or framework.

The breakdown of this model is due to several things:

• Successful applications grow which mean more complex applications and more developers
• More complex applications often mean larger code bases
  • Deprecating code is good, but not always easy in large code bases
  • More code means more testing, but slower releases

At Heroku one way we often describe the platform to others is “A distributed Unix in the cloud.” There may be many reasons for this, but one of which is that we love the Unix approach and philosophy of Small sharp tools. Sticking to that, many of our internal pieces are small individual apps that talk across defined contracts or APIs.

Back to Django’s app structure… Many people build apps and re-use them and often share them with the world. This is truly great for re-usability, which means you can focus on building key features. However, this does not enable your application to be more maintainable in the future nor does it enable scalability. Yes, you can absolutely scale a monolithic application, but it doesn’t mean you should. This doesn’t mean the app structure is entirely broken, it just means that it is a partial step to where you should be. The real solution is to build more of these pieces of your greater application as services.

A Django app is defined as A web application that does something. I.e. Weblog, Poll, Ticket system. Within Django an app contains:

• Models
• Views
• URLs

I couldn’t find a great definition of a Service that was succinct and also said something of value (If you have one please pass along as I’d love to have a definition from a source other than myself). For the sake setting something in place I’m defining a service as Method of communication over the web with a provider using a defined contract. By this definition a service contains:

• Provider
• Endpoint
• Contract

Let me clarify this a bit further…

Tangible example/parable:

Django Apps::

• Ticket
• FAQ

Company Teams::

• Support
• Community Evangelist

You start with two apps, that maybe share a little code. Moreover they at least exist in a central code base. Then you deploy something and the Ticket app can no longer create FAQ, due to a change in one or the other. There’s no finger to point, but more importantly, you don’t know how to contact to resolve. Neither team wants to deploy, so you test more. Before every deploy you run tests… and validate a build… and deployment slows… well maybe not with two teams. But as you get to 5 teams it does, and more so with 15, and more so with 30 teams. Then you hire a build master and release master, who really wants that?

So within Django maybe you go from apps all in the same codebase to releasing private versions of apps…

Your requirements.txt for a main site looks like:

1

FAQ==0.2

You have 3 apps which depend on it, support, marketing, billing. You bump a version FAQ==0.3 but then all three or no teams have to upgrade the version to the new APIs. However if your interface was:

1
2
3
4
5

data = {
‘question’: “my question”,
‘source’: 123
}
requests.POST(os.environ[‘FAQ_API’] + ‘/v1/create’, data=data)

You could also have:

1
2
3
4
5
6

data = {
‘question’: “my question”,
‘source’: 123,
‘related’: [456, 789]
}
requests.POST(os.environ[‘FAQ_API’] + ‘/v2/create’, data=data)

Then you can easily support both, deprecate v1, and track its usage easily. This doesn’t guarantee, but it does enable re-usability, scalability, maintainability. And of course continues to let you build features instead of maintaining software.

In the next post I’ll go into a bit more detail of how a real example looks with apps in both forms, using a set of Django Apps and using a set of Services built on Django Apps.

Slides from a corresponding talk at DjangoCong are here

• Python Build Pack
• Ruby Build Pack
• Scala Build Pack

There have even been some created as fun weekend hacks such as the NES Rom Buildpack.

Recently at Heroku my teams have started exploring new forms of collaborating and documenting. In particular editing a wiki for communication is contrary to our regular workflow. Much of our day is spent in code and git. To edit a wiki within a web browser and using some markup we’re less familiar with is an overhead we were aiming to reduce. As a result we’ve tried a few things, the first was simply using a github repo to edit markdown.

Personally I have always been a fan of Sphinx documentation. However, Sphinx has no means to secure a site out of the box. Generating the static site then running it being a Rack app to secure it seemed like a few extra steps that would hinder workflow. As a result I set out to build the Sphinx buildpack which would let you push a Sphinx project to Heroku and automatically run your documentation. The buildpack itself supports two modes, public documentation and a private documentation. To have your documentation secured in private mode you simple need to add your google apps domain as a config var heroku config:add DOMAIN=mydomain.com.

If you need more information about setting up OpenID check out my recent post Securing your organization with OpenID

1
2
3
4
5
6
7

$ sphinx-quickstart
$ git init .
$ git add .
$ git commit -m initial
$ heroku create -s cedar -b http://github.com/craigkerstiens/heroku-buildpack-sphinx.git
$ git push heroku master
$ heroku open

In our personal lives we use email as our keys to the kingdom. In fact I now almost refuse to sign up for any service that doesn’t let me use oauth, so why should a work place be much different. So I inquired with a few companies to see if they were fine with securing things like documentation or wiki’s being google auth, they indeed were. Yet they still seem to have users keep one more username and password for their VPN to be able to login to access internal docs/tools.

Most tech centric companies grow their own apps for many things they do within a company. Even the heavier adopters of SaaS still end up building a lot of internal systems. So why not secure them with your email domain just as you commonly would if it were a public service?

The problem comes in that OpenId with google has an initial setup overhead, but after that works unbelievably well.

The catch

In some cases you currently have to identify your domain as an OpenId provider. This means that @yourname.com is an OpenId provider. This simply means creating a url route for openid in your base site similar to the below:

1
2
3
4
5
6
7
8
9

<?xml version="1.0" encoding="UTF-8"?>
<xrds:XRDS xmlns:xrds="xri://$xrds" xmlns="xri://$xrd*($v*2.0)">
  <XRD>
    <Service priority="0">
      <Type>http://specs.openid.net/auth/2.0/signon</Type>
      <URI>https://www.google.com/a/craigkerstiens.com/o8/ud?be=o8</URI>
    </Service>
  </XRD>
</xrds:XRDS>

This is due to an issue of OpenID discovery which you can read more on at: https://groups.google.com/group/google-federated-login-api/browse_thread/thread/4a7dd2312a47a082/9285cec18a30b9d3?lnk=gst&q=apps+discovery&pli=1#9285cec18a30b9d3. In short, setting up the above can save you a lot of time

Setting up in apps

Most web frameworks have libraries that make it easy to secure your apps with openid/oauth. In particular Django and Rails both make this pretty easy. To make this even simpler for you below is code to actually secure an internal app for both Django and Rails. You can do similar with Flask or Sinatra as well.

Rails

In case your admin controller isn’t already generated:

1

rails g controller admin/users

Then anything you want to secure:

1
2
3
4
5
6
7
8
9

module Admin
  class UsersController < ApplicationController
    before_filter :admin_required

    def index
      render :text => 'Hello from the admin panel!'
    end
  end
end

Django

Finally sync your database:

1

python yourapp/manage.py syncdb

Secure any view with the login_required decorator as your typically would with Django.

Summary

In short with some very basic app setup you can have an internal workflow thats just as good as what you use in your day to day outside the office.

Our goal in hiring is seldom to fill a role, but more commonly to find more talented people share our goal (changing the world for developers).

So what’s our hiring process look like….

1. Review Resume/Github Profile
2. Initial Screen
3. Second Screen
4. Starter Project

While there’s definitely a process that we follow that’s not what’s interesting. We way too often get worried about the steps 1, 2, 3… Instead you should focus on what’s important: are they a fit? Can they get shit done? Who cares about how many phone screens someone goes through?! Five phone screens instead of two doesn’t make them a better fit for your company. The short of it is they go through enough screens that you feel comfortable and you progress them through the process. For us at any point in the process if someone is determined to not be a fit the process ends there. If the process does end the hiring manager will relay this in the appropriate form, though always in writing via email as well.

The hiring manager could debatably be the biggest difference between our process and others. When a candidate applies to a position it goes to the hiring manager (not an HR person). The hiring manager will be your manager once at Heroku it’s one in the same and this ensures from the start of the process the candidate and the manager mesh well. Yes, having the manager of a group review github profiles and resumes is extra effort, but who better to judge from a quick glance than engineers. In general as a manager you’re evaluated on the success of those you manage, as such you should be invested heavily in those you hire. In addition to this we find a big difference in the on boarding process and how quickly someone can succeed. We have used many approaches, but the success of someone at Heroku based on having their hiring manager and manager be the same individual is best highlighted below:

While every step in the hiring process is valuable starter projects may be the most valuable to ensuring quality. The final step with nearly everyone we hire is to invite them to come hack with us. Instead of parading someone around for a day long interview we get down to business and write some code. It could be something internal to Heroku, it could be an open source project we use, it could be something interesting that the candidate feels would add value to Heroku. Starter Projects vary slightly between each hiring manager.

Several of our managers prefer to lay out several potential interesting projects, talk through them with the candidate, and then let the candidate decide what they’d like to work on. Sometimes there’s a pressing need that the candidate can jump right in and add some value. It’s always important that the starter project is achievable, if it’s too broad of difficult for a 1-2 day period then the manager has failed in the hiring process. Regardless of the project it’s far more than an exercise on a white board, it’s actually what life is like at Heroku. We have lunch at the same table that we eat at every other day, we interact just as we normally would, and after work there may or may not be drinks just like any other day.

As a slight aside, we even conduct starter projects when current Herokai move from one team to another

Starter projects usually last anywhere from a day to several days. At the end of a starter project the candidate presents what they did, in a similar fashion to weekly demos that occur at workshop (more on that some other time). In earlier days it was nearly the entire company that would sit in, ask questions and give feedback. Now it’s a bit harder for all us to fit into one conference room, though there’s an open invite and anyone that wishes can sit in (often 10-20 Herokai). At the end of the starter project there’s no question that the candidate fits or doesn’t, often from both sides. Of course if it’s a fit we make an offer and welcome them into the family.

Maker’s Day ensures that engineers get a full day of uninterrupted time to focus on making things.

The more consistent interruptions are throughout an engineer’s day, the more time will be lost due to context switching in addition to the time spent on those other activities. These interruptions may include a quick question from a manager, a question on a code problem someone else is working through, or an email or IM from a coworker. Regardless of the type of interruption, it causes an engineer to lose focus. According to Peopleware: Productive Projects and Teams, in a study regarding productivity among engineers, the top performers when surveyed said they were interrupted regularly 38% of the time versus the bottom performers, who were interrupted 76%. Context switching should be counted as fully wasted time for an engineer, and all too often as the number of meetings increases, the time involved with context switching is increased similarly to the following:

For more on how interruptions or context switching decreases productivity, Jeff Atwood has a great post about The Multi-Tasking Myth, which demonstrates …

Most people understand that context switching is bad, but another team may still have valid demands on your time. Pushing back against another team or manager isn’t always feasible; after all, we do work together, and each team at times may need something from another team. This is where Maker’s Day starts to come in. Every Thursday at Heroku is Maker’s Day.

Maker’s day is meant for making shit. Meetings don’t happen on Maker’s Day. If someone asks if that time on your calendar works for a meeting, the simple response is no–it’s Maker’s Day. Because Maker’s Day has been ingrained into our culture, engineers have no problem giving that response when there’s a request on their time on Maker’s Day. If someone in marketing, sales, or another non-engineering role wants to book meetings, they’re welcome to do so, but they’re going to be without engineers. However, even for non-engineers, Maker’s Day is equally invaluable; uninterrupted hours of focus at a time are amazing for productivity in any role.

Maker’s Day varies in how it is executed from person to person. Often the office is slightly less busy due to some engineers working from home or coffee shops to maximize their productivity. To an outsider, the office may appear business as usual: engineers sit at their desks, working. At lunch, everyone is sitting around the lunch table eating together. To the unobservant eye it may appear to be just any other day, but the engineers notice the difference. There will be significantly less interruptions by someone walking over to your desk, you won’t be pulled into meetings that distract you from features, and you know it’s an opportunity to accomplish a bulk of work laid out from your weekly planning meeting.

As Heroku has grown, meetings have increased, and the value of Maker Day’s has increased exponentially.

Whether you’re in the early stages of bootstrapping a company or at a large company of thousands of engineers, one of the best practices anyone can put into place is dedicated quality time for engineers to produce code. Maker’s Day is a fantastic way to ensure this happens on a weekly basis.

Most packages you install should be on pip and an actual released version of a package. There are times however when you may need to test out a package that is still being worked on. For these cases there’s what developers commonly do and what you should do.

If you must install a package from git then you might use something similar to the following:

1

git+https://github.com/kennethreitz/requests.git#egg=requests

Even more ideal is specifying the version:

1

git+https://github.com/kennethreitz/requests.git==0.6.0#egg=requests

The above should never be done when deploying to production, it reduces predictable behavior and requires that github is available for you to deploy. You should never be dependent on more than PyPi and your own system to deploy.

Instead when you deploy to production you should point to an internal link. This looks something similar to:

1

--find-links http://path/to/package/

The other item thats not currently (though will be soon) a default behavior is mirrors. For those not familiar, a mirror is essentially a copy of a site (in this case PyPi). At some point in the future the site may be down preventing you from installing packages or deploying code. If this is the case you’re able to use a flag to use these mirrors. If you simple add the flag below it will automatically use the mirrors, allowing you to continue:

1

pip --use-mirrors install Django

I’ll be diving into many various practices that enable Heroku to put quality engineering above all else, but first let me highlight the team structure and tools that enable this.

Heroku is comprised of many small teams internally, each team operates much like an individual entity. The team chooses its own tools and best method for communication, though as a whole some form of Scrum is run throughout teams. Think of the unix philosophy of small sharp tools as in The Art of Unix Programming applied to teams and people.

For most teams this involves a weekly planning meeting earlier in the week. In such a meeting teams may conduct a retrospective, opportunities to improve the process the coming week, and of course plan tasks for the coming week. Its very important to note that planning tasks for the week doesn’t necessarily involve planning the deadline for them, but rather simply laying out what people are working on (more on this in a future post). Each team will record and track this in a tool of their own choosing. Several use pivotal tracker, one uses scrumy, some use email to distribute and track against personal to do lists. The method for tracking issues is again entirely up to the individual team. A one person team may choose to use a simple to do list, larger teams commonly use github issues and pull requests. Given the team is the one responsible for their own productivity the team is the one to choose what tools they use.

Meeting loads vary from person to person depending on what is the demands are on their time, though everyone at Heroku participates in some form of standup. Most teams do these daily as quick status stand-ups of what was worked on the day before and whats to be worked on the next day. In addition to the planning meeting and stand-ups, there is often collaborative engineering, and company wide gatherings.

Collaborative engineering once again varies depending on which engineers are working together. Engineers may get in front of a white board or in front of machines and simple collaborate. For engineers together in the office this is often the most productive way. These practices work the same for remote employees, though slightly modified for the high touch interaction. For remote employees this often works as pair programming via Skype. Skype is indispensable for allowing remote workers to feel far less remote. Skype alongside typewith.me and you have an unbelievable easy to collaborate not just with 1 other, but with multiple parties to work through a document on a given topic. For smaller activities of communication asynchronous is key. This ranges from campfire most commonly during common working hours when someone is likely to be at a machine, to email when the return on a request may take slightly longer.

Finally there is the all common company wide meeting, which occurs weekly. The structure of this varies from status updates to broader ongoings. Its often the perfect time for engineers to hear about what sales is doing or get updates on teams you don’t commonly interact with. Along with common status updates there will be broader company updates.

Consistently across all teams you’ll find these principles which allow us to ensure the quality of engineering as we continue to grow:

• Small teams that talk across defined API’s and data contracts
• Teams using the tool that they believe is best for the job
• Frequent asynchronous communication
• Collaboration (including for remote employees)

The key in Heroku running efficiently is primarily allowing each team to run as it chooses. Heroku works because we have talented engineers, the best thing we can do for those engineers is allow them to work productively. Often only they know the best way to accomplish this, so who better to let them accomplish it than themselves.

All major Python packages are hosted on PyPi (Pronounced Pi-P or Cheeseshop). When you use a common python installer it will:

1. Search for the package you specify
2. If you specify a version will use it, otherwise will use the latest
3. Will download the source for that package
4. Install it into your Python environment

Now for actually installing… Lets get started with installing the three packages below. At this point you should at least have a fresh Python environment, however you don’t have an immediate way to install packages. The defacto Python package installer is pip.

Earlier we setup virtualenv to help isolate our python packages we were working with. First lets go ahead and create a folder for our project then setup a new environment for the project we’ll work on:

1
2
3

$ mkdir myapp
$ cd myapp
$ virtualenv --no-site-packages venv

If we list the contents of the directory you’ll now see a folder venv. Within this folder you’ll find all the parts of the environment that virtualenv just created:

1
2
3
4

$ ls
venv
$ ls venv
bin   include lib

Now you’ve got a sandboxed environment that exists but you haven’t loaded it. You can now activate and deactivate this any time you like. Once you do this it customizes your path to use the packages you’ve installed for this environment. To load your environment when in the myapp directory:

1

$ source venv/bin/activate

To deactivate this simple:

1

$ deactivate

Now that we’ve got your environment loaded installing your packages should be pretty simple. Ensure that you have your virtualenv loaded and then run:

1
2
3

$ pip install requests
$ pip install PIL
$ pip install Django

Now that you’ve installed your packages you want to be able to share this with others to make it easy to get setup. You could provide a list of everything your application needs to run manually, or because its Python you can expect it to make it easy for you. Pip has a wonderful command freeze that will show all of your packages and their versions that are installed. Simply run:

1

$ pip freeze

However, this only outputs the information. Along with this pip has a canonical form for listing requirements and installing them from a file. The filename is commonly a requirements.txt. To create this we simply pipe the results of pip freeze to this file.

1

$ pip freeze > requirements.txt

Next we’ll talk about a few more advanced items in dependency management, then finally we’ll get started on building an application.

1
2

$ python --version
Python 2.7.2

As long as you see a Python version 2.5.x-2.7.x you should be fine to continue. From here we’re going to work through setting up your Python project environment. For this we’re going to use virtualenv. For those of you not familiar virtualenv is a self-contained python environment. It holds its own copy of python and any libraries you install. This allows you to work on multiple projects with different versions of libraries.

While we’re installing virtualenv we’re also going to go ahead and setup PostgreSQL as we’ll be using it later. If you’re on a mac you’ll first need to setup homebrew. Homebrew is used for installing various system packages. If you’re on linux, in particular Ubuntu you can skip down to the steps for setting up your environment.

First for Mac users lets setup homebrew which will allow us to install various system packages:

1

$ /usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)"

OSX with Brew:

1
2
3

$ curl -O http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg | sh setuptools-0.6c11-py2.7.egg
$ easy_install virtualenv
$ brew install postgresql

Ubuntu:

Now for setting up your project:

1
2

$ sudo apt-get install virtualenv
$ sudo apt-get install postgresql

Now you have Python, virtualenv, and postgresql all installed. We can now focus on setting up the initial start to a project.

In the next part we can start with installing some Python packages.

• Amazing documentation, there’s not 30 different blog posts with different ways to do things, either read the django project documentation or the app documentation
• They don’t consistently break backwards compatibility. While at times they do this, it is not the norm or standard, which isn’t the case for some other unnamed frameworks
• DRY, Don’t Repeat Yourself. If there’s an app that exists people don’t have a burning need to recreate the same functionality, resulting in lower number but higher quality pluggable apps.

However, after using the framework for nearly 4 years I’m just now discovering my preferred way of managing environments. I know there’s still a bit of back and forth on development environment/IDE, but as far as configuring actual project environment I’ve become very comfortable with what I’ve now been using for many months. It also allows for someone else bootstrapping their environment incredible quickly as well. Below is a quick cookbook of how to do this on OSX and Ubuntu.

OSX with Macports:

1
2
3

$ sudo port install python27
$ sudo port install py27-virtualenv
$ sudo port install postgresql90

Now for setting up your project:

1
2
3
4

$ mkdir example
$ cd example
$ virtualenv-2.7 --no-site-packages .
$ source bin/activate

OSX with Brew:

1
2
3

$ sudo brew install python
$ sudo brew install virtualenv
$ sudo brew install postgresql

Now for setting up your project:

1
2
3
4

$ mkdir example
$ cd example
$ virtualenv --no-site-packages .
$ source bin/activate

Ubuntu:

Now for setting up your project:

1
2
3

$ sudo aptitude install python
sudo aptitude install virtualenv
sudo aptitude install postgresql

Now for setting up your project:

1
2
3
4

$ mkdir example
$ cd example
$ virtualenv --no-site-packages .
$ source bin/activate

For those of you not familiar virtualenv is a self-contained python environment. It holds its own copy of python and any libraries you install. Now that you’ve setup your virtualenv we can go through the process of installing django and setting up your repository. This is the same across all of the above platforms:

Add to a .gitignore file:

1
2
3
4
5
6

bin
build
include
lib
.Python
*.pyc

Add to a requirements.txt file:

1
2

Django==1.3.1
psycopg2==2.4.1

Then run:

1

$ bin/pip install -r requirements.txt

This should fully installed any of your required apps and makes it easy for others to do the same to begin contributing to a larger app. Finally if you like you can create your git repo from this an make your first commit:

$ git init
$ git add .
$ git commit -m 'my first django virtualenv'

The very general meaning behind attribution is to give credit. When it comes to web products this can be giving credit for lots of things:

• A visit to your site
• A purchase to your site
• Other events, such as sign up, referring friends, etc.

Its really wonderful when there’s a direct mapping in correlation. Take for example the case where:

1. User sees your ad on facebook
2. User clicks on ad
3. User signs up for an account
4. User pays you money

In this case it’s clear that its worthwhile to advertise on facebook. The hard part becomes when there’s a non-linear path in some form (which is almost always the case). Now take the case:

1. User sees your ad on facebook
2. User clicks on ad, then leaves
3. User comes back a day later directly, and spends 10 minutes on the site reading
4. User comes back a day later after searching via google, and creates an account
5. User sees a re targeting ad 1 month later and visits
6. User pays for account

In a perfect world you have case 1 and can give the full credit or attribution of the purchase to your facebook ad. However, in a real world you have case 2 and it becomes more complicated. Perhaps its something like this though:

• Account creation - 100% to facebook ad
• Paying for account - 50% to facebook ad, 50% to retargetting?

This is where there is not an exact science to attribution, as it could be justified to give 100% of the credit for paying for an account to the re targeting. So if this has you pondering on attribution here’s the downer part…. The tools that allow you to attribute a goal or action to multiple sources:

• Omniture
• Web Trends

Most other tools are the last place the user ever came from, some occasionally are the first place a user came from, but without pony-ing up for a high end product your options on actually getting data don’t really exist.

Disclaimer: I have little to no formal training in stats or analytics, have simply learned through launching products so take this for what its worth, someone that has been there and done it.

Part 1 - Focus on SEO

• Ensure you have a sitemap; http://www.google.com/support/webmasters/bin/answer.py?answer=183668
  • Submit it to google
  • Submit it to yahoo
  • Submit it to bing
• Use meta tags and ensure they’re unique
• Use unique titles and h1/h2 tags in your pages

Part 2 - Find Influencers

• Find the key bloggers
• Be brief and to the point
• Provide value to them, often by an exclusive

Part 3 - Traditional Advertising

• Contextual - AdSense, target for competitors
• Demographic - Cast wide net for demographic info, narrow on interests/hobbies
• Vertical - Ensure to filter out noise and focus on brand

Part 4 - Retargetting

• No matter what do this step, it make’s you appear to have a larger presence than you do to users
• Most recommended service: AdRoll

Again this is just a very high level recap. If any of these items strikes a chord with you please dive deeper. The amount of time it takes to learn about how to market and target your users is a fraction of the return you get from doing it effectively. Finally, if anyone out there needs direction on how to target and measure more effectively always feel free to reach out to me, i’m easily found on twitter among many other services.

Within measuring you’re metrics you’ll have abandonment at each level. You may have some visitors that never register, and many that register but never purchase anything. It’s wonderful if you’re able to immediately have full insight of the best means of marketing to drive revenue, however realistically it occurs in a more phased approach. The first step is to drive visitors and almost immediate second is to convert those users as registered.

Acquisition -> Activation

This at a high level makes sense, as does at a high level knowing you should be targeting users in your target market. However, even slightly drilling into this you realize that all traffic is not equal. It’s often known that CPC and CPA advertising does not convert well for users, though can drive traffic. This may not always be the case. For Registry Stop to make this analysis easier we’ve created a custom report in google.

To create your own custom report simply click Manage Custom Reports under the Custom Reports area then “Create new custom report”. The custom report ability in google gives you much more ability to drill into the data that you already have at a higher level. To track effectiveness of converting visitors to registered users and which sources are effective at this you’d create something that looks like:

Here while very simple we’re able to see some very key information quickly. Here’s an example of how it would appear over a few day period:

As you progress in your stage of bootstrapping or growing your startup you’ll want to grow custom reports that allow you to report against as many metrics as possible. The first and last part of my day is spent pouring over these reports. Having this data readily available allows us to drive our business based on data. Perhaps the hardest part of all of this is admitting when the data is counter to what we expect and following its advice.

A personal recommendation is actually to use both, goals and funnels. The key a funnel is that you need to have successive steps that occur in some order. With regards to metrics tracking this is absolutely needed, but typically you may have 1-2 total funnels with many steps in your site versus goals where you could have 10-15 single goals. For Registry Stop we’ve structured our site so that our earlier stage goals become the same as steps in later stage funnels. For us in almost all cases the first part of the funnel is the visit, the second is registering for an account. We do have independent goals for visits and registrations as well, but we do not have funnels on those goals.

A key to getting the most use out of your funnels is to know that there is a workflow to follow to getting to that end goal. To highlight this slightly more visually let me walk through an example:

1. User visits your site.
Here you’ll want to create a goal. Likely this is tracking a page view of the homepage.
2. User registers for an account.
Here again, you’ll create a goal and nothing more. If there is a confirmation page that will be your goal, if there is not, then you’ll want to fake a page view in order to track that goal.
3. User adds a product
This is the first place you want to create both a goal and a funnel.

With regards to the goal, it should be pretty simple, tracking some page view. For the funnel though you will likely have 4 parts.

The parts of your funnel will be:

1. User visits your site.
2. User registers for an account.
3. User begins process of adding a product
4. User completes product add

For Registry Stop we have a similar funnel for users which includes:

1. Registering for an account
2. Visiting the sync page
3. Adding a synced registry

Now as for actually setting up a goal or funnel. This is slightly unintuitive as it’s not from inside a specific site on Google Analytics. At the home level of google analytics you’ll have all of your sites setup. For each site to the right you’ll have actions, click the Edit to go to the goal area.

Once here it becomes a bit more intuitive. You can begin by simply adding a goal. Your goal types should seem mostly intuitive, as mentioned in an earlier post you cannot use an Event in a goal. For this reason you can fake a pageview as if it actually occurred and then create your goal against that non-existent page view. If you want a few more details of how to do this check out the previous post . So for an example, we have a goal on Registry Stop that detects when a page view occurs as a result of a registry being synced. Because this workflow is heavily javascript and flow based we fake the page view and track it as if the page was actually visited. The goal itself looks like:

For setting up your funnel as we mentioned above its generally a set of page views. A very key item is the check box next to the first item in your funnel. If you check this is means any other steps in the funnel are not counted unless you the first step is completed. If you have a very structured 1, 2, 3 workflow this makes sense. However, if there are various ways for the goal to complete then be very careful about selecting this.

For this same goal above we have a corresponding funnel to track in detail how our conversion flows. The funnel itself looks like this once setup:

This results in a funnel report that looks like:

• Django
• JQuery
• JQuery UI

For a view to search within your django model it would look something like:

1
2
3
4
5
6
7
8

from django.utils import simplejson
def autocompleteModel(request):
    search_qs = ModelName.objects.filter(name__startswith=request.REQUEST['search'])
    results = []
    for r in search_qs:
        results.append(r.name)
    resp = request.REQUEST['callback'] + '(' + simplejson.dumps(result) + ');'
    return HttpResponse(resp, content_type='application/json')

For the jQuery autocomplete and call:

In contrast the wedding industry is very much a pay to play space. If you give some money you can get some attention.

First things first, contact the blog you’re interested in being written up in and ask for their media kit. If they welcome sponsored posts, then it is likely called out in their media kit. However, this isn’t always the case, if you’ve noticed posts on their blog that have been sponsored posts but pricing isn’t called out in their blog then email them explicitly and inquire.

Once you’ve got their media kit its time to do some digging. My process has first been to validate their numbers. Most blogs include their unique visitors and page views in their media kits. I immediately jump over to compete to check if their numbers are even in the ballpark. To clarify in the ballpark can be somewhere around 1.5x of compete. There are a surprising number of blogs that may be 20x off in the numbers they are stating. This could mean you’re outright lying on your stats, or that you’re not running a solid enough business that you know how to effectively track your numbers. It could be, because your blog exists on 5 different domains, while compete I’m checking only the primary. Regardless, if you’re numbers aren’t close, it often means you’re not as together as we’d like.

If they pass the first smoke screen of the stats being in the ballpark, then we can move on to evaluating sponsoring a post. Traffic’s a big factor, unique visitors are important as well as page views. Next we’ll typically look for how active your users are. Do users actively engage in comments, where there are active commenters there’s usually opportunity to get a bit more out of your post.

Next would be, how frequent are posts. Are you looking at 1-2 posts that go up per day, or 10. If 10, it simply means your content will be pushed to the bottom of the page pretty quickly, in this case if page views are exceptionally high, it may mean that users only view 1-2 posts per day and miss the others. I’ve historically done this on a subjective basis, but it could easily be a number that is calculated and factored in.

So you start with this basic methodology for one blog, then do it for a few more. Its pretty simple to compare 3-4 blogs on their potential value, but when you really start expanding this you could be looking at 100 blogs. If that’s the case it does help to have some structured method. We typically weight their unique visitors, page views, a factor of how accurate they are against compete, their commenter level, and finally their post frequency. We multiply that weight against the cost of a sponsored post and there you have your priority in terms of which blogs to begin advertising on. Its usually best to try 2-3 blogs to determine your return. But even one can give you an idea of results.

A quick recap of the basic formula:

• Factor of stated traffic compared to compete traffic
• Post Frequency (less is better)
• User engagement (measured by comments)
• Page views relative to unique visits
• And of course, cost of the post

As a tangible example, we’ve had variations of 2x from something that ranked nearly the same through the above calculation. Another interesting note, is even months after a post we have seen slow and steady referral on some posts. This method is definitely not ideal for all companies to acquire users, but for many should definitely be explored. And remember, you should always track these against your metrics to know true effectiveness.

Retargetting Retargetting is the idea of showing an ad to a user that has already visited your site. It’s pretty basic, someone comes to your site and you have a pixel that loads telling your ad network they’ve visited. From then on they may see your ad when randomly browsing the web. With my most recent venture into the online registry space I was browsing Chiacgo Tribune and Slashdot and came across our ads. There’s absolutely no contextual relevance there, but I checked and it was due to our retargetting. This will make it appear as if you are everywhere to your users. If you do want to heavily monitor what retargetting is doing for you, the best place to do it is around your retention metric.

How do you do retargetting? Sounds like a complicated process slightly… Well it’s simple you don’t, let one of the major ad networks do it for you. The first step for it, is to use image ads. Text based ads may work great for google and facebook, but on most of the websites your retargetting will run on you want to have images that the blogs typically serve. The 3 key form factors you’ll want are:

• 300x250
• 728x90
• 160x600

As for who to use for a service? Google recently launched their service in the past year and called it ReMarketing. I’ve had mixed results with google with regards to their approval process on images. Many times I’ve had images sit in a queue for a full month, then finally reached out to their support and have it take another 3 days. For this reason alone, it almost makes it worthwhile to go with the better customer service. I’ve used AdRoll for basic ad campaigns as well as ReTargetting. I must say I absolutely love them for ReTargetting. For standard ad campaigns they do a good amount to optimize performance, but you are also paying a premium for it. Those are the two I’m most aware of, but if others know of great retargetting services I’d love to hear about them and give them a try.

If you want your shop to seem like a mom and pop site that has 100 visitors a month, then retargetting will hold absolutely no value for you. But even if you do only have 100 visitors a month, with retargetting they’ll get some confidence that you’re a real online brand with a presence.