Python at Redfin

### Doug Wade Platforms team [doug.wade@redfin.com](mailto:doug.wade@redfin.com) Aside: I'm mostly a javascriptista, but Python was the first language I liked I've been at Redfin almost a year I work mostly on front end build, especially performance

Python

Redfin Official Scripting Language

nagios alerts
endpoint testing
data imports
walk score deployments

# Agenda - Build images: dirpy - Build: Bazel - Deploy: Fabric - Test: ETL Testing - Test: REF Aside: We're currently working on creating a continuous deployment pipeline Many parts of the are written in Python Our Python is rapidly growing

dirpy

Server-side image cropping + resizing

Resizes arbitrary images from MLS to fit standard sizes
Handles 200 images a second in prod
Resizes a single image in 6 - 8 ms
built with Pillow

Bazel

Fast, reproducible build system

Written in a subset of Python
Builds api server
Builds distibution tarballs
Adding building node modules

Bazel


    def _external_npm_module(ctx):
    return struct(
        internal = False,
        transitive_internal_deps = [],
        transitive_external_deps = [ctx.attr.raw_target] + ctx.attr.runtime_deps
    )
    external_npm_module = rule(
        implementation = _external_npm_module,
        attrs = {
            "raw_target": attr.label(allow_files = True),
            "runtime_deps": attr.label_list(allow_files = True)
        },
    )

Fabric

streamlines deploy and sysadmin tasks

Manages: servers - databases - solr - ci fleet - deployments

Fabric


    from fabric.api import *
    import utils.configuration
    from utils.deploylogger import log_run
    import time
    import os

    '''Start apache on each server.'''
    def do_work(config, silent_mode=False, args=None):
        retstring, retval = log_run(config, 'apachectl start', args)
        if retval:
            return (retstring, retval)
        return ('Success', 0)

ETL Testing

Black box verification of ETL pipelines

defines test cases
manages creation and deletion of source data
manages execution and failure detection of the pipeline
is extensible to new sources/destinations
built with py.test and Spring Python

ETL Testing

REF

Redfin Experiments Framework

performs A/B testing
integrates with our feature toggle, Bouncer
used to test major features before release
counts events from Redshift table and generates static site

We're Hiring!

Jobs site

Contact us: techrecruiting@redfin.com

Questions?

This couldn't possibly work, could it? It can't be that easy, can it? But it works!

- Dan Fabulich

Slides: http://redfin.github.io/slides/python-at-redfin/