We’re currently looking for interns for spring 2016!

We want to see your best work.

Here’s how.

Cover letters

All candidates must submit a cover letter. Your cover letter should be a statement of purpose. We’re interested in what you’re passionate about and why you’re passionate about it. (Most cover letters tell us that you are hardworking, passionate and talented, etc. And that you love NPR. We don’t need you to tell us that.)

• Tell us what you care about and work on.
• Tell us why you are passionate about your work.
• Tell us why this opportunity will help you reach your potential.
• Tell us how you will contribute to our team.

Other expectations

• Photo internship candidates must have a portfolio.
• Programming/design candidates with either projects on Github or a personal site are strongly preferred.

Selection process

After you submit a resume and cover letter, our selection committee will read through all the applications. We’ll reduce the list to approximately 8-10 candidates by eliminating applications that don’t have a cover letter and resume or who clearly aren’t a good fit for the team.

If you’re one of these candidates, two or three folks from the Visuals team will conduct a 30 minute Skype interview with you. You’ll get an email before your interview with outline of the questions you’ll be asked in the interview and also given the opportunity to ask any questions beforehand. The questions may vary a bit from interview to interview based on your professional experience, but we will be as consistent as possible.

Then we’ll call references and conduct some follow-up via email, possibly asking one or two more substantial, interview-style questions. Email communication is crucial in our workplace, and gives us an opportunity to see how you communicate in writing. We expect that answers are prompt, succinct, and clear.

We’ll follow up with all of our finalists with some constructive criticism about their application and interview.

Who we are

We’re a small group of photographers, videographers, photo editors, developers and designers in the NPR newsroom who make visual journalism. (Yeah, NPR is a radio thing, and yeah, it’s weird sometimes.) Check out our latest stuff!

Why we’re doing this

Everyone on the Visuals team wants to open our field to the best people out there, but the process doesn’t always work that way. So we’re trying to make the job application process more accessible.

Applicants with strong cover letters and good interview skills naturally tend to do well in this process. Often, those skills are a result of coaching and support — something that not all students are privileged to have. To help candidates without those resources, we’re being more transparent about our process and expectations.

We’re certain that we’re missing out on candidates with great talent and potential who don’t have that kind of support in their lives. We think knowing our cover letter expectations and interview questions ahead of time will help level the playing field, keep our personal bias out of the interview process, and allow better comparisons between candidates.

Apply!

Photo editing

Our photo editing intern will work with our digital news team to edit photos for npr.org. It’ll be awesome. There will also be opportunities to research and pitch original work.

Please…

• Love to write, edit and research
• Be awesome at making pictures

Are you awesome? Apply now!

Design and code

This intern will work as a designer and/or developer on graphics and projects for npr.org. It’ll be awesome.

Please…

• Our work is for the web, so be a web maker!
• We’d especially love to hear from folks who love illustration, news graphics and information design.

Are you awesome? Apply now!

What will I be paid? What are the dates?

The deadline for applications is November 1, 2015.

Check out our careers site for much more info.

Thx!

I didn’t always have a mission. For the first seven years of my career, I worked in the software industry. The work was interesting, and I had a craft, for sure, but not a mission. All that changed when I quit my job to become a journalist.

Our mission at NPR Visuals is to make people care. Everything we do: The things we make, our design process, and how we measure success, all flow from that mission. It’s awesome.

We’re here to create empathy. To introduce you to somebody you’ve never met, and think for a few minutes about life in their shoes. We’re here to open your eyes and make you give a shit.

Are you ready for a change?

The Knight-Mozilla fellowship is an awesome opportunity — for you and for us. It’s a chance for you to change your life, to try out working in a newsroom. You’ll learn a ton, and we’ll learn from you.

We’re open to folks from all walks of life, but if you’re a filmmaker, graphic designer, or involved in the digital humanities, we’d especially love to hear from you. No sweat if you can’t code or haven’t reported a story before — we’ll teach you.

(As for the specific work you’ll be doing… it’s hard to say! That’s one of the joys of working in a newsroom. We work on short schedules, and news deadlines. But I can say that you’ll work with us to report and tell important, impactful, visual stories, online.)

Want to join our mission? APPLY NOW!

We want to see your best work.

Here’s how.

(In case you missed it, applications are currently open for our fall internships.)

Cover letters

All candidates must submit a cover letter. Your cover letter should be a statement of purpose. We’re interested in what you’re passionate about and why you’re passionate about it. (Most cover letters tell us that you are hardworking, passionate and talented, etc. And that you love NPR. We don’t need you to tell us that.)

• Tell us what you care about and work on.
• Tell us why you are passionate about your work.
• Tell us why this opportunity will help you reach your potential.
• Tell us how you will contribute to our team.

Other expectations

• Photo internships candidates must have a portfolio.
• Programming/design candidates with either projects on Github or a personal site are strongly preferred.

Selection process

After you submit a resume and cover letter, our selection committee will read through all the applications. We’ll reduce the list to approximately 8-10 candidates by eliminating applications that don’t have a cover letter and resume or who clearly aren’t a good fit for the team.

If you’re one of these candidates, two or three folks from the Visuals team will conduct a 30 minute Skype interview with you. You’ll get an email before your interview with outline of the questions you’ll be asked in the interview and also given the opportunity to ask any questions beforehand. The questions may vary a bit from interview to interview based on your professional experience, but we will be as consistent as possible.

Then we’ll call references and conduct some follow-up via email, possibly asking one or two more substantial, interview-style questions. Email communication is crucial in our workplace, and gives us an opportunity to see how you communicate in writing. We expect that answers are prompt, succinct, and clear.

We’ll follow up with all of our finalists with some constructive criticism about their application and interview.

Why we’re doing this

Everyone on the Visuals team wants to open our field to the best people out there, but the process doesn’t always work that way. So we’re trying to make the job application process more accessible.

Applicants with strong cover letters and good interview skills naturally tend to do well in this process. Often, those skills are a result of coaching and support — something that not all students are privileged to have. To help candidates without those resources, we’re being more transparent about our process and expectations.

We’re certain that we’re missing out on candidates with great talent and potential who don’t have that kind of support in their lives. We think knowing our cover letter expectations and interview questions ahead of time will help level the playing field, keep our personal bias out of the interview process, and allow better comparisons between candidates.

Apply for this fall!

If you’re looking for a gig, please apply. If you know somebody who may be, please pass this along.

Our dailygraphics rig has been around for more than a year and in that time we’ve used it to make hundreds of responsive rectangles of good internet, but we’ve never made it easy for others to use. The rig is heavily customized for our needs and includes our organization-specific styles and templates. Despite this, a handful of hardy news organizations have made efforts to adopt it. In order to better facilitate this, today we are releasing our first fixed “version” of the rig: 0.1.0.

This isn’t a traditional release. The rapid pace of development and the pace of our news cycle makes it impossible for us to manage normal open source releases. Instead, we will tag selected commits with version numbers, and maintain a detailed CHANGELOG of everything that happens between those commits. This way users who want to use and stay up to date with the rig will have a clear path to do so.

As part of this release we’ve folded in a number of changes that make dailygraphics better than ever.

Block histogram

This block histogram is a format we’ve used several times to display discrete “binned” data. It works especially well for states or countries. Aly has turned it into a new graphic template so we can spin them up quickly. Run fab add_block_histogram to make one now!

Negative numbers and smart label positioning

The bar_chart, column_chart, grouped_bar_chart, stacked_bar_chart and stacked_column_chart graphic templates have all been updated to gracefully support negative numbers.

These five templates are also now much smarter about positioning labels so they always fit within the confines of the chart or hiding them if there is no way to make them fit in the available space.

(Curious how we did this? Here is the relevant code for bar charts. And here it is for column charts.)

Custom Jinja filters

Lastly, we’ve added support for defining custom Jinja filter functions in graphic_config.py. This allows for, among other things, much more complex formatting of numbers in Jinja templates. For example, to print comma-formatted numbers you can add this filter function:

def comma_format(value):
    return locale.format('%d', float(value), grouping=True)

JINJA_FILTER_FUNCTIONS = [
    comma_format
]

And then use it in your template, like this:

{{ row.value|comma_format }}

Documention for this feature has been added to the README.

Please see the CHANGELOG for a more complete list of changes we’ve made. We hope this new release process allows more news organizations to experience the joy of using a code-driven process for making daily charts and graphics.

Hey!

Are you a student?

Do you design? Develop? Love the web?

…or…

Do you make pictures? Want to learn to be a great photo editor?

If so, we’d very much like to hear from you. You’ll spend the fall working on the visuals team here at NPR’s headquarters in Washington, DC. We’re a small group of photographers, videographers, photo editors, developers and designers in the NPR newsroom who work on visual stuff for npr.org. Our work varies widely, check it out here.

Before you apply, please read our guide about what we expect in an internship application.

Photo editing

Our photo editing intern will work with our digital news team to edit photos for npr.org. It’ll be awesome. There will also be opportunities to research and pitch original work.

Please…

• Love to write, edit and research
• Be awesome at making pictures

Are you awesome? Apply now!

Design and code

This intern will work as a designer and/or developer on graphics and projects for npr.org. It’ll be awesome.

Please…

• Our work is for the web, so be a web maker!
• We’d especially love to hear from folks who love illustration, news graphics and information design.

Are you awesome? Apply now!

What will I be paid? What are the dates?

The deadline for applications is July 31, 2015.

Check out our careers site for much more info.

Thx!

Love to design and code?

Want to use your skills to make the world a better place?

We’re a crew of visual journalists (developers, designers, photojournalists…lots of things) in the newsroom at NPR headquarters in sunny Washington, DC. We make charts and maps, we make and edit pictures and video, we help reporters with data, and we create all sorts of web-native visual stories.

(And yeah, sometimes it’s kind of weird to be a visuals team at a radio organization. But there’s this special thing about audio. It’s intimate, it’s personal. Well, visual storytelling is really similar. It’s power is innate. Humans invented writing — visual and audio storytelling are built in, deep in our primordial lizard brains. So, anyway, yeah, we fit right in.)

Pictures and graphics are little empathy machines. And that’s our mission. To create empathy. To make people care.

It’s important work, and great fun.

And we’d love it if you’d join us.

We believe strongly that…

• We must put our audience’s needs first
• The right way to resolve an argument is with evidence
• It’s our duty to serve the public, so we give away our data and write about our work

You must have…

• Strong design skills, and experience implementing your designs on the web
• A steely and unshakable sense of ethics
• Attention to detail and love for making things
• A genuine and friendly disposition

Bonus points for…

• Serious front-end coding skills
• Experience running user-centered design processes

Allow me to persuade you

The newsroom is a crucible. We work on tight schedules with hard deadlines. That may sound stressful, but check this out: With every project we learn from our mistakes and refine our methods. It’s a fast-moving, volatile environment that drives you to be better at what you do, every day. It’s awesome. Job perks include…

• Live music at the Tiny Desk
• All the tote bags you can eat
• A sense of purpose

Know somebody who’d love this job?

Maybe it’s you?

Email bboyer@npr.org! Thanks!

Posted To: Ideas & Innovation > Blogically Thinking

This article first appeared June 22, 2015 in Nieman Labs.

Nowadays, we often seek to measure media engagement by social media activity, web metrics or attention minutes.

But there was a time in the not-so-distant past – before the Internet and social media disrupted traditional media – when genuine engagement really worked.  A period when news organizations actually involved people in their communities so successfully it triggered impact.

With last week’s celebration of the tremendous journalism contributions of Ed Fouhy, the award-winning broadcast executive and founder of the Pew Center for Civic Journalism, it seemed like a good time to revisit what we already learned – but may have forgotten.

During the heyday of civic journalism, which spanned a decade starting in the early ‘90’s, the Pew Center funded 120 newsroom projects and rewarded scores more with the James K. Batten Awards. More than 600 CJ initiatives were counted and studied by U-Wisconsin researchers, who found a pattern of outcomes. Some 78 percent of the projects studied offered solutions, and more than half included solutions offered by citizens themselves.

I was on the frontlines of this activity. Fouhy hired me in 1994 to be his Pew Center deputy. A couple years later, I took his place at the helm.

I find it striking how many of these efforts foreshadowed what we now call interactive and participatory journalism.

Civic journalism began as a way to get political candidates to address the public’s agenda in running for office. News organizations soon adapted its techniques, starting with polls and town hall meetings, to difficult problems in their communities. Later on-ramps involved civic mapping, email, voice mail, cutting-edge video technologies, and eventually, of course, the Internet.

Key hallmarks of these civic journalism initiatives included:

• Building specific ways to involve readers and viewers.
• Deliberately positioning ordinary people as capable of some action.
• Inviting the community to identify solutions.

Consider how some of these efforts played out:

Taking Back Our Neighborhoods: This seminal initiative, a finalist for a Pulitzer Public Service Award, set the bar high for CJ projects.  It evolved from the 1993 shooting of two Charlotte, N.C. police officers.

Determined to address the root cause of crime, The Charlotte Observer partnered with WSOC-TV to synchronize in-depth coverage and give people things they could do reclaim their communities.

Elements included data analysis, which identified patterns of crime and the most violent neighborhoods to spotlight. A poll asked residents how crime affected them, why crime was happening and what were possible solutions. Town hall meetings and neighborhood advisory panels in 10 targeted communities contributed very specific lists of neighborhood “needs” that were published with each community’s narrative.

Outcomes were impressive:  Some 700 volunteers stepped up to fulfill the needs on those lists – from opening new recreation centers to making uniforms for a fledgling drill team. Eighteen law firms filed pro bono nuisance suits to close crack houses. New community centers were built and neighborhoods were cleaned up. Eight years later, crime was still down and the quality of life had improved in eight of the 10 neighborhoods.

West Virginia After Coal: The Herald-Dispatch in Huntington, W.Va., and West Virginia Public Broadcasting joined forces in 2000-01 to examine one of the state’s biggest issues: Its future without coal.  

The partners developed a groundbreaking database that exposed how virtually none of the $18 million in coal severance taxes distributed to the state’s 55 counties and 234 municipalities were being used for economic development. Instead, the funds paid for such things as dog wardens or postage. The media partners used statewide polls and an interactive town hall involving audience input from 10 different sites via cutting-edge video conferencing technology. By the project’s end, the state was promising more job training and more revenue targeted to economic development.

Waterfront Renaissance: In 2001 The Herald of Everett, Wa. engaged the community in plans to remake its waterfront. It held a town hall meeting on development plans and created online clickable maps with moveable icons to give residents a virtual vote on what should be built along the Snohomish River and Port Gardner Bay. Some 1,200 people participated. The Herald tabulated the results of these maps and submitted their findings to city officials. A prevailing theme was that people wanted any development to give them access to their riverfront. Their wishes were ultimately included in city plans. The project today remains a prime example of how to involve citizens in surrogate “public hearings.”

Neighbor to Neighbor: In 2002, after the shooting of an unarmed teenager in Cincinnati sparked allegations of police misconduct and major rioting, The Cincinnati Enquirer embarked on an ambitious project. It held solutions-oriented conversations on how to improve race relations in every municipality and neighborhood in the regions – some 145 in all.  

Each group was asked to answer:

• What three things can people do to ease racial tensions?
• What three things would we like to see our community leaders do?
• How can we make it happen?

Some 1,838 people participated; 300 people volunteered to host or facilitate the conversations. The project inspired much grassroots volunteerism and efforts among black and whites to interact. The project "started people talking together, going to dinner, meeting in their homes and going to school and churches together,” said then-managing editor Rosemary Goudreau at the time.

There were scores of similar robust projects:

• The Savannah Morning News involved a large citizen task force in discussions and visits to 15 U.S. schools to figure out how to improve local education.
• A 1997 series on alcoholism, “Maine’s Deadliest Drug,” by The Portland Press Herald and Maine Sunday Telegram led to citizen forming 29 study circles that concluded with an action plan to stem alcohol abuse.
• We the People/Wisconsin, involving the Wisconsin State Journalism and the state’s public broadcaster, engaged in some of longest-running efforts to bring citizens face-to-face with candidates running for statewide office.

To be sure, journalism investigations often lead to widespread change. But, to me, so many of today’s journalism success stores seem pallid by comparison to what I saw during the period of civic journalism experimentation.

Simply put: civic journalism worked.  Readers and viewers got it.

We learned that if you deliberately build in simple ways for people to participate – in community problems or elections – many will engage.  Particularly if they feel they have something to contribute to the problem.

Nowadays, this is so much easier than it used to be. All that is needed is the creativity to make it happen.

Jan Schaffer is executive director of J-Lab, a successor to the Pew Center and an incubator for entrepreneurial news startups.

Posted To: Ideas & Innovation > Blogically Thinking

This was first published March 10, 2015 on Mediashift.

When I launched J-Lab in 2002, the best piece of advice I received was to have a lawyer draft a Memorandum of Understanding outlining the relationship between my center and its soon-to-be home, the University of Maryland.

The agreement detailed how I would support my startup, who owned the intellectual property, how much the university would charge for administering my incoming grants – and how I might spin the center into its own 50s(c)3 or affiliate with another fiscal agent in the future.

Thanks to that MOU, when U-MD changed its rules for grant-supported centers, I was able to seamlessly transition to American University. The MOU basically served as a pre-nup agreement.

I never really expected to need the MOU – until I did. So, too, are new media startups finding themselves in situations where they need to know about, and plan for, an array of legal issues.  Many of these issues particularly affect digital-first news sites.  

With this, and many more experiences under my belt, I approached Jeff Kosseff, a Washington, D.C., media lawyer and fellow A.U. adjunct, about co-authoring “Law for Media Startups.” We wanted to make it a user-friendly e-guide to what news entrepreneurs need to know and also help them identify when they needed professional help.

Next, I recruited CUNY’s Tow-Knight Center for Journalism Entrepreneurship to help support the project.  The result: our 12-chapter guide that we hope will be as helpful to educators teaching media law courses and it will be to startup founders themselves.  A downloadable PDF is coming soon.

Most journalists are used to working with legal counsel for such things as pre-publication review of important stories.  But legal issues for digital-first news startups extend far beyond such traditional First Amendment issues as defamation, privacy and open government.

“New media has not changed the law of libel at all,” said Richard Tofel, ProPublica’s president and its resident lawyer. “But it has changed the breadth of laws entrepreneurs need to know about.”

How should you respond when someone is demanding the IP address or the identity of a commenter on your sites. How should you flag sites that steal your content? How can your make sure, in a rush to add an image to an article, that you are not posting a copyrighted photo?  How to deal with a freelancer’s request to use for another assignment research gathered for a story you commissioned? When it is OK for someone to be a freelancer and when do they have a right to be an employee?

“The No. 1 question by far that we hear from our members is about freelancer contracts and rights,” said Kevin Davis, executive director of the Investigative News Network.

The IRS sets out very specific guidelines addressing who should be an employee and who can be an independent contractor.  As important, it requires all unpaid interns to meet six specific conditions.

All digital-first news startups are collecting some type of data on their users, and while most journalists advocate for openness and transparency, as an Internet-based business, you have a number of legal obligations to keep certain formation private. You also need to tell your users how you will use their data.

Certainly, one of the biggest misconceptions some online publishers have is that you websites will only have immunity if you take a hands-off approach, and don’t edit or moderate any comments. Indeed, according to the e-guide, “service providers have wide latitude to edit, delete, or even select user content without being held responsible for it.”

Again and again, I have reviewed applications for J-Lab funding that promised that the startup would get grants to support its work.  However, the applicant was neither a nonprofit nor affiliated with one and, therefore. was not eligible for the grants they wanted to support their business.  News entrepreneurs need to understand what being a nonprofit entails or pick another business structure.

As our guide notes, “journalism is not something the IRS recognizes as having a tax-exempt purpose.” So, if you embark on applying for 501(c)3 status, you need to flesh out how you will be different from a regular commercial publisher.

In the media startup space, legal needs can be surprising. Lorie Hearn, founder of inewsource.org, has partnered with a number of media outlets to amplify her investigative stories in the San Diego area.

But she says she has begun to feel the need to craft written distribution agreements to cover inewsource partnerships with other news outlets, especially pertaining to how they credit her material on their websites. Some “want their own correspondents to come in and interview our people and make like this is a joint investigation,” she said.

For that, she will likely seek out a lawyer who has worked closed with her site over the years.

To read about more issues, see the full guide here.

• Start with the goal of helping as many listeners as possible learn to create a quality listening experience on a computer, to listen via an app, to use external speakers at home and in the car, and to find and listen to a podcast or on demand content.
• Have up-to-date and easy to use digital listening options.
• Develop a simple narrative describing the benefits of using the station’s digital offerings, including step-by-step instructions on how to get the most out of each option.
• Promote the heck out of it using every possible touch point, including on-air.
• Provide prompt individualized customer service when needed.
• Rinse and Repeat.

• 43 of the breaks (86%) had absolutely no promotion for the station's digital listening offerings.
• 8 of the 14 stations had no digital listening promotion. I listened to at least 3 breaks in one hour for each station.
• Of the 5 stations that had some sort of digital listening promotion, 3 mentioned more than one type of digital listening in the same break.  For example, the website was promoted as a way to stream the station and as a way to hear the station's new podcast.
• 1 station qualified as promoting digital listening only because it included the website in its legal ID, "...and online at WXZY.org."  That's more of a throw away mention than a promotion, but I still counted it.

When news happens in locations that our audience may not know very well, a map seems like a natural thing to include as part of our coverage.

But good maps take time.*

In ArcMap, I’ll assemble the skeleton of my map with shapefiles from Natural Earth and other sources and find an appropriate projection. Then I’ll export it to .AI format and bring it into Adobe Illustrator for styling. (In the example below, I also separately exported a raster layer for shaded relief.) And then I’ll port the final thing, layer by layer, to Adobe Photoshop, applying layer effects and sharpening straight lines as necessary.

(* Note: I enjoy making maps, but I am unqualified to call myself a cartographer. I owe much, though, to the influence of cartographer colleagues and GIS professors.)

I concede that this workflow has some definite drawbacks:

• It’s cumbersome and undocumented (my own fault), and it’s difficult to train others how to do it.

• It relies on an expensive piece of software that we have on a single PC. (I know there are free options out there like QGIS, but I find QGIS’s editing interface difficult to use and SVG export frustrating. ArcMap has its own challenges, but I’m used to its quirks and the .AI export preserves layers better.)

• This reliance on ArcMap means we can’t easily make maps from scratch if we’re not in the office.

• The final maps are flat images, which means that text doesn’t always scale readably between desktop and mobile.

• Nothing’s in version control.

So for the most recent round of Serendipity Day at NPR (an internal hackday), I resolved to explore ways to improve the process for at least very simple locator maps – and maybe bypass the expensive software altogether.

Filtering And Converting Geodata

My colleague Danny DeBelius had explored a little bit of scripted mapmaking with his animated map of ISIS-claimed territory. And Mike Bostock has a great tutorial for making maps using ogr2ogr, TopoJSON and D3.

(ogr2ogr is a utility bundled with GDAL that converts between geo formats. In this case, we’re using it to convert GIS shapefiles and CSVs with latitude/longitude to GeoJSON format. TopoJSON is a utility that compresses GeoJSON.)

Danny figured out how to use ogr2ogr to clip a shapefile to a defined bounding box. This way, we only have shapes relevant to the map we’re making, keeping filesize down.

ogr2ogr -f GeoJSON -clipsrc 77.25 24.28 91.45 31.5 data/nepal-geo.json ../_basemaps/cultural/ne_10m_admin_0_countries_v3.1/ne_10m_admin_0_countries.shp

We applied that to a variety of shapefile layers — populated places, rivers, roads, etc. – and then ran a separate command to compile and compress them into TopoJSON format.

ogr2ogr -f GeoJSON -clipsrc 77.25 24.28 91.45 31.5 data/nepal-geo.json ../_basemaps/cultural/ne_10m_admin_0_countries_v3.1/ne_10m_admin_0_countries.shp

ogr2ogr -f GeoJSON -clipsrc 77.25 24.28 91.45 31.5 data/nepal-cities.json -where "adm0name = 'Nepal' AND scalerank < 8" ../_basemaps/cultural/ne_10m_populated_places_simple_v3.0/ne_10m_populated_places_simple.shp

ogr2ogr -f GeoJSON -clipsrc 77.25 24.28 91.45 31.5 data/nepal-neighbors.json -where "adm0name != 'Nepal' AND scalerank <= 2" ../_basemaps/cultural/ne_10m_populated_places_simple_v3.0/ne_10m_populated_places_simple.shp

ogr2ogr -f GeoJSON -where "featurecla = 'River' AND scalerank < 8" -clipsrc 77.25 24.28 91.45 31.5 data/nepal-rivers.json ../_basemaps/physical/ne_10m_rivers_lake_centerlines_v3.1/ne_10m_rivers_lake_centerlines.shp

ogr2ogr -f GeoJSON -clipsrc 77.25 24.28 91.45 31.5 data/nepal-lakes.json ../_basemaps/physical/ne_10m_lakes_v3.0/ne_10m_lakes.shp

ogr2ogr -f GeoJSON -clipsrc 77.25 24.28 91.45 31.5 data/nepal-roads.json ../_basemaps/cultural/ne_10m_roads_v3.0/ne_10m_roads.shp

topojson -o data/nepal-topo.json --id-property NAME -p featurecla,city=name,country=NAME -- data/nepal-geo.json data/nepal-cities.json data/nepal-neighbors.json data/nepal-rivers.json data/nepal-lakes.json data/nepal-roads.json data/nepal-quakes.csv

(Why two separate calls for city data? The Natural Earth shapefile for populated places has a column called scalerank, which ranks cities by importance or size. Since our example was a map of Nepal, I wanted to show a range of cities inside Nepal, but only major cities outside.)

Mapturner

Christopher Groskopf and Tyler Fisher extended that series of ogr2ogr and TopoJSON commands to a new command-line utility: mapturner.

Mapturner takes in a YAML configuration file, processes the data and saves out a compressed TopoJSON file. Users can specify settings for each data layer, including data columns to preserve and attributes to query. The config file for our Nepal example looked like this:

bbox: '77.25 24.28 91.45 31.5'
layers:
    countries:
        type: 'shp'
        path: 'http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_admin_0_countries.zip'
        id-property: 'NAME'
        properties:
            - 'country=NAME'
    cities:
        type: 'shp'
        path: 'http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_populated_places_simple.zip'
        id-property: 'name'
        properties:
            - 'featurecla'
            - 'city=name'
            - 'scalerank'
        where: adm0name = 'Nepal' AND scalerank < 8
    neighbors:
        type: 'shp'
        path: 'http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_populated_places_simple.zip'
        id-property: 'name'
        properties:
            - 'featurecla'
            - 'city=name'
            - 'scalerank'
        where: adm0name != 'Nepal' AND scalerank <= 2
    lakes:
        type: 'shp'
        path: 'http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/physical/ne_10m_lakes.zip'
    rivers:
        type: 'shp'
        path: 'http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/physical/ne_10m_rivers_lake_centerlines.zip'
        where: featurecla = 'River' AND scalerank < 8
    quakes:
        type: 'csv'
        path: 'data/nepal.csv'
        properties:
            - 'date'
            - '+intensity'

Mapturner currently supports SHP, JSON and CSV files.

Drawing The Map

I’ve been pretty impressed with the relative ease of using D3 to render maps and test projections. Need to adjust the scope of the map? It might just be a matter of adjusting the map scale and centroid (and, if necessary, expanding the overall bounding-box and re-running the mapturner script) — much faster than redrawing a flat map.

Label positioning is a tricky thing. So far, the best way I’ve found to deal with it is to set up an object at the top of the JS with all the nit-picky adjustments, and then checking for that when the labels are rendered.

var CITY_LABEL_ADJUSTMENTS = [];
CITY_LABEL_ADJUSTMENTS['Biratnagar'] = { 'dy': -3 };
CITY_LABEL_ADJUSTMENTS['Birganj'] = { 'dy': -3 };
CITY_LABEL_ADJUSTMENTS['Kathmandu'] = { 'text-anchor': 'end', 'dx': -4, 'dy': -4 };
CITY_LABEL_ADJUSTMENTS['Nepalganj'] = { 'text-anchor': 'end', 'dx': -4, 'dy': 12 };
CITY_LABEL_ADJUSTMENTS['Pokhara'] = { 'text-anchor': 'end', 'dx': -6 };
CITY_LABEL_ADJUSTMENTS['Kanpur'] = { 'dy': 12 };

Responsiveness makes label positioning even more of a challenge. In the Nepal example, I gave each label a class corresponding to its scalerank, and then used LESS in a media query to hide cities above a certain scalerank on smaller screens.

@media screen and (max-width: 480px) {
    .city-labels text,
    .cities path {
        &.scalerank-4,
        &.scalerank-5,
        &.scalerank-6,
        &.scalerank-7,
        &.scalerank-8 {
            display: none;
        }
    }
}

Our finished example map (or as finished as anything is at the end of a hackday):

There’s still more polishing to do — for example, the Bangladesh country label, even abbreviated, is still getting cut off. And the quake dots need more labelling and context. But it’s a reasonable start.

Drawing these maps in code has also meant revisiting our map styles — colors, typography, label and line conventions, etc. Our static map styles rely heavily on Helvetica Neue Condensed, which we don’t have as a webfont. We do have access to Gotham, which is lovely but too wide to be a universal go-to. So we may end up with a mix of Gotham and Helvetica — or something else entirely. We’ll see how it evolves.

Locator Maps And Dailygraphics

We’ve rolled sample map code into our dailygraphics rig for small embedded projects. Run fab add_map:$SLUG to get going with a new map. To process geo data, you’ll need to install mapturner (and its dependencies, GDAL and TopoJSON). Instructions are in the README.

Caveats And Next Steps

• This process will NOT produce finished maps — and is not intended to do so. Our goal is to simplify one part of the process and get someone, say, 80 percent of the way to a basic map. It still requires craft on the part of the map-maker — research, judgement, design and polish.

• These maps are only as good as their source data and the regional knowledge of the person making them. For example, the Natural Earth country shapefiles still include Crimea as part of Ukraine. Depending on where your newsroom stands on that, this may mean extra work to specially call out Crimea as a disputed territory.

• When everything’s in code, it becomes a lot harder to work with vague boundaries and data that is not in geo format. I can’t just highlight and clip an area in Illustrator. We’ll have to figure out how to handle this as we go. (Any suggestions? Please leave a comment!)

• We’ve figured out how to make smart scale bars. Next up: inset maps and pointer boxes. I’d also like to figure out how to incorporate raster topo layers.

A hexagon tile grid, square tile grid and geographic choropleth map. Maps by Danny DeBelius and Alyson Hurt.

As the saying goes, nothing is certain in this life but death, taxes and requests for geographic data to be represented on a map.

For area data, the choropleth map is a tried and true visualization technique, but not without significant dangers depending on the nature of the data and map areas represented. Clarity of mapped state-level data, for instance, is frequently complicated by the reality that most states in the western U.S. carry far more visual weight than the northeastern states.

Are more northeastern states shaded than western? That’s hard to say with this type of choropleth. Whatever, though. West coast, best coast, right?

While this presentation is faithful to my Californian perception of the U.S. where the northeast is a distant jumble of states I pay little attention to, I’ve learned in four years of living in D.C. that there are actually a lot of people walking around that jumble, and they’d prefer not to be ignored in mapped data visualizations. There are approximately 74 million people living in the thirteen states the U.S. Census Bureau defines as the Western United States, while around 42 million people live just in the combined metropolitan statistical areas of New York, Washington, Boston and Philadelphia.

One popular solution to this problem is the cartogram — maps where geography is distorted to correspond with some data variable (frequently population). By shading and sizing map areas, a cartogram can display two variables simultaneously. In this New York Times example from the 2012 election, the size of the squares corresponds to the number of electoral votes assigned to each state, while the shade represents possible vote outcomes. NPR’s Adam Cole used this technique to size states according to electoral votes and ad spending, as seen in the map below. Cartograms can be a great solution with some data sets, but they introduce complexity that might not serve our ultimate goal of clarity.

A cartogram of the U.S. with states sized proportionally by electoral votes. Map by Adam Cole.

Recently, a third variation of choropleth has gained popularity — the tile grid map. In this version, the map areas are reduced to a uniform size and shape (typically a square) and the tiles are arranged to roughly approximate their real-world geographic locations. It’s still a cartogram of sorts, but where the area sizing is based on the shared value of one “map unit.” Tile grid maps avoid the visual imbalances inherent to traditional choropleths, while keeping the map a quick read by forgoing the complexity of cartograms with map areas sized by a variable data point.

Tile grid maps are a great option for mapped state data where population figures are not part of the story we’re trying to tell with the map. Several news organizations have used this approach to great effect, including FiveThirtyEight, Bloomberg Business, The Guardian, The Washington Post and The New York Times.

A square tile grid map.

Here at NPR, we recently set out to create a template for quickly producing this type of map, but early in the process my editor Brian asked, “Do the tiles have to be squares?”

More specifically, Brian was interested in exploring the possibility of using hexagons instead of squares, with the assumption that two additional sides would offer greater flexibility in arranging the tiles and a better chance at maintaining as many border adjacencies as possible.

The idea was intriguing, but I had questions about sacrifices we might make in scanability by trading the squares for hexagons. The columns and rows of a square grid lend to easy vertical and horizontal scanning, and I wondered if the tessellation of hexagons would provide a comfortable reading experience for the audience.

Here is Brian’s first quick pencil sketch of a possible state layout using hexagons:

Brian’s hex grid sketch.

That proof of concept was enough to convince me that the idea was worth exploring further. I opened up Sketch and redrew Brian’s map with the polygon tool so we could drag the states around to experiment with the tile layout more easily. We tried several approaches in building the layout, starting from each coast and building from the midwest out, to varying degrees of success.

Ultimately, I decided to prioritize accuracy in representing the unique geographic features of the U.S. border (Texas and Florida as the southernmost tips, notches for the Great Lakes) and making sure the four “corners” of the country were recognizable for orientation.

The final layout that will power our tile grid map template looks like this:

Six sides instead of four! That means it’s two better.

This map still has many of the same problems that other attempts at a tile layout of the U.S. have fallen into — the relationship of North and South Carolina, for one example — but we like the increased fidelity of the country’s shape the hex grid makes possible.

In case you were wondering, news dev Twitter loves talking about maps:

Hex Tile Grid Maps

We recently published our first use of the hexagon tile grid map to show the states that currently have laws restricting discrimination in employment, housing and public accommodations based on sexual orientation, gender identity and gender expression. The hex grid tile map also made appearances in several presentations of last week’s U.K. election results, including those by The Guardian, Bloomberg Business and The Economist.

What do you think? Vote in the poll below!

Amazon’s Relational Database Service (RDS) is an excellent way to host databases. The service is affordable, low-maintenance, and self-contained. If you use the Amazon cloud, there are precious few reasons to maintain your own database server.

At some point, Amazon started requiring RDS instances to use Virtual Private Cloud (VPC) networking. However, if you’re like the NPR Visuals team, you might have older Amazon Elastic Cloud Compute (EC2) server instances that don’t use VPC but need to connect to RDS databases. Even if you don’t, you might need to connect to your RDS instance locally.

As is often the case with Amazon, it’s not entirely clear how to configure the correct security rules to allow access from outside the VPC. Here’s what worked for us.

During creation, make sure your RDS instance is publicly accessible. This setting cannot be edited later.

For the security group setting, either option will suffice, though creating a new security group will help isolate the network access rules for this database instance.

Once created, click on the security group from the instance details:

A new tab or window will open with the security group selected. Click the “Inbound” tab in the lower window pane, then click the “Edit” button to add rules to allow the IP addresses you want to access the RDS instance

Now you can configure the inbound rules in the modal that opens:

I found a lot of places in the VPC interface to set inbound rules, but only the security group rules actually worked to allow local machines and non-VPC EC2 instances access to the RDS database.

If you know a better way to handle this, let us know in the comments!

In the past couple weeks the Visuals team has consciously shifted resources to focus on the parts of our work that have the highest impact. As part of this reorganization the graphics team has grown from one (Graphics Editor Alyson Hurt) to two—the second being me! Having a dedicated engineer working on daily graphics means we are doubling down both the amount of content we can create and on the tools we use to create it. For the last week I’ve been sprinting on a slew of improvements to our dailygraphics rig. Most of these are small changes, but collectively they represent the biggest iteration we’ve made to dailygraphics since creating it over a year ago.

OAuth

Amongst a group of features we’ve ported over from the app-template is the addition of an OAuth support for access to our “copytext” Google spreadsheets. This means Google credentials no longer need to be stored in environment variables, which increases security and portability. (Hat tip to David Eads for untangling OAuth for all of our projects.)

This change also allowed us to implement a more significant feature: automatically creating copytext spreadsheets. Each time you add a graphic a spreadsheet will be automatically created. (You can opt out of this by not specifying a COPY_GOOGLE_DOC_KEY in your base graphic template or by deleting graphic_config.py entirely.)

Rewriting the copytext workflow has also allowed as to add a “refresh flag” to the preview. Now anytime you pass ?refresh=1 with your graphic preview url, the preview app will automatically re-download the latest copytext before rendering. This can tremendously accelerate editing time for text-heavy graphics.

Advanced graphic templates

As our graphics pipeline has matured we’ve started to run into many of the same limitations that prompted development of the app-template. As a result, we’ve reincorporated features such as template inheritance, asset compression and LESS support.

The base template

All graphic templates now “inherit” from a base template, which is found in graphic_templates/_base. When a new graphic is created, this folder is copied to the new graphic’s path before the normal graphic template (e.g. graphic_templates/bar_chart). This base template can house files common to all templates for easy updates. (The individual graphic templates can copy over any or all of them.)

The base template also includes a base_template.html which the original child_template.html now inherits from using Jinja2 template inheritance. This change means you can now make a change to the header or footer of your graphics and have it instantly incorporated in all your graphic templates. (Not retroactively though, every graphic is still a copy of all assets and templates.)

LESS and asset compression

All CSS files in graphic templates can now be LESS files, which will be automatically compiled during preview and deployment. The resulting CSS assets will be automatically compiled into a single file and compressed by using this code in the base template:

<!-- CSS + LESS -->
{{ CSS.push('css/base.less') }}
{{ CSS.push('css/graphic.less') }}
{{ CSS.render('css/graphic-header.css') }}

Mirroring the app-template, this same pattern is followed for compressing Javascript assets:

{{ JS.push('js/lib/jquery.js') }}
{{ JS.push('js/lib/d3.min.js') }}
{{ JS.push('js/lib/modernizr.svg.min.js') }}
{{ JS.push('js/lib/pym.js') }}
{{ JS.push('js/base.js') }}
{{ JS.push('js/graphic.js') }}
{{ JS.render('js/graphic-footer.js') }}

Google Analytics support

Our new base template also now includes code for embedding Google Analytics with your graphics. We’ve long wanted to be able to track detailed analytics for our graphics, but putting analytics inside in the iframe would have resulted in impressions being counted twice—once for the parent page and once for the child page. To avoid this we’ve recently begun tracking our project analytics on a separate Google property from that used for NPR.org. This allows us to put our custom analytics tag inside the iframe while our traditional pageviews are captured by the parent analytics tags.

Improvements to the graphic viewer (parent.html)

Perhaps the most obvious changes to the dailygraphics rig are our suite of improvements to the graphic preview template (a.k.a. parent.html). These changes are aimed at making it easier to see how the final graphic will work and making it faster to test. They include:

• Resize buttons for quickly testing mobile and column layouts.
• Border around the graphic so you can see how much margin you’ve included.
• An obvious label so you know which environment you’re working in (local, staging, production).
• One-click links to other environments and to the copytext spreadsheet (if configured).
• Easy-to-copy Core Publisher embed code (for NPR member stations).

Other improvements

In addition to these larger improvements we’ve also made a couple of smaller improvements that are worth noting:

• A stacked bar chart template.
• Upgrading to pym 0.4.3, which includes shortcuts for scrolling and navigating the parent frame from within the graphic.

Upgrading

If you’re a user of the dailygraphics rig we strongly encourage you to upgrade and incorporate these new improvements into your process. I think they’ll make your graphics workflow smoother and much more flexible. After pulling the latest code you’ll need to install new requirements. Node.js is now a dependency, so if you don’t have that you’ll need to install it first:

brew install node
curl https://npmjs.org/install.sh | sh

Then you can update your Node and Python dependencies by running the following commands:

pip install -Ur requirements.txt
npm install

Please remember that everything in the dailygraphics rig still works on copies, so upgrading will not retroactively change anything about your existing graphics.

If you’re using the improved dailygraphics rig, let us know!

Mapping the spread of Wal-Mart

For a recent story on the growth of Wal-Mart in urban areas we set out to map Wal-Marts across the US and over time. Due to limitations with our dataset, we only ended up mapping three cities. Here is the graphic we produced:

Automation is key to generating these sorts of maps. There are huge number of things that could go wrong if each one was produced by hand. For this story the automated process involved connecting several different tools and many different data sources. In this post I’m going to set that complexity aside and focus on just the final part of the toolchain: outputting SVG maps for final styling in Illustrator. If you’re interested in the complete process, we’ve open sourced the code here.

Why use many little maps?

For this story the maps we produced were used as “small multiples”, that is, many small images that collectively illustrate a something larger. However, there are many other occasions where producing small maps is useful, such as when illustrating city or county-level data for many hundreds of places. Sometimes it’s necessary to generate these maps dynamically, but in many cases they can be pre-generated and “looked up” as needed.

From XML to SVG

To generate map images we used a tool I originally wrote over four years ago, when I was working for the Chicago Tribune: invar.

invar is a suite of three command line tools:

• ivtile generates map tiles suitable making slippy maps.
• ivframe generates individual maps centered on locations.
• ivs3 bulk uploads files (such as map tiles) to Amazon S3 for distribution.

Both ivtile and ivframe use Mapnik as a rendering engine. Mapnik allows you to input an XML configuration file specifying styles and datasources and output map images as PNGs or SVGs. For example, here is a fragment of the configuration for the circles (“buffers”) around each store:

<Layer name="buffers" status="on" srs="+init=epsg:4269">
    <StyleName>buffer-styles</StyleName>
    <Datasource>
        <Parameter name="type">postgis</Parameter>
        <Parameter name="host">localhost</Parameter>
        <Parameter name="dbname">walmart</Parameter>
        <Parameter name="table">(select * from circles where year::integer &lt;= 2005 order by range desc) as buffers</Parameter>
    </Datasource>
</Layer>

<Style name="buffer-styles">
<Rule>
    <Filter>[range] = 1</Filter>
    <PolygonSymbolizer fill="#28556F" />
</Rule>
<Rule>
    <Filter>[range] = 2</Filter>
    <PolygonSymbolizer fill="#3D7FA6" />
</Rule>
</Style>

In this example we query a PostGIS table called circles to get buffers for stores opened before or during 2005. (The < escaping is an unfortunate necessity for getting the XML to parse correctly.) We then color the circles differently based on whether they represent a one or two mile range. To render the map for Chicago we would run:

ivframe -f svg --name chicago_2005.svg -z 10 -w 1280 -t 1280 map.xml . 41.83 -87.68

(You can also render a series of images using coordinates from a CSV. See the invar docs for more examples and more details on the flags being used here.)

Documentation of the Mapnik XML format is relatively sparse, but Googling frequently turns up working examples. If the XML annoys you too badly, there are also Python bindings for Mapnik though personally I’ve never had much luck generating maps from scratch with them.

Using invar to make your own maps

invar is easy to install, just pip install invar. Unfortunately, the Mapnik dependency is notoriously difficult. You’ll find instructions specific to your platform on the Mapnik wiki. (If you’re on OSX I recommend brew!) This is the only time I will ever suggest not using virtualenv to manage your Python dependencies. Getting Mapnik to work within a virtualenv is a painful process and you’re better off simply installing everything you need globally. (Just this once!)

Dusting off invar after not having used it for a long time gave me a good opportunity to fix some critical bugs and the new 0.1.0 release should be the most stable version ever. More importantly, it now supports rendering SVG images, so you can produce rough maps with invar and then refine them with Illustrator, which is what we did for the Wal-Mart maps. Go ahead and give it a spin: the full documentation is here. Let us know how you use it!

Posted To: Press Releases

J-Lab partners with CUNY to create e-guide

Washington, D.C.– “Law for Media Startups,” a new resource for entrepreneurs launching news ventures and educators teaching students how to do it, was published today by CUNY’s Tow-Knight Center for Entrepreneurial Journalism.

The 12-chapter web guide was written by Jan Schaffer, J-Lab executive director, and Jeff Kosseff, media lawyer for Covington & Burling LLP law firm in Washington, D.C. The Tow-Knight Center supported the research, writing and production as part of its suite of entrepreneurial journalism resources.

“This guide goes beyond traditional First Amendment law to address every-day issues news entrepreneurs confront,” Schaffer said. “From native advertising to labor law and fair use, it supplies what I found missing for my students.”

"Every day, innovators are developing new ways to deliver news and content to consumers," Kosseff said. "I hope this guide helps them identify the legal issues that they should be considering as they build their business models."

Small digital news startups are facing a range of legal issues as they set up their business operations, gather and report the news, protect their content, and market and support their news ventures. They need to know classic First Amendment law – and much more. This guide offers an introduction to many of those issues, from hiring freelancers and establishing organizational structures, to native advertising and marketing, to maintaining privacy policies and dealing with libel accusations. It seeks to help jumpstart the launch of news ventures and help entrepreneurs know when to seek professional legal help.

“The news ecosystem of the future will be made up of enterprises of many sizes, shapes, and forms, including journalistic startups that need help with their businesses and the law” said Jeff Jarvis, Director of the Tow-Knight Center. “Jan Schaffer and Jeff Kosseff provide an invaluable guide to help them recognize legal pitfalls. It complements other research from Tow-Knight on a variety of business practices.”

Jeff Kosseff is a communications and privacy attorney in Covington & Burling’s Washington, D.C. office, where he represents and advises media and technology companies. He is co-chair of the Media Law Resource Center’s legislative affairs committee. He clerked for Judge Milan D. Smith, Jr. of the U.S. Court of Appeals for the Ninth Circuit and Judge Leonie M. Brinkema of the U.S. District Court for the Eastern District of Virginia. He is an adjunct professor of communications law at American University, where he teaches in its MA in Media Entrepreneurship program. Before becoming a lawyer, Kosseff was a journalist for The Oregonian and was a finalist for the Pulitzer Prize and recipient of the George Polk Award for national reporting.

Jan Schaffer is executive director of J-Lab, an incubator for news entrepreneurs and innovators, and Entrepreneur in Residence at American University’s School of Communication, where she also teaches in its MA in Media Entrepreneurship program. She launched J-Lab in 2002 to help newsrooms use digital technologies to engage people in public issues. It has funded 100 news startups and pilot collaboration projects and it has commissioned and developed a series of online journalism resources that include Launching a Nonprofit News Site, Top Ten Rules for Limited Legal Risk and The Journalist’s Guide to Open Government. As the federal court reporter for The Philadelphia Inquirer, she was part of a team awarded the Pulitzer Gold Medal for Public Service for a series of stories that won freedom for a man wrongly convicted of five murders and led to the civil rights convictions of six Philadelphia homicide detectives.

The “Law for Media Startups” guide also invites media lawyers around the country to contribute information on state-specific laws that apply to news entrepreneurs, following the guide’s template for laws in Virginia.

J-Lab is a journalism catalyst that has provided funding and resources for news startups. It has funded 100 startups and pilot projects since 2005.

The Tow-Knight Center for Entrepreneurial Journalism offers educational programs, supports research, and sponsors events to foster sustainable business models for quality journalism. It is part of the City University of New York's Graduate School of Journalism, and funded by The Tow Foundation and The Knight Foundation.

Suyeon Son and David Eads re-worked the authentication mechanism for accessing Google Spreadsheets with the NPR Visuals App Template. This is a significant change for App Template users. Here’s why we did it and how it works.

Most App Template developers only need to consult the Configuring your system and Authenticating sections of this post, provided someone on your team has gone through the process of creating a Google API application and given you credentials.

Why OAuth?

Prior to this change, the App Template accessed Google spreadsheets with a user account and password. These account details were accessed from environment variables stored in cleartext. Storing a password in cleartext is a bad security practice, and the method led to other dubious practices like sharing credentials for a common Google account.

OAuth is a protocol for accessing online resources on behalf of a user without a password. The user must authenticate with the service using her password to allow the app to act on her behalf. In turn the app receives a magic access token. Instead of directly authenticating the user with the service, the application uses the token to access resources.

There are many advantages to this approach. These access tokens can be revoked or invalidated. If used properly, OAuth credentials are always tied to an individual user account. An application can force all users to re-authenticate by resetting the application credentials. Accessing Google Drive resources with this method is also quite a bit faster than our previous technique.

Setting up the Google API application

To use the new OAuth feature of the App Template, you will need to create a Google API project and generate credentials. Typically, you’ll only need to do this once for your entire organization.

Visit the Google Developer’s Console and click “Create Project”.

Give the project a name for the API dashboard and wait for the project to be created:

Give the project a name again (oh, technology!) by clicking “Consent screen” in the left hand toolbar:

Enable the Drive API by clicking “APIs” in the left hand toolbar, searching for “Drive” and enabling the Drive API:

You can optionally disable the default APIs if you’d like.

Finally, create client credentials by clicking “Credentials” in the left hand toolbar and then clicking “Create New Client ID”:

Make sure “Web application” is selected. Set the Javascript origins to “http://localhost:8000”, “http://127.0.0.1:8000”, “http://localhost:8888” and “http://127.0.0.1:8888”. Set the Authorized Redirect URIs to “http://localhost:8000/authenticate/”, “http://127.0.0.1:8000/authenticate/”, “http://localhost:8888/authenticate/” and “http://127.0.0.1:8888”.

Now you have some credentials:

Configuring your system

Whew! Happily, that’s the worst part. Typically, you should only do this once for your whole organization.

Add some environment variables to your .bash_profile or current shell session based on the client ID credentials you created above:

export GOOGLE_OAUTH_CLIENT_ID="825131989533-7kjnu270dqmreatb24evmlh264m8eq87.apps.googleusercontent.com"
export GOOGLE_OAUTH_CONSUMER_SECRET="oy8HFRpHlJ6RUiMxEggpHaTz"
export AUTHOMATIC_SALT="mysecretstring"

As you can see above, you also need to set a random string to act as cryptographic salt for the OAuth library the App Template uses.

Authenticating

Now, run fab app in your App Template project and go to localhost:8000 in your web browser. You’ll be asked to allow the application to access Google Drive on behalf of your account:

If you use multiple Google accounts, you might need to pick one:

Google would like you to know what you’re getting into:

That’s it. You’re good to go!

Bonus: Automatically reloading the spreadsheet

Any route decorated with the @oauth_required decorator can be passed a refresh=1 querystring parameter which will force the latest version of the spreadsheet to be downloaded (e.g. localhost:8000/?refresh=1).

This is intended to improve the local development experience when the spreadsheet is in flux.

Behind the scenes

The new system relies on the awesome Authomatic library (developed by a photojournalist!).

We provide a decorator in oauth.py that wraps a route with a check for valid credentials, and re-routes the user through the authentication workflow if the credentials don’t exist.

Here’s an example snippet to show how it works:

from flask import Flask, render_template
from oauth import oauth_required

app = Flask(__name__)

@app.route('/')
@oauth_required
def index():
    context = {
        ‘title’: ‘My awesome project’,
    }
    return render_template(‘index.html’, **context)

Authomatic provides an interface for serializing OAuth credentials. After successfully authenticating, the App Template writes serialized credentials to a file called ~/.google_oauth_credentials and reads them when needed.

By using the so-called “offline access” option, the credentials can live in perpetuity, though the access token will change from time-to-time. Our implementation hides this step in a function called get_credentials which automatically refreshes the credentials if necessary.

By default, credentials are global – once you’re authenticated for one app template project, you’re authenticated for them all. But some projects may require different credentials – perhaps you normally access the project spreadsheet using your USERNAME@YOURORG.ORG account, but for some reason need to access it using your OTHERUSERNAME@GMAIL.COM account. In this case you can specify a different credentials file in app_config.py by changing GOOGLE_OAUTH_CREDENTIALS_PATH:

GOOGLE_OAUTH_CREDENTIALS_PATH = '~/.special_project_credentials'

Finally, the Google Doc access mechanism has changed. If you need to access a Google spreadsheet that’s not involved with the default COPY rig, use the new get_document(key, file_path) helper function. The function takes two parameters: a spreadsheet key and path to write the exported Excel file. Here’s an example of what you might do:

from copytext import Copy
from oauth import get_document

def read_my_google_doc():
    file_path = 'data/extra_data.xlsx'
    get_document('0AlXMOHKxzQVRdHZuX1UycXplRlBfLVB0UVNldHJYZmc', file_path)
    data = Copy(file_path)

    for row in data['example_list']:
        print '%s: %s' % (row['term'], row['definition'])

read_my_google_doc()