Journocode_
WorkshopsAbout Us

JournoCon 2018: A Recap of our first data-driven conference

by Kira Schacht 0 Comments
JournoCon 2018: A Recap of our first data-driven conference

On March 24th, we hosted the JournoCon 2018. A data journalism bootcamp-slash-conference for students, freelance journalists and anyone wanting to learn more about the field. To those who were there: Thank you for making it an amazing day! To those who weren’t: Here’s a recap.

For beginners, it’s not always easy to get into data-driven work. You need skills you usually don’t learn in journalism schools, and in many newsrooms, there’s no infrastructure in place for journalists to augment their skills and connect with designers and developers. Still, what most journalists, in our experience, have, is motivation. What they generally do not have are

  1. time
  2. money.

So, when planning our first ddj conference, we wanted to make it affordable, even for students and trainees. You can’t teach everything there is to know about data-driven journalism in a day, of course. But we wanted to give learners a starting point, an overview of the steps involved. Basically: If people knew what to google when they got home, we would have done our job for the day. The result – after three months of planning – is the JournoCon 2018: one day, 60 participants, 7 squirrels, 5 amazing speakers from data journalism teams around Germany. We held it in German, so the slides and resources we’ll link to at the bottom aren’t available in English at the moment. But we’ll have more events, no doubt. For now, those of you who don’t speak German will have to make do with this recap, though.

The conference was structured around the data-driven workflow:

This is the simplified version, of course, but it’ll work for our purposes. So, we organized the conference into five blocks, corresponding to the five main steps in working with data. For each block, there were two tracks:

  1. a keynote track with two half-hour segments: Talks, discussions and presentations on aspects of the workflow step
  2. a workshop track with one hour-long course for those who want to try the hands-on approach to ddj.

Thanks to our amazing host, the Infographics Group, the conference took place in a beautiful space in the heart of Berlin. We were also happy to have some of our other supporters present at the conference. For our introduction, we decided to play a little warm-up game to get talking about data-driven journalism.

Finding ideas

For our first segment, we invited leaders from German ddj teams to talk about their process of coming up with ideas. Did you know that, in many cases, data journalism teams are also the ones bringing new technologies to the rest of their newsrooms? Thanks so much to Christina Elmer (Spiegel Online), Götz Gringmuth (rbb) and Michael Kreil (Infographics Group) for participating.

Researching data

Michael Hörz, data journalist at rbb and ddj lecturer, held the first workshop of the day: For some hands-on data research, he gave an overview of data sources available in Germany (You can find his slides here). In the meantime, our Sophie Rotgeri gave a short introduction to data sources in the keynote track as well, supplemented by an introduction to scraping from our hacker-in-chief, Sakander Zirai. If you want to try some hands-on scraping yourself, check out our tutorial on how to scrape a website with Python, R and the tool import.io.

Cleaning messy data

First things first: What is clean data, anyway? In his talk, Journocode squirrel Moritz Zajonz explained what “readable” means to a machine versus a human, and why CSVs and Excel often don’t get along. And cleaning data can be especially tricky when dealing with large amounts of it, as Vanessa Wormer, leader of the Süddeutsche Zeitung data team, knows from her work on the Panama Papers and the Paradise Papers. In her keynote, she presented the most helpful tools to deal with big datasets. For those who wanted to dig into data right away, Kira Schacht held a workshop on how to use Open Refine for data cleaning.

Analyzing data

When Journocode started out, our main goal was to teach each other about data analysis in the statistical programming language R. We haven’t forgotten our roots, so of course, the JournoCon featured an R crash course held by our founder Marie-Louise Timcke. We will publish the course as a tutorial on our website soon, so stay tuned! Until then, you can find all of our previous R tutorials here.

But R is far from the only thing journos can learn about analyzing data: In the keynote track, Moritz Zajonz gave an overview of the key lessons every journalist should know about statistics, and our computer scientists Elena Erdmann and Sakander Zirai talked about how to use machine learning methods in a journalistic way.

 

Data visualization

Finally, we were glad to have Lisa Charlotte Rost on board with a workshop on Datawrapper, a data visualization tool designed to be used in the newsroom. As for talks, we wrapped up the day with a look into the hottest trends in dataviz: Phillip Hafellner of Infographics Group talked all things AR, VR and Gamification. While opening the first beers of the evening, we ended on a little contest: In “painting by numbers”, our participants had five minutes to create a visualization for a dataset they had just learned about. And they came up with some amazing ideas. Take a look:

 

To JournoCon 2019… and beyond!

It was great fun seeing so many motivated people come together. All in all, we’re definitely doing this again! We also got some really helpful feedback from attendees after the conference. We’re going to take all of it to heart for JournoCon 2019, the main points being:

  • more space for workshops
  • longer and more pauses
  • more time for networking. Turns out, journalists really like talking to each other.

Some people asked for more in-depth ddj talks, others were glad we kept it simple, still others thought some points were too specific already. We definitely want to keep the JournoCon beginner-friendly, but we’d also love to give more advanced people the opportunity to talk shop. Maybe that’s going to mean a longer conference, maybe it means more tracks tailored to different levels of expertise. Maybe we’re going to organize more mini-conferences with more in-depth scopes – we don’t know yet. What we do know is that we’re only just getting started.

So, folks: See you for #JournoCon19 – at the latest!

More Info

You can find our conference pad here. It contains links to all our presentations, as well as further resources for many of the conference’s talks and workshops. For more impressions from the conference, be sure to check out the contents of #JournoCon18 on twitter. The complete program and more information on the speakers can also be found on the conference website.

On the website, you’ll also find more info on our supporters, without which the JournoCon 2018 would not have been possible. Special thanks go to Infographics Group Berlin for providing location and logistics.

 

JournoCon 2018: A data-driven conference

by Kira Schacht 0 Comments
JournoCon 2018: A data-driven conference
JournoCon 2018

We are excited to announce our first one-day data journalism boot camp/conference in Berlin! If you’re a German-speaking person interested in data-driven stuff, be sure to check it out! We offer workshops, talks and discussions around every step of the data-driven workflow.

Note: Dear English-speaking person! We’re sorry, but this event will be held in German. Stay tuned for our next events, though. There will definitely be some in English!

Go to Project

Scraping for everyone

by Elena Erdmann 1 Comment
Scraping for everyone

by Sophie Rotgeri, Moritz Zajonz and Elena Erdmann

One of the most important skills for data journalists is scraping. It allows us to download any data that is openly available online as part of a website, even when it’s not supposed to be downloaded: may it be information about the members of parliament or – as in our christmas-themed example – a list of christmas markets in Germany.

There are some tools that don’t require any programming and which are sufficient for many standard scraping tasks. However, programmed scrapers can be fitted precisely to the task at hand. Therefore, some more complicated scraping might require programming.

Here, we explain three ways of scraping: the no-programming-needed online tool import.io and writing a scraper both in R and Python – since those are the two most common programming languages in data journalism. However, if you read the different tutorials, you will see that the steps to programming a scraper are not that different after all. Make your pick:

No programming needed Python R

Scraping with Import.io

There are two steps to scraping a website. With import.io, you first have to select the information important to you. Then the tool will extract the data for you so you can download it.

You start by telling import.io which URL it should look at, in this case the address “http://weihnachtsmarkt-deutschland.de”. When you start a new “Extractor”, as import.io calls it, you will see a graphical interface. It has two tabs: “Edit” will display the website. Import.io analyses the website’s structure and automatically tries to find and highlight structured information for extraction. If it didn’t select the correct information , you can easily change the selection by clicking on the website’s elements you are interested in.

In the other tab, “Data”, the tool shows the selected data as a spreadsheet, which is how the downloaded data will look like. If you are satisfied, click “Extract data from website”.

Now, the actual scraping begins. This might take a few seconds or minutes, depending on the amount of data you told it to scrape. And there we go, all the christmas markets are already listed in a file type of your choosing.

Scraping with Python

There are two steps to scraping a website. First, you download the content from the web. Then, you have to extract the information that is important to you. Good news: Downloading a website is easy as pie. The main difficulty in scraping is the second step, getting from the raw HTML-file to the content that you really want. But no need to worry, we’ll guide you with our step-by-step tutorial.

Let’s get to the easy part first: downloading the html-page from the Internet. In Python, the requests library does that job for you. Install it via pip and import it into your program. To download the webpage, you only have to call requests.get and hand over the url, in this case the address “http://weihnachtsmarkt-deutschland.de”. Get() will return an object and you can display the downloaded website by calling response.text.

Now, the entire website source, with all the christmas markets, is saved on your computer. You basically just need to clean the file, remove all the HTML-markers, the text that you’re not interested in and find the christmas markets. A computer scientist would call this process parsing the file, so you’re about to create a parser.

Before we start on the parser, it is helpful to inspect the website you’re interested in. A basic understanding of front-end technologies comes very handy, but we’ll guide you through it even if you have never seen an HTML-file before. If you have some spare time, check out our tutorial on HTML, CSS and Javascript.

Every browser has the option to inspect the source code of a website. Depending on your Browser, you might have to enable Developer options or you can inspect the code straight away by right-clicking on the website. Below, you can see a screenshot of the developer tools in Chrome. On the left, you find the website – in this example the christmas markets. On the right, you see the HTML source code. The best part is the highlighting: If you click on a line of the source code, the corresponding unit of the website is highlighted. Click yourself through the website, until you find the piece of code that encodes exactly the information that you want to extract: in this case the list of christmas markets.

All the christmas markets are listed in one table. In html, a table starts with an opening <table> tag, and it ends with </table>. The rows of the table are called <tr> (table row) in html. For each christmas market, there is exactly one row in the table.

Now that you have made yourself familiar with the structure of the website and know what you are looking for, you can turn to your Python-Script. Bs4 is a python library, that can be used to parse HTML-files and extract the HTML-elements from it. If you haven’t used it before, install it using pip. From the package, import BeautifulSoup (because the name is so long, we renamed it to bs in the import). Remember, the code of the website is in response.text. Load it by calling BeautifulSoup. As bs4 can be used for different file types, you need to specify the parser. For your HTML-needs, the ‘lxml’ parser is best suited.

Now the entire website is loaded into BeautifulSoup. The cool thing is, BeautifulSoup understands the HTML-format. Thus, you can easily search for all kinds of HTML-elements in the code. The BeautifulSoup method ‘find_all()’ takes the HTML-tag and returns all instances. You can further specify the id or class of an element – but this is not necessary for the christmas markets. As you have seen before, each christmas market is listed as one row in the table, and marked by a <tr> tag. Thus, all you have to do is find all elements with the </tr> tag in the website.

And there we go, all the christmas markets are already listed in your Python output!

However, the data is still quite dirty. If you have a second look at it, you find that each row consist of two data cells, marked by <td>. The first one is the name of the christmas market, and links to a separate website with more information. The second data cell is the location of the christmas market. You can now choose, which part of the data you are interested in. BeautifulSoup lets you extract all the text immediately, by calling .text for the item at hand. For each row, this will give you the name of the christmas market.

Done! Now you have the list of christmas markets ready to work with.

Scraping with R

There are two steps to scraping a website. First, you download the content from the web. Then, you have to extract the information that is important to you. Good news: Downloading a website is easy as pie. The main difficulty in scraping is the second step, getting from the raw HTML-file to the content that you really want. But no need to worry, we’ll guide you with our step-by-step tutorial.

Let’s get to the easy part first: Downloading the HTML-page from the Internet. In R, the “rvest” package packs all the required functions. Install() and library() it (or use “needs()”) and call “read_html()” to.. well, read the HTML from a specified URL, in this case the address “http://weihnachtsmarkt-deutschland.de”. Read_html() will return an XML document. To display the structure, call html_structure().

Now that the XML document is downloaded, we can “walk” down the nodes of its tree structure until we can single out the table. it is helpful to inspect the website you’re interested in. A basic understanding of front-end technologies comes very handy, but we’ll guide you through it even if you have never seen an HTML-file before. If you have some spare time, check out our tutorial on HTML, CSS and Javascript.

Every browser has the option to inspect the source code of a website. Depending on your Browser, you might have to enable Developer options or you can inspect the code straight away by right-clicking on the website. Below, you can see a screenshot of the developer tools in Chrome. On the left, you find the website – in this example the christmas markets. On the right, you see the HTML source code. The best part is the highlighting: If you click on a line of the source code, the corresponding unit of the website is highlighted. Click yourself through the website, until you find the piece of code that encodes exactly the information that you want to extract: in this case the list of christmas markets.

All the christmas markets are listed in one table. In html, a table starts with an opening <table> tag, and it ends with </table>. The rows of the table are called <tr> (table row) in html. For each christmas market, there is exactly one row in the table.

Now that you have made yourself familiar with the structure of the website and know what you are looking for, you can go back to R.

First, we will create a nodeset from the document “doc” with “html_children()”. This finds all children of the specified document or node. In this case we call it on the main XML document, so it will find the two main children: “<head>” and “<body>”. From inspecting the source code of the christmas market website, we know that the <table> is a child of <body>. We can specify that we only want <body> and all of its children with an index.

Now, we have to further narrow it down, we really only want the table and nothing else. To achieve that, we can navigate to the corresponding <table> tag with “html_node()”.This will return a nodeset of one node, the “<table>”. Now, if we just use the handy “html_table()” function –- sounds like it was made just for us! –- we can extract all the information that is inside this HTML table directly into a dataframe.

Done! Now you have a dataframe ready to work with.

Extracting geodata from OpenStreetMap with Osmfilter

by Journocode 2 Comments
Extracting geodata from OpenStreetMap with Osmfilter

A guest post by Hans Hack

When working on map related projects, I often need specific geographical data from OpenStreetMap (OSM) from a certain area. For a recent project of mine, I needed all the roads in Germany in a useful format so I can work with them in a GIS program. So how do I do I get the data to work with? With a useful little program called Osmfilter.

There are various sites that provide OSM datasets for certain areas. However, these datasets include ALL the geodata OSM provides. That means houses, streets, rivers etc – basically everything you see on a normal map. In addition, the file is in a rather inconvenient format for direct usage. A quite complicated way to proceed with the dataset would be to load it into a database and query what you need. This can be time consuming and – depending on the power of your PC – impossible.

If you only need to get info about small areas, I recommend using the site overpass turbo. For larger datasets, there is Osmfilter, which easily lets you filter OSM geodata. With a little help from two other free tools, you will have a dataset you can work with in no time.

 

What you’ll need for this tutorial

 

Get an OSM Dataset

To start out, we need to download an OSM dataset, which is saved in a format called pbf (a format to compress large data sets). For this tutorial, I will use a dataset provided by geofabrik, but there are other sources, too. Lets download the pbf file for Liechtenstein and save it in a folder of your choice. Once you have downloaded the data, open the shell and go to the folder where your new dataset is stored with the command

Prepare your data

Osmfilter only supports the file formats osm and o5m. For fast data processing, using o5m is recommended. You can convert your pbf file to o5m with osmconvert in your shell like this:

This translates to: Use the program osmconvert and convert the file called liechtenstein-latest.osm.pbf to a o5m file called liechtenstein. The -o stands for output.

You will now have the same dataset in the o5m format in the same folder.

Filter your data

Now, you can filter your geodata using the shell that should still be open. The osmfilter command logic is built up as follows:

Let’s look at the part about the filter commands. Here is where you can tell the program which parts of the dataset you need by writing --keep=DATA_I_WANT . Here is an example which creates a file for you called buildings.osm that contains all the buildings (and only the buildings) from the Liechtenstein dataset:

To find out how features are stored and classified in OSM, you can go to this site and look up the feature you want. Tip: You can head over to overpass turbo to test your query on a small area of your choice by using the ‘wizard’.

Of course you can do much more with Osmfilter. You can specify which building type you want. For example, you might only want to look at schools:

You can query multiple features by chaining them. For example if you want all the schools and universities:

You can exclude things by adding the flag –drop. For example, if you don’t want to have buildings that are warehouses but keep everything else:

You can reduce the final file size by dropping extra data on the authors and the version by adding:

You can, of course, combine these flags. Here is a query that gives you all the highway types that cars can use in Lichtenstein without the ones where motor vehicles are not allowed:

You find more on the filtering options on the Osmfilter site with some examples too.

Convert to a useful format

As a final step, you can convert your osm file to the most widely supported geodata format called a Shapefile (the GIS program QGIS can handle osm files, but it sometimes doesn’t work well with large datasets). You can convert your osm file to a Shapefile with the program ogr2ogr like this:

The above command converts the file streets_liechtenstein.osm to the shapefile format and tells it to store it in a folder called streets_shapefiles. In the newly created folder you will find 4 different shapefiles (one for every geometry type). In case of the streets, we are only interested in the file lines.shp. You can open this file in a GIS program of your choice.

Ogr2org also allows you to convert your newly created Shapefiles to other geodata formats that you may need, such as GeoJSON, CSV and many more. Have a look at the ogr2ogr website for more info. If you’re tired of using the shell, try the online tool Mapshaper which allows you to convert your Shapefile file to formats such as GeoJSON, SVG and CSV. The file size for Mapshaper is limited but I have tried it with files bigger than 1 GB.

 

Have fun filtering OSM and happy mapping!

 

From the data to the story: A typical ddj workflow in R

by Marie-Louise Timcke 2 Comments
From the data to the story: A typical ddj workflow in R

R is getting more and more popular among Data Journalists worldwide, as Timo Grossenbacher from SRF Data pointed out recently in a talk at useR!2017 conference in Brussels. Working as a data trainee at Berliner Morgenpost’s Interactive Team, I can confirm that R indeed played an important role in many of our lately published projects, for example when we identified the strongholds of german parties. While we also use the software for more complex statistics from time to time, something that R helps us with on a near-daily basis is the act of cleaning, joining and superficially analyzing data. Sometimes it’s just to briefly check if there is a story hiding in the data. But sometimes, the steps you will learn in this tutorial are just the first part of a bigger, deeper data analysis.

JavaScript: Interactive Leaflet.js map with search bar

by Marie-Louise Timcke 3 Comments
JavaScript: Interactive Leaflet.js map with search bar

In our previous posts about Leaflet.js, we coded an interactive marker map and learned how to update our data with a google spreadsheet. In this short tutorial, we will show you how to make your map searchable so users can find a specific marker.

We’ll use the code of the already finished map that’s hosted on our GitHub page so we don’t have to start at the very beginning again. Simply download the directory to your computer and open it in your text editor. If you’ve never created an interactive map with Leaflet.js before, please have a look at the two tutorials mentioned above before proceeding with this one.

HTML

Adding a search bar to the map isn’t hard at all, thanks to Italian programmer Stefano Cudini. He is the author of Leaflet.Control.Search, a JavaScript library that will do the trick for us. First, download the libraries’ JS and CSS code from his GitHub repository and save it in the map’s directory. It is currently named loop_google_spreadsheet, but you can rename it to whatever you want.

Now, open index.html in the editor and include the two files you just downloaded:

Done. Now we get to the part where we’ll change a little bit more of our map code.

JavaScript

Adding a visual search bar to the map is easy. First, we have to create a new layer which will be searchable later. Then we add the search control. This is how it’s done:

If you want a nice search icon to appear on the bar, download the images directory of Stefano Cudinis library and copy it into your map directory. Now, when you open the index file in a web browser, you should see the search bar on your map.

The search bar is useless as of now, since we haven‘t given the search bar any data to look through. So in the next step, we want to fill markersLayer with the search keywords and the markers’ location. We use the code of the Google Spreadsheet map, so we’ll load the data with Tabletop, like we did in the previous tutorial, and set the URL, size and orientation of our icons. But we’ll delete or comment out the entire part where we actually add the markers on the map:

Now, to add the markers with matching popups to the map and also to the search corpus, we have to add the markers to markersLayer. We define the value searched as the markers’ title and the found position as loc. Then we’ll save every new L.marker as marker, bind the popup and add the whole thing to our layer.

And that’s basically it! Now you can search for the newsrooms’ names and have the map zoom to their corresponding location. You can also add the snippet

to zoom back to the normal level once you close the search bar.

And this is what your result should look like. Click on the search symbol, type in “Dat” and see what happens.

As always, you’ll find the entire code for this example on our GitHub page. If you have any questions, suggestions or feedback, feel free to leave a comment! We’ll try to answer fast.

JavaScript: Loop through a Google Spreadsheet

by Marie-Louise Timcke 2 Comments
JavaScript: Loop through a Google Spreadsheet

In our post Coding marker maps with Leaflet.js we filled a map visualization with information given directly in the code, but we also learned how to loop through data in a JSON file to create markers and pop-ups. This time, we will loop through an public Google Spreadsheet. We will do this with the same example map as in the previous post. But this little trick will not only help you with building marker maps but can come in handy for any JavaScript application you need to automate the handling of your data.

So let’s start right away with setting up a data table with Google Spreadsheets.

The Journocode Advent calendar is here!

by Moritz Zajonz 1 Comment
The Journocode Advent calendar is here!

Presenting you a different ddj tool every day – for 24 days.

It’s been over a year now since we started our weekly meet-up. In this time, we discovered a bunch of awesome, helpful tools. And because it’s cozy winter Christmas time, we wanted to tell you about them! There will be something for everyone – from data scraping, extraction and cleaning to location, visualization and web development tools. So if you want to have a look at the different data journalism tools we discovered, visit advent.journocode.com!

PS: There’s also music! Don’t worry – it’s not ‘Last Christmas’, we swear. To listen to the Christmas carol, simply hover over the speaker icon in the bottom left corner.

Go to Project