earth-osm

📚 Overview

earth-osm downloads, filters, cleans and exports infrastructure data from OpenStreetMap (OSM). It provides a Python API and a CLI interface to extract data for various infrastructure types, such as power lines, substations, and more.

🌟 Key Features

• 🔌 Extracts infrastructure data from OSM
• 📅 Historical data support (specify target dates)
• 🧹 Cleans and standardizes the data (coming soon)
• 🚀 No API rate limits (data served from GeoFabrik)
• 🐍 Provides a Python API
• 🖥️ Supports multiprocessing for faster extraction
• 📊 Outputs data in .csv and .geojson formats
• 🌍 Supports planetary data extraction
• 🖱️ Easy-to-use CLI interface
• 🌐 Dual data sources: GeoFabrik (default) and live Overpass API

🚀 Getting Started

Installation

Install earth-osm using pip (recommended):

pip install earth-osm

Or with conda:

conda install --channel=conda-forge earth-osm

Basic Usage

Extract OSM data using the CLI:

earth_osm extract power --regions benin monaco --features substation line

This command extracts power infrastructure data for Benin and Monaco, focusing on substations and power lines. By default, the resulting .csv and .geojson files are stored in ./earth_data/out.

Load the extracted data using pandas:

import pandas as pd
import geopandas as gpd

# For Pandas
df_substations = pd.read_csv('./earth_data/out/BJ_raw_substations.csv')

# For GeoPandas
gdf_substations = gpd.read_file('./earth_data/out/BJ_raw_substations.geojson')

Optional: Speed up downloads with aria2c

Install aria2c (for example brew install aria2, sudo apt-get install aria2, or choco install aria2) and set EO_PARALLEL_DOWNLOADS greater than 1; earth-osm will then use multiple connections for .osm.pbf downloads and otherwise falls back automatically.

export EO_PARALLEL_DOWNLOADS=8    # fish: set -x EO_PARALLEL_DOWNLOADS 8
earth_osm extract power --regions germany --features line

🛠️ CLI Reference

Extract Command

earth_osm extract <primary> --regions <region1> <region2> ... [options]

Arguments:

• <primary>: Primary feature to extract (e.g power)

Required Options:

• --regions: Specify one or more regions using ISO 3166-1 alpha-2, ISO 3166-2 codes, or full names

Tip: A list of regions is available at regions.md

Optional Arguments:

Argument	Description	Default
--features	Specify sub-features of the primary feature	All features
--update	Update existing data	False
--mp [N]	Enable multiprocessing, optionally capped at N processes	Auto (all CPUs but one)
--no_mp	Disable multiprocessing	False (MP enabled)
--data_dir	Path to data directory	‘./earth_data’
--out_dir	Path to output directory	Same as data_dir
--out_format	Export format(s): csv and/or geojson	[‘csv’, ‘geojson’]
--agg_feature	Aggregate outputs by feature	False
--agg_region	Aggregate outputs by region	False
--source	Data source: geofabrik (default) or overpass	geofabrik

ℹ️ When using the Overpass backend, wildcard feature selections such as ALL_power are not supported. Specify concrete feature values instead, or switch to the Geofabrik source for wildcard exports.

Overpass request settings can be overridden with environment variables:

Environment variable	Description	Default
EO_OVERPASS_ENDPOINT	Overpass API interpreter endpoint	https://overpass-api.de/api/interpreter
EO_OVERPASS_REQUEST_TIMEOUT	HTTP request timeout in seconds	600
EO_OVERPASS_QUERY_TIMEOUT	Overpass query timeout in seconds	300

Planet-wide extractions

Use the special earth region to export the full OpenStreetMap planet snapshot.

earth_osm extract power --regions earth --features line 

🐍 Python API

For more advanced usage, you can use the Python API:

import earth_osm as eo

eo.save_osm_data(
    primary_name='power',
    region_list=['benin', 'monaco'],
    feature_list=['substation', 'line'],
    update=False,
    mp=True,
    data_dir='./earth_data',
    out_format=['csv', 'geojson'],
    out_aggregate=True,
    data_source='geofabrik'  # or 'overpass'
)

For historical data, add the target_date parameter:

from datetime import datetime

# Download historical data for a specific date
eo.get_osm_data(
    region_str='malta',
    primary_name='power',
    feature_name='line',
    target_date=datetime(2020, 1, 1)  # Jan 1, 2020
)

🛠️ Development

To contribute to earth-osm, follow these steps:

1. (Optional) Install a specific version of earth_osm:

   pip install git+https://github.com/pypsa-meets-earth/earth-osm.git@<required-commit-hash>
   

2. (Optional) Create a virtual environment for Python >=3.10:

   python3 -m venv .venv
   source .venv/bin/activate
   

3. Install the development dependencies:

   pip install git+https://github.com/pypsa-meets-earth/earth-osm.git
   pip install -e .[dev]
   

4. Read the CONTRIBUTING.md file for more detailed information on how to contribute to the project.

🤝 Community

Join our Discord community to connect with other users and contributors, ask questions, and get support.

📚 Documentation

For more detailed information, check out our full documentation.

---

Made with ❤️ by the PyPSA meets Earth team