Useful Data Sources

--- layout: page title: Data permalink: /data/ --- ## Open Data Everyone enjoys discovering [interesting datasets](http://rs.io/100-interesting-data-sets-for-statistics/), but useful datasets are even better. The problem is that the open data movement has been too successful by some measures. We have gone from a relatively data poor environment in government and nonprofits to an amazing repositories of open data assets, including [Socrata](https://socrata.com/solutions/open-data-citizen-engagement/) and [CKAN's](https://ckan.org/about/instances/) Open Government data portals, the US Federal portal at [www.data.gov](https://www.data.gov/), and over 80,000 academic datasets posted on [Dataverse](https://dataverse.harvard.edu/). Data is more useful when it comes with a [vignette](https://ds4ps.github.io/PROG-EVAL-III/Datasets_in_R.html) that provides some information about the nature of the data and potential uses. We will work to update the site with some helpful datasets for public affairs programs in the near future. In the meantime, enjoy some of these resources: **OPEN DATA** ----------------------- * TOC {:toc} ----------------------- ### Overview * Background on the Open Data Movement [ [link](http://www.urban.org/sites/default/files/alfresco/publication-pdfs/413153-Putting-Open-Data-to-Work-for-Communities.PDF) ] * Ben Wellington's TED Talk on Open Data in NYC [ [link](https://www.ted.com/talks/ben_wellington_how_we_found_the_worst_place_to_park_in_new_york_city_using_big_data?) ] ### DATA Act * The Data Transparency Act [ [overview](/s/Electronic_Version___DATA_Act_Vision_and_Value.pdf) ] [ [link](http://www.datacoalition.org/what-is-data-transparency/data-act/) ] [ [link](http://labs.data.gov/dashboard/offices) ] [ [link](http://www.forbes.com/sites/techonomy/2014/09/12/how-open-data-is-transforming-city-life/) ] * Keynote Speech on Importance of DATA Act [ [link](https://www.volckeralliance.org/publications/data-act-good-use-scarce-government-resources) ] * Progress Tracker on Federal Open Data Compliance [ [link](http://labs.data.gov/dashboard/offices) ] ### Impact of Open Data * I Quant NY [ [budget error](https://iquantny.tumblr.com/post/147446103684/open-data-reveals-791-million-error-in-newly) ] [ [metro fares](https://iquantny.tumblr.com/post/114470101209/i-quant-a-victory-mta-adds-new-button-for) ] [ [parking tickets](https://gizmodo.com/justice-the-fire-hydrant-that-earned-nyc-33-000-a-yea-1585633742) ] * Realizing the Promise of Big Data: IBM Center for Gov. [ [link](http://www.businessofgovernment.org/sites/default/files/Realizing%20the%20Promise%20of%20Big%20Data_0.pdf) ] * Data Used in 2017 Public Policy Dissertations [ [link](https://ds4ps.org/2019/03/20/data-for-policy-dissertations.html) ][ [broken link](http://publicmanagementresearch.com/2017/12/18/data-for-dissertations-december-18-2017/) ] ### Guides & Best Practices * Project Open Data [ [link](https://project-open-data.cio.gov/) ] [ [principles](https://project-open-data.cio.gov/principles/) ] * Open North standards [ [link](http://www.opennorth.ca/publications) ] * Sunlight Foundation's Open Data Guidelines [ [link](http://sunlightfoundation.com/opendataguidelines/) ] * Global Impact of Open Data Book: GovLab / O'Reilly [ [link](http://www.oreilly.com/data/free/files/the-global-impact-of-open-data.pdf) ] * The Hidden Cost (and Benefits) of Open Data [ [link](http://www.governing.com/columns/tech-talk/gov-open-data-cost-problems.html) ] * Data Maturity Framework [ [link](https://dsapp.uchicago.edu/home/resources/datamaturity/) ] * How to Share Data for Collaboration [ [link](https://peerj.com/preprints/3139v5.pdf) ] ### Government Portal and Resources * How to Make Government Data Sites Better [ [link](http://flowingdata.com/2014/06/10/how-to-make-government-data-sites-better/) ] [ [link](http://blogs.scientificamerican.com/guest-blog/what-s-wrong-with-open-data-sites-and-how-we-can-fix-them/) ] * US Cities Open Data Census [ [link](http://us-city.census.okfn.org/) ] * Statewide Portal Tested in California [ [link](http://www.governing.com/topics/mgmt/tns-california-open-data.html) ] * Five Largest Cities Now Have Open Data Policies [ [link](http://sunlightfoundation.com/blog/2014/10/15/all-five-of-the-largest-u-s-cities-now-have-open-data-policies/) ] * 40 Brilliant Open Data Projects for Smart Cities [[ link](https://carto.com/blog/forty-brilliant-open-data-projects) ] ### Machine Learning Training Data * Top Sources for Machine Learning Datasets [ [link](https://vas3k.com/blog/machine_learning/) ] ----------------------- # Useful Data Sources ### APIs * Awesome Public Datasets Page [ [GitHub](https://github.com/awesomedata/awesome-public-datasets) ] * Quandl API (many data sources) [ [link](https://www.quandl.com/) ] [ [r package](https://www.quandl.com/help/r) ] * Census Data API [ [acs package](http://eglenn.scripts.mit.edu/citystate/wp-content/uploads/2013/02/wpid-working_with_acs_R2.pdf) ] [ [census api](http://rstudio-pubs-static.s3.amazonaws.com/19337_2e7f827190514c569ea136db788ce850.html) ] * TwitteR Package API [ [link](http://davetang.org/muse/2013/04/06/using-the-r_twitter-package/) ] * 19 Free Public Datasets (Springboard blog) [ [link](https://www.springboard.com/blog/free-public-data-sets-data-science-project/) ] * ckanr [ [github](https://github.com/ropensci/ckanr) ] [ [vignette](https://cran.r-project.org/web/packages/ckanr/vignettes/ckanr_vignette.html) ] * Rsocrata [ [github](https://github.com/Chicago/RSocrata) ] * censusapi Package [ [github](https://github.com/hrecht/censusapi) ] [ [slides](http://urbaninstitute.github.io/R-Trainings/accesing-census-apis/presentation/index.html#/) ] [ [tutorial](/s/CensusAPI_Package.html) ] * @unitedstates [ [about](https://sunlightfoundation.com/2013/08/20/a-modern-approach-to-open-data/) ] [ [github](https://github.com/unitedstates) ] * Data USA [ [link](http://datausa.io/) ] [ [documentation](https://gist.github.com/lecy/0aa782a873cd174573f32d243233ca5b) ] * Data Science Toolkit [ [link](http://www.datasciencetoolkit.org/) ] [ [rpackage](http://files.meetup.com/1696476/DRUG.pdf) ] * Federal Government APIs [ [link](https://github.com/unitedstates/APIs) ] * Strava GPS Data of Athletes by City [ [blog](http://www.databrew.cc/posts/strava.html) ] * rtimes Package: NYTimes API for government data [ [link](https://github.com/ropengov/rtimes) ] * rsunlight Package: Wrapper for the Open Congress and Open States APIs [ [link](https://github.com/ropengov/rsunlight) ] ### Data for Teaching * A Little Stats [ [link](http://alittlestats.blogspot.com/p/data-sources.html) ] * Fun Data for Teaching [ [link](http://bartomeuslab.com/2016/01/21/fun-data-for-teaching-r/) ] * Forbes: 35 Open Data Sources of Note [ [link](http://www.forbes.com/sites/bernardmarr/2016/02/12/big-data-35-brilliant-and-free-data-sources-for-2016/#2a8d98876796) ] * 100 Interesting Datasets [ [link](http://rs.io/100-interesting-data-sets-for-statistics/) ] * Data and Story Library [ [link](https://dasl.datadescription.com/) ] ### Open Data for the Nonprofit Sector * Urban Institute's NCCS Data [ [link](https://nccs-data.urban.org/index.php) ] * Nonprofit Open Data Collective [ [link](https://nonprofit-open-data-collective.github.io/overview/) ] * Bureau of Labor Statistics Employment Data by Sector [ [link](https://www.bls.gov/bdm/nonprofits/nonprofits.htm) ] * Association of Religious Archives Congration Data by County 1950-2010 [ [link](http://www.thearda.com/archive/browse.asp) ] ### Poverty Action Lab Catalog of Administrative Data A guide to data sources that have been used as the basis of sampling frameworks for randomized control trials (RCTs) in the US. * JPAL Catalog of Data [ [link](https://www.povertyactionlab.org/sites/default/files/documents/AdminDataCatalog.pdf) ] * JPAL Publications with Data Available [ [link](https://www.povertyactionlab.org/evaluations?f[0]=field_external_data:title:1) ] ### Data-Driven Journalism Project Portals Data journalists are making their stories transparent by posting the data and code used for their work so that it can be easily replicated or the work can be extended. * BBC creates graphics cookbook [ [link](https://medium.com/bbc-visual-and-data-journalism/how-the-bbc-visual-and-data-journalism-team-works-with-graphics-in-r-ed0b35693535) ] [ [cookbook](https://bbc.github.io/rcookbook/) ] * Buzzfeed [ [all projects on GitHub](https://github.com/BuzzFeedNews/everything) ] * LA Times [ [datadesk on GitHub](https://github.com/datadesk) ] * Washington Post [ [projects on GitHub](https://github.com/washingtonpost) ] * Associated Press [ [GitHub](https://github.com/associatedpress) ] [ [project template](https://github.com/associatedpress/datakit-core) ] [ [example](http://data.ap.org/projects/2017/federal-judges/processed/Federal_Judiciary_Diversity.html) ] * The Economist [ [GitHub](https://github.com/TheEconomist) ] * Center for Public Integrity [ [GitHub](https://github.com/PublicI) ] [ [Workplace Descrimination Story](https://www.buzzfeednews.com/article/lamvo/eeoc-sexual-harassment-data) ] ### Disaster Management SHELDUS Database [ [link](https://cemhs.asu.edu/node/7) ] ### Police Data Initiative The Police Data Initiative is a law enforcement community of practice that includes leading law enforcement agencies, technologists, and researchers committed to engaging their communities in a partnership to improve public safety that is built on a foundation of trust, accountability and innovation. The PDI represents the great work and leadership of more than 130 law enforcement agencies who have released more than 200 datasets to date, and originated as a result of several recommendations in the Task Force on 21st Century Policing that focused on technology and transparency. [Available Datasets](https://www.policedatainitiative.org/datasets/) ### Data Blogs * Data is Plural by Jeremy Singer-Vine [ [archive](https://tinyletter.com/data-is-plural/archive) ] ----------------------- # "Awesome Data" Catalog This is just a sample of some datasets that would be relevant to the public and nonprofit sectors from the larger catalog of open public sources curated and managed by [AwesomeData](https://github.com/awesomedata/awesome-public-datasets). Agriculture ----------- * [Hyperspectral benchmark dataset on soil moisture](https://doi.org/10.5281/zenodo.1227837) * [U.S. Department of Agriculture's Nutrient Database](https://www.ars.usda.gov/northeast-area/beltsville-md/beltsville-human-nutrition-research-center/nutrient-data-laboratory/docs/sr28-download-files/) * [U.S. Department of Agriculture's PLANTS Database](http://www.plants.usda.gov/dl_all.html) Climate+Weather --------------- * [Actuaries Climate Index](http://actuariesclimateindex.org/data/) * [Australian Weather](http://www.bom.gov.au/climate/dwo/) * [Aviation Weather Center - Consistent, timely and accurate weather [...]](https://aviationweather.gov/adds/dataserver) * [Brazilian Weather - Historical data (In Portuguese) - Data related to [...]](http://sinda.crn.inpe.br/PCD/SITE/novo/site/historico/index.php) * [Canadian Meteorological Centre](http://weather.gc.ca/grib/index_e.html) * [Climate Data from UEA (updated monthly)](http://www.cru.uea.ac.uk/data/) * [European Climate Assessment & Dataset](http://eca.knmi.nl/) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//Climate+Weather/European-Climate-Assessment-&-Dataset.yml) ] * [Global Climate Data Since 1929](http://en.tutiempo.net/climate) * [NASA Global Imagery Browse Services](https://wiki.earthdata.nasa.gov/display/GIBS) * [NOAA Bering Sea Climate](http://www.beringclimate.noaa.gov/) * [NOAA Climate Datasets](http://www.ncdc.noaa.gov/data-access/quick-links) * [NOAA Realtime Weather Models](http://www.ncdc.noaa.gov/data-access/model-data/model-datasets/numerical-weather-prediction) * [NOAA SURFRAD Meteorology and Radiation Datasets](https://www.esrl.noaa.gov/gmd/grad/stardata.html) * [The World Bank Open Data Resources for Climate Change](http://data.worldbank.org/developers/climate-data-api) * [UEA Climatic Research Unit](http://www.cru.uea.ac.uk/data) * [WU Historical Weather Worldwide](https://www.wunderground.com/history/index.html) * [WorldClim - Global Climate Data](http://www.worldclim.org) ComplexNetworks --------------- * [DBLP Citation dataset](https://kdl.cs.umass.edu/display/public/DBLP) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//ComplexNetworks/DBLP-Citation-dataset.yml) ] * [DIMACS Road Networks Collection](http://www.dis.uniroma1.it/challenge9/download.shtml) * [NBER Patent Citations](http://nber.org/patents/) DataChallenges -------------- * [DrivenData Competitions for Social Good](http://www.drivendata.org/) * [Localytics Data Visualization Challenge](https://github.com/localytics/data-viz-challenge) * [Netflix Prize](http://netflixprize.com/leaderboard.html) Economics --------- * [American Economic Association (AEA)](https://www.aeaweb.org/resources/data) * [EconData from UMD](http://inforumweb.umd.edu/econdata/econdata.html) * [Economic Freedom of the World Data](http://www.freetheworld.com/datasets_efw.html) * [Historical MacroEconomic Statistics](http://www.historicalstatistics.org/) * [INFORUM - Interindustry Forecasting at the University of Maryland](http://inforumweb.umd.edu/) * [International Economics Database](https://db.nomics.world/) * [International Trade Statistics](http://www.econostatistics.co.za/) * [Internet Product Code Database](http://www.upcdatabase.com/) * [Joint External Debt Data Hub](http://www.jedh.org/) * [Jon Haveman International Trade Data Links](http://www.macalester.edu/research/economics/PAGE/HAVEMAN/Trade.Resources/TradeData.html) * [OpenCorporates Database of Companies in the World](https://opencorporates.com/) * [Our World in Data](http://ourworldindata.org/) * [SciencesPo World Trade Gravity Datasets](http://econ.sciences-po.fr/thierry-mayer/data) * [The Atlas of Economic Complexity](http://atlas.cid.harvard.edu) * [The Center for International Data](http://cid.econ.ucdavis.edu) * [The Observatory of Economic Complexity](http://atlas.media.mit.edu/en/) * [UN Commodity Trade Statistics](http://comtrade.un.org/db/) * [UN Human Development Reports](http://hdr.undp.org/en) Education --------- * [College Scorecard Data](https://collegescorecard.ed.gov/data/) * [Student Data from Free Code Camp](https://github.com/freeCodeCamp/open-data) Foundations ----------- * International Aid Transparency Initiative (iati) [ [ database of grants](https://iatistandard.org/en/) ] * Ford Foundation Grants [ [database](https://www.fordfoundation.org/work/our-grants/grants-database/grants-all) ] * Hewlett Foundation Grants [ [database](https://hewlett.org/grants/?sort=date) ] GIS --- * [ArcGIS Open Data portal](http://opendata.arcgis.com/) * [Cambridge, MA, US, GIS data on GitHub](http://cambridgegis.github.io/gisdata.html) * [Factual Global Location Data](https://places.factual.com/data/t/places) * [IEEE Geoscience and Remote Sensing Society DASE Website](http://dase.grss-ieee.org) * [Geo Maps - High Quality GeoJSON maps programmatically generated](https://github.com/simonepri/geo-maps) * [Geo Spatial Data from ASU](http://geodacenter.asu.edu/datalist/) * [Geo Wiki Project - Citizen-driven Environmental Monitoring](http://geo-wiki.org/) * [GeoFabrik - OSM data extracted to a variety of formats and areas](http://download.geofabrik.de/) * [GeoNames Worldwide](http://www.geonames.org/) * [Global Administrative Areas Database (GADM) - Geospatial data organized [...]](https://gadm.org/) * [Homeland Infrastructure Foundation-Level Data](https://hifld-geoplatform.opendata.arcgis.com/) * [Landsat 8 on AWS](https://aws.amazon.com/public-data-sets/landsat/) * [List of all countries in all languages](https://github.com/umpirsky/country-list) * [National Weather Service GIS Data Portal](http://www.nws.noaa.gov/gis/) * [Natural Earth - vectors and rasters of the world](http://www.naturalearthdata.com/) * [OpenAddresses](http://openaddresses.io/) * [OpenStreetMap (OSM)](http://wiki.openstreetmap.org/wiki/Downloading_data) * [Pleiades - Gazetteer and graph of ancient places](http://pleiades.stoa.org/) * [Reverse Geocoder using OSM data](https://github.com/kno10/reversegeocode) * [Robin Wilson - Free GIS Datasets](http://freegisdata.rtwilson.com) * [TIGER/Line - U.S. boundaries and roads](https://www.census.gov/geo/maps-data/data/tiger-line.html) * [TZ Timezones shapfiles](http://efele.net/maps/tz/world/) * [TwoFishes - Foursquare's coarse geocoder](https://github.com/foursquare/twofishes) * [UN Environmental Data](http://geodata.grid.unep.ch/) * [World boundaries from the U.S. Department of State](http://geonode.state.gov/layers/?limit=100&offset=0) * [World countries in multiple formats](https://github.com/mledoze/countries) Government ---------- * [Alberta, Province of Canada](http://open.alberta.ca) * [Antwerp, Belgium](http://opendata.antwerpen.be/datasets) * [Argentina (non official)](http://datar.noip.me/) * [Datos Argentina - Portal de datos abiertos de la República Argentina. [...]](http://datos.gob.ar/) * [Austin, TX, US](https://data.austintexas.gov/) * [Australia (abs.gov.au)](http://www.abs.gov.au/AUSSTATS/[email protected]/DetailsPage/3301.02009?OpenDocument) * [Australia (data.gov.au)](https://data.gov.au/) * [Austria (data.gv.at)](https://www.data.gv.at/) * [Baton Rouge, LA, US](https://data.brla.gov/) * [Belgium](http://data.gov.be/) * [Brazil](http://dados.gov.br/dataset) * [Buenos Aires, Argentina](http://data.buenosaires.gob.ar/) * [Calgary, AB, Canada](https://data.calgary.ca/) * [Cambridge, MA, US](https://data.cambridgema.gov/) * [Canada](http://open.canada.ca/) * [Chicago](https://data.cityofchicago.org/) * [Chile](http://datos.gob.cl/dataset) * [China](http://data.stats.gov.cn/english/) * [Dallas Open Data](https://www.dallasopendata.com/) * [DataBC - data from the Province of British Columbia](http://www.data.gov.bc.ca/) * [Denver Open Data](http://data.denvergov.org//) * [Durham, NC Open Data](https://live-durhamnc.opendata.arcgis.com/) * [Edmonton, AB, Canada](https://data.edmonton.ca/) * [England LGInform](http://lginform.local.gov.uk/) * [EuroStat](http://ec.europa.eu/eurostat/data/database) * [EveryPolitician - Ongoing project collating and sharing data on every [...]](http://everypolitician.org/) * [Federal Committee on Statistical Methodology (FCSM) (formerly FedStats)](https://nces.ed.gov/FCSM/index.asp) * [Finland](https://www.opendata.fi/en) * [France](https://www.data.gouv.fr/en/datasets/) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//Government/France.yml) ] * [Fredericton, NB, Canada](http://www.fredericton.ca/en/citygovernment/Catalogue.asp) * [Gatineau, QC, Canada](http://www.gatineau.ca/donneesouvertes/default_fr.aspx) * [Germany](https://www-genesis.destatis.de/genesis/online) * [Ghent, Belgium](https://data.stad.gent/data) * [Glasgow, Scotland, UK](https://data.glasgow.gov.uk/) * [Greece](http://www.data.gov.gr/) * [Guardian world governments](http://www.guardian.co.uk/world-government-data) * [Halifax, NS, Canada](https://www.halifax.ca/home/open-data) * [Helsinki Region, Finland](http://www.hri.fi/en/) * [Hong Kong, China](https://data.gov.hk/en/) * [Houston, TX, US](http://data.houstontx.gov/) * [Indian Government Data](https://data.gov.in/) * [Indonesian Data Portal](http://data.go.id/) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//Government/Indonesian-Data-Portal.yml) ] * [Ireland's Open Data Portal](https://data.gov.ie/data) * [Italy - Il Portale dati.gov.it è il catalogo nazionale dei metadati [...]](https://www.dati.gov.it/) * [Japan](http://www.e-stat.go.jp/SG1/estat/eStatTopPortalE.do) * [Laval, QC, Canada](http://www.laval.ca/Pages/Fr/Citoyens/donnees.aspx) * [Lexington, KY](http://data.lexingtonky.gov/) * [London Datastore, UK](http://data.london.gov.uk/dataset) * [London, ON, Canada](http://www.london.ca/city-hall/open-data/Pages/default.aspx) * [Los Angeles Open Data](https://data.lacity.org/) * [Luxembourg - Luxembourgish Open Data Portal](https://data.public.lu/en/) * [MassGIS, Massachusetts, U.S.](http://www.mass.gov/anf/research-and-tech/it-serv-and-support/application-serv/office-of-geographic-information-massgis/) * [Metropolitain Transportation Commission (MTC), California, US](http://mtc.ca.gov/tools-resources/data-tools/open-data-library) * [Mexico](https://datos.gob.mx/busca/dataset) * [Missisauga, ON, Canada](http://www.mississauga.ca/portal/residents/publicationsopendatacatalogue) * [Moldova](http://data.gov.md/) * [Moncton, NB, Canada](http://open.moncton.ca/) * [Montreal, QC, Canada](http://donnees.ville.montreal.qc.ca/) * [Mountain View, California, US (GIS)](http://data-mountainview.opendata.arcgis.com/) * [NYC Open Data](https://opendata.cityofnewyork.us/) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//Government/NYC-Open-Data.yml) ] * [NYC betanyc](http://betanyc.us/) * [Netherlands](https://data.overheid.nl/) * [New Zealand](http://www.stats.govt.nz/browse_for_stats.aspx) * [OECD](https://data.oecd.org/) * [Oakland, California, US](https://data.oaklandnet.com/) * [Oklahoma](https://data.ok.gov/) * [Open Data for Africa](http://opendataforafrica.org/) * [Open Government Data (OGD) Platform India](https://data.gov.in/) * [OpenDataSoft's list of 1,600 open data](https://www.opendatasoft.com/a-comprehensive-list-of-all-open-data-portals-around-the-world/) * [Oregon](https://data.oregon.gov/) * [Ottawa, ON, Canada](http://data.ottawa.ca/en/) * [Palo Alto, California, US](http://data.cityofpaloalto.org/home) * [OpenDataPhilly - OpenDataPhilly is a catalog of open data in the [...]](https://www.opendataphilly.org/) * [Portland, Oregon](https://www.portlandoregon.gov/28130) * [Portugal - Pordata organization](http://www.pordata.pt/en/Home) * [Puerto Rico Government](https://data.pr.gov//) * [Quebec City, QC, Canada](http://donnees.ville.quebec.qc.ca/) * [Quebec Province of Canada](https://www.donneesquebec.ca/en/) * [Regina SK, Canada](http://open.regina.ca/) * [Rio de Janeiro, Brazil](http://www.data.rio/) * [Romania](http://data.gov.ro/) * [Russia](http://data.gov.ru) * [San Diego, CA](https://data.sandiego.gov) * [San Antonio, TX - Community Information Now - CI:Now is a nonprofit [...]](http://cinow.info/) * [San Francisco Data sets](http://datasf.org/) * [San Jose, California, US](http://data.sanjoseca.gov/) * [San Mateo County, California, US](https://data.smcgov.org/) * [Saskatchewan, Province of Canada](http://opendatask.ca/data/) * [Seattle](https://data.seattle.gov/) * [Singapore Government Data](https://data.gov.sg/) * [South Africa Trade Statistics](http://www.econostatistics.co.za/) * [South Africa](http://www.statssa.gov.za/) * [State of Utah, US](https://opendata.utah.gov/) * [Switzerland](http://www.opendata.admin.ch/) * [Taiwan gov](https://data.gov.tw/) * [Taiwan](http://data.gov.tw/) * [Tel-Aviv Open Data](https://opendata.tel-aviv.gov.il/index_en.html#/) * [Texas Open Data](https://data.texas.gov/) * [The World Bank](https://openknowledge.worldbank.org/handle/10986/2124) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//Government/The-World-Bank.yml) ] * [Toronto, ON, Canada](https://portal0.cf.opendata.inter.sandbox-toronto.ca/) * [Tunisia](http://www.data.gov.tn/) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//Government/Tunisia.yml) ] * [U.K. Government Data](http://data.gov.uk/data) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//Government/U.K.-Government-Data.yml) ] * [U.S. American Community Survey](https://www.census.gov/programs-surveys/acs/data.html/) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//Government/U.S.-American-Community-Survey.yml) ] * [U.S. CDC Public Health datasets](https://www.cdc.gov/nchs/data_access/ftp_data.htm) * [U.S. Census Bureau](http://www.census.gov/data.html) * [U.S. Department of Housing and Urban Development (HUD)](http://www.huduser.gov/portal/datasets/pdrdatas.html) * [U.S. Federal Government Agencies](http://www.data.gov/metrics) * [U.S. Federal Government Data Catalog](http://catalog.data.gov/dataset) * [U.S. Food and Drug Administration (FDA)](https://open.fda.gov/index.html) * [U.S. National Center for Education Statistics (NCES)](http://nces.ed.gov/) * [U.S. Open Government](http://www.data.gov/open-gov/) * [UK 2011 Census Open Atlas Project](https://data.cdrc.ac.uk/product/cdrc-2011-census-open-atlas) * [U.S. Patent and Trademark Office (USPTO) Bulk Data Products](https://www.uspto.gov/learning-and-resources/bulk-data-products) * [Uganda Bureau of Statistics](http://www.ubos.org/unda/index.php/catalog) * [Ukraine](https://data.gov.ua/) * [United Nations](http://data.un.org/) * [Uruguay](https://catalogodatos.gub.uy/) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//Government/Uruguay.yml) ] * [Valley Transportation Authority (VTA), California, US](https://data.vta.org/) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//Government/Valley-Transportation-Authority-VTA-California-US.yml) ] * [Vancouver, BC Open Data Catalog](http://data.vancouver.ca/datacatalogue/) * [Victoria, BC, Canada](http://opendata.victoria.ca/) * [Vienna, Austria](https://open.wien.gv.at/site/open-data/) * [U.S. Congressional Research Service (CRS) Reports](https://www.everycrsreport.com/) Healthcare ---------- * [Composition of Foods Raw, Processed, Prepared USDA National Nutrient Database for Standard [...]](https://data.nal.usda.gov/dataset/composition-foods-raw-processed-prepared-usda-national-nutrient-database-standard-reference-release-27) * [EHDP Large Health Data Sets](http://www.ehdp.com/vitalnet/datasets.htm) * [GDC - GDC supports several cancer genome programs for CCG, TCGA, TARGET etc.](https://gdc.cancer.gov/) * [Gapminder World demographic databases](http://www.gapminder.org/data/) * [MeSH, the vocabulary thesaurus used for indexing articles for PubMed](https://www.nlm.nih.gov/mesh/filelist.html) * [Medicare Coverage Database (MCD), U.S.](https://www.cms.gov/medicare-coverage-database/) * [Medicare Data Engine of medicare.gov Data](https://data.medicare.gov/) * [Medicare Data File](http://go.cms.gov/19xxPN4) * [Number of Ebola Cases and Deaths in Affected Countries (2014)](https://data.humdata.org/dataset/ebola-cases-2014) * [Open-ODS (structure of the UK NHS)](http://www.openods.co.uk) * [OpenPaymentsData, Healthcare financial relationship data](https://openpaymentsdata.cms.gov) * [PhysioBank Databases - A large and growing archive of physiological data.](https://www.physionet.org/physiobank/database/) * [The Cancer Imaging Archive (TCIA)](https://www.cancerimagingarchive.net) * [The Cancer Genome Atlas project (TCGA)](https://portal.gdc.cancer.gov/) * [World Health Organization Global Health Observatory](http://www.who.int/gho/en/) * [Informatics for Integrating Biology & the Bedside](https://www.i2b2.org/NLP/DataSets/Main.php) PublicDomains ------------- * [Amazon](http://aws.amazon.com/datasets/) * [Archive.org Datasets](https://archive.org/details/datasets) * [Archive-it from Internet Archive](https://www.archive-it.org/explore?show=Collections) * [CMU JASA data archive](http://lib.stat.cmu.edu/jasadata/) * [CMU StatLab collections](http://lib.stat.cmu.edu/datasets/) * [Data.World](https://data.world) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//PublicDomains/Data.World.yml) ] * [Data360](http://www.data360.org/index.aspx) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//PublicDomains/Data360.yml) ] * [Enigma Public](https://public.enigma.com/) * [Google](http://www.google.com/publicdata/directory) * [Grand Comics Database - The Grand Comics Database (GCD) is a nonprofit, [...]](https://www.comics.org) * [Infochimps](http://www.infochimps.com/) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//PublicDomains/Infochimps.yml) ] * [KDNuggets Data Collections](http://www.kdnuggets.com/datasets/index.html) * [Microsoft Azure Data Market Free DataSets](https://azuremarketplace.microsoft.com/en-us/marketplace/apps?source=datamarket&filters=pricing-free&page=1) * [Microsoft Data Science for Research](http://aka.ms/Data-Science) * [Microsoft Research Open Data](https://msropendata.com/) * [Numbray](http://numbrary.com/) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//PublicDomains/Numbray.yml) ] * [Open Library Data Dumps](https://openlibrary.org/developers/dumps) * [Reddit Datasets](https://www.reddit.com/r/datasets) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//PublicDomains/Reddit-Datasets.yml) ] * [RevolutionAnalytics Collection](http://packages.revolutionanalytics.com/datasets/) * [Sample R data sets](http://stat.ethz.ch/R-manual/R-patched/library/datasets/html/00Index.html) * [StatSci.org](http://www.statsci.org/datasets.html) * [Stats4Stem R data sets (archived)](https://web.archive.org/web/20151024082129/http://www.stats4stem.org:80/data-sets.html) * [The Washington Post List](http://www.washingtonpost.com/wp-srv/metro/data/datapost.html) * [UCLA SOCR data collection](http://wiki.stat.ucla.edu/socr/index.php/SOCR_Data) * [UFO Reports](http://www.nuforc.org/webreports.html) * [Wikileaks 911 pager intercepts](https://911.wikileaks.org/files/index.html) * [Yahoo Webscope](http://webscope.sandbox.yahoo.com/catalog.php) SearchEngines ------------- * [Academic Torrents of data sharing from UMB](http://academictorrents.com/) * [DataMarket (Qlik)](https://datamarket.com/data/list/?q=all) * [Datahub.io](https://datahub.io/dataset) * [Harvard Dataverse Network of scientific data](https://dataverse.harvard.edu/) * [ICPSR (UMICH)](http://www.icpsr.umich.edu/icpsrweb/ICPSR/index.jsp) * [Institute of Education Sciences](http://eric.ed.gov) * [National Technical Reports Library](https://ntrl.ntis.gov/NTRL/) * [Open Data Certificates (beta)](https://certificates.theodi.org/en/datasets) * [OpenDataNetwork - A search engine of all Socrata powered data portals](http://www.opendatanetwork.com/) * [Statista.com - statistics and Studies](http://www.statista.com/) * [Zenodo - An open dependable home for the long-tail of science](https://zenodo.org/collection/datasets) SocialNetworks -------------- * [72 hours #gamergate Twitter Scrape](http://waxy.org/random/misc/gamergate_tweets.csv) * [Ancestry.com Forum Dataset over 10 years](http://www.cs.cmu.edu/~jelsas/data/ancestry.com/) * [CMU Enron Email of 150 users](http://www.cs.cmu.edu/~enron/) * [Cheng-Caverlee-Lee September 2009 - January 2010 Twitter Scrape](https://archive.org/details/twitter_cikm_2010) * [EDRM Enron EMail of 151 users, hosted on S3](https://aws.amazon.com/datasets/enron-email-data/) * [Facebook Data Scrape (2005)](https://archive.org/details/oxford-2005-facebook-matrix) * [Facebook Social Networks from LAW (since 2007)](http://law.di.unimi.it/datasets.php) * [Foursquare from UMN/Sarwat (2013)](https://archive.org/details/201309_foursquare_dataset_umn) * [GitHub Collaboration Archive](https://www.githubarchive.org/) * [Google Scholar citation relations](http://www3.cs.stonybrook.edu/~leman/data/gscholar.db) * [High-Resolution Contact Networks from Wearable Sensors](http://www.sociopatterns.org/datasets/) * [Indie Map: social graph and crawl of top IndieWeb sites](http://www.indiemap.org/) * [Mobile Social Networks from UMASS](https://kdl.cs.umass.edu/display/public/Mobile+Social+Networks) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//SocialNetworks/Mobile-Social-Networks-from-UMASS.yml) ] * [Network Twitter Data](http://snap.stanford.edu/data/higgs-twitter.html) * [Reddit Comments](http://files.pushshift.io/reddit/comments/) * [Skytrax' Air Travel Reviews Dataset](https://github.com/quankiquanki/skytrax-reviews-dataset) * [Social Twitter Data](http://snap.stanford.edu/data/egonets-Twitter.html) * [SourceForge.net Research Data](http://www3.nd.edu/~oss/Data/data.html) * [Twitter Data for Online Reputation Management](http://nlp.uned.es/replab2013/) * [Twitter Data for Sentiment Analysis](http://help.sentiment140.com/for-students/) * [Twitter Graph of entire Twitter site](http://an.kaist.ac.kr/traces/WWW2010.html) * [Twitter Scrape Calufa May 2011](http://archive.org/details/2011-05-calufa-twitter-sql) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//SocialNetworks/Twitter-Scrape-Calufa-May-2011.yml) ] * [UNIMI/LAW Social Network Datasets](http://law.di.unimi.it/datasets.php) * [Yahoo! Graph and Social Data](http://webscope.sandbox.yahoo.com/catalog.php?datatype=g) * [Youtube Video Social Graph in 2007,2008](http://netsg.cs.sfu.ca/youtubedata/) SocialSciences -------------- * [ACLED (Armed Conflict Location & Event Data Project)](http://www.acleddata.com/) * [Canadian Legal Information Institute](https://www.canlii.org/en/index.php) * [Center for Systemic Peace Datasets - Conflict Trends, Polities, State Fragility, etc](http://www.systemicpeace.org/) * [Correlates of War Project](http://www.correlatesofwar.org/) * [Cryptome Conspiracy Theory Items](http://cryptome.org) * [Datacards](https://www.datacards.org/login/) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//SocialSciences/Datacards.yml) ] * [European Social Survey](http://www.europeansocialsurvey.org/data/) * [FBI Hate Crime 2013 - aggregated data](https://github.com/emorisse/FBI-Hate-Crime-Statistics/tree/master/2013) * [Fragile States Index](http://fundforpeace.org/fsi/) * [GDELT Global Events Database](http://gdeltproject.org/data.html) * [General Social Survey (GSS) since 1972](http://gss.norc.org) * [German Social Survey](http://www.gesis.org/en/home/) * [Global Religious Futures Project](http://www.globalreligiousfutures.org/) * [Gun Violence Data - A comprehensive, accessible database that contains [...]](https://github.com/jamesqo/gun-violence-data) * [Humanitarian Data Exchange](https://data.humdata.org/) * [INFORM Index for Risk Management](http://www.inform-index.org/Results/Global) * [Institute for Demographic Studies](http://www.ined.fr/en/) * [International Networks Archive](http://www.princeton.edu/~ina/) * [International Social Survey Program ISSP](http://www.issp.org) * [International Studies Compendium Project](http://www.isacompendium.com/public/) * [James McGuire Cross National Data](http://jmcguire.faculty.wesleyan.edu/welcome/cross-national-data/) * [MIT Reality Mining Dataset](http://realitycommons.media.mit.edu/realitymining.html) * [MacroData Guide by Norsk samfunnsvitenskapelig datatjeneste](http://nsd.uib.no) * [Minnesota Population Center](https://www.ipums.org/) * [Notre Dame Global Adaptation Index (ND-GAIN)](https://gain.nd.edu/our-work/country-index/download-data/) * [Open Crime and Policing Data in England, Wales and Northern Ireland](https://data.police.uk/data/) * [OpenSanctions - A global database of persons and companies of political, [...]](http://www.opensanctions.org/#downloads) * [Paul Hensel General International Data Page](http://www.paulhensel.org/dataintl.html) * [PewResearch Internet Survey Project](http://www.pewinternet.org/?post_type=dataset) * [PewResearch Society Data Collection](http://www.pewresearch.org/data/download-datasets/) * [Political Polarity Data](http://www3.cs.stonybrook.edu/~leman/data/14-icwsm-political-polarity-data.zip) * [StackExchange Data Explorer](http://data.stackexchange.com/help) * [Terrorism Research and Analysis Consortium](http://www.trackingterrorism.org/) * [Texas Inmates Executed Since 1984](http://www.tdcj.state.tx.us/death_row/dr_executed_offenders.html) * [Titanic Survival Data Set](https://github.com/awesomedata/awesome-public-datasets/tree/master/Datasets) * [UCB's Archive of Social Science Data (D-Lab)](http://ucdata.berkeley.edu/) * [UCLA Social Sciences Data Archive](https://dataverse.harvard.edu/dataverse/ssda_ucla) * [UN Civil Society Database](http://esango.un.org/civilsociety/) * [UPJOHN for Labor Employment Research](http://www.upjohn.org/services/resources/employment-research-data-center) * [Universities Worldwide](http://univ.cc/) * [Uppsala Conflict Data Program](http://ucdp.uu.se/) * [World Bank Open Data](http://data.worldbank.org/) * [WorldPop project - Worldwide human population distributions](http://www.worldpop.org.uk/data/get_data/) TimeSeries ---------- * [Databanks International Cross National Time Series Data Archive](http://www.cntsdata.com) * [Hard Drive Failure Rates](https://www.backblaze.com/hard-drive-test-data.html) * [Heart Rate Time Series from MIT](http://ecg.mit.edu/time-series/) * [Time Series Data Library (TSDL) from MU](https://datamarket.com/data/list/?q=provider:tsdl) * [UC Riverside Time Series Dataset](http://www.cs.ucr.edu/~eamonn/time_series_data/) Transportation -------------- * [Airlines OD Data 1987-2008](http://stat-computing.org/dataexpo/2009/the-data.html) * [Ford GoBike Data (formerly Bay Area Bike Share Data)](https://www.fordgobike.com/system-data) * [Bike Share Systems (BSS) collection](https://github.com/BetaNYC/Bike-Share-Data-Best-Practices/wiki/Bike-Share-Data-Systems) * [GeoLife GPS Trajectory from Microsoft Research](http://research.microsoft.com/en-us/downloads/b16d359d-d164-469e-9fd4-daa38f2b2e13/) * [German train system by Deutsche Bahn](http://data.deutschebahn.com/datasets/) * [Hubway Million Rides in MA](http://hubwaydatachallenge.org/trip-history-data/) * [Montreal BIXI Bike Share](https://montreal.bixi.com/en/open-data) * [NYC Taxi Trip Data 2009-](http://www.nyc.gov/html/tlc/html/about/trip_record_data.shtml) * [NYC Taxi Trip Data 2013 (FOIA/FOILed)](https://archive.org/details/nycTaxiTripData2013) * [NYC Uber trip data April 2014 to September 2014](https://github.com/fivethirtyeight/uber-tlc-foil-response) * [Open Traffic collection](https://github.com/graphhopper/open-traffic-collection) * [OpenFlights - airport, airline and route data](http://openflights.org/data.html) * [Philadelphia Bike Share Stations (JSON)](https://www.rideindego.com/stations/json/) [[fixme](https://github.com/awesomedata/apd-core/tree/master/core//Transportation/Philadelphia-Bike-Share-Stations-JSON.yml) ] * [Plane Crash Database, since 1920](http://www.planecrashinfo.com/database.htm) * [RITA Airline On-Time Performance data](http://www.transtats.bts.gov/Tables.asp?DB_ID=120) * [RITA/BTS transport data collection (TranStat)](http://www.transtats.bts.gov/DataIndex.asp) * [Renfe (Spanish National Railway Network) dataset](data.renfe.com) * [Toronto Bike Share Stations (JSON and GBFS files)](https://www.toronto.ca/city-government/data-research-maps/open-data/open-data-catalogue/#84045f23-7465-0892-8889-7b6f91049b29) * [Transport for London (TFL)](https://tfl.gov.uk/info-for/open-data-users/our-open-data) * [Travel Tracker Survey (TTS) for Chicago](http://www.cmap.illinois.gov/data/transportation/travel-tracker-survey) * [U.S. Bureau of Transportation Statistics (BTS)](http://www.rita.dot.gov/bts/) * [U.S. Domestic Flights 1990 to 2009](http://academictorrents.com/details/a2ccf94bbb4af222bf8e69dad60a68a29f310d9a) * [U.S. Freight Analysis Framework since 2007](http://ops.fhwa.dot.gov/freight/freight_analysis/faf/index.htm) -------------------------------------- <style> p, li { font-family:system-ui,-apple-system,"Segoe UI",Roboto,Helvetica,Arial,sans-serif; font-size:calc(0.85em + 0.25vw); font-weight:300; line-height:1.7; -webkit-font-smoothing:antialiased; -moz-osx-font-smoothing:grayscale; margin-left:1%; margin-right:0%; } h2{ font-size:calc(2em + 0.25vw) !important; color: #337ab7; font-weight:300; margin-top:60px !important; margin-bottom:20px; } h3{ font-size:calc(1.4em + 0.25vw); font-weight:300; margin-top:20px !important; margin-bottom:10px;} ul { list-style-type:none; margin: 0; padding: 0; font-size:calc(0.75em + 0.25vw); line-height:1.2; } ul a { color: rgba(229,144,42,.7); font-size:calc(0.75em + 0.25vw); line-height:1.2; text-decoration: none; font-weight: normal; } ul a:hover { color: #337ab7; text-decoration: none; font-weight: normal; } #markdown-toc ul { font-size:calc(0.85em + 0.25vw); line-height:1.2; font-weight: bold; } #markdown-toc ul li { list-style-type: disc !important; font-size:calc(0.65em + 0.25vw); line-height:1.2; margin-left: 20px; } #markdown-toc a { color: black; font-size:calc(0.65em + 0.25vw); line-height:1.2; font-weight: normal; } #markdown-toc a:hover { color: black; text-decoration: none; font-weight: bold; } .collapsible { background-color: #fff; color: #444; cursor: pointer; padding: 18px; width: 20%; border: none; text-align: left; outline: none; font-size: 15px; } .active, .collapsible:hover { background-color: #ccc; } .active, .collapsible:hover { background-color: #ccc; } .content { display: none; overflow: hidden; } </style>

Related Documents

Testing

Multi-class: exactly one of the sentiment labels applies

HPC (High Performance Computing) bookmarks

Ruby 2.7