開放大數據集

開放大數據集

https://github.com/caesar0301/awesome-public-datasets

 

Awesome Public Datasets

 

This list of public data sources are collected and tidied from blogs, answers, and user responses. Most of the data sets listed below are free, however, some are not. Other amazingly awesome lists can be found in the awesome-awesomenessand sindresorhus's awesome list.

Table of Contents

• Agriculture
• Biology
• Climate/Weather
• Complex Networks
• Computer Networks
• Contextual Data
• Data Challenges
• Economics
• Education
• Energy
• Finance
• Geology
• GIS/Environment
• Government
• Healthcare
• Image Processing
• Machine Learning
• Museums
• Natural Language
• Physics
• Psychology/Cognition
• Public Domains
• Search Engines
• Social Networks
• Social Sciences
• Software
• Sports
• Time Series
• Transportation
• Complementary Collections

Agriculture

• U.S. Department of Agriculture's PLANTS Database

Biology

• 1000 Genomes
• American Gut (Microbiome Project)
• Broad Cancer Cell Line Encyclopedia (CCLE)
• Broad Bioimage Benchmark Collection (BBBC)
• Cell Image Library
• Collaborative Research in Computational Neuroscience (CRCNS)
• Complete Genomics Public Data
• EBI ArrayExpress
• EBI Protein Data Bank in Europe
• Electron Microscopy Pilot Image Archive (EMPIAR)
• ENCODE project
• Ensembl Genomes
• Gene Expression Omnibus (GEO)
• Gene Ontology (GO)
• Global Biotic Interactions (GloBI)
• Harvard Medical School (HMS) LINCS Project
• Human Genome Diversity Project
• Human Microbiome Project (HMP)
• ICOS PSP Benchmark
• International HapMap Project
• Journal of Cell Biology DataViewer
• MIT Cancer Genomics Data
• NCBI Proteins
• NCBI Taxonomy
• NeuroData
• NIH Microarray data or FTP
• OpenSNP genotypes data
• Pathguid - Protein-Protein Interactions Catalog
• Protein Data Bank
• Psychiatric Genomics Consortium
• PubChem Project
• PubGene (now Coremine Medical)
• Sanger Catalogue of Somatic Mutations in Cancer (COSMIC)
• Sanger Genomics of Drug Sensitivity in Cancer Project (GDSC)
• Sequence Read Archive(SRA)
• Stanford Microarray Data
• Stowers Institute Original Data Repository
• Systems Science of Biological Dynamics (SSBD) Database
• Temple University Hospital EEG Database
• The Cancer Genome Atlas (TCGA), available via Broad GDAC
• The Catalogue of Life
• The Personal Genome Project or PGP
• UCSC Public Data
• Universal Protein Resource (UnitProt)
• UniGene

Climate/Weather

• Australian Weather
• Brazilian Weather - Historical data (In Portuguese)
• Canadian Meteorological Centre
• Climate Data from UEA (updated monthly)
• European Climate Assessment & Dataset
• Global Climate Data Since 1929
• NASA Global Imagery Browse Services
• NOAA Bering Sea Climate
• NOAA Climate Datasets
• NOAA Realtime Weather Models
• The World Bank Open Data Resources for Climate Change
• UEA Climatic Research Unit
• WorldClim - Global Climate Data
• WU Historical Weather Worldwide

Complex Networks

• AMiner Citation Network Dataset
• CrossRef DOI URLs
• DBLP Citation dataset
• NBER Patent Citations
• Network Repository with Interactive Exploratory Analysis Tools
• NIST complex networks data collection
• Protein-protein interaction network
• PyPI and Maven Dependency Network
• Scopus Citation Database
• Small Network Data
• Stanford GraphBase (Steven Skiena)
• Stanford Large Network Dataset Collection
• Stanford Longitudinal Network Data Sources
• The Koblenz Network Collection
• The Laboratory for Web Algorithmics (UNIMI)
• The Nexus Network Repository
• UCI Network Data Repository
• UFL sparse matrix collection
• WSU Graph Database
• DIMACS Road Networks Collection

Computer Networks

• 3.5B Web Pages from CommonCraw 2012
• 53.5B Web clicks of 100K users in Indiana Univ.
• CAIDA Internet Datasets
• ClueWeb09 - 1B web pages
• ClueWeb12 - 733M web pages
• CommonCrawl Web Data over 7 years
• CRAWDAD Wireless datasets from Dartmouth Univ.
• Criteo click-through data
• Open Mobile Data by MobiPerf
• Rapid7 Sonar Internet Scans
• UCSD Network Telescope, IPv4 /8 net

Contextual Data

• Context-aware data sets from five domains or GitHub

Data Challenges

• Challenges in Machine Learning
• CrowdANALYTIX dataX
• D4D Challenge of Orange
• DrivenData Competitions for Social Good
• ICWSM Data Challenge (since 2009)
• Kaggle Competition Data
• KDD Cup by Tencent 2012
• Localytics Data Visualization Challenge
• Netflix Prize
• Space Apps Challenge
• Telecom Italia Big Data Challenge
• Yelp Dataset Challenge
• Bruteforce Database

Economics

• American Economic Ass (AEA)
• EconData from UMD
• Economic Freedom of the World Data
• Historical MacroEconomc Statistics
• International Trade Statistics
• Internet Product Code Database
• Joint External Debt Data Hub
• Jon Haveman International Trade Data Links
• OpenCorporates Database of Companies in the World
• Our World in Data
• SciencesPo World Trade Gravity Datasets
• The Atlas of Economic Complexity
• The Center for International Data
• The Observatory of Economic Complexity
• UN Commodity Trade Statistics
• UN Human Development Reports

Education

• Student Data from Free Code Camp

Energy

• AMPds
• BLUEd
• COMBED
• Dataport
• ECO
• EIA
• HFED
• iAWE
• Plaid
• REDD
• UK-Dale

Finance

• CBOE Futures Exchange
• Google Finance
• Google Trends
• NASDAQ
• OANDA
• OSU Financial data
• Quandl
• St Louis Federal
• Yahoo Finance
• NYSE Market Data

Geology

• Earth Models
• Smithsonian Institution Global Volcano and Eruption Database
• USGS Earthquake Archives

GIS/Environment

• BODC - marine data of ~22K vars
• Cambridge, MA, US, GIS data on GitHub
• EOSDIS - NASA's earth observing system data
• Factual Global Location Data
• Geo Spatial Data from ASU
• Geo Wiki Project - Citizen-driven Environmental Monitoring
• GeoFabrik - OSM data extracted to a variety of formats and areas
• GeoNames Worldwide
• Global Administrative Areas Database (GADM)
• Homeland Infrastructure Foundation-Level Data
• Integrated Marine Observing System (IMOS) - roughly 30TB of ocean measurements or on S3
• International Institute for Systems Analysis - GIS Datasets
• Landsat 8 on AWS
• List of all countries in all languages
• Marinexplore - Open Oceanographic Data
• National Weather Service GIS Data Portal
• Natural Earth - vectors and rasters of the world
• OpenAddresses
• OpenStreetMap (OSM)
• Pleiades - Gazetteer and graph of ancient places
• Reverse Geocoder using OSM data & additional high-resolution data files
• TIGER/Line - U.S. boundaries and roads
• TwoFishes - Foursquare's coarse geocoder
• TZ Timezones shapfiles
• UN Environmental Data
• World boundaries from the U.S. Department of State
• World countries in multiple formats

Government

• OpenDataSoft's list of 1,600 open data portals
• A list of cities and countries contributed by community

Healthcare

• EHDP Large Health Data Sets
• Gapminder World demographic databases
• Medicare Coverage Database (MCD), U.S.
• Medicare Data Engine of medicare.gov Data
• Medicare Data File
• MeSH, the vocabulary thesaurus used for indexing articles for PubMed
• Number of Ebola Cases and Deaths in Affected Countries (2014)
• Open-ODS (structure of the UK NHS)
• OpenPaymentsData, Healthcare financial relationship data
• The Cancer Genome Atlas project (TCGA) and BigQuery table
• World Health Organization Global Health Observatory

Image Processing

• 10k US Adult Faces Database
• 2GB of Photos of Cats or Archive version
• Affective Image Classification
• Animals with attributes
• Face Recognition Benchmark
• ImageNet (in WordNet hierarchy)
• Indoor Scene Recognition
• International Affective Picture System, UFL
• Massive Visual Memory Stimuli, MIT
• Several Shape-from-Silhouette Datasets
• Stanford Dogs Dataset
• SUN database, MIT
• The Oxford-IIIT Pet Dataset
• YouTube Faces Database
• Adience Unfiltered faces for gender and age classification
• The Action Similarity Labeling (ASLAN) Challenge
• Violent-Flows - Crowd Violence Non-violence Database and benchmark

Machine Learning

• Delve Datasets for classification and regression (Univ. of Toronto)
• Discogs Monthly Data
• eBay Online Auctions (2012)
• IMDb Database
• Keel Repository for classification, regression and time series
• Labeled Faces in the Wild (LFW)
• Lending Club Loan Data
• Machine Learning Data Set Repository
• Million Song Dataset
• More Song Datasets
• MovieLens Data Sets
• RDataMining - "R and Data Mining" ebook data
• Registered Meteorites on Earth
• Restaurants Health Score Data in San Francisco
• UCI Machine Learning Repository
• Yahoo! Ratings and Classification Data

Museums

• Canada Science and Technology Museums Corporation's Open Data
• Cooper-Hewitt's Collection Database
• Minneapolis Institute of Arts metadata
• Natural History Museum (London) Data Portal
• Rijksmuseum Historical Art Collection
• Tate Collection metadata
• The Getty vocabularies

Natural Language

• Blogger Corpus
• CLiPS Stylometry Investigation Corpus
• ClueWeb09 FACC
• ClueWeb12 FACC
• DBpedia - 4.58M things with 583M facts
• Flickr Personal Taxonomies
• Freebase.com of people, places, and things
• Google Books Ngrams (2.2TB)
• Google Web 5gram (1TB, 2006)
• Gutenberg eBooks List
• Hansards text chunks of Canadian Parliament
• Machine Comprehension Test (MCTest) of text from Microsoft Research
• Machine Translation of European languages
• Personae Corpus
• SaudiNewsNet Collection of Saudi Newspaper Articles (Arabic, 30K articles)
• SMS Spam Collection in English
• USENET postings corpus of 2005~2011
• Wikidata - Wikipedia databases
• Wikipedia Links data - 40 Million Entities in Context
• WordNet databases and tools

Physics

• CERN Open Data Portal
• Crystallography Open Database
• NASA Exoplanet Archive
• NSSDC (NASA) data of 550 space spacecraft
• Sloan Digital Sky Survey (SDSS) - Mapping the Universe

Psychology/Cognition

• OSU Cognitive Modeling Repository Datasets

Public Domains

• Amazon
• Archive-it from Internet Archive
• Archive.org Datasets
• CMU JASA data archive
• CMU StatLab collections
• Data360
• Datamob.org
• Google
• Infochimps
• KDNuggets Data Collections
• Microsoft Azure Data Market Free DataSets
• Numbray
• Open Library Data Dumps
• Reddit Datasets
• RevolutionAnalytics Collection
• Sample R data sets
• Stats4Stem R data sets
• StatSci.org
• The Washington Post List
• UCLA SOCR data collection
• UFO Reports
• Wikileaks 911 pager intercepts
• Yahoo Webscope

Search Engines

• Academic Torrents of data sharing from UMB
• Datahub.io
• DataMarket (Qlik)
• Harvard Dataverse Network of scientific data
• ICPSR (UMICH)
• Institute of Education Sciences
• National Technical Reports Library
• Open Data Certificates (beta)
• OpenDataNetwork - A search engine of all Socrata powered data portals
• Statista.com - statistics and Studies
• Zenodo - An open dependable home for the long-tail of science

Social Networks

• 72 hours #gamergate Twitter Scrape
• Ancestry.com Forum Dataset over 10 years
• Cheng-Caverlee-Lee September 2009 - January 2010 Twitter Scrape
• CMU Enron Email of 150 users
• EDRM Enron EMail of 151 users, hosted on S3
• Facebook Data Scrape (2005)
• Facebook Social Networks from LAW (since 2007)
• Foursquare from UMN/Sarwat (2013)
• GetGlue - users rating TV shows
• GitHub Collaboration Archive
• Google Scholar citation relations
• High-Resolution Contact Networks from Wearable Sensors
• Mobile Social Networks from UMASS
• Network Twitter Data
• Reddit Comments
• Skytrax' Air Travel Reviews Dataset
• Social Twitter Data
• SourceForge.net Research Data
• Twitter Data for Sentiment Analysis
• Twitter Data for Online Reputation Management
• Twitter Graph of entire Twitter site
• Twitter Scrape Calufa May 2011
• UNIMI/LAW Social Network Datasets
• Yahoo! Graph and Social Data
• Youtube Video Social Graph in 2007,2008

Social Sciences

• ACLED (Armed Conflict Location & Event Data Project)
• Canadian Legal Information Institute
• Center for Systemic Peace Datasets - Conflict Trends, Polities, State Fragility, etc
• Correlates of War Project
• Cryptome Conspiracy Theory Items
• Datacards
• European Social Survey
• FBI Hate Crime 2013 - aggregated data
• GDELT Global Events Database
• General Social Survey (GSS) since 1972
• German Social Survey
• Global Religious Futures Project
• Humanitarian Data Exchange
• Institute for Demographic Studies
• International Networks Archive
• International Social Survey Program ISSP
• International Studies Compendium Project
• James McGuire Cross National Data
• MacroData Guide by Norsk samfunnsvitenskapelig datatjeneste
• MIT Reality Mining Dataset
• Open Crime and Policing Data in England, Wales and Northern Ireland
• Paul Hensel General International Data Page
• PewResearch Internet Survey Project
• PewResearch Society Data Collection
• Political Polarity Data
• StackExchange Data Explorer
• Terrorism Research and Analysis Consortium
• Texas Inmates Executed Since 1984
• Titanic Survival Data Set
• UCB's Archive of Social Science Data (D-Lab)
• UCLA Social Sciences Data Archive
• UN Civil Society Database
• Universities Worldwide
• UPJOHN for Labor Employment Research
• World Bank Data
• WorldPop project - Worldwide human population distributions

Software

• FLOSSmole data about free, libre, and open source software development

Sports

• Basketball (NBA/NCAA/Euro) Player Database and Statistics
• Betfair Historical Exchange Data
• Cricsheet Matches (cricket)
• Ergast Formula 1, from 1950 up to date (API)
• Football/Soccer resources (data and APIs)
• Lahman's Baseball Database
• Pinhooker: Thoroughbred Bloodstock Sale Data
• Retrosheet Baseball Statistics

Time Series

• Databanks International Cross National Time Series Data Archive
• Hard Drive Failure Rates
• Heart Rate Time Series from MIT
• Time Series Data Library (TSDL) from MU
• UC Riverside Time Series Dataset

Transportation

• Airlines OD Data 1987-2008
• Bay Area Bike Share Data
• Bike Share Systems (BSS) collection
• GeoLife GPS Trajectory from Microsoft Research
• German train system by Deutsche Bahn
• Hubway Million Rides in MA
• Marine Traffic - ship tracks, port calls and more
• Montreal BIXI Bike Share
• NYC Taxi Trip Data 2009-
• NYC Taxi Trip Data 2013 (FOIA/FOILed)
• NYC Uber trip data April 2014 to September 2014
• Open Traffic collection
• OpenFlights - airport, airline and route data
• Philadelphia Bike Share Stations (JSON)
• Plane Crash Database, since 1920
• RITA Airline On-Time Performance data
• RITA/BTS transport data collection (TranStat)
• Toronto Bike Share Stations (XML file)
• Transport for London (TFL)
• Travel Tracker Survey (TTS) for Chicago
• U.S. Bureau of Transportation Statistics (BTS)
• U.S. Domestic Flights 1990 to 2009
• U.S. Freight Analysis Framework since 2007

Complementary Collections

• Data Packaged Core Datasets
• Database of Scientific Code Contributions
• DataWrangling: Some Datasets Available on the Web
• Inside-r: Finding Data on the Internet
• OpenDataMonitor: An overview of available open data resources in Europe
• Quora: Where can I find large datasets open to the public?
• RS.io: 100+ Interesting Data Sets for Statistics
• StaTrek: Leveraging open data to understand urban lives