Working with Open Data

Untitled document

Course Outline

Last revised: 2013.05.27

Old plans:

Review of Python & iPython (Chap 3 and Chap 13) + setup EPD
NumPy and Getting started with pandas I (Chap 4, 5)
Open Government Data I: US Census data: introduction how to work with
Plotting and Visualization (Chap 8)
Data Loading, Storage, and File Formats (Chap 6)
Wikipedia: API, data structure
NumPy and Getting started with pandas II (Chap 4, 5)
Freebase, dbpedia, wikidata
Data Wrangling: Clean, Transform, Merge, Reshape (Chap 7)
working with geodata I
JavaScript-based visualization I
Publishing open data including LOD
Google Refine
working with geodata II
JavaScript-based visualization II
MIDTERM (Day 17, 2013-03-19)
????
Wikipedia II
Data Aggregation and Group Operations (Ch 9)
Time Series (Ch 10)
Financial and Economic Data Applications (Ch 11)
????
Open Government Data II
Advanced NumPy (Ch 12)
PyData Misc I / Working on Projects
PyData Misc II / Working on Projects
Project Poster Session

To be scheduled:

open scientific data (w/ guest lectures)
using licensed UC Data Lab resources
1 to 3 more outside speakers

Can we fit other topics?

open music data, open bibliographic data, etc (- open bibliographic data (e.g, the Harvard dataset,OCLC, HathiTrust))

slide 8/25

* help? contents?

Similar

Search Help

Non Advanced Search or Natural Search

The documents are returned sorted on relevance depending on order, proximity, frequency of terms.

Advanced Search or Boolean Search

Default search behavior

By default, all search terms are optional. It behaves like an OR logic. Objects that contain the more terms are rated higher in the results and will appear first in their type. For example, wiki forum will find:

objects that include both terms
objects that include the term wiki
objects that include the term forum

Requiring terms

Add a plus sign ( + ) before a term to indicate that the term must appear in results. Example: +wiki forum will find objects containing at least wiki. Objects with both terms and many occurences of the terms will appear first.

Excluding terms

Add a minus sign ( - ) before a term to indicate that the term must not appear in the results. To reduce a term's value without completely excluding it, use a tilde. Example: -wiki forum will find objects that do not contain wiki but contain forum

Grouping terms

Use parenthesis ( ) to group terms into subexpressions. Example: +wiki +(forum blog) will find objects that contain wiki and forum or that contain wiki and blog in any order.

Finding phrases

Use double quotes ( " " ) around a phrase to find terms in the exact order, exactly as typed. Example: "Alex Bell" will not find Bell Alex or Alex G. Bell.

Using wildcards

Add an asterisk ( * ) after a term to find objects that include the root word. For example, run* will find:

objects that include the term run
objects that include the term runner
objects that include the term running

Reducing a term's value

Add a tilde ( ~ ) before a term to reduce its value indicate to the ranking of the results. Objects that contain the term will appear lower than other objects (unlike the minus sign which will completely exclude a term). Example: +wiki ~forum will rate an object with only wiki higher that an object with wiki and forum.

Changing relevance value

Add a less than ( < ) or greater than ( > ) sign before a term to change the term's contribution to the overall relevance value assigned to a object. Example: +wiki +(>forum < blog) will find objects that contain wiki and forum or wiki and blog in any order. wiki forum will be rated higher.