CRAIGLIST CAR FOR SALE DATA EXPLORATION¶
The data set contains about vehicles (cars, trucks, vans, ...) for sale on a Web site. Individuals (owners) or dealers create a post for a vehicle, and provide information about it such as the make, model and year of the vehicle, put it in a broad category for condition (e.g., excellent, good, ...), how many miles it has on the odometer, the type of vehicle (e.g. hatchback, sudan), what type of fuel it uses (e.g., gas or diesel). We also have information about the location of the poster/vehicle - the city, and often the latitude and longitude. Importantly, we have the price being sought by the seller. We also have a short free-form description of the vehicle.
I. SETUP:¶
# Load the data file called vehicles.rda from the subdirectory called data.
load("./data/vehicles.rda")
library(lattice)
library(maps)
library(ggplot2)
library(gmodels)
library(RColorBrewer)
# Create ownerDealer variable
vposts$ownerDealer = factor(vposts$byOwner, levels = c(TRUE, FALSE), labels = c("Owner", "Dealer"))
II. DATA EXPLORATION:¶
How many observations are there in the data set?¶
# We can get the number of observations by counting the number of rows
nrow(vposts)
## [1] 34677
# Alternatively, we could get the dimensions of the dataset and look at the number of rows (observations)
dim(vposts)
## [1] 34677 27
What are the names of the variables? and what is the class of each variable?¶
# Names of the variables
names(vposts)
## [1] "id" "title" "body" "lat"
## [5] "long" "posted" "updated" "drive"
## [9] "odometer" "type" "header" "condition"
## [13] "cylinders" "fuel" "size" "transmission"
## [17] "byOwner" "city" "time" "description"
## [21] "location" "url" "price" "year"
## [25] "maker" "makerMethod" "ownerDealer"
# Get the class of each variable using sapply() and then unlist() the results
unlist( sapply(X = vposts, FUN = class) )
## id title body lat long
## "character" "character" "character" "numeric" "numeric"
## posted1 posted2 updated1 updated2 drive
## "POSIXct" "POSIXt" "POSIXct" "POSIXt" "factor"
## odometer type header condition cylinders
## "integer" "factor" "character" "factor" "integer"
## fuel size transmission byOwner city
## "factor" "factor" "factor" "logical" "factor"
## time1 time2 description location url
## "POSIXct" "POSIXt" "character" "character" "character"
## price year maker makerMethod ownerDealer
## "integer" "integer" "character" "numeric" "factor"
# Alternative solution that is cleaner as suggested by Ian Heath on Piazza
print.table( sapply(X = vposts, FUN = class) )
## id title body lat
## character character character numeric
## long posted updated drive
## numeric POSIXct, POSIXt POSIXct, POSIXt factor
## odometer type header condition
## integer factor character factor
## cylinders fuel size transmission
## integer factor factor factor
## byOwner city time description
## logical factor POSIXct, POSIXt character
## location url price year
## character character integer integer
## maker makerMethod ownerDealer
## character numeric factor
What is the average price of all the vehicles? the median price? and the deciles? Displays these on a plot of the distribution of vehicle prices.¶
# The densityplot (with rug plot) indicates that there are a few extreme outliers in the right tail.
densityplot(vposts$price, main = "Price", xlab = "Price")
# Let's see the 50 largest prices
tail( sort(vposts$price), 50 )
## [1] 95593 96590 97000 97500 97911 98000 99560
## [8] 99999 100000 100000 100000 100000 104800 105000
## [15] 105500 107000 112000 116100 116491 120000 122950
## [22] 123981 125000 129950 129990 138500 139000 139950
## [29] 143000 143000 143950 147000 149890 149995 150000
## [36] 152900 159000 169000 177588 202455 240000 286763
## [43] 359000 400000 559500 569500 9999999 30002500 600030000
## [50] 600030000
# For sure, the prices 9999999 30002500 600030000 600030000 are highly suspicious.
# Let's look at any car more than $100,000.
idx = which( vposts$price >= 100000 & !is.na(vposts$price) )
# There are 42 cars > $100,000
length( idx )
## [1] 42
# Let's order the row indices by price low-to-high
idx = idx[order(vposts[ idx, "price"])]
# Let's look at some of the actual data for those cars > $100,000
vposts[ idx, c("header", "price") ]
## header price
## posted12461 2000 Mack RD688S 100000
## posted24081 2003 lincoln navigator 100000
## posted181511 1965 porsche 911 100000
## posted245512 1961 Maserati 151 100000
## posted212510 2014 Porsche 911 104800
## posted13245 2010 ford fusion 105000
## posted7225 1967 chevrolet corvette 105500
## posted20867 2015 mercedes-benz s550 107000
## posted194311 2005 TOYOTA AVALON 112000
## posted222912 2015 Porsche Panamera 116100
## posted37310 2014 Audi RS 7 4.0T quattro 116491
## posted95215 1988 porsche 911 Carrera Targa TL 120000
## posted18546 2013 Mercedes-Benz G63 122950
## posted106214 2014 Land Rover Range 5.0L V8 123981
## posted212613 1941 willys 125000
## posted11066 2009 Lamborghini Gallardo 129950
## posted16934 2013 Isuzu NRR 129990
## posted191712 2015 Hyundai Sonata 138500
## posted112215 1976 Porsche 930 139000
## posted18506 2009 Mercedes-Benz SL65 139950
## posted163710 2015 Mercedes-Benz S-Class 143000
## posted231215 2015 Mercedes-Benz S-Class 143000
## posted214110 2011 Bentley Mulsanne 143950
## posted238613 2015 Porsche GT3 147000
## posted5038 2012 Mercedes-Benz SLS AMG 2dr Roadster SLS AMG 149890
## posted129214 2007 Lamborghini Gallardo Spyder 149995
## posted22621 2013 Ford 150000
## posted231114 2015 Porsche 911 152900
## posted220311 2011 toyota rav4 159000
## posted6747 2004 Lexus 470 169000
## posted698 2008 BMW X5 177588
## posted121313 2016 porsche 911 202455
## posted12630 2014 ferrari 458 italia 240000
## posted9976 2004 Toyota Corolla 286763
## posted1460 2010 CHEVROLET SILVERADO 359000
## posted23788 2006 FORD GT 400000
## posted21402 2009 CHEVROLET IMPALA 559500
## posted21422 2007 CHEVROLET MONTE CARLO 569500
## posted16005 2001 Honda Accord 9999999
## posted6903 2002 Caddy Seville sls 30002500
## posted22491 1969 Pontiac GTO 600030000
## posted23881 1969 Pontiac GTO 600030000There are some very expensive cars in there like Mercedes-Benz G63 AMG, Bentley Mulsanne, Maserati 3500 GT, Porsche GT, etc. which are probably legitimately more than \$100,000. However, there are also cars in there such as “2015 Hyundai Sonata Call/SMS 650.445.0890 (12) - \$138500” which probably has an extra zero at the end and even a “2007 CHEVROLET MONTE CARLO LT - Easy Financing! Any Credit Auto Loans! BHPH! - \$569500” which probably has two extra zeros at the end. We will ignore them for now, but acknowledge that these are also problems and that they should be cleaned up.
# Let's look at the worst price errors
vposts[ vposts$price >= 9999999 & !is.na(vposts$price), c("header", "body")]
## header
## posted22491 1969 Pontiac GTO
## posted23881 1969 Pontiac GTO
## posted6903 2002 Caddy Seville sls
## posted16005 2001 Honda Accord
## body
## posted22491 \n We have 1968 & 1969 Pontiac GTO's.\nCurrently we are working on a 1968 end a 1969 Gto project is almost complete.\nOur Intention is the custom to specification by owner.\nCost will be between $6000 & $30,000. This will be depending on the car in the condition and the Owner financial capabilities. \nSerious inquires only inquiries only.. please call Tony at \n show contact info\n\n
## posted23881 \n We have 1968 & 1969 Pontiac GTO's.\nCurrently we are working on a 1968 end a 1969 Gto project is almost complete.\nOur Intention is the custom to specification by owner.\nCost will be between $6000 & $30,000. This will be depending on the car in the condition and the Owner financial capabilities. \nSerious inquires only inquiries only.. please call Tony at \n show contact info\n\n
## posted6903 \n clean, fully loaded, nice shine, good running engine and trans, willing to trade for old school or truck?????????????????? Mounted on 22 inch rims new tires no bends no cracks\n
## posted16005 \n Selling my car for some lunch money. $20 OBO. Comes with complimentary Oboe.\nThe two records with price of 600030000 are 1968 and 1969 Pontiac GTO - \$600030000, which can see from reading the body of the post that these are offers to customize GTO’s for somewhere between \$6,000 and \$30,000 and these values seem to be pasted together. We can revisit this in question #8, but for now, let’s exclude these.
Best guess for the Seville SLS with price 30002500 looks like a typo by the person who posted the ad. Perhaps they were originally thinking that they would sell it for \$3,000, but then changed their mind and lowered the price to \$2,500 but did not properly clear the field for price before they editing the posting price. We can revisit this in question #8, but for now, let’s exclude this.
The 2001 Honda Accord with price 9999999 looks like a deliberate misleading price by the person who posted the ad given that they failed to fill in several other fields.
As a side note, numbers like 99, or 999 or 99999999, etc. are often default entries for missing values, much like NA, #N/A, , Null, NULL, etc. Depending on the data, even numbers like -1 could be defined as missing data. We can revisit this in question #8, but for now, let’s exclude this.
For Question #3 here, all cars with a price greater than \$240,000 seem to have data errors so we will exclude all of them here.
We also note several other problems like lots of missing prices in general. These could be corrected one-by-one perhaps.
There are also lots of cars for sale at the price of \$1. This is a common advertising tactic to post for the minimum price since most people sort prices lowest-to-highest and thus these ads get seen at the top more often. Most of these are misleading ads by dealers, some are for car parts, some are offers for car financing. The same holds for almost all ads less than \$500. There is just too much data to clean up manually here so we will exclude them. SURELY we are throwing away some good data here and biasing our results.
# How many prices are NA?
sum( is.na(vposts$price) )
## [1] 3328
# How many cars for sale at $1?
sum( vposts$price == 1 & !is.na(vposts$price) )
## [1] 612
# Let's take a look 10% randomly selected $1 cars
idx = which( vposts$price == 1 & !is.na(vposts$price) )
idx = sample(x = idx, size = 60, replace = FALSE)
# Omitted here for brevity
# vposts[ idx, "body"]
# Only include prices between $500 and $240,000
idx = (vposts$price >= 500 & vposts$price <= 240000 & !is.na(vposts$price))
densityplot(vposts$price[ idx ], xlab = "Price", main = "Price")
# 94% of the data is between about $500 and $50,000
quantile(x = vposts$price, probs = c(0.05,0.99), na.rm = TRUE)
## 5% 99%
## 499 47000
# Final plot
avg = mean( vposts$price[ idx ] )
med = median( vposts$price[ idx ] )
dec = quantile(x = vposts$price[ idx ], probs = seq(from = 0.1, to = 0.9, by = 0.1) )
idx = (vposts$price >= 500 & vposts$price <= 50000 & !is.na(vposts$price))
plot(density(vposts$price[ idx ]), xlab = "Price", main = "Price", lwd = 2)
abline(h = 0)
points(x = c(avg, med), y = c(0,0), col = c("forestgreen", "darkorange"), pch = 18, cex = 3.0)
points(x = dec, y = rep(x = 0, times = length(dec)), col = "blue2", pch = 18, cex = 1.5)
legend("topright", legend = c("Mean", "Median", "Deciles"), col = c("forestgreen", "darkorange", "blue2"),
pch = 18)
What are the different categories of vehicles, i.e. the type variable/column? What is the proportion for each category ?¶
# Different types. Note the NA's
names( table(vposts$type, useNA = "ifany") )
## [1] "bus" "convertible" "coupe" "hatchback" "mini-van"
## [6] "offroad" "other" "pickup" "sedan" "SUV"
## [11] "truck" "van" "wagon" NA
# Proportions table, sorted
sort( round( x = prop.table( x = table(vposts$type, useNA = "ifany") ), digits = 4) )
##
## bus offroad mini-van van wagon other
## 0.0006 0.0019 0.0131 0.0146 0.0161 0.0192
## convertible hatchback pickup truck coupe SUV
## 0.0204 0.0236 0.0262 0.0347 0.0469 0.1214
## sedan <NA>
## 0.2030 0.4583
# Let's plot the proportions
# Base dotchart() will not show the label NA and Lattice dotplot() will not plot the NA at all so force it
t = prop.table( x = table(vposts$type, useNA = "ifany") )
names(t)[ is.na(names(t)) ] = "NA"
# dotchart(x = sort(t), xlim = c(0,1), main = "Proportions of Car Types")
dotplot(x = sort(t), xlim = c(-0.05, 1.05), cex = 1.5, main = "Proportions of Car Types")
# Let's look at proportions *among* the non-missing
t = prop.table( x = table(vposts$type[ !is.na(vposts$type) ], useNA = "ifany") )
dotplot(x = sort(t), xlim = c(-0.05, 1.05), cex = 1.5, main = "Proportions of Car Types Among Non-Missing")
Close to half of the data is missing the vehicle type. Among those not missing the vehicle type, about 40% are sedans.
Display the relationship between fuel type and vehicle type. Does this depend on transmission type?¶
What we can see from the mosaic plots below overall and by transmission type and ignoring vehicle type other and fuel type other, is that gas vehicles dominate across vehicle types and across transmission types, with the notable exception that trucks have higher percent diesel than other vehicle types, as do buses with automatic transmissions - this is expected.
It might be easier to see these same relationships in dotplots than mosaicplots.
# Overall relationship fuel type and vehicle type
tbl = table(vposts$fuel, vposts$type)
row.order = order( rowSums( tbl ), decreasing = TRUE )
col.order = order( colSums( tbl ), decreasing = TRUE )
tbl = tbl[ row.order, col.order ]
col.palette = colorRampPalette(brewer.pal(9,"Blues"))(length(col.order))
mosaicplot(tbl, las = 2, color = col.palette,
main = "Overall Fuel Type by Vehicle Type", cex = 0.7
)
# Split by transmission type
fuelVehicleBytransmission = split( vposts[ , c("fuel", "type")], vposts$transmission)
invisible(
lapply( 1:length(fuelVehicleBytransmission),
FUN = function(x){
tbl = table(fuelVehicleBytransmission[[x]]$fuel, fuelVehicleBytransmission[[x]]$type)
row.order = order( rowSums( tbl ), decreasing = TRUE )
col.order = order( colSums( tbl ), decreasing = TRUE )
tbl = tbl[ row.order, col.order ]
# Get color palette and then reverse the order darkest to lightest
col.palette = colorRampPalette(brewer.pal(9,"Blues"))(length(col.order))
col.palette = col.palette[ length(col.palette):1 ]
mosaicplot(tbl, las = 2, color = col.palette,
main = paste0("Fuel and Vehicle by Transmission = ",
names(fuelVehicleBytransmission)[x]), cex = 0.7
)
}
)
)
dotplot(
prop.table( table(vposts$type , vposts$transmission, vposts$fuel, useNA = "ifany"),
margin = c(1,2) ),
xlim = c(-0.05,1.05), auto.key = list(columns = 3), par.settings = simpleTheme(cex=1.2, pch=16),
xlab = "Percent"
)
How many different cities are represented in the dataset?¶
# levels() gives us the cities
levels(vposts$city)
## [1] "boston" "chicago" "denver" "lasvegas" "nyc" "sac"
## [7] "sfbay"
# length() tells us how many there are
length( levels(vposts$city) )
## [1] 7
Visually display how the number/proportion of “for sale by owner” and “for sale by dealer” varies across city?¶
It is a bit suspicious that all cities have roughly 5000 observations and that the percentages are almost perfectly 50/50 within each city. Smells like the data was filtered first…
Note that we also created a new variable called ownerDealer in the setup section.
# Each city has ~5000 observations
table(vposts$city)
##
## boston chicago denver lasvegas nyc sac sfbay
## 4958 4886 4979 4963 4983 4966 4942
# prop.table() with margin = 2 gives the column percentages
prop.table(table(vposts$ownerDealer, vposts$city), margin = 2)
##
## boston chicago denver lasvegas nyc sac
## Owner 0.4975797 0.4901760 0.4994979 0.4984888 0.4992976 0.5000000
## Dealer 0.5024203 0.5098240 0.5005021 0.5015112 0.5007024 0.5000000
##
## sfbay
## Owner 0.4991906
## Dealer 0.5008094
# Plot the counts from table() (Note: almost identical graphs as percentages)
plot( table(vposts$city, vposts$ownerDealer, useNA = "ifany"), main = "Owner/Dealer by City",
xlab = "City", ylab = "Owner or Dealer", col=c("darkblue","darkviolet"))
# Plot the percentages from prop.table (Note the switch of the variables)
plot( prop.table(table(vposts$city, vposts$ownerDealer), margin = 2), main = "Owner/Dealer by City",
xlab = "City", ylab = "Owner or Dealer", col=c("darkblue","darkviolet"))
# Use a dotplot since there are only two categories
tbl = prop.table( table(vposts$ownerDealer, vposts$city, useNA = "ifany"), margin = 2)
dotplot( sort(tbl[ 1, ]), xlim = c(-0.05, 1.05), xlab = "% Owner for Sale by Owner", ylab = "City",
main = "Percent for Sale by Owner by City", cex = 1.5)
The barplots basically capture the key information, but notice that the bar widths do not really represent anything meaningful. Since we are interested in the percentage for sale by owner within each city, a dotplot probably captures this the best since there are only two categories (byOwner = TRUE = owner, and byOwner = FALSE = dealer) and not missing data.
We can see from the tables and very clearly from the plots that the percentages of owner posting and dealer postings are almost perfectly 50/50 and this does not seem to vary across the different cities at all.
What is the largest price for a vehicle in this data set? Examine this and fix the value. Now examine the new highest value for price.¶
# Being pedantic, let's get the max price
max(vposts$price, na.rm = TRUE)
## [1] 600030000
# Let's remind ourselves of the few worst prices
idx = which( vposts$price >= 9999999 & !is.na(vposts$price) )
# Let's look at the actual data for those cars > $100,000
vposts[ idx, c("header", "price", "maker", "year") ]
## header price maker year
## posted22491 1969 Pontiac GTO 600030000 pontiac 1969
## posted23881 1969 Pontiac GTO 600030000 pontiac 1969
## posted6903 2002 Caddy Seville sls 30002500 cadillac 2002
## posted16005 2001 Honda Accord 9999999 honda 2001
# Let's fix the two records for the 1969 Pontiac GTO with price = 600030000
# We could remove it completely - maybe not the best approach.
# We could average the prices from the description, $6000 and $30,000 = $18,000 - not bad approach.
# We could research this car online and try to find some average price - pretty good approach.
# Let's see if we can't find a decent point estimate from within the dataset itself:
idx = ( vposts$maker == "pontiac" & vposts$year %in% c(1968, 1969) &
vposts$price < 9999999 & vposts$price > 1 &
grepl(pattern = "GTO", x = vposts$header, ignore.case = TRUE) &
!is.na(vposts$maker) & !is.na(vposts$price) & !is.na(vposts$header) )
dat = vposts[ idx, c("header", "price", "maker", "year") ]
dat[ order(dat$price), ]
## header price maker year
## posted4991 1968 pontiac gto 15995 pontiac 1968
## posted5371 1968 pontiac gto 15995 pontiac 1968
## posted231214 1968 Pontiac GTO 24500 pontiac 1968
## posted16497 1969 Pontiac GTO 25000 pontiac 1969
## posted201111 1969 Pontiac GTO 25000 pontiac 1969
## posted7355 1968 pontiac gto 30000 pontiac 1968
## posted16701 1968 Pontiac gto 38500 pontiac 1968
## posted40911 1968 GTO 38500 pontiac 1968
# Let's use a rounded mean price
newPrice = round( mean(vposts$price[idx]), digits = -3)
vposts$price[vposts$price == 600030000 & !is.na(vposts$price)] = newPrice
# Now let's fix the 2002 Caddy Seville sls with price = 30002500
# Let's see if we can't find a decent point estimate from within the dataset itself:
idx = ( vposts$maker == "cadillac" & vposts$year %in% c(2002) &
vposts$price < 9999999 & vposts$price > 1 &
grepl(pattern = "Seville", x = vposts$header, ignore.case = TRUE) &
!is.na(vposts$maker) & !is.na(vposts$price) & !is.na(vposts$header) )
dat = vposts[ idx, c("header", "price", "maker", "year") ]
dat[ order(dat$price), ]
## header price maker year
## posted20963 2002 cadillac seville 700 cadillac 2002
## posted20691 2002 Cadillac Seville 1500 cadillac 2002
## posted9323 2002 cadillac seville 2100 cadillac 2002
## posted16927 2002 cadillac seville 3495 cadillac 2002
# Average price estimate is lower than the posting price of $2500 or $3000, so use the lower $2500
round( mean(vposts$price[idx]), digits = -3)
## [1] 2000
vposts$price[vposts$price == 30002500 & !is.na(vposts$price)] = 2500
# The 2001 Honda Accord with price = 9999999 is a junk post
# "Selling my car for some lunch money. $20 OBO. Comes with complimentary Oboe."
# Remove it completely.
idx = which(vposts$price == 9999999 & !is.na(vposts$price))
vposts = vposts[ -idx, ]
Again, SURELY there are more that need to be fixed, but that was enough pain for one day.
What are the three most common makes of cars in each city for “sale by owner” and for “sale by dealer”? Are they similar or quite different?¶
cities = levels(vposts$city)
# We could do the inner function in one line, but it is hard to read so break it down into steps.
# names( head( sort( table(vposts$maker[ vposts$city == x & vposts$byOwner == y & !is.na(vposts$city) ]),
# decreasing = TRUE), 3) )
makeByCityByOwner = lapply(X = c(TRUE, FALSE), FUN = function(y){
sapply(X = cities, FUN = function(x){
t = table(vposts$maker[ vposts$city == x & vposts$byOwner == y & !is.na(vposts$city) ])
s = sort(t, decreasing = TRUE)
h = head(s, 3)
n = names(h)
})
})
names(makeByCityByOwner) = c("Owner", "Dealer")
makeByCityByOwner
## $Owner
## boston chicago denver lasvegas nyc sac
## [1,] "ford" "chevrolet" "ford" "ford" "nissan" "toyota"
## [2,] "honda" "ford" "chevrolet" "chevrolet" "toyota" "ford"
## [3,] "chevrolet" "honda" "toyota" "toyota" "honda" "chevrolet"
## sfbay
## [1,] "toyota"
## [2,] "honda"
## [3,] "ford"
##
## $Dealer
## boston chicago denver lasvegas nyc sac
## [1,] "ford" "chevrolet" "ford" "ford" "nissan" "ford"
## [2,] "toyota" "ford" "chevrolet" "nissan" "toyota" "toyota"
## [3,] "chevrolet" "nissan" "dodge" "chevrolet" "honda" "chevrolet"
## sfbay
## [1,] "toyota"
## [2,] "ford"
## [3,] "bmw"
# Check if the top in each city by owner and by dealer match
makeByCityByOwner$Owner[ 1, ] == makeByCityByOwner$Dealer[ 1, ]
## boston chicago denver lasvegas nyc sac sfbay
## TRUE TRUE TRUE TRUE TRUE FALSE TRUE
A quick visual scan shows that 2 of the top 3 makes by city for sale by owner are among the top 3 for sale by dealer within the same city. The top in each city by owner is the same as by dealer in all cities except SacTown. They are fairly similar.
Visually compare the distribution of the age of cars for different cities and for “sale by owner” and “sale by dealer”. Provide an interpretation of the plots, i.e., what are the key conclusions and insights?¶
There are a few clear data errors with years in c(4, 1900, 2022), potentially others as well. We could fix them, or just ignore them since there are only a few data points that will not affect our analysis much.
The 2022 Honda Odyssey “has only 117102 miles” so it is probably a typo for 2002, so let’s fix it that way.
The year = 1900 ads are for wheels and buying back cars that fail the smog test so they are not cars at all. Remove them.
The Jeep with year = 4 is probably 2004 since it has a “AM/FM cassette player-muli CD player”, so let’s fix it that way.
# Omitted here for brevity
# vposts[ vposts$year %in% c(4, 1900, 2022) & !is.na(vposts$year), ]
# Clean up
vposts$year[ vposts$year == 2022 & !is.na(vposts$year) ] = 2002
vposts = vposts[ -which(vposts$year == 1900 & !is.na(vposts$year)), ]
vposts$year[ vposts$year == 4 & !is.na(vposts$year) ] = 2004
# Create an age variable
vposts$age = 2016 - vposts$year
# Overall by owner has older cars, dealers have newer cars
histogram( ~ age | byOwner, data = vposts, main = "Age by Owner/Dealer", col = "lightblue")
# Subset to zoom in a bit on the plots
idx = ( vposts$age < 25 & !is.na(vposts$age))
# Zoomed in
histogram( ~ age | byOwner, data = vposts, subset = idx, main = "Age by Owner/Dealer Zoomed",
col = "lightblue")
# Looking by city, there is not much difference at all across cities for owner vs. dealer
histogram( ~ age | byOwner + city, data = vposts, subset = idx, main = "Age by City by Owner/Dealer Zoomed",
col = "lightblue")
It does seem that cars for sale by owners tend to be older than cars for sale by dealers. However, this does not seem to vary much by city.
Plot the locations of the posts on a map? What do you notice?¶
Remember that the data was limited to just 7 major cities. Thus it would be a mistake to conclude something like almost all used cars on Craigslist (or from whatever website this data comes) sell in these 7 cities. We can conclude that the location of the people (and/or the cars themselves) who sell used cars in these major cities tend to be clustered fairly tightly around the major cities. Most buyers may be unlikely to travel far to look at, let alone, buy a used car.
There could be several explanations for the points far from one of the major cities. They could be travelling when they actaully posted the ad, for example. But in general, the location of the person posting the car is generally the same as the city in which they are attempting to sell the vehicle.
We also notice in the second plot of California that it seems that many people in Sac Town post their car on SF Bay Area, which we can see by the SF Bay points bleeding into the Sacramento area on the map. Perhaps they are trying to reach a bigger audience in the Bay.
We also notice that there are a few people in Reno / Tahoe posting ads in Sacramento and SF Bay. And there also appears to be somebody posting off the coast of Monterrey, but perhaps that is a data error :)
We could make one more improvement to this plot by using an alpha parameter to control the transparancy of the plotting points to better see density and bleeding into other regions.
# Split the data by city to see if location of poster is clustered around posting city
locationByCity = split(vposts[ , c("long", "lat")], vposts$city)
# Get color palette and then reverse the order darkest to lightest
col.palette = brewer.pal( length(locationByCity), "Purples")
col.palette = col.palette[ length(col.palette):1 ]
map('state', mar = c(0,0,0,0))
invisible(
lapply( 1:length(locationByCity),
FUN = function(x){
points(x = locationByCity[[ x ]]$long, y = locationByCity[[ x ]]$lat,
pch = ".", cex = 5, col = col.palette[x] )
}
)
)
legend("bottomleft", legend = names(locationByCity), col = col.palette, pch = 15, cex = 0.9)
map('state', region = 'california', mar = c(0,0,0,0))
points(x = locationByCity[[ "sac" ]]$long, y = locationByCity[[ "sac" ]]$lat,
pch = ".", cex = 4, col = col.palette[1] )
points(x = locationByCity[[ "sfbay" ]]$long, y = locationByCity[[ "sfbay" ]]$lat,
pch = ".", cex = 4, col = col.palette[5] )
# reset margins
par(mar = c(5, 4, 4, 2) + 0.1 )
Summarize the distribution of fuel type, drive, transmission, and vehicle type. Find a good way to display this information.¶
Notice that in the dotplot below, that the distributions in the different panels are all almost identical excpet that the distribtion shows some variation in the middle column where fuel type = "gas". Thus, we can essentially drop the fuel type from the plot, subset to just fuel type = "gas" and consider the relationship among the remaining three variables.
# 9 panels; circle colors are the vehicle type
# dotplot( table(vposts$fuel, vposts$drive, vposts$transmission, vposts$type))
# 15 panels
dotplot( table(vposts$type, vposts$fuel, vposts$drive, vposts$transmission),
auto.key = list(columns = 3), par.settings = simpleTheme(cex=1.2, pch=16) )
# fuel = "gas" for almost all the data
table(vposts$fuel, useNA = "ifany")
##
## diesel electric gas hybrid other <NA>
## 1071 68 29222 350 1187 2771
Subsetting to just fuel == "gas" we see that automatic transmissions are the most common across all types of cars and across drive trains, followed by manual then other. Within the rear wheel drive vehicles, manual stick shifts do have a higher percentage than other verhicle types for hatchback, coupe and convertible, which makes sense since coupe and convertibles tend to be sports cars and hatchbacks tend to be cheaper, entry level models. Among 4 wheel ddrive, offroad vehicles have a higher
idx = (vposts$fuel == "gas" & !is.na(vposts$fuel))
dotplot(
prop.table( table(vposts$type[idx], vposts$drive[idx], vposts$transmission[idx], useNA = "ifany"),
margin = c(1,2) ),
xlim = c(-0.05,1.05), auto.key = list(columns = 3), par.settings = simpleTheme(cex=1.2, pch=16),
xlab = "Percent"
)
Plot odometer reading and age of car? Is there a relationship? Similarly, plot odometer reading and price? Interpret the result(s). Are odometer reading and age of car related?¶
Previously, we fixed some prices and we also concluded that we would subset price to those prices >= 500 and <= 50,000, which captures about 94% of the data.
We should spend some time cleaning up the odometer readings. For example, the max odometer reading of 1234567890 is just some annoying poster. But to keep things simple here, we see that 99th percentile of odometer readings is 2.610^{5}, thus we will trim the data at 500,000 to capture almost all of the distribtion.
As we can see from the smoothScatter plot below, there generally does seem to be an upward trend for the overwhelming majority of the data. However, notice the dense shading between about 5 years of age and 20 years of age that have low odometer readings. Also notice that once age of the vehicle gets beyond about 25 years or so that there is actually a downward trend in that subset of the data. This is likely because very old antique cars just sit in some old man’s garage getting polished and buffed and only taken out for a ride during antique car shows.
quantile(vposts$odometer, probs = 0.99, na.rm = TRUE)
## 99%
## 260000
idx = ( vposts$odometer < 500000 & !is.na(vposts$odometer) & !is.na(vposts$age) )
smoothScatter(x = vposts$age[idx], y = vposts$odometer[idx], xlab = "Age of Vehicle",
ylab = "Odometer Reading", main = "Odometer Readings by Age of Vehicle" )
As we can see in the smoothscatter plot below, there is a general negative relationship between odometer reading and price, but note that there are some very expensive cars with low odometer readings, many of those are antique cars.
idx = ( vposts$odometer < 500000 & vposts$price >= 500 & vposts$price <= 100000 &
!is.na(vposts$odometer) & !is.na(vposts$price) )
smoothScatter(x = vposts$odometer[idx], y = vposts$price[idx], xlab = "Odometer Reading",
ylab = "Price of Vehicle", main = "Price by Odometer Readings of Vehicle" )
Identify the “old” cars. What manufacturers made these? What is the price distribution for these?¶
From the first smoothScatter plot below, it would seem that cars older than say 35 years are “old.”
From the table below, Chevy and Ford make up more than 50% of the “old” cars and contains most of the classic American car brands, as one might expect. In particular, there are not many “old” Japanese cars since the U.S. did not start importing Japanese cars on a large scale until the oil shocks of the 1970’s, whereas our cutoff for “old” is approximately 1970.
Comparing the price distribution of “old” cars compared to all cars, “old” cars seem have more density at higher prices with the majority of the prices less than \$40,000 whereas the bulk of the overall data tends to be less than \$20,000.
# Check overall age and price relationship
idx = (vposts$price >= 500 & vposts$price <= 100000 &
!is.na(vposts$price) & !is.na(vposts$age) )
smoothScatter(x = vposts$age[idx], y = vposts$price[idx], xlab = "Age of Vehicle",
ylab = "Price of Vehicle", main = "Price by Age of Vehicle" )
# Look at makers and price distribution for "old" cars
idx = (vposts$age >= 35 & !is.na(vposts$age))
# Look at the 10 largest makers of "old" cars
round( sort( tail( sort(table(vposts$maker[ idx ])) / sum(table(vposts$maker[ idx ])), 10)
, decreasing = TRUE), digits = 3 )
##
## chevrolet ford volkswagen dodge pontiac cadillac
## 0.315 0.213 0.055 0.038 0.038 0.037
## buick gmc plymouth oldsmobile
## 0.027 0.026 0.025 0.024
idx = (vposts$age >= 35 & !is.na(vposts$age) & !is.na(vposts$price))
smoothScatter(x = vposts$age[idx], y = vposts$price[idx], xlab = "Age of Vehicle",
ylab = "Vehicle Price", main = "Price by Age of Vehicle" )
I have omitted one important variable in this data set. What do you think it is? Can we derive this from the other variables? If so, sketch possible ideas as to how we would compute this variable.¶
When searching for cars on websites, the most important filters are usually year, make and model, in that order. Notice that year and make (i.e. maker) are stand-alone variables in the dataset, but model is not. However, notice that variable called header in the dataset is year, make, then model. Thus if we could parse the text string of each header to pull out the model, we could derive our own stand-alone variable for model. In fact, a future homework assignment will focus on using regular expressions to do just this type of string parsing.
head(vposts$header, 20)
## [1] "2012 Chevrolet Camaro SS"
## [2] "2013 Chevrolet Equinox LT"
## [3] "2013 Nissan Altima 2.5 SV"
## [4] "2009 Infiniti M35x X"
## [5] "2013 Infiniti G37x X"
## [6] "2012 Acura MDX 3.7L"
## [7] "2010 Toyota Yaris"
## [8] "2012 Acura RDX Base"
## [9] "2014 Chevrolet Cruze 1LT Auto"
## [10] "2009 Lexus IS 250"
## [11] "2013 Toyota Camry SE"
## [12] "2014 Honda CR-Z"
## [13] "2008 BMW 335 xi"
## [14] "2013 Nissan Juke SL"
## [15] "2013 Dodge Durango SXT"
## [16] "2012 Toyota Camry 4dr Sedan I4 Automatic SE"
## [17] "2012 Toyota Venza LE"
## [18] "2014 Nissan Juke S"
## [19] "2012 Toyota Corolla"
## [20] "2012 Honda Pilot EX-L"
Display how condition and odometer are related. Also how condition and price are related. And condition and age of the car. Provide a brief interpretation of what you find.¶
# Print out conditions so we can cut and paste them into smaller
# categories. There's really no way out of this since a human has to decide
# what the new categories should be.
conditions = levels(vposts$condition)
conditions = sprintf('"%s",\n', conditions)
cat(conditions)
## "0used",
## "207,400",
## "ac/heater",
## "carfax guarantee!!",
## "certified",
## "complete parts car, blown engine",
## "excellent",
## "fair",
## "front side damage",
## "good",
## "hit and run :( gently",
## "honnda",
## "like new",
## "mint",
## "muscle car restore",
## "needs bodywork",
## "needs restoration!",
## "needs restored",
## "needs total restore",
## "needs work",
## "needs work/for parts",
## "new",
## "nice",
## "nice rolling restoration",
## "nice teuck",
## "not running",
## "parts",
## "pre owned",
## "pre-owned",
## "preowned",
## "preownes",
## "project",
## "project car",
## "rebuildable project",
## "restoration",
## "restoration project",
## "restore",
## "restored",
## "rough but runs",
## "salvage",
## "superb original",
## "used",
## "very good",
# We'll base the new categories on the most common existing categories.
sort(table(vposts$condition))
##
## 0used 207,400
## 1 1
## ac/heater complete parts car, blown engine
## 1 1
## front side damage hit and run :( gently
## 1 1
## honnda mint
## 1 1
## muscle car restore needs restoration!
## 1 1
## needs total restore needs work
## 1 1
## needs work/for parts nice
## 1 1
## nice rolling restoration nice teuck
## 1 1
## not running pre-owned
## 1 1
## preownes project car
## 1 1
## rebuildable project restoration
## 1 1
## restoration project restore
## 1 1
## restored rough but runs
## 1 1
## carfax guarantee!! needs restored
## 2 2
## pre owned superb original
## 2 2
## needs bodywork project
## 3 3
## parts preowned
## 5 10
## very good certified
## 10 54
## salvage new
## 61 273
## fair used
## 770 1180
## like new good
## 2898 4663
## excellent
## 7543
# We'll base the new categories on the most common existing categories.
sort(table(vposts$condition))
##
## 0used 207,400
## 1 1
## ac/heater complete parts car, blown engine
## 1 1
## front side damage hit and run :( gently
## 1 1
## honnda mint
## 1 1
## muscle car restore needs restoration!
## 1 1
## needs total restore needs work
## 1 1
## needs work/for parts nice
## 1 1
## nice rolling restoration nice teuck
## 1 1
## not running pre-owned
## 1 1
## preownes project car
## 1 1
## rebuildable project restoration
## 1 1
## restoration project restore
## 1 1
## restored rough but runs
## 1 1
## carfax guarantee!! needs restored
## 2 2
## pre owned superb original
## 2 2
## needs bodywork project
## 3 3
## parts preowned
## 5 10
## very good certified
## 10 54
## salvage new
## 61 273
## fair used
## 770 1180
## like new good
## 2898 4663
## excellent
## 7543
# Define new categories.
new_cats = list(
excellent = c("excellent"),
good = c("good", "very good"),
"like new" = c("like new", "mint", "new", "pre owned", "pre-owned", "preowned", "preownes"),
used = c("0used", "used"),
fair = c("fair", "nice", "nice teuck"),
salvage = c("complete parts car, blown engine", "front side damage", "hit and run :( gently",
"muscle car restore", "needs bodywork", "needs restoration!", "needs restored",
"needs total restore", "needs work", "needs work/for parts", "nice rolling restoration",
"not running", "parts", "project", "project car", "rebuildable project", "restoration",
"restoration project", "restore", "restored", "salvage", "rough but runs"),
other = c("207,400", "ac/heater", "carfax guarantee!!", "certified", "honnda", "superb original" )
)
# Convert conditions to new categories.
vposts$new_cond = vposts$condition
levels(vposts$new_cond) = c(levels(vposts$new_cond), "other")
for (i in seq_along(new_cats)) {
new_cat = names(new_cats)[[i]]
vposts$new_cond[vposts$new_cond %in% new_cats[[i]]] = new_cat
}
vposts$new_cond = factor(vposts$new_cond)
# Restrict to odometer less than 5 million.
sane_odo = subset(vposts, odometer < 5e6)
odo_by_cond = split(sane_odo$odometer, sane_odo$new_cond)
boxplot(odo_by_cond, col = "salmon")
title("Odometer Distribution by Condition", ylab = "Miles")
# Make a second plot to show the distribution better.
boxplot(odo_by_cond, col = "skyblue1", ylim = c(0, 5e5))
title("Odometer Distribution by Condition", ylab = "Miles")
# Now we can see that the highest odometer readings seem to be for the "fair"
# and "good" conditions, which is a little surprising. It's possible people
# overstate the condition of the car when the odometer is higher, to try to
# make it sound more appealing. The condition with the most spread-out
# distribution is "salvage," which makes sense because salvage cars could be
# very old, or they could be new cars that were damaged somehow.
sane_price = subset(vposts, price < 2e5)
price_by_cond = split(sane_price$price, sane_price$new_cond)
boxplot(price_by_cond, col = "thistle")
title("Price Distribution by Condition", ylab = "Dollars")
# Make an age column.
vposts$age = 2015 - vposts$year
sane_age = subset(vposts, age < 200)
age_by_cond = split(sane_age$age, sane_age$new_cond)
boxplot(age_by_cond, col = "aquamarine")
title("Age Distribution by Condition", ylab = "Years")
# The price and age distributions don't show anything too surprising. Price and
# condition appear to be directly related, with salvage cars having the lowest
# prices. Cars that are "like new" or "excellent" are sometimes offered for
# extremely high prices, whereas this is less common with cars in worse
# condition. Age and condition, are inversely related: older cars seem to have
# worse conditions. As with odometer and condition, salvage cars exhibit the
# weakest relationship.