Pricing the Internet

More or less by accident I ran across http://www.dtc.umn.edu/~odlyzko/doc/smart.pricing.pdf
an analysis of prospects for internet pricing by Andrew Odlyzko, an old Bell Labs hand now
at the U. of Minnesota.  He makes the case against congestion pricing, arguing that consumers
and perhaps business as well prefer simple flat rates, i.e. usage independent pricing and that
technology is capable of expanding to meet this preference.  Having argued the other side
of this case in http://comjnl.oxfordjournals.org/content/27/1/8.full.pdf  long ago, I was a bit
surprised,  and am still not entirely persuaded.  Maybe this only reflects that I haven't fully
accepted recent developments in behavioral economics?

Amsterdam Econometrics Games 2015

This year for the first time we sent a UIUC team
to the Econometric Games in Amsterdam.  This is a 3 day competition among
30 Universities; there is a preliminary round to select 10 finalists and then a
second round to determine the three prize winners.  The topic this year was
Longevity and Longevity Risk.  Details of this year's competition are available
here   http://www.econ.uiuc.edu/~roger/EGames/Case1.pdf   and
here  http://www.econ.uiuc.edu/~roger/EGames/Case2.pdf

Our team:   Nicolas Bottan, Cesare Buiatti, Julia Gonzalez and Jiaying Gu
placed second in this competition outperforming perennial favorites like
Copenhagen, Aarhus, Amsterdam, Carlos III as well as Oxford, Cambridge
and Harvard.  Our team deserves a ticker tape parade when they return
this weekend, unfortunately ticker tape is scarce these days.  So we will have
a celebration of their remarkable achievement on April 17th as part of the
Fourth Annual Boneyard Conference of Econometrics.  Details here:
http://www.econ.uiuc.edu/~roger/seminar/Boneyard15.pdf

So the time has come put your brackets on the table, here is a random realization of a bracket from
the QBracketology model complete with scores, and survival curves for the various teams based on
1000 realizations of the Tournament.  As you can see, not surprisingly, Kentucky is an overwhelming
favorite, with a probability of 0.407 of winning.  Only time will tell whether the estimated probabilities of the model are competitive in the Kaggle competition.

Mea Copula Redux

Never one to leave bad enough alone,  I've yet again returned to Quantile Bracketology as a
March diversion,  in lieu of watching any basketball.  This year Kaggle (sponsored by HP) is
again offering modest prizes for predicting probabilities of the NCAA Tournament, and this
year they provide somewhat better data:  30 years of past seasonal results as well as the tournament
outcome data.

I gave a talk about my technology for this a week or so ago in the Stat Department, slides for
that talk are linked here:  http://www.econ.uiuc.edu/~roger/research/bracketology/MM.html

A question that puzzled me last year was whether it might help to shrink the model's predicted
probabilities toward 1/2.  In an effort to explore this, I estimated models and predicted tournament
probabilities for each of the last 30 years.  A naive shrinkage rule was employed which simply
took the model's original probabilities as if they were based on the frequency of successes out
of 20 trials, and used a Beta(a,a) prior to do the shrinkage.  This yields a linear shrinkage of
modest amout for a taking values 0, 1, ... , 5,  where 0 corresponds to no shrinkage at all.
The results are rather mixed.  In the figure below I plot the logistic loss achieved for each
year for the various levels of a, with the black line representing no shrinkage and the various
colored lines the various degrees of shrinkage.  In 2/3 of the years no shrinkage is best.  In
one year, 2011, in which the model is most disastrous, maximal shrinkage is optimal.  Comparison
of the mean performance over the whole 30 year span yields:

   a =  0    a =  1    a =  2    a =  3    a =  4    a =  5 

0.5403810 0.5419280 0.5464682 0.5515166 0.5566610 0.5617073 

So, it would appear that shrinkage just tends to make the predictions more bland without really

helping much at all.

Mea Copula

The usual craziness last Sunday drove me to consider updating the bracketology methods that Gib
Bassett and I described here:
http://www.tandfonline.com/doi/abs/10.1198/jbes.2009.07093#.Uys1PVxTq6Q

Since the Intel (Kaggle.com) contest had reasonable looking data structures, it was reasonably
straightforward to modify my old R software to do 2013-14 version of what we had done
earlier for 2003-4.  The Kaggle competition wanted submissions that made probability estimates
for all 2278 possible match ups for this years tournament.  Our approach was to estimate a
pairwise comparison model for team scores for the entire pre-tournament season  for a relatively
fine grid of tau values -- by (what else?) quantile regression.  The resulting QR models have a
design matrix that is 10724 by 704, but are very sparse so on a grid 1:199/200 the estimation
can all be done in about a minute on my desktop machine.  Then comes the fun of simulating
tournament brackets, and estimating probabilities.  For any pairing, ij,  we have an estimated
quantile function for team i's score and another estimated quantile function for team j's score.
Yesterday afternoon as the deadline for the Intel contest loomed closer and closer, I lost focus
and decided to compute probability estimates based on what one might call the comonotonic
copula -- that is under the assumption that if in our hypothetical game team i achieves quantile
tau in its scoring performance, then team j will also achieve quantile tau performance in this
game.  Thus, if team i's quantile function lies above the quantile function for team j,  the
predicted probability will be 1, that is we will assert that team i will always beat team j.
But clearly team i might have a bad day, when team j has a good one, so another limiting
option would be to consider an independent copula:  each team gets an independent draw
from a uniform that then gets plugged into their quantile function.  Various compromises then
suggest themselves.  In the JBES paper we used a Frank copula model that implied a Kendall
correlation of about 0.27, so mild positive correlation between performance of the two teams.
In contrast to the initial probability estimates with the comonotonic model that produced roughly
half of the 2278 phats at 0 or 1, the Frank copula model produced a much more uniform distribution
of them, as illustrated by the histogram below.  Unfortunately, this didn't occur to me until after
the deadline passed for the Intel submissions.

Empirical Bayes Bakeoff

The ISI has a new journal called Stat,  I'm not sure this is a good marketing appellation in the modern world of search engines, but it has an interesting premise.  It promises quick turnaround, prompt electronic publication and a sanctioned blog forum in which readers can comment on published articles.  It isn't intended for long expository pieces, rather it is meant for pithier, perhaps more provocative pieces that wouldn't fit well in the current journals.  This is a niche that Biometrika's Miscellanea section used to fill.  I decided to give this a whirl recently, submitted a 5 page paper comparing some empirical Bayes methods in several simulation settings of the Johnstone and Silverman "needles in haystacks" type.  In a couple of weeks I received a favorable, and very well informed referee report* and the paper was accepted.  It appeared after a bit of copy-editing a week or two later.  Whether it attracts any Statblog attention remains to be seen,  but I certainly hope so.  It is freely available at:http://onlinelibrary.wiley.com/doi/10.1002/sta4.v3.1/issuetoc  for a while at least.


* In the realm of referee reports, "favorable" and "well informed" are synonyms, so this sentence was a bit redundant.

Winter Warming?

With all the complaining about the winter weather this year,  I became curious about trends in winter
weather in my home town.  So here is a plot of the last 100 years, or so, of minimum January temperatures in Grand Forks, North Dakota.  The three black lines are regression quartile fits
and the red line is the mean regression fit.  The mean and median fits both indicate that there is roughly a 0.1  degree F per year warming trend over the entire period -- so 6 degrees F since I was in first grade.   This year the minimum was -30F, so quite bit below the median trend line.