Data is currently at
https://data.giss.nasa.gov/gistemp/tabledata_v4/GLB.Ts+dSST.csv
or
https://data.giss.nasa.gov/gistemp/tabledata_v4/GLB.Ts+dSST.txt
(or such updated location for this Gistemp v4 LOTI data)
January 2024 might show as 124 in hundredths of a degree C, this is +1.24C above the 1951-1980 base period. If it shows as 1.22 then it is in degrees i.e. 1.22C. Same logic/interpretation as this will be applied.
If the version or base period changes then I will consult with traders over what is best way for any such change to have least effect on betting positions or consider N/A if it is unclear what the sensible least effect resolution should be.
Numbers expected to be displayed to hundredth of a degree. The extra digit used here is to ensure understanding that +1.20C resolves to an exceed 1.195C option.
Resolves per first update seen by me or posted as long, as there is no reason to think data shown is in error. If there is reason to think there may be an error then resolution will be delayed at least 24 hours. Minor later update should not cause a need to re-resolve.
@gonnarekt for me the variance noticably went up today from yesterday's run due to model variance (edit2: causing the std. dev to rise from yesterday by ~0.007 C).
These are the LOTI temps (superensemble debiased, without adjustment of own past loti prediction errors) that I've been getting since I've started using the super ensemble method I describe below (ended up back where I started today).
So far the little on-going validation has shown a slight bias to be being a bit cold (MAE ERA5-predicted is +0.02 C), while the unbiased version (MAE ERA5-predited is +0.01C), but this is still fairly small as it doesn't result in that different LOTI temps (difference of 0.002C in the LOTI temp being data for the latter half of the month).
When I use my own past errors in these markets as a further adjustment the LOTI temps go up by ~ 0.011 C.:
LOTI predictions for July:
# 07/14 - 07/24 # 0.966 # 0.959 # 0.954 # 0.950 # 0.948 # 0.948 # 0.954 # 0.956 # 0.956 # 0.957 # 0.966
chart from (mostly 00Z) ensembles today (predicting an ERA5 July temp of 16.591 C):
Polymarket has shifted largely in favor of 1.00-1.04.
This highseems unjustified based on the forecast I have presently, and even though I have zero YES shares in it I've put even more limit orders for NO if anyone wants to bet with Polymarket and bet it up a bit.
@gonnarekt with new data and little correction i get 0.955-0.995 range, i will check it one more time at the end of the next week
@gonnarekt switched to a new model to combine ensemble forecasts, optimizing on bias-variance on arimax adjusted forecasts of ensemble means (this forms a super ensemble, weighting the arimax mean forecasts from EPS, GEPS, GEFS, with weights as 0.57, 0.33, 0.10 respectively, and adjusting based on biases found in training; this matches roughly in line with own experience of ranking these models performance by forecasts error).
Right now the expected value overall is still 0.966 C.
I take the variance from the actual ensemble forecasts and weight them to come up with an estimate for the super ensemble's contribution to the overall variance (0.023) for the month. Right now its contribution is slightly larger than the estimate for the contribution of the variance for the ERA5->gistemp model using own prediction errors (0.021).
I don't have confidence that this total overall variance is actually too large, so I'm reducing my bet sizes for a while accordingly.
Edit: Here are the ensemble mean biases estimated from training: (model - era5)
EPS: -0.022
GEPS: +0.047
GEFS: + 0.041
EPS predicted temps are slightly cooler than ERA5, while GEPS and GEFS are quite a bit hotter than ERA5.
Unfortunately I lack a diverse set of forecasts for GEPS so its performance and bias is only based on the last few months (where as for EPS/GEFS I have samples from at least 1 year).
The trend has been downward (although slowing) for the last 5 days for the (unadjusted by own errors) LOTI values..
07/14 - 07/18:
0.966
0.959
0.954
0.950
0.948
After adjusting for own past errors, expectation (0.958 C) is still in the second bin, but still with a wide CI estimated for the std. dev.: 99.5% Confidence Interval for Standard Deviation: (0.1044, 0.1611), which puts most of the probability (~45%) still in the lowest bin.
Right now the variance breaks down into the components as such (at least now the super ensemble variance is lower than the ERA5 model variance):
Own error variance (ERA5 only) 0.0207
Super ensemble var from ensembles: 0.0157
@gonnarekt Confused about that shift upwards for you.
I ran the numbers again without the super ensemble bias adjustment and it didn't move much: the (unadjusted on own error) LOTI is 0.951 C, based on the arimax adjusted ensemble means as weighted as I mentioned in the comment above.
For reference what are you getting for your ERA5 month average if you output that in your model?
(super ens bias adjusted): Absolute ERA5: 16.573
(super ens bias unadjusted): Absolute ERA5: 16.576
@gonnarekt From that ERA5 temp you provided my linear ERA5->GISTEMP model indicates something like a GISTEMP of ~1.01 C for July, so it looks like its mostly disagreement about our forecasts for the ERA5 temp, not the actual ERA->GISTEMP fitting.
Polymarket right now is leaning towards your estimate of the LOTI.
Updated after today's 00Z ensembles, the temp is holding mostly steady at 16.573 C (with the super ensemble debiasing) corresponding to an (unadjusted by own errors) LOTI of 0.948 C (adjusted upwards to 0.959 C by own errors).
Super ens. without debiasing ERA5 is 16.575 C (so very close).
An updated chart with forecast (still showing a drop coming):
@gonnarekt the upwards adjustment on own errors I refer to is simply a small sample debiasing based on my own ERA5-> Gistemp predictions once all the ERA5 data has come in … in other words most of the recent predictions that I’ve made after the month has finished that I’ve made have been a bit under.
(I use my own errors to come up with a CI for the std. dev of this error, rather than relying on the model std. dev. from the historical data which has distributional shifts and larger variance historically — this variance along with the variance from the super ensemble (the variance plotted) is what I’m using to come up with probabilities).
Models now all agree in next 15 days of a major rise... after that who knows (the decline at the end of June/beginning of July was very abnormal to me based on some simple statistical ARIMA models)...
can see how far away from the 95% CI of the model before the drop by all the horizontal blue lines (monthly mean prediction) above the yellow current 95% region:
Right now my statistical model and the mid-range-only split from the models are in good agreement on the center of the distribution (~1.06 and 1.07 C), so I'm increasing the sharpness of my bet a bit.
Long range GEFS shows an increasing trend (its already 0.1-0.2 C above mid range forecasts towards end of mid range forecast); Prophet's continuation is based solely on ERA5 data (discontinuous jump).