Tuesday, June 9, 2020

11. Correlation between station temperature records

Central to statistical processes such as homogenization and breakpoint adjustment that I have mentioned in previous posts is the concept of correlation. In climate science it is used to help compare temperature records and generate regional trends, expectations or benchmarks against which the accuracy of local station records can be judged or measured (in theory).
A correlation coefficient is a mathematical measure of the degree by which two sets of data move in the same way. To put it simply, if you have two sets of data that you can plot as a scatter graph, if the points on the graph follow a straight line then the two sets of data are perfectly correlated. If the line of points slopes upwards then the correlation coefficient, ρ, will be +1.0, and if it slopes down then ρ = -1.0 . This rarely happens, though. Instead the points are usually scattered about a line or direction, and the more the points diverge from the line of perfect correlation (or best fit, because in that instance the two functions will be the same) the lower the correlation coefficient will be in magnitude.

Correlation coefficients are useful because the two sets of data that they compare don’t need to be of similar magnitudes, or even have the same units. It is the relative change of each that is measured and compared. And they are used a lot in climate science, particularly to compare temperature records.

As I pointed out in the last post, most climate groups use correlation coefficients to construct expected regional temperature trends against which individual temperature records are compared for the purpose of quality control (i.e. identifying measurement errors, station moves etc.). The assumption is that weather stations that are close together should be more strongly correlated than those that are further apart. But the questions that interest me are these.

(i) By how much does the degree of correlation change with increasing station separation?

(ii) What is be the impact of data smoothing on the correlation coefficient?

(iii) What effects do breakpoint adjustments have on the correlation coefficient?

The first of these questions is significant because it will determine the relative weighting of each record in the regional expectation trend, and more importantly indicate how independent these stations really are from one another, or to what degree they are coupled. The second question was initially included as an curiosity, but now has greater significance given the scaling behaviour of the Berkeley Earth adjusted data highlighted in the last post. The third question is about testing the quality of Berkeley Earth adjusted data, and in particular the validity of breakpoint adjustments.

In order to answer these questions I have calculated the correlation coefficients between most combinations of the 50 New Zealand stations that have more than 180 months of temperature data. This results in a potential 1275 pairs. However, only those pairs of records where the overlap of data within a specified time-frame exceeded a certain threshold have been included. Typically, this threshold is a minimum overlap of 200 months between 1971 and 2010. As a result 660 pairs of records qualified for this analysis (or 52%). In order to determine the temperature anomaly for each set of data, the monthly reference temperatures (MRTs) were calculated for the period 1981-2000. It was found that having consistency in the choice of MRT period for all station records was a major factor in minimizing the spread of values for the correlation coefficient.

Once the qualifying stations had been determined, the correlation coefficient for each pair of stations was calculated. There are a number of different types of correlation coefficient that are widely used. I have settled for the most commonly used variant, Karl Pearson's product moment correlation coefficient (PMCC).



 Fig. 11.1: Correlation (PMCC) between raw temperature records for the period 1971-2010 for all stations in New Zealand with a minimum overlap of 200 months.

The data used in the correlation calculation was the anomaly data, not the raw temperature data. The main reason for this is that the raw temperature data is dominated by seasonal oscillations (the MRTs) that are far bigger than either the underlying trend or the monthly anomalies. As a result, applying a correlation test to the raw temperature data will yield consistently large positive correlations of more than 0.8 (as shown above in Fig. 11.1). This highlights one of the difficulties with correlation coefficients; while they seem quantitative, the interpretation of their values can be very subjective or relative. In some instances a correlation coefficient of 0.7 may be very good, in others like that in Fig. 11.1 above, it may be quite poor. Ideally, the maximum range for your input variable (in this case the station separation) should produce a correspondingly large change in correlation coefficient. While this is not true for raw temperature data, it most certainly is true for the anomaly data (see Fig. 11.2 below).



Fig. 11.2: Correlation (PMCC) for the period 1971-2010 between temperature anomalies for all stations in New Zealand with a minimum overlap of 200 months.


The data in Fig. 11.2 is split into two columns. On the left are plots of the PMCC versus station separation for the original anomaly data, on the right are the same plots but for Berkeley Earth adjusted data. In each case three sets of data are compared; (i) the monthly anomaly, (ii) the 12 month moving average of the anomaly, and (iii) the 5 year moving average of the anomaly.

If we first consider the data in Fig. 11.1(a) we see that the PMCC for the monthly anomaly data exhibits a clear downward trend with increasing station separation. It is also possible to identify a boundary line above which almost no data points are present. This boundary line for the maximum observed PMCC at that station separation (denoted as ρmax) has the following approximate functional dependence on the station separation d in kilometres,

(11.1)

It should be noted that not only is this limit on correlation coefficients seen for New Zealand data, but a near identical trend is also seen for data from the Antarctic, thereby suggesting this behaviour may be more universal. This boundary line is illustrated graphically below in Fig. 11.3 along with the monthly anomaly data from Fig. 11.2(a).



Fig. 11.3: Data from Fig. 11.1(a) with maximum trend line (in red).


When it comes to data smoothing ( see Fig. 11.2(b) and Fig. 1.2(c) ), it appears this has two influences on the correlation coefficient. First it reduces the gradient of the boundary line, and second it increases the random scatter of points below this line. The amount by which the slope of the boundary line changes is broadly (but not exactly) consistent with the scaling behaviour outlined in Post 9. As for the increased scatter, this at first glance appears counter-intuitive. One might instead expect the smoothing to increase the degree of correlation by removing fluctuations to reveal a regional trend that is the more or less the same on all temperature records. But remember, we are probably dealing with a fractal here. Just like the Julia set, it may be that the smaller the level of detail, the more dissimilar some adjacent locations can be.

So what about the Berkeley Earth adjusted anomaly data? The 12 month moving average data in Fig. 11.2(e) clearly has a stronger trend and less scatter than the two other data sets in Fig. 11.2(d) and Fig. 11.2(f). This is very different to what is seen with the unadjusted data. In addition the adjusted monthly anomaly data in Fig. 11.2(d) has more scatter than the unadjusted monthly anomaly data. On this evidence the breakpoint adjustment has (again) made the unsmoothed data worse, not better. Yet the 12 month moving average data in Fig. 11.2(e) appears to be the "best" data of all: it has a higher mean correlation coefficient, less scatter and is generally closer to the boundary line. Why?



 Fig. 11.4: Comparison of 12 month smoothed data from Paraparaumu aerodrome with regional expectation.


Well the answer may lie in Fig. 11.4 above. This is a comparison of the 12 month moving average data and the regional expectation for the station at Paraparaumu aerodrome taken from the Berkeley Earth site. This graph implies that it may be the 12 month moving average data that is used to construct the regional expectation (and hence the breakpoint adjustment) and not the monthly anomaly data. So, if the breakpoint adjustments are optimized to match the 12 month moving average data to the regional expectation, it is perhaps hardly a surprise that these datasets produce the highest correlation coefficients between stations. The surprise is that the improvement is not replicated in either the adjusted monthly anomaly data or other moving average data, which rather undermines the credibility of the whole breakpoint adjustment process.



Fig. 11.5: Square of the correlation coefficients in Fig. 11.2(a).


One of the purposes of the correlation coefficients is to quantify the weighting coefficients that are used to combine station records into regional expectations (see Fig. 11.4). This involves squaring the correlation coefficients to generate the weighting coefficient. The effect of squaring these coefficients is shown for the original monthly anomalies in Fig. 11.5 above and for the 12 month moving average of adjusted data in Fig. 11.6 below. In Fig. 11.5 only 29% had weightings of more than 0.5, while for the adjusted data in Fig. 11.6 it is significantly larger at 56%. This again has implications for how the regional expectation trends are constructed, and how representative they really are. A regional expectation based on the original monthly anomalies in Fig. 11.5 will rely on fewer datasets that are located closer to the target station. In Fig. 11.5 all data points with ρ2 > 0.5 represent station pairs that are less than 900 km apart. In Fig. 11.6 the equivalent distance is 1500 km.




Fig. 11.6: Square of the correlation coefficients in Fig. 11.2(e).


Finally, I analysed the long stations of New Zealand in isolation. This data is important for two reasons. First, these are the only stations with a significant amount of data that precedes 1940. That means they are essential for the construction of the regional expectation prior to 1940. The second reason is that they are fairly evenly spread across New Zealand. While this is a strength in that it means that the country is evenly covered with temperature data, on the down side it means that these stations are all (bar one pair) at least 200 km apart. I find it hard to reconcile that with the assertion that these stations could effectively error check each other for bad data, and that the resulting breakpoint adjustments that they would produce could be considered reliable. For supporting corroboration I would look at places like Greenland where the levels of snowfall, and the degree of glacier melt, have very local rather than regional behaviours.

Perhaps unsurprisingly then, the data in Fig. 11.7 below has a number of similarities with the data in Fig. 11.2, namely the effect of smoothing on the scatter of unadjusted data, and the effect of distance on the maximum correlation (as outlined in Eq. 11.1). It also has the same deficiencies; the totally random nature of the adjusted monthly data in Fig. 11.7(d), and adjusted anomalies that appear less reliable than their unadjusted counterparts.



Fig. 11.7: Correlation (PMCC) for the period 1854-2013 between temperature anomalies for long stations in New Zealand with a minimum overlap of 1080 months.


So, what of the three questions I asked at the start? Well the major breakthrough result from this analysis has been the discovery of the dependence of the correlation coefficient on station separation for the monthly anomalies as quantified in Eq. 11.1. What was surprising was the absence of any similar structure to the adjust monthly anomalies.

The impact of smoothing on the PMCC has been conflicting. The adjusted data has followed an expected trend, but possibly for the wrong reason (see Fig. 11.4). The unadjusted data still requires a degree of explanation, particularly the increasing randomness. While the gradient of the boundary line in Fig. 11.3 reduces (in magnitude) and the line moves to higher correlations as expected, the data below becomes more random. Is this because of fractal structure?

As for breakpoint adjustments, there was little of a positive slant found here to support their use. Their net impact appears to have been to decouple local station records and decrease overall levels of correlation. It is, therefore, difficult to reconcile the reality of their impact with the official viewpoint that they are supposedly correcting for data imperfections.



Sunday, June 7, 2020

10. Breakpoint adjustment and its effect on the scaling behaviour of anomalies

In my previous post (see here) I demonstrated that the noise or random temporal fluctuations in the temperature anomaly from weather station data were not consistent with white noise, but were instead characteristic of a one-dimensional time-dependent fractal with a 1/√ω power spectrum (where ω is the frequency of the oscillations in the noise signal). I then showed that these temperature fluctuations exhibit self-similarity with a fractal dimension of 0.25 (approximately). I also showed that one consequence of this is that the expected long term temperature fluctuations (i.e. those with timescales greater than 100 years) would be at least six times greater in magnitude (almost 1.0 °C) than most climate scientists probably realize. This analysis was done on the original raw unadulterated data (or at least what I assume is the original raw unadulterated data). In this post I shall apply the same technique to data that has been processed and adjusted by Berkeley Earth and look at the difference.

One of the most frequent claims made against climate science by sceptics is that the temperature data is manipulated. It is of course a claim that those climate scientists deny, but ever since the Climategate email scandal there has been a bad smell. It was, after all, that bad smell that prompted Richard Muller to set up Berkeley Earth in order to provide some independent verification of the results that the other groups (NOAA, GISS, HADCRUT) were producing.

That said, the claim that some climate scientists manipulate the data has never been completely rejected by climate scientists themselves. In fact they justify much of it on the grounds that the data is unreliable and needs to be “cleaned up”. Their rationale for these statistical interventions is this: some temperatures records are very long and therefore it is inevitable that something will go wrong with the data in that time. People will change, instruments will change (e.g. changes from Stevenson screens to electronic systems), procedures will change (e.g. time of data collection), and locations will change, so error checking needs to be put in place to compensate for any potential errors that may arise. This is particularly important since the primary purpose of most climate research is to determine the temperature change over time. This in turn requires the scientist to measure two temperatures, under identical conditions, at the same location, but a long time apart. It is therefore no good having very accurate measurements now if what you are comparing them against is uncertain.

In addition, data is often missing or incorrectly entered (e.g. in Fahrenheit rather than Celsius). The result of all this is that the climate scientists believe the data needs to be subject to a strict quality control during the analysis process, and two of the “data cleaning” techniques that they rely on to do this are homogenization and breakpoint adjustment.

Homogenization is a primarily used for two outcomes. The first is to use extrapolation or interpolation to fill in missing data points in the record. The second is to compare records from neighbouring stations and construct a regional average that can be used as a more reliable reference dataset.

Breakpoint adjustment is a data manipulation technique that first uses homogenization to construct an idealized reference for the region and then compares the actual station data to the reference. It then looks for points in the dataset of the target station where the difference between the target and the reference changes over a short time interval by an amount that is much larger than is normally seen. At this point a cut or break in placed in the data and data each side of the cut is shifted up or down. An example (taken from Berkeley Earth) is shown in Fig. 10.1 below.


Fig. 10.1: Anomaly data for Waiouru Airstrip.


In the above data for Waiouru Airstrip (Berkeley Earth ID - 172950) a breakpoint was detected by Berkeley Earth between October and November 1986, as shown below in Fig. 10.2. The red lines in Fig. 10.2 indicate the amount by which the data on the right (post October 1986) is deemed to be too low in value relative to the data before November 1986 when compared with a regional average.


Fig. 10.2: Difference in station anomaly and regional anomaly for Waiouru Airstrip.


This regional average appears to be constructed from the surrounding station data weighted by the square of their correlation coefficient with the target station (Waiouru Airstrip). There is only one problem with this, though; there is only one other station within 100 km with data spanning 1986 and that is at Ohakea 87 km away with only 234 months of data (see below). So how reliable is the weighting process? That is a major issue, particularly in the Southern Hemisphere where station density is much lower than in the USA or Europe.


 Fig. 10.3: Anomaly data for Ohakea.


Nevertheless, it can be seen that the breakpoint correction in Fig. 10.2 is an impressive 1.2 °C. However, this adjustment is not applied evenly. First, it equates to a different amount for each month (January - December) as shown in Fig. 10.4 (see below).


 Fig. 10.4: Total breakpoint correction for Waiouru Airstrip.


This effectively means that while the breakpoint is set at the same point for each of the twelve monthly datasets that make up the whole, the adjustment is different for each. Why? I’m still not sure about this and can see no obvious rationale. But surely if the cause is the same same (e.g. a station move), then the impact on all monthly readings should be the same.


Fig. 10.5: Actual breakpoint correction for Waiouru Airstrip.


The second feature of note is that adjustments are made to data on both sides of the breakpoint, but by different amounts (see Fig. 10.5 above). Those months before November 1985 are adjusted down, while those after are adjusted up. These adjustments are not equal in magnitude, but are scaled in proportion to the number of data points on the opposite side of the breakpoint so that the mean reading remains unchanged but the difference in adjustments equals the total adjustment. For example, the total adjustment for January is 1.21 °C, but there are 12 January data readings before November 1986, and 25 after. So the January readings before November 1986 are reduced by 0.82 °C ( = 1.21x25÷37), with the January readings after November 1986 being increased by 0.39 °C ( = 1.21x12÷37). The mean adjustment is therefore zero but the difference between the two is 0.39 - (-0.82) = 1.21 °C.


 Fig. 10.6: Berkeley Earth adjusted anomaly for Waiouru Airstrip.


The impact of the adjustment on the anomaly data in shown above in Fig. 10.6, the outcome of which is to change the gradient of the best-fit trend line from a negative cooling of -3.24 °C per century (the green line in Fig. 10.1) to a positive warming of +0.82 °C per century (the red line in Fig. 10.6).

Variants on breakpoint adjustment are used by most of the major climate science groups that analyse the global temperature record. In the case of NOAA they are called changepoints and are identified by pair-wise comparisons of neighbouring stations. The effect of these adjustments is to reduce long temperature records to a series of short ones. In the case of NOAA, the records are reduced to a typical length of 15-20 years; for Berkeley Earth it is 13.5 years. You can see this in the New Zealand data I have analysed in the last three posts. Of the ten long records (1200+ months of data) and seventeen medium ones (400+ months of data), all had breakpoints inserted by the Berkeley Earth group, while of the thirty short records only nine did. In the long records alone, there are a total of 46 breakpoints. That is about one breakpoint every 28.6 years and does not include the record gaps and documented station moves that also effectively serve as breakpoints. So the obvious question is, are they justified?



Fig. 10.7: Scaling behaviour of the stemperature anomaly noise for Auckland with and without breakpoint adjustment.


The assumption is that breakpoint adjustments correct the data and return it to its true values and behaviour that would be the case before human error was introduced. Yet if we look at the scaling behaviour I described in the last post, this suggests that the impact may be more negative than positive.

The data in Fig. 10.7 above compares the scaling behaviour of the temperature anomaly for Auckland-Albert Park (Berkeley Earth ID - 157062) before and after the breakpoint adjustments were added. It can clearly be seen that the original data is less scattered than the adjusted data. The gradient for the original data is -0.248 while it is -0.266 for the adjusted. But it is the scatter of the data that is most striking. If we calculate the percentage error (or residual) by comparing the r.m.s. value of the difference between the y-values and the best fit line with the r.m.s. value of the y-values themselves, the results are 4.8% for the original data and 8.1% for the adjusted data.



Fig. 10.8: Scaling behaviour of the temperature anomaly noise for Wellington with and without breakpoint adjustment.

A similar pattern is seen for Wellington-Kelburn (Berkeley Earth ID - 18625) above. The gradient of the best fit line is -0.235 for the original data and -0.239 for the adjusted data. While the gradients are almost identical the residuals are again different, with the adjusted data having a residual (3.3%) that is more than double that of the original data (1.6%).



Fig. 10.9: Scaling behaviour of the temperature anomaly noise for Christchurch with and without breakpoint adjustment.

Finally, if we look at the data above (Fig. 10.9) from the station at Christchurch AP-Harewood (Berkeley Earth ID - 157045), similar differences are seen with the gradient of the best fit to the original data being -0.275 while it is -0.382 for the adjusted data. Here, though, the adjusted data has a residual (20.3%) that is more than seven times that of the original data (2.6%), an astonishing disparity.




Fig. 10.10: Scaling behaviour of the mean temperature anomaly noise for all ten long stations in New Zealand.

It is, though, not just individual stations where this disparity between the scaling behaviour of the original and adjusted data is seen. It is seen in the regional trends as well. So in Fig. 10.10 the trend data from Fig. 7.6 in Post 7 is analysed. This data represents the mean anomaly of all ten long stations in New Zealand. Here the scaling follows a best fit line with a slope of -0.219 and a residual of 1.4%, while the breakpoint adjusted data for all ten long stations (see Fig. 7.8 in Post 7) has a scaling behaviour with a best fit line of slope -0.198 and a residual of 3.2% (see Fig. 10.11 below). So again the residual for the adjusted data is more than double that of the original data.



Fig. 10.11: Scaling behaviour of the mean temperature anomaly noise for all ten long stations in New Zealand after breakpoint adjustment.

What I think this shows is that breakpoints are unsubtle in their impact. The claims that they are surgical in their precision and benign with regard to the general temperature trend are dubious at the very least. At best they may be seen as blunt statistical interventions that correct some of the worst measurement errors, and at worst as being a greater corrupting influence on the data than the errors that they were intended to eliminate.


Saturday, May 30, 2020

9. Fooled by randomness

Is global warming real? That is probably a justifiable question given what I revealed in the last post about breakpoint alignment. But what I am going to demonstrate here and over the next two or three posts should also make you question everything you think you know about climate change. The first topic I am going to explore is a concept that most physicists and mathematicians are all too familiar with, but which appears to be totally off the radar of climate scientists: chaos theory and fractal geometry.


Fig. 9.1:  Record 1.


First a test. Look at the dataset above (Fig. 9.1) and the one below (Fig. 9.2). Can you tell which one is a real set of temperature data and which one is fake?


Fig. 9.2:  Record 2.


Okay, so actually it was a trick question because they are both real sets of data. In fact they are both from the same set of station data, and they are partially from the same time period as well, but there is clearly a difference. The difference is that the data in Fig. 9.1 above is only a small part of the actual temperature record but the data from Fig. 9.2 is from the entire record. The data in Fig. 9.1 is taken from the Christchurch station (Berkeley Earth ID - 157045) and is monthly data for the period 1974 - 1987. The data in Fig. 9.2 is from the same record but for the time interval 1864 - 2013: it has also been smoothed with a 12 month moving average. Yet they look the very similar in terms of the frequency and height of their fluctuations - why? Well, what you are seeing here is an example of self-similarity or fractal behaviour. The temperature record for Christchurch is a one-dimensional fractal, and so for that matter is every other temperature record.

Self-similarity is common in nature. You see it everywhere from fern leaves and cauliflowers to clouds and snowflakes. It is observed when you magnify some objects and look at them in greater detail, only to find, to your surprise, that the detail looks just like a smaller version of the original object. This is known as self-similarity: the object looks like itself but in microcosm. It is also an example of scaling behaviour. There is usually a fixed size ratio between the original and the smaller copies from which it is made.

In order to make the smoothed data in Fig. 9.2 look similar to the original data in Fig. 9.1 two scaling adjustments were made. First the time scale on the horizontal axis in Fig. 9.2 was shrunk by a factor of twelve. This is to compensate for the smoothing process which effectively combines twelve points into one. The second was to scale up the temperature axis in Fig. 9.2 by a factor 12 0.275. The reason for the power of 0.275 will become apparent shortly, but it is important as it has profound implications for the noise level we see in temperature records over long time periods (i.e. centuries).

To demonstrate the scaling behaviour of the temperature record we shall do the following. First we smooth the data with a moving average of length say two points and then calculate the standard deviation of the smoothed data. Then we repeat this for the original data, but with a different number of data points in the moving average and again calculate the standard deviation of the new smoothed data. After doing this for six or seven different moving averages we plot a graph of the logarithm of the standard deviation versus log(N) where N is the number of points used each time for the moving average. The result is shown below in Fig. 9.3.


Fig. 9.3:  Plot of the standard deviation of the smoothed  anomaly data against the smoothing interval N for temperature data from Christchurch (1864-2013).


The important feature of the graph in Fig. 9.3 is that the data lies on an almost perfect straight line of slope -0.275 (remember that number)? I have to confess that even I was shocked by how good the fitting was when I first saw it, particularly given how imperfect temperature data is supposed to be. What this graph is illustrating is that as we smooth the data by a factor N, the noise level is reducing by a factor N-0.275. But is this reproducible for other data? Well the answer appears to be, yes.


Fig. 9.4:  Plot of the standard deviation of the smoothed  anomaly data against the smoothing interval N for temperature data from Auckland (1853-2013).


The graph above (Fig. 9.4) shows the same scaling behaviour for the station at Auckland (Berkeley Earth ID = 157062) while the one below (Fig. 9.5) illustrates it for the station at Wellington (Berkeley Earth ID = 18625). The gradients of the best fit lines (i.e. the power law index in each case) are -0.248 and -0.235 respectively. This suggests that the real value is probably about -0.25.


Fig. 9.5:  Plot of the standard deviation of the smoothed  anomaly data against the smoothing interval N for temperature data from Wellington (1863-2005).


But it is the implications of this that are profound. Because the data is such a perfect fit in all three cases, we can extrapolate to longer smoothing operations such as one hundred years. That corresponds to a scaling term of 1200 (because it is equal to 1200 months and thus is 1200 greater in period than the original data) and a noise reduction of 1200 0.25 = 5.89. In other words, the noise level on the underlying one hundred year moving average is expected to be about six times less than for the monthly data. This sounds like a lot but the monthly data for Christchurch has a noise range of up to 5 °C (see Fig. 9.6 below), so this implies that the noise range on a 100 year trend will still be almost 1 °C. Now if that doesn’t grab your attention, I have to wonder what will? Because it implies that the anthropogenic global warming (AGW) that climate scientists think they are measuring is probably all just low frequency noise resulting from the random fluctuations of a chaotic non-linear system.


Fig. 9.6:  The temperature anomaly data from Christchurch (1864-2013) plus a 5-year smoothing average.


What we are seeing here is a manifestation of the butterfly effect which, put simply, says that there is no immediate causal link between some current phenomena such as the temperature fluctuations we see today and current global events. This is because the fluctuations are actually the result of dynamic effects that played out long ago but which are only now becoming visible.


Fig. 9.7:  Typical mean station temperatures for each decade over time.


To illustrate the potential of this scaling behaviour further we can use it to make other predictions. Because the temperature record exhibits self-similarity on all timescales, it must do so for long timescales as well, such as centuries. So we can predict what the average temperature over hundreds of years might look like (qualitatively but not precisely) just by taking the monthly data in Fig. 9.6, expanding the time axis by a factor of 120 and shrinking the amplitude of the fluctuations by a factor of 120 0.25 = 3.310. The result is shown in Fig. 9.7 above. Because of the scaling by a factor of 120, each monthly data point in Fig. 9.6 becomes a decade in Fig. 9.7. The data in Fig. 9.7 thus indicates that the average temperature for each decade can typically fluctuate by about ±0.5 °C or more over the course of time.


Fig. 9.8:  Typical mean station temperatures over 100 years over time.


Then, if we smooth the data in Fig. 9.7, we can determine the typical fluctuations over even longer timescales. So, smoothing with a ten point moving average will yield the changes in mean temperature for 100 year intervals as shown in the graph above (Fig. 9.8). This again shows large fluctuations (up to 0.5 °C) over large time intervals. But what we are really interested in from a practical viewpoint is the range of possible fluctuations over 100 years as this corresponds to the timeframe most quoted by climate scientists.

To examine this we can subtract from the value at current time t the equivalent value from one hundred years previous, i.e. ∆T = T(t) - T(t-100).


Fig. 9.9:  Typical change in the 100-year mean temperature for a time difference of 100 years.


So, as an example we may wish to look at the change in mean temperature from different epochs, say from one century to the next. Well the data in Fig. 9.9 shows just that. Each data point represents difference between the mean temperature over a hundred years at that point in time with the same value for a hundred years previous. Despite the large averaging periods we still see significant temperature changes of ± 0.25 °C or more. However, if we compare decades in different centuries it is even more dramatic.

For example, Fig. 9.10 below predicts the range of changes in the average decadal temperatures from one century to the next, in other words, the difference between the 10-year mean temperature at a given time t and the equivalent decadal mean for a time one hundred years previous. What Fig. 9.10 indicates is that there is a high probability that the mean temperature in the 1990s could be 0.5 °C higher or lower that the mean temperature in the 1890s, and this is just as a consequence of low frequency noise.


Fig. 9.10:  Typical change in mean decadal temperature for a time difference of 100 years.


So why have climate scientists not realized all this? Maybe it's because their cadre comprise more geography graduates and marine biologists than people with PhDs in quantum physics. But perhaps it is also due to the unique behaviour of the noise power spectrum.

If the noise in the temperature record behaved like white noise it would have a power spectrum that is independent of frequency, ω. If we define P(ω) to be the total power in the noise below a frequency, ω, then the power spectrum is the differential of P(ω). For white noise this is expected to be constant across all frequencies up to a cutoff frequency ωo.


(9.1)

This in turn means that P(ω) has the following linear form up to the cutoff frequency ωo.

P(ω) = aω

(9.2)

where a is a constant. The cutoff frequency is the maximum frequency in the Fourier spectrum of the data and is set by the inverse of the temporal spacing of the data points. If the data points are closer together then the cutoff frequency will be higher. Graphically P(ω) looks like the plot shown below in Fig. 9.11, a continuous horizontal line up to the cutoff frequency ωo.


Fig. 9.11: The frequency dependent power function P(ω) for white noise.


The effect of smoothing with a moving average of N points is to effectively reduce the cutoff frequency by a factor of N because you are merging N points into one. And because the noise power is proportional to the noise intensity, which is proportional to the square of the noise amplitude, this means that the noise amplitude (as well as the standard deviation of the noise) will reduce by a factor equal to √N when you smooth by a factor of N.

For a 100-year smoothing the scaling factor compared to a monthly average is 1200, and so the noise will therefore reduce by a factor of 1200 0.5 = 34.64 . That means the temperature fluctuations will be typically less than 0.1 °C. This is probably why climate scientists believe that the long term noise will always be smoothed or averaged out, and therefore why any features that remain in the temperature trend must be "real". The problem is, this does not appear to be true.

Instead the standard deviation varies as N -0.25. So the intensity of the noise varies as N -0.5 and P(ω) will increase as √N. It therefore follows that the power spectrum is not independent of frequency as is the case for white noise, but instead varies with frequency as


(9.3)

and P(ω) will look like the curve shown in Fig. 9.12 below.


Fig. 9.12:  The frequency dependent power function P(ω) for temperature data.


The net result is that the random fluctuations in temperature seen over timescales of 100 years or more are up to six times greater in magnitude than most climate scientists probably think they will be. So the clear conclusions is this: most of what you see in the smoothed and averaged temperature data is noise not systemic change (i.e. warming). Except, unfortunately, most people tend to see what they want to see.


Thursday, May 28, 2020

8. New Zealand - trend due to long and medium stations

In the last post I looked at the long weather station records from New Zealand (i.e. those with over 1200 months of data) and showed how they could be combined to give a temperature trend for climate change using the theory outlined in Post 5. The result was a trend line that looked nothing like the ones Berkeley Earth and other climate science groups claim to have uncovered (compare Fig. 7.6 with Fig. 7.2 in Post 7).

Some may argue that part of the reason for this difference lies in the number of stations used (only ten), or the lack of data from the much larger group of seventeen medium length stations (with 401-1200 months of data each) that could have been utilized. To see if data from these stations does make a significant difference I have repeated the analysis process from the previous post, but with the seventeen medium length stations included alongside the original ten long stations.

The first problem we have, though, is that most of these medium stations have data that only stretches back in time to about 1960 and most of their data is post 1970. That means we cannot use the 1961-1990 period to determine the monthly reference temperatures (MRTs) that are needed to remove the seasonal variations (for an explanation of the MRT see Post 4). So instead I have chosen to use the period 1981-2000, which while not as long, is a period for which all 27 stations have at least 80% data coverage. After calculating the anomalies for each station, as before, the trend was determined by using the averaging method represented by Eq. 5.11 in Post 5. This gives the trend profile shown in Fig. 8.1 below.


Fig. 8.1: Average warming trend of for long and medium stations in New Zealand.


This trend is virtually identical to the one presented in the last post (see Fig. 7.6), which suggests that the additional data makes little difference to the slope, although there is a slight reduction in the noise level post-1970. The best fit line indicates a warming trend of only 0.27 ± 0.04 °C per century, again almost identical to that from the long stations alone (0.29 ± 0.04 °C per century). This also suggests that the choice of time period for the MRT has little effect as well. Yet if we look at the Berkeley Earth adjusted data we get a different picture.


Fig. 8.2: Average warming trend of for long and medium stations in New Zealand using Berkeley Earth adjusted data.


If we use Berkeley Earth adjusted data for both long and medium stations to determine the local warming trend for New Zealand we get the data in Fig. 8.2 above. The best fit line shows a significant +0.60 ± 0.04 °C per century upward slope.


Fig. 8.3: Smoothed warming trends of for long and medium stations in New Zealand using Berkeley Earth adjusted data.


This is even more evident if we look at the 1-year and 5-year moving averages (see Fig. 8.3 above). But if we look at the real data in Fig. 8.1 and plot the 1-year and 5-year moving averages (see Fig. 8.4 below) we again get a different trend entirely from that in Fig. 8.3.


Fig. 8.4: Smoothed warming trends of for long and medium stations in New Zealand using original data.


The question is, why are the data in Fig. 8.3 and Fig. 8.5 so different? The answer is breakpoint alignment.


Fig. 8.5: Difference (Berkeley Earth adjusted data - original data) in the smoothed warming trends for long and medium stations in New Zealand.



If we subtract the data in Fig. 8.4 from that in Fig. 8.3 we get the curves in Fig. 8.5 above. This data is the result of corrections that have been imparted into the data by Berkeley Earth, via a technique called breakpoint alignment or breakpoint adjustment, supposedly to correct for systematic data errors, such as those described above: station moves, changes in instruments and changes to the time of day of measurement. These adjustment are in effect attempting to identify systematic errors between temperature records or within temperature records, and compensate for them.

Yet these changes by Berkeley Earth are clearly not neutral. They do not merely iron out undulations in order to reveal the trend more clearly, they actually add to the trend. In this case these adjustments add 0.33 °C per century to the overall trend. That is more than the original trend in Fig. 8.1, and is why the gradient of the Berkeley Earth best fit line in Fig. 8.3 and Fig. 8.4 is more than double that for the original data in Fig. 8.1 and Fig. 8.4. This is why there is so much scepticism about global warming. Many people outside the climate science community do not trust the data or the analysis. And this is not just a problem with Berkeley Earth. All the major groups do it; it is just that Berkeley Earth are more transparent about it.

What the analysis here has shown is that having more data for recent epochs does not really improve the quality of data in the overall trend, or the confidence level of the conclusions that can be derived from that data. It is more important to have ten long temperature records than twenty (or even a hundred) short ones. Yet herein lies a paradox. The longer the temperature record, the less its quality is trusted by the climate scientists, and the more they seek to fragment if into shorter records via the use of breakpoints. We will see this more clearly later when I look in more detail at the Horlicks that is breakpoint alignment.

 

Addendum

Close inspection of Fig. 8.1 suggests that the spread of the data is greater before 1940 than it is thereafter. This is a consequence of the increased number of datasets that are used to calculate the trend in the latter half of the 20th century compared to the first half and the 19th century. The number of datasets involved in constructing the average temperature trend shown in Fig. 8.1 is indicated below in Fig. 8.6.

 

Fig. 8.6: The number of sets of station data included each month in the temperature trend for New Zealand.


In summary, there were between 5 and 11 datasets used to calculate the trend between 1870 and 1940, and up to 25 thereafter. Given that the standard deviation of the anomalies in most individual temperature records is approximately 1.0 °C, this implies that the standard deviation of the monthly data in the temperature trend should be about 60% greater before 1940 compared to 1980 and later. It also means the uncertainty in the mean trend will increase slightly as you go back in time, from about ±0.2 °C after 1980, to about ±0.35 °C before 1940. Nevertheless, in my view, this indicates that the temperature trend from 1860-1940 is almost as reliable as that for much later years (1960-2010) despite the reduced amount of data. For while the uncertainty before 1940 may be almost double its post 1960 value, it is still significantly less than the natural variation seen in the 5-year moving average temperature.

Tuesday, May 26, 2020

7. New Zealand - trend due to long stations SLIGHT WARMING

As I pointed out last time, New Zealand has some very good temperature data, or at least good in comparison to most other countries in the Southern Hemisphere. It has over 50 sets of station data, of which ten have over 1200 months of data, and about a dozen have more than 400 months of data, yet even that is not enough.

The stations with over 1200 months of data I shall denote as long stations, those with over 400 months I denote as medium stations. Their geographical locations are shown in Fig. 6.1 in the last post. To start with I shall analyse the long station records, (a) because there are a large number of them, and (b) because they are the records that will show the most discernible trend, yet even with many of these stations the trend can be ambiguous.

If we start with the individual stations, a typical temperature anomaly, i.e. the mean temperature each month minus the monthly reference temperature (MRT), is shown below together with its best fit line and a 5-year moving average. This is the data for Auckland (Albert Park). It dates back to 1853 and it is one of the oldest, longest and most complete records in New Zealand.


Fig. 7.1: Temperature anomaly for Albert Park, Auckland (1853-2013). The best fit line has a positive gradient of 0.18 ± 0.05 °C per century.


Two things are immediately apparent. The first is that the noise level on the anomaly data makes it difficult to discern the overall trend in the data even though the seasonal variation (the MRT) has been removed, and remember, the data represents the average temperature over a whole month not just a typical day. In fact the standard deviation of the data in Fig. 7.1 is 0.95 °C and still almost 25% of data lie outside this range.

The second is that, while the trend does become more discernible if a 5-year moving average is performed (the yellow curve), the trend is still not uniform, nor does it represent a single continuous rise or fall. In fact the temperature in 2013 appears to be no higher than in the 1850s and there is a clear sign of a longer term (150 year) oscillation.

Also shown in Fig. 7.1 is the best fit to the anomaly data. While this fit line has a positive slope of 0.18±0.05 °C per century it needs to be acknowledged that this is not due to a warming trend, but is purely as a consequence of the 150 year oscillation. If you look back at Fig. 4.7 you will recall that the best fit to a full sine wave always has a positive slope. What is clear, however, is that the trend shown in Fig. 7.1 bears no resemblance to the one expected for the region, as illustrated below and on the Berkeley Earth site, either in shape or magnitude.


Fig. 7.2: Berkeley Earth warming trend for New Zealand (1853-2013).


So what about the other nine stations? Well the next three longest are shown in Fig. 7.3. These are the records from Dunedin (ID 18603), Christchurch (157045) and Wellington (18625).


Fig. 7.3: Smoothed temperature anomalies and best fit lines for three stations.


Here the results are even more contradictory. In Fig. 7.3 each set of data is plotted as in the form of the 5-year moving average together with the best fit line. The legend on the graph indicates the slope of each best fit line in degrees Celsius per century, therefore the data for Dunedin-Musselburgh (Berkeley Earth ID = 18603) has a warming trend of 0.64°C per century, while that for Wellington-Kelburn (Berkeley Earth ID = 18625) has a cooling trend of -0.25 °C per century. That for Christchurch (Berkeley Earth ID = 157045) is slightly warming. Of the three datasets, those for Dunedin and Christchurch do appear to offer a passing resemblance to the Berkeley Earth trend in Fig. 7.2, although neither exhibits a temperature gradient as high as that in Fig. 7.2, while Christchurch and Wellington both appear to have a 150 year oscillation that is responsible for most of the slope in the best fit.


Fig. 7.4: Smoothed temperature anomalies and best fit lines for three stations.


If we consider the three next longest records we observe a common theme (see Fig. 7.4). After 1920 there is a distinct warming trend (as was seen in most of the previous data above) but before 1920 much of the important data is missing. Given what we have seen in previous data, it is reasonable to suppose that this data would be of a higher temperature than that which is present, and therefore that the warming trends indicated by the best fit lines in Fig. 7.4 are over-estimates.


Fig. 7.5: Smoothed temperature anomalies and best fit lines for three stations.


For the remaining stations the lack of early data becomes an even bigger problem (see Fig. 7.5), and while a trend post-1940 can be discerned, the early data is highly fragmented. Nevertheless, the last two datasets in Fig. 7.5 both have a peak at about 1890 which is clearly evident on all the data in Fig. 7.1 and Fig. 7.3, and which is at a comparable height relative to the data around 1960 in each case. That suggests that most of this data is consistent and sound.

As I pointed out in post 5, if we wish to derive a regional trend such as that shown in Fig. 7.2, all we need to do is average the anomalies (provided all the data has been processed in a consistent manner and is reliable). When we do this for the ten stations described above, we get the dataset illustrated below in Fig. 7.6.


Fig. 7.6: The warming trend for New Zealand (1853-2013) based on the averaging of long station anomalies. The best fit line has a positive gradient of 0.29 ± 0.04 °C per century.


The mean temperature change or anomaly plotted in Fig. 7.6 has a slight upward trend and its best fit line has a gradient of 0.29±0.04 °C per century. However, this variation is clearly in two parts. From 1860 to 1940 the trend is clearly downwards, while from 1940 to 2000 it is upwards.

What is also clear is that the temperature trends in Fig. 7.6 are very similar to those shown for Auckland in Fig. 7.1 (and actually most of the other individual station datasets) but very dissimilar to that advanced by Berkeley Earth in Fig. 7.2. The question is why?

Well, there are two reasons, and they are both to do with how the temperature data is handled and processed. The graphs I have presented here in Fig. 7.1 and Figs. 7.3-7.6 all use the original temperature data as is. I first calculate the MRT following the method outlined here and subtract it from the original data to obtain the anomaly without the problem of the large seasonal variations. That is a standard procedure that all climate science groups should do. The time base I used for the calculation of the MRT was 1961-1990 for reasons outlined here. Again, this time frame appears to be fairly standard. However, despite this my MRT values differ slightly from those of Berkeley Earth. The reason for this I have not discovered yet, but it may be that old favourite of climate scientists, homogenization, or it may be a different choice of time frame. Whatever the reason, fortunately it makes little difference. If you sum the Berkeley Earth anomalies the result is very similar as shown in below in Fig. 7.7.


Fig. 7.7: The warming trend for New Zealand (1853-2013) based on the averaging of the Berkeley Earth anomalies for long stations.


That, however, is where the similarity ends because Berkeley Earth then play their joker: breakpoint alignment. This is a mathematical device that is supposed to account for imperfections in the data due to human measurement error, changes in instruments, location moves and changes in the time of day when the measurements were made. How this is implemented I will discuss at a later date. What is important here is the net result and that is shown in Fig. 7.8.


Fig. 7.8: The warming trend for New Zealand (1853-2013) based on the averaging of the adjusted Berkeley Earth anomalies for long stations after breakpoint alignment.


The differences between Fig. 7.6 (or Fig. 7.7) and Fig. 7.8 are subtle but clear when you finally see it. In Fig. 7.6 there is a discontinuity or kink in the gradient of the general trend around 1940 and the slope is shallow. In Fig. 7.8 the slope is more uniform and steeper. The gradient of the best fit line has now more than doubled to 0.60 ± 0.04 °C per century and the temperature rise from 1860 to 2010 has gone from virtually zero in Fig. 7.6 to an impressive 0.9 °C. Now, if we smooth the data in Fig. 7.8 using a 12-month and a 10-year moving average, we get the curves shown in Fig. 7.9 below which look very similar to the Berkeley Earth summary trend in Fig. 7.2.


Fig. 7.9: The smoothed warming trend for New Zealand (1853-2013) based on the averaging of the adjusted Berkeley Earth anomalies for long stations after breakpoint alignment.



This validates our averaging process, but the problem is that Fig. 7.9 bears very little resemblance to the original data in Fig. 7.6. This difference cannot be due to a difference in the averaging process, otherwise Fig. 7.9 would not resemble Fig. 7.2 so closely. That only leaves homogenization and breakpoint adjustments as the possible causes of the difference. The equally worrying question is, why are these adjustments being made? Those reasons may become apparent as we look at more of the global temperature data.


Sunday, May 24, 2020

6. New Zealand station profile

New Zealand is probably most famous for two things: sheep and rugby (not necessarily in that order). I’m not sure what impact rugby has had on global warming, but sheep are are not exactly carbon-neutral. I shall leave further discussion regarding the methane problem until another day though.

New Zealand is, however, surprising in one sense: despite being a small country with an even smaller population, it has the second highest number of long temperature records in the Southern Hemisphere. Only Australia has more station records with more than 1200 monthly measurements each (1200 being the equivalent of more than 100 years of data). New Zealand therefore seems like a good place to start analysing regional temperature trends.

According to Berkeley Earth, New Zealand has about 64 station records (it may be slightly more or less depending on whether you include some near to Antartica or some South Pacific Islands). Of these, ten have more than 1200 months of temperature data stretching back to the 19th Century, including two that date back to January 1853. I shall characterize these as long stations due to the length of their records. In addition there are a further 27 stations with more than 240 months of data which could be characterized as medium length stations. This includes a further dozen or so stations that contain data covering most of the period from 1973 to 2013.

In my previous post I explained how temperature data can be processed into a usable form comprising the temperature anomaly, and how these anomalies can be combined to produce a global warming trend for the region (see Eq. 5.11 here). This process involves combining multiple temperature records from the same country or region into a single numerical series by adding the anomaly data from the same month in each record and taking the average. This new average should, in theory, have less noise that the individual records from which it is constructed because the averaging process should lead to a regression towards the mean. What is left should be a general trend curve that consists of the signal of long-term climate change for that region together with a depreciated noise component. As a starting point we shall in the next post look at combining the ten longest data sets for New Zealand and seeing how the warming trend it produces compares with the trend as advertised by climate scientists.

As I noted last time, we need to be careful in regard to how the different records are combined, and in particular, to consider two main issues. The first is the evenness of the distribution of the stations across the region in question. If stations are too close together they will merely reproduce each other’s data and render one or more of them redundant. Ideally they should be evenly spaced, otherwise they should be weighted by area (see Eq. 5.15 here).




Fig. 6.1: Geographical distribution of long (1200+ months), medium (400+ months) and short (240+ months) temperature records in New Zealand.


If we look at the spatial distribution of the long stations in New Zealand (see Fig. 6.1.), we see that they are indeed distributed very evenly across the country. This means that weighting coefficients are unnecessary and the local warming trend can be approximated to high precision merely by adding the anomalies from each station record.

The second issue regards the construction of the temperature anomalies themselves (how these anomalies are derived has been discussed here). These anomalies are the amount by which the average temperature for a particular month in a given year has deviated from the expected long-term value for that month. In other words, by how much does the average temperature for this month (May 2020) differ from the average temperature for all months of May over the last 30 years or so? Central to this derivation is the construction of a set of monthly average temperatures, which involves finding the mean temperature for each of the 12 months over a pre-defining time interval of about 30 years as outlined here and in Fig. 4.2 here. I call these averages the monthly reference temperatures (MRTs) because they are temperatures against which the actual data is compared in order to determine the monthly change in temperature. These temperature changes or anomalies are in essence a series of random fluctuations about the mean value, but they may also exhibit an underlying trend over time. It is this trend that climate scientists are seeking to identify and measure.

This immediately raises an important question: over what period should the reference temperature for each month be measured? Most climate science groups seem to favour a thirty year period from 1961-1990. It appears that this is chosen because it tends to correspond to a period with a high number of active stations, and this is certainly true for New Zealand. As the Berkeley Earth graph in Fig. 6.2 below shows, the number of active stations in New Zealand has risen over time, peaking at over 30 in the last few decades. However, when it comes to finding the optimum period for the MRT calculation, relying on station population frequencies is not always the best illustrator.


 Fig. 6.2: New Zealand stations used in the Berkeley Earth average.

What we really require is a time period which allows us to incorporate the maximum number of data points into our analysis. This can be achieved, not by summing the number of active stations each month, but instead by summing the total number of data points that each of the stations present in that month possesses. Such a graph of the sum of station frequency x data length versus time is shown below in Fig. 6.3.


Fig. 6.3: Data frequency over time.


Fig. 6.3 shows more clearly that the period 1970-2000 is the Goldilocks zone for calculating the MRT. Choosing this time period for the MRT not only allows us to incorporate a large number of stations, but it also means we will have a large number of data points per temperature record, and hence a longer trend.  Nevertheless, not all temperature records will have enough data in this region, and some useful data could still be lost. So why not choose a different period, say 10 years, or a longer period, say 50 years or 100 years that could access the lost data? And how much effect would this choice make on the overall warming trend?

The problem is that there are various competing drivers at play here. One is the need to have the longest temperature record, as that will yield the most detectable temperature trend. But measurement accuracy also depends on having the highest number of stations in the calculation, and in having an accurate determination of the MRT for each. And of course, ideally the same time-frame should be used for all the different temperature records that are to be combined in order to maintain data consistency and accuracy. Unfortunately, this is not always possible as most temperature records tend to have different time-frames. When faced with the need to compromise, it is generally best to try different options and seek the optimal one.