Academic Research: Basic Referencing 101 -- A case study of Bernstein's Beveridge, Zhu & Pun (2016)

Image from virtualhighschool

"If something is difficult but important and necessary for the future, do it now and don't do it later. Down the road, nobody will be sorry that they took early action, most people are sorry that they take late action." -- Elon Musk

Everybody knows that the future is renewable energy because it is sustainable energy production, and everybody knows that the future has no internal combustion engine (ICE) vehicles because what we need is sustainable energy consumption. Then why are there still people who want to protect the oil industry? Do they even care about our children, grandchildren and future generations?

***************************************

And I'm not even going to talk about distortion of data and deceptions in argument today. If you are interested in those, please read Letters to the editor, April 26 2016 on SCMP written by Charged Hong Kong chairman, Mark Webb-Johnson.

I'm just going to talk about Referencing 101 for research students.

As a lecturer/tutor at a university, I always use the first lesson of any course to talk about the importance of referencing and citation, no matter it is a Bachelor's or a Master's degree course. Cite clearly and only the creditable and authoritative. It's fine to cite yourself, but only if that has been published and peer reviewed. Most important of all, be aware of what you make reference to at all time, read thoroughly what you cite, ask where the data come from, how it was analysed and how it comes to its conclusion. Why? Because if what you cite is based on outdated data, faulty method of analysis or conclusion, and if you develop your study on it, then your study is pretty much worthless. My students are pretty awesome, they often understand the logic behind, but sometimes, even the professional researchers can go wrong with the basics.

I refer to the Bernstein report titled "Bernstein Energy: Oops - Hong Kong (and China) EV Subsidies Are Leading to More CO2 Emissions, Not Less" by Neil Beveridge, Ph.D., Robin Zhu and Tracy Pun (2016).

Before we even go into the data, let's talk about general referencing issues with this report Beveridge, Zhu & Pun (2016, p.4).

Figure from Beveridge, Zhu & Pun (2016) Exhibit 4

Now, if you pay attention to the 'Source' in the above Exhibit 4, you will see it comes from "Bernstein analysis and estimates". The problem lies in what analysis and estimates they have done. When I do my analysis and estimates, I will publish it, every single step and detail of it, and I will cite myself by offering every bit of detail to readers such that they can find it, and I'll make sure they are publicly available. Now if I want to look for the analysis and estimates performed by Bernstein, where can I find it? Who can I look for? Now, in "Fuel Efficiency of Electric Vehicles (2) 18 kWh per 100km, it is made reference to "Bernstein Technology Team", which means basically, it is either not published or it is published but readers won't be able to find it due to the lack of source in this 'Source'.

Then there are 3 items which are not even cited, they are "CO2 produced per 100km", "Total Lifetime Distance", Total Fuel Related C)2 Emissions". Already, I am getting a feeling of 'Don't ask, just trust me'.

Under "Battery Production CO2 Emissions (3)   5.6 tonnes CO2", we finally get a bit of reference, which goes to McManus (2012) "Environmental Impacts of batteries for low carbon technologies compared" published by Applied Energy, so we take a look at this reference, turns out, it is not even the full report but just a newsletter summary and it is not published by Applied Energy, but by European Commission.

Bernstein Energy: Oops - Bernstein Research Team cited Wrong Reference, Not Correct (Recreativity of their original report title).

So, I went the extra mile and try to find where that McManus (2012) really is, turns out the title is different when it was published, instead it is "Environmental consequences of the use of batteries in low carbon systems: The impact of battery production". The name is different in the journal version, changed from "Impact" to "consequence". Now if I were marking this paper, I would have stopped here and sent it back to my student and asked them to re-do it before resubmission. But, because I'm a nice teacher, I let-it-go for once.

Before we go into looking for the number from McManus, I need to know what number I should be looking for, so I looked at p.4 of the report, an excerpt is shown and highlighted below.

******************************************************************

Excerpt from Beveridge, Zhu & Pun (2016, p.4) 

******************************************************************

So despite lots of assumptions based on no reference other than Bernstein Technology Team ('Don't ask, just trust me') and a comparison of a car which hasn't even in the production stage and has not released details of its battery size, it is this line -- "a typical lithium ion battery produces 12.5kg of CO2 for each kg of battery produced" that I should be looking for. So, I went into the FULL report, which all students should, to search for the quoted figure. And here is Table 2 in McManus (2012, p.293).

McManus (2012, p.293)

Before we go deeper into asking McManus where he got his number from, take a look at the unit "kg CO2 eq". CO2e or carbon dioxide equivalent is not exactly the same as carbon dioxide because the former includes other greenhouse gases such as methane, perfluorocarbons, and nitrous oxide. Okay, but I'll let-it-go again.

Looking at these numbers from McManus (2012, p.293), if I were going to use them, I would first ask myself, how did McManus arrive at these numbers? Did he make reference to each/some of these numbers? No. Could he have calculated these numbers from some raw data? Maybe, because I can't find the exact match for these numbers even going all 31 references cited by McManus (2012). But where do the raw data come from? No idea. What's the formula used in calculating each one of the above item? No idea.

Hey, Locky, maybe you don't know how to calculate because you are not in this field! True, I admit, but back to the basics of referencing, it is clarity that matters most, because any papers published in good journals are supposed to be comprehensible by newbies like me, if I have to go and ask all these questions and do my own calculation of each of the figures, then something has gone very wrong with this paper. Or again, should I 'Don't ask, just trust me'?

Okay, let's assume McManus's figures are brilliant and very much reliable, but how can he get Climate change (kg Co2 eq) right? Because emission of CO2 eq differs from country to country, grid to grid. What fuel mix is used in calculating this? 99% coal or 55% coal for Hong Kong's average? Some might say, hey, what does it matter? They are taking a constant across all battery types. True, but doesn't that affect the Fossil fuel depletion (kg Oil eq) because that is exactly a dependent variable of Climate change (kg CO2 eq)? Okay, say I let it go again! Say I trust McManus (2012) 100%, without knowing the fuel mix used in this paper, do you think Bernstein's calculation will be problematic in their Exhibit 4? Obviously Beveridge, Zhu & Pun (2016, p.4) haven't considered this, otherwise they wouldn't have just done that simple calculation on p.4.

One last point I have to make in terms of good referencing especially when dealing with technology-related numbers: technology advances quickly, numbers go down in reliability with time. For Beveridge, Zhu & Pun (2016) to be citing McManus (2012) who used NMP data from Zackrisson, Avellan, Orlenius (2010) who used numbers from Rydh & Sanden (2005a, 2005b) who used numbers from Almemark, Granath & Setterwall (1999) whose report is available only in Swedish and can no longer be found on the internet.

Now, do you see a problem here?


All in all, when writing an article, one needs to realise the reliability of your calculation if you have based it on certain references and assumptions. For the former, if you don't know how the author arrives at that number and you used it, the reliability of your calculation has just dropped at least 50% and it can go all the way to 0%. For the latter, if your assumption has about 90% reliability, but you make 4 of these in the same calculation, then it only has 66% reliability (=90% x 90% x 90% x 90%). Personally, anything less than 80% reliability is unreliable. Don't agree? Ask people who complain about the weather forecast in Hong Kong.


PS: BTW, Zackrisson, Avellan, Orlenius (2010) actually realised the problem with fuel mix but McManus (2012) didn't. That being said, their paper is actually on plug-in hybrid, and they also took numbers from Saft (2008), a battery making company that does NOT supply batteries to the company Bernstein's report is comparing with, Tesla.

Resources:
Elon Musk talks sustainability with Norway Prime Minister (2016.4.21) @YouTube
https://www.youtube.com/watch?v=0pqTHBFDjVc

Letters to the editor, April 26, 2016 @SCMP
http://www.scmp.com/comment/letters/article/1938603/letters-editor-april-26-2016

Beveridge, Zhu & Pun (2016) "Bernstein Energy: Oops - Hong Kong (and China) EV Subsidies Are Leading to More CO2 Emissions, Not Less" published by Bernstein, not freely available to public.

Environmental impacts of batteries for low carbon technologies compared +European Commission
http://ec.europa.eu/environment/integration/research/newsalert/pdf/303na1_en.pdf

McManus (2012) Environmental consequences of the use of batteries in low carbon systems: The impact of battery production
http://www.sciencedirect.com/science/article/pii/S0306261911008580

Zackrisson M, Avellan L, Orienius J. Life cycle assessment of lithium-ion batteries for plug-in hybrid electric vehicles – critical issues. J Cleaner Prod 2010;18:1519–29
http://www.sciencedirect.com/science/article/pii/S0959652610002167

Rydh CJ, Sanden BA. Energy analysis of batteries in photovoltaic systems. Part 1: Performance and energy requirements. Energy Conserv Manage 2005;46:1957–79.
http://www.sciencedirect.com/science/article/pii/S0196890404002407

Rydh CJ, Sanden BA. Energy analysis of batteries in photovoltaic systems. Part 2: Energy return factors and overall battery efficiencies. Energy Conserv Manage 2005;46:1980–2000.
http://www.sciencedirect.com/science/article/pii/S0196890404002419

Saft, 2008. Annual Report, Saft Batteries.
http://www.saftbatteries.com/investor-centre/financial-results/financial-reports

Almemark M, Granath J, Setterwall C. Electricity for vehicles—comparative life cycle assessment for electric and internal combustion vehicles for Swedish conditions [in Swedish]. Elforsk report 99:30, ELFORSK, Stockholm, Sweden, 1999.