#ThikingHat - Are opinion polls any good ?
Ep # 1 - Analysis of CVoter Survey Results from assembly elections
In #ThinkingHat series we ruminate, reflect and rethink .
As the elections of 5 states are around the corner, television channels and news portals are belting out daily shows based on opinion polls .
Are these opinion polls any good ? It turns out not so much .
Hang on, as we dive in to the details.
I have randomly picked opinion poll data published for 3 states, that went to polls recently.
All three opinion polls or surveys were conducted by C Voter for the states Uttar Pradesh, Uttarakhand and Karnataka.
Uttar Pradesh & Uttarakhand went to polls in 2022 and Karnataka earlier this year in 2023. All 3 states have different voter demographics, population and assembly seats. Uttar Pradesh is a very large state with 403 Assembly seats, in comparison Uttarakhand is small with only 68 Assembly seats and Karnataka is relatively moderate in size as it has 224 assembly seats.
Now why did I choose C Voter ?
First of all, C Voter conducts and publishes opinion polls regularly throughout India, it has been in the business for 30 years.
Second, C Voter data is readily available for reference through ABP’s online portal.
Last but not the least, Yashwant Deshmukh founder of CVoter would be a good sport.
Opinion polls is not an exact science, it is a field of social research that depends a lot not only on the analysts and field researchers collecting the data, but also on how respondents answer the questions that are posed to them .
Suffice to say, work of a pollster is an art, as much as it is science.
For the purpose of our analysis, we will look at Margin of Error and Confidence Interval that CVoter publishes with its opinion polls, but of-course in the Disclaimer section .
According to Cvoter’s own admission, their Margin of Error is + or - 3% at a Micro Level and + or - 5% at a Macro Level. Data published is published with 95% of Confidence Interval.
What does that mean for the viewer or reader ?
It means that the vote swing for a single party should not be greater than + or - 3 %, at an aggregate level, it should not be greater than + or - % 5 % and the seats predicted should have a difference of more than 5 %. That means , if an assembly has 100 seats, seat forecast will be accurate for at least 95 seats.
Before we move on :
If you would like to know, the nitti gritties of an opinion in more detail I would recommend watching a conversation between Yogendra Yadav and Dr. Vasu. on the art of conducting election surveys.
Yogendra Yadav is an ex-psephologist and Dr Vasu is with eeDina, a Kannada news portal that forecasted the Karnataka Election outcome exactly right.
Let’s look at Uttarakhand first. It’s a small state and small margin of error can cause huge variations. CVoter called it a close fight in its opinion polls between INC and incumbent BJP.
Results turned to be very different though. BJP received a vote share of 44.3 % and INC received a vote sahre of 37.9% . This lead to an 6.4% of actual vote share difference, as opposed to 2 percent that was initially predicted.
It gave a huge advantage to AAP, a newcomer of 13 % compared to 3.3% vote share it actually received and other received 14.5 vote share, far more than initially anticipated by 4% .
Now as you can see from the charts, at the Micro level - it was way off the mark of its + or - 3 % accuracy - in the case of AAP actual was 9.7 % below and 10.6 % more in the case of others. At a Macro Level this turned out to be 12.3 % instead of 5 % as claimed by C Voter.
How these vote shares are translated into Seats, is an area surveying agencies would know better. What we know though is CVOTER prediction were incorrect. BJP won 47 , 14 more than 33 that it had forecasted - leading to an Error % of 22.06, well beyond the limits of 95 % that it claims for itself.
I must mention that we have considered median values of the range of seats that’s usually published for its surveys .
Now, lets look at Uttar Pradesh -
How did CVoter fair in comparison to its prediction versus its claimed margin of ERRor % and Confidence Interval?
CVoter claimed BJP was ahead in the UP Assembly Elections, which it of-course was looking at the results, however the results reflected another anomaly. It gave SP+ Alliance 20 more seats than it actually won, BSP had 16 more seats and Congress 4 more than the median .
If you look at the vote percentages, these are well within the margin of error for the major players that is BJP +, SP + and BSP. However CVOter Survey could not predict the vote share accurately for INC or Others . The margin of error was greater than 4 % in each case.
At a macro level this became a difference of 6 % .
Ultimately BJP+ won 42 seats more than what was predicted , and the error in seat prediction was 10.42 %, way more than compared to 95 % of
Cvoter survey gave a vote percentage of 17 % far more than 13.29 % that was actually received by JDS, and the Seat prediction was again way of the mark as Congress won 135 Seats in comparison to 118. CVoter had predicted a Congress win with a wafer thin margin however it turned out to be a lanslide victory for congress, BJP relegated to 66 seats and JDS winning only 19 seats, 6 less than what was predicted for them.
Here is a summary of Incorrect Seat Predictions by CVOTER for Karnataka, Uttar Pradesh and Uttarakhand. In all 3 states that we studied, it’s Seats predicted by CVOTER are well beyond the 95% confidence interval mark and sufficient to create doubts about its accuracy and efficacy .
Do let us know what do you think about the episode in the comments section and what you would like to be discussed in the Thinking Hat.
Until next time.