As we move into a digital age, online surveys are growing in popularity due to a perception that they are cost and time effective. While potentially offering accurate data and the immediacy of data for analysis and feedback, online surveys are still approached with caution in China due to issues of data reliability and quality - based on experience!!As we move into a digital age, online surveys are growing in popularity due to a perception that they are cost and time effective. While potentially offering accurate data and the immediacy of data for analysis and feedback, online surveys are still approached with caution in China due to issues of data reliability and quality - based on experience!

The following discusses the issues for both quantitative and qualitative research utilizing this methodology currently in China.

Our first concern is simply poor panel management.

There are a plethora of panel providers in China and this alone should tell us that there will be concerns about panel management.  Too many people have jumped into this market thinking it would be a business gold mine and the resultant panel offerings are highly dubious.  A successful panel requires a lot of management – an opt-in system, regular contact and usage to keep the panel alive and a reliable rewards system for participation.  Many local panels fail across all three of these imperatives.

Our second concern is poor panel quality.

Given that recruitment quality in China is potentially an issue – lacking the standard screening and opt-in requirements of most quality global panels.  The fact is it’s hard to be sure exactly who is in many panels in China in terms of key demographics and participation interest.  The chance to earn quick money from panel inclusion by faking profiles is well known in the China market.

Most panels also lack one key indicator – personal income, which is a requirement for many consumer studies and an excellent quality control indicator. Where this does exist, a quick scan of the distribution by city and by age is a useful request prior to any study commitments and will on occasions raise serious concerns (e.g. 21-29 year olds earning over RMB15k a month!).

Our third concern is the quality of respondent participation.

In a study we commissioned to test some panel suppliers we used an image grid task which is always the one area where panel respondent interest is pushed to the limits resulting in no consideration responses and even jumping of sections of a study.

Lack of real involvement in the research task by respondents will often be seen in pattern responses such as random all over the scale responses, straight-lining (ticking all the boxes in the same spot) or other linear patterns (e.g. V-shapes or diagonals).   It was clear in our quality checking that some panels had respondents who were not committed targets for research and were just there to complete surveys for quick money. Typically it is those panels with poor opt-in and low to no levels of panel management that show these patterns.

ONLINE QUANTITATIVE

An Alternative Approach – The Online/Offline Interview

For any complex, major and detailed quantitative online survey it is always useful to address the issue of data reliability and quality, Anovax has suggested to some clients intent on using panels that they first consider using the panels more for recruitment purposes.  This of course reduces some of the cost savings benefits of panels, but does go a long way towards guaranteeing quality of the final sample.  

In this approach, the panel source can be used to identify qualified targets based on a short questionnaire capturing key sample criteria.  These identified targets can be further screened using a follow-up survey to identify even more granular sample requirements e.g. recent brand users. Anovax then incentivizes these panelists to attend a personal screening and then online interview at an internet cafe or coffee shop of their choice. 

At these venues they can be more correctly screened for demographic criteria and claimed behaviour or other criteria before participating in the full online survey.  As the survey is based on self-completing an online questionnaire, these targets can also be observed for quality with the assistance of the interviewer who will help to ensure that they are not rushing through and have answered all questions correctly.  The interviewer can also help clarify questions and issues which typically arise in understanding the questions in more complex studies.

Many of our clients have found this alternative approach to be effective in their quantitative online research especially when the survey extends beyond say 10-15 minutes or the interview topic or processes for interview are complex.

ONLINE QUALITATIVE

Some good news and not so good news!

With regards to online “qualitative” project in China, there are several issues that need to be considered:

First the Good News – "A Common Language"

Luckily the common language used by Chinese consumers is Mandarin Chinese despite wide differences in terms of dialect usage across China.  So online questionnaires and qualitative moderators need to read and write Chinese characters if only to communicate with respondents across the whole of China.

The does mean that the survey system also needs to support Chinese characters and not “Pinyin” (Romanized Chinese).  In Pinyin, the same word can have many different meanings depending on its tone (4 tones for each word) and context.  For example, the pinyin word “yao” can mean any of the following - will, want, bite, medicine, waist, need, sway, etc.

Other Good News

• A major positive of online qualitative is that the usual reticence of the Chinese can be overcome to some extent using online focus groups.  This is a useful tool for exploratory work where the “hidden” presence of their fellow participants allows the consumer to express often real but personal negatives about certain brands and other stimuli since the risk of embarrassing another group respondent is minimal.
• Online qualitative panels are very good for “communal” panels or those where the participants share some behavioural activity or lifestyle (e.g. McDonald Happy Meal buyers, Mothers with infants under 6 months, etc.).  These panels tend to have a longer “shelf life” and the interaction tends to be richer.  However recruiting these panelist and confirming legitimacy will always be an issue in China!

Problems With Online Qualitative in China

As this is a relatively new data collection methodology in China, there are some possible reasons for being careful with this methodology and these needs to be considered:

• Quality of responses may be an issue for some longitudinal qualitative methodologies because Chinese respondents tend not to be committed to research that requires extended time commitments, especially with the Bulletin Board approach.  Drop-offs can be a major issue in China! 
• Difficulty in verifying that the respondent is the one answering the questions – again this relates to panel quality!  That short survey quality check can be useful here, but making sure it is the same person throughout may be an issue.  It’s not unusual for some respondents to delegate the task to their children with little briefing of them at all.
• High costs for setting up and hosting an online discussion – this needs a dedicated server and software system, a cost many companies will want to pass on to the client.  Not a big costs, but it soon adds up.
• A government permit is required as the discussion platform is present in the public arena on the web. Not really an issue if the topics are not political in any way.
• Online qualitative that uses text moderation rather than verbal response can be an issue in China with response patterns typically lacking any rich textual insights, despite the best efforts of the moderator.

Most of the concerns we have raised here suggests clients should focus on two key areas when deciding to use a panel supplier in China.  The first is to check the opt-in process used by panel providers as thoroughly as is possible.  Every panel provider claims this, but very few really follow the process strictly.

The second area is to run a short test using an image grid and look at response patterns in the image grids.  Once you see overly random or linear patterns of any kind, start questioning the panel quality! At the same time you can check some of the demographics looking for clearly impossible relationships between things like age, education, gender and income.

As the old Latin saying goes, Caveat Emptorm let the buyer beware!