Using Large-Scale Social Media Experiments in Public Administration: Assessing Charitable Consequences of Government Funding of Nonprofits

Using Large-Scale Social Media Experiments in Public Administration: Assessing Charitable... Abstract In this article, we introduce and showcase how social media can be used to implement experiments in public administration research. To do so, we pre-registered a placebo-controlled field experiment and implemented it on the social media platform Facebook. The purpose of the experiment was to examine whether government funding to nonprofit organizations has an effect on charitable donations. Theories on the interaction between government funding and charitable donations stipulate that government funding of nonprofit organizations either decreases (crowding-out), or increases (crowding-in) private donations. To test these competing theoretical predictions, we used Facebook’s advertisement facilities and implemented an online field experiment among 296,121 Facebook users nested in 600 clusters. Through the process of cluster-randomization, groups of Facebook users were randomly assigned to different nonprofit donation solicitation ads, experimentally manipulating information cues of nonprofit funding. Contrary to theoretical predictions, we find that government funding does not seem to matter; providing information about government support to nonprofit organizations neither increases nor decreases people’s propensity to donate. We discuss the implications of our empirical application, as well as the merits of using social media to conduct experiments in public administration more generally. Finally, we outline a research agenda of how social media can be used to implement public administration experiments. Introduction Using social media has become an increasingly common activity as platforms like Facebook have become ubiquitous. For instance, to date there are 218 million Facebook users in the United States (Statistia 2017) with one-third of the US adult population logging into Facebook at least once a day (Public Religion Research Institute 2012). Social media platforms like Facebook or Twitter provide brand-new opportunities to implement online experiments studying citizen–state interactions. However, within public administration this potential has been largely untapped.1 Therefore, the purpose of this article is to introduce and showcase how social media, particularly Facebook, can be used to implement experiments in public administration research. As a consequence of digitization, large-scale social media experiments have been used increasingly in neighboring fields like political science, economics, or marketing (for an overview see Aral 2016). Examples include a wide range of topics, including political behavior (e.g., Bond et al. 2012), advertising (e.g., Bakshy et al. 2012a), product pricing (e.g., Ajorlou, Jabdabaie, and Kakhbod 2016), information propagation (e.g., Bakshy et al. 2012b), or emotional contagion (e.g., Kramer, Guillory and Hancock 2014). This trend is encouraging but it misses an important component of people’s online behavior, namely citizen–state interactions. Indeed, the advent of e-government and an increased presence of government agencies on social media platforms has led to the rise of online citizen–state interactions (Thomas and Streib 2003; Wukich and Mergel 2015). These interactions range from gathering information, for example, about how to fill out online applications, to crisis communication, and even complaints about poor services. In this study, we illustrate how to implement large-scale social media experiments on Facebook by examining interactions between nonprofit organizations seeking donations and their potential donors. In particular, we assess whether changes in government funding affect the levels of charitable income of nonprofit organizations. Theoretically, two competing mechanisms have been distinguished in explaining why levels of government funding may have an effect on private donations to nonprofit organizations. The crowding-out perspective argues that government funding would decrease people’s willingness to donate because donors as taxpayers perceive government funding as a substitution to their donations (Andreoni 1989; Warr 1982; Kim and Van Ryzin 2014). If they contributed already via taxes, why should they give in addition to them? Therefore, the crowding-out model predicts a decrease in private donations as a result of government funding. In contrast, the signaling model of crowding-in suggests that government funding is used by potential donors as an imperfect signal of an organization’s effectiveness (e.g., Borgonovi and O’Hare 2004; Rose-Ackerman 1981). In the absence of complete information about how a nonprofit organization will perform and operate with the funds at its disposal, government funding serves as an organization’s “quality stamp,” signaling the organization is not only trustworthy but also effective because it managed to receive competitive government grants. The crowding-in perspective, therefore, predicts an increase in private donations as a result of government funding. In this study, we test these somewhat competing claims in the context of a large-scale social media experiment. Conducted in a naturalistic setting, social media experiments, similar to conventional field experiments, combine high levels of internal validity with external validity. This allows us to test whether government funding crowds-in, or crowds-out, private donations. We implemented a field experiment on the social media platform Facebook by assigning clusters of approximately 300,000 Facebook users to donation solicitations of groups of real food banks. Using a pre-registered, placebo-controlled between-subjects design, groups of users were randomly allocated to three experimental conditions: (1) the control group (i.e., no funding information), (2) the placebo group (i.e., donation funded), and (3) the treatment group (i.e., government funded). As outcome measures, we monitored people’s revealed donation intentions by their click-through-rates (i.e., the frequency people clicked on the links in the ad solicitations), but also other behavioral measures such as website visits. Consistently, we find no direct evidence for either model, suggesting that public and private funding streams of nonprofit organizations do not seem to interact in the real world. In addition to these findings, we provide an overview of social media experiments and how they can be implemented in public administration research, including an agenda for studying online citizen–state interactions using large-scale social media experiments. The remainder of the study is as follow: in the next section, we review empirical applications of social media experiments in neighboring fields to provide an overview of the applicability of social media to conduct experiments in public administration. We then discuss our empirical application by first reviewing the literature on the crowding-out and crowding-in hypotheses. On this basis, we introduce our experimental research design and report the results of the experiment subsequently. In the final section, we draw implications for public administration research and practice from both, our empirical application and the review of innovative social media experiments. Conducting Social Media Experiments Before turning to the empirical application in this article, we provide an overview of the potential of social media to conduct experiments. Recent years have seen an increase in online field experiments implemented on social media platforms (Aral 2016). Indeed, companies like Amazon, Google, and Facebook constantly perform small experiments on their clients, for example through randomly altered website content where two different versions of the same website, online ad, or any other online parameter are randomly assigned to service users—a procedure marketers commonly refer to as A/B testing. In the past, social scientists have collaborated with major social media platforms to implement experiments. For example, Bond et al. (2012) implemented a 61-million-person political mobilization experiment on Facebook; similarly, Kramer, Guillory, and Hancock (2014) have implemented an online experiment to study emotional contagion among 690,000 Facebook users. Most studies on Facebook include advertisement related topics, however. For example, Bakshy and colleagues (2012a) study the effectiveness of social cues (i.e., peers’ associations with a brand) on consumer responses to ads for more than 5 million Facebook users. In another study, Bakshy et al. (2012b) look at the randomized exposure of links shared by peers of more than 250 million Facebook users and how it affects information sharing behavior on Facebook. In all of these cases, researchers had to work closely with Facebook to implement the process of randomization at the individual level of Facebook users. But, this would also mean that experimenting on Facebook would be limited to those with industry contacts. In the following, we report from recent experiments that have been conducted without industry collaboration. We aim to showcase how social media platforms like Facebook or Twitter can be used by scholars or government agencies to implement experiments in a relatively straightforward manner. Ryan (2012) was one of the first social scientists to use Facebook’s advertisement facilities to conduct research without having to collaborate with Facebook directly. Similar to the empirical application we report in this article, he randomly assigned clusters of individuals to different advertisements instead of randomizing on the user level (see Teresi and Michelson 2015 for an alternative approach2). To do so, he used Facebook’s advertisement facilities, which allow targeting ads to Facebook users on a number of demographic characteristics, such as age and gender, but also zip code (see also Ryan and Brockman 2012). Based on these user parameters, researchers can predetermine clusters of users and randomly allocate them to varying ad content. This is what Ryan did in his study. In particular, he looked at how advertisements that evoke emotions such as anger or anxiety affect information seeking behavior. He then used cluster-level click-through-rates as a dependent variable. Across three studies and more than 1.8 million impressions grouped into 360 clusters in total, he found consistent evidence that political advertisements that ought to evoke anger increase users’ proclivity to click through to a website. In other words, anger makes us click. A similar methodological approach was also used by Ryan and Brader (2017) who studied partisan selective exposure to election messages during the 2012 US presidential elections, using a total of 846 clusters of Facebook users. Similar applications also exist in fields like marketing or economics. Aral and Walker (2014) for instance report from an experiment conducted with 1.3 million users of a Facebook application to test how social influence in online networks affects consumer demand. They experimentally manipulated users’ network embeddedness and the strength of their social ties, finding that both increase influence in social networks. Wang and Chang (2013) studied a similar topic looking at whether social ties and product-related risks influence purchase intentions of Facebook users that were recruited via an online application. Although they found that online tie strength leads the higher purchase intentions, product-related risks had no direct effect on purchase intentions. In another Facebook study by Broockman and Green (2013), users were exposed to different types of political candidate advertisements over the course of 1 week. Like Ryan (2012), they randomized clusters of individuals, instead of individuals themselves. However, since they had access to public voter registries, they targeted 32,000 voters, which they assigned to 1,220 clusters across 18 age ranges, 34 towns, and 2 genders. These clusters of Facebook users were assigned to one of four experimental conditions: a control group with no advertisement, and three different types of advertisements that ought to increase Facebook users’ name recognition of the candidate. The innovation that Brookman and Green’s study introduced was that they used contact information from public voter records to gather public opinion data from these individual voters through telephone interviews later on. Since the creation of clusters was done on the basis of assigning 32,000 registered voters to 1,220 clusters, they had detailed contact information of registered voters that belong to each respective cluster. In other words, they were able to link cluster assignment on Facebook with attitudinal outcomes from survey data, such as candidate’s name recognition, positive impression of the candidate, whether people voted for the candidate, and whether they recall having seen the ad. Social media experiments exist outside of Facebook also. Gong et al. (2017) conducted a large-scale experiment among the social microblogging service Sina Weibo (i.e., the “Chinese Twitter”). They examined the return-on-investment of company tweets on viewers of TV shows. To do so, they randomly assigned 98 different TV shows into three experimental conditions: (1) the control group, where there is no tweet sent out about the particular TV shows; (2) the tweet condition, where each show is tweeted by the company; and (3) a tweet and retweet condition, where each show is tweeted by a company and retweeted by a so-called “influencer.” TV show viewing percentages were used as an outcome measure, finding that both tweeting and tweeting coupled with retweeting boost TV show views relative to the shows in the control group. In other words, social media efforts of TV companies result in a significant increase in viewers. They also found that retweets of influencers are more effective in generating new viewers than tweets by the companies. In another Twitter experiment, Coppock, Guess, and Ternovski (2016) looked at online mobilization behavior. In particular, authors were interested in whether Twitter users could be encouraged to sign a petition. To do so, they randomly divided 8,500 followers of a US nonprofit advocacy group into three experimental conditions. In the first stage, the nonprofit organization published a tweet in which its followers were encouraged to sign a petition. All three groups were exposed to the public tweet. In the second stage, a treatment condition received a direct message with a similar request, referring to them as “followers,” another treatment condition got the same direct message, but they were referred to as “organizers,” whereas the control group received no direct message. On this basis, authors checked whether subjects either retweeted or signed the petition. Other notable examples of using social media platforms like Twitter to implement experiments involve studying social media censorship in China (King, Pan, and Roberts 2013), the effectiveness of repeated political messages on twitter followers of politicians (Kobayashi and Ichifuji 2015), the effectiveness of news media (King, Schneer, and White 2017), or racist online harassment (Munger 2017). The aforementioned examples provide rich inspiration for conducting social media experiments in public administration research. Social media experiments have the distinct advantage that they combine the internal validity of experiments with an increased realism and external validity. In this sense, they are a subtype of conventional field experiments, which are conducted in an online environment where people interact via social media. In addition, social media experiments can easily be conducted on large-scale samples using a variety of unobtrusive outcome measures to assess respondents’ revealed behaviors. They are therefore a viable option to complement survey-based experiments that often employ stated preferences (i.e., attitudes, evaluative judgments, or behavioral intentions), which make up the majority type of experiments implemented in public administration to date (Li and Van Ryzin 2017). People increasingly interact with government and government agencies using social media platforms like Twitter and Facebook (Mergel 2012). Scholars and government agencies alike can implement social media experiments to test the effectiveness of using these relatively new channels of communication and information provision. Examples may include assessing whether providing information on social media about the performance of government agencies affect citizen trust in those agencies, or may lead citizens to desirable behaviors, including coproduction. Indeed, implementing such innovative experimental designs in the context of online citizen–government interactions may be a viable avenue for future experimentation in public administration research. In the following, we introduce an empirical application of an online social media experiment that examines in how far government funding and charitable donation intentions interact. Empirical Application: How Government Funding and Private Donations Interact An impressive body of literature has emerged from various disciplines that focus on the issue of whether government funding would displace (crowding-out effect) or leverage (crowding-in effect) private contributions to nonprofit organizations. The impact of government funding on private donations to nonprofits largely rests on how potential donors and nonprofits themselves strategically respond to government funding of nonprofit activities (Tinkelman 2010; Lu 2016). We focus on the strategic responses of private donors in the present analysis: how would donors change their levels of charitable giving when a nonprofit organization is supported by government funding. The literature distinguishes two contrasting models of how government-funded nonprofits are perceived, which ultimately affects charitable donations via processes of crowding-in or crowding-out. Early crowding-out theory assumes that private donors are altruistic and care about the optimal level of public goods provision. Donors as taxpayers would consider government funding as their contributions through taxation and thus perceive it as a perfect substitute for voluntary donations. In this way, increases in government support would lower the need for additional private contributions. Therefore, when a nonprofit receives more support from the government, private donors would consciously reduce their giving to this organization. As a result, there is a dollar-for-dollar replacement between private giving and government funding (Roberts 1984; Warr 1982). This pure altruism assumption was later challenged by Andreoni’s (1989) model of impure altruism, which predicts that donors are also motivated by a “warm-glow”—the utility from the act of giving to help others. In this impure altruism line of reasoning, government funding and private giving would not completely substitute each other. As a result, there may exist a crowding-out effect between these two funding sources, but the magnitude of the effect is less than the dollar-for-dollar model that pure altruism would predict. On the other hand, private donors might consider government funding favorably and become more willing to contribute to government-funded nonprofits because they perceive them as more competent and/or needy. Crowding-in theory proposes that government funding may stimulate charitable contributions in two ways. First, when donors do not have complete knowledge concerning beneficiary nonprofits and their programs, government funding serves as a direct signal of the nonprofit’s quality and reliability (Rose-Ackerman 1981). Indeed, to be funded by government agencies, nonprofit organizations have to go through a competitive merit-based selection process and meet financial and programmatic requirements (Lu 2015; Suárez 2011). Therefore, the receipt of government funding can be perceived by uninformed donors as an indicator of trustworthiness and competence. Second, government funding also is considered as a signal of unmet social needs, calling for more donor attention and further facilitating the leveraging effect of government funding (Brooks 1999; Okten and Weisbrod 2000). There exist a rich body of empirical studies in support of the crowding-out hypothesis (e.g., Andreoni 1993; Andreoni and Payne 2011; Brooks 2000; De Wit, Bekkers, and Broese van Groenou 2017; Dokko 2009; Hughes, Luksetich, and Rooney 2014; Kingma 1989; Steinberg 1987) and the crowding-in model (e.g., Borgonovi and O’Hare 2004; De Wit and Bekkers 2016; Heutel 2014; Khanna and Sandler 2000; Lu 2016; Okten and Weisbrod 2000; Smith 2007). Most recently, De Wit and Bekkers (2016) and Lu (2016) respectively employed meta-analytical techniques to aggregate existing empirical evidence on crowding-in and crowding-out. Both studies conclude a significant positive association between government funding and private donations, even though the magnitude of the relationship is trivial. The above-mentioned body of literature on crowding-out and crowding-in greatly advances our understanding of the complex interaction between government funding and private donations. However, it generally suffers from two limitations. First, the majority of existing empirical literature testing the crowding-in or crowding-out effect employs observational data. Although observational studies enable scholars to explore the association between the two revenue sources, drawing causal inferences remains challenging (Blom-Hansen, Morton, and Serritzlew 2015; James, Jilke, and Van Ryzin 2017a). Second, both crowding-in and crowding-out lines of reasoning assume that potential donors possess perfect information about the nonprofits they might want to donate to, especially whether these organizations are funded by government and to what extent. However, this assumption might not be true in the real world (De Wit et al. 2017; Horne, Johnson, and Van Slyke 2005; Krasteva and Yildirim 2013). For example, Horne, Johnson, and Van Slyke (2005) used public opinion data to demonstrate that individual donors do not necessarily have complete information on the financial structures of their beneficiary organizations and subsequently link donation decisions to the levels of government funding. In recent years, scholars began to employ experimental designs to address these two limitations (i.e., endogeneity and imperfect information) in the crowding-out and crowding-in literature (e.g., Eckel, Grossman, and Johnston 2005; Kim and Van Ryzin 2014; Ottoni-Wilhelm, Vesterlund, and Xie 2017; Wasif and Prakash 2017). Existing experimental studies testing crowding-out/-in effects usually include manipulations of the existence and the level of direct government support to beneficiaries, and then measure variations in of subjects’ donations. Methodologically, experimental designs are advantageous over observational settings in terms of their internal validity when testing crowding-out/-in effects because experimental studies create a controlled environment of information exchange to rule out confounding factors. As a result, scholars are provided with more direct evidence of the causal linkage between government support and charitable giving (Blom-Hansen, Morton, and Serritzlew 2015; James, Jilke, and Van Ryzin 2017a). Table 1 reviews the experimental studies of crowding-out/-in effects to date, including type, setting, and results. As can be seen in Table 1, most experimental studies employ laboratory experimental designs, primarily using two specific experimental paradigms. One is the public goods game (e.g., Andreoni 1993; Eckel et al. 2005; Isaac and Norton 2013) and the other type of experimental setting employed in the literature is the dictator game (e.g., Blanco, Lopez, and Coleman 2012; Konow 2010; Korenok, Millner, and Razzolini 2012). Despite different experimental paradigms, most of the laboratory experiments report a partial crowding-out effect between government funding and charitable contributions (see also De Wit and Bekkers 2016). Table 1. Experimental Studies on Crowding-in and Crowding-out Effects Reference  Type of experiment  Experimental setting  Subjects  Finding  Andreoni (1993)  Lab  Public goods game  Students  Crowding-out  Chan et al. (1996)  Lab  Public goods game  Students  No effect  Bolton and Katok (1998)  Lab  Dictator game  Students  Crowding-out  Chan et al. (2002)  Lab  Public goods game  Students  Crowding-out  Sutter and Weck-Hannemann (2004)  Lab  Public goods game  Students  Crowding-out  Eckel et al. (2005)  Lab  Public goods game  Students  Crowding-out  Güth et al. (2006)  Lab  Public goods game  Students  Crowding-out  Galbiati and Vertova (2008)  Lab  Public goods game  Students  Crowding-in  Hsu (2008)  Lab  Public goods game  Students  Crowding-out  Reeson and Tisdell (2008)  Lab  Public goods game  Students  Crowding-out  Konow (2010)  Lab  Dictator game  Students  Crowding-out  Blanco et al. (2012)  Lab  Dictator game  Tourists  No effect  Gronberg et al. (2012)  Lab  Public goods game  Students  Crowding-out  Korenok et al. (2012)  Lab  Dictator game  Students  Crowding-out  Issac and Norton (2013)  Lab  Public goods game  Students  Crowding-out  Lilley and Slonim (2014)  Lab  Contribution decision making  Students  Crowding-out  Galbiati and Vertova (2014)  Lab  Public goods game  Students  Crowding-in  Kim and Van Ryzin (2014)  Survey  Hypothetical vignette  General population sample (US)  Crowding-out  Wasif and Prakash (2017)  Survey  Hypothetical vignette  General population sample (Pakistan)  No effect  Ottoni-Wilhelm et al. (2017)  Lab  Contribution decision making  Students  Crowding-out  Reference  Type of experiment  Experimental setting  Subjects  Finding  Andreoni (1993)  Lab  Public goods game  Students  Crowding-out  Chan et al. (1996)  Lab  Public goods game  Students  No effect  Bolton and Katok (1998)  Lab  Dictator game  Students  Crowding-out  Chan et al. (2002)  Lab  Public goods game  Students  Crowding-out  Sutter and Weck-Hannemann (2004)  Lab  Public goods game  Students  Crowding-out  Eckel et al. (2005)  Lab  Public goods game  Students  Crowding-out  Güth et al. (2006)  Lab  Public goods game  Students  Crowding-out  Galbiati and Vertova (2008)  Lab  Public goods game  Students  Crowding-in  Hsu (2008)  Lab  Public goods game  Students  Crowding-out  Reeson and Tisdell (2008)  Lab  Public goods game  Students  Crowding-out  Konow (2010)  Lab  Dictator game  Students  Crowding-out  Blanco et al. (2012)  Lab  Dictator game  Tourists  No effect  Gronberg et al. (2012)  Lab  Public goods game  Students  Crowding-out  Korenok et al. (2012)  Lab  Dictator game  Students  Crowding-out  Issac and Norton (2013)  Lab  Public goods game  Students  Crowding-out  Lilley and Slonim (2014)  Lab  Contribution decision making  Students  Crowding-out  Galbiati and Vertova (2014)  Lab  Public goods game  Students  Crowding-in  Kim and Van Ryzin (2014)  Survey  Hypothetical vignette  General population sample (US)  Crowding-out  Wasif and Prakash (2017)  Survey  Hypothetical vignette  General population sample (Pakistan)  No effect  Ottoni-Wilhelm et al. (2017)  Lab  Contribution decision making  Students  Crowding-out  View Large Table 1. Experimental Studies on Crowding-in and Crowding-out Effects Reference  Type of experiment  Experimental setting  Subjects  Finding  Andreoni (1993)  Lab  Public goods game  Students  Crowding-out  Chan et al. (1996)  Lab  Public goods game  Students  No effect  Bolton and Katok (1998)  Lab  Dictator game  Students  Crowding-out  Chan et al. (2002)  Lab  Public goods game  Students  Crowding-out  Sutter and Weck-Hannemann (2004)  Lab  Public goods game  Students  Crowding-out  Eckel et al. (2005)  Lab  Public goods game  Students  Crowding-out  Güth et al. (2006)  Lab  Public goods game  Students  Crowding-out  Galbiati and Vertova (2008)  Lab  Public goods game  Students  Crowding-in  Hsu (2008)  Lab  Public goods game  Students  Crowding-out  Reeson and Tisdell (2008)  Lab  Public goods game  Students  Crowding-out  Konow (2010)  Lab  Dictator game  Students  Crowding-out  Blanco et al. (2012)  Lab  Dictator game  Tourists  No effect  Gronberg et al. (2012)  Lab  Public goods game  Students  Crowding-out  Korenok et al. (2012)  Lab  Dictator game  Students  Crowding-out  Issac and Norton (2013)  Lab  Public goods game  Students  Crowding-out  Lilley and Slonim (2014)  Lab  Contribution decision making  Students  Crowding-out  Galbiati and Vertova (2014)  Lab  Public goods game  Students  Crowding-in  Kim and Van Ryzin (2014)  Survey  Hypothetical vignette  General population sample (US)  Crowding-out  Wasif and Prakash (2017)  Survey  Hypothetical vignette  General population sample (Pakistan)  No effect  Ottoni-Wilhelm et al. (2017)  Lab  Contribution decision making  Students  Crowding-out  Reference  Type of experiment  Experimental setting  Subjects  Finding  Andreoni (1993)  Lab  Public goods game  Students  Crowding-out  Chan et al. (1996)  Lab  Public goods game  Students  No effect  Bolton and Katok (1998)  Lab  Dictator game  Students  Crowding-out  Chan et al. (2002)  Lab  Public goods game  Students  Crowding-out  Sutter and Weck-Hannemann (2004)  Lab  Public goods game  Students  Crowding-out  Eckel et al. (2005)  Lab  Public goods game  Students  Crowding-out  Güth et al. (2006)  Lab  Public goods game  Students  Crowding-out  Galbiati and Vertova (2008)  Lab  Public goods game  Students  Crowding-in  Hsu (2008)  Lab  Public goods game  Students  Crowding-out  Reeson and Tisdell (2008)  Lab  Public goods game  Students  Crowding-out  Konow (2010)  Lab  Dictator game  Students  Crowding-out  Blanco et al. (2012)  Lab  Dictator game  Tourists  No effect  Gronberg et al. (2012)  Lab  Public goods game  Students  Crowding-out  Korenok et al. (2012)  Lab  Dictator game  Students  Crowding-out  Issac and Norton (2013)  Lab  Public goods game  Students  Crowding-out  Lilley and Slonim (2014)  Lab  Contribution decision making  Students  Crowding-out  Galbiati and Vertova (2014)  Lab  Public goods game  Students  Crowding-in  Kim and Van Ryzin (2014)  Survey  Hypothetical vignette  General population sample (US)  Crowding-out  Wasif and Prakash (2017)  Survey  Hypothetical vignette  General population sample (Pakistan)  No effect  Ottoni-Wilhelm et al. (2017)  Lab  Contribution decision making  Students  Crowding-out  View Large In addition to laboratory experiments, there are a few experimental studies that employ survey experiments to test the crowding-in and crowding-out propositions. For example, Kim and Van Ryzin (2014) conducted an online survey experiment with 562 participants and found that an arts nonprofit with government funding would receive about 25% less private donations than an identical hypothetical organization without government funding. In contrast, Wasif and Prakash’s (2017) survey experiment with 530 respondents in Pakistan reported that federal funding would not change respondents’ willingness to donate to a hypothetical faith-based educational nonprofit. When meta-analyzing results from experimental studies only, De Wit and Bekkers (2016) find substantially different results compared to observational studies, with experimental studies showing a considerable crowding-out effect and nonexperimental studies a very small crowding-in effect. There are two potential explanations for these differences. A first possibility would be that observational studies on crowding-out/-in are plagued by endogeneity, and hence the discrepancies in results may be a product of the comparatively poor internal validity of observational research designs. A second possibility would be that findings predominately produced within stylized settings such as economic games or hypothetical scenarios may hardly extrapolate beyond the laboratory. Or in other words, people may behave differently in lab and survey experiments as in the real world. Indeed, a recent systematic comparison between laboratory and field experiments concluded that the ability of stylized experiments to extrapolate social preferences from the lab to the field is limited, at best (Galizzi and Navarro-Martinez, Forthcoming; see also Levitt and List 2007). We respond to these competing interpretations by implementing a naturalistic field experiment using social media. Method In the following, we report the experimental design for this study. Doing so, we follow James, Jilke, and Van Ryzin’s (2017b) recommended reporting guidelines for experiments in public administration research. Experimental Design, Subject Recruitment, and Treatment We designed a large-scale placebo-controlled online field experiment on the social networking platform Facebook by purchasing ads using Facebook’s advertisement facilities. These ads were designed to solicit donations to human service nonprofits in New York City (NYC). We randomly allocated groups of Facebook users to three different ads of local human service organizations. Depending on experimental conditions, these ads included information cues about (1) government funding (treatment group), (2) donation funding (placebo group), or (3) no such information (control group) as depicted in Figure 1. The experimental design was pre-registered at the AEA RCT registry3 and received ethics approval from Rutgers University’s Institutional Review Board. Figure 1. View largeDownload slide Experimental Design. Figure 1. View largeDownload slide Experimental Design. We chose to target Facebook users who live in NYC because this allowed us to leverage local information about government funding of human service organizations (i.e., real food banks in NYC) in the experimental design. All ads included a donation appeal, encouraging Facebook users to donate to local food banks through a webpage they could access by clicking on the ad. Once clicked, and depending on the experimental condition, users were directed to one of three web pages where Food Banks from NYC4 were listed and hyperlinked. Facebook users in the control condition (i.e., no funding information) were directed to the Rutgers Observatory of Food Banks webpage. Here, all NYC Food Banks were listed and hyperlinked. Those subjects who were allocated to the placebo condition (i.e., donation-funded), were directed to the Rutgers Observatory of Donation-Funded Food Banks webpage. The presence of a placebo condition (i.e., donation-funded) allows us to test whether any differences between treatment and control is due to the mere provision of funding information. On this website, all donation-funded5 food banks from NYC were listed and hyperlinked. In the treatment condition (i.e., government-funded), Facebook users were sent to the Rutgers Observatory of Government-Funded Food Banks website, where all Food Banks that were classified as government-funded were listed and hyperlinked. Unique ads were developed for the experiment. Figure 2 shows the ads that were randomly presented to groups of Facebook users. Taking the ad of the control condition as a baseline, the treatment and placebo ads look exactly like the control ad, except for four important differences: Figure 2. View largeDownload slide Facebook Ads (Control versus Placebo versus Treatment Conditions). Figure 2. View largeDownload slide Facebook Ads (Control versus Placebo versus Treatment Conditions). 1) A different sponsor was used: i. Rutgers Observatory of Food Banks for the control group; ii. Rutgers Observatory of Donation-Funded Food Banks for the placebo group; iii. Rutgers Observatory of Government-Funded Food Banks for the treatment group. 2) The ad’s headline was changed from “Donate to Food Banks in your area” to “Donate to government-funded Food Banks in your area,” or “Donate to donation-funded Food Banks in your area” respectively. 3) The heading under the image changed from “Food Banks in NYC” to “Government-funded Food Banks in NYC,” or “Donation-funded Food Banks in NYC” respectively. 4) The two-sentence-description used “government-funded Food Banks in NYC” and “donation-funded Food Banks in NYC” instead of “Food Banks in NYC.” Cluster Randomization Randomization into experimental conditions was not performed on the individual level, but on the level of groups of Facebook users (see also Ryan 2012 for a similar approach). This was done because Facebook’s commercial advertisement facilities do not permit different ads to be randomized on the individual level (Ryan and Brockman 2012). Instead, users were classified into groups. This approach makes use of Facebook’s ability to target users based on demographics. More precisely, we stratified Facebook users into clusters of users by zip code, age, and gender. Cluster-randomized experiments are widely used across the social and medical sciences where groups of individuals—for example, people living in an area, or classmates—are jointly randomized into experimental conditions (e.g., Campbell and Walters 2014; Hayes and Moulton 2017). For our experiment, we randomized demographic clusters of Facebook users, rather than users nested within these clusters. Doing so, we first collected all 177 NYC zip codes and randomly chose 100. In a second step, we grouped respondents of 18 to 60 years into three age categories (18–27, 28–38, and 39–60). We composed these three groups to have an approximately equal “potential reach” across strata (potential reach refers to the approximate number of Facebook users that can be exposed to an ad). Each of the three groups has a potential reach of about 1.5 to 1.6 million users who live in NYC. Next, we created 600 clusters by taking 100 zip codes * 3 age categories * 2 genders. These 600 clusters were randomized into one of the three experimental conditions, so that each experimental condition ended up with 200 clusters with a potential reach of about 1 million Facebook users6. An important concern when analyzing data from cluster-randomized experiments is that clusters vary in size so that the conventional difference-in-means estimator would be biased (Middleton and Arronow 2015). Indeed, the 600 clusters we have come up with vary in (anticipated) size, with a potential reach between fewer than 1,000 to 110,000 Facebook users (mean potential reach of approximately 5,000 users). Therefore, we followed Gerber and Green’s (2012) suggestion and blocked on cluster size (200 groups of 3 clusters each) in the randomization procedure, so that the difference-in-means estimator can be used without risk of bias. The blocked cluster-randomization was performed using Stata 14. The actual implementation of the experiment was done using Facebook’s Adverts Manager interface. We purchased 600 ads and targeted them to specific demographic clusters (stratified by zip code, age categories, and gender). A total of 600 clusters (i.e., 200 per experimental condition) were purchased for $10 each and our ads were constructed to pay per impression7. All ads ran on the same day for 24 h, from 6 a.m. EST on August 21, 2017 until 6 a.m. the following day. Although the sample size of the clusters was pre-determined, the number of Facebook users who have been actually exposed to the ads was not; however, the potential reach across experimental conditions was approximately the same (see footnote 5). This is so because Facebook uses its own “Optimization for Ad Delivery” algorithm (which is the same across experimental conditions) to determine who will be exposed to an ad. In essence, Facebook users are shown ads which—according to Facebook’s algorithm—have the highest probability of resulting in an impression or click.8 The 600 clusters had a joint potential reach of 2,972,500, but 296,121 Facebook users were actually exposed to the ads as a result the Facebook’s ad bidding procedure. Outcomes Our primary outcome of interest is subjects’ revealed intention to donate. Or in other words, we look at whether the ads encourage people to click on them. We interpret link clicks as people’s intention to donate because the ads explicitly solicit donations by encouraging people to click. Indeed, clicking is commonly seen as a crucial antecedent for purchase intentions in the marketing literature (Zhang and Mao 2016). Against this background and given the explicit ad solicitations for donations we used in the ads (i.e., “click on one of the listed websites to make a donation”), we assume that ad clicks are a motivational precursor for donation intentions. Doing so, we use the so-called unique-outbound-link-click-rate per cluster as our primary dependent variable. The unique-outbound-click-rate (also referred to as click-through-rate) denotes a cluster’s actual reach (i.e., the number of people who saw our ads at least once) divided through its unique outbound clicks (i.e., the number of people who performed a click that took them off Facebook-owned property). In our case, the outbound click led users to one of three web pages—depending on the experimental condition they were assigned to—we have developed for this study. On these web pages, which were labeled as either the Rutgers Observatory of Food Banks (control group), the Rutgers Observatory of Food Banks (placebo group), or the Rutgers Observatory of Government-Funded Food Banks (treatment group), subjects were informed about the purpose of the study and provided with a hyperlinked list of either government-funded, donation-funded, or all food banks in NYC. As a secondary, though not pre-registered, measure, we analyze unique page views (i.e., the number of times people visited their assigned Observatory webpage at least once) as provided by Google Analytics for the three web pages. We calculated the percentage of unique page views relative to the number of people who saw the ads at least once. This measure serves as a robustness check for our primary measure of interest as it provides the actual number of users who actually ended up on the web pages. Results The ads we ran on Facebook reached a total 296,121 NYC Facebook users, resulting in a mean click-through-rate (CTR) of 0.48 (standard deviation of 0.49) for a total of 600 clusters (which is our unit of analysis). This means that of the approximately 300,000 Facebook users who saw the ads at least once, 0.48% performed at least one click which was intended to take them off Facebook and to one of the respective Observatory web pages that were designed for this study. To determine whether we find evidence for a crowding-out/-in effect of government funding on donation intentions, we analyzed whether there are systematic differences in users’ click-through-rates between experimental conditions. The left panel in Figure 3 depicts the results of tests on the equality of means between control, placebo (i.e., donation), and treatment (i.e., government) groups. Independent two-sample t-tests were conducted to compare (1) control and treatment conditions (mean difference of 0.00, p = 0.99; Cohen’s d = 0.00), (2) placebo and treatment conditions (mean difference of 0.06, p = 0.21; Cohen’s d = −0.13), and (3) control and placebo conditions (mean difference of −0.06, p = 0.17; Cohen’s d = −0.14) (see also Table A1 in the Appendix). In sum, all three experimental conditions are statistically indistinguishable from each other in terms of their CTRs. In addition, they are of trivial magnitude with very small values in terms of effect sizes (i.e., Cohen’s d). This means that we neither find support for the crowding-out nor the crowding-in hypothesis. In addition, there is also no significant difference between control and placebo conditions. Figure 3. View largeDownload slide Experimental Results. Figure 3. View largeDownload slide Experimental Results. When analyzing unique page views of the Observatory web pages that were created for this study, we used page visit data from Google Analytics, which provided us with the total number of unique page visits for each experimental condition. Therefore, the unit of analysis for this measure is individual Facebook users (n = 296,121). In total, 524 page views were recorded representing 0.18% of our sample of Facebook users.9 In the control condition, there were a total of 162 page visits, representing 0.16% of Facebook users that have been assigned to the control condition. In the placebo and treatment conditions there were respectively a total of 168 (0.17%) and 194 (0.19%) page views (see also the right panel on Figure 3). When analyzing whether these somewhat-minor differences are of systematic nature, we performed x2 tests. The percentage points difference in page visits between control and treatment condition is 0.03 (x2(1) = 2.30, p = 0.13; Odds Ratio = 1.18), and the difference between placebo and treatment condition is 0.02 (x2(1) = 1.16, p = 0.28; Odds Ratio = 1.12). Between control and placebo conditions, the percentage point difference is even smaller (i.e., 0.01) (x2(1) = 1.87, p = 0.66; Odds Ratio = 1.05). In sum, all three experimental conditions are statistically indistinguishable from each other. In addition, differences are small in magnitude with small effect sizes (i.e., Odds Ratios), confirming prior results from analyzing Facebook’s CTRs. We can therefore confidently reject both, the crowding-out and the crowding-in hypotheses. Conclusion This study has developed and implemented a large-scale but relatively low-cost online experiment in the context of public administration research, where about 300,000 Facebook users were randomly assigned to different donation solicitation ads. The study has found no evidence for crowding-in, nor the crowding-out model. In this sense, our results differ from prior experimental studies that stem from the behavioral laboratory or have been produced in the context of hypothetical vignette designs. While our study is certainly not the first that finds neither evidence for the crowding-out, nor the crowding-in hypotheses (see Blanco et al. 2012; Chan et al. 1996; Wasif and Prakash 2017), we provide—to our knowledge—the first experimental evidence that has been produced within a naturalistic context. However, our results have to be taken with a fair grain of salt. Our effective sample size of 600 clusters may not be able to have detected a difference in groups for our secondary outcome measure. Future studies are advised to replicate our design using a larger number of clusters, possibly by stratifying clusters for single age categories (i.e., one for each year) instead of grouping them as we did. In this sense, our findings do not provide a strict theory test of crowding-in versus crowding-out but provide some suggestive evidence that government funding does not seem to matter in the case of online donation solicitations of local food banks. Future studies may extend this to other areas of human service delivery, including more direct cues of government funding because sometimes people do not pay appropriate attention to informational cues. Although there is a long-standing scholarly debate on crowding-out and crowding-in between government funding and private donations, our study adds to this literature that in an online setting, whether a nonprofit is supported by government funding seems not to affect donors’ behaviors. Therefore, it is possible that, from donors’ perspective, government funding and charitable donations are two independent revenue streams. Donors do not necessarily consider a nonprofit’s revenue structure, especially whether it is supported by government funding when making donation decisions. If so, the crowding-out versus crowding-in debate over the last several decades might be overstated. Certainly, we did not examine in the present study how nonprofits strategically modify their fundraising efforts when they receive government funding, which might actually contribute to the crowding-in or crowding-out effect. For example, Andreoni and Payne (2011) argued that the crowding-out effect is mostly driven by nonprofits’ decreased fundraising efforts as a response to government funding. Again, we cannot guarantee the generalizability of our findings from food banks to other service areas. The primary aim of this study was to showcase how public administration scholars and government agencies can utilize social media to conduct experimental research in the realm of online citizen–government interactions. Social media experiments enable public administration researchers to ensure both internal and external validity because they have the distinct advantage that they combine the internal validity of experiments with an increased realism and ecological validity. Furthermore, as we demonstrate in this study, industry contacts are no longer necessary for implementing large-scale social media experiments. Social media service platforms such as Twitter or Facebook can be easily used to test a variety of important questions to public administration scholars and practitioners. Here, online experiments can be conducted on large-scale samples using a variety of unobtrusive outcome measures to assess respondents’ revealed behaviors. A research agenda for applying social media experiments in studying online citizen–state interactions has implications for a wide range of areas. Possible applications may include assessing the effectiveness of different public communication strategies on social media and how these strategies affect, for example, agency reputation. Indeed, bureaucratic reputation is a core topic area in public administration, with studies examining how news media shape agencies’ reputation, including trust (e.g., Grimmelikhuijsen, de Vries, and Zijlstra 2018). Little is known about how social media communication strategies may alter agency reputation. Social media experiments may assess the effect of different types of communication strategies on reputation—examples may include diffusing responsibility attributions for service failure across actors, active agency branding, or just randomly publishing information about administrative procedures. Another possible area of application may include assessing the effectiveness of different targeted job advertisements for government agencies, using, for instance, Facebook’s advertisement facilities to get underrepresented groups to apply. Recent scholarship has exemplified the importance of signaling personal benefits to increase the number of women and people of color to apply for government jobs (e.g., Linos 2017). Testing and refining such targeted job advertisements and/or messages via social media platforms like Facebook may be a great way to further this promising line of scholarship, but also be an excellent evaluation tool in seeking the most effective online recruitment strategy to increase the demographic representation of government agencies. A more obvious area of application may include assessing whether different ways of how public performance information is presented online affects citizens’ attitudes and subsequent behaviors. Indeed, a growing literature in behavioral public administration examines the psychology of performance information (e.g., James and Olsen 2017), assessing whether different ways of presenting numbers affects perceptions of public service performance or satisfaction evaluations of government agencies. This agenda may be extended to social media experiments where large-scale experiments could be conducted on social media platforms to probe for the external validity of prior survey experiments. Social media experiments may also be useful in simply testing the effectiveness of governmental online crisis warning systems by assessing which types of messages on social media are most likely to be shared across users. Such an agenda would have important practical implications for the effectiveness of early crisis warning systems. The emergence of Internet-based social media has made it possible for researchers to easily interact with thousands or even millions of citizens. Implementing such innovative experimental designs in the context of online citizen–government interactions may be a viable avenue for future experimentation in public administration research both, for academics and practitioners alike. We hope this contribution sparks a wider implementation of public administration experiments on social media platforms. FUNDING This study received funding from the Rutgers SPAA Faculty Research Funds AY16-17. This study was supported by a National Research Foundation of Korea Grant from the Korean Government [NRF-2017S1A3A2065838]. Footnotes 1 Despite a number of important studies on social media use in government agencies and online citizen–government interactions (e.g., Mergel 2012; Grimmelikhuijsen and Meijer 2015; Porumbescu 2015; Im et al. 2014), to date no experimental applications on social media platforms exist in public administration scholarship (for a notable exception see a recent conference contribution by Olsen 2017). 2 Teresi and Michelson (2015) randomized individual Facebook users with whom they connected via a Facebook profile (i.e., becoming “friends”) into experimental conditions. While one group of “friends” received mainly apolitical status updates from the host account, the treatment group received political messages about the upcoming 2010 elections. After the election, authors searched for each “friend” in the state list of registered voters using information provided via Facebook’s profile (i.e., names, age, gender, etc.) to examine whether these online get-out-the-vote messages distributed through social media encouraged subjects to vote. 3 AEARCTR-0002345 (see https://www.socialscienceregistry.org/trials/ 2345). 4 The food banks listed on the websites are all real food banks that operate in NYC, as collected through the National Center for Charitable Statistics (NCCS) database. 5 We distinguished donation-funded and government-funded food banks based on the revenue information provided in food banks’ latest 990 forms in the NCCS database. For this study, we defined government-funded food banks as the ones that receive government funding in addition to private donations, and donation-funded food banks as the ones that do not receive government funding but private donations. 6 988,200 in the control group; 985,300 in the placebo group; 999,000 in the treatment group. 7 This makes a total project cost of $6,000. However, from the purchased $6,000, Facebook ran our ads for a total value of $3,299, or a mean value of $5.50 per cluster. 8 Further information can be found here: https://www.facebook.com/business/help/1619591734742116. 9 There is an inconsistency between the number of unique outbound clicks reported by Facebook and actual unique page visits as provided by Google Analytics. This may be the result of some Facebook clicks not reaching its intended destination with users potentially aborting connecting with a website outside Facebook, possibly because of fear of virus-infected websites. References Ajorlou, A., A. Jadbabaie, and A. Kakhbod. 2016. Dynamic pricing in social networks: The word-of-mouth effect. Management Science  64: 971– 79. Google Scholar CrossRef Search ADS   Andreoni, J. 1989. Giving with impure altruism: Applications to charity and Ricardian equivalence. The Journal of Political Economy  97: 1447– 58. Google Scholar CrossRef Search ADS   ———. 1993. An experimental test of the public-goods crowding-out hypothesis. The American Economic Review  83: 1317– 27. Andreoni, J., and A. A. Payne. 2011. Is crowding out due entirely to fundraising? Evidence from a panel of charities. Journal of Public Economics  95: 334– 43. Google Scholar CrossRef Search ADS   Aral, S. 2016. Networked experiments. In The Oxford Handbook of the Economics of Networks , ed. Yann Bramoullé, Andrea Galeotti, and Brian Rogers, New York: Oxford University Press. Aral, S., and D. Walker. 2014. Tie strength, embeddedness, and social influence: A large-scale networked experiment. Management Science  60: 1352– 70. Google Scholar CrossRef Search ADS   Bakshy, E., D. Eckles, R. Yan, and I. Rosenn. 2012a. Social influence in social advertising: evidence from field experiments. arXiv:1206.4327v1. Bakshy, E., I. Rosenn, C. Marlow, and L. Adamic. 2012b. The role of netwroks in information diffusion. arXiv:1207.4145v5. Blanco, E., M. C. Lopez, and E. A. Coleman. 2012. Voting for environmental donations: Experimental evidence from Majorca, Spain. Ecological Economics  75: 52– 60. Google Scholar CrossRef Search ADS   Blom-Hansen, Jens, Rebecca Morton, and Soren Serritzlew. 2015. Experiments in public management research. International Public Management Journal  18: 151– 70. Google Scholar CrossRef Search ADS   Bolton, G. E., and E. Katok. 1998. An experimental test of the crowding out hypothesis: The nature of beneficent behavior. Journal of Economic Behavior & Organization  37( 3): 315– 31. Google Scholar CrossRef Search ADS   Bond, R. M., C. Fariss, J. Jones, A. Kramer, C. Marlow, J. Settle, and J. Fowler. 2012. A 61-million-person experiment in social influence and political mobilization. Nature  489: 295– 98. Google Scholar CrossRef Search ADS   Borgonovi, F., and M. O’Hare. 2004. The impact of the national endowment for the arts in the United States: Institutional and sectoral effects on private funding. Journal of Cultural Economics  28: 21– 36. Google Scholar CrossRef Search ADS   Broockman, David and Donald Green. 2014. Do online advertisements increase political candidates’ name recognition or favorability? Evidence from randomized field experiments. Political Behavior  36: 263– 89. Google Scholar CrossRef Search ADS   Brooks, A. C. 1999. Do public subsidies leverage private philanthropy for the arts? Empirical evidence on symphony orchestras. Nonprofit and Voluntary Sector Quarterly  28: 32– 45. Google Scholar CrossRef Search ADS   ———. 2000. Is there a dark side to government support for nonprofits? Public Administration Review  60: 211– 18. CrossRef Search ADS   Campbell, Michael and Stephen Walters. 2014. How to design, analyse and report cluster randomised trials in medicine and health related research . Hoboken: Wiley. Google Scholar CrossRef Search ADS   Chan, K. S., S. Mestelman, R. Moir, and R. A. Muller. 1996. The voluntary provision of public goods under varying income distributions. Canadian Journal of Economics  29( 1): 54– 69. Google Scholar CrossRef Search ADS   Chan, K. S., R. Godby, S. Mestelman, and R. A. Muller. 2002. Crowding-out voluntary contributions to public goods. Journal of Economic Behavior & Organization  48( 3): 305– 17. Google Scholar CrossRef Search ADS   Coppock, A., A. Guess and J. Ternovski. 2016. When treatments are tweets: A network mobilization experiemnt over twitter. Political Behavior  38: 105– 2018. Google Scholar CrossRef Search ADS   De Wit, A., and R. Bekkers. 2016. Government support and charitable donations: A meta-analysis of the crowding-out hypothesis. Journal of Public Administration Research and Theory  27: 301– 19. De Wit, A., R. Bekkers, and M. Broese van Groenou. 2017. Heterogeneity in crowding-out: When are charitable donations responsive to government support? European Sociological Review  33: 59– 71. Dokko, J. K. 2009. Does the NEA crowd out private charitable contributions to the arts? National Tax Journal  62: 57– 75. Google Scholar CrossRef Search ADS   Eckel, C. C., P. J. Grossman, and R. M. Johnston 2005. An experimental test of the crowding out hypothesis. Journal of Public Economics  89: 1543– 60. Google Scholar CrossRef Search ADS   Galbiati, R., and P. Vertova. 2008. Obligations and cooperative behaviour in public good games. Games and Economic Behavior  64( 1): 146– 70. Google Scholar CrossRef Search ADS   Galbiati, R., and P. Vertova. 2014. How laws affect behavior: Obligations, incentives and cooperative behavior. International Review of Law and Economics  38: 48– 57. Google Scholar CrossRef Search ADS   Galizzi, Mateo and Daniel Navarro-Martinez. Forthcoming. On the external validity of social preference games: A systematic lab-field study. Management Science . Gerber, Alan and Donald Green. 2012. Field experiments: design, analysis, and interpretation . New York: Norton. Gong, S., J. Zhang, P. Zhao, and X. Jiang. 2017. Tweeting as a marketing tool: A field experiment in the TV industry. Journal of Marketing Research  54: 833– 50. Google Scholar CrossRef Search ADS   Grimmelikhuijsen, Stephan and Albert Meijer. 2015. Does twitter increase perceived police legitimacy? Public Administration Review  75: 598– 607. Google Scholar CrossRef Search ADS   Grimmelikhuijsen, S., F. de Vries, and W. Zijlstra. 2018. Breaking bad news without breaking trust: The effects of a press release and newspaper coverage on perceived trustworthiness. Journal of Behavioral Public Administration  1. Gronberg, T. J., R. A. LuccasenIII, T. L. Turocy, and J. B. Van Huyck. 2012. Are tax-financed contributions to a public good completely crowded-out? Experimental evidence. Journal of Public Economics  96( 7–8): 596– 603. Google Scholar CrossRef Search ADS   Güth, W., M. Sutter, and H. Verbon. 2006. Voluntary versus compulsory solidarity: Theory and experiment. Journal of Institutional and Theoretical Economics  162( 2): 347– 63. Google Scholar CrossRef Search ADS   Hayes, Richard and Lawrence Moulton. 2017. Cluster randomised trials . Boca Raton: Chapman & Hall/CRC. Heutel, G. 2014. Crowding out and crowding in of private donations and government grants. Public Finance Review  42: 143– 75. Google Scholar CrossRef Search ADS   Horne, C. S., J. L. Johnson, and D. M. Van Slyke. 2005. Do charitable donors know enough—and care enough—About government subsidies to affect private giving to nonprofit organizations? Nonprofit and Voluntary Sector Quarterly  34: 136– 49. Google Scholar CrossRef Search ADS   Hsu, L. C. 2008. Experimental evidence on tax compliance and voluntary public good provision. National Tax Journal  61( 2): 205– 23. Google Scholar CrossRef Search ADS   Hughes, P., W. Luksetich, and P. Rooney. 2014. Crowding‐Out and fundraising efforts. Nonprofit Management and Leadership  24( 4): 445– 64. Im, T., W. Cho, G. Porumbescu, and J. Park. 2014. Internet, trust in government, and citizen compliance. Journal of Public Administration Research and Theory  24( 3): 741– 63. Google Scholar CrossRef Search ADS   Isaac, R. M., and D. A. Norton. 2013. Endogenous institutions and the possibility of reverse crowding out. Public Choice  156: 253– 84. Google Scholar CrossRef Search ADS   James, Oliver, Sebastian Jilke, and Gregg Van Ryzin, eds. 2017a. Experiments in public management research: challenges and opportunities . Cambridge: Cambridge University Press. Google Scholar CrossRef Search ADS   James, Oliver, Sebastian Jilke, and Gregg Van Ryzin. 2017b. Causal inference and the design and analysis of experiments. In Experiments in public management research: Challenges and opportunities , ed. Oliver James, Sebastian Jilke, and Gregg Van Ryzin, 59– 88. Cambridge: Cambridge University Press. Google Scholar CrossRef Search ADS   James, Oliver, and Asmus Leth Olsen. 2017. Citizens and public performance measures: making sense of performance information. In Experiments in public management research: Challenges and opportunities , ed. Oliver James, Sebastian Jilke, and Gregg Van Ryzin, 270– 89. Cambridge: Cambridge University Press. Google Scholar CrossRef Search ADS   Khanna, J., and T. Sandler. 2000. Partners in giving: The crowding-in effects of UK government grants. European Economic Review  44: 1543– 56. Google Scholar CrossRef Search ADS   Kim, Mirae, and Gregg G. Van Ryzin. 2014. Impact of government funding on donations to arts organizations: A survey experiment. Nonprofit and Voluntary Sector Quarterly  43: 910– 25. Google Scholar CrossRef Search ADS   King, G., J. Pan, and M. E. Roberts. 2013. How censorship in China allows government criticism but silences collective expression. American Political Science Review  107( 2): 1– 18. Google Scholar CrossRef Search ADS   King, G., B. Schneer, and A. White. 2017. How the news media activate public expression and influence national agendas. Science  358: 776– 80. Google Scholar CrossRef Search ADS   Kingma, B. R. 1989. An accurate measurement of the crowd-out effect, income effect, and price effect for charitable contributions. Journal of Political Economy  97: 1197– 207. Google Scholar CrossRef Search ADS   Kobayashi, T. and Y. Ichifuji. 2015. Tweets that matter: Evidence from a randomized field experiment in Japan. Political Communication  32: 574– 93. Google Scholar CrossRef Search ADS   Konow, J. 2010. Mixed feelings: Theories of and evidence on giving. Journal of Public Economics  94(3–4): 279– 97. Google Scholar CrossRef Search ADS   Korenok, O., E. L. Millner, and L. Razzolini. 2012. Are dictators averse to inequality? Journal of Economic Behavior & Organization  82: 543– 47. Google Scholar CrossRef Search ADS   Kramer, A., J. Guillory and J. Hancock. 2014. Experimental evidence of massice-scale emotional contagion through social networks. Proceedings of the National Academy of Science of the United States of America  111: 8788– 90. Google Scholar CrossRef Search ADS   Krasteva, S., and H. Yildirim. 2013. (Un) informed charitable giving. Journal of Public Economics  106: 14– 26. Google Scholar CrossRef Search ADS   Ledyard, J. 1995. Public goods: A survey of experimental research. In Handbook of experimental economics , ed. J. Kagel and A. Roth, 111– 94. Princeton: Princeton University Press. Levitt, Steven and John List. 2007. What do laboratory experiments measuring social preferences reveal about the real world? Journal of Economic Perspectives  21: 153– 74. Google Scholar CrossRef Search ADS   Li, H. and G. Van Ryzin. 2017. A systematic review of experimental studies in public management journals. In Experiments in public management research: challenges and opportunities , ed. James, O., S. Jilke, and G. Van Ryzin. Cambridge: Cambridge University Press. Google Scholar CrossRef Search ADS   Lilley, A., and R. Slonim. 2014. The price of warm glow. Journal of Public Economics  114: 58– 74. Google Scholar CrossRef Search ADS   Linos, E. 2017. More than public service: A field experiment on job advertisements and diversity in the police. Journal of Public Administration Research and Theory  28( 1): 67– 85. Google Scholar CrossRef Search ADS   Lu, J. 2015. Which nonprofit gets more government funding? Nonprofit Management and Leadership  25: 297– 312. Google Scholar CrossRef Search ADS   ———. 2016. The Philanthropic consequence of government grants to nonprofit organizations. Nonprofit Management and Leadership  26: 381– 400. CrossRef Search ADS   Mergel, I. 2012. Social media in the public sector: A guide to participation, collaboration and transparency in the networked world . Hoboken: Wiley: Jossey-Bass. Middleton, Joal and Peter Arronow. 2015. Unbiased estimation of the average treatment effect in cluster-randomized experiments. Statistics, Politics and Policy  6: 39– 75. Google Scholar CrossRef Search ADS   Munger, K. 2017. Tweetment effects on the tweeted: experimentally reducing racist harassment. Political Behavior  39( 3): 629– 49. Google Scholar CrossRef Search ADS   Okten, C., and B. A. Weisbrod. 2000. Determinants of donations in private nonprofit markets. Journal of Public Economics  75: 255– 72. Google Scholar CrossRef Search ADS   Olsen, A. 2017. Adventures into the negativity bias . Paper presented at EGPA conference in Milan, Italy. Ottoni-Wilhelm, M., L. Vesterlund, and H. Xie. 2017. Why do people give? Testing pure and impure altruism. American Economic Review  107: 3617– 33. Google Scholar CrossRef Search ADS   Porumbescu, Gregory. 2015. Comparing the effects of E-Government and social media use on trust in government: Evidence from Seoul, South Korea. Public Management Review  18: 1308– 34. Google Scholar CrossRef Search ADS   Public Religion Research Institute. 2012. Few Americans use social media to connect with their faith communities. http://publicreligion.org/research/2012/08/july-2012-tracking-survey/ (accessed April 30, 2018). Reeson, A. F., and J. G. Tisdell. 2008. Institutions, motivations and public goods: An experimental test of motivational crowding. Journal of Economic Behavior & Organization  68( 1): 273– 81. Google Scholar CrossRef Search ADS   Roberts, R. D. 1984. A positive model of private charity and public transfers. The Journal of Political Economy  92: 136– 48. Google Scholar CrossRef Search ADS   Rose-Ackerman, S. 1981. Do government grants to charity reduce private donations? In Nonprofit firms in a three sector economy , ed. White M. J., 95– 114. Washington, DC: Urban Institute. Ryan, T. 2012. What makes us click? Demonstrating incentives for angry discourse with digital-age field experiments. The Journal of Politics  74: 1138– 52. Google Scholar CrossRef Search ADS   Ryan, T. J., and Brader, T. 2017. Gaffe appeal a field experiment on partisan selective exposure to election messages. Political Science Research and Methods  5: 667– 87. Google Scholar CrossRef Search ADS   Ryan, T. and D. Broockman. 2012. Facebook: A new frontier for field experiments. The Experimental Political Scientist: Newsletter of the ASPA Experimental Section  3: 2– 8. Smith, T. M. 2007. The impact of government funding on private contributions to nonprofit performing arts organizations. Annals of Public and Cooperative Economics  78: 137– 60. Google Scholar CrossRef Search ADS   Statistia. 2017. Number of Facebook users by age in the U.S. as of January 2017. https://www.statista.com/statistics/398136/us-facebook-user-age-groups/ (accessed April 30, 2018). Steinberg, R. 1987. Voluntary donations and public expenditures in a federalist system. The American Economic Review  77: 24– 36. Sutter, M., & H. Weck-Hannemann. 2004. An experimental test of the public goods crowding out hypothesis when taxation is endogenous. FinanzArchiv: Public Finance Analysis  60( 1): 94– 110. Google Scholar CrossRef Search ADS   Suárez, D. F. 2011. Collaboration and professionalization: The contours of public sector funding for nonprofit organizations. Journal of Public Administration Research and Theory  21: 307– 26. Google Scholar CrossRef Search ADS   Teresi, H. and M. Michelson. 2015. Wired to mobilize: The effect of social networking messages on voter turnout. The Social Science Journal  52: 195– 204 Google Scholar CrossRef Search ADS   Thomas, J. C., and G. Streib. 2003. The new face of government: Citizen-initiated contacts in the era of E-Government. Journal of public administration research and theory  13: 83– 102. Google Scholar CrossRef Search ADS   Tinkelman, D. 2010. Revenue interactions: crowding out, crowding in, or neither? In Handbook of research on nonprofit economics and management , ed. Seaman, B. A., and D. R. Young, pp. 18– 41. Northampton, MA: Edward Elgar. Google Scholar CrossRef Search ADS   Wang, J. C., and C. H. Chang. 2013. How online social ties and product-related risks influence purchase intentions: A Facebook experiment. Electronic Commerce Research and Applications  12: 337– 46. Google Scholar CrossRef Search ADS   Warr, P. G. 1982. Pareto optimal redistribution and private charity. Journal of Public Economics  19: 131– 38. Google Scholar CrossRef Search ADS   Wasif, R., and A. Prakash. 2017. Do government and foreign funding influence individual donations to religious nonprofits? A survey experiment in Pakistan. Nonprofit Policy Forum  8( 3): 237– 73. Google Scholar CrossRef Search ADS   Wukich, C., and I. Mergel. 2015. Closing the citizen-government communication gap: Content, audience, and network analysis of government tweets. Journal of Homeland Security and Emergency Management  12: 707– 35. Google Scholar CrossRef Search ADS   Zhang, J., and E. Mao. 2016. From online motivations to ad clicks and to behavioral intentions: An empirical study of consumer response to social media advertising. Psychology & Marketing  33: 155– 64. Google Scholar CrossRef Search ADS   Appendix Table A1. Mean difference tests CTR (independent two-sample t-test; n = 600)   Control  Placebo  Treatment                Mean  Standard error  Mean  Standard error  Mean  Standard error  Mean difference; control vs. treatment  p value  Mean difference; placebo versus treatment  p value  Mean difference; control versus placebo  p value  CTR  0.50  0.03  0.44  0.03  0.50  0.04  0.00  0.99  0.06  0.21  −0.06  0.17    Control  Placebo  Treatment                Mean  Standard error  Mean  Standard error  Mean  Standard error  Mean difference; control vs. treatment  p value  Mean difference; placebo versus treatment  p value  Mean difference; control versus placebo  p value  CTR  0.50  0.03  0.44  0.03  0.50  0.04  0.00  0.99  0.06  0.21  −0.06  0.17  View Large Table A1. Mean difference tests CTR (independent two-sample t-test; n = 600)   Control  Placebo  Treatment                Mean  Standard error  Mean  Standard error  Mean  Standard error  Mean difference; control vs. treatment  p value  Mean difference; placebo versus treatment  p value  Mean difference; control versus placebo  p value  CTR  0.50  0.03  0.44  0.03  0.50  0.04  0.00  0.99  0.06  0.21  −0.06  0.17    Control  Placebo  Treatment                Mean  Standard error  Mean  Standard error  Mean  Standard error  Mean difference; control vs. treatment  p value  Mean difference; placebo versus treatment  p value  Mean difference; control versus placebo  p value  CTR  0.50  0.03  0.44  0.03  0.50  0.04  0.00  0.99  0.06  0.21  −0.06  0.17  View Large © The Author(s) 2018. Published by Oxford University Press on behalf of the Public Management Research Association. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com. This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices) http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Journal of Public Administration Research and Theory Oxford University Press

Using Large-Scale Social Media Experiments in Public Administration: Assessing Charitable Consequences of Government Funding of Nonprofits

Loading next page...
 
/lp/ou_press/using-large-scale-social-media-experiments-in-public-administration-f5DoU6wNvC
Publisher
Oxford University Press
Copyright
© The Author(s) 2018. Published by Oxford University Press on behalf of the Public Management Research Association. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.
ISSN
1053-1858
eISSN
1477-9803
D.O.I.
10.1093/jopart/muy021
Publisher site
See Article on Publisher Site

Abstract

Abstract In this article, we introduce and showcase how social media can be used to implement experiments in public administration research. To do so, we pre-registered a placebo-controlled field experiment and implemented it on the social media platform Facebook. The purpose of the experiment was to examine whether government funding to nonprofit organizations has an effect on charitable donations. Theories on the interaction between government funding and charitable donations stipulate that government funding of nonprofit organizations either decreases (crowding-out), or increases (crowding-in) private donations. To test these competing theoretical predictions, we used Facebook’s advertisement facilities and implemented an online field experiment among 296,121 Facebook users nested in 600 clusters. Through the process of cluster-randomization, groups of Facebook users were randomly assigned to different nonprofit donation solicitation ads, experimentally manipulating information cues of nonprofit funding. Contrary to theoretical predictions, we find that government funding does not seem to matter; providing information about government support to nonprofit organizations neither increases nor decreases people’s propensity to donate. We discuss the implications of our empirical application, as well as the merits of using social media to conduct experiments in public administration more generally. Finally, we outline a research agenda of how social media can be used to implement public administration experiments. Introduction Using social media has become an increasingly common activity as platforms like Facebook have become ubiquitous. For instance, to date there are 218 million Facebook users in the United States (Statistia 2017) with one-third of the US adult population logging into Facebook at least once a day (Public Religion Research Institute 2012). Social media platforms like Facebook or Twitter provide brand-new opportunities to implement online experiments studying citizen–state interactions. However, within public administration this potential has been largely untapped.1 Therefore, the purpose of this article is to introduce and showcase how social media, particularly Facebook, can be used to implement experiments in public administration research. As a consequence of digitization, large-scale social media experiments have been used increasingly in neighboring fields like political science, economics, or marketing (for an overview see Aral 2016). Examples include a wide range of topics, including political behavior (e.g., Bond et al. 2012), advertising (e.g., Bakshy et al. 2012a), product pricing (e.g., Ajorlou, Jabdabaie, and Kakhbod 2016), information propagation (e.g., Bakshy et al. 2012b), or emotional contagion (e.g., Kramer, Guillory and Hancock 2014). This trend is encouraging but it misses an important component of people’s online behavior, namely citizen–state interactions. Indeed, the advent of e-government and an increased presence of government agencies on social media platforms has led to the rise of online citizen–state interactions (Thomas and Streib 2003; Wukich and Mergel 2015). These interactions range from gathering information, for example, about how to fill out online applications, to crisis communication, and even complaints about poor services. In this study, we illustrate how to implement large-scale social media experiments on Facebook by examining interactions between nonprofit organizations seeking donations and their potential donors. In particular, we assess whether changes in government funding affect the levels of charitable income of nonprofit organizations. Theoretically, two competing mechanisms have been distinguished in explaining why levels of government funding may have an effect on private donations to nonprofit organizations. The crowding-out perspective argues that government funding would decrease people’s willingness to donate because donors as taxpayers perceive government funding as a substitution to their donations (Andreoni 1989; Warr 1982; Kim and Van Ryzin 2014). If they contributed already via taxes, why should they give in addition to them? Therefore, the crowding-out model predicts a decrease in private donations as a result of government funding. In contrast, the signaling model of crowding-in suggests that government funding is used by potential donors as an imperfect signal of an organization’s effectiveness (e.g., Borgonovi and O’Hare 2004; Rose-Ackerman 1981). In the absence of complete information about how a nonprofit organization will perform and operate with the funds at its disposal, government funding serves as an organization’s “quality stamp,” signaling the organization is not only trustworthy but also effective because it managed to receive competitive government grants. The crowding-in perspective, therefore, predicts an increase in private donations as a result of government funding. In this study, we test these somewhat competing claims in the context of a large-scale social media experiment. Conducted in a naturalistic setting, social media experiments, similar to conventional field experiments, combine high levels of internal validity with external validity. This allows us to test whether government funding crowds-in, or crowds-out, private donations. We implemented a field experiment on the social media platform Facebook by assigning clusters of approximately 300,000 Facebook users to donation solicitations of groups of real food banks. Using a pre-registered, placebo-controlled between-subjects design, groups of users were randomly allocated to three experimental conditions: (1) the control group (i.e., no funding information), (2) the placebo group (i.e., donation funded), and (3) the treatment group (i.e., government funded). As outcome measures, we monitored people’s revealed donation intentions by their click-through-rates (i.e., the frequency people clicked on the links in the ad solicitations), but also other behavioral measures such as website visits. Consistently, we find no direct evidence for either model, suggesting that public and private funding streams of nonprofit organizations do not seem to interact in the real world. In addition to these findings, we provide an overview of social media experiments and how they can be implemented in public administration research, including an agenda for studying online citizen–state interactions using large-scale social media experiments. The remainder of the study is as follow: in the next section, we review empirical applications of social media experiments in neighboring fields to provide an overview of the applicability of social media to conduct experiments in public administration. We then discuss our empirical application by first reviewing the literature on the crowding-out and crowding-in hypotheses. On this basis, we introduce our experimental research design and report the results of the experiment subsequently. In the final section, we draw implications for public administration research and practice from both, our empirical application and the review of innovative social media experiments. Conducting Social Media Experiments Before turning to the empirical application in this article, we provide an overview of the potential of social media to conduct experiments. Recent years have seen an increase in online field experiments implemented on social media platforms (Aral 2016). Indeed, companies like Amazon, Google, and Facebook constantly perform small experiments on their clients, for example through randomly altered website content where two different versions of the same website, online ad, or any other online parameter are randomly assigned to service users—a procedure marketers commonly refer to as A/B testing. In the past, social scientists have collaborated with major social media platforms to implement experiments. For example, Bond et al. (2012) implemented a 61-million-person political mobilization experiment on Facebook; similarly, Kramer, Guillory, and Hancock (2014) have implemented an online experiment to study emotional contagion among 690,000 Facebook users. Most studies on Facebook include advertisement related topics, however. For example, Bakshy and colleagues (2012a) study the effectiveness of social cues (i.e., peers’ associations with a brand) on consumer responses to ads for more than 5 million Facebook users. In another study, Bakshy et al. (2012b) look at the randomized exposure of links shared by peers of more than 250 million Facebook users and how it affects information sharing behavior on Facebook. In all of these cases, researchers had to work closely with Facebook to implement the process of randomization at the individual level of Facebook users. But, this would also mean that experimenting on Facebook would be limited to those with industry contacts. In the following, we report from recent experiments that have been conducted without industry collaboration. We aim to showcase how social media platforms like Facebook or Twitter can be used by scholars or government agencies to implement experiments in a relatively straightforward manner. Ryan (2012) was one of the first social scientists to use Facebook’s advertisement facilities to conduct research without having to collaborate with Facebook directly. Similar to the empirical application we report in this article, he randomly assigned clusters of individuals to different advertisements instead of randomizing on the user level (see Teresi and Michelson 2015 for an alternative approach2). To do so, he used Facebook’s advertisement facilities, which allow targeting ads to Facebook users on a number of demographic characteristics, such as age and gender, but also zip code (see also Ryan and Brockman 2012). Based on these user parameters, researchers can predetermine clusters of users and randomly allocate them to varying ad content. This is what Ryan did in his study. In particular, he looked at how advertisements that evoke emotions such as anger or anxiety affect information seeking behavior. He then used cluster-level click-through-rates as a dependent variable. Across three studies and more than 1.8 million impressions grouped into 360 clusters in total, he found consistent evidence that political advertisements that ought to evoke anger increase users’ proclivity to click through to a website. In other words, anger makes us click. A similar methodological approach was also used by Ryan and Brader (2017) who studied partisan selective exposure to election messages during the 2012 US presidential elections, using a total of 846 clusters of Facebook users. Similar applications also exist in fields like marketing or economics. Aral and Walker (2014) for instance report from an experiment conducted with 1.3 million users of a Facebook application to test how social influence in online networks affects consumer demand. They experimentally manipulated users’ network embeddedness and the strength of their social ties, finding that both increase influence in social networks. Wang and Chang (2013) studied a similar topic looking at whether social ties and product-related risks influence purchase intentions of Facebook users that were recruited via an online application. Although they found that online tie strength leads the higher purchase intentions, product-related risks had no direct effect on purchase intentions. In another Facebook study by Broockman and Green (2013), users were exposed to different types of political candidate advertisements over the course of 1 week. Like Ryan (2012), they randomized clusters of individuals, instead of individuals themselves. However, since they had access to public voter registries, they targeted 32,000 voters, which they assigned to 1,220 clusters across 18 age ranges, 34 towns, and 2 genders. These clusters of Facebook users were assigned to one of four experimental conditions: a control group with no advertisement, and three different types of advertisements that ought to increase Facebook users’ name recognition of the candidate. The innovation that Brookman and Green’s study introduced was that they used contact information from public voter records to gather public opinion data from these individual voters through telephone interviews later on. Since the creation of clusters was done on the basis of assigning 32,000 registered voters to 1,220 clusters, they had detailed contact information of registered voters that belong to each respective cluster. In other words, they were able to link cluster assignment on Facebook with attitudinal outcomes from survey data, such as candidate’s name recognition, positive impression of the candidate, whether people voted for the candidate, and whether they recall having seen the ad. Social media experiments exist outside of Facebook also. Gong et al. (2017) conducted a large-scale experiment among the social microblogging service Sina Weibo (i.e., the “Chinese Twitter”). They examined the return-on-investment of company tweets on viewers of TV shows. To do so, they randomly assigned 98 different TV shows into three experimental conditions: (1) the control group, where there is no tweet sent out about the particular TV shows; (2) the tweet condition, where each show is tweeted by the company; and (3) a tweet and retweet condition, where each show is tweeted by a company and retweeted by a so-called “influencer.” TV show viewing percentages were used as an outcome measure, finding that both tweeting and tweeting coupled with retweeting boost TV show views relative to the shows in the control group. In other words, social media efforts of TV companies result in a significant increase in viewers. They also found that retweets of influencers are more effective in generating new viewers than tweets by the companies. In another Twitter experiment, Coppock, Guess, and Ternovski (2016) looked at online mobilization behavior. In particular, authors were interested in whether Twitter users could be encouraged to sign a petition. To do so, they randomly divided 8,500 followers of a US nonprofit advocacy group into three experimental conditions. In the first stage, the nonprofit organization published a tweet in which its followers were encouraged to sign a petition. All three groups were exposed to the public tweet. In the second stage, a treatment condition received a direct message with a similar request, referring to them as “followers,” another treatment condition got the same direct message, but they were referred to as “organizers,” whereas the control group received no direct message. On this basis, authors checked whether subjects either retweeted or signed the petition. Other notable examples of using social media platforms like Twitter to implement experiments involve studying social media censorship in China (King, Pan, and Roberts 2013), the effectiveness of repeated political messages on twitter followers of politicians (Kobayashi and Ichifuji 2015), the effectiveness of news media (King, Schneer, and White 2017), or racist online harassment (Munger 2017). The aforementioned examples provide rich inspiration for conducting social media experiments in public administration research. Social media experiments have the distinct advantage that they combine the internal validity of experiments with an increased realism and external validity. In this sense, they are a subtype of conventional field experiments, which are conducted in an online environment where people interact via social media. In addition, social media experiments can easily be conducted on large-scale samples using a variety of unobtrusive outcome measures to assess respondents’ revealed behaviors. They are therefore a viable option to complement survey-based experiments that often employ stated preferences (i.e., attitudes, evaluative judgments, or behavioral intentions), which make up the majority type of experiments implemented in public administration to date (Li and Van Ryzin 2017). People increasingly interact with government and government agencies using social media platforms like Twitter and Facebook (Mergel 2012). Scholars and government agencies alike can implement social media experiments to test the effectiveness of using these relatively new channels of communication and information provision. Examples may include assessing whether providing information on social media about the performance of government agencies affect citizen trust in those agencies, or may lead citizens to desirable behaviors, including coproduction. Indeed, implementing such innovative experimental designs in the context of online citizen–government interactions may be a viable avenue for future experimentation in public administration research. In the following, we introduce an empirical application of an online social media experiment that examines in how far government funding and charitable donation intentions interact. Empirical Application: How Government Funding and Private Donations Interact An impressive body of literature has emerged from various disciplines that focus on the issue of whether government funding would displace (crowding-out effect) or leverage (crowding-in effect) private contributions to nonprofit organizations. The impact of government funding on private donations to nonprofits largely rests on how potential donors and nonprofits themselves strategically respond to government funding of nonprofit activities (Tinkelman 2010; Lu 2016). We focus on the strategic responses of private donors in the present analysis: how would donors change their levels of charitable giving when a nonprofit organization is supported by government funding. The literature distinguishes two contrasting models of how government-funded nonprofits are perceived, which ultimately affects charitable donations via processes of crowding-in or crowding-out. Early crowding-out theory assumes that private donors are altruistic and care about the optimal level of public goods provision. Donors as taxpayers would consider government funding as their contributions through taxation and thus perceive it as a perfect substitute for voluntary donations. In this way, increases in government support would lower the need for additional private contributions. Therefore, when a nonprofit receives more support from the government, private donors would consciously reduce their giving to this organization. As a result, there is a dollar-for-dollar replacement between private giving and government funding (Roberts 1984; Warr 1982). This pure altruism assumption was later challenged by Andreoni’s (1989) model of impure altruism, which predicts that donors are also motivated by a “warm-glow”—the utility from the act of giving to help others. In this impure altruism line of reasoning, government funding and private giving would not completely substitute each other. As a result, there may exist a crowding-out effect between these two funding sources, but the magnitude of the effect is less than the dollar-for-dollar model that pure altruism would predict. On the other hand, private donors might consider government funding favorably and become more willing to contribute to government-funded nonprofits because they perceive them as more competent and/or needy. Crowding-in theory proposes that government funding may stimulate charitable contributions in two ways. First, when donors do not have complete knowledge concerning beneficiary nonprofits and their programs, government funding serves as a direct signal of the nonprofit’s quality and reliability (Rose-Ackerman 1981). Indeed, to be funded by government agencies, nonprofit organizations have to go through a competitive merit-based selection process and meet financial and programmatic requirements (Lu 2015; Suárez 2011). Therefore, the receipt of government funding can be perceived by uninformed donors as an indicator of trustworthiness and competence. Second, government funding also is considered as a signal of unmet social needs, calling for more donor attention and further facilitating the leveraging effect of government funding (Brooks 1999; Okten and Weisbrod 2000). There exist a rich body of empirical studies in support of the crowding-out hypothesis (e.g., Andreoni 1993; Andreoni and Payne 2011; Brooks 2000; De Wit, Bekkers, and Broese van Groenou 2017; Dokko 2009; Hughes, Luksetich, and Rooney 2014; Kingma 1989; Steinberg 1987) and the crowding-in model (e.g., Borgonovi and O’Hare 2004; De Wit and Bekkers 2016; Heutel 2014; Khanna and Sandler 2000; Lu 2016; Okten and Weisbrod 2000; Smith 2007). Most recently, De Wit and Bekkers (2016) and Lu (2016) respectively employed meta-analytical techniques to aggregate existing empirical evidence on crowding-in and crowding-out. Both studies conclude a significant positive association between government funding and private donations, even though the magnitude of the relationship is trivial. The above-mentioned body of literature on crowding-out and crowding-in greatly advances our understanding of the complex interaction between government funding and private donations. However, it generally suffers from two limitations. First, the majority of existing empirical literature testing the crowding-in or crowding-out effect employs observational data. Although observational studies enable scholars to explore the association between the two revenue sources, drawing causal inferences remains challenging (Blom-Hansen, Morton, and Serritzlew 2015; James, Jilke, and Van Ryzin 2017a). Second, both crowding-in and crowding-out lines of reasoning assume that potential donors possess perfect information about the nonprofits they might want to donate to, especially whether these organizations are funded by government and to what extent. However, this assumption might not be true in the real world (De Wit et al. 2017; Horne, Johnson, and Van Slyke 2005; Krasteva and Yildirim 2013). For example, Horne, Johnson, and Van Slyke (2005) used public opinion data to demonstrate that individual donors do not necessarily have complete information on the financial structures of their beneficiary organizations and subsequently link donation decisions to the levels of government funding. In recent years, scholars began to employ experimental designs to address these two limitations (i.e., endogeneity and imperfect information) in the crowding-out and crowding-in literature (e.g., Eckel, Grossman, and Johnston 2005; Kim and Van Ryzin 2014; Ottoni-Wilhelm, Vesterlund, and Xie 2017; Wasif and Prakash 2017). Existing experimental studies testing crowding-out/-in effects usually include manipulations of the existence and the level of direct government support to beneficiaries, and then measure variations in of subjects’ donations. Methodologically, experimental designs are advantageous over observational settings in terms of their internal validity when testing crowding-out/-in effects because experimental studies create a controlled environment of information exchange to rule out confounding factors. As a result, scholars are provided with more direct evidence of the causal linkage between government support and charitable giving (Blom-Hansen, Morton, and Serritzlew 2015; James, Jilke, and Van Ryzin 2017a). Table 1 reviews the experimental studies of crowding-out/-in effects to date, including type, setting, and results. As can be seen in Table 1, most experimental studies employ laboratory experimental designs, primarily using two specific experimental paradigms. One is the public goods game (e.g., Andreoni 1993; Eckel et al. 2005; Isaac and Norton 2013) and the other type of experimental setting employed in the literature is the dictator game (e.g., Blanco, Lopez, and Coleman 2012; Konow 2010; Korenok, Millner, and Razzolini 2012). Despite different experimental paradigms, most of the laboratory experiments report a partial crowding-out effect between government funding and charitable contributions (see also De Wit and Bekkers 2016). Table 1. Experimental Studies on Crowding-in and Crowding-out Effects Reference  Type of experiment  Experimental setting  Subjects  Finding  Andreoni (1993)  Lab  Public goods game  Students  Crowding-out  Chan et al. (1996)  Lab  Public goods game  Students  No effect  Bolton and Katok (1998)  Lab  Dictator game  Students  Crowding-out  Chan et al. (2002)  Lab  Public goods game  Students  Crowding-out  Sutter and Weck-Hannemann (2004)  Lab  Public goods game  Students  Crowding-out  Eckel et al. (2005)  Lab  Public goods game  Students  Crowding-out  Güth et al. (2006)  Lab  Public goods game  Students  Crowding-out  Galbiati and Vertova (2008)  Lab  Public goods game  Students  Crowding-in  Hsu (2008)  Lab  Public goods game  Students  Crowding-out  Reeson and Tisdell (2008)  Lab  Public goods game  Students  Crowding-out  Konow (2010)  Lab  Dictator game  Students  Crowding-out  Blanco et al. (2012)  Lab  Dictator game  Tourists  No effect  Gronberg et al. (2012)  Lab  Public goods game  Students  Crowding-out  Korenok et al. (2012)  Lab  Dictator game  Students  Crowding-out  Issac and Norton (2013)  Lab  Public goods game  Students  Crowding-out  Lilley and Slonim (2014)  Lab  Contribution decision making  Students  Crowding-out  Galbiati and Vertova (2014)  Lab  Public goods game  Students  Crowding-in  Kim and Van Ryzin (2014)  Survey  Hypothetical vignette  General population sample (US)  Crowding-out  Wasif and Prakash (2017)  Survey  Hypothetical vignette  General population sample (Pakistan)  No effect  Ottoni-Wilhelm et al. (2017)  Lab  Contribution decision making  Students  Crowding-out  Reference  Type of experiment  Experimental setting  Subjects  Finding  Andreoni (1993)  Lab  Public goods game  Students  Crowding-out  Chan et al. (1996)  Lab  Public goods game  Students  No effect  Bolton and Katok (1998)  Lab  Dictator game  Students  Crowding-out  Chan et al. (2002)  Lab  Public goods game  Students  Crowding-out  Sutter and Weck-Hannemann (2004)  Lab  Public goods game  Students  Crowding-out  Eckel et al. (2005)  Lab  Public goods game  Students  Crowding-out  Güth et al. (2006)  Lab  Public goods game  Students  Crowding-out  Galbiati and Vertova (2008)  Lab  Public goods game  Students  Crowding-in  Hsu (2008)  Lab  Public goods game  Students  Crowding-out  Reeson and Tisdell (2008)  Lab  Public goods game  Students  Crowding-out  Konow (2010)  Lab  Dictator game  Students  Crowding-out  Blanco et al. (2012)  Lab  Dictator game  Tourists  No effect  Gronberg et al. (2012)  Lab  Public goods game  Students  Crowding-out  Korenok et al. (2012)  Lab  Dictator game  Students  Crowding-out  Issac and Norton (2013)  Lab  Public goods game  Students  Crowding-out  Lilley and Slonim (2014)  Lab  Contribution decision making  Students  Crowding-out  Galbiati and Vertova (2014)  Lab  Public goods game  Students  Crowding-in  Kim and Van Ryzin (2014)  Survey  Hypothetical vignette  General population sample (US)  Crowding-out  Wasif and Prakash (2017)  Survey  Hypothetical vignette  General population sample (Pakistan)  No effect  Ottoni-Wilhelm et al. (2017)  Lab  Contribution decision making  Students  Crowding-out  View Large Table 1. Experimental Studies on Crowding-in and Crowding-out Effects Reference  Type of experiment  Experimental setting  Subjects  Finding  Andreoni (1993)  Lab  Public goods game  Students  Crowding-out  Chan et al. (1996)  Lab  Public goods game  Students  No effect  Bolton and Katok (1998)  Lab  Dictator game  Students  Crowding-out  Chan et al. (2002)  Lab  Public goods game  Students  Crowding-out  Sutter and Weck-Hannemann (2004)  Lab  Public goods game  Students  Crowding-out  Eckel et al. (2005)  Lab  Public goods game  Students  Crowding-out  Güth et al. (2006)  Lab  Public goods game  Students  Crowding-out  Galbiati and Vertova (2008)  Lab  Public goods game  Students  Crowding-in  Hsu (2008)  Lab  Public goods game  Students  Crowding-out  Reeson and Tisdell (2008)  Lab  Public goods game  Students  Crowding-out  Konow (2010)  Lab  Dictator game  Students  Crowding-out  Blanco et al. (2012)  Lab  Dictator game  Tourists  No effect  Gronberg et al. (2012)  Lab  Public goods game  Students  Crowding-out  Korenok et al. (2012)  Lab  Dictator game  Students  Crowding-out  Issac and Norton (2013)  Lab  Public goods game  Students  Crowding-out  Lilley and Slonim (2014)  Lab  Contribution decision making  Students  Crowding-out  Galbiati and Vertova (2014)  Lab  Public goods game  Students  Crowding-in  Kim and Van Ryzin (2014)  Survey  Hypothetical vignette  General population sample (US)  Crowding-out  Wasif and Prakash (2017)  Survey  Hypothetical vignette  General population sample (Pakistan)  No effect  Ottoni-Wilhelm et al. (2017)  Lab  Contribution decision making  Students  Crowding-out  Reference  Type of experiment  Experimental setting  Subjects  Finding  Andreoni (1993)  Lab  Public goods game  Students  Crowding-out  Chan et al. (1996)  Lab  Public goods game  Students  No effect  Bolton and Katok (1998)  Lab  Dictator game  Students  Crowding-out  Chan et al. (2002)  Lab  Public goods game  Students  Crowding-out  Sutter and Weck-Hannemann (2004)  Lab  Public goods game  Students  Crowding-out  Eckel et al. (2005)  Lab  Public goods game  Students  Crowding-out  Güth et al. (2006)  Lab  Public goods game  Students  Crowding-out  Galbiati and Vertova (2008)  Lab  Public goods game  Students  Crowding-in  Hsu (2008)  Lab  Public goods game  Students  Crowding-out  Reeson and Tisdell (2008)  Lab  Public goods game  Students  Crowding-out  Konow (2010)  Lab  Dictator game  Students  Crowding-out  Blanco et al. (2012)  Lab  Dictator game  Tourists  No effect  Gronberg et al. (2012)  Lab  Public goods game  Students  Crowding-out  Korenok et al. (2012)  Lab  Dictator game  Students  Crowding-out  Issac and Norton (2013)  Lab  Public goods game  Students  Crowding-out  Lilley and Slonim (2014)  Lab  Contribution decision making  Students  Crowding-out  Galbiati and Vertova (2014)  Lab  Public goods game  Students  Crowding-in  Kim and Van Ryzin (2014)  Survey  Hypothetical vignette  General population sample (US)  Crowding-out  Wasif and Prakash (2017)  Survey  Hypothetical vignette  General population sample (Pakistan)  No effect  Ottoni-Wilhelm et al. (2017)  Lab  Contribution decision making  Students  Crowding-out  View Large In addition to laboratory experiments, there are a few experimental studies that employ survey experiments to test the crowding-in and crowding-out propositions. For example, Kim and Van Ryzin (2014) conducted an online survey experiment with 562 participants and found that an arts nonprofit with government funding would receive about 25% less private donations than an identical hypothetical organization without government funding. In contrast, Wasif and Prakash’s (2017) survey experiment with 530 respondents in Pakistan reported that federal funding would not change respondents’ willingness to donate to a hypothetical faith-based educational nonprofit. When meta-analyzing results from experimental studies only, De Wit and Bekkers (2016) find substantially different results compared to observational studies, with experimental studies showing a considerable crowding-out effect and nonexperimental studies a very small crowding-in effect. There are two potential explanations for these differences. A first possibility would be that observational studies on crowding-out/-in are plagued by endogeneity, and hence the discrepancies in results may be a product of the comparatively poor internal validity of observational research designs. A second possibility would be that findings predominately produced within stylized settings such as economic games or hypothetical scenarios may hardly extrapolate beyond the laboratory. Or in other words, people may behave differently in lab and survey experiments as in the real world. Indeed, a recent systematic comparison between laboratory and field experiments concluded that the ability of stylized experiments to extrapolate social preferences from the lab to the field is limited, at best (Galizzi and Navarro-Martinez, Forthcoming; see also Levitt and List 2007). We respond to these competing interpretations by implementing a naturalistic field experiment using social media. Method In the following, we report the experimental design for this study. Doing so, we follow James, Jilke, and Van Ryzin’s (2017b) recommended reporting guidelines for experiments in public administration research. Experimental Design, Subject Recruitment, and Treatment We designed a large-scale placebo-controlled online field experiment on the social networking platform Facebook by purchasing ads using Facebook’s advertisement facilities. These ads were designed to solicit donations to human service nonprofits in New York City (NYC). We randomly allocated groups of Facebook users to three different ads of local human service organizations. Depending on experimental conditions, these ads included information cues about (1) government funding (treatment group), (2) donation funding (placebo group), or (3) no such information (control group) as depicted in Figure 1. The experimental design was pre-registered at the AEA RCT registry3 and received ethics approval from Rutgers University’s Institutional Review Board. Figure 1. View largeDownload slide Experimental Design. Figure 1. View largeDownload slide Experimental Design. We chose to target Facebook users who live in NYC because this allowed us to leverage local information about government funding of human service organizations (i.e., real food banks in NYC) in the experimental design. All ads included a donation appeal, encouraging Facebook users to donate to local food banks through a webpage they could access by clicking on the ad. Once clicked, and depending on the experimental condition, users were directed to one of three web pages where Food Banks from NYC4 were listed and hyperlinked. Facebook users in the control condition (i.e., no funding information) were directed to the Rutgers Observatory of Food Banks webpage. Here, all NYC Food Banks were listed and hyperlinked. Those subjects who were allocated to the placebo condition (i.e., donation-funded), were directed to the Rutgers Observatory of Donation-Funded Food Banks webpage. The presence of a placebo condition (i.e., donation-funded) allows us to test whether any differences between treatment and control is due to the mere provision of funding information. On this website, all donation-funded5 food banks from NYC were listed and hyperlinked. In the treatment condition (i.e., government-funded), Facebook users were sent to the Rutgers Observatory of Government-Funded Food Banks website, where all Food Banks that were classified as government-funded were listed and hyperlinked. Unique ads were developed for the experiment. Figure 2 shows the ads that were randomly presented to groups of Facebook users. Taking the ad of the control condition as a baseline, the treatment and placebo ads look exactly like the control ad, except for four important differences: Figure 2. View largeDownload slide Facebook Ads (Control versus Placebo versus Treatment Conditions). Figure 2. View largeDownload slide Facebook Ads (Control versus Placebo versus Treatment Conditions). 1) A different sponsor was used: i. Rutgers Observatory of Food Banks for the control group; ii. Rutgers Observatory of Donation-Funded Food Banks for the placebo group; iii. Rutgers Observatory of Government-Funded Food Banks for the treatment group. 2) The ad’s headline was changed from “Donate to Food Banks in your area” to “Donate to government-funded Food Banks in your area,” or “Donate to donation-funded Food Banks in your area” respectively. 3) The heading under the image changed from “Food Banks in NYC” to “Government-funded Food Banks in NYC,” or “Donation-funded Food Banks in NYC” respectively. 4) The two-sentence-description used “government-funded Food Banks in NYC” and “donation-funded Food Banks in NYC” instead of “Food Banks in NYC.” Cluster Randomization Randomization into experimental conditions was not performed on the individual level, but on the level of groups of Facebook users (see also Ryan 2012 for a similar approach). This was done because Facebook’s commercial advertisement facilities do not permit different ads to be randomized on the individual level (Ryan and Brockman 2012). Instead, users were classified into groups. This approach makes use of Facebook’s ability to target users based on demographics. More precisely, we stratified Facebook users into clusters of users by zip code, age, and gender. Cluster-randomized experiments are widely used across the social and medical sciences where groups of individuals—for example, people living in an area, or classmates—are jointly randomized into experimental conditions (e.g., Campbell and Walters 2014; Hayes and Moulton 2017). For our experiment, we randomized demographic clusters of Facebook users, rather than users nested within these clusters. Doing so, we first collected all 177 NYC zip codes and randomly chose 100. In a second step, we grouped respondents of 18 to 60 years into three age categories (18–27, 28–38, and 39–60). We composed these three groups to have an approximately equal “potential reach” across strata (potential reach refers to the approximate number of Facebook users that can be exposed to an ad). Each of the three groups has a potential reach of about 1.5 to 1.6 million users who live in NYC. Next, we created 600 clusters by taking 100 zip codes * 3 age categories * 2 genders. These 600 clusters were randomized into one of the three experimental conditions, so that each experimental condition ended up with 200 clusters with a potential reach of about 1 million Facebook users6. An important concern when analyzing data from cluster-randomized experiments is that clusters vary in size so that the conventional difference-in-means estimator would be biased (Middleton and Arronow 2015). Indeed, the 600 clusters we have come up with vary in (anticipated) size, with a potential reach between fewer than 1,000 to 110,000 Facebook users (mean potential reach of approximately 5,000 users). Therefore, we followed Gerber and Green’s (2012) suggestion and blocked on cluster size (200 groups of 3 clusters each) in the randomization procedure, so that the difference-in-means estimator can be used without risk of bias. The blocked cluster-randomization was performed using Stata 14. The actual implementation of the experiment was done using Facebook’s Adverts Manager interface. We purchased 600 ads and targeted them to specific demographic clusters (stratified by zip code, age categories, and gender). A total of 600 clusters (i.e., 200 per experimental condition) were purchased for $10 each and our ads were constructed to pay per impression7. All ads ran on the same day for 24 h, from 6 a.m. EST on August 21, 2017 until 6 a.m. the following day. Although the sample size of the clusters was pre-determined, the number of Facebook users who have been actually exposed to the ads was not; however, the potential reach across experimental conditions was approximately the same (see footnote 5). This is so because Facebook uses its own “Optimization for Ad Delivery” algorithm (which is the same across experimental conditions) to determine who will be exposed to an ad. In essence, Facebook users are shown ads which—according to Facebook’s algorithm—have the highest probability of resulting in an impression or click.8 The 600 clusters had a joint potential reach of 2,972,500, but 296,121 Facebook users were actually exposed to the ads as a result the Facebook’s ad bidding procedure. Outcomes Our primary outcome of interest is subjects’ revealed intention to donate. Or in other words, we look at whether the ads encourage people to click on them. We interpret link clicks as people’s intention to donate because the ads explicitly solicit donations by encouraging people to click. Indeed, clicking is commonly seen as a crucial antecedent for purchase intentions in the marketing literature (Zhang and Mao 2016). Against this background and given the explicit ad solicitations for donations we used in the ads (i.e., “click on one of the listed websites to make a donation”), we assume that ad clicks are a motivational precursor for donation intentions. Doing so, we use the so-called unique-outbound-link-click-rate per cluster as our primary dependent variable. The unique-outbound-click-rate (also referred to as click-through-rate) denotes a cluster’s actual reach (i.e., the number of people who saw our ads at least once) divided through its unique outbound clicks (i.e., the number of people who performed a click that took them off Facebook-owned property). In our case, the outbound click led users to one of three web pages—depending on the experimental condition they were assigned to—we have developed for this study. On these web pages, which were labeled as either the Rutgers Observatory of Food Banks (control group), the Rutgers Observatory of Food Banks (placebo group), or the Rutgers Observatory of Government-Funded Food Banks (treatment group), subjects were informed about the purpose of the study and provided with a hyperlinked list of either government-funded, donation-funded, or all food banks in NYC. As a secondary, though not pre-registered, measure, we analyze unique page views (i.e., the number of times people visited their assigned Observatory webpage at least once) as provided by Google Analytics for the three web pages. We calculated the percentage of unique page views relative to the number of people who saw the ads at least once. This measure serves as a robustness check for our primary measure of interest as it provides the actual number of users who actually ended up on the web pages. Results The ads we ran on Facebook reached a total 296,121 NYC Facebook users, resulting in a mean click-through-rate (CTR) of 0.48 (standard deviation of 0.49) for a total of 600 clusters (which is our unit of analysis). This means that of the approximately 300,000 Facebook users who saw the ads at least once, 0.48% performed at least one click which was intended to take them off Facebook and to one of the respective Observatory web pages that were designed for this study. To determine whether we find evidence for a crowding-out/-in effect of government funding on donation intentions, we analyzed whether there are systematic differences in users’ click-through-rates between experimental conditions. The left panel in Figure 3 depicts the results of tests on the equality of means between control, placebo (i.e., donation), and treatment (i.e., government) groups. Independent two-sample t-tests were conducted to compare (1) control and treatment conditions (mean difference of 0.00, p = 0.99; Cohen’s d = 0.00), (2) placebo and treatment conditions (mean difference of 0.06, p = 0.21; Cohen’s d = −0.13), and (3) control and placebo conditions (mean difference of −0.06, p = 0.17; Cohen’s d = −0.14) (see also Table A1 in the Appendix). In sum, all three experimental conditions are statistically indistinguishable from each other in terms of their CTRs. In addition, they are of trivial magnitude with very small values in terms of effect sizes (i.e., Cohen’s d). This means that we neither find support for the crowding-out nor the crowding-in hypothesis. In addition, there is also no significant difference between control and placebo conditions. Figure 3. View largeDownload slide Experimental Results. Figure 3. View largeDownload slide Experimental Results. When analyzing unique page views of the Observatory web pages that were created for this study, we used page visit data from Google Analytics, which provided us with the total number of unique page visits for each experimental condition. Therefore, the unit of analysis for this measure is individual Facebook users (n = 296,121). In total, 524 page views were recorded representing 0.18% of our sample of Facebook users.9 In the control condition, there were a total of 162 page visits, representing 0.16% of Facebook users that have been assigned to the control condition. In the placebo and treatment conditions there were respectively a total of 168 (0.17%) and 194 (0.19%) page views (see also the right panel on Figure 3). When analyzing whether these somewhat-minor differences are of systematic nature, we performed x2 tests. The percentage points difference in page visits between control and treatment condition is 0.03 (x2(1) = 2.30, p = 0.13; Odds Ratio = 1.18), and the difference between placebo and treatment condition is 0.02 (x2(1) = 1.16, p = 0.28; Odds Ratio = 1.12). Between control and placebo conditions, the percentage point difference is even smaller (i.e., 0.01) (x2(1) = 1.87, p = 0.66; Odds Ratio = 1.05). In sum, all three experimental conditions are statistically indistinguishable from each other. In addition, differences are small in magnitude with small effect sizes (i.e., Odds Ratios), confirming prior results from analyzing Facebook’s CTRs. We can therefore confidently reject both, the crowding-out and the crowding-in hypotheses. Conclusion This study has developed and implemented a large-scale but relatively low-cost online experiment in the context of public administration research, where about 300,000 Facebook users were randomly assigned to different donation solicitation ads. The study has found no evidence for crowding-in, nor the crowding-out model. In this sense, our results differ from prior experimental studies that stem from the behavioral laboratory or have been produced in the context of hypothetical vignette designs. While our study is certainly not the first that finds neither evidence for the crowding-out, nor the crowding-in hypotheses (see Blanco et al. 2012; Chan et al. 1996; Wasif and Prakash 2017), we provide—to our knowledge—the first experimental evidence that has been produced within a naturalistic context. However, our results have to be taken with a fair grain of salt. Our effective sample size of 600 clusters may not be able to have detected a difference in groups for our secondary outcome measure. Future studies are advised to replicate our design using a larger number of clusters, possibly by stratifying clusters for single age categories (i.e., one for each year) instead of grouping them as we did. In this sense, our findings do not provide a strict theory test of crowding-in versus crowding-out but provide some suggestive evidence that government funding does not seem to matter in the case of online donation solicitations of local food banks. Future studies may extend this to other areas of human service delivery, including more direct cues of government funding because sometimes people do not pay appropriate attention to informational cues. Although there is a long-standing scholarly debate on crowding-out and crowding-in between government funding and private donations, our study adds to this literature that in an online setting, whether a nonprofit is supported by government funding seems not to affect donors’ behaviors. Therefore, it is possible that, from donors’ perspective, government funding and charitable donations are two independent revenue streams. Donors do not necessarily consider a nonprofit’s revenue structure, especially whether it is supported by government funding when making donation decisions. If so, the crowding-out versus crowding-in debate over the last several decades might be overstated. Certainly, we did not examine in the present study how nonprofits strategically modify their fundraising efforts when they receive government funding, which might actually contribute to the crowding-in or crowding-out effect. For example, Andreoni and Payne (2011) argued that the crowding-out effect is mostly driven by nonprofits’ decreased fundraising efforts as a response to government funding. Again, we cannot guarantee the generalizability of our findings from food banks to other service areas. The primary aim of this study was to showcase how public administration scholars and government agencies can utilize social media to conduct experimental research in the realm of online citizen–government interactions. Social media experiments enable public administration researchers to ensure both internal and external validity because they have the distinct advantage that they combine the internal validity of experiments with an increased realism and ecological validity. Furthermore, as we demonstrate in this study, industry contacts are no longer necessary for implementing large-scale social media experiments. Social media service platforms such as Twitter or Facebook can be easily used to test a variety of important questions to public administration scholars and practitioners. Here, online experiments can be conducted on large-scale samples using a variety of unobtrusive outcome measures to assess respondents’ revealed behaviors. A research agenda for applying social media experiments in studying online citizen–state interactions has implications for a wide range of areas. Possible applications may include assessing the effectiveness of different public communication strategies on social media and how these strategies affect, for example, agency reputation. Indeed, bureaucratic reputation is a core topic area in public administration, with studies examining how news media shape agencies’ reputation, including trust (e.g., Grimmelikhuijsen, de Vries, and Zijlstra 2018). Little is known about how social media communication strategies may alter agency reputation. Social media experiments may assess the effect of different types of communication strategies on reputation—examples may include diffusing responsibility attributions for service failure across actors, active agency branding, or just randomly publishing information about administrative procedures. Another possible area of application may include assessing the effectiveness of different targeted job advertisements for government agencies, using, for instance, Facebook’s advertisement facilities to get underrepresented groups to apply. Recent scholarship has exemplified the importance of signaling personal benefits to increase the number of women and people of color to apply for government jobs (e.g., Linos 2017). Testing and refining such targeted job advertisements and/or messages via social media platforms like Facebook may be a great way to further this promising line of scholarship, but also be an excellent evaluation tool in seeking the most effective online recruitment strategy to increase the demographic representation of government agencies. A more obvious area of application may include assessing whether different ways of how public performance information is presented online affects citizens’ attitudes and subsequent behaviors. Indeed, a growing literature in behavioral public administration examines the psychology of performance information (e.g., James and Olsen 2017), assessing whether different ways of presenting numbers affects perceptions of public service performance or satisfaction evaluations of government agencies. This agenda may be extended to social media experiments where large-scale experiments could be conducted on social media platforms to probe for the external validity of prior survey experiments. Social media experiments may also be useful in simply testing the effectiveness of governmental online crisis warning systems by assessing which types of messages on social media are most likely to be shared across users. Such an agenda would have important practical implications for the effectiveness of early crisis warning systems. The emergence of Internet-based social media has made it possible for researchers to easily interact with thousands or even millions of citizens. Implementing such innovative experimental designs in the context of online citizen–government interactions may be a viable avenue for future experimentation in public administration research both, for academics and practitioners alike. We hope this contribution sparks a wider implementation of public administration experiments on social media platforms. FUNDING This study received funding from the Rutgers SPAA Faculty Research Funds AY16-17. This study was supported by a National Research Foundation of Korea Grant from the Korean Government [NRF-2017S1A3A2065838]. Footnotes 1 Despite a number of important studies on social media use in government agencies and online citizen–government interactions (e.g., Mergel 2012; Grimmelikhuijsen and Meijer 2015; Porumbescu 2015; Im et al. 2014), to date no experimental applications on social media platforms exist in public administration scholarship (for a notable exception see a recent conference contribution by Olsen 2017). 2 Teresi and Michelson (2015) randomized individual Facebook users with whom they connected via a Facebook profile (i.e., becoming “friends”) into experimental conditions. While one group of “friends” received mainly apolitical status updates from the host account, the treatment group received political messages about the upcoming 2010 elections. After the election, authors searched for each “friend” in the state list of registered voters using information provided via Facebook’s profile (i.e., names, age, gender, etc.) to examine whether these online get-out-the-vote messages distributed through social media encouraged subjects to vote. 3 AEARCTR-0002345 (see https://www.socialscienceregistry.org/trials/ 2345). 4 The food banks listed on the websites are all real food banks that operate in NYC, as collected through the National Center for Charitable Statistics (NCCS) database. 5 We distinguished donation-funded and government-funded food banks based on the revenue information provided in food banks’ latest 990 forms in the NCCS database. For this study, we defined government-funded food banks as the ones that receive government funding in addition to private donations, and donation-funded food banks as the ones that do not receive government funding but private donations. 6 988,200 in the control group; 985,300 in the placebo group; 999,000 in the treatment group. 7 This makes a total project cost of $6,000. However, from the purchased $6,000, Facebook ran our ads for a total value of $3,299, or a mean value of $5.50 per cluster. 8 Further information can be found here: https://www.facebook.com/business/help/1619591734742116. 9 There is an inconsistency between the number of unique outbound clicks reported by Facebook and actual unique page visits as provided by Google Analytics. This may be the result of some Facebook clicks not reaching its intended destination with users potentially aborting connecting with a website outside Facebook, possibly because of fear of virus-infected websites. References Ajorlou, A., A. Jadbabaie, and A. Kakhbod. 2016. Dynamic pricing in social networks: The word-of-mouth effect. Management Science  64: 971– 79. Google Scholar CrossRef Search ADS   Andreoni, J. 1989. Giving with impure altruism: Applications to charity and Ricardian equivalence. The Journal of Political Economy  97: 1447– 58. Google Scholar CrossRef Search ADS   ———. 1993. An experimental test of the public-goods crowding-out hypothesis. The American Economic Review  83: 1317– 27. Andreoni, J., and A. A. Payne. 2011. Is crowding out due entirely to fundraising? Evidence from a panel of charities. Journal of Public Economics  95: 334– 43. Google Scholar CrossRef Search ADS   Aral, S. 2016. Networked experiments. In The Oxford Handbook of the Economics of Networks , ed. Yann Bramoullé, Andrea Galeotti, and Brian Rogers, New York: Oxford University Press. Aral, S., and D. Walker. 2014. Tie strength, embeddedness, and social influence: A large-scale networked experiment. Management Science  60: 1352– 70. Google Scholar CrossRef Search ADS   Bakshy, E., D. Eckles, R. Yan, and I. Rosenn. 2012a. Social influence in social advertising: evidence from field experiments. arXiv:1206.4327v1. Bakshy, E., I. Rosenn, C. Marlow, and L. Adamic. 2012b. The role of netwroks in information diffusion. arXiv:1207.4145v5. Blanco, E., M. C. Lopez, and E. A. Coleman. 2012. Voting for environmental donations: Experimental evidence from Majorca, Spain. Ecological Economics  75: 52– 60. Google Scholar CrossRef Search ADS   Blom-Hansen, Jens, Rebecca Morton, and Soren Serritzlew. 2015. Experiments in public management research. International Public Management Journal  18: 151– 70. Google Scholar CrossRef Search ADS   Bolton, G. E., and E. Katok. 1998. An experimental test of the crowding out hypothesis: The nature of beneficent behavior. Journal of Economic Behavior & Organization  37( 3): 315– 31. Google Scholar CrossRef Search ADS   Bond, R. M., C. Fariss, J. Jones, A. Kramer, C. Marlow, J. Settle, and J. Fowler. 2012. A 61-million-person experiment in social influence and political mobilization. Nature  489: 295– 98. Google Scholar CrossRef Search ADS   Borgonovi, F., and M. O’Hare. 2004. The impact of the national endowment for the arts in the United States: Institutional and sectoral effects on private funding. Journal of Cultural Economics  28: 21– 36. Google Scholar CrossRef Search ADS   Broockman, David and Donald Green. 2014. Do online advertisements increase political candidates’ name recognition or favorability? Evidence from randomized field experiments. Political Behavior  36: 263– 89. Google Scholar CrossRef Search ADS   Brooks, A. C. 1999. Do public subsidies leverage private philanthropy for the arts? Empirical evidence on symphony orchestras. Nonprofit and Voluntary Sector Quarterly  28: 32– 45. Google Scholar CrossRef Search ADS   ———. 2000. Is there a dark side to government support for nonprofits? Public Administration Review  60: 211– 18. CrossRef Search ADS   Campbell, Michael and Stephen Walters. 2014. How to design, analyse and report cluster randomised trials in medicine and health related research . Hoboken: Wiley. Google Scholar CrossRef Search ADS   Chan, K. S., S. Mestelman, R. Moir, and R. A. Muller. 1996. The voluntary provision of public goods under varying income distributions. Canadian Journal of Economics  29( 1): 54– 69. Google Scholar CrossRef Search ADS   Chan, K. S., R. Godby, S. Mestelman, and R. A. Muller. 2002. Crowding-out voluntary contributions to public goods. Journal of Economic Behavior & Organization  48( 3): 305– 17. Google Scholar CrossRef Search ADS   Coppock, A., A. Guess and J. Ternovski. 2016. When treatments are tweets: A network mobilization experiemnt over twitter. Political Behavior  38: 105– 2018. Google Scholar CrossRef Search ADS   De Wit, A., and R. Bekkers. 2016. Government support and charitable donations: A meta-analysis of the crowding-out hypothesis. Journal of Public Administration Research and Theory  27: 301– 19. De Wit, A., R. Bekkers, and M. Broese van Groenou. 2017. Heterogeneity in crowding-out: When are charitable donations responsive to government support? European Sociological Review  33: 59– 71. Dokko, J. K. 2009. Does the NEA crowd out private charitable contributions to the arts? National Tax Journal  62: 57– 75. Google Scholar CrossRef Search ADS   Eckel, C. C., P. J. Grossman, and R. M. Johnston 2005. An experimental test of the crowding out hypothesis. Journal of Public Economics  89: 1543– 60. Google Scholar CrossRef Search ADS   Galbiati, R., and P. Vertova. 2008. Obligations and cooperative behaviour in public good games. Games and Economic Behavior  64( 1): 146– 70. Google Scholar CrossRef Search ADS   Galbiati, R., and P. Vertova. 2014. How laws affect behavior: Obligations, incentives and cooperative behavior. International Review of Law and Economics  38: 48– 57. Google Scholar CrossRef Search ADS   Galizzi, Mateo and Daniel Navarro-Martinez. Forthcoming. On the external validity of social preference games: A systematic lab-field study. Management Science . Gerber, Alan and Donald Green. 2012. Field experiments: design, analysis, and interpretation . New York: Norton. Gong, S., J. Zhang, P. Zhao, and X. Jiang. 2017. Tweeting as a marketing tool: A field experiment in the TV industry. Journal of Marketing Research  54: 833– 50. Google Scholar CrossRef Search ADS   Grimmelikhuijsen, Stephan and Albert Meijer. 2015. Does twitter increase perceived police legitimacy? Public Administration Review  75: 598– 607. Google Scholar CrossRef Search ADS   Grimmelikhuijsen, S., F. de Vries, and W. Zijlstra. 2018. Breaking bad news without breaking trust: The effects of a press release and newspaper coverage on perceived trustworthiness. Journal of Behavioral Public Administration  1. Gronberg, T. J., R. A. LuccasenIII, T. L. Turocy, and J. B. Van Huyck. 2012. Are tax-financed contributions to a public good completely crowded-out? Experimental evidence. Journal of Public Economics  96( 7–8): 596– 603. Google Scholar CrossRef Search ADS   Güth, W., M. Sutter, and H. Verbon. 2006. Voluntary versus compulsory solidarity: Theory and experiment. Journal of Institutional and Theoretical Economics  162( 2): 347– 63. Google Scholar CrossRef Search ADS   Hayes, Richard and Lawrence Moulton. 2017. Cluster randomised trials . Boca Raton: Chapman & Hall/CRC. Heutel, G. 2014. Crowding out and crowding in of private donations and government grants. Public Finance Review  42: 143– 75. Google Scholar CrossRef Search ADS   Horne, C. S., J. L. Johnson, and D. M. Van Slyke. 2005. Do charitable donors know enough—and care enough—About government subsidies to affect private giving to nonprofit organizations? Nonprofit and Voluntary Sector Quarterly  34: 136– 49. Google Scholar CrossRef Search ADS   Hsu, L. C. 2008. Experimental evidence on tax compliance and voluntary public good provision. National Tax Journal  61( 2): 205– 23. Google Scholar CrossRef Search ADS   Hughes, P., W. Luksetich, and P. Rooney. 2014. Crowding‐Out and fundraising efforts. Nonprofit Management and Leadership  24( 4): 445– 64. Im, T., W. Cho, G. Porumbescu, and J. Park. 2014. Internet, trust in government, and citizen compliance. Journal of Public Administration Research and Theory  24( 3): 741– 63. Google Scholar CrossRef Search ADS   Isaac, R. M., and D. A. Norton. 2013. Endogenous institutions and the possibility of reverse crowding out. Public Choice  156: 253– 84. Google Scholar CrossRef Search ADS   James, Oliver, Sebastian Jilke, and Gregg Van Ryzin, eds. 2017a. Experiments in public management research: challenges and opportunities . Cambridge: Cambridge University Press. Google Scholar CrossRef Search ADS   James, Oliver, Sebastian Jilke, and Gregg Van Ryzin. 2017b. Causal inference and the design and analysis of experiments. In Experiments in public management research: Challenges and opportunities , ed. Oliver James, Sebastian Jilke, and Gregg Van Ryzin, 59– 88. Cambridge: Cambridge University Press. Google Scholar CrossRef Search ADS   James, Oliver, and Asmus Leth Olsen. 2017. Citizens and public performance measures: making sense of performance information. In Experiments in public management research: Challenges and opportunities , ed. Oliver James, Sebastian Jilke, and Gregg Van Ryzin, 270– 89. Cambridge: Cambridge University Press. Google Scholar CrossRef Search ADS   Khanna, J., and T. Sandler. 2000. Partners in giving: The crowding-in effects of UK government grants. European Economic Review  44: 1543– 56. Google Scholar CrossRef Search ADS   Kim, Mirae, and Gregg G. Van Ryzin. 2014. Impact of government funding on donations to arts organizations: A survey experiment. Nonprofit and Voluntary Sector Quarterly  43: 910– 25. Google Scholar CrossRef Search ADS   King, G., J. Pan, and M. E. Roberts. 2013. How censorship in China allows government criticism but silences collective expression. American Political Science Review  107( 2): 1– 18. Google Scholar CrossRef Search ADS   King, G., B. Schneer, and A. White. 2017. How the news media activate public expression and influence national agendas. Science  358: 776– 80. Google Scholar CrossRef Search ADS   Kingma, B. R. 1989. An accurate measurement of the crowd-out effect, income effect, and price effect for charitable contributions. Journal of Political Economy  97: 1197– 207. Google Scholar CrossRef Search ADS   Kobayashi, T. and Y. Ichifuji. 2015. Tweets that matter: Evidence from a randomized field experiment in Japan. Political Communication  32: 574– 93. Google Scholar CrossRef Search ADS   Konow, J. 2010. Mixed feelings: Theories of and evidence on giving. Journal of Public Economics  94(3–4): 279– 97. Google Scholar CrossRef Search ADS   Korenok, O., E. L. Millner, and L. Razzolini. 2012. Are dictators averse to inequality? Journal of Economic Behavior & Organization  82: 543– 47. Google Scholar CrossRef Search ADS   Kramer, A., J. Guillory and J. Hancock. 2014. Experimental evidence of massice-scale emotional contagion through social networks. Proceedings of the National Academy of Science of the United States of America  111: 8788– 90. Google Scholar CrossRef Search ADS   Krasteva, S., and H. Yildirim. 2013. (Un) informed charitable giving. Journal of Public Economics  106: 14– 26. Google Scholar CrossRef Search ADS   Ledyard, J. 1995. Public goods: A survey of experimental research. In Handbook of experimental economics , ed. J. Kagel and A. Roth, 111– 94. Princeton: Princeton University Press. Levitt, Steven and John List. 2007. What do laboratory experiments measuring social preferences reveal about the real world? Journal of Economic Perspectives  21: 153– 74. Google Scholar CrossRef Search ADS   Li, H. and G. Van Ryzin. 2017. A systematic review of experimental studies in public management journals. In Experiments in public management research: challenges and opportunities , ed. James, O., S. Jilke, and G. Van Ryzin. Cambridge: Cambridge University Press. Google Scholar CrossRef Search ADS   Lilley, A., and R. Slonim. 2014. The price of warm glow. Journal of Public Economics  114: 58– 74. Google Scholar CrossRef Search ADS   Linos, E. 2017. More than public service: A field experiment on job advertisements and diversity in the police. Journal of Public Administration Research and Theory  28( 1): 67– 85. Google Scholar CrossRef Search ADS   Lu, J. 2015. Which nonprofit gets more government funding? Nonprofit Management and Leadership  25: 297– 312. Google Scholar CrossRef Search ADS   ———. 2016. The Philanthropic consequence of government grants to nonprofit organizations. Nonprofit Management and Leadership  26: 381– 400. CrossRef Search ADS   Mergel, I. 2012. Social media in the public sector: A guide to participation, collaboration and transparency in the networked world . Hoboken: Wiley: Jossey-Bass. Middleton, Joal and Peter Arronow. 2015. Unbiased estimation of the average treatment effect in cluster-randomized experiments. Statistics, Politics and Policy  6: 39– 75. Google Scholar CrossRef Search ADS   Munger, K. 2017. Tweetment effects on the tweeted: experimentally reducing racist harassment. Political Behavior  39( 3): 629– 49. Google Scholar CrossRef Search ADS   Okten, C., and B. A. Weisbrod. 2000. Determinants of donations in private nonprofit markets. Journal of Public Economics  75: 255– 72. Google Scholar CrossRef Search ADS   Olsen, A. 2017. Adventures into the negativity bias . Paper presented at EGPA conference in Milan, Italy. Ottoni-Wilhelm, M., L. Vesterlund, and H. Xie. 2017. Why do people give? Testing pure and impure altruism. American Economic Review  107: 3617– 33. Google Scholar CrossRef Search ADS   Porumbescu, Gregory. 2015. Comparing the effects of E-Government and social media use on trust in government: Evidence from Seoul, South Korea. Public Management Review  18: 1308– 34. Google Scholar CrossRef Search ADS   Public Religion Research Institute. 2012. Few Americans use social media to connect with their faith communities. http://publicreligion.org/research/2012/08/july-2012-tracking-survey/ (accessed April 30, 2018). Reeson, A. F., and J. G. Tisdell. 2008. Institutions, motivations and public goods: An experimental test of motivational crowding. Journal of Economic Behavior & Organization  68( 1): 273– 81. Google Scholar CrossRef Search ADS   Roberts, R. D. 1984. A positive model of private charity and public transfers. The Journal of Political Economy  92: 136– 48. Google Scholar CrossRef Search ADS   Rose-Ackerman, S. 1981. Do government grants to charity reduce private donations? In Nonprofit firms in a three sector economy , ed. White M. J., 95– 114. Washington, DC: Urban Institute. Ryan, T. 2012. What makes us click? Demonstrating incentives for angry discourse with digital-age field experiments. The Journal of Politics  74: 1138– 52. Google Scholar CrossRef Search ADS   Ryan, T. J., and Brader, T. 2017. Gaffe appeal a field experiment on partisan selective exposure to election messages. Political Science Research and Methods  5: 667– 87. Google Scholar CrossRef Search ADS   Ryan, T. and D. Broockman. 2012. Facebook: A new frontier for field experiments. The Experimental Political Scientist: Newsletter of the ASPA Experimental Section  3: 2– 8. Smith, T. M. 2007. The impact of government funding on private contributions to nonprofit performing arts organizations. Annals of Public and Cooperative Economics  78: 137– 60. Google Scholar CrossRef Search ADS   Statistia. 2017. Number of Facebook users by age in the U.S. as of January 2017. https://www.statista.com/statistics/398136/us-facebook-user-age-groups/ (accessed April 30, 2018). Steinberg, R. 1987. Voluntary donations and public expenditures in a federalist system. The American Economic Review  77: 24– 36. Sutter, M., & H. Weck-Hannemann. 2004. An experimental test of the public goods crowding out hypothesis when taxation is endogenous. FinanzArchiv: Public Finance Analysis  60( 1): 94– 110. Google Scholar CrossRef Search ADS   Suárez, D. F. 2011. Collaboration and professionalization: The contours of public sector funding for nonprofit organizations. Journal of Public Administration Research and Theory  21: 307– 26. Google Scholar CrossRef Search ADS   Teresi, H. and M. Michelson. 2015. Wired to mobilize: The effect of social networking messages on voter turnout. The Social Science Journal  52: 195– 204 Google Scholar CrossRef Search ADS   Thomas, J. C., and G. Streib. 2003. The new face of government: Citizen-initiated contacts in the era of E-Government. Journal of public administration research and theory  13: 83– 102. Google Scholar CrossRef Search ADS   Tinkelman, D. 2010. Revenue interactions: crowding out, crowding in, or neither? In Handbook of research on nonprofit economics and management , ed. Seaman, B. A., and D. R. Young, pp. 18– 41. Northampton, MA: Edward Elgar. Google Scholar CrossRef Search ADS   Wang, J. C., and C. H. Chang. 2013. How online social ties and product-related risks influence purchase intentions: A Facebook experiment. Electronic Commerce Research and Applications  12: 337– 46. Google Scholar CrossRef Search ADS   Warr, P. G. 1982. Pareto optimal redistribution and private charity. Journal of Public Economics  19: 131– 38. Google Scholar CrossRef Search ADS   Wasif, R., and A. Prakash. 2017. Do government and foreign funding influence individual donations to religious nonprofits? A survey experiment in Pakistan. Nonprofit Policy Forum  8( 3): 237– 73. Google Scholar CrossRef Search ADS   Wukich, C., and I. Mergel. 2015. Closing the citizen-government communication gap: Content, audience, and network analysis of government tweets. Journal of Homeland Security and Emergency Management  12: 707– 35. Google Scholar CrossRef Search ADS   Zhang, J., and E. Mao. 2016. From online motivations to ad clicks and to behavioral intentions: An empirical study of consumer response to social media advertising. Psychology & Marketing  33: 155– 64. Google Scholar CrossRef Search ADS   Appendix Table A1. Mean difference tests CTR (independent two-sample t-test; n = 600)   Control  Placebo  Treatment                Mean  Standard error  Mean  Standard error  Mean  Standard error  Mean difference; control vs. treatment  p value  Mean difference; placebo versus treatment  p value  Mean difference; control versus placebo  p value  CTR  0.50  0.03  0.44  0.03  0.50  0.04  0.00  0.99  0.06  0.21  −0.06  0.17    Control  Placebo  Treatment                Mean  Standard error  Mean  Standard error  Mean  Standard error  Mean difference; control vs. treatment  p value  Mean difference; placebo versus treatment  p value  Mean difference; control versus placebo  p value  CTR  0.50  0.03  0.44  0.03  0.50  0.04  0.00  0.99  0.06  0.21  −0.06  0.17  View Large Table A1. Mean difference tests CTR (independent two-sample t-test; n = 600)   Control  Placebo  Treatment                Mean  Standard error  Mean  Standard error  Mean  Standard error  Mean difference; control vs. treatment  p value  Mean difference; placebo versus treatment  p value  Mean difference; control versus placebo  p value  CTR  0.50  0.03  0.44  0.03  0.50  0.04  0.00  0.99  0.06  0.21  −0.06  0.17    Control  Placebo  Treatment                Mean  Standard error  Mean  Standard error  Mean  Standard error  Mean difference; control vs. treatment  p value  Mean difference; placebo versus treatment  p value  Mean difference; control versus placebo  p value  CTR  0.50  0.03  0.44  0.03  0.50  0.04  0.00  0.99  0.06  0.21  −0.06  0.17  View Large © The Author(s) 2018. Published by Oxford University Press on behalf of the Public Management Research Association. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com. This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices)

Journal

Journal of Public Administration Research and TheoryOxford University Press

Published: May 12, 2018

There are no references for this article.

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off