Advisory committee for environmental research and education september 12, 2012

DR. TRAVIS: Indeed. Mary Catherine.

DR. BATESON: This is a response, both to what you've been telling us and earlier discussions today. I'm an anthropologist, and I'm troubled by the fact that you're pigeon-holing things, which the system does, and then basing counts on that. It seems to me that the question of interdisciplinary work is so urgent that you need to build in a process of observation, what might be called an ethnographic process or a narrative process, as to how the interactions involved took place, and how people remember the moments of insight and new ideas. This could be done by an observer on the spot, actually sitting in on meetings. It could be done with oral histories that could be gone back over, but the importance of what can be learned about the productivity of interdisciplinary interaction and international also, is so important that things should not go forward without a rather deeper and more open-ended self-observation process.

DR. RUSSELL: Well, those are good points. I really didn't talk about the evaluation and assessment activities that are also part of this initiative. So, you know, we are formulating plans for surveys internally and externally, to confront some of those kinds of issues that you're talking about. In other words, whenever NSF initiates something like this, the whole issue of accountability goes with that, and we need to find ways of assessing what is the impact of the program, so some of this will be rather technical in the sense of trying to see what sorts of projects get supported that really were not being supported in the past that's -- this kind of text analysis can help with that, because we can, you know, go to the existing database and then see what's novel about what's coming out of this. But another key part of all of that will be surveys of the community and of the program directors internally. So, those are -- those plans are in the works, and you know, we would certainly be very welcoming of any suggestions about what could be included in that. But that's a part of this whole picture.

DR. BATESON: Well, one suggestion that I would make, which has to do with my own background, obviously you need quantitative data. You need surveys. You need ways to mechanize your analysis of results and end points, and so on and so forth, but one of the things that I find myself saying to all social scientists is that do some qualitative research before you do -- before you design your surveys, and that's what really is critical, because you want to talk to people about, what are the blocks? What are the difficulties? What are the moments of breakthrough? In order to be -- you have to listen to what they say before you decide surveys and count the results.

DR. RUSSELL: Well, let me just say that there is -- there is an internal review for OIA, the Office of Integrative Activities, that will be coming up within -- well, we don't know exactly when it will be, but probably the next month or two. And as far as explaining or speaking to the internal community about what is going on in INSPIRE, we were wrestling with how to go about that, because obviously it's too early for there to be results from the scientific projects that just got funded within the past month. But the idea that occurred to us was, in fact, to interview some of these PIs and ask them, did this opportunity cause you to do some things, make some connections, try some ideas, that you would not have otherwise? In other words, what -- how did this affect your mental processes of the scientific curiosity and creativity? So that's, you know, that's exactly the sort of thing that we're talking about doing internally. But we haven't done it yet, but that's the idea that we put on the table within the past two weeks perhaps.

DR. TRAVIS: So let's go down this part of the table first and then we'll cross over. Fred and then David and then Stephanie. I have three others waiting.

[laughter]

DR. ROBERTS: So Tom, maybe it's too early to answer this, but what kinds of things worked and what kinds of things didn't in terms of the process and are there any changes you anticipate?

DR. RUSSELL: Actually, we thought the process worked surprisingly well. I mean, we were worried that we might be off by, you know, a factor of three in either direction in terms of interest and capacity. Some -- we decided that this had to be done internally as a bottom-up process. In other words, it's really in the hands of the program directors. The -- there's a lot of concern about workload at NSF. People are stretched really thin, and you know, some of the program directors have very, you know, I've seen -- some of them have negative views about INSPIRE, because it's adding to work load. What some of those people don't realize is how hard we tried to think about how to do this in a way that would, you know, mitigate that as much as we could. We knew it was going to represent workload, but we felt that an inquiry where you could basically in a pretty short manner say, “No, that's not an INSPIRE proposal,” but that would be something that could be turned around, you know, without a huge amount of effort.

So by and large we felt that it worked quite well. We had some bumps in the road. Initially -- we have these inquiries coming in to multiple people, and there was a question of self -- were they self-organized? You know, we started from the perspective, “It should be bottom-up. Let's let them figure it out.” Eventually we and they concluded that that didn't work, you know. It needed -- there needed to be somebody designated of being in charge, a program director being in charge of handling the inquiry, because otherwise, you know, it will just get lost in their crush of other business and nobody would respond to the PI and, you know, so that was one of the kinds of adjustments we made. But I'd say that, you know, as far as the stuff that, where we had to sort of, you know, to make course corrections, it was things of that nature. It wasn't really anything fundamental that went wrong.

DR. TRAVIS: David?

DR. BLOCKSTEIN: My question I guess is a follow-up, just maybe a little more open-ended. What in particular surprised you and your colleagues about this whole process?

DR. RUSSELL: Well, I think as I said a second ago, we were surprised that it worked as smoothly as it did. I mean, we thought, you know, we're trying to do something unconventional that's going to cause people to think in different ways, and you know, we got -- there was inevitable amounts of rumbling and so on, but we think by and large most people ended up feeling pretty positively about how it worked out. So, maybe in that sense we were somewhat pleasantly surprised. There were, you know, some units were more -- some program directors were more active than others. You can say that that could be a concern, because it, you know, if the program director says, “Oh, forget about this,” you know, what effect does that have on the community that that person serves? Again, we, after a lot of thought, concluded that this is -- it has got to be a bottom-up thing. We have to get the buy-in of the program directors, and the centralized funds that we have residing in the Office of Integrative Activities provided some incentive for people, you know, to leverage investment for their community, and it seemed as if most program directors in some ways, you know, bought into that. So, I don't know, I guess -- but that's not surprising when you see that sort of individual variation. So I guess that wouldn't really qualify as a surprise.

I don't know. I think by and large things went -- I mean, when we were trying to design this we had our theories about how it would go, and the actual sequence was not that different. Perhaps we thought that we might be -- we might get our money over-committed earlier in the fiscal year, and that didn't happen. Most of it -- a lot of the proposal activity and inquiry activity came, you know, as the deadline approached, even though the window was wide open from November to June. So maybe that was a little bit of a surprise that, you know, we weren't running out of money in March or April or something.

DR. TRAVIS: Stephanie?

DR. PFIRMAN: I wanted to follow up on Mary Catherine's recommendation. This committee has been very interested obviously in interdisciplinary research and education and, as Marge said, we were the first advisory committee that was established specifically, I think with that -- within our mandate. And we've also been calling for NSF to do something along these lines for, you know, a long time now. So, I think there's a lot of expertise and interest, you know, on the part of this committee that could help you out when you try to understand how the community is responding, and potentially actually how NSF is responding also.

DR. RUSSELL: Please do.

DR. PFIRMAN: So, I think, you know, as you're thinking about how you're -- how to evaluate that, you know, I think it would be really useful to tap into sort of our interests in this area, and of course, it's somewhat self-serving, because we want to do interdisciplinary research and education ourselves. So, you know, it would be nice for us to understand the process better. But the other thing I wanted to bring up is when biocomplexity first came up, which was one of the big cross, you know, directorate interdisciplinary initiatives, an award was given to some -- to a couple of PIs to analyze the awards, and so they went out and they did network analyses and interviews and everything of the PIs. It was very interesting analysis, and so I'm just wondering if maybe something could be considered along those lines as well. This is not self-serving and I hopefully will not be the one to do this.

[laughter]

DR. RUSSELL: And how long into biocomplexity did they do that?

DR. PFIRMAN: Yeah, I think in the initial award, so in the first batch, so they got funding at the same time as the other PIs did. So, but I think, you know, what people always say is you want to get a baseline, so you know, we should've done this already, anyway. You know what I mean? So, it's --

DR. RUSSELL: We're working on it. We're having a statement at work right now to do baseline data gathering.

DR. PFIRMAN: -- yeah, I'm just -- like with -- I didn't mean that to sound, you know, judgmental or anything. Just the earlier that the evaluation is done the better, and I think, you know, awarding, you know, some PIs support to actually conduct something like this could be really interesting research in and of itself.

DR. RUSSELL: And we'll think about that.

DR. TRAVIS: Let me -- Molly, Tony, Bruce, Ivor. I have Bruce, Lil, Ivor. Sorry. Then Mary Catherine.

DR. BROWN: Hi. So my question is on how did -- what are your thoughts about avoiding the old boys’ network, being not an old boy myself.

[laughter]

Because it strikes me that this is a perfect opportunity for all your favorite PIs to go and say, you know, your NSF program manager called you up, “Hey, don't you have something in your toolbox there? Can't you submit something?” And then you get exactly the same people that you've always gotten, and how do you know that you're not -- do you have any insight to whether or not that sort of process occurs sort of excluding people you don't know or you're not sure about or you don't -- you've never heard of, you know, as a post-doc who's in the closet kind of thing and never gets to come out. It would be really interesting to see, because that's really what concerns me with this kind of process.

DR. RUSSELL: Sure. Yeah, we were very concerned about that. I mean, I think, you know, Alan's in the back. He can verify that this topic came up many times in our discussions.

DR. BROWN: Okay.

DR. RUSSELL: And we thought we really might see some program directors trying to, you know, grease the way for their old boy network, as you say. From what we could tell, we really did not see much evidence of that.

DR. BROWN: Okay.

DR. RUSSELL: Every inquiry that -- well, the way the process worked internally was that every inquiry that program directors were intrigued by and wanted to invite a proposal for, came to our group --

DR. BROWN: Okay.

DR. RUSSELL: -- as a check in a way, because we were responsible, NSF-wide, for enforcing the criteria, the interdisciplinary -- in other words, it's a pilot that's going outside standard policy, so somebody had to be accountable for seeing that the pilot was being properly administered, and that was our group. So, the way we designed the process was we told the program directors if you've got something that you want to invite a proposal for, bring it to us right away before you even invite the proposal so we can look at it and see if we see any red flags about it. And in particular, if it didn't look interdisciplinary enough, or if there was something about it that didn't seem to be along the lines of what would be appropriate, we would tell them right then so that they could either change course or adjust or whatever before they put a lot of extra work into it.

So, we were very active in this process, our group, and I can say that, you know, our members were probably, you know, tougher on inquiries involving their own areas than they were on ones that, you know, were more far afield from what they knew about. So, this stuff -- these kinds of issues got discussed all the time, and --

DR. BROWN: Okay. That's all.

DR. RUSSELL: -- and we really didn't see -- we had been afraid of that, too, and we really didn't see it.

DR. BROWN: Okay. Great.

DR. TRAVIS: Tony?

DR. JANETOS: One of the things you said early on is something I've heard -- we've heard in lots of discussions of interdisciplinary research, which is sort of conflating the idea of interdisciplinary proposal with risk -- with risky research. They're somehow inherently more risky, and maybe it's true, but I'm not -- it would be really interesting to hear, you know, a year from now, three years from now, whether or not you continue to think it's true, or whether there's any evidence for it at all in the proposals that you fund compared to proposals that have gone through ordinary panel reviews. I’m not even sure that I know what risk means in this context, you know, as somebody won't write a paper. And the proposal -- it's a really vague concept in this arena, and I just think you guys have been so thorough in putting this together and so clear in what you've done that you've -- if anybody can tease this apart you should be able to do it, and if you can't do it I'm pretty sure it can't be done.

DR. RUSSELL: Well, I think -- I guess you just provided some evidence that maybe I wasn't that clear, because in our minds we don't conflate those two things, interdisciplinary and, you know, potentially transformative or high-risk high-reward whatever. Now we think of them as being actually quite different. So, if I gave the impression that we were conflating that, then I wasn't being clear enough. So, no, I think the sort of inquiries that got the kind of routine “Send a regular proposal response” were ones of the nature yes, it's interdisciplinary, but no, it's not really out of the box. It's something that we would be able to deal with through the regular process.

And again, the whole thing, you know, this whole bottom-up view was, our sense, was the best way to judge that would be to rely on your expert colleagues or program directors. They would know it when they see it, and so that's the way we went.

DR. TRAVIS: Bruce.

DR. LOGAN: Okay. I'll try and be real quick. Question, two comments, and a short story in about one and a half minutes.

[laughter]

DR. RUSSELL: Oh, I got to hear this.

DR. LOGAN: Can you imagine a young INSPIRE? Okay, that's the question. I'm concerned about creativity. When you go collaborative how do you instill creativity or how do you cultivate an environment towards creativity? And I'm also very concerned about success rates being only 10 percent and young researchers having to compete in that network. And so, the way I looked at it, when I was an assistant professor I got a phone call from a full professor and said, “Would you be willing to do some oceanography?” I was sitting in the middle of Arizona thinking about particle dynamics, and I said, “Sure.” So, three months later I was on a ship off of Santa Barbara doing oceanography. And that connection inspired me to do work that I had never done before. It was different from post-doc. Post-doc you have to plop down somewhere and you do the work in a laboratory. And 20 some-odd years later I try and get young researchers to come into my lab, young professors, assistant professors and kind of see what's going on and look for creative. So, my question again is, could you imagine where instead of having a bigger round you might actually have a smaller round where you fund somebody based upon their connection only to facilitate some sort of creative interaction?

DR. RUSSELL: Based on --

DR. LOGAN: So, you would fund somebody who said, “I'm working in, you know, this field, but I want to go into an anthropology lab and learn what they do, so that we can work together.”

DR. RUSSELL: Okay. I think there -- I mean, there are some other NSF activities that are aimed at that sort of career move, you know, work with somebody in a different field.

DR. LOGAN: It's not a career move. It's a career launch. I'm talking about somebody who might, you know, definitely nobody who has tenure to do this.

MALE SPEAKER: Well, that's true.

DR. RUSSELL: We have not yet analyzed the, you know, the PhD age of this set of awards. There are some younger people in there. As for the specific issue that you bring up, I'm not sure whether it couldn't be handled at the kind of scale you're talking about through the EAGER mechanism, which is already out there, but I may not be completely tuning into what --

DR. LOGAN: Yeah. Just my point is that it's different from Eager, because I can go to my program manager -- a person can go to a program manager with an idea.

DR. RUSSELL: -- yeah.

DR. LOGAN: This would be where you needed two program managers to support our idea, right? Because you need somebody, you know, to accept that, “Yeah, this person would be proposing to go into that other field.”

DR. RUSSELL: Okay. So you're talking about an interdisciplinary context or --

DR. LOGAN: I'm just saying you're thinking to the next level, but I do think bring it down to an earlier level, I think maybe like a phase three or something.

DR. RUSSELL: -- yeah, well, you know, there are some thoughts around of actually -- the director has made public statements about individual INSPIRE, let's call it.

DR. LOGAN: Okay. That's good. That's probably enough. I think we need to return --

DR. RUSSELL: Yeah, I mean, because in his mind there are, you know, individuals who are interdisciplinary and need that kind of support, and so that very well -- we don't have any -- there's no public statements about that in any of the written documents, but...

DR. TRAVIS: Lil?

DR. ALESSA: Okay, real fast. So, comment and question. I think we should be held to a higher standard, higher accountability, how are you going to do it?

DR. RUSSELL: Higher than?

DR. ALESSA: For a variety of reasons, because of some perceptions in the community that Molly has touched on, and some of those perceptions are pretty severe, by the way.

DR. RUSSELL: So, higher standard of accountability than what?

DR. ALESSA: Than a regular proposal that's gone through a peer review process -- panel review process. How's it going to be -- these are supposed to be groundbreaking, cross-cutting, you know, completely move us forward, so how are they going to be evaluated in terms of being productive and --

DR. RUSSELL: Well, we expect that there will be a great many failures. If the projects are all successful, then we failed, because these are meant to be trying some things that are out of the box. So, you know, how you structure accountability is a tricking question, but as I said, you know, we are putting together plans for surveys and data analysis aimed at, you know, precisely trying to assess the impact of the program. I don't -- but if it's all, you know -- if everything is succeeding then we've played it too safe.

DR. ALESSA: No, no, no. Success doesn't mean that you've -- everything worked perfectly. Success means that process --

DR. TRAVIS: Use your mic, please.

DR. ALESSA: Sorry. I mean, success doesn't mean that everything worked perfectly and all the pieces fit together. Success is a process. It's not a thing. We shouldn't get into it here, because I think we're talking from two different sort of philosophical fields, so I'll move on.

DR. TRAVIS: Ivor.

DR. KNIGHT: Yeah, I just wanted to pick up from what Tony was saying about the risk and interdisciplinary. At Canon we mitigate risk by doing interdisciplinary research. I mean, that's how we do it. It's conflated, I guess, because it's a mitigation process.

And the question at hand, though, is the last point in the acronym, education. Are you doing anything to look at the effect on students involved in these programs and how they behave perhaps differently? I'm just thinking that, you know, when we hire people out of post-docs and PhD programs we usually spend the first three years teaching them how to do interdisciplinary work. And so I'm wondering if there's any -- you know, it's a very small sample size you have, but is there any kind of tracking that's going on with the students maybe long term into careers?

DR. RUSSELL: Well, that will certainly be, you know, part of the assessment that we will do. And, you know, those of us whose brains have, you know, ossified by now --

[laughter]

-- really appreciate the importance of inducing interdisciplinary thinking in people when they're young. And so yeah, that's definitely on the table. There are a few of these projects that involve educational research. I thin primarily the Division of Research on [unintelligible] under the director of education and human resources, but I think your point is more general than that. I mean, what happened to students in post-docs, you know, in all of these projects in all the different domains that they cover. Yes, I mean, that will be part of what we look at, because, you know -- we're trying to help induce some cultural changes here, both within the NSF and in the community and to have people see that, you know, there is reward and opportunity and excitement in approaching things this way. Certainly, you know, the young people are a key part of that.

DR. TRAVIS: Mary Catherine and then Eric.

DR. BATESON: I'll try and be very brief, because related things have been said. First of all, take a look at the MacArthur quote, unquote, genius grants. It's widely felt that they have become progressively more conservative, because they're so anxious that they won't be productive. So they're being given to people very much on the basis of work already done rather than work they might do, and beware, because that's a real problem.

Second, I want to second the career sequence issue and suggest that you think now about looking at the people you -- whose work you’ve supported, five years from now to see whether they're colleagues in their institutions support the kind of thing that's being done. Because the institution needs to be educated, not just the NSF, the universities, the departments.

Then the last thing that I wanted to mention is we heard earlier today from Maria Uhle? Is that the right pronunciation? About the Belmont Forum. And they require international connections to be made for research that they fund. Now, that's a little trickier mechanically than interdisciplinary, but -- so they had a registration process, as I understand it, that would facilitate people finding partners to work with. And the analogy that was used was a dating service. It's actually a brilliant idea, you know? If I want to work on something and I think well gee, there must be somebody who knows about X and I don't know that person, but that person is interested in interdisciplinary work, you could build that.