Maintaining the bibliography of academic work about asexuality

[Lord Happy Toast]

I wanted to repost here something from my blog, concerning the asexuality bibliography on my website, to get more input from AVENites. (The bibliography is linked in my signature.)

When I first started my bibliography on asexuality, there were very few academic publications out there. Since then, the number of publications has increased enormously, and so has the rate at which articles are published. I've fallen behind in maintaining it. Partly, this is because there is a lot that has come out. Partly, this is because the whole thing is in need of a serious overhaul and I would like to develop a better method for maintaining it (especially now that, in my own writing, I've learned how awesome bibtex is). I was curious if any readers would be interested in collaborating with me to help maintain the bibliography, and help make it more easy to update in the future. If anyone would be interested in helping, that would be greatly appreciated. If anyone who is familiar with bibtex and/or Python is interested in helping out, that would be even better.

Another reason that I've been behind on updating the bibliography is that I'm seriously questioning the value of having an exhaustive bibliography as the primarily bibliography for asexuality publications on my site.

For a long time, I attempted to treat all publications equally regardless of what I thought of the quality. I made a slight modification to this a few years later by putting a couple of recommended starting places at the top of the bibliography. But some of the publications on asexuality are so utterly awful that I feel the authors are doing a disservice to people by publishing them (i.e. they're wasting the time of potential readers), and I feel like I may be doing a disservice to users of the bibliography by even including these.

Something that has become clear to me from following the academic publications about asexuality is that the peer-review process and other quality control methods are failing miserably in at least some parts of academia. I've seen articles that do "history of asexuality" with virtually no use of primary sources. I've seen sweeping generalizations about the field of psychiatry by people who give no evidence of having ever read anything from that field. Probably the most epic example was that one paper who quoted the AVENites "Megan Mitosis" and "Asexy A-postle".* Evidently, their understanding of AVEN was so limited that they didn't know the difference between the post ranking field and the username field.

More generally, I feel that a lot of the papers and chapters out there...basically fail to make any meaningful contribution to our understanding of asexuality. They don't present any original research (quantitative, qualitative, or historical/archival). They don't present any original ideas about asexuality. They just...say stuff.

So my question for readers is what would be most useful to readers of my bibliography who are wanting to have a better understanding of asexuality. Should it continue to try to be as exhaustive as possible (which will likely became unmanagable after a few more years)? Should there be somewhat greater restrictions on what is included? If so, what sort of standard would be fair? (I certainly don't think that "Did I like that paper?" is a good standard.) Should I have two bibliographies--an exhaustive-as-possible one and another for things that pass some sort of standard of quality control?)

I know that some of these issues are closely related to larger issues in academia today: Many people feel that the peer-review process is not doing a good job of quality control, but quality control is vitally important for intellectually credible scholarship.

[Member54880]

On my blog, I posted about your request, and had suggestions about how to handle quality control.

There are two possible approaches I see, in regards to quality control:
The first is to review all of the publications, and rate them by their intellectual merits, originality, and the quality of their research, perhaps rating them with a 1-5 star rating. The best publications overall can be put as the recommended readings. This is one way for externally-imposed quality control to happen, with lower-rated publications showing severely flawed research methods, or even a lack of original research. These could serve as examples for how to avoid those problems for future researchers, or for current researchers preventing themselves from making the same mistakes.
The second would to be review the publications, rating them in the same way as above, but on a pass/fail basis, and only including those that pass.

Elaborating on it more, I'd use these criteria:
Originality: Original research presented, originality of ideas.
Quality of research methods: I'd grade this differently, starting with a perfect score, and deducting points for any flaws detected. Flaws that are severe enough, like a blatant lack of research, and suggesting that correlation = causation, would be an automatic fail.
Quality of the writing: Is it coherent? This is a harder criterion to judge, because some asexuality research papers are in the same format as other gender/sexuality studies papers, while others are in the same format as psychology papers, and the latter are written under the assumption that the reader knows how to read empirical research papers.
Sources used: Looking for primary sources, the quality of sources, and how well the sources are used. Are they well-integrated, as opposed to just being tacked on?
Accuracy in regards to both asexuality itself, and the current state of asexuality research: Self-explanatory, major inaccuracies would be an automatic fail.
How meaningful is the content of the publication: Related to the "originality" criterion, did the publication seem like it had something meaningful, and had depth to it?

From there, you should decide if you want to rate each of those qualities on a scale, or on a pass/fail basis. Personally, I'd like to include all the publications and give numerical ratings, or grades to each of them, because it could be useful to researchers, but only if trying to include them all isn't overwhelming to anyone who'll be involved in this project.

[Lord Happy Toast]

Elaborating on it more, I'd use these criteria:

Originality: Original research presented, originality of ideas.

I think that "originality" needs to be assessed differently depending on the type of article. For example, here are some statements from some of the major (scientific) sexuality journals:

Ex 1: Journal of Sex Research

The Journal of Sex Research (JSR) is a scholarly journal devoted to the publication of articles relevant to the variety of disciplines involved in the scientific study of sexuality. JSR is designed to stimulate research and promote an interdisciplinary understanding of the diverse topics in contemporary sexual science. JSR publishes empirical reports, theoretical essays, literature reviews, methodological articles, historical articles, teaching papers, book reviews, and letters to the editor. JSR actively seeks submissions from researchers outside of North America. The JSR audience is researchers and practitioners in the fields of psychology, sociology, education, psychiatry, communication, and allied health.

Ex 2: Archives of Sexual Behavior

The official publication of the International Academy of Sex Research, is dedicated to the dissemination of information in the field of sexual science, broadly defined. Contributions consist of empirical research (both quantitative and qualitative), theoretical reviews and essays, clinical case reports, letters to the editor, and book reviews.

I think that originality (in some sense) is important--i.e. Does the article/book/thesis/dissertation/etc. make some sort of substantive contribution? Although it can be complicated because the sort of "originality" can be hugely variable. e.g. If a study replicates the results from a previous study, current publishing pressures in academia don't value this very much (because of it's lack of "originality"), even though replication is extremely important for scientific progress. Also, a good review article should be (in some sense) "original" even though it's basically covering things that have already been done. A review article doesn't have to be especially original in terms of the ideas that are presented, although it should be original in some sense. i.e. If a lot of research has been published, and there is need to a) provide a readable summary for experts in other domains wanting to know the state of the research, and b) establish what is known and what isn't and what the important issues needing further research are.

Quality of research methods: I'd grade this differently, starting with a perfect score, and deducting points for any flaws detected. Flaws that are severe enough, like a blatant lack of research, and suggesting that correlation = causation, would be an automatic fail.

My thinking about the quality of scientific research tends to start from the opposite end of the quality spectrum. In some sense, all scientific research is "deeply flawed," (i.e. far from perfect) and this is especially true for the social and biological sciences. (If we use a "levels of analysis" approach to understanding the world, then, generally speaking, the higher up you go in terms of the complexity of systems, the harder it is to do good research. It's well known that, when studying humans, data based on how people actually behave in the world are full of all sorts of variables that make it extremely hard to establish the role of any one variable (especially because of so many potential confounds). By contrast, experimental data "in the lab" about human behavior or cognition can give a better idea of the role of one specific variable, but the very fact that they've controlled all sorts of potential confounds often means that the setting is rather artificial, and so we don't know how well it generalizes to the real world.

People who are ideologically opposed to science (or at least to science in certain domains) sometimes highlight the many problems with scientific research in order to cast suspicion on the whole enterprise. However, they don't provide any better ways of doing things. Generally, the main alternatives to scientific approaches to topics of interest in the social sciences are rigid adherence to dogma, or just saying a lot of bullshit. So, in my view, the relevant comparisons for the quality of research are "How well does it compare with what we already have?" and "Given practical constraints, how well does it compare with reasonably possible alternatives?" (I would add that the same applies to research in general, regardless of whether or not it is "science" or "humanities" or whatever.)

In terms of work on asexuality, all of it is far from perfect: Asexuals recruited online are likely a lot different from asexuals more generally; we don't know how people understand the "asexual" items in the large probability surveys; even in the large studies with probability sampling, participation rates are well below 100% and the subsamples of asexuals are still not all that big. Because of these and other problems, I don't put that much confidence in any single study about asexuality (unless it's doing some sort of general description that doesn't aim to be generalizable beyond a certain sort of subset of asexuals). However, if a lot of studies--all having problems, but not all having the same problems--all point in the same general direction, then I think we can be much more confident in that finding.

My inclination is to divide "quality of research" into (at least) two sorts of categories: First, ow good was the methodology (as compared with what's already been done, what was reasonably possible, and whether it has a sufficiently high degree of prima facia plausibility)? Secondly, do their conclusions follow from the evidence given, do they adequately discuss alternate explanations, and do they provide appropriate levels tentativeness? In addition, it's also important that "Do their conclusions follow from the evidence given?" is only answered affirmatively for non-vacuous claims. i.e. Several papers have actually "argued" for the position that our understandings of asexuality are culturally dependent. However, this claim is to utterly obvious (and true for pretty much all of our concepts about anything) so such "arguments" should be seen more along the lines knocking down straw-men than substantive theoretical or empirical progress.

Quality of the writing: Is it coherent? This is a harder criterion to judge, because some asexuality research papers are in the same format as other gender/sexuality studies papers, while others are in the same format as psychology papers, and the latter are written under the assumption that the reader knows how to read empirical research papers.

I think this would be a sensible criterion to include.

Sources used: Looking for primary sources, the quality of sources, and how well the sources are used. Are they well-integrated, as opposed to just being tacked on?

The main place where lack of use of primary sources has been a problem involves talking about the history of asexuality, HSDD, the DSM, etc. In the context of historical research, theoretical articles about the medicalization of categories similar to asexuality, etc. I would consider this to be a part of "quality of research", so I don't think it would need to be a separate field.

Accuracy in regards to both asexuality itself, and the current state of asexuality research: Self-explanatory, major inaccuracies would be an automatic fail.

This also seems sensible, provided that there is a caveat that "accuracy" doesn't mean "agrees with my (or our) understanding of asexuality."

How meaningful is the content of the publication: Related to the "originality" criterion, did the publication seem like it had something meaningful, and had depth to it?

I think that this could be combined with one of the above ones (i.e. the part about whether the claims are supported by the evidence, and its requirement of non-vacuous claims).

From there, you should decide if you want to rate each of those qualities on a scale, or on a pass/fail basis. Personally, I'd like to include all the publications and give numerical ratings, or grades to each of them, because it could be useful to researchers, but only if trying to include them all isn't overwhelming to anyone who'll be involved in this project.

After thinking about this some more and talking to a number of people, I think that, in the short term, I'd like to just get a system up and running for keeping the bibliography up-to-date. In the long run, I'd like to have a searchable database for articles about asexuality (or somehow especially relevant). For the articles about asexuality, I think it would make sense to attempt to do quality control by having something like what you're proposing. Presumably, we'd need to have a scoring system that involves several criteria, each of which is rated on some sort of scale (e.g. give it 1-5 stars, give it a number from 0 to 10, etc.) and then several people willing to help rate articles according to that system. This information could be given along with other relevant information about articles matching a user's search.