Peer Review

[funkstar]

As that link says, "... both terms may apply to a particular act ...". Suing would be justified.
I'm not sure it would. Copyright infringement and plagiarism are not the same thing, even though they can overlap - the offender would have to be criminally stupid to have copied verbatim. I'm unsure of what civil suits one can bring in the case of plagiarism.

Can you prove that? The research I quoted above questions that.

Please don't call a polemic opinion piece research. Furthermore, I don't see how Tipler's rant questions what I said. He points out that the peer review process can be problematic, which is true, though I don't agree with a lot of his points. I don't see how that supports that no editing at all would be better.

The criticism there looks like sour grapes to me. The Mona Lisa can be created by a very simple computer algorithm. Would it be Wolfram's fault if most scientists didn't grasp that?

Don't disagree with me merely to disagree: It makes you look silly when you get caught using examples you don't understand. That genetic algorithm has nothing to do with the simple cellular automata of Wolfram's door stopper.

Red herring. I didn't suggest otherwise.

Sure you did. Your very first post in this thread strongly implies it.

 

[BertBonsai]

I'm not sure it would. Copyright infringement and plagiarism are not the same thing, even though they can overlap - the offender would have to be criminally stupid to have copied verbatim.

If I felt a wrongdoing occurred, I'd let the judge or jury decide in civil court.

I don't see how that supports that no editing at all would be better.

It's not that research I was talking about. It was the one where "the conclusions add to the growing debate among editors of leading medical and scientific journals over whether existing peer review is in fact damaging, rather than enhancing, the credibility of research."

That genetic algorithm has nothing to do with the simple cellular automata of Wolfram's door stopper.

It was an analogy. I applaud Wolfram writing his book. Lots of things that are generally accepted today were highly criticized at first. It doesn't matter if the whole thing really is a door stopper. Likely peer review would have censored the work and the rest of us wouldn't have any chance to evaluate it. I wouldn't trust peer reviewers to fairly evaluate it. Since Wolfram bypassed this process, if I don't like the book I can simply put it back on the shelf. Someone else, if only years from now, may find a nugget therein to use as a stepping stone to a groundbreaking idea.

Sure you did. Your very first post in this thread strongly implies it.

You jumped to that conclusion and couldn't show otherwise.

 

[funkstar]

If I felt a wrongdoing occurred, I'd let the judge or jury decide in civil court.

You'd have to do better than that. It's not the court's problem to define your suit.

It's not that research I was talking about. It was the one where "the conclusions add to the growing debate among editors of leading medical and scientific journals over whether existing peer review is in fact damaging, rather than enhancing, the credibility of research."

If you look at the actual paper, rather than The Scientist's report, the conlusions are somewhat more modest.

In any case, I still don't agree with you, obviously: I still think that peer-reviewed papers are more credible than non-reviewed ones. The fact that some good papers failed the peer-review process doesn't harm this. False negatives (good rejected papers) mean that the process is not as sensitive as we'd want. But I would argue that 1) it is still fairly sensitive (few good papers are rejected as compared to the may good ones that are accepted), and 2) it is still fairly specific (few bad papers are accepted as compared to the many bad ones that are rejected).

Surely any relatively sensitive and/or specific test is better than none at all? Of course, what is and isn't a good or bad paper is somewhat subjective, so YMMV as to just how sensitive and specific peer review is.

I'm a bit surprised Tipler doesn't address such issues in a formal manner (or, indeed, at all). He seems quite content to quote cherry-picked rejections (and a fairly large piece of the polemic is about himself), which doesn't really say much about the usefulness of the peer review process at all. Pretty dodgy and not .

Again: This doesn't mean that I consider the peer review system perfect, or immune to criticism. But I think it is better than nothing, and I haven't seen anything to convince me otherwise. The cochrane study is a step towards an actual evaluation, but I haven't looked to deeply at that.

It was an analogy.

Ok.

Lots of things that are generally accepted today were highly criticized at first.

Of course! That's another thing that strikes me as odd: It's not a problem that new ideas are given a hard time. Most new ideas are obviously bad, and those that aren't of course need to vetted more critically than those "merely" building on existing work. It's basically built into the scientific method that that's how it should be!

You jumped to that conclusion and couldn't show otherwise.

Come on: In your first post in this thread you complained "no major journal today is going to consider anything that challenges general relativity, even though that theory makes some bold claims that haven't been experimentally confirmed" which is an obvious allusion that you think your dream theory should get the time of day in a major journal.

 

[BertBonsai]

Surely any relatively sensitive and/or specific test is better than none at all? Of course, what is and isn't a good or bad paper is somewhat subjective, so YMMV as to just how sensitive and specific peer review is.

I think the best test is done once a paper is published. The fact that peer review doesn't allow for rebuttal, or hardly does, is ridiculous. The fact that reviews are not public (anonymity is okay) is ridiculous. Both allow too much room for abuse, and abuse is what I see happening all over when I read about peer review.

I'm a bit surprised Tipler doesn't address such issues in a formal manner (or, indeed, at all). He seems quite content to quote cherry-picked rejections (and a fairly large piece of the polemic is about himself), which doesn't really say much about the usefulness of the peer review process at all. Pretty dodgy and not .

I agree that Tipler talks too much about himself there. I still think he raises good points though.

Again: This doesn't mean that I consider the peer review system perfect, or immune to criticism. But I think it is better than nothing, and I haven't seen anything to convince me otherwise.

It might be better than nothing, but clearly something else would be an improvement. Yet science will always be a popularity contest (even if what's popular is what makes the most careers or money, regardless of validity), so I won't hold my breath for something better. Anyone can publish on the web, so we already have a better system. It's just not popular or well-organized.

Of course! That's another thing that strikes me as odd: It's not a problem that new ideas are given a hard time. Most new ideas are obviously bad, and those that aren't of course need to vetted more critically than those "merely" building on existing work. It's basically built into the scientific method that that's how it should be!

The problem arises when new ideas are not vetted beyond the title, and instead summarily dismissed.

Come on: In your first post in this thread you complained "no major journal today is going to consider anything that challenges general relativity, even though that theory makes some bold claims that haven't been experimentally confirmed" which is an obvious allusion that you think your dream theory should get the time of day in a major journal.

It makes perfect sense that I'd say that, given that I asked every major physics journal if they accepted challenges against GR, and they all said no.

 

[AlphaNumeric]

I think the best test is done once a paper is published. The fact that peer review doesn't allow for rebuttal, or hardly does, is ridiculous.

Peer review does allow for rebuttal, you can respond to all reviewers, the editor, ask for new reviewers and can take it to other journals.

The fact that reviews are not public (anonymity is okay) is ridiculous.

What would making them public achieve?

Both allow too much room for abuse, and abuse is what I see happening all over when I read about peer review.

Why don't you try actually having your work peer reviewed, rather than simply 'reading about it'? You seem to be trying to tell people who have been peer reviewed and have peer reviewed other people how the peer review process, which you've never personally experienced, is done!

It might be better than nothing, but clearly something else would be an improvement. Yet science will always be a popularity contest (even if what's popular is what makes the most careers or money, regardless of validity), so I won't hold my breath for something better. Anyone can publish on the web, so we already have a better system. It's just not popular or well-organized.

You sound like you've got a case of sour grapes. Your work fails to meet basic standards and you simply can't accept its your fault. I really think you just need to grow up, you're throwing your toys out of the pram because you have failed to put in any time or effort, you have failed to produce work of any decent level of quality, you have failed.

The problem arises when new ideas are not vetted beyond the title, and instead summarily dismissed.

This is a mute point when it comes to your work because even when someone does read past the title it fails to be worth reading.

It makes perfect sense that I'd say that, given that I asked every major physics journal if they accepted challenges against GR, and they all said no.

We've been through your issue with paraphrasing. When talking to me you couldn't even honestly paraphrase things I'd said properly and obviously the axe you've got to grind with regards to journals means no one should take your comments about what they said at face value. Speaking as someone who has been in the academic research community I know the attitude of researchers with regards to various things you make claims about and I know you to be wrong.

Take some damn responsibility for your failures to produce worthwhile work, stop trying to blame everyone else for your ignorance and your unwillingness to learn or understand.

 

[BertBonsai]

What would making them public achieve?

Then the public (the larger scientific community, at least) could see when they're unscientific. Kinda like BP's webcam of the oil spill. Before they were pressured to make it public, they lied about the volume.

Why don't you try actually having your work peer reviewed, rather than simply 'reading about it'?

I've determined it would be a waste of my time.

You sound like you've got a case of sour grapes.

Yeah, like it was sour grapes when blacks complained about segregation. It's always sour grapes to you, isn't it?

Your work fails to meet basic standards and you simply can't accept its your fault.

You talking about the work in the thread where you were refuted over & over again, and my equations were independently verified by someone you lauded? Get over it already.

Speaking as someone who has been in the academic research community I know the attitude of researchers with regards to various things you make claims about and I know you to be wrong.

Just like you know SR can be used in only a point-sized region below an event horizon? Excuse me if I don't give much weight to what you think you know.

It's not just me who questions peer review, as the links I gave above show. Nothing I say about it need have anything to do with my work.

 

[funkstar]

I think the best test is done once a paper is published.

In some sense I agree - the importance of a paper can, by the nature of things, only be judged after the fact. However, the peer review system is great for a first approximation of the quality of a paper. Almost immediately, you can make a guess at the worth of the paper: It was good enough to get published, wasn't it? And how reputable was the outlet? No, these are not perfect measures, but they're a start. You tell me: Which is more likely to be a good paper? Something on GeoCities, or something from Science? It's pretty obvious, no?

This has massive benefits: It is a huge time-saver for the working scientist, who then doesn't necessarily have to slog through the details of every piece of material he comes across, but can approach it with some confidence as to its validity. Even more importantly, it also allows you limit your search to primarily established scientific outlets. Yet another point is that published papers are a lot more polished than memos, essays, technical reports, and pre-prints, and reviews have a lot to do with that: If you cannot get your point across to an expert in the field, you have serious work yet in store. In that sense, peer review is a "free editing service" for scientific work.

The fact that peer review doesn't allow for rebuttal, or hardly does, is ridiculous.

As far as I know, it always does, at least for journals! If you get as far as having your paper reviewed, you always get the opportunity to reply. Even if you're rejected.

So this is false.

(There are some journals which allow you to "go for broke" and get an unconditional accept or reject. But these are, to my knowledge, always electoral, and seen as a service to the submitter, in the sense that you avoid the usual 2 or 3 revisions and reviews.)

The fact that reviews are not public (anonymity is okay) is ridiculous.

I don't think so - most research fields are not large (in terms of the number of researchers). As a reviewer, it is likely that you know, or have met, the person(s) who's paper you're reviewing. Anonymity allows you to give bad reviews to people you interact with, with less of the unfortunate side effects; scientists are only human, after all. Part of the point about anonymity is, in fact, to diminish the "popularity contest" aspects you complain about, below!

(There are also sometimes double blind reviews. It's a lot more difficult to hide your identity as an author, but I haven't been able to guess all (or even most) of the blinded papers I've personally reviewed. Yet another point is that I think that a few papers actually publish the reviews along with the paper, and that not all reviews are anonymous.)

Both allow too much room for abuse, and abuse is what I see happening all over when I read about peer review.

Well, you're hardly going to find essays on peer review that are all glowing praise. Like with Tipler, you're going to get a lot of sour grapes.

It might be better than nothing, but clearly something else would be an improvement.

I honestly can't imagine anything that wouldn't be, in some sense, merely a refinement of peer review.

Yet science will always be a popularity contest (even if what's popular is what makes the most careers or money, regardless of validity), so I won't hold my breath for something better. Anyone can publish on the web, so we already have a better system. It's just not popular or well-organized.

Hey, if you want to publish your stuff randomly on the web, go ahead. No one's stopping you. Just don't expect anyone to take it seriously.

And I seriously doubt that the internet that brings us time cube is a better system than peer review.

The problem arises when new ideas are not vetted beyond the title, and instead summarily dismissed.

I think that very rarely, if ever, happens. In fact, I don't think that's a serious problem at all.

(As far as yourself goes, that doesn't count. I'm assuming that you haven't actually submitted any material; merely queried whether they would accept such a submission for consideration, which is not the same.)

It makes perfect sense that I'd say that, given that I asked every major physics journal if they accepted challenges against GR, and they all said no.

Exactly: you're criticizing the peer review system because you don't think you can get through it. Obviously, you are now denigrating peer review to make your own work look better.

 

[BertBonsai]

Funkstar, good stuff, I learned some new things, that some of my understanding is wrong, so thanks.

I don't have a problem with anonymity, I think that's a good idea. But I think all the reviews should ideally be public as they're made, so anyone (and not a select group) can comment. A web interface could easily let anyone separate the wheat from the chaff. For example, you think someone's a crackpot, just check the Ignore User box, or check the Ignore Users My Colleagues Are Ignoring box. I think cosmology in particular is complex enough now, that it's easy for even the experts to make mistakes. (Like my example above, where a major major physics journal editor argued against direct quotes by the authors of Gravitation, and essentially called me a crackpot because he thought they were my words.) Scientists are only human, as you say. A public interface could reveal those mistakes before a paper is rejected due to them.

I know it's hard for people to believe, but not everyone seeks fame or whatever. When major journals told me they aren't interested in challenges to GR (about half of them rudely so, even though I just asked a question politely), I lost interest in journals. There's no sense in me trying to work with someone who has already told me they're not interested; that's a lost cause. The only reason I would go to a journal is if I thought someone else would be interested in the subject matter. Clearly, the majority of "scientists" want to treat established theories are facts, and don't want to see alternatives. I don't see small journals as being much better than writing a blog.

Peer review is probably fine for incremental work. From what I've seen I doubt we'll see any revolutionary work again, at least not in physics. That's a shame, but the world is what it is and I accept that.

 

[Guest254]

I'm a scientist, and I long for papers that surprise me. As do all my colleagues. Surprisingly enough, the majority of papers I've rejected as a reviewer are for reasons such as "no novel content", or "no significantly new result". This idea that scientists are only interested in the well-established theories is completely foreign to me. In fact, I don't of a scientist who holds such a position. All the ones I know are extremely interested in new physics/mathematics/ideas - as long as they are good ideas.

Perhaps your should let the scientific community decide if your work is worth reading - instead of making the decision for them?

 

[BertBonsai]

Perhaps your should let the scientific community decide if your work is worth reading - instead of making the decision for them?

They (major journals) already told me it's not worth reading, based on the subject. They made that decision. What else are you proposing?

 

[Guest254]

I don't believe their rejection (of "revolutionary work") was quite so forthright. Without seeing your e-mail to them and their reply, I'm afraid I can't comment. Perhaps name the journal and/or editors. I will happily e-mail them.

 

[BertBonsai]

I don't believe their rejection (of "revolutionary work") was quite so forthright. Without seeing your e-mail to them and their reply, I'm afraid I can't comment. Perhaps name the journal and/or editors. I will happily e-mail them.

See above; there was no work to consider. All I initially asked was this: "Does your journal consider arguments against general relativity?" This was to each of the editor(s) of the major relevant journals on a list I found somewhere on the web, e.g. Physical Review D. About 15 of them, as I recall. I counted the answer as a "no" if they said yes but only with unscientific requirements, e.g. wouldn't accept a thought experiment as the argument, or would require a replacement theory. (A thought experiment can certainly be valid logic.) The rude ones made it clear they were paying the idea only lip service in any case, to pretend to be scientific. One said it was policy that they don't consider any challenge to Einstein; I thought that was the most honest answer, but thought they should advertise that on their web site. I could look up the emails but I won't betray a confidence.

 

[Guest254]

See above; there was no work to consider. All I initially asked was this: "Does your journal consider arguments against general relativity?" This was to each of the editor(s) of the major relevant journals on a list I found somewhere on the web, e.g. Physical Review D. About 15 of them, as I recall. I counted the answer as a "no" if they said yes but only with unscientific requirements, e.g. wouldn't accept a thought experiment as the argument, or would require a replacement theory. (A thought experiment can certainly be valid logic.) One said it was policy that they don't consider any challenge to Einstein; I thought that was the most honest answer, but thought they should advertise that on their web site. I could look up the emails but I won't betray a confidence.

I'm not willing to accept your paraphrasing I'm afraid. If you want to give me details via a PM, that's fine.

As I mentioned, your claims are diametrically opposed to my experience of the scientific community.

 

[AlphaNumeric]

I (A thought experiment can certainly be valid logic.)

A thought experiment is only of use to convey the logical implications and structure of a model. A thought experiment tells you what the model says should happen, it does not tell you what the universe says happens. If a model is inconsistent then a thought experiment might be used to demonstrate as much. If you can't find a logical inconsistency then its not right as the thing which matters is whether the model and universe agree.

If a model is mathematically/logically consistent then a thought experiment alone won't be able to falsify it. As such you must include a comparison to experiments if you're going down the route of thought experiments, demonstrating the universe is not as the model's prediction in the thought experiment. You haven't done this. Further more SR and GR are mathematically sound. This is easily seen in SR as its the application of a well known and understood mathematical structure to space-time. In GR even black hole singularities are mathematically sound, though the specifics are much much more advanced than most other things in GR. Thus we need to include an experimental comparison somehow, but all possible ways of testing GR within the capability of an individual have been done, GR passed. Thus a paper written by someone outside a research group will not include new experimental data.

So we're left with new models. A new model would have to be demonstrated to work for all known experiments, at the very least. No crank I've ever come across has demonstrated they are aware of the majority of phenomena their 'model' must work for, never mind all of them better than GR. You included.

As Guest says, your paraphrasing is insufficient. You've previously paraphrased me incorrect to my face so lord knows what you're like when paraphrasing people not in the discussion.

 

[Jack_]

A thought experiment is only of use to convey the logical implications and structure of a model. A thought experiment tells you what the model says should happen, it does not tell you what the universe says happens. If a model is inconsistent then a thought experiment might be used to demonstrate as much. If you can't find a logical inconsistency then its not right as the thing which matters is whether the model and universe agree.

If a model is mathematically/logically consistent then a thought experiment alone won't be able to falsify it. As such you must include a comparison to experiments if you're going down the route of thought experiments, demonstrating the universe is not as the model's prediction in the thought experiment. You haven't done this. Further more SR and GR are mathematically sound. This is easily seen in SR as its the application of a well known and understood mathematical structure to space-time. In GR even black hole singularities are mathematically sound, though the specifics are much much more advanced than most other things in GR. Thus we need to include an experimental comparison somehow, but all possible ways of testing GR within the capability of an individual have been done, GR passed. Thus a paper written by someone outside a research group will not include new experimental data.

So we're left with new models. A new model would have to be demonstrated to work for all known experiments, at the very least. No crank I've ever come across has demonstrated they are aware of the majority of phenomena their 'model' must work for, never mind all of them better than GR. You included.

As Guest says, your paraphrasing is insufficient. You've previously paraphrased me incorrect to my face so lord knows what you're like when paraphrasing people not in the discussion.

I would like to see your proof of the logical consistency of SR.

You keep saying it, so I assume you can prove it and know what that means in model theory.

http://en.wikipedia.org/wiki/Gödel's_completeness_theorem

Since you keep harping on the word crank, I assume you and others here will get right to it.

 

[BertBonsai]

As I mentioned, your claims are diametrically opposed to my experience of the scientific community.

You can ask them yourself and get similar responses. It should be a repeatable test. I would ask a slightly different question though, which is "Does your journal accept refutations of GR that consist of only a thought experiment." If no, that journal is not open-minded, not fully scientific. You wouldn't need a copy of my emails.

 

[Jack_]

You can ask them yourself and get similar responses. It should be a repeatable test. I would ask a slightly different question though, which is "Does your journal accept refutations of GR that consist of only a thought experiment." If no, that journal is not open-minded, not fully scientific. You wouldn't need a copy of my emails.

Feel free to say no, but can I see these thought experiments?

 

[BertBonsai]

If a model is inconsistent then a thought experiment might be used to demonstrate as much.

True. Then a thought experiment should be enough to refute GR. But major physics journals are not interested in considering that. Which (to be on topic) seems indicative of a big problem with peer review. And that need not have anything to do with my work.

Further more SR and GR are mathematically sound.

This line of thinking is probably why major physics journals are not interested in considering a challenge to GR. But a thought experiment can show that the reality is more subtle than that. There is room for one to refute the theory.

A new model would have to be demonstrated to work for all known experiments, at the very least.

No new model is required to refute a theory, scientifically speaking.

 

[BertBonsai]

Feel free to say no, but can I see these thought experiments?

Maybe. I've got an open thread here in pseudoscience that may get to that.

 

[Jack_]

Maybe. I've got an open thread here in pseudoscience that may get to that.

Well, if you do I will read it not in a nasty way.

And, unlike cranks around here, I would not tell you you are wrong unless I could prove it with math or an experiment for all to see.


Otherwise, I would say I cannot find a problem or it is beyond me.