How to do Research
An Article of the Modern Survival Guide
We are in the midst of an absolutely magical time — a time when data of all conceivable sorts is widely available and, for the first time in human history, accessible to the greater mass of humanity. Obviously we are making a complete mess of this opportunity, but it is still an enormous opportunity, as long as you know how to conduct research.
Judging by the response to the pandemic, precious few people do. You can be one of those few, though, and all it takes is time. If you don’t have that, you’re screwed. I’ll put that right out front. If you don’t have a few hours every month to look stuff up, then you should go the Associated Press, listen to what an expert is saying on the subject you’re interested in, and trust them.¹ Then go back to living. This is the easiest way to stay informed, and I’d say it’s about 80% effective on average.
If you do have time, first of all, congratulations. It’s one of the three things that really matter, and you have some.² You lucky dog. If you want to use that time doing research, bully for you. This article will tell you how to do it. We’re going to start with a quick statement about what research is and is not, then dive into the realities of conducting research in this wonderful, hazardous world of competing information.
What Research Is (and Isn’t)
Let’s start with a definition, because it’s always fun to have a jumping-off point:
Research is the collection, organization, and analysis of information to increase one’s knowledge of a topic.
We’re going to go through these points really quick, and that should set us on a solid path.
- Collection of information: To properly collect information, the collection process must be done scientifically. The scientific collection of information yields data. The non-scientific collection of information yields anecdotes. The difference is that data can be proven to be reliable, and anecdotes cannot. If you’re not sure what a scientific data collection methodology looks like, you’re hardly alone, and here’s a quick primer.
- Organization of information: To properly organize information, the data you collect must be assessed to determine what type of thing it is. If I am collecting demographic information, for example, I organize data based on sex, gender, age, income, etc., so that I can perform the next step with any degree of relevancy. Professional researchers pay very close attention to data organization, because the categories into which you lump data have a tendency to determine your findings.
- Analysis of information: Once you have organized data, you can conduct a proper analysis of that data. Analysis is the act of rendering data into information — the step required to take facts and turn them into stories. If you do it right, the stories you can tell based on your facts are true stories. If you do it wrong, they aren’t. Here’s a quick summary of data analysis methodologies, for the curious.
These are the three steps of basic research, the process that scientists use to try to learn new things using the scientific method. Now here’s the fun bit: you aren’t likely going to be doing basic research. Basic research is for things like figuring out the Ph of seawater, or taking a survey, or conducting a key-word meta analysis of news sites — small nuggets of information, in other words. You are probably doing research to try to figure out which car is right for you, or which candidate to vote for, or whether Diet Coke has nanobot tracking chips in it (it does, it totally does). You’re probably looking for aggregated information, in other words. Some people would call this applied research; for the purposes of this article I prefer to think of it as amateur research.
I should hasten to add that there is no stigma to this “amateur” label; it just means that most of us aren’t using our research in professional scientific or business settings. I am an amateur researcher on most topics. In my job, I am occasionally a basic or applied researcher, but most of the time in life I’m performing amateur research, trying to tease meaning out of all the information that is out there in the world and on the internet.
To that end, I’ll add two steps to the definition of research: curation and understanding. Here’s what those mean:
- Curation of information: The process of collecting a set of analyses on the information in question and developing methods to remove bad actors. We can’t get away from the reality that some analyses running around out there are either deliberately or accidentally incorrect; therefore we need ways to reliably separate the wheat from the chaff.
- Understanding information: The process of assimilating several analyses into a clear picture of the topic. We’ll discuss this more in a minute, but the one-line version is that you always want to confirm your understanding through multiple valid sources, which is why curation is so important (and dangerous).
To sum up, my understanding of amateur research is that you look up facts, organize those facts, analyze those facts, curate analyses into sets of data you’re pretty confident are “true,” and use those analytic sets to summarize the information into a broad understanding of the topic.
Research doesn’t have to go deep (although it can). Researching to understand the highlights of a topic is just as valid, and often more immediately useful, compared to researching to understand the topic in depth. It all depends on what you’re going to use the research for. To take an example, if I’m going to develop a vaccine then I definitely need to research in depth, almost certainly starting with basic research. If I’m trying to choose which vaccine to take, on the other hand, I don’t necessarily need to know how the vaccine works. I just need to know its efficacy and side effects, and the consequences of not taking it. That’s probably something I can find out through a skim of existing sources.
Now let’s briefly discuss some things that research is not.
- Research is not reading a post on Facebook, much less a meme.
- Research is not reading a magazine article or random website.
- Research is not watching a TV show.
- Research is not watching a YouTube channel.
- Research is not listening to a celebrity.
- For that matter, research is not listening to any one person on any given topic.
These things are, at best, anecdotes, because they represent someone else’s curation and understanding of a topic. Critically, they represent the end state of a process that you did not get to see, and therefore cannot reasonably judge without additional evidence.
To put that another way: don’t believe everything you hear, see, or read. Approach life with the attitude that all information is suspect until verified, and you will rarely be disappointed. If this breaks your optimistic view of the world, tough cookies. There are people out there who make their living lying to you. These careers are called “advertising,” “marketing,” “sales,” “politics,” and “public relations,” to name a few. Don’t believe everything that comes your way, because some people are highly incentivized to ensure that you draw the conclusions they want you to draw on, well, just about every subject imaginable.³
Be wary, or be wrong.
How to Conduct Amateur Research
Ok, so with all of that out of the way, let’s dive into how you can actually conduct your very own amateur research. If we follow the template discussed earlier, we are collecting, organizing, analyzing, curating, and then attempting to understand information.
I’m going to make two assumptions here. First, I’m going to assume that you’re not doing basic research, you’re doing amateur research. You are trying to find the correct answer to a question, sifted from the sea of bad answers surrounding you. That cuts out a good bit of the collection, organization, and analysis aspects, since you aren’t doing research from scratch, you’re trying to understand other people’s basic research.
Second, I’m going to assume you have access to a library, which helps immensely with curation. That point is critically important, because the very first thing we all have to realize when it comes to research is that unless you’re viewing primary sources, you’re viewing someone else’s opinion.
I just introduced a new term, so let’s define primary sources: A “primary source” is firsthand testimony or direct experimental evidence of the topic you are researching. Primary sources include things like witness interviews, logbooks, test results, and verifiable histories written at the time events occurred. They’re important because they represent data in the raw, with only limited interpretation assigned to the events they describe.⁴
Primary sources, unfortunately, are usually hidden behind paywalls called academic journals. There are some exceptions, but that’s the rule. Academic journals exist to provide standards of practice and peer reviews for the primary sources they collect, and as such they are a critically important part of the scientific process because they serve as the first screen that filters out quacks and salesmen from scientific inquiry. A good scientific journal performs this function reliably; a bad one does not. Both good and bad scientific journals exist, so finding a good list of reliable ones is very important. Fortunately, Wikipedia has you pretty well covered.
So, why are libraries important? For two reasons: they have subscriptions to most of the journals or journal aggregation sites you might need, and they have librarians. A librarian’s job is partly to put books on the shelf and track their use, but also to serve as a source of knowledge to point researchers in the direction of valid journals and primary sources. Unless you want to pay for your very own subscription to a scientific journal, go to your local library and ask the librarian about the topic of your interest.⁵
They will most likely point you to an academic journal aggregation site, such as JSTOR, give you some hints about keywords, and leave you to searching. Now, remember the part at the start of the article where I said you needed time above all other things when performing research? This is why. Your next task is to sift through the sea of journal articles that any search will pull up, find the ones that interest you, and read them (again, the librarian may be able to help with this, at which point you most certainly owe them a coffee).⁶
After you find some articles which seem to answer your question, you should try to find other articles either confirming or refuting the information you just uncovered. You also should spend a moment or two verifying the validity of the author(s) and seeing if they are the font of any major controversies. If they are, you might want to double-down on your search for supporting sources.
That all takes time. It takes even more time to do the next step, which is to write down what you have learned. If you don’t do this, you will forget, and you will waste time. Write down the conclusions of at least three articles on the subject you’re researching, and then see if that answers your question. If it doesn’t, read three more and repeat. After six articles or so, you should start to see is one of two things:
- You’ll start to see a clear pattern that shows a conclusion, or,
- You’ll start to see a clear pattern that researchers disagree.
If you see option #1, congratulations, you have probably found a verifiable and defensible answer to your question. If you see option #2, it means that scientific minds probably don’t have a clear conclusion, and the answer is open to interpretation. Most often, articles you read will reference other works, which can be helpful if you start seeing the same sources come up over and over (this might mean there isn’t a lot of information on the topic, or it might mean that source is the definitive source, which can save a lot of time).
For reference, things like global warming and vaccine efficacy fall squarely into camp #1. The overwhelming vast majority of research shows that global warming exists and is human-caused, and the overwhelming vast majority of research shows that vaccines are good health policy. These things are no longer up for reasonable debate, which means that anyone who is debating you on these subjects is subscribing to an unreasonable belief. I have advice on dealing with that situation here, if you’re interested.
On the other hand, things like the ideal tax rate⁷ and the precise morality of a justifiable war fall squarely into camp #2. These are subjects where, if you read three articles, you’ll get four opinions. They’re either too complex or too ambiguous for experts to form a unified opinion, and consequently you should feel free to believe whatever makes the most sense to you, since that’s what all the experts are doing anyway. By all means continue to educate yourself, though, and bear in mind that anyone who says they are absolutely certain about one of these subjects is probably full of shit.
And that’s… it. That’s all you can really do in amateur research. But if you devote an afternoon to amateur researching on an important topic, you’re already doing more than just about all of your peers, so hey, go you.
Finally, a word of warning: sometimes you simply will not have the expertise required to gain an informed opinion from primary sources, and that’s all there is to it. I can read a primary source paper on quantum mechanics, and it simply won’t contribute much to my understanding of the topic because I lack the necessary grounding in the subject. Know your limits, and when you run up against them either work to expand them or use the advice from the following section.
What if You Don’t Have Time?
Right, so what if you don’t have time to go to a library for an afternoon every other week? Let’s be honest, that’s going to be most of us — we either don’t have the time or don’t want to spend it in doing research. What then?
You are now squarely back in the jaws of having to parse through the news media to find expert opinions. Again, the AP is your best bet, but there are other publications that are also pretty good. As it happens, someone has done… drumroll… research on this point! And here it is. If you’re curious, the University of Michigan also has a nice political alignment page to show which way various sites and organizations tend to lean.
Your very best bet, in the absence of time (or expertise) to read primary sources and gain an actual informed opinion, is to rely on expert opinions from a non-partisan, reliable news organization. This is not research, but it is information, and it is usually reliable. News articles give you more information to help you verify their authenticity than TV or radio shows, as a general rule. Some things to watch for:
- Does an article list its author? If it doesn’t, that’s a red flag. Valid publications tell you whose analysis you’re reading.
- Does an article have a publication date? If it doesn’t, again, red flag. If it does, is it reasonably close to the current date? If not, that’s also a red flag. News has a shelf life.
- Does the article list sources and identify the experts it references? If it doesn’t, serious red flag. If it does, you can check to see whether these sources are reliable or biased. While it is a logical fallacy to disbelieve someone just because of who they are, it is not fallacious to be suspicious of known bad actors and seek additional verification of their statements.
- Does the article use another news article as a reference? If it does, be suspicious. The information might be right, but it also might be a lazy copy/paste without verification. News sites sometime copy from each other and accidentally spread bad information.
- Does the article seem to reflect reality? E.g., if an article states definitively that COVID is over, but your doctor states that hospitals are full of COVID patients, and someone you know has COVID, the most reasonable conclusion is that the article was wrong, not that reality is wrong.
Using these points, and understanding the biases of various news sources, you should curate the news you listen to. For example, I simply do not listen to Fox News, and I do not consider any opinion aired by a Fox News commentator to be worthwhile. Not because they are correct or incorrect on any particular point, but because the overall bias of their organization is such that I know they are pushing an agenda with each story. The same goes for MSNBC, by the by.
Doubt is Good
Let’s end on another word of warning: no matter whether you read primary sources or rely on expert opinions, you should always maintain a degree of doubt regarding conclusions. Note that this doesn’t have to be a high degree of doubt.
For example, I’m like 99% sure Dr. Fauci is giving me the best information he has on COVID. But that doesn’t mean he’s always right, it just means that I can have high confidence that he’s right. It also doesn’t mean that I immediately curate him out of my information intake if he’s wrong; it simply means that I should take everything I see and hear with a grain of salt.
Similarly, if I’ve spent all day reading primary sources, I might be 80% certain that I’ve got good information. But I have to acknowledge that a day’s worth of research isn’t the same thing as an expert opinion. If I think one thing, and the acknowledged experts are saying something else, it’s more likely that I am wrong and they are right than it is that they are wrong and I am right. An afternoon on JSTOR is not the same thing as a doctorate. My next move is to understand why I’m wrong, and try to broaden my understanding that way.
It’s a wild, crazy, baffling universe out there, and it is under no obligation to make sense to you or anyone else. Maintaining a thread of doubt is a very, very healthy way to approach anything so fraught with peril as attempting to impost order on the cosmos (or, to put that another way, conduct research). Most of life is about balancing risk vs. reward, knowledge vs. uncertainty, and then moving forward on our path; research is no different.
When in doubt, stay that way. At least until you gain enough of an understanding to have at least a couple of nines of confidence on a subject. Doing amateur research is a great way to get there, but only if you’re careful, and only if you can acknowledge your own limits. As usual, know thyself first, then try to understand the rest.
¹That’s the AP for short. Not your local news channel. Not a national news channel. Not, under any circumstances, Fox News. If you’re a Fox News junkie, you are actively being misled on a daily basis, and I’m sorry, that’s just the way the cookie crumbles. The Associated Press is the one you want. AP news is direct from the reporter, and the closest you’re going to get to an unbiased report. https://apnews.com/
²The other two being money and interpersonal relationships.
³Which leads to the natural question, should you trust me? For fuck’s sake, NO. I’m just some person on Medium. My name isn’t really Allen Faulton, for all you know I don’t even look like my picture; hell, I might be several authors writing under a pen name. I’m intentionally anonymous, for Pete’s sake. Verify everything I tell you through some other source.
⁴To put that in perspective, a news program is not a primary source. Nor is the Bible, or any YouTube video you care to name. A primary source is usually a scholarly article or essay, and has a title like “Understanding the Role of Raw Potatoes in the Irish Potato Famine of 1845,” as opposed to the average YouTube video which has a title like “Sixteen Things the Illuminati Don’t Want You to Know!!!” Generally speaking, the more boring the title, the more scientific the information. It’s a rough measure, true, but a good rule of thumb that a good scientific article is going to tell you exactly what you will get out of it in the title, as opposed to leading you on.
⁵And do not support any politician who wants to cut library funding, because what they’re really cutting is your access to information.
⁶There’s a classic problem in the realm of information verification that the Romans summed up as “Quis custodiet ipsos custodes?” We can loosely translate that as “Who guards the guards?” or “Who watches the watchmen?” and it highlights the problem that anytime you try to verify data, you have to then verify the source of verification, and then verify the source of the verification of the verification, and so on until it’s turtles all the way down. That being said, having a couple of degrees of validity, in this case academic journals and a trained, dispassionate professional to point you to good journals, should be enough to result in a pretty good degree of confidence.
⁷Notice that I said “ideal,” as opposed to “whether or not taxes should exist.” The experts are solidly on the side of “yes, they should.”
