StatCastsWe’ve delayed a lot of our work on this site. There are good reasons. Rex and I have been working hard on some projects that are being launched in the near future. Meanwhile, you can hear what half of us are saying at stuffdavidsays.com.
Sometimes funny, possibly clear talk of analytical stuff read more »
One thing we want to do at StatCasts.com is train in bite-sized portions. Here’s a smallish bite: sensitivity and specificity.
Say you have an idea for a new product. Some BS features for the product might be:
- seize innovative bandwidth
- reintermediate seamless metrics
- repurpose magnetic paradigms
- synergize cross-media systems
We all tend to have BS detectors when people talk like this around us. But hey, you’re the one pitching, maybe you really mean it. If I were a foolish investor, I might give you some dollars to deliver some of these features. I particularly like the idea of synergizing cross-media systems, so I’ll give you a million dollars to do that.
So, now you have a million dollars from a foolish rich person. You’ve got a real problem, though. What on earth do you mean by synergizing cross-media systems? That is a very vague claim to make. So, you devise a test to see whether you’ve done just that. How good is this test?
Finally, the punch line: use sensitivity and specificity to measure the value of your test.
When you’re testing something, or you have a model of the real world, or you are using some sort of math that might not always be 100% accurate, you can measure four pieces of information:
- true positives: when a test/model/measure says something is true, and it actually is.
- false positives: when a test/model/measure says something is true, but it actually is not.
- true negatives: when a test/model/measure says something is false, and it actually is.
- false negatives: when a test/model/measure says something is false, but it actually is not.
Now, in our example, we might be sued for breach of contract if we say we synergized cross-media systems, but we actually did not. So, we don’t want to have many false positives. Also, we want to show that our new feature is pretty snazzy, so if we actually did synergize cross-media systems, we want credit for it, so we want to limit our false negatives too.
Our first measure, sensitivity, looks at how accurate we are with positives: the number of times we predicted a true positive over all actual positives. So, a sensitivity of 20% would show that 4 times out of 5, when we said that we had synergized cross-media systems, we actually did not. That’s probably a lawsuit on its way. A sensitivity of 95% would be much better: we only exaggerated our position 5% of the time.
Our next measure, specificity, looks at how accurate we are with negatives: the number of times we predicted a true negative and it actually was. This is our measure of how much money we are leaving on the table. A specificity of 20% would mean that we were 5 times more useful than we though we were. A specificity of 95% means that we are pretty good at beating our own chests when we did some good.
You might want to look at these gists. They describe how these four numbers (true positives, false positives, true negatives, and false negatives) relate to each other.
Now, imagine that this wasn’t a silly example. What if we were testing for cancer or mortality in newborns? There are many things in our life where we want to know how much risk we are living with.
In a world where I’m testing for something bad, I don’t want to have many false negatives. If I have to tradeoff between sensitivity and specificity, I’d choose specificity. I don’t want to miss the warning signs of life.
In a world where I’m testing for something good (ostensibly this is synergized cross-media systems), I am more concerned with sensitivity. I don’t want to worry about false positives. My credibility is at stake. In this case, I’d choose sensitivity over specificity.
It’s interesting to know that in the real world, profit is usually half of the equation. A highly specific test will catch as many infants as possible with risk of serious health issues and a poorly sensitive test will have us working on a lot of healthy babies for problems they don’t have. A highly sensitive tests for delivering good features will make sure our credibility is intact and a poorly specific test will keep us from taking credit for all the good we’ve done.
If we’re really smart, we might be able to come up with tests or models with both a high sensitivity and a high specificity. This is exactly what Daphne Koller did with PhysiScore vs. the traditional APGAR. Both tests are for predicting a high risk of mortality or serious physical ailment in newborn babies.
The traditional APGAR has a sensitivity of 43% and a specificity of 95%. That means that 5% of all babies that the APGAR says is fine actually aren’t. It also means that 57% of all babies that the APGAR says have problems are fine. That could be a lot of unnecessary hours for babies in the ICU.
The PhysiScore, on the other hand, has a sensitivity of 86% and a specificity of 96%. That means that only 4% of babies tested have missed a risk of some serious health issues. It also means that only 14% of babies receiving additional attention for serious health issues are actually fine. Hospital administrators must be relieved that their resources are managed fairly well.
So, that’s the gist of things. I’m not sure if I boiled things down, but hopefully this is a fairly-readable explanation of these two concepts.
I’ve been thinking for a few days about who needs analytics. Google has been improving its Google Prediction API. Fathom is being designed as an analytics platform (working title: Awesome Sauce Analytics Platform). Alpine Miner is an up and coming company that’s created incredible ROI for some data mining projects done by a client of mine. Ubalo has some incredible ideas that could dwarf any offerings I’ve considered in this arena.
Which leads me to ask: what is the use case for analytics?
The ideas from Competing on Analytics suggest that companies can evolve a competitive strategic advantage with analytics. They also say that good analytics evolve, rather than just appear in an organization. It is as much an organizational behavior problem as it is a technical problem.
This is what makes this problem a little sticky. What are people ready for today?
As someone who gives away his free time to develop analytic systems and teach people how to use analytics, I really want to know the answer to this question. I have my ideas: clinical decision support for medical offices, customer discovery tools for product or service owners, business value optimization tools.
What are your ideas? If you want to take a little initiative today, send me a 3+ paragraph email describing the insight you wish you had in your organization. You can send it directly to me. At a minimum, I’ll send you a $25 gift card to Amazon.com. Additionally, your ideas could be the seed of some open source code I write or some content I deliver at this blog. This is a semi-limited offer. There is a budget I have for giveaways. It’s kind of a soft budget, however, so don’t be bashful with your ideas for making our world better with some well-applied analytics.
I had a fascinating conversation with a veterinarian today. He works for the state preventing outbreaks of disease. There are few issues in Utah, thank goodness, but he told a story that captured my imagination. I wanted to understand the analysis these people do to prevent outbreaks of disease.
One story he told was of a local horse stable that had an outbreak of Vesicular Stomatitis (VS). This is a disease that looks somewhat like foot and mouth disease: livestock catch this virus which forms sores around their hooves, mouths, and other areas of their bodies that are painful and prevent them from eating. The animals lose weight and without treatment, can die. VS is a much slower-moving virus than foot and mouth disease, which makes it an economically devastating disease.
This is part of what fascinated me. If a herd were to contract foot and mouth disease, the whole herd could be wiped out quickly, the spread of the disease very difficult to stop. In the case of VS, the danger is in having the disease stick around for round after round of outbreak. VS can be carried by insects for up to three weeks after becoming host to the disease.
If this doesn’t make a lot of sense, don’t worry about it. It didn’t to me either. This is what I think I know now about the spread of viruses.
- A virus is merely a packet of DNA or RNA.
- A virus is unable to survive without a host.
- In order for a virus to survive, it must be exposed to an animal.
- Susceptible cells must allow the virus entry into the cells.
- The virus must replicate inside of the cell by taking over the cell’s reproductive functions
- The cell must shed the virus in order for the process to continue.
- Defenses against viruses start by limiting exposure to the virus.
- Cell susceptibility is whether a cell will allow viral entry.
- Most viruses have very narrow target cells, meaning most animal cells are not susceptible to most viruses.
- Cell permissibility is when a cell allows a virus to take over its reproductive system.
- Immune systems combat viral replication.
So, this is a bit of a story, but understanding what is going on when a virus is spread helps me understand what the analysts are thinking when they quarantine a stable. When VS is found in a stable, it must be quarantined for 21 days after the last animal shows signs of the disease. My question was how they came up with 21 days.
My first thought was they must have measured VS’s survivability outside of a host, taken a number of standard deviations from the mean of their observations, and come up with 21 days. But VS can only survive without a host for 1 to 2 hours.
It turns out that VS lives in insects after it exits larger animals. It kills most insects it invades, but the grasshopper can live for quite some time with the virus. The animals still need to be exposed to the grasshoppers with VS, which comes from grasshoppers in their feed. Grasshoppers have been observed to live three weeks after being caught in a bale of hay, which may be the reason why a VS quarantine lasts so long.
So, why did I relate this story about Vesicular Stomatitis? We live in a complex world where decisions are made every day about how to keep everyone safe and alive. For a layperson to understand what’s going on, it may take just a little time with Google and a little logic to follow the numbers. In this case, I didn’t find typical statistical models to describe the policy, just a few basic observations and a watered-down understanding of how viruses work.
The other day, Rex and I recorded our first video podcast for this site. Upon review, our format needed some work. We’ve come up with a better style of covering our topics. Think punctuated. That’s the picture. It’ll be time-boxed segments in quick succession. We’ll get to the facts quickly, and let you move on with your day. So, we’re down from 1-hour format to a 15-minute format. Believe me, you’ll thank us for the refactoring. Anyway, I thought the delay in our release deserved a public explanation.
I downloaded Gephi 0.8 today in order to talk about it on our pod call (will be published later tonight or in the morning). I thought I’d add a more robust review here since we won’t spend more than a few minutes talking about it on the call.
Summary: Gephi, an open-source graph visualization tool, has a lot of promise despite it’s quirks.
Gephi is all about graph visualization. You can visualize graph data, filter it, re-order it with various algorithms, clean it up, and then export it as a PDF or SVG file. It can be mesmerizing to play with some of the data. I used the Les Miserables data that comes with the package and found it predictable but interesting anyway. I found that this was a good place to try out the slides in the Quick Start guide. After playing with zoom, drag, node color schemes, and various organization algorithms, I felt like I could use Gephi for many other applications.
The new version of Gephi is mainly about localization and the email spigot. Localization isn’t an issue with any of my data right now, and I didn’t take a look at that. I did try to use their email spigot, but couldn’t get it configured right so I gave up.
While searching for some more configuration help on the email spigot, I found a social network import plugin that allows me to import Twitter and New York Times articles. It took me a while to figure out how to get that plugin installed, I couldn’t find any information on that in the application or online. The Plugin menu option is empty, but the Tools -> Plugin option does what you’d expect. This plugin was fun to play with and some of the screenshots I’ve included use this plugin.
Exporting the visualizations as SVG was a little quirky. My labels were dropped and some of my formatting didn’t carry over. Also, I couldn’t figure out how to get the preview working.
Overall, the architecture of the package is really open. There are plugins that the community can develop and share. There are open standards for importing and exporting edge lists. There is an active community behind the system.
I don’t know if it’s the Java, or if there are some memory issues with Gephi, but I had to kill the process a couple of times when it took up all of my laptop’s memory. The first time was on a randomly-generated graph with 5,000 nodes, and another when trying to import a New York Times query of recent articles on the shutdown. I’m not sure how many returned results I was trying to process when that crapped out on me.
Going forward, I think my main use case will be to export graph data from other applications and then import them either via CSV or RDF. I think even the social network imports would be easier to manage if they were coming from a database I created before trying to visualize them. There is a larger list of supported formats found here. Alternatively, you can import data from MySQL, PostgreSQL, or even SQL Server.
The RDF support was just added, and I haven’t tested that out yet. Possibly I’ll do that when Ben Lavender and I do a pod call to review his work with Dydra and Semantic data. I’ll get to that as soon as he finishes saving data.gov (no kidding).
Gephi is released under the GNU Affredo GPL version 3. It is a Java-based program that runs on Windows, Linux, and OS X.
Not nearly as stunning as some of the images found on the Gephi site, here are a few of my own from just a few minutes’ worth of tinkering:
With all the quirks that I found, I’m not disappointed. This is an Alpha version 0.8. I’ll be very happy to continue to follow Gephi as it continues to mature.
This, from Richard Brautigan:
One April morning in the sixth grade, we became, first by accident and then by premeditation, trout fishing in America terrorists.
We were always being called in before the principal for daring and mischievous deeds. The principal was a young man and a genius in the way he handled us. One April morning we were standing around in the play yard, acting as if it were a huge open-air poolhall with the first-graders coming and going like poolballs. We were all bored with the prospect of another day’s school, studying Cuba.
One of us had a piece of white chalk and as a first-grader went walking by, the one of us absentmindedly wrote “Trout fishing in America” on the back of the first-grader. The first-grader strained around, trying to read what was written on his back, but he couldn’t see what it was, so he shrugged his shoulders and went off to play on the swings.
We watched the first-grader walk away with “Trout fishing in America” written on his back. It looked good and seemed quite natural and pleasing to the eye that a first-grader should have “Trout fishing in America” written in chalk on his back.
The next time I saw a first-grader, I borrowed my friend’s piece of chalk and said, “First-grader, you’re wanted over here.”
The first-grader came over to me and I said, “Turn around.”
The first-grader turned around and I wrote “Trout fishing in America” on his back. It looked even better on the second first-grader. We couldn’t help but admire it. “Trout fishing in America.” It certainly did add something to the first-graders. It completed them and gave them a kind of class
“It really looks good, doesn’t it?”
“Yeah.”
“There are a lot more first-graders over there by the monkey-bars.”
“Yeah. ”
“Lets get some more chalk.”
“Sure.”
We all got hold of chalk and later in the day, by the end of lunch period, almost all of the first-graders had “Trout fishing in America” written on their backs, girls included.
Complaints began arriving at the principal’s office from the first-grade teachers. One of the complaints was in the form of a little girl.
“Miss Robins sent me, ” she said to the principal. “She told me to have you look at this.”
“Look at what?” the principal said, staring at the empty child.
“At my back, ” she said.
The little girl turned around and the principal read aloud, “Trout fishing in America.”
“Who did this?” the principal said.
That gang of sixth-graders,” she said. “The bad ones.”
You look good and seem quite natural and pleasing to the eye with a statcasts.com T-shirt on your back. Rex put these together. Ask, and we’ll send you one. At least until we blow our T-shirt budget.
What a day.
I got up today thinking about how I could assist more people with my machine learning and analytical work. I had ideas about better data visualization for science fair projects and public health initiatives supported by evidence-based research.
I then contacted Dave Brady (@dbrady) about some data that he used for a blog article, Terrible Beauty. Dave started giving me advice for blogging like I talk in real life–relaxed and authentic.
Next, I started going over things with Rex Griffiths (@griffpup) about some projects we’re working on. I proposed we start recording some of our conversations and post them here. We laugh and joke about all things data and visualization and usability. He agreed. We’ll be speaking next week about Dave Brady’s Sex Offender data, ways to understand, visualize, and explore it. Expect some laughs and an interesting time. We wanted to have more time to play with the data before we produced our first video.
The rest of my day was spent contacting smart people I’d like to have participate here, setting up details for the site, reading Crush It! to make sure I’ve got a well-rounded perspective on this stuff.
This blog was born in a day. Or, delivered in a day. I guess I’ve been pregnant with this for decades.
Here, you can find Rex Griffiths (@griffpup), David Richards (@drichards), and others talk about data and statistics and useful data representations and funny things going on around us. At least, we’ll make fun of a few things, talk about whatever comes to mind, and possible teach something along the way.
Polls
Loading ...